we all know that Intel's got a problem with the 13900 K and the 14900 K being unstable. It's been months. Do we really blame Gamers for being impatient? At this point, I'm not so sure that this is really just a boost motherboard voltage clock problem. I'm starting to think there's something deeper wrong with the chips. I mean, it's Thursday, all I want to do is just play Dwarf Fortress of ridiculously high single-thread clock speeds. I get intermittent problems. They are the worst to troubleshoot. But it occurred to me, we have another source of crashes, the games themselves. Loads of games have Telemetry in them to track their crashes, a data source from the crashes. I mean, so I started digging into that, and what I found will shock you. I'm much less sure that we can fix this with a bioer microcode update because I started digging into this, and I went on a journey, and now, now, now I'm troubled. Come with me as we attempt a

level one text armchair diagnostic. It's a level one diagnostic, but from the armchair, right? So take that with a bit of a grain of salt. [Music] I needed data from thousands of systems. How am I going to do that? Well, did you know when you play a game that most of the time it logs usage data? A lot of the times one set of analytics goes to the marketing team, like how long you're going to play the game, when do you open it, what all is going on with it, but there's another set of data that goes to the dev team around when the game crashes. The game developers will get a crash report. So I reached out to my contact list and I found two different people from two different games that were willing to give me access to their crash database so I could look around for interesting stuff. Okay, full disclosure, it took a little bit of finagling and convincing because I said, Hey, I think this weird thing is going on, and they said, Nah, that seems weird.

I was like, No, the errors are probably not what you think they are, and they said, Hm, you might be right. See, the problem with the 14900K and the 1300K and other CPUs in the 13th and 14th generation are crashing is that they're not super consistent. I needed to know system configuration, Play Time, crash rate. I need to know about the population of just people playing the game without errors in order to make suppositions, and these databases are large and they're also rolling. Neither company really hangs on to untagged or untriaged events more than about 3 months, and some there's some exceptions to that for really outliers, but mostly it's a three-month time window. The instability is not particular as well. It's not like the ancient problem, you know, the Pentium foof bug. This is a hardware problem which actually led to Intel creating their whole microcode system so they could patch CPU errata and software even when you had a hardware problem. In this case, the errors are

all over the place. You're even getting GPU errors. GPU errors, yeah, you know the out of vram error that has now become infamous. A lot of armchair experts and forums saying, Hey, your game sucks. You've got a vram leak. It's got to be you. And that was actually not the case. It is actually a really common error with Intel CPUs that have this problem out of vram error when you're not actually out of vram. And these crashes aren't so bad that, you know, the game totally crashes always, though we know that users are experiencing worse crashes, things like blue screens and that sort of thing. But we can't get a crash report if the computer blue screens or at least most of these games are not set up to be able to do that. So there is a little bit of survivorship bias in all the stuff that I'm talking about today. So again, grain of salt. There's also one operation that stresses the CPU in a particular way, that is decompression. It is a common

feature of game and check out this statement from Ood regarding decompression failures. When we're talking about their game Tools Red V game tools, yeah, check this out. This is an Intel-specific thing. Now, this decompression library is ubiquitous, it's well-used, it's probably nearly bug-free. This is a hardware issue that's creating this, and you can see that from this bulletin. This article suggests some BIOS changes and some clock changes that'll help users mitigate, and most of these changes are around power and clock settings. Okay, fine, maybe motherboard manufacturers pushed things too far. Okay, maybe we'll come back to that, but I've got some big databases here. Let's see if I can find this particular error, the Udle error. How many decompression errors are there logged in their respective game databases for the past 90 days? Okay, the answer to that is 1,584. How many of those are Intel 13th or 14th generation decompression errors? 1,431. What's the next most high CPU with an error? It's an i79750h with just 11.

instances of decompression failure. What about AMD CPUs? Maybe AMD. I only saw four entries from any AMD CPU, and you know, that's pretty awesome. Don't assume that that means that AMD CPUs are better. I mean, I think they are. In this particular case, they're not experiencing these kinds of issues, but there could be way less AMD CPUs in this population. We don't, we don't know. I decided to check out those things and check out the distribution. And then there's handhelds to worry about. Almost all handhelds are AMD CPUs. Not all of them, but a lot of them. Okay, so what's the breakdown? 'Cause, you know, if we're only working with five players on AMD systems, so like, we, we need to know, right? Okay, the breakdown between Intel CPUs and AMD CPUs in the crash database was about 70/30 in favor of Intel, which suggests something about 70% of players are using Intel, 60% on Nvidia, and the rest were among AMD and literally everything else. This is also another instance where it's weird.

misleading because you know this is game. Telemetry data from Windows, but sometimes this is actually game. Telemetry data from Linux, and this reporting tool reports, uh, GPUs grouped together in the Linux scenario differently than it does on Windows, and so you end up with some percentage of the 40% of users actually being Linux users that could be using an Nvidia or an AMD CPU. So, 60% really is the floor for NVIDIA GPUs in this scenario, which is weird. I know it doesn't matter for this video. I didn't, I, I chose not to clean up the data for GPU distribution. I just wondered something about the population. So we get it, the data here is not super clean, not ideal, and this is also why I think this is flown under the radar a little bit. Fortunately, the rate of errors per unique player was also not super high. There was a small number of people, less than 200, that were suffering tremendously. Don't get me wrong, but like the PC World article where they had

actually SWA their CPUs. I think those 200 people actually would be better off just swapping their CPUs, just getting an RMA. No amount of microcode is going to help those users. I think for the problem users, I used those as a springboard to look at what other errors their systems had logged because they've logged a lot of them. And I saw a lot of IO errors or so IO errors. The game, these are nvme errors, but the game doesn't log that. The Telemetry tool doesn't look at system errors as a thing that we should log. It just says it can't retrieve a game asset, which is not super uncommon. Like an nvme error is not really a super uncommon thing. And once again, the population of all systems that had any IO error grouped by CPU, the percent of Intel 13900 K and 14900 K were definitely overrepresented. They had a much higher uh rate of errors in all errors for those CPUs versus any other CPU. It's very odd. The other reason I say that this is odd is because when

we're talking about I/O and PCIe devices. Those are a different clock domain on the CPU. Like if every little overclock instability would result in file system corruption, you'd be reinstalling Windows a lot. And yeah, sure, a severe overclock can cause problems, but most of the time you're not going to corrupt your SSD from a bad overclock. If a gamer experienced 10 or more errors in the last 90 days, you could bet at least one of them was an I/O error of some type in the game where it couldn't retrieve an asset. At least in the filtered cohort of users experiencing four or more errors in 90 days, there were no I/O errors of this type from AMD systems at all in the last 90 days. There was, you know, maybe there aren't enough AMD systems. I mean, I really, really dug through both game companies' databases, and I only found four errors that could possibly be attributed to an I/O error or an I/O error in that sort of a context. That's not really enough data.

in my opinion, to make any sweeping conclusions, but this I/O error mostly does seem confined to people that are having real earnest hardware errors. I decided to window the data down to people with at least 20 gigs of GPU memory next because it's just not possible for either one of these games to need more than that much. Intel has also accepted that this error likely stems from instability specific to the problems we're talking about with 13th and 14th generation enthusiast CPUs, so you don't have to take my word for it that this error is probably not actually related to out of VRAM in these specific games. So selecting for everybody that had a 390, 490, 7900 XT, 7900 XTX, which is actually surprisingly difficult because the Telemetry tool doesn't always record the correct GPU when you have more than one GPU and iGPU plus everything else. The oldest error that I found was one from more than six months ago, which was logged and the dev team had spent a

lot of time on so it wasn't cycled out of the system but one user had experienced uh about two crashes um or about one crash for every two hours of playtime. That seems like a lot trying to find a cohort of players that play regularly and then grouping them by CPU and crash rate showed that at least for this game AMD players had fewer crashes than 13900 K and 14900 K players per unit of play time. At least when we take into account the crash rate if the crash rate is consistent and also the 12900 K was shockingly better. The 12900 K was about equivalent for AMD CPUs and about equivalent to everything that was in a 13th or 14th gen CPU which is interesting. It's also interesting that you can't just check 13th gen CPUs in general because some 13th gen CPUs are actually rebadged 12th gen CPUs so you can't just say all 13th gen CPUs you have to look at specific ones so for.

this video, I tried to stick to just 13900 K, KS, KF, and 4900 K, ksk, KF. So, in other words, just because it says it's Intel 13th gen on the box, doesn't actually mean it's Intel 13th gen. I mean, that's true for the K series CPUs, it is what it says, but OEM processors and variants, like, it just depends. So, based on this data and a lot of data that I'm not commenting on, I would think around 20 to 30% of players with one of those two CPUs have experienced at least one crash that can be attributed to the CPU or the motherboard during their lifetime of play. I think that the more that these CPUs are used, the rate of error is increasing over time. At least, I spent a lot of time trying to find a Smoking Gun here that would look like that, and it seems like there's some data that fits that supposition, but the analysis of that is very hard because the data is not organized in a way to really be

conducive to that kind of searching for the players. That play consistently. The number of errors that they are encountering with our systems are definitely increasing over time for those two CPUs. I can say that at least and the intermittent problems. They're the worst to troubleshoot. It's like there's intermittent contact in the input polarizers. Did somebody sticking iron filing in the CPU vat. Is it some sort of contaminant. I don't know. It's a leathery burnt bacon enigma wrapped in a mystery. Uh maybe the problem here is self-inflicted. I mean that's certainly some of the messaging from Intel users are overclocking their system or motherboard makers are overclocking the motherboard. That's got to be the source of the problem right. Well let's use our brain and think about counter examples of that oh look on that Quest. I did actually find something interesting. It turns out the 13900 K and the 14900 K do have a place in the data center. They're they're not just for gaming CPUs. They're

also for gaming servers. And guess what? In the data center, most systems are deployed with a motherboard that is based around the W680 chipset, a W series chipset, not a Z-series chipset. Z-series chipsets are for overclocking, totally different motherboard, totally different power phase designs. Maybe Intel is just rebadging a Z series, and they've been super lazy about it, but I don't think so, 'cause it's designed for Xeon. I mean, these are alternative motherboards for LGA 1700 that is different than their desktop class counterparts. And so, one might expect that the CPUs would behave differently on these much more conservative motherboards, because these motherboards are designed to operate well within the specifications of the CPU. So, I said about gathering crash data from thousands of systems already deployed in the data center around the W series chipsets. And here's a screenshot of one of the systems that had crashed hard and rebooted. Notice a few things on this screenshot. One, the insanely conservative DDR5 memory speed. This may

be a result of automatic crashing and the Asus bios backing off a memory speed. Two Asus w680 chipset like I was saying. I've seen similar crash screens from the super micro w680 boards, which is the other game provider they use. Super micro one of them uses Asus and the Crash rate is pretty similar between these two. I mean the reason these are uses because of the insanely High single thread clock speed and for Game servers it turns out it's actually useful. W680 was created to go along with motherboards designed for maximum stability. Neither Asus nor super micro motherboards really support giving tons of extra power to the CPU or doing insane overclocking for things on a desktop. So I really don't think both Asus and super micro have colored really far outside the lines on this motherboard. And I really don't think Super Micro or Asus have just lazily copy-paste the voltage settings from their desktop motherboards to the server-class motherboards. Fully 50% of the systems deployed for both companies with either one of these processors to

within one percentage point are experiencing the same stability issues. Even disabling ecores has not fully resolved the issue. For one of these companies, the error rate also seems to be going up over time on the server side as well. Oh, and get this, it gets better. One of the companies is negotiating for another $100,000 of servers, and this time a line item popped up in the proposal from the provider. The Intel systems are more than $1,000 more expensive than their equivalent AMD counterparts, and they let me insert myself in the negotiation process. I was able to get on the phone and talk to the data center provider. Oh boy, they dropped a lot of interesting nuggets. The way that the sales work here is you buy a system and it gets deployed in the data center, but if there's an issue, it's a support contract that the owner of the system never actually really touches it, at least until it's time to be retired. So I asked, Okay, why is the support cost on these Intel systems so much higher?

than it was for roughly the same systems that were bought in 2023. I mean there's a couple minor Hardware changes, but similar systems, and they said, and I quote, support incidents have been unusually high for that configuration. So recently we've had to update the BIOS, disable ecores, or do CPU swaps to get the issues resolved, and we're not sure that the issues are fully resolved. So we are charging a support premium for those systems right now. Huh, isn't that interesting, $1,000 extra that wasn't there six months ago. So I asked, is that normal? Have you had a lot of s like what's going on? And they said we had really good luck with the 12900 K based systems that we had, and we always had good luck with the Xeon. Something just isn't right with the 13900 K and 14900 K. We already replaced a lot of customer 13 900k systems with 14 900k systems, and the issues don't seem to be fully resolved. We've been

steering customers toward 7950 X systems. Instead, they're almost always faster anyway. Neat, we talking one of the game developers about this. They said, I think I'm going to lose about $100,000 in Lost players from their multiplayer server crashes. Yeah, if you were a game player, I'd you'd be frustrated too. Makes sense. This game is terrible. It just crashes all the time, and I get there. There's a certain cohort of you watching this video out there that are going to say, Oh, this is what you get for not going with Zeon. But this wasn't a problem with 12th or 11th or 10th or 9th generation Intel Desktop CPUs. For this use case, relatively high single-core performance and relatively low cost. It makes sense for Game servers, even if you trade minor stability issues for the cost and the performance. But the stability tradeoff for 13th to 14th generation, at least in these scenarios, at least for the last six months to a year for them, I guess.

has been terrible for at least 6 months or so. I tried to prod them a little bit on what the data center was experiencing from, you know, messaging from Intel and support, and they really didn't seem like they were getting much support. Beyond just here having a tray of extra CPUs, swap CPUs and hope for the best. It really doesn't add up, especially when you consider W680 and its more conservative power and clock targets. That got me to thinking, what is Intel telling large system integrators like Dell, HP, and Lenovo? So I reached out to contacts that I have inside of those companies, which required a little bit more intrigue on my part. And I'm not really sure that I got good Intel from those companies, but the Intel that I did get said that, well, you can expect between 10 and 25% of CPUs have a problem or are marginal in some way. And we're not really sure what the root issue is. Do they say clearly if it is a motherboard problem or an Intel CPU?

problem. The messaging seems to be that it's a little column A and a little column B, even for OEM systems, which also, like W680, tend to be a little more conservative in power and clock performance configurations, even when you're buying a K, not necessarily when you're buying a non-K. Based on what I saw from game telemetry and game server crash data, I would say that 10 to 25% is much less than I would have guessed. I would have guessed that about half of these CPUs have some type of issue, with some clearly a lot worse than others. Is that attributable to power on time or how much they've been used or some overclock attempt? I don't know. I don't know. In terms of specifics and crashing, the two populations of systems were a little different. The one provider uses dual DIM configurations, and that seemed to suffer a lot. The single DIM configurations seem to work a little better. 2 48 gig DIMs versus 4 32 gig DIMs, opt for 2 48 gig DIMs every

time. The most stable configuration for testing YC cruncher 24 hours at a time on the Linux side was definitely configuring a Max multiplier of 53 and configuring the DDR5 speed to 4200 for the 4 DIMM configuration. 5200 was fine for single DIMM but 4200, you know, that's it technically. Yes, the W680 does support XMP, but it's not recommended, especially in a game server context. So, in order to find failures, we would use a combination of decompression tests for the phonics test suite with automation and Y cruncher 'cause Y cruncher is always pretty great. And a lot of the time, the failures were just random. The core was random, everything else is random. There were a handful of machines that would have specific failures. We had one machine that when doing S&T testing, it would always fail at the S&T test almost immediately, no matter what. It's kind of wild, but mostly the failures were random. And sometimes Y cruncher would pass, but Compress -7 zip and PTs.

would fail was very interesting. Oh, and in case you're wondering, one of the first things that I did in setting up both machines for both providers was to fully update the BIOS to whatever was current as of June 25th, 2024, which were quite a lot of BIOS updates, and that did help, but did not fully resolve the issues. In the end, DDR5 4200 and disabling ECores were the most drastic things that positively impacted stability, but mostly disabling ECores didn't have as much impact as making the memory run dog slow. One last thing I'll leave you with on the Linux servers because the way that one of the particular games works. It logs a number of Game World ticks per second, and it turned out that I could use this to work backwards from crash events. Sometimes when a system crashes, the tick rate in the world would drop to about 50% of normal, and this I couldn't attribute this to EC cor problems. I couldn't attribute this to power core, uh, power problems. See, the power is actually

logged at the socket by smart power strips in the data center, which is very useful. I couldn't figure out anything to correlate this to like thermal monitoring because we log that. Just there wasn't anything. Just every now and then the CPU gets miserably slow and for up to a minute before actual hard crash. And I don't have any explanation for that or theory as to why that might be. For about half the systems that are working fine, there's no such slowdown happening. That is unexplainable. If Intel knows what the root cause is, it could be that half of the CPUs that have the issue can be mitigated in software. Maybe that's it. So maybe that's, you know, that leaves 25%. Half of the half that are going to have to be physically replaced. I don't know. But you know, the fact that we're still doing guesswork months after the fact is not ideal. I mean, the ambiguity here is the problem. Consider that when motherboard makers were caught juicing AMD CPUs to the point of

catastrophic failure. AMD was quick to make a statement that anyone affected by this issue would be made whole. I really don't know that Intel has done the same in a clear and concise way like you want to take care of your enthusiasts first, right? I mean, that's sort of the first and biggest killer of this situation. It's the uncertainty. Intel should step up with clear messaging that Gamers and enthusiasts with these CPUs, with an affected CPU, will be made whole with a replacement CPU if that's what it comes to. If they're still experiencing issues after that, the updates are in and the clock for getting, you know, updates in, almost out. In my opinion, customers that are buying system and CPUs by the thousands, um, they don't seem to all have the same message. I mean, they think the problem rate is going to be like maybe 10%. If you're a smaller provider, larger providers are being told a larger number. From what I can tell, w680 experiencing problems with a similar

error rate as desktop. At least for the gaming stuff is interesting because the w680 power targets are so much more conservative. Eventually, I think that the Enterprise customers and the corporate customers that are involved, they're going to start comparing notes, and they're going to start leaking things to randos on the internet that make videos to try to get some relief or to try to get, you know, more eyes on the problem, especially if some percentage of CPUs are going to end up having to be swapped and it's not able to be fixed in software or microcode or anything else. I merely, I mean, for this video, really what I want is to point out that we do in fact have the data. There's a lot of data from Steam games, and there's a lot of data from, you know, just games in general, game Telemetry, crash data, the windows event logs. Uh, yeah, it's a little prickly and rough around the edges, but you know, I mean, you're not going to be able to just do select star from where CPU, you know, from

game problems where CPU is equal to 3900k, and then just look at the data. Uh, some of these crashes are because people are overclocking their system, but the data center error rate is alarmingly high from these CPUs, and there's got to be something to that. I will not in the least be surprised if there is a deeper issue with the power voltage, you know, timing problem, and maybe the CPU have degraded over time, some of them to the point that they can't be salvaged. Like that would not surprise me in the least at this point in time, whereas you know a month ago I might have said, Ah, it's probably just motherboards and microcode. It's not. I don't believe that that's the issue anymore. And what I found was interesting, and I think based on this video that uh, smarter people are going to do their homework and go digging and try to find stuff, and they'll probably find some really interesting stuff. How what this is level one. This has just been some some

rambling on what I found doing analysis of game failure databases and listening to some customers of Intel complain about things a lot of things. Oh, and in case you're wondering that the game provider they're going with AMD 7950 X systems for their new game servers. They're being provisioned right now so that they have a little bit of breathing room to move over their multiplayer servers so that they can troubleshoot and update BIOSes and do whatever else on the Intel systems. I mean they've already been doing the BIOS update square dance for a couple of months now. Just get tired of it. I'm what this level one. I'm signing out. You can find me in the level one forums. [Music] [Music]

Intel has a Pretty Big Problem

We are passionate about technology and how it shapes our world. We create videos to share our knowledge about tech, science and design.

+ Business Inquiries/Brand Integrations: Queries@level1techs.com
*IMPORTANT* Any email lacking “level1techs.com” should be ignored and immediately reported to Queries@level1techs.com

Intel has a Pretty Big Problem

Intel has a Pretty Big Problem

Scripts