Server Cooling System Corrosion Investigation and Resolution

Lius experienced a server cooling system failure due to galvanic corrosion, leading to extensive troubleshooting and potential equipment loss.

### Server Water Cooling Disaster: Troubleshooting Galvanic Corrosion
* 😱  Linus recounts a disastrous day where he discovered a **water leak in his server cooling loop**, potentially **damaging multiple systems**.
* 🔍  Linus suspects galvanic corrosion as the cause, but is puzzled by the lack of **sacrificial anodes** in the loop.
* 🔧  He disassembles the affected components, finding **unusual corrosion patterns** that don't align with **typical galvanic corrosion mechanisms**.
* 🖥️  Linus and his son begin checking other systems in the loop, discovering that **only one out of six systems experienced significant corrosion**.
* 🛠️  The disassembly process is detailed, with Linus explaining steps to **drain the water** and **assess damage to components**.
* 💡  Despite the damage, Linus considers the outcome "**about the best I could have possibly hoped for**," with **most systems surviving**.
* 🤔  The exact cause of the corrosion remains a mystery, with Linus planning to **follow up with manufacturers for further investigation**.
* 👨‍👩‍👦  Linus concludes by thanking the audience and promoting a sponsor, while **hinting at a potential future video on rebuilding the affected systems**.

June the 14th of this year was probably the worst day of my life so far. The previous evening, I had found out that my sister passed away. I had spent the day dealing with the family drama and grief that comes with an unexpected loss like that. I made my way through one of the hardest W shows that I've ever done, um, just focusing on the routine. And when I came home, there was an unexplained wine that I couldn't really nail down the source of. I made my way downstairs and I found out that the pump that runs the cooling system for my server rack here was running dry. That means one of two things: the reservoir is leaking or the systems are leaking. And I'll let this water mark on the floor answer the question for you. So instead of eating, I began the painstaking process of tearing everything out of the rack as safely as possible. I made my way all the way through both storage and utility servers, my main and my backup, mine and Yvon's office machines, and all three of my kids.

gaming rigs. Until I found this, the culprit: a completely corroded off fitting that dumped gallons of water on everything underneath it. How could this happen? I mean, that's galvanic corrosion plain as the nose on my face, but this is a loop that has nothing but Copper, Brass, plastic. What the is going on here? I've had to sit and wonder from then until now so that I could get someone over here to point a camera at this because I'm sure that you guys want to know the answer as badly as I do. And if I'm going to lose a bunch of server equipment, well, I might as well at least help pay for it with this message from our sponsor. If I had time to untangle cables, would my hair look like this? Why not switch to Magnetic cables from our sponsor, mag tame instead. They pull apart effortlessly, roll up perfectly, and stick to any magnetic surface. No stress, no mess, no worries. We'll have them linked in the video description. Now, I know what you guys are thinking. Ah, it's lius, he's

always doing janky projects, and I'm sure he overlooked something. I mean, galvanic corrosion can't happen by accident. That's science. But listen to this. You want to talk science? How about the scientific method for a control system? I grabbed one of the other ones in the loop and look at those fittings. Those are the same two fittings, and they're immaculate. So, if it was a configuration error in the loop, these should be eaten, and yet they're pristine. Our goal today is twofold. My son obviously wants his gaming rig back up and running, which is why he's helping. But before we do that, we need to know what the heck happened here so that we can ensure that this won't happen again. The first step of disassembling these blocks is popping the stainless steel cover off. It's important to note that this is purely cosmetic since steel is not a compatible metal with the ones that are typically used for water cooling like Copper, Brass, and nickel. Now let's go a layer deeper and

remove the delin top. Okay, not the cleanest. Loop I've ever seen. Why is there so much dust on it? Well, I can tell what some of this is. These look like little plastic filings from working with the tubing that we installed. Oh, there's a little metal shaving. Uh, that could be left over from manufacturing of something. Radiator heat exchanger. There's a surprising amount of this that looks like hair or other organic matter, but there's iodine in the loop. And what kind of contradicts that theory is that this tube over here, which is running water to my system right now, is Crystal Clear. Well, maybe one of the other systems will give us some hints as to what that is. Actually, we don't need to go that far. Around the edges of the reservoir, there is a little bit of something. Not much, but I think I might need to up my iodine dosage and maybe throw a filter in there. Either way, none of that would explain the leak. Let's take apart the other block. Here we go. What the crap? What is this? Ew, why does

it look burned. It kind of does look burned. It's definitely not. It was underwater, but the kind of oxidation that could occur when burning, I guess, could also happen when it's being corroded. But the funny thing is, is that this is where the metal, and you can actually see metallic elements here, this is where the metal was being deposited, but there's no metal for it to deposit onto. Normally in galvanic corrosion, you have a sacrificial anode, and then you have a cathode that the metal deposits onto, but it's depositing onto the delrin or the space above it. You can see that's what was being corroded away, right, which is why the fitting broke off and liquid gushed out into the system. But the deposit was so close, it was right here, the mid plate, that fitting crumbled, I think. Who, okay, well, that's something that happened. Now EK had a theory when I contacted them, and that was that this stainless steel cover was somehow participating in the galvanic corrosion process because

it was bridged to the fluid somehow or something, but I just don't see how it could connect. None of these screw holes externally pass through to the inside of the block, and you can clearly see with their decades of experience designing these things, there's no connection here either. Even if I screw this in all the way, not going to lie boy, I was hoping it would be a little bit more obvious at this point. Can we just fix it and then not care what happens? We got really lucky this time. We only lost the big ups at the bottom, and I lost one of the rows of NVMe backplane slots on one of the servers. The other row fully works, and I don't need the second row. Um, also, your computer is dead. We won't necessarily get that lucky next time. There were pools of water on the motherboards of some of the systems under this one, so we need to figure out what happened, is what I'm trying to say, which means we need to start dissecting some more victims. Actually, wait before

we look at a different system. We should look at a second block within this system to see if there's something unique about this environment. These fuel line hoses are a bit of a nightmare to get off. You can see the reinforcing string that makes even cutting them a bit of a chore. Okay, what do you think we're going to find a lot more corrosion and metal. Oh, crud, I called it 'cause this means your computer's not getting fixed today. Oh, still happy. No, that might do it for EK theory about that top stainless steel thing. Maybe this is a more widespread issue. Now we really need to start cracking open more systems and seeing what's up. One rail came out. Hold on, you got to come straight out. There you go. Oh, my goodness, the plot thickens. The second fitting on this GPU is pristine. What is going on here? Could it be the Worm Drive clamps? They're not touching though, and this, this one hasn't been moved. I unscrewed it by just twisting up the tube. This is exactly as

it was pretty nasty. What this means to me is every one of these systems has to have every fitting popped to check if it's corroding. A couple of the screws going into the top of this GPU block are rusted. This is not a damp environment, so the only moisture they should have access to is from in here. I didn't know which holes they came out of though. Unfortunately, I don't think your computer is going to be working for a while. Hey, thank goodness it's summer and there's lots else to do, right? Yeah, I wouldn't let my dad get away with that, but hey, thanks. What do we have to do to get it working again? It relies on a cooling system that it shares with six other systems. All six systems need to be checked to ensure that they don't have corrosion in them, because if one of them corrodes, it takes down the whole thing. So all the systems need to be taken right down to the disassembly of the bare parts. Okay, so like 2 months, I'm the

kind of dad who believes in honoring my promises, so I am not going to promise anything. Okay, this is shaping up to look like it is going to be an absolutely terrifying amount of work. Like better off starting over amount of work here. Then shouldn't we just start over? Well, I mean it'd be kind of wasteful, don't you think? And this is your gaming rig. This has seen you through many battles. Well, you just going to leave it behind on the field? Yeah, now it's clear there are issues with the loop not caused by anything to do with block design. Uh, there is some organic matter in here. You can really tell now that it's still wet, but I reiterate that would not cause a leak. So then it is ek's Fault. Well, we can't say that definitively. I think what we're going to do is we're going to investigate one or two of the other systems, send them our findings, and then we'll go from there. The worst corroded one in both cases was nearest the

acceleration or impingement plate, but on the GPU, the deposition didn't take place on the plate itself. It took place here before it. You're probably wondering what to do next. What we need is a corrosion check on this, this, this, and this. You're going to start by blowing all the water out of the system. How do I do that? I'm glad you asked. When they aren't coupled, these quick connects stop the flow of water and air, so there's no way for us to blow out the system unless we put a female fitting on them. Now she should just kind of drain. Oh, hey, there we go. But that's slow, and we can accelerate the process using under normal circumstances. I'd be giving you guys a bunch of lectures about air compressors and computers. There are concerns with moisture being blown onto the components, uh, static buildup, but in this case, we're just blowing out tubing, so we don't need to worry about any of that. What you're going to do is you're going to take this, you're going to hold

it tight up against the in one as you can, and all the water is going to blow out the other side when you pull the trigger. [Applause] W now we don't need to worry about water leaking all over our system as we disconnect our hoses realistically. The system is not alive anymore, but there could still be components that are good. Remember that I said that your SSD might be okay. Well, that's because it's over here on the other side from where the leak was, so as long as the water didn't come up that high or as long as there wasn't some kind of malfunction of the motherboard that caused a cascading malfunction to your SSD, you might still have all your games and data and all your saves and all that kind of stuff. Get this hex head and pull the block off. Clean the thermal goop off the bottom of the block while you're handling it so you don't make a mess. Okay, next thing you want to do is get these fittings off. Yeah, you might

need to take off these worm drives first. So you need that Flathead. There's certain things that I really hope are not dead. I mean a motherboard's a relatively simple replacement. I even have a cold spare of these just in case, you know, we experienced a failure. But like these pcie Gen4 extensions and stuff like that H, it'll just probably take a little while to get new ones. Then again, I've made no commitments about when this will be done. So now we're just going to play twisty twist until this comes out. Not the world's most fun game. Twisty twist is very effective. How'd you get to be so sarcastic? Bad parenting, I guess. Yeah, yeah, yeah, yeah, yeah, yeah, yeah. Speaking of bad parenting, I think I'm just going to have to give you a really sharp knife to cut the tubing off. Thank you. Perfect. Okay, so now we just put it back. Well, I don't know, man. Like this is raising more questions than it's answering. I think what we need to do is

have a look at the fittings on the GPU now. So this one's perfect on both. Yeah, maybe this is a system by system thing. Then the fittings are perfect on the GPU on the second system. Remember our first control system as well. Got two more to go though. Let's see what's shaken in those phrase. Interesting. So yours is the bad one. Oh man, I guess this is. It's kind of good news, but it also just deepens the mystery. And there you have it. Looking at the last two systems, the ones above the leak flawless, flawless, flawless. Only one system experienced corrosion out of the five, and it was on two different blocks. For real chemistry majors, get out of your armchairs. Or rather, get in your armchairs. What happened here? So you're probably wondering, well, what now then? Well, now I'm going to test the two systems that were under number three and see if they still work, which means maybe. Ooh, caps up white light good. Hey desktop, better as unbelievable as it might seem. The only system

casualty is my son's gaming rig. Almost everything else survived. Is it possible that this system still functions? I kind of doubt it. It gushed water over the motherboard, like right over the VRMs. Let everything else survive. What do you think, Andrew? Is she just going to like post? I'm not. I think this one's [Music] dead now. Modern Electronics got a lot of protections, you know, short circuit, over current, over voltage, but dump gallons of flipping water over it. Protection. Obviously, I can't deploy it the way it is. It's got its blocks and fittings all corroded to heck and back. So there's definitely going to be a follow-up where I hear from some of you folks. I go back to the manufacturer. We figure out exactly what's going on. We rebuild this mess. But I'd say this is about the best outcome I could have possibly hoped for. Just like I couldn't hope for a better sponsor than our sponsor next to go. Have you ever hopped on a meeting and thought your screen was

frozen because the person you're talking to his picture is moving at 5 FPS. How embarrassing! Enhance your work-from-home setup with the Nexa Hello Cam Pro. It features Windows Hello facial recognition for an effortless login experience, and it also has an automated privacy cover that closes when the camera is not active. Plus, you'll look great at your next meeting thanks to the 1080p 60fps resolution and refresh rate. Learn more using the link in the description. If you enjoyed this video and you like this kind of raw troubleshooting SL building with the kids, then maybe check out fixing the gaming minivan. They were helpful.

Is this my Fault?

Linus Tech Tips is a passionate team of "professionally curious" experts in consumer technology and video production who aim to educate and entertain.


Linus Tech Tips

A Day of Unexpected Loss and Technical Disaster

The speaker experienced the sudden loss of their sister and then discovered a major technical issue with their server rack, leading to a frantic and painstaking process of investigation and repair.

The issue of galvanic corrosion causing a fitting to dump water on their equipment raises questions and prompts the need for a solution to prevent future occurrences.

The speaker's determination to understand what happened is driven by the desire to avoid a repeat of the disaster and to get their systems back up and running.

Troubleshooting a Leaking Water Cooling System

The system has a buildup of dust, plastic filings, metal shavings, and organic matter, possibly causing the leak.

The block shows signs of galvanic corrosion, with the metal being deposited onto the delrin instead of a cathode.

The cause of the corrosion, possibly involving the stainless steel cover, needs further investigation to prevent future leaks and damage.

Corrosion Issues in Computer Cooling Systems

The speaker discovers extensive corrosion in a computer cooling system, leading to the system's failure.

They realize that all six systems sharing the cooling system need to be checked and disassembled, which is a daunting task.

The possibility of starting over is considered, but the sentimental value of the current system is also taken into account.

Troubleshooting Water Deposition in GPU System

Water deposition in the GPU system requires a corrosion check on multiple components.

Blowing out the water from the system using a female fitting can help accelerate the process without causing damage.

It is important to consider the possibility of components being salvageable, such as the SSD, and to carefully clean the thermal paste off the components during the troubleshooting process.

Using a Flathead to remove worm drives is crucial in the replacement process.

The speaker has a cold spare for potential failures and is unsure about the timeline for completion.

The GPU fittings are found to be flawed in one system, leading to corrosion, while the other systems remain flawless, prompting further testing.

Possibility of a Water-Damaged Gaming Rig Still Functioning

The gaming rig suffered water damage directly to the motherboard, but other components survived.

The creator doubts the system will still function and plans to rebuild it.

The video ends with a sponsorship message for a webcam, followed by an invitation to watch another video.