Computer problems are a fact of life, and sometimes the fix is as simple as just turning it off and turning it back on again, but other times it's not.

And when the system you're talking about is running an air traffic control system, controlling a bunch of ATMs, or say routing 911 calls, keeping them up and running can be a matter of life and death.

Now, the stakes aren't nearly as high for us, but this server here runs multiple apps that we rely on every day, accelerates our game downloads with Steam caching, and it runs our DNS.

If that service goes down, it breaks literally everyone in the company's internet, which my boss informs me isn't great.

So how do we make it more reliable? It's already a server, we build more servers.

And what's really cool about this is everything we're about to show you, courtesy of Intel, who sponsored this video and sent over their new Emerald Rapids Xeon CPUs, can be done on nearly any computer, even your dad's old Dell, that is as long as you have more than one.

So if one leaves for cigarettes, we can still play catch.

More than one Dell, not more than one dad. Oh well, anyways, I'm done, you want to check this out?

Yeah, let's have a look. You got your lovely cat picture, your crab rave on that computer. M watch this like I can yeah, I can interact with this just just give it a second.

Okay, it's it's going hey now it's on this computer and like no bamboozle here look watch whoa buddy watch watch watch boom unplugged I can just completely interact with this as I normally would.

So what's going on here? What you guys just saw was the programs, the lovely drawing, the entire operating system just teleporting from the computer over here to the one over here, no trickery.

This is possible thanks to the magic of virtualization. We've talked about it before, but if you're not familiar, virtualization allows you to slice up a single machine into multiple less powerful virtual machines, and this setup leverages that technology to allow us to move these virtual machines between multiple physical computers.

That way, if one breaks, another one can immediately take its place. And the best part is that while this all sounds super fancy, all the software we're using is both open source and free, and we're going to show you guys how the setup works in a little bit.

First, I want to take a look at the servers we're going to be using for our setup.

Gigabyte sent over four of their R163 SG2 AAC1 servers. These are bare bones, so we're going to have to add a few of our own parts, but we should be able to build this in what, like 5 minutes?

We're going to add some of our own parts, starting with a pair of Patriot 480 gig SATA SSDs that will function as a mirrored boot drive.

This kind of per machine redundancy isn't strictly speaking necessary because we could lose an entire machine in our configuration without having any issues, but having them in pairs makes our lives easier in the future potentially, since if one of them fails, we can just replace it and then rebuild it from the other one.

Then on the other side of the machine, we're installing two of these Kioxia CD6 7 TB drives for fast bulk storage.

That leaves us six more SATA bays to do nothing with and two more NVMe bays for potential future expansion. Moving back, let's get our CPU installed.

We're using a Xeon Platinum 8562Y+ in each node. These were graciously provided by Intel, and with 32 cores, 64 threads, and 4.1 GHz max turbo clock speeds, these are going to give us a ton of compute to share between our virtual machines, all at a modest 300 watt TDP.

We're going to have it and the rest of the parts linked in the video description.

Now, I've never installed in this socket before, so good luck me. Step one is to install the carrier on the CPU, and you can tell which one of the three you're supposed to use by the little marking right there on the CPU IHS.

Line up our little golden triangle with our gigantic, gargantuan hole in the whole thing.

Triangle. Oh, this is adorable. It's got a cute little arm so you can break the thermal paste seal with the cooler so you can get the cooler and the CPU separated more easily.

Speaking of thermal paste, we're going to be using a Honeywell PTM 7950 pad available at LTTstore.com. This stuff is absolutely perfect for a server install because it lasts not forever, but for a very, very, very long time without maintenance.

Now you might think, okay, go ahead, put it onto the CPU socket.

You'd be wrong. Instead, I'm going to install it onto the cooler. I'm going to know how to do that in a sec.

So arrow and arrow. So maybe, ah, ah, ah, hey, there we go. Damn, look at that vapor chamber. Love me a vapor chamber. Okay.

We're going to make sure all these are clicked into place. Look for our little arrow here, line that up with the arrow on the socket and make sure that the locks are in their unlocked position, then we should be able to just, that's it, it's locked.

Oh, that's it. Okay, next comes something you don't see me do very often, and that is use a screwdriver other than the LTT screwdriver, and that's because these need to be torqued to a specific value, that is 6.9 inch lbs.

Nice. It's so cool to think that if I was doing this, you know, performing maintenance on the server or upgrading a bad RAM stick, our entire operation could be chugging along as if nothing happened.

Speaking of RAM, we've gone with four 96 gig Micron 5600 megatransfer per second registered ECC DIMMs. That's a somewhat unconventional choice because, especially in a server, giving up half of the memory channels means that we will be giving up some performance, but we don't really need all of the performance for now, and 384 gigs is a ton of capacity for our needs at the moment, and of course, if anything changes, we can always add more without any downtime to our services.

The only thing that's really important here then is making sure that we install our sticks in the correct slots, which is not always super intuitive, so make sure to consult your manual.

We don't need a GPU for now, though we could add one in the future.

So that means all that's really left is these Nvidia ConnectX-6 cards. Now 100 Gig networking might seem a bit overkill, but because our setup uses high-speed drives in four servers and we want to be able to withstand two server failures, anytime we're writing data, it has to be simultaneously written to the drives on at least three machines.

That ensures we have three up-to-date copies in the event of an unexpected failure.

Now, if you were doing this at home, you obviously wouldn't want to spend this kind of money, but the good news is that you can do this with as few as two machines.

And if you're not trying to run a high-speed caching server for 100 people, 10 or 25 gig cards are available for a fraction of the price, and you can connect them directly to each other without an expensive switch.

I mean, even one gig could work for light applications like ensuring that your home automation system never goes down.

Enough chitchat, though, let's get on with the demo and show you what happens if one of these things goes to heaven in a live environment, but not before we get them in the rack and set up specifically here in the lab server room.

Because if you didn't notice earlier, the studio server room is kind of running out of space, at least until these machines are up and running and we can take the machine they're replacing out.

Let's go grab the service. Unfortunately, the rest of the machines are now magically built off of camera and we can just slide them in.

What the hell is going on? Oh, there we go. Beautiful. These Gigabyte chassis come with nice toolless rails, so installing these in our nice ginormous Hammond rack should be pretty easy.

Yeah, look at that. It's getting close. I can taste it.

We just need networking. Like we mentioned before, 100 gig, but what we didn't mention before is that each is getting two of them, specifically one to each of the network switches in the rack.

That way, if one of those switches has a problem, the servers will stay up and we even get an added bonus because some fancy Dell magic called VT, we get the throughput of both of these cables, so 200 gig to each surface, pretty sick.

All that's left then is power and like any other good server, IPMI, which is a management interface and allows us to control the machines even if they're not working.

They have like a hardware problem, we can still access them, we can turn them on, turn them off. It's kind of magic.

If you have a server that doesn't have IPMI, I don't know, I don't even know if that's a server really.

There are two main elements to making this setup work: clustering the hypervisor, which controls our virtual machines, and clustering the storage, which you can skip if you have existing network storage you want to use instead.

If you're not not interested in how to set this up, you can skip ahead to here to see what it's like when it's up and running.

This isn't going to be a perfect step-by-step guide, but with the documentation you can find down in the description, you should be able to replicate this setup pretty easily.

Starting with networking, we added both of our 100 Gig ports to a bond, created a bridge, and then added a VLAN for three different networks: one for our VMs to use, one for cluster communication, and one for the storage.

They can all technically run on the same network, but the cluster needs low latency and the storage ideally uses jumbo frames, so splitting it up like this is best practice.

You'll also need to add each node's cluster network IP address to the host file on each node.

With the networking up and running, enable the no subscription repo and disable the Enterprise repo. It's not recommended by the Proxmox team, for production, they want you to pay for the Enterprise repo, which is a bit more stable, but the free one is totally fine for a home setup.

Run any pending updates before proceeding. Then make sure you have a reliable and ideally local time server configured on each of your individual servers, as the clustering software wants the time very closely in sync to stay happy.

With that out of the way, we can set up our cluster, which handles syncing the configuration and management of any virtual machines between our physical machines, and it also orchestrates migrating or restoring them when a machine goes down.

Creating the cluster just takes actually a few clicks, but you might want to consider the size of your setup before you continue. That's because in order to make sure everything stays in sync in case of an issue with a machine, you need the majority of servers online and available to be able to say, hey, I see that one's offline, but you know, we're we're still good.

They call this Quorum. If you have an even number of machines, let's say four, like we do, and each server gets the default single say or vote, the minimum possible majority is then three servers, so that means we can only withstand one going down, which is the same amount of redundancy you'd get if you had three machines, 'cause you can only lose one to have two.

If you only have two computers, then you only ever have a majority when both are online, which obviously doesn't work, that's not safe.

But you can screw it around this by adding a third machine, like say a Raspberry Pi to be a tiebreaker, but that's kind of beyond the scope of this video.

Once you're ready, select the cluster network in the creation menu, and then join the other machines to the cluster.

Once they're in, you should be able to see them in the web GUI of any of the machines. Now, on to clustering our storage.

By default, Proxmox is very heavily integrated with SEF, an open source distributed storage system that's pretty easy to set up and maintain.

With that in mind, newbies should start with Seth, and you can follow the great tutorial on their Wiki, but it isn't the most performant in a small cluster like this, so we're going to be using something called Linode with DRBD, or distributed replicated block device, another open source storage system.

It requires a bit more manual configuration, but they do have a purpose-built tutorial for Proxmox and host the files for free with an optional paid Enterprise version that operates on a similar model as Proxmox itself.

Unlike SEF, it doesn't handle its own storage devices, so we mirrored our two Kioxia SSDs with ZFS first and then pointed Lin store to that.

Once it's installed and configured, then you can add the clustered storage to Proxmox, create a virtual machine with that storage, and it'll automatically be replicated in real time to the number of other nodes you specify.

And if you happen to migrate a VM to a server that doesn't have a copy on it, it'll automatically stream the data over the network from one of those nodes in what they call dis list mode, but let's just try it.

Hey, pretty nice, right? Looking good. It's like even cable managed. I know, right? So 200 gig on each of them.

Nice. Who are you people and what have you done with our infer team? I made one small adjustment just for you.

Look at the drives. They're in the same spot. No, they're not. The top one's different. I hate you so much. Why?

would you do that but more importantly does it work? Uh, yeah, obviously. Okay, well here's your Windows desktop. Obviously, he says, what editor? A super cut of things not working here.

Please, Jake, we have a leak. Oh God, one failure. You just downgraded my Wi-Fi. Four drives aren't working.

Did you actually break it? Anyways, you see our Windows, right? Yeah, our Windows is running right now on number four, which is the bottom server.

Yes. Now, obviously, remoting into the machine over Wi-Fi. Okay, the video playback's a little bit choppy. That's not going to affect the type of workload we would normally be running on something like this, like a DNS server or like, are we finally doing Active Directory?

Not today, but we can now. But this is the kind of setup that you want for something like AD live, playing the video. Let's migrate to number one, which is the top one.

The process will be a little bit faster, but basically what it's doing is copying the memory, like the RAM, what's actually in memory.

And then once it's done most of it, it pauses the operating system for a split second, copies the last tiny little bit and boom.

That is so cool. You're exactly where you were before because the storage is already there, right? So in terms of actual downtime, like interruption to that experience, 17 seconds, no, 270 milliseconds.

Oh, I thought you were pointing the other. No, 17 seconds is that whole process.

But oh yeah, yeah, well, that's kind of downtime, I guess. No, cuz that if there was somebody using this like as a virtual desktop for instance, they would see like a quarter of a second blink and otherwise like nothing changed.

I wanted to show a more realistic to us demo. Sure, come hither.

here's a Plex server, we've got some videos on it, and this is on server number one. Okay, let's play a video now.

We go and move our Plex server to a different machine, so it's copying the RAM at 2.5 gigabytes a second, so it's like 2.8 gigabytes a second, that's pretty good.

We haven't done any actual, oh, it's already done and no interruption because video playback, like many other applications, uses buffers to hide small interruptions in the service.

In this case, downloading the video in small chunks a little bit at a time, yeah, roughly 10-second chunks it looks like here, which is plenty to cover that 146 milliseconds of downtime.

Wow. You want to try Steam download with LAN cache? I mean, we should. Yeah, why not? Yep, we're CPU bottleneck for sure, using, you know, 80 to 90% of a 24 core Threadripper.

But I realized I made a little bit of an oopsy here.

Look, you can see the CPU usage. We're using % of our eight CPUs that I assigned to this Steam cache. We can see our network traffic's going up, sick.

Except I made this as a container, not a VM, and the thing with containers, they're great, they're a little bit lighter weight, better performance, but they run within the kernel of the main system.

It'll shut down that container and then just reboot on the other machine, right, which means it's fine, but there will be a longer downtime delay, but way less than, hey, is that thing working?

Oh, I think the internet is not working. Somebody should go look at that. Yeah, trying to figure out what's going on, fixing the machine, getting the machine back going so cool.

You're talking about the matter of a couple minutes maybe. Yeah. Now for the most impressive demo yet, the unexpected migration.

Which one am I yanking? Okay, so number one has three VMs on it. They're all in the high availability Jake's chain.

Ah, what means? Oh, I get it. Okay, sorry. Which one? I wasn't even listening to number one, number one and we'll see how how fast it does.

I Built a PC that CAN’T Fail… and You Can Too!

Linus Tech Tips is a passionate team of "professionally curious" experts in consumer technology and video production who aim to educate and entertain.


I Built a PC that CAN’T Fail… and You Can Too!

Scripts