Enhancing Server Reliability with Virtualization Techniques

Virtualization enables seamless server management, ensuring reliability and minimal downtime for critical applications.

### Building a Reliable Server Cluster with Virtualization and High Availability
* 💻  The video discusses the importance of **server reliability** for critical systems like air traffic control, ATMs, and 911 call routing, as well as for **everyday business operations**.
* 🔧  The setup involves four Gigabyte servers with Intel Xeon Platinum 8562Y+ CPUs, **384GB of RAM**, and **100Gb networking cards**, creating a **powerful cluster for virtual machines**.
* 🛠️  The servers are configured with Proxmox VE, an open-source virtualization platform, and LinStore with DRBD for clustered storage, ensuring **high availability** and **real-time data replication**.
* 🌐  Networking is set up with VLANs for different types of traffic, and the cluster is created to **manage VM migration and failover** between the physical nodes.
* 🔄  A live demo showcases **seamless VM migration between servers**, with **minimal downtime and interruption** to services like Plex media streaming and Steam game downloads.
* 💡  The setup is highlighted as a **game-changer for infrastructure management**, with potential applications ranging from **home automation to enterprise systems**.
* 💰  The video emphasizes that similar setups can be achieved with older, more affordable hardware, making **high availability clusters accessible to a wider range of users**.
* 🔗  The description includes links to the components used in the build, as well as additional information on **Intel's Emerald Rapids processors** and **other related products**.

computer problems are a fact of life, and sometimes the fix is as simple as just turning it off and turning it back on again. But other times, it's not, and when the system you're talking about is running an air traffic control system, controlling a bunch of ATMs, or say routing 911 calls, keeping them up and running can be a matter of life and death. Now, the stakes aren't nearly as high for us, but this server here runs multiple apps that we rely on every day, accelerates our game downloads with Steam caching, and it runs our DNS. If that service goes down, it breaks literally everyone in the company's internet, which my boss informs me isn't great. So, how do we make it more reliable? It's already a server. We build more servers. And what's really cool about this is everything we're about to show you, courtesy of Intel, who sponsored this video and sent over their new Emerald Rapid Xeon CPUs, can be done on nearly any computer, even your dad's old Dell, that is, as long as you have more than one. So, if one

leaves for cigarettes. We can still play catch. More than one dell, not more than one dad. Oh well, anyways, I'm done. You want to check this out? Yeah, let's have a look. You got your lovely cat picture, your crab rave on that computer. M, watch this. Like, I can, yeah, I can interact with this. Just give it a second, okay? It's going. Hey, now it's on this computer. And like, no bamboozle here. Look, watch. Whoa, buddy. Watch, watch, watch. Boom, unplugged. I can just completely interact with this as I normally would. So, what's going on here? What you guys just saw was the programs, the lovely drawing, the entire operating system just teleporting from the computer over here to the one over here. No trickery. This is possible thanks to the magic of virtualization. We've talked about it before, but if you're not familiar, virtualization allows you to slice up a single machine into multiple less powerful virtual machines. And this setup leverages that technology to allow us to move these

virtual machines between multiple physical computers. That way, if one breaks, another one can immediately take its place. And the best part is that while this all sounds super fancy, all the software we're using is both open-source and free. And we're going to show you guys how the setup works in a little bit. First, I want to take a look at the servers we're going to be using for our setup. Gigabyte sent over four of their R163 SG2 AAC1 servers. These are bare bones, so we're going to have to add a few of our own parts, but we should be able to build this in what, like 5 minutes? I'd like to see you try this, guy. We're going to add some of our own parts, starting with a pair of Patriot 480 gig SATA SSDs that will function as a mirrored boot drive. This kind of per-machine redundancy isn't strictly speaking necessary because we could lose an entire IRE machine in our configuration without having any issues, but having them in pairs makes our lives

easier in the future potentially since if one of them fails we can just replace it and then rebuild it from the other one. Then on the other side of the machine, we're installing two of these Kokia CD6 7 TBTE drives for fast bulk storage. That leaves us six more SATA bays to do nothing with and two more NVMe bays for potential future expansion. Moving back, let's get our CPU installed. We're using a Xeon Platinum 8562 Y+ in each node. These were graciously provided by Intel, and with 32 cores, 64 threads, and 4.1 GHz max turbo clock speeds, these are going to give us a ton of compute to share between our virtual machines, all at a modest 300 watt TDP. We're going to have it and the rest of the parts linked in the video description. Now, I've never installed in this socket before, so good luck me. Step one is to install the carrier on the CPU, and you can tell which one of the three you're supposed to use by the little

marking right there on the CPU IHS. Line up our little Golden Triangle with our gigantic gargantuan hole in the whole thing triangle. Oh, this is adorable. It's got a cute little arm so you can break the thermal paste seal with the cooler so you can get the cooler and the CPU separated more easily. Love to see it. Speaking of thermal paste, we're going to be using a Honeywell PTM 7950 pad available at LTTstore.com. This stuff is absolutely perfect for a server install because it lasts not forever but for a very, very, very long time without maintenance. Now you might think, Okay, go ahead, put it onto the CPU socket. You'd be wrong. Instead, I'm going to install it onto the cooler. I'm going to know how to do that in a sec. So arrow and arrow, so maybe ah ah ah hey, there we go. Damn, look at that vapor chamber. Love me a vapor chamber. Okay, we're going to make sure all these are clicked into place. Look for our little

arrow here line that up with the arrow on the socket and make sure that the locks are in their unlocked position. Then we should be able to just that's it's locked. Oh, that's it. Okay, next comes something you don't see me do very often, and that is use a screwdriver other than the LTT screwdriver. And that's because these need to be torqued to a specific value that is 6.9 inch PBS. Nice. It's so cool to think that if I was doing this, you know, performing maintenance on the server or upgrading a bad RAM stick, our entire operation could be chugging along as if nothing happened. Speaking of RAM, we've gone with four 96 gig Micron 5600 megat transfer per second registered ECC dims. That's a somewhat unconventional choice because especially in a server, giving up half of the memory channels means that we will be giving up some performance. But we don't really need all of the performance for now, and 384 gigs is a ton of

capacity for our needs at the moment, and of course, if anything changes, we can always add more without any downtime to our services. The only thing that's really important here then is making sure that we install our sticks in the correct slots, which is not always super intuitive, so make sure to consult your manual. We don't need a GPU for now, though we could add one in the future, so that means all that's really left is these Nvidia connect X6 cards. Now, 100 Gig networking might seem a bit overkill, but because our setup uses high-speed drives in four servers, and we want to be able to withstand two server failures, anytime we're writing data, it has to be simultaneously written to the drives on at least three machines. That ensures we have three up-to-date copies in the event of an unexpected failure. Now, if you were doing this at home, you obviously wouldn't want to spend this kind of money, but the good news is that you can do this with as few as two machines, and if you're not trying to run a high-speed

caching server for 100 people. 10 or 25 gig cards are available for a fraction of the price, and you can connect them directly to each other without an expensive switch. I mean, even one gig could work for light applications like ensuring that your home automation system never goes down. Enough chitchat though, let's get on with the demo and show you what happens if one of these things goes to heaven in a live environment, but not before we get them in the rack and set up specifically here in the lab server room. Because if you didn't notice earlier, the studio server room is kind of running out of space, at least until these machines are up and running, and we can take the machine they're replacing out. Let's go grab the service. Unfortunately, the rest of the machines are now magically built off-camera, and we can just slide them in. What the hell is going on? Oh, there we go, beautiful. These gigabyte chassis come with nice toolless rails, so installing these in our nice

ginormous hammonded rack should be pretty easy. Yeah, look at that. It's getting close. I can taste it. We just need networking, like we mentioned before, 100 Gig. But what we didn't mention before is that each is getting two of them, specifically one to each of the network switches in the rack. That way, if one of those switches has a problem, the servers will stay up, and we even get an added bonus because some fancy Dell magic called VT. We get the throughput of both of these cables, so 200 gig to each surface. Pretty sick. All that's left then is power, and like any other good server, IPMI, which is a management interface, and allows us to control the machines even if they're not working. They have like a hardware problem. We can still access them. We can turn them on, turn them off. It's kind of magic. If you have a server that doesn't have IPMI, I don't know. I don't even know if that's a server, really. There are two main elements to making this setup work: clustering the

hypervisor, which controls our virtual machines, and clustering the storage, which you can skip if you have existing network storage you want to use instead. If you're not interested in how to set this up, you can skip ahead to here to see what it's like when it's up and running. This isn't going to be a perfect step-by-step guide, but with the documentation you can find down in the description, you should be able to replicate this setup pretty easily. Starting with networking, we added both of our 100 Gig ports to a bond, created a bridge, and then added a VLAN for three different networks: one for our VMs to use, one for cluster communication, and one for the storage. They can all technically run on the same network, but the cluster needs low latency, and the storage ideally uses jumbo frames, so splitting it up like this is best practice. You'll also need to add each node's cluster Network IP address to the host file on each node. With the networking up and running, enable the no subscription repo and disable the

Enterprise repo. It's not recommended by the Proxmox team for production. They want you to pay for the Enterprise repo, which is a bit more stable, but the free one is totally fine for a home setup. Run any pending updates before proceeding, then make sure you have a reliable and ideally local time server configured on each of your individual servers, as the clustering software wants the time very closely in sync to stay happy. With that out of the way, we can set up our cluster, which handles syncing the configuration and management of any virtual machines between our physical machines, and it also orchestrates migrating or restoring them when a machine goes down. Creating the cluster just takes actually a few clicks, but you might want to consider the size of your setup before you continue. That's because in order to make sure everything stays in sync in case of an issue with a machine, you need the majority of servers online and available to be able to say, Hey, I see that one's offline, but you know, we're still good. They call

this Quorum. If you have an even number of machines, let's say four like we do, and each server gets the default single say or vote, the minimum possible majority is then three servers. So, that means we can only withstand one going down, which is the same amount of redundancy you'd get if you had three machines, 'cause you can only lose one to have two. If you only have two computers, then you only ever have a majority when both are online, which obviously doesn't work. That's not safe. But you can screw it around this by adding a third machine, like say a Raspberry Pi, to be a tiebreaker. But that's kind of beyond the scope of this video. Once you're ready, select the cluster Network in the creation menu, and then join the other machines to the cluster. Once they're in, you should be able to see them in the web GUI of any of the machines. Now, on to clustering our storage. By default, Proxmox is very heavily integrated with Ceph and open-source distributed storage.

system that's pretty easy to set up and maintain. With that in mind, newbies should start with Seth, and you can follow the great tutorial on their Wiki. But it isn't the most performant in a small cluster like this, so we're going to be using something called Linore with DRBD, or Distributed Replicated Block Device, another open-source storage system. It requires a bit more manual configuration, but they do have a purpose-built tutorial for Proxmox and host the files for free, with an optional paid Enterprise version that operates on a similar model as Proxmox itself. Unlike Seph, it doesn't handle its own storage devices, so we mirrored our two Kokia SSDs with ZFS first and then pointed Lin store to that. Once it's installed and configured, then you can add the clustered storage to Proxmox, create a virtual machine with that storage, and it'll automatically be replicated in real time to the number of other nodes you specify. And if you happen to migrate a VM to a server that doesn't have a

copy on it. It'll automatically stream the data over the network from one of those nodes in what they call dis list mode. But let's just try it. Hey, pretty nice, right? Looking good. It's like even cable managed. I know, right? So, 200 gig on each of them. Nice. Who are you people and what have you done with our infer team? I made one small adjustment just for you. Look at the drives. They're in the same spot. No, they're not. The top one's different. I hate you so much. Why would you do that? But more importantly, does it work? Uh, yeah, obviously. Okay, well, here's your Windows desktop. Obviously, he says what editor. A super cut of things not working here. Please, Jake, we have a leak. Oh God, one failure. You just downgraded my Wi-Fi. Four drives aren't working. Did you actually break it? Anyways, you see our Windows, right? Yeah, our Windows is running right now on number four, which is the bottom server. Yes. Now, obviously, remoting into the machine over Wi-Fi. Okay, the video playback's a little

bit choppy. That's not going to affect the type of workload we would normally be running on something like this, like a DNS server, or like, are we finally doing active directory? We will not, not today, not today, but we can now, but this is the kind of setup that you want for something like AD live. Playing the video. Let's migrate to number one, which is the top one. The process will be a little bit faster, but basically what it's doing is copying the memory, like the RAM, what's actually in memory. And then once it's done, most of it, it pauses the operating system for a split second, copies the last tiny little bit, and boom, that is so cool. You're exactly where you were before because the storage is already there, right? So in terms of actual downtime, like interruption to that experience, 17 seconds, no, 270 milliseconds. Oh, I thought you were pointing the other, no, 17 seconds is that whole process, but oh yeah, yeah, well that's kind of downtime, I guess. No, 'cause that if there was somebody

using this like as a virtual desktop, for instance, they would see like a quarter of a second blink and otherwise like nothing changed. I wanted to show a more realistic to us demo. Sure, come hither. Here's a Plex server. We've got some videos on it, and this is on server number one. Okay, let's play a video now. We go and move our Plex server to a different machine, so it's copying the RAM at 2.5 gigabytes a second. So, it's like 2.8 gigabytes a second. That's pretty good. We haven't done any actual... Oh, it's already done. And no interruption because video playback, like many other applications, uses buffers to hide small interruptions in the service. In this case, downloading the video in small chunks, a little bit at a time. Yeah, roughly 10C chunks, it looks like here, which is plenty to cover that 146 milliseconds of downtime. Wow, you want to try Steam download with Landan Cach? I mean, we should. Yeah, why not? Yep, we're

CPU bottleneck for sure, using, you know, 80 to 90% of a 24-core Threadripper. But I realized I made a little bit of an oopsy here. Look, you can see the CPU usage. We're using % of our eight CPUs that I assigned to this steam cache. We can see our network traffic's going up, sick, except I made this as a container, not a VM. And the thing with containers, they're great, they're a little bit lighter weight, better performance, but they run within the kernel of the main system. It'll shut down that container and then just reboot on the other machine, right? Which means it's fine, but there will be a longer downtime delay, but way less than, Hey, is that thing working? Oh, I think the internet is not working. Somebody should go look at that. Yeah, trying to figure out what's going on, fixing the machine, getting the machine back going. So cool, you're talking about the matter of a couple minutes, maybe. Yeah, now for the most impressive demo yet, the unexpected migration. Which one am I yanking? Okay, so

number one has three VMs on it. They're all in the high availability Jake's chain. Ah, what means. Oh, I get it. Okay, sorry. Which one? I wasn't even listening to number one. Number one. And we'll see how fast it does. We're looking at server one from server two. So go for [Applause]. It, from my understanding, this process takes a minute or two. Okay, to go. Oh, it's already detected the no is offline. Sure is. If you're doing scheduled maintenance, you can actually go and just shut off a machine. And then it will just be like, oh crap, I need to move all those things before I shut off, which is a little bit nicer. In this case, it has to be like, sure that the server is down. So all three of those are yelling at what was this. Say hello. What happened? Are you alive? What's going on? I can hear them. Hello. What happened? Are you alive? What's going on? Oh, hey, hey. Oh, it did something. So in theory, it should distribute them evenly because that's the option that's

set right now, right in terms of its workload, you mean. Yeah, there is also a mode that does like resource checking. Sure, but right now, it's going how many VMs are in each one and just like filling that number. So, it's even that is so cool. Okay, so what service was running on that one? Was that the Steam cache? So, we should go download a game. You could do Plex right now too. Let's go do it. Let's go do it. Come on, let's go. Wait, we can take this door and no movie magic, but also magic virtualization magic. This is flipping awesome, and it's going to be an absolute game changer for the way that we manage our infrastructure. And like I said at the beginning, I think the coolest thing about it is that this type of architecture doesn't even have to run on the kind of Emerald Rapids latest server technology that Intel and Gigabyte and Micron and Nvidia all sent over here. So, the takeaway for you guys is whether it's for work or whether it's just for

your home automation or your Plex server at home. Something like this is absolutely attainable with potentially very little financial outlay. Like go buy some used, like eighth-gen Intel core processors. Those are pretty cheap, some cheap DDR4, and you're off to the races. Or if you're doing this more properly for your business, check out Intel Emerald Rapids and their whole line of Xeon and GPU products down below. Where were you pointing? Down below, that's the description. Get your mind out of the description.

I Built a PC that CAN’T Fail… and You Can Too!

Linus Tech Tips is a passionate team of "professionally curious" experts in consumer technology and video production who aim to educate and entertain.


Linus Tech Tips

Computer problems are common, but for critical systems like air traffic control or 911 call routing, reliability is crucial.

The speaker's server runs multiple essential apps and services, so they aim to build more servers for redundancy.

Using Intel's Emerald Rapid Xeon CPUs and virtualization, they plan to move virtual machines between physical computers, ensuring seamless operation even if one server fails.

The installation process begins with aligning the CPU's Golden Triangle with the socket and using a Honeywell PTM 7950 thermal paste pad for long-lasting maintenance.

The server is equipped with four 96GB Micron ECC DIMMs for a total of 384GB of memory, allowing for future expansion without downtime.

100 Gig networking is used for high-speed data writing across four servers, ensuring three up-to-date copies in case of a server failure, with the option to use lower-cost networking solutions for smaller setups.

The setup involves two 100 Gig networking cables for each server, providing a total of 200 Gig throughput.

Power and IPMI management interface are essential for controlling the servers, even during hardware issues.

The setup also requires clustering the hypervisor and storage, with a focus on low latency and jumbo frames for optimal performance.

A cluster with four machines needs a minimum of three servers for a majority, allowing for one server to go down.

Proxmox integrates with Ceph for storage, but the tutorial recommends using Linore with DRBD for a small cluster.

Linore requires manual configuration and mirrors SSDs with ZFS, offering real-time replication of virtual machines and data streaming in case of a migration.

The server migration process involves copying the RAM and pausing the operating system for a split second, resulting in minimal downtime.

The demo with a Plex server shows seamless video playback during the migration, thanks to buffering.

Using containers for migration offers lighter weight and better performance, but may result in a longer downtime delay compared to virtual machines.

High Availability VMs and Infrastructure Management

Number one has three VMs in a high availability chain, which quickly detects when a server goes offline.

The system distributes the workload evenly among the VMs and has a resource-checking mode.

This architecture is accessible for both home and business use, with affordable options like used Intel processors and DDR4 memory.