satisfying support vector classifier to separate the patients that were cured from the patients that were not cured. One way to deal with overlapping data is to use a support vector machine with a radial kernel, aka the radial basis function (RBF). Because the radio Colonel finds support vector classifier z' in infinite dimensions, it's not possible to visualize what it does. However, when using it on a new observation like this, the radio kernel behaves like a weighted nearest neighbor model. In other words, the closest observations, aka the nearest neighbors, have a lot of influence on how we classify the new observation, and the observations that are further away have relatively little influence on the classification. So, since these observations are the closest to the new observation, the radio kernel uses their classification for the new observation. Now let's talk about how the radio Colonel determines how much influence each observation in the training dataset has on classifying new observations. Just like with the polynomial kernel, A and B refer to two different dosage measurements. The difference between the measurements is then squared, giving us the squared

the X evaluated at a divided by 1 factorial and multiplied by X minus a note in case you don't already know the derivative of e to the X equals e to the X. So taking the derivative of e to the X is super easy. Now we plug in the second derivative of e to the X evaluated at a divided by 2 factorial and multiplied by X minus a squared. Then we keep adding terms based on higher derivatives until we get to infinity. Thus, this is the Taylor series expansion of e to the X. Now the question is, what is a? The definition of the Taylor series says that a can be any value as long as F of a exists. And since e to the zero equals one, e to the zero exists. So we will set a equal to zero and simplify peepee poopy poopy poop. Thus, if we can accept that the Taylor series expansion does what it says it does, e to the X is equal to this infinite sum large BAM. Going back to the radial kernel, we can now create the Taylor series expansion.

are relatively far from each other. We get a number very close to zero. Thus, the further two observations are from each other, the less influence they have on each other. Note, just like with the polynomial kernel, when we plug values into the radial kernel, we get the high-dimensional relationship. Thus, 0.11 is the high-dimensional relationship between these two observations that are relatively close to each other, and a number very close to zero is the high-dimensional relationship between these two observations that are relatively far from each other. Medium Bamm. Now, before we move on, I want to simplify the training data set to just two observations and use the polynomial kernel to give us intuition into how the radio kernel works in infinite dimensions. When R equals zero, the polynomial kernel simplifies to a single term, and that gives us a dot product with a single coordinate. When D equals 2, we get a squared times B squared, which is equal to the dot product of A squared and B squared. Since this dot product only has one coordinate, the new coordinate is just

the square of the original measurement on the original axis. For example, if we plug these two dosages into the kernel, we get this dot product, and that means the new XS coordinate for this point is 2.5 squared, which equals 6.25. And the new x-axis coordinate for this point is 4 squared, which equals 16. In other words, when R equals 0 and D equals 2, all the polynomial kernel does is shift the data down the original axis. When R equals 0 and D equals 3, we still only have one coordinate in the dot product, and we shift the data down the X access further because now we are cubing each value. Lastly, when R equals 0 and D equals 1, we still only have one coordinate in the dot product, but now the data just stays in its original location. So setting R equals 0 seems silly because no matter what values we use for D, the dot products leave the data in the original dimension. And in this example, the data stays on the same one-dimensional line. However, it turns

coordinates for an infinite number of dimensions. That would be awesome, right? Well, that's exactly what the radial kernel does. So, let's talk about it. Warning: this part gets very mathy, so feel free to skip to the end if this is not your thing. Let's start with the radial kernel and multiply out the square beep OOP OOP OOP OOP. Now, because we can set gamma equal to anything, let's set it to 1/2 so that this 2 goes away. Now, let's create the Taylor series expansion of this last term. Wait a minute, what's a Taylor series expansion? This big thing is a Taylor series. Although there are exceptions, the main idea is that a function f of X can be split into an infinite sum. Since this is very abstract, let's walk through how to convert e to the X into an infinite sum. In other words, we are setting f of X equal to e to the X, and that means F of a equals e to the a. Now, we plug in the derivative of e to

of this last term to create the Taylor series expansion of e to the a B. We plug in a B for X. Now we have the Taylor series expansion of the last part of the radial kernel. Okay, time to take a deep breath. We've done a lot, but we still have a few more steps before we are done and get to eat snacks. Before we move on, let's remember that when we added up a bunch of polynomial kernels with R equals 0 and D going from 0 to infinity, we got a dot product with coordinates for an infinite number of dimensions. Now observe that a polynomial kernel with R equals 0 and D equals 0 is equal to 1, and a polynomial kernel with R equals 0 and D equals 1 is equal to a times B, and a polynomial kernel with R equals 0 and D equals 2 is equal to a times B squared, etc., etc., etc. Thus, each term in this Taylor series expansion contains a polynomial kernel with R equals 0 and D going from 0 to

has coordinates for an infinite number of dimensions. That means when we plug numbers into the radial kernel and do the math, be boo-boo-boo, the value we get at the end is the relationship between the two points in infinite dimensions. Triple bam! Now we can go eat snacks. Hooray! We've made it to the end of another exciting stat quest. If you like this stat quest and want to see more, please subscribe. And if you want to support stack quest, consider contributing to my Patreon campaign, becoming a channel member, buying one or two of my original songs, or a t-shirt or a hoodie, or just donate. The links are in the description below. Alright, until next time, quest on.

infinity now just to remind you. Converting this sum to a dot product was easy because the dot product tells us to multiply each term together and then add up all the terms. With that in mind, the dot product for e to the a B is this. We can verify that the dot product is correct by multiplying each term together and adding up the new terms to get the Taylor series expansion of e to the a B double bam back to the original radial kernel. We can plug in the dot product for e to the a B. Now the radial kernel is equal to this term times the dot product to make the radio Colonel all one dot product instead of something times a dot product. We just multiply both parts of the dot product by the square root of this term. So we can fit everything onto the screen. Let's let s equal the square root of the first term. Now we multiply the dot product by s be boo boo boo boo boo boo boo boo boo boo and at long last we see that the radial kernel is equal to a dot product that

out that setting R equals zero can result in some pretty awesome stuff. Going back to the original training data set, let's talk about what happens if we take a polynomial kernel with R equals 0 and D equals 1, and add another polynomial kernel with R equals 0 and D equals 2. This gives us a dot product with coordinates for two dimensions. The first coordinate is the original dosage, and the second coordinate is dosage squared. Now we can plot the transformed data on XY axes and find a support vector classifier to separate the data. And by now, you know that we don't actually do the transformation; we just solve for the dot product to get the high-dimensional relationships. Now, if we added another polynomial kernel with R equals zero and D equals three, then the dot product has coordinates for three dimensions, and we can plot the transformed data on XYZ axes and find a support vector classifier to separate the data. Now, what if we just kept adding polynomial kernels with R equals zero and increasing D until D equals infinity? That would give us a dot product with

distance between the two observations. Thus, the amount of influence one observation has on another is a function of the squared distance gamma, which is determined by cross-validation. Scales the squared distance and thus it scales the influence. For example, if we set gamma equal to one, plug in the dosages from two observations that are relatively close to each other, and do the math peepee poopy poop, we get 0.11 when gamma equals 1. So, let's put 0.11 here. Now, let's set gamma equal to two and plug in the same two dosages as before and do the math beep boop boop. When gamma equals 2, we get 0.01, which is less than when gamma equals 1. So, we see that by scaling the distance, gamma scales the amount of influence two points have on each other. Small BAM. Now, let's set gamma equal to 1 again and determine how much influence two observations have when they are relatively far from each other. So, we plug in the two dosages, do the math.

the radio Colonel works in infinite dimensions. I know that sounds kind of crazy, but it's actually not that bad. Stat Quest. Hello, I'm Josh Armour, and welcome to Stat Quest. Today, we're gonna talk about support vector machines, part 3, the radial kernel. Specifically, we're going to talk about the radial kernel's parameters, how the radial kernel calculates high-dimensional relationships, and then show you how the radial kernel works in infinite dimensions. Note this Stat Quest assumes that you are already familiar with support vector machines and the polynomial kernel. If not, check out the quests. The links are in the description below. In the Stat Quest on support vector machines, we had a training data set based on drug dosages measured in a bunch of patients. The red dots represented patients that were not cured, and the green dots represented patients that were cured. In other words, the drug doesn't work if the dosage is too small or too large. It only works when the dosage is just right. Because this training data set had so much overlap, we were unable to find a

Support Vector Machines Part 3: The Radial (RBF) Kernel (Part 3 of 3)

Statistics, Machine Learning and Data Science can sometimes seem like very scary topics, but since each technique is really just a combination of small and simple steps, they are actually quite simple. My goal with StatQuest is to break down the major methodologies into easy to understand pieces. That said, I don't dumb down the material. Instead, I build up your understanding so that you are smarter.

Contact, Video Index, Etc: https://statquest.org

Support Vector Machines Part 3: The Radial (RBF) Kernel (Part 3 of 3)

Scripts