Now, moving on to the next topic, we have explored how artificial intelligence is developed, and among that, machine learning has been identified as the core technology we are focusing on. I will primarily discuss an overview of machine learning and how it progresses. 

First of all, it is inevitable that artificial intelligence, which we are working on, is done through computers, and that involves mathematical concepts. Therefore, a mathematical expression is necessarily included. Let’s first look at how we can express artificial intelligence in basic mathematical terms. As mentioned earlier, artificial intelligence aims to mimic human intelligence. This requires input, so we need to consider what kind of input is involved.

We refer to it as environmental information or the observed data we have. Based on this data, we need to determine what actions we should take; making judgments about these actions is essentially what artificial intelligence is. AI receives input environmental information and makes necessary judgments regarding these actions. Therefore, through this x, we can derive this y, and the function that connects x and y is what we call artificial intelligence. Mathematically, we can express this as y equals the function f of x. This is the most basic way we can represent a function. For example, in image recognition, the environmental data would consist of a photo. Based on this photo information, the AI might determine that the probability of the image being a cat is 0.9 and the probability of it being a dog is 0.1. So ultimately, it concludes that...

Now, we are making a judgment that it is a cat. So, we can express this mathematically as y = f(x). To reiterate, the key to implementing artificial intelligence is how we find that f in the equation y = f(x), which is the most fundamental aspect. Now, how do we find this apple? As mentioned earlier, there are two approaches. One approach is rule-based. We set certain rules in advance. For example, if x is less than 3, we go this way; if x is greater than 3, we evaluate whether the maximum value of x is greater than 5. We continue making judgments in this manner, consistently applying these rules.

이제 우리가 f라는 것을 만들어 낼 수 있습니다. 하지만 이렇게 하기에는 모든 것을 이런 식으로 해결할 수 없으므로 데이터 기반의 인공지능도 필요하게 됩니다. 데이터 기반 인공지능이란, 명확한 규칙을 정할 수 없기 때문에 이미 인간이 내리고 있던 판단들을 모두 모아 놓는 것입니다. 예를 들어, 어떤 x의 값일 때 y는 어떤 값이 되며, 또 다른 x의 값일 때 y는 또 다른 값이 되는 데이터를 쭉 모읍니다. 이 데이터로부터 x에서 y로 가는 함수를 적절히 찾도록 컴퓨터에 지시하는 것이 바로 데이터 기반의 인공지능입니다. 따라서 데이터 기반의 인공지능은 기본적으로 데이터가 필요합니다. 이러한 x와 y를 나타내는 데이터가 필요하게 됩니다. 그러다 보니 우리가 지금 하려는 이러한 기계 학습은 데이터가 필수적입니다.

It is based on fundamentals. So, the learning we talked about is based on data, which raises the important question: what is data? For example, let's say we have data on the height of adult males in South Korea. Typically, we would gather a sample of, say, 100 male university students from Korea University, and measure their heights. We might find that one person is 168 cm tall, another is 181 cm, another is 170 cm, and another is 178 cm, resulting in a total of 100 height measurements. This collection of 100 numbers is what we refer to as data. However, if we think about what data really is, we can easily realize that it consists of randomly observed data. Therefore, we refer to this as observed random data.

In fact, in South Korea, the adult male population is roughly around 20 million, considering our total population is about 25 million and excluding children. Therefore, there are more than 20 million adult men. This means there are more than 20 million possible height measurements. However, we do not examine all 20 million data points; instead, we only look at a sample of 100 data points. So, what this means is that the figure of 20 million is what we commonly refer to as the population, which is called population in English. From this, we randomly select data, like drawing balls from a jar or rolling dice. For instance, we randomly select one measurement of 168, then another one of 180, and then another one of 178, and so on, selecting them randomly.

So, what we have gathered here becomes data. Now, what is the nature of this data? The data we see may appear to be 100 fixed numbers, but in reality, it is not fixed data. If we were to draw data 100 more times from here, the new data would be very similar to the previously drawn data, but they would actually be different data sets. Therefore, the property itself, which we refer to as distribution, is similar, but the actual values are all different. Thus, the data we are looking at is not fixed data; it is data obtained merely by coincidence. It is more accurate to think of it this way. So, what we usually talk about in terms of learning objectives is to identify some properties that are valid only from the observed data.

The goal of learning is not to identify this data, but to find something fundamental that can be applied to the entire dataset from which this data originated. Therefore, it is not just about the observed data; the goal is to find valid data from the entire population, which we refer to as estimation. In English, it's called estimation, and what this means is that the objective is to find some actual values in the population based on the observed data. However, the problem is that it would be ideal if we could directly observe this population, but doing so is practically impossible. This creates a consistently challenging issue. So, if we think more about the goal of learning, as I mentioned earlier, it is to find a function f that represents the relationship between x and y when we have both of them from the data.

Finding is our goal. So, if we approach this more mathematically, we have the output variable, which is the result of the judgment regarding the behavior we want to target, denoted as y. There are input variables, x, that can influence the result of this judgment. Usually, there are multiple x variables, such as x1, x2, and so on, up to xp. When we consider the relationship between them, we express it as y = f(x), and our goal is to find f. The objective of learning is to track this function f from the given observational data because we cannot see all the data in the population. Therefore, the aim is to find a relationship that is valid not just within the observational data but also in the population. For example, suppose we collected data with x and y, and after gathering about 30 data points, we ended up with a certain set of data. This is the data we collected. Now, we need to represent the true relationship between x and y.

The purpose is to find what a true relationship is. This true relationship could be represented as a bent S-shaped curve or as a straight line. So, which one represents a better relationship? Which one is a more genuine relationship? Finding this is essentially about determining which function is better, and that is the goal of this learning process. Now, how do we find this? Typically, it can vary depending on the purpose of our analysis. Are we analyzing this for the sake of prediction, meaning that when a new x value is given, we want to better match the y value? Or do we have a goal of interpreting the relationship in the model? It depends on that. However, modern artificial intelligence tends to focus more on prediction.

We are placing more emphasis on this. So, if we take a closer look at the problem of learning, we see that there are countless x and y values in a given population. There exists a true f that represents the actual relationship between these two within this population. However, since this population contains an infinite amount of data, we cannot observe all of it. Therefore, we collect what is commonly referred to as data. The process of collecting data involves random sampling, where we select a finite number of samples randomly. Given that we are considering an infinite amount of data, it is impossible to observe all of it directly. Thus, we obtain a random sample, usually consisting of n elements.

We extract finite data that is represented. Then, for some x, there is a corresponding y, and in Excel, we can find various wines. So, usually, when we look at this data from the perspective of analyzing it, the collection of this data typically happens in the past. Right? And currently, through a learning process, we are trying to match a true function f with our estimated function F. This is what we refer to as estimation. We are looking for an estimated function f hat. The term hat indicates that if the true value is f, then f hat is what we have estimated. If the true value is a, then a hat is what we have estimated from the data. Therefore, we commonly use the notation with hat. So, we are trying to find f hat. Now, we proceed to find f hat.

When considering what kind of F hat we should look for, it's obvious that we need to find something that is most similar to f. So how can we determine this similarity? If we input some x into f, we get the true f, and this gives us the true y. However, since we do not know f, we look for F hat. By inputting the same x, the output we get is the estimate of the true y, which we call y hat. The difference between y and y hat represents the error. Therefore, minimizing this error is what leads us to the best F hat. To minimize this, we can use various mathematical methods, but if you have learned about probability, we can utilize the concept of expectation. I won’t go into detail about what expectation is here, but we typically think of it in this way.

What this means is that we think of it as minimizing the internal error. We want to minimize the average error as much as possible. However, minimizing the error itself is a bit challenging because the error can be both positive and negative. When you average the two, it converges to zero, which means it will always be minimized. That's not the approach we take; instead, we use the square of the error. 

So, the dataset we use to find this model, denoted as F hat, is usually referred to as the training data. In English, it is called training data because there is a concept of training the model. Our goal is to find something that minimizes the error across the entire population from this training dataset. However, there is a problem here. The issue is that when we actually create something, for example, regardless of how we use it, we have F s.

We created this to see whether this FN truly minimizes errors in the population or not. However, when we tried to measure the errors, we found it impossible to do so. The reason is that we cannot observe the entire population. If we could see it, we would have already trained on it. Since we cannot see it, we have extracted what we call a training set, and we used only this to find our model. However, we cannot determine how good it is or whether it is truly the optimal FN because we cannot look at it again. So, the question arises: what should we do? One method is to take the training set we have already extracted and measure the errors against it. This way, we can measure the errors from our training.

Although we referred to it as ms, it's simply an error; it's a training error, which we usually call. When we measure this training error, we might wonder if that is the correct approach, but it is well-known that measuring the training error is not the right method. We will explain this later, but the reason is that the way we usually look for models is by finding the one that minimizes the error on the training data. Therefore, all the models we find are optimal for the training dataset. However, whether these models are truly optimal for the entire population is a completely different question. Hence, the common approach people use is to create an additional dataset, independent of the training dataset, which we refer to as the evaluation dataset. So,

We evaluate how good the estimated model we found in the training data is by putting it into the evaluation dataset. So, we don't actually see the population, but we create a model from two data points that come from the population. Then, we assess how well this model performs on a new dataset, going through the process of actually evaluating the training. The problem with learning is that we want to find fx that minimizes the error in the evaluation data, but fl is only found from the training data. This is the goal we aim to achieve. If we look at the temporal setting on the board, if we are trying to create a model right now, the training set we will use consists of data that has already been collected in the past, and we are training the model with that data. Then the model...

Once everything is trained, we will gather the dataset again to create actual evaluation data, which will be used for assessment. Typically, training data is created from previously collected data, measurements are taken at the current point in time, and then a model is built based on that. Once the model is completed, it will be evaluated using test data collected from a future point in time, ahead of the current moment. This process goes through these three stages. Therefore, conceptually, I mentioned that the training set is created first, then the model is trained, and finally, the test set is collected and created in the future. However, in reality, it is quite challenging to gather data twice. Generally, data is collected only once.

Because watching it twice is very difficult, what we usually do is gather data from the past once. After that, we divide this data into two parts, and we think of one part as if it hasn't been gathered yet. So, we take only a portion of it to create a training set for the model to learn from. The remaining part, which we have already gathered in the past, is treated as if it hasn't been gathered yet. This way, it will be used later, as if it were gathered in the future. The reason is that all processes here are essentially the same. It's just like rolling a die or drawing a ball; the process is fundamentally identical. The only difference is whether we did it in the past or will do it in the future. So, we do everything in the past and simply act as if we haven't done it.

If you don't do it, you won't do it, right? If you don't see it, you won't do it. So, thinking of it as something that can be achieved by not seeing it is a bit more realistic. Therefore, when we consider what kind of model is the most important, the concept of model complexity comes into play. Model complexity, also known as model complexity, is not precisely defined, but generally refers to the range of patterns that a model can express. In other words, if a model can represent a variety of patterns in the data, then it is considered complex. Conversely, if a model cannot express simple forms of data, then it is regarded as simple. Although model complexity is not quantitatively well-defined, we can compare models with each other.

Through comparison, we can qualitatively or relatively verify certain aspects. In other words, if one model can express a certain pattern of data and another model can express that same pattern as well as others, then the first model is considered more complex. The complexity of such models is often referred to in terms of flexibility or degrees of freedom. 

For example, consider a quadratic function and a linear function. If we were to ask which of the two is the more complex model, it is clear that the first model (the quadratic function) is more complex. This is because the first and second models cannot express linear data patterns. However, if we set a in the first model f1 to 0, it becomes bx + c, which means that this model can also express linear patterns.

If \( a \) is not zero, then such a curve-shaped model can also be expressed. This means that since it can represent both straight lines and curves, \( f_1 \) becomes a more complex model than \( f_2 \). Similarly, as I will explain later, the linear regression model we usually discuss represents only straight lines, whereas the artificial neural network model will be examined and found to be generally more complex. Therefore, assessing the complexity of these models plays a crucial role in determining which model we use. Generally, when we talk about complex data, we mean that the representation of the data and the patterns in which the data is distributed are very intricate. Thus, to learn from complex data, a more complex model is necessary. However, in practice, when we evaluate the data...

When we evaluate a model with data, we can understand that a more complex model is not necessarily better. This can be represented by the performance of a model based on its complexity. Generally speaking, as we mentioned, the more complex a model is, the better its performance on training data; the error consistently decreases. However, the overall performance, which is ultimately evaluated, refers to the population's performance. Performance on evaluation data does not always follow this trend. This means that there exists an optimal model that is neither too complex nor too simple. For example, let's look at this data. Here, we have blue and red data points, and our goal is to find an appropriate boundary that can separate the blue and red points.

However, if we use a very simple model, such as a linear model, we would draw a boundary for judgment, saying that everything on this side is red and everything on that side is blue. As you can see, while we mostly get it right on one side, there are errors occurring in certain areas. On the other side, we also get most of it right, but errors still happen in some places. As you can see, a simple linear model cannot classify this data well, no matter how we draw the line. The errors always tend to increase because the model is too simple to capture the complexity of the data. For example, if you look closely, this data shows a certain complexity.

The lines separate the two data points in a curved shape, which reflects the complexity of the data. However, the boundary we use for judgment is a straight line, which is too simple. As a result, the error is significant in the data we are currently examining, and when other general data comes in, the error increases as well. Ultimately, if you look at a graph, this represents the model's complexity. Here, we have low complexity and high complexity, and this represents the error. In this case, the model's complexity is too low. As you can see, the blue graph represents the training dataset, while the other graph represents the evaluation dataset. If the model is too simple and has low complexity, it cannot capture the patterns inherent in the data. Consequently, a phenomenon occurs where the error increases in both the training and evaluation datasets across all fields.

This phenomenon is referred to as underfitting, which in Korean is called 과소 적합. However, the term underfitting seems much more natural. This occurs when a model is too simple, resulting in poor performance on both training and evaluation data. To address this issue, we can use increasingly complex models. As we increase the complexity, the model transitions from a straight line to a more wavy form, leading to a decrease in error for both the training and evaluation data. Eventually, we reach a point where both the training data error is low and the evaluation data shows good general performance. So, perhaps at this point...

Here, when you look at it, you might see a black dot that appears as a line. However, if we increase the complexity of the model, what happens is that the boundaries of judgment become more and more wavy. This means that it starts to distinguish the data in a way that becomes increasingly intricate, trying to match each piece of the given data. As it matches all the provided data, the performance on the training data continues to decrease. However, when new data comes in, there is no guarantee that this point will necessarily be a blue dot; it could be a red dot. And over here, there might be a new red dot.

It can happen that, in such cases, the model does not make accurate predictions. When the model becomes too complex, it leads to an increase in errors when new data is introduced. Therefore, if the model is overly complex, it fits the training data too well, resulting in poor predictions for new data. This phenomenon is referred to as overfitting. Overfitting occurs when the model is so complex that it performs excellently on the training data but poorly on evaluation data. Our goal is to find a model with an appropriate level of complexity, neither underfitting nor overfitting. If we can see the evaluation data at this point...

It can be very easy to live, but the key issue becomes that we do not look at the evaluation data and only continuously reduce the training data. So, to summarize again, our goal is to find the relationship f between the true population and y within a certain population. To achieve this, we create a training dataset, which contains finite data of x and y. From this data, we determine which model is the best, and this decision-making process is referred to as model selection. The criterion for selecting a model during this model selection process is to minimize the error on the evaluation dataset that we have not yet seen. Thus, we aim to find such a function F hat.

You can now think of this as our purpose. This is the overall process of the learning we are trying to undertake, and it represents the goal of our learning. This becomes a fundamental aspect of machine learning. When we classify machine learning, we can broadly categorize it into three types. The first is supervised learning, which we refer to as supervised learning. What we have explained so far is essentially the concept of supervised learning. In this case, the data's y is provided. As I mentioned earlier, everything you have seen so far pertains to supervised learning. The core issue in supervised learning is finding a function f that predicts y based on the given x, where both x and y are provided. Therefore, the problem of finding a model that accurately predicts or explains y is typically what we deal with in this context.

This is supervised learning. Therefore, this supervised learning is the main topic of this course and is also a problem that is mainly addressed in many practical applications. Next, there is another major category called unsupervised learning. So, what is unsupervised learning? It often occurs when the output variable (y) is not specified or when there is no interest in it. There are also cases where the interest lies in the patterns of the input variables (x) themselves. We refer to such cases as unsupervised learning. While unsupervised learning can be important on its own, it typically appears as a preliminary process for examining data before moving on to supervised learning. To perform unsupervised learning, representative methods such as clustering analysis and dimensionality reduction are used, which will also be covered in this course. Additionally, while it may not fall under the major categories of machine learning classification, one of the subcategories that we often evaluate is semi-supervised learning.

학습의 중간 정도에 해당합니다. 하지만 이는 우리가 보통 이야기할 때 지도 학습, 즉 기계 학습의 주요한 분류에 속하지 않습니다. 왜냐하면 주요한 분류인 지도 학습과 비지도 학습의 성격을 모두 갖고 있기 때문입니다. 그럼 세 번째는 무엇일까요? 지도 학습과 기계 학습의 큰 분류 중 세 번째는 바로 강화 학습입니다. 앞서 설명한 Reinforcement Learning이라 부르는 것인데요, 이는 지도 학습과 비지도 학습과는 약간 다릅니다. 약간 다르게 설명하자면, 주어진 상황에서 나중에 강화 학습에 대해서도 조금 다룰 예정이지만, 지도 학습과 비지도 학습은 기본적으로 사용하는 데이터가 임의로 관측된 데이터를 사용합니다. 하지만 강화 학습은 그렇지 않습니다. 강화 학습은 전략적으로 선택된 데이터를 사용하여 학습하게 됩니다.

What this means is that, fundamentally, reinforcement learning is the same. In reinforcement learning, when some input x is given, the core idea is to match it to some output y through a function f. However, the actual goal of reinforcement learning is not to find this function f, but rather to derive the input x that leads us to the desired output y. That is the essence of reinforcement learning. 

For example, when we play Go, what is x? It is the moves we make. If we make certain moves, then what is y? It is victory or defeat. Naturally, the way we make our moves will determine the outcome of victory or defeat through some function f. So, while this is the case, reinforcement learning is not just about this; it is fundamentally about how we can create x to achieve the result of y, which is winning. That is the basic approach that reinforcement learning pursues. That is the goal.

To do this, if we randomly consider all the possible combinations of x and whether we won or lost, we can find f through this method, which would be supervised learning. However, instead of finding it that way, we design what y will be when we input a certain x and provide that as data. This leads to the idea that we are using strategically selected data. So, we can classify machine learning into three categories. Next, I will briefly explain the procedure of machine learning. The first step in machine learning is to define the problem. Defining the problem essentially means determining what the dependent variable y is. Once y is determined, we then look at the related data.

We collect data to think about what x can influence y. We will now gather data for both x and y, resulting in hundreds of data points. However, in traditional analysis, this data was not available. When we set up the problem, it was generally the case that we designed experiments to collect this data and performed these experiments to gather it. But nowadays, in big data analysis, we collect a lot of relevant data from existing databases. The data is already available in these databases. By extracting data from these databases, we now gather our data. Therefore, traditional analysis methods and big data analysis differ slightly. Once the data is collected, the next step is to undergo a process called exploratory data analysis. Exploratory data analysis fundamentally involves examining the data.

EDA, or Exploratory Data Analysis, is the process of learning about the data and understanding its characteristics. When we first collect data, we often don't have a clear idea of what it actually looks like; we lack intuition about it. Therefore, the process of gaining a rough understanding of the data is precisely what exploratory data analysis involves. This process includes visualization, exploring missing values and outliers, and also encompasses data preprocessing, often referred to as data wrangling. After this stage, we move on to more formal data analysis. Formal data analysis typically follows exploratory data analysis, and once we preprocess the data into a more usable format, we usually start with what we call clean data—data that is neat and tidy.

I think about it. So the first thing to do here is to separate the training set into the training data and evaluation data, as mentioned earlier. This is necessary because we need two sets of data. Ideally, the process should involve collecting the data once, then using it for evaluation, creating the model, and finally gathering more data to use as evaluation data. However, saying we need to collect data twice means we have to go through the data collection process two times, which is quite challenging. This is a very difficult process, so usually, data is collected once and then divided into two parts, treating one part as unseen data. Therefore, after collecting the data, we split it into two sets, and initially, we only use the training dataset to create the model. After that, if the data is too abundant, unnecessary data may be excluded.

After removing the dependent variable, we select a few candidates for the learning model. Later, I will explain the process in more detail, but we typically choose about four or five models. For example, we might consider whether a linear model is better or if a more complex model would be preferable. After selecting several algorithms, we conduct tests on them. Among these candidate models, we use a validation technique to ultimately select one model. This selection is made without having seen the evaluation data. Once we choose a model, it undergoes various validation processes to determine if it is indeed the best option we can conceive of. If we conclude that this model is truly outstanding, we then use the test set to evaluate its final performance. This is how data analysis is completed. After that,

This means that, from our perspective, data analysis has come to a conclusion. However, in reality, we are now looking at how to apply this in a specific field. This is where the model is developed, leading to the creation of an artificial intelligence service. After that, we will apply it in the field and test it in practice. At this point, we will be using a completely new dataset that we have not seen before, evaluated not by developers but by actual users. This is where true evaluation takes place. Through this process, machine learning is carried out, and we typically refer to this more concretely from a business perspective as CRISP-DM. CRISP-DM is a standard for data mining or data analysis developed by IBM.

약자는 크로스 인더스트리 스탠다드 프로세스 프로 데이터 마인이라고 보통 이야기합니다. 보시면 대략 제가 앞서 설명드린 것과 굉장히 유사합니다. 비즈니스 언더스탠딩이란 말은 우리가 어떤 사업에서 풀고 싶은 문제가 있을 때, 그 문제를 이해하는 것이 먼저라는 것입니다. 그게 바로 우리가 무엇을 y로 해야 할지를 찾는 문제와 유사합니다. 그 다음에는 데이터 언더스탠딩이라는 표현이 나오는데, 이는 우리가 실제로 갖고 있는 데이터가 무엇인지에 대한 것입니다. 실제로 갖고 있는 데이터가 무엇인지 파악하는 것이 중요합니다. 이 과정에서 우리는 x라고 하는 데이터를 찾아서 모으고, 이를 실제로 사용할 수 있는지를 비교하게 됩니다. 그래서 두 가지 사이를 왔다 갔다 하면서 우리가 풀고자 하는 문제가 데이터적으로 풀 수 있는 문제인지 생각해 보게 됩니다.

After that, we will actually prepare the data. This process can be seen as the data collection phase, and during this data preparation process, we will also perform exploratory data analysis (EDA). As a result, the data will be established. Following this, we will move into the actual analysis phase and begin modeling. By modeling, we will create a model using the training data without looking at the evaluation data. After selecting several candidate models based on the methods mentioned earlier, we will finalize one of these models, which will yield an 'f'. Then, we will input this 'f' into the final evaluation dataset to assess the model's performance. If good results come from this evaluation, then...

In reality, deployment means applying it in the field, in actual business. This is where the so-called field test takes place, and the build test is conducted. If the evaluation is conducted and the results are not good, then the model has failed. Since it has failed, we need to go back and restart the process of developing the model, going through the steps again. This is what we commonly refer to as the CRISP-DM process, which can be said to be very similar to the process I mentioned earlier. So, up to this point, we have discussed machine learning, its general concepts, what the goals of machine learning are, and then the general...

데이터 학습과 지능 1 2

데이터 학습과 지능 1 2

Scripts