a new language model learning technique just dropped and it made a huge jump in the AGI benchmarks so I'm going to break it all down for you and I'm going to show you why this might be another lever that we can pull to scale and

potentially reach AGI let me start from the beginning before I get into the research paper it's kind of crazy to think about but the 01 family of models came out just 2 months ago and 01 unlocked something incredible by giving models the ability to think to take

their time to not give you the first thing that they predict but rather to use Chain of Thought to actually think through problems in a longer period of time whether that's seconds or minutes or hours open AI showed us that by throwing more compute at test time or

inference time models can become much more intelligent and then as I said 01 showed us well we can now scale up post-training time we can scale up during inference time and it turns out there's actually something else we can do which is kind of in the middle of training and

test time compute and it is called test time training and it is exactly like it sounds so let's take a look so I want to thank slow developer for sharing this research paper this is how I found it and we had this massive jump in the ARC

prize where this new paper was able to achieve 61.9% on the ARC prize and for those of you who aren't familiar with the ARC prize let me show that to you really quickly so I know there's a lot of

benchmarks out there and it seems like those benchmarks are just getting absolutely destroyed by new models nowadays they are passing with flying colors there are a ton of different benchmarks Math and Science and writing and reading comprehension and coding and everything in between but really how do

you define artificial general intelligence and that is what the ARC prize aimed to do and I think they've done it quite well so let me just read this really quickly ARC prize is a million dollar public competition to beat and open source a solution to the ARC

AGI Benchmark and so it is testing for AGI and how it does that is by generalization and that is artificial general intelligence so what they do is they give you a few examples which for the most part should be simple for humans to complete and then they say

Okay based on these examples how would you transform this 7x7 grid into to what the output should be so as we can see here on the left in the examples we have three blue dots in the shape of an L and then invert that and we have three blue

dots right there then the output this is the example we have a dark blue dot that essentially fills the three to make it four and a square in both so if we come over here logically it's pretty easy to figure out as a human we basically just

put a dark blue square in each of these and then that would be the right answer well it turns out large language models and artificial intelligence in general really struggle with this and it's generalization it's saying hey here's something that I'm teaching you

now use that to complete this problem now so far the top score on the leaderboard is only 42% by Ryan Greenblatt and different approaches have been taken you could basically use any approach you want as long as you open source it and show what you've done and

so 42% is the best and I will say that the average human score on these types of tests is about 60% and the best human score is nearly 98% so that kind of gives you a baseline as to where we are

with artificial intelligence and now with this new technique of test time training it was able to achieve the average human score all right so enough talk let me show you what they've actually accomplished so here's the research paper it is out of MIT and it is called the surprising effectiveness

of test time training for abstract reasoning so I read through this whole thing I highlighted the most interesting bits and that's what I'm going to talk to you about right now so it starts out in the abstract with language models often struggle with novel problems

requiring complex reasoning and by novel problems they mean hey you have this quote-unquote built-in intelligence now use that intelligence and I'm going to give you a new problem that you haven't seen but you should be able to infer how to solve based on your existing

knowledge and most of the time it can't and the ARC prize is the perfect example of that but they have shown that their new technique called test time training is very effective getting 60% and if you remember going back to the current top score of about 42% that is a drastic

improvement so what is test-time training? It basically means updating model parameters temporarily during inference. If you've been following along with AI over the last couple years, you've probably heard of LoRA. LoRA is low-rank adaptation, and it's just a really lightweight and efficient way to

fine-tune models, and that is the technique they use to basically fine-tune models during test time or during inference time. So the core concept is LoRA is a method that allows efficient customization of pre-trained neural networks by training only a small number of parameters while freezing the

original model weights. We've seen LoRAs with text-to-image models, we've seen LoRAs with large language models, so they've been around, they work really well. So they used this new test-time training against the ARC prize, and again they got amazing results. Let's keep

going. So there are three crucial components for test-time training: one, initial fine-tuning on similar tasks. So the model has to be capable in itself before the test-time training even happens. Then next, auxiliary task format and augmentations: how do we actually generate the data to then fine-tune the

model with? And three, per-instance training. And based on this technique, they have seen a 6X improvement in accuracy compared to the base fine-tuned models. For example, applied to an 8 billion parameter language model, they achieve a 53% accuracy on the ARC's

public validation set, improving the state-of-the-art by nearly 25%. And that's an 8 billion parameter model, a relatively small model, and I know I've said this a million times, small models are getting so good, especially when we have all of these other techniques that

we can throw at them I really like small models that are really efficient that I can run on my computer and so here is the gist of the problem with today's large language models they excel at performing tasks that occur in their training data so the stuff that they

were trained on and often Elementary variations or compositions of those tasks so very small differences so if a model knows 1 + 1 = 2 it probably will also know 1 + 2 = 3 but if you give it a vastly different math problem it

probably won't know how to do that but can LLMs also solve new problems involving non-trivial reasoning planning or string manipulation of a kind very different from their pre-training data and that is where test time training comes in they don't need to know these

things they could actually generate data based on these new novel problems train itself and then solve them for complex and novel tasks it is often difficult to obtain a correct answer simply by sampling from an LLM however a significant finding in recent years has been that LLM performance can be

substantially improved by augmenting LLM decoding with additional test time computation now that is not test time training test time computation is what we previously talked about with the 01 models giving it time to think using Chain of Thought and other techniques so

methods in this category include Chain of Thought which we talked about sampling with majority voting code execution and self-executing architecture giving models additional tools to think and research on their own all of which has been available using agents but now they're kind of getting built into the models

themselves, but now we have test-time training, which updates through explicit gradient steps based on test-time inputs. This method differs from standard fine-tuning as it operates in an extremely low data regime—basically very light fine-tuning—and so it can be done

at inference time, at test time. And so what they have found is this: these results challenge the assumption that symbolic components are strictly necessary for solving such complex tasks. Instead, they suggest that the critical factor in solving novel reasoning problems may be the allocation of proper computational resources during test time.

Perhaps independently of whether these resources are deployed through symbolic or neural mechanisms. So basically, again, being able to scale up compute at different points in the language model life cycle. So how does it actually work? Let me show you that. So test-time training enables parametric models to

adapt during inference through dynamic parameter updates. I know that sounds very complex, but it basically just means it's updating itself based on the problem it sees. The model leverages the test data structure to improve its prediction. So starting with initial model parameters, so the fine-tuned kind

of frozen-in-time model for each test input, so the problem, we first generate training data. Now what does that actually mean? That means that they're taking the problem and they're creating lots of different variations of it, and then using all of that training data to

fine-tune itself in a really light way using—or we then optimize these parameters to minimize a loss function, producing temporarily updated parameters for prediction. So it updates itself for that problem, and then when it gets to the next problem, it just erases all of

that and goes back to that base model and does that process again so it's very dynamic at inference time and after generating predictions the model is restored to the original parameters for the next instance or batch thus test time training trains a specialized prediction model for each test input

obtained by fine-tuning a base model on a test time data set generated from the test input so here it is this is the original test task and they give four four examples one of which they say okay solve this last one so they actually

give three complete examples and then they give the actual problem and then they're waiting for the solution then from that they drop the problem without a solution and generate more of these leave one out tasks then they use geometric transformations to change it and basically just create a lot more

training data or examples then they put all of those together and use it for the fine-tuning data but that's not it they didn't just do test time training they actually came up with other augmentation methods as well so let's take a look at

that so first augmented inference recent work has shown that scaling test time compute can significantly improve the performance of language models that's what we've been talking about with O1 one of the most common techniques is to sample multiple responses and then selecting the best response using a

ranker so come up with a bunch of different examples and we're going to have a judge determine which one is best however while sampling is very effective in domains with multiple possible solutions coding or multiple possible paths to the final answer with math it can be detrimental when generating

answers directly where there's only really one answer as there is no way to directly enforce diversity across samples while ensuring coherence within samples as an alternative to inference time scaling we use augmented inference which generates multiple prediction candidates by using geometric transformations like inversion and then

is combined with a greedy decoding scheme basically just creating a lot of different variations of the same problem to give the model a lot more knowledge they also came up with ensembling predictions or a voting strategy so this approach involves two stages of voting to progressively narrow down the best

candidates first by selecting the most frequent predictions within each transformation and then by conducting an overall vote across transformation specific candidates to identify the top two most frequent predictions so this is very common come up with a bunch of examples vote on which one is best and

they actually do that in multiple iterations all right so let's look at the scores now all of that aside what were they actually able to accomplish so what we see here is that using the TT method test time training a fine-tuned language model based on this new BART

technique BART and the program synthesizer of BART you could see that they were able to achieve a score of 61.9 beating the average human score of 60.2 now the best human 97.8 so you can see some of these new

techniques are really pushing the boundaries of what's possible using smaller models and throwing a bunch of compute at it post training so that's it that is the gist of this paper I'm going to drop a link to the paper in the description below if you want to check

it out in full this is fascinating to me and I feel like since we've essentially run out of public data that we can train models on we're really limited in how much more we can scale training time within the context of new data now of course we have synthetic data but the

other option rather than just creating synthetic data which really hasn't been proven is to be able to do a lot more with the existing data that we have and Sam Altman talked about that months ago right we either have synthetic data to continue scaling or we just do more with

the data we have and this is an extension of doing more with the data that we already have and I really think that this is going to be the key to reaching AGI according to the AI Prize if you enjoyed this video please consider giving a like and subscribe and

Strawberry 2.0 - AI Breakthrough Unlocks New Scaling Law

Artificial Intelligence (AI), Open Source, Generative Art, AI Art, Futurism, ChatGPT, Large Language Models (LLM), Machine Learning, Technology, Coding, Tutorials, AI News, and more

** Exclusive Pinecone Partner **

Matthew Berman

Introduction to Test-Time Training and its Impact on AGI Benchmarks

A new language model learning technique, test-time training, significantly improved AGI benchmarks.

The 01 family of models enabled models to think and use Chain of Thought for problem-solving.

OpenAI demonstrated that increased compute during inference improves model intelligence.

Test-time training is a technique that updates model parameters during inference.

Test-time training achieved 61.9% on the ARC prize, surpassing previous best scores.

The ARC prize is a million-dollar competition to create an open-source solution for the ARC AGI benchmark.

The benchmark tests for AGI through generalization, using examples to solve new problems.

The benchmark involves transforming grids based on given examples, a task challenging for LLMs.

The previous top score was 42%, while the average human score is around 60%.

Details of the Test-Time Training Research Paper

The MIT research paper highlights the effectiveness of test-time training for abstract reasoning.

Language models struggle with novel problems requiring complex reasoning.

Test-time training achieved a drastic improvement over the previous top score (42% to 60%).

The technique uses LoRA (low-rank adaptation) for efficient fine-tuning during inference.

Three crucial components are initial fine-tuning, auxiliary task format, and per-instance training.

Limitations of Current LLMs and How Test-Time Training Addresses Them

Current LLMs excel at tasks similar to their training data but struggle with novel problems.

Test-time training allows LLMs to generate data, train themselves, and solve novel problems.

LLM performance improves by augmenting decoding with additional test-time computation.

Test-time training updates parameters through gradient steps based on test-time inputs.

The results challenge the assumption that symbolic components are necessary for complex tasks.

Test-time training dynamically updates model parameters during inference based on the problem.

For each test input, training data is generated and used for light fine-tuning.

After prediction, the model is restored to its original parameters for the next instance.

The process involves generating variations of the problem to create more training data.

Augmentation methods like geometric transformations and ensembling predictions are also used.

Test-time training achieved a score of 61.9%, exceeding the average human score.

The technique is seen as a key to reaching AGI by doing more with existing data.

The approach is presented as an alternative to relying on synthetic data for scaling.

Strawberry 2.0 - AI Breakthrough Unlocks New Scaling Law

Scripts