The rocks can talk, but they can also read; they can see, and now they can use a computer, browsing the web, clicking buttons, typing text all by itself. The age of AI agents is here. One of the first out the gate is Claude, a computer user, Anthropic's brand-new AI agent. Let's dive

into how it works, what it can do, and how it may change AI forever. [Music] In October, Anthropic made waves when it released a set of upgraded models: Claude 3.5, Haiku, and a new 3.5 Sonic. They also released something special: Claude, computer use,

but they're not the only ones in the space. We already know Sam Altman is working to recreate Samantha from the movie *Her*, and OpenAI is said to be releasing its own agent, Operator, in the new year. Google is working on something similar, too. The landscape for AI agents

is growing fast, and so far, Anthropic is the first of the big AI labs to get into the game. Right now, Claude, computer use, is still in public beta as developers put it to the test, but already it's looking like a complete game-changer. So,

how does it work? Claude had the ability to understand images for a while, so the next step was to train it on how and when to perform specific actions like clicking buttons or writing text based on what's displayed on the screen. Claude has had, for a long time since

Claude 3 back in March, the ability to analyze images and respond to them with text. The only new thing we added is that those images can be screenshots of a computer, and in response, we train the model to give a location on the screen where you can click and/or buttons on the

Keyboard shortcuts you can press in order to take action. And it turns out that, with actually not all that much additional training, the models can get quite good at that task. It's a good example of generalization. For this, Anthropic needed to train Claude to recognize exact

locations on the screen, down to the pixel. Anthropic was then able to train Claude to understand what's happening on screen and to reason about how it should use its software tools to do tasks. For example, it might help you automate boring and repetitive tasks. Claude's going

to start taking screenshots of my screen and quickly realizes that the ant equipment company isn't actually in the spreadsheet. Luckily, we get a search match, and Claude then starts scrolling through the page, looking for all the information it needs to fill out this form to get started with computer use.

Developers have to run it in a virtual machine or container like Docker. You'll also need an Anthropic API key. Once that's all set, you can then open a dedicated browser window, which shows the user prompt on the left and Claude's activity on the right. Claude starts by

analyzing the prompt and deciding which tool to use. As it works, it takes a screenshot at each step to check its progress, making sure the task is on track. If adjustments are needed, Claude loops back to try different actions or tools until it completes the task. This

repeatable loop of deciding, evaluating, and acting is called the agent loop, and it's how Claude handles complicated, step-by-step tasks all on its own. So what else can computer use make possible? In their own demos, Anthropic shows us a few different tasks like this one of Claude

Helping to plan a sunrise hike at the Golden Gate Bridge, it searches the web, figures out some important details, and then creates an event in Google Calendar. In another example, Wharton Professor Ethan Mollick puts Claude's computer use to the test by feeding it a video of

a construction site and prompting Claude to monitor the site and look for safety issues. You'll see Claude takes screenshot after screenshot, analyzing different parts of the site, making note of all the gear and materials and trying to spot any potential issues. It even finishes up by putting everything

together in a nice, neat spreadsheet—an automated OSHA compliance check. By now, it should be clear that computer use is a step forward for AI. Up until now, developers have had to make tools to fit the model, coming up with custom environments where AIs use specially

designed tools to do various tasks. Now we can make the model fit the tools. That's a powerful change. Computer use opens up so many applications. Businesses can automate repetitive tasks and increase efficiency, while the average user can save time on routine things like booking flights or ordering

food. It's easy to see a future where AI agents handle most of the drudge work for us. And for developers, computer use massively lowers the barriers to entry. LLMs have already made tasks like coding way more accessible to the average person, and computer use takes that a

whole step further. Computer use is still a work in progress, so it has some bugs and limitations. It's much slower than typical models and has a tendency to crash from time to time, so reliability is still an early concern. Occasionally, Claude will misstep in its tool

Selection can get confused or even sometimes veer off task. During one session that Anthropic shared on YouTube, Claude unexplainably started searching for pictures of Yellowstone National Park out of nowhere, in the middle of its task. To be fair, humans get distracted and sometimes do that too. Claude does have

guardrails, since it could easily be used for abuse. It steers clear of things like account creation or content generation for social media. It's also vulnerable to prompt injection, a security risk where the model can be tricked into following different information or prompts embedded in the online

sources it visits, rather than sticking to the original prompt. Imagine a website prompt injecting Claude to upload the contents of your password manager; that'd be bad. Anthropic thought about this and tries to keep users safe by keeping actions contained to a secure virtual machine, limiting access to sensitive data and

strictly controlling approved sites. However, many of these limitations could be lifted soon because this beta is just the beginning. Anthropic's already said that computer use will rapidly improve to become faster, more reliable, and more useful for the tasks users want to complete. Plenty of startups are getting

into the mix too. Just recently, a YC company, Kura, released their own browser agents that seem to outperform Claude's computer use on the Web Voyager Benchmark, achieving a new state-of-the-art. In the near future, LLMs with the full ability to use and

control computers will reshape everything: how developers write software, how CEOs run their companies, and even how we all live our daily lives. Each new groundbreaking application will transform how we work, connect, and live. This kind of AI won't just be an assistant; it'll take on entire tasks.

that once needed whole teams or companies, so what will you build with computer use? [Music]

Anthropic’s Claude Computer Use Is A Game Changer | YC Decoded

Videos to help you build a successful startup. Subscribe for startup advice, founder stories, and a look inside Y Combinator.

What is Y Combinator?
We invest $500,000 in every startup and work intensively with the founders for three months. For the life of their company, founders have access to the most powerful community in the world, essential advice, later-stage funding and programs, recruiting resources, and exclusive deals. 

Visit ycombinator.com to learn more.

Video Team:
Zach Both - Video and Content Lead
Ryan Loughlin - Senior Producer



Y Combinator

Introduction to Claude, Anthropic's AI Agent

Claude, an AI agent, can browse the web, click buttons, and type text.

Anthropic released upgraded models, including Claude, a computer user AI agent.

Other companies like OpenAI and Google are also developing similar AI agents.

How Claude Works: Image Understanding and Action Execution

Claude's ability to understand images was enhanced to perform actions like clicking buttons and writing text based on screen displays.

Training involved teaching Claude to recognize exact locations on the screen and reason about using software tools.

The model's ability to generalize to new tasks with minimal additional training is highlighted.

Example of automating tasks: filling out a form by using screenshots and web searches.

The process involves analyzing prompts, selecting tools, taking screenshots to check progress, and repeating the loop until the task is complete.

Examples of tasks: planning a hike, monitoring a construction site for safety issues.

Computer use allows models to fit the tools, unlike previous approaches where tools were made to fit the model.

Potential applications include automating business tasks and assisting average users with routine tasks.

Current limitations: slow speed, occasional crashes, and potential for errors in tool selection or task completion.

Security concerns: vulnerability to prompt injection and the need for guardrails to prevent misuse.

Anthropic plans to improve speed, reliability, and functionality in future updates.

Other startups are also developing similar browser agents.

LLMs with computer use capabilities will significantly impact software development, business operations, and daily life.

This technology will automate complex tasks previously requiring teams or companies.