The rocks can talk, but they can also read; they can see, and now they can use a computer, browsing the web, clicking buttons, typing text all by itself. The age of AI agents is here. One of the first out the gate is Claude, a computer user, Anthropic's brand-new AI agent. Let's dive

into how it works, what it can do, and how it may change AI forever. [Music] In October, Anthropic made waves when it released a set of upgraded models: Claude 3.5, Haiku, and a new 3.5 Sonic. They also released something special: Claude, computer use,

but they're not the only ones in the space. We already know Sam Altman is working to recreate Samantha from the movie *Her*, and OpenAI is said to be releasing its own agent, Operator, in the new year. Google is working on something similar, too. The landscape for AI agents

is growing fast, and so far, Anthropic is the first of the big AI labs to get into the game. Right now, Claude, computer use, is still in public beta as developers put it to the test, but already it's looking like a complete game-changer. So,

how does it work? Claude had the ability to understand images for a while, so the next step was to train it on how and when to perform specific actions like clicking buttons or writing text based on what's displayed on the screen. Claude has had, for a long time since

Claude 3 back in March, the ability to analyze images and respond to them with text. The only new thing we added is that those images can be screenshots of a computer, and in response, we train the model to give a location on the screen where you can click and/or buttons on the

Keyboard shortcuts you can press in order to take action. And it turns out that, with actually not all that much additional training, the models can get quite good at that task. It's a good example of generalization. For this, Anthropic needed to train Claude to recognize exact

locations on the screen, down to the pixel. Anthropic was then able to train Claude to understand what's happening on screen and to reason about how it should use its software tools to do tasks. For example, it might help you automate boring and repetitive tasks. Claude's going

to start taking screenshots of my screen and quickly realizes that the ant equipment company isn't actually in the spreadsheet. Luckily, we get a search match, and Claude then starts scrolling through the page, looking for all the information it needs to fill out this form to get started with computer use.

Developers have to run it in a virtual machine or container like Docker. You'll also need an Anthropic API key. Once that's all set, you can then open a dedicated browser window, which shows the user prompt on the left and Claude's activity on the right. Claude starts by

analyzing the prompt and deciding which tool to use. As it works, it takes a screenshot at each step to check its progress, making sure the task is on track. If adjustments are needed, Claude loops back to try different actions or tools until it completes the task. This

repeatable loop of deciding, evaluating, and acting is called the agent loop, and it's how Claude handles complicated, step-by-step tasks all on its own. So what else can computer use make possible? In their own demos, Anthropic shows us a few different tasks like this one of Claude

Helping to plan a sunrise hike at the Golden Gate Bridge, it searches the web, figures out some important details, and then creates an event in Google Calendar. In another example, Wharton Professor Ethan Mollick puts Claude's computer use to the test by feeding it a video of

a construction site and prompting Claude to monitor the site and look for safety issues. You'll see Claude takes screenshot after screenshot, analyzing different parts of the site, making note of all the gear and materials and trying to spot any potential issues. It even finishes up by putting everything

together in a nice, neat spreadsheet—an automated OSHA compliance check. By now, it should be clear that computer use is a step forward for AI. Up until now, developers have had to make tools to fit the model, coming up with custom environments where AIs use specially

designed tools to do various tasks. Now we can make the model fit the tools. That's a powerful change. Computer use opens up so many applications. Businesses can automate repetitive tasks and increase efficiency, while the average user can save time on routine things like booking flights or ordering

food. It's easy to see a future where AI agents handle most of the drudge work for us. And for developers, computer use massively lowers the barriers to entry. LLMs have already made tasks like coding way more accessible to the average person, and computer use takes that a

whole step further. Computer use is still a work in progress, so it has some bugs and limitations. It's much slower than typical models and has a tendency to crash from time to time, so reliability is still an early concern. Occasionally, Claude will misstep in its tool

Selection can get confused or even sometimes veer off task. During one session that Anthropic shared on YouTube, Claude unexplainably started searching for pictures of Yellowstone National Park out of nowhere, in the middle of its task. To be fair, humans get distracted and sometimes do that too. Claude does have

guardrails, since it could easily be used for abuse. It steers clear of things like account creation or content generation for social media. It's also vulnerable to prompt injection, a security risk where the model can be tricked into following different information or prompts embedded in the online

sources it visits, rather than sticking to the original prompt. Imagine a website prompt injecting Claude to upload the contents of your password manager; that'd be bad. Anthropic thought about this and tries to keep users safe by keeping actions contained to a secure virtual machine, limiting access to sensitive data and

strictly controlling approved sites. However, many of these limitations could be lifted soon because this beta is just the beginning. Anthropic's already said that computer use will rapidly improve to become faster, more reliable, and more useful for the tasks users want to complete. Plenty of startups are getting

into the mix too. Just recently, a YC company, Kura, released their own browser agents that seem to outperform Claude's computer use on the Web Voyager Benchmark, achieving a new state-of-the-art. In the near future, LLMs with the full ability to use and

control computers will reshape everything: how developers write software, how CEOs run their companies, and even how we all live our daily lives. Each new groundbreaking application will transform how we work, connect, and live. This kind of AI won't just be an assistant; it'll take on entire tasks.

that once needed whole teams or companies, so what will you build with computer use? [Music]

Anthropic’s Claude Computer Use Is A Game Changer | YC Decoded

Videos to help you build a successful startup. Subscribe for startup advice, founder stories, and a look inside Y Combinator.

What is Y Combinator?
We invest $500,000 in every startup and work intensively with the founders for three months. For the life of their company, founders have access to the most powerful community in the world, essential advice, later-stage funding and programs, recruiting resources, and exclusive deals. 

Visit ycombinator.com to learn more.

Video Team:
Zach Both - Video and Content Lead
Ryan Loughlin - Senior Producer



Y Combinator

앤트로픽의 AI 에이전트 클로드 소개

AI 에이전트인 Claude는 웹을 탐색하고 버튼을 클릭하고 텍스트를 입력할 수 있습니다.

앤트로픽은 컴퓨터 사용자 AI 에이전트인 클로드 등 업그레이드된 모델을 출시했습니다.

OpenAI 및 Google과 같은 다른 회사에서도 유사한 AI 에이전트를 개발하고 있습니다.

Claude는 현재 공개 베타 버전입니다.

Claude의 작동 방식: 이미지 이해 및 작업 실행

클로드의 이미지 이해 능력이 향상되어 화면 디스플레이에 따라 버튼을 클릭하고 텍스트를 작성하는 등의 작업을 수행할 수 있게 되었습니다.

교육에는 클로드에게 화면의 정확한 위치를 인식하고 소프트웨어 도구 사용에 대해 추론하는 방법을 가르치는 것이 포함되었습니다.

최소한의 추가 교육만으로 새로운 업무에 일반화할 수 있는 모델의 능력이 강조됩니다.

작업 자동화의 예: 스크린샷과 웹 검색을 사용하여 양식 작성하기.

클로드의 에이전트 루프 및 애플리케이션

이 프로세스에는 프롬프트를 분석하고, 도구를 선택하고, 스크린샷을 찍어 진행 상황을 확인하고, 작업이 완료될 때까지 이 과정을 반복하는 것이 포함됩니다.

작업의 예: 하이킹 계획 세우기, 건설 현장의 안전 문제 모니터링.

컴퓨터를 사용하면 도구가 모델에 맞춰 제작되던 이전 방식과 달리 모델을 도구에 맞출 수 있습니다.

비즈니스 작업 자동화, 일반 사용자의 일상적인 작업 지원 등 다양한 용도로 활용될 수 있습니다.

현재 제한 사항: 느린 속도, 가끔 발생하는 충돌, 도구 선택 또는 작업 완료 시 오류 발생 가능성.

보안 문제: 프롬프트 주입에 대한 취약성 및 오용 방지를 위한 가드레일 필요성.

향후 업데이트를 통해 속도, 안정성, 기능을 개선할 계획입니다.

다른 스타트업도 유사한 브라우저 에이전트를 개발하고 있습니다.

컴퓨터 사용 능력을 갖춘 LLM은 소프트웨어 개발, 비즈니스 운영 및 일상생활에 큰 영향을 미칩니다.

이 기술은 기존에 팀이나 회사가 수행해야 했던 복잡한 작업을 자동화합니다.

Anthropic’s Claude Computer Use Is A Game Changer | YC Decoded

Anthropic’s Claude Computer Use Is A Game Changer | YC Decoded

스크립트