Free Open-Source Secure Computer Use with Gemini 2.0 Flash: AI Agent Controls Linux Desktop

Secure Computer Use offers free, open-source AI agents controlling Linux desktops for various tasks.

### Gemini 2.0 Powered Free Open-Source Computer Control Agent
* 💻 The video introduces a free, open-source agent, "Secure Computer Use,"  that controls a computer within an e2b cloud VM, similar to Anthropic's Computer Use, but significantly cheaper.
* 💡  Unlike the original, which uses Anthropic's model, this version leverages open-source models like **OS Atlas**, **Llama 2 3B**, and requires two providers (OpenAI and Firework) for vision and LLM tasks.
* ⚙️ The tutorial demonstrates configuring the agent to use **Gemini 2.0 Flash** for free, requiring installation of FFmpeg and Poetry, cloning the repo, checking out the `os_computer_use` branch, and modifying the LLM Python file and `.env` file with API keys.
* 🏃‍♂️ The setup involves installing packages via `poetry install` and running the agent with `poetry start`, displaying a live stream of the desktop environment.
* 🔍 The agent successfully performs tasks such as searching "cats" on Google and using the calculator, showcasing its ability to interact with local applications unlike browser-only alternatives.
* ⚠️ While functional, the agent's interaction with local apps is less polished than its Anthropic counterpart, occasionally terminating, but remains a valuable free alternative.
* ✨  The agent boasts a **one-million-context window**, enabling it to handle complex tasks requiring multiple steps and a deeper understanding.
* 👍 The video concludes by highlighting the agent's free nature, strong vision capabilities, and potential for further development, emphasizing its role as a compelling open-source alternative to closed-source options.

[Music] [Applause] Hi, welcome to another video. So, Computer Use came out some time ago and worked pretty well, although it has been a lot more expensive to use. So, as with anything, multiple open-source projects came out that try to do the same and do it much better than Claude and make it quite cool to use.

But there's another agent that I saw, and it's this Secure Computer Use one. This is by e2b, and what it basically does is that it allows you to do what Anthropic's Computer Use does, but in a cloud VM that is given by e2b.

If you know e2b, then you'll know that they are widely used for their code interpreter sandbox thing, but they also recently launched their whole virtual sandbox thing for agents that want to control a computer.

So it basically gives you AI agents with Linux desktops that they can control and use to do anything. Now, this is an implementation of that by them, and if you'll see this, then you can see that this is based on the main Anthropic Computer Use thing, which means that it will be very token-heavy.

But actually, there's another version of this implementation which you can view by changing the Git branch here to OS Computer Use. Here you'll see that the implementation in this branch is different, and this is based on open-source models instead of Anthropic's model. It uses the same sandbox, but it also uses OpenAI's and Firework's as well.

Now you may ask why it needs two providers. Well, that's a good question, and I don't know why either. From what I was able to understand from the code, it uses three models: the first one is the OS Atlas model.

If you don't know about OS Atlas, then it's an open-source vision model that basically is trained to give...

It outputs pixel addresses and controls.  Its output is like a computer using pixel coordinates

to click, type, or do anything like that. But to control this, you need a proper LLM that can be asked to give the coordinates for

the tasks. So for that, it uses Llama 2 3B via Firework, as it's a great open-source model. But as you know, it

cannot do vision things. So for vision help, it uses Llama 2 to analyze images as well, via OpenRouter. So that's why it

needs two providers. But I'll tell you how you can get better results than this by changing the models to Gemini 2.0

Flash and using it technically for free. So let's get right into it and let me show you how you can make it work.

First of all, you'll need to install the stuff that it needs, which is FFmpeg and Poetry. So just get it installed by

running this command. Once done, clone the repo onto your computer. Now change the directory to 'secure_computer_use'. Once done, we'll need

to change the branch to the 'os_computer_use' branch by running the `git checkout` command. This will get the branch

changed to it. Now, first let me tell you how you can configure it to use Gemini 2.0 Flash,

because I like using it with that since it's free. So to do that, just go to this LLM

Python file. Here, first change the base URL in this to the Gemini OpenAI compatible endpoint, and in this also

change it to OpenAI compatible like this. Once that's done, also update this line to correctly work with the

OpenAI SDK. Once that's done, just go back and create an .env file. Now here, enter the

Ollama API key that you can get for free from their site. And there's no apparent limit as well, so just get it. Once done,

Just enter the Gemini API key in both variables. Once done, we can start using it. So, first of all, to use it the

first time, you'll need to make sure that you install every package by running the `poetry install` command. Then, once that's done, just run

it with `poetry start`, and it will get started. Now, this will start it, and you'll see the e2e sandbox open up here.

This is a live stream of the desktop, so you cannot click and do stuff like that. Anyway, now here you can just enter what

you want it to do. Let me show you something simple, which is also shown in their demo, and that's to search for cats

on Google. So let's ask it here. Once we do that, you'll see that it starts doing it. The live stream here is a bit delayed

than the prints on the terminal that you see, so there's that. But it's in a proper, safe, sandboxed environment, which is a

great thing. So you can see that it's doing it here. Let's wait a bit. [Music]

And it's now done. So it did this quite well, which is pretty great. Another thing that I want to try

is that I wanted to use the calculator and make a simple calculation in it. Because the best part about it is that

it can use local apps, unlike Browser Use, which was another open-source project that was really very great but could

only use browsers. Anyway, it's now doing that. Let's wait a bit. [Music] And it's now done. So it can use the

local apps quite well, so that's great. I have tried it on some other stuff as well, and it works

well. Although, because it uses local apps, it's not as polished as its Anthropic counterpart because it terminates

sometimes and stuff like that, which is fine considering that we are using it all for free. This is a great

implementation, and you can surely extend it even more if you want, as the code is also very simple to understand and

extend. Navigation is a challenging thing using it with Gemini 2.  It's also great because it's free and has really

great vision capabilities, and it also has a quite big, one-million-context window, which means that it can also do some big tasks that

may require more clicks or a deeper understanding of something. So it works pretty well and is super cool to use, and it's a really

great open-source version of the Anthropic Claude thing. It's great to see some new alternatives to closed

source things popping up that people can utilize in different ways. Overall, it's pretty cool. Anyway, let me know your thoughts in

the comments. If you liked this video, consider donating to my channel through the Super Thanks option below, or you can

also consider becoming a member by clicking the Join button. Also, give this video a thumbs up and subscribe to my channel.

I'll see you in the next video. Till then, bye. [Music]

E2B Gemini 2.0 Computer Use: This FULLY FREE Agent can Control Whole Computer & DO ANYTHING!

Facing the future with AI! 

Here you'll find content about multiple AI Tools that are actually useful and sometimes free.

AICodeKing

Introduction to Secure Computer Use and its functionality

Introduction to Secure Computer Use, an open-source alternative to Anthropic's Computer Use.

Secure Computer Use operates within an e2b cloud VM, providing AI agents with Linux desktops for control.

E2b's virtual sandbox allows AI agents to control and use computers to perform various tasks.

The initial implementation is based on Anthropic's Computer Use, making it token-heavy.

An alternative implementation uses open-source models like OS Atlas, Llama 2, and OpenRouter.

Explanation of the models used in Secure Computer Use

OS Atlas, an open-source vision model, outputs pixel addresses and controls for computer interaction.

Llama 2 3B via Firework provides the LLM for task coordination and coordinate generation.

Llama 2 is also used for image analysis via OpenRouter.

Two providers are needed due to the combination of vision and LLM capabilities.

Improved results are achieved using Gemini 2.0 Flash.

Setup and configuration instructions for Secure Computer Use

Cloning the repository and changing to the 'os_computer_use' branch.

Configuring the LLM Python file to use Gemini 2.0 Flash.

Creating an .env file with Ollama and Gemini API keys.

Installing packages and running the application using poetry.

Demonstration of Secure Computer Use functionalities

Live stream demonstration of the desktop environment.

Example task: Searching for 'cats' on Google.

Using the local calculator application for calculations.

Assessment of the application's performance and limitations.

Highlights the advantages of using Gemini 2.0 and the large context window.

Overall assessment of the application as a great open-source alternative.

Encourages viewers to share their thoughts and support the channel.