NVIDIA Jetson Orin: Micro Supercomputer Runs LLMs at the Edge for $249

Nvidia's Jetson Orin, a micro supercomputer, brings powerful, affordable edge computing for large language models and robotics.

### Nvidia's Game-Changing Jetson Orin: The Future of AI at the Edge
* 🚀 Nvidia has unveiled a revolutionary supercomputer, the **Jetson Orin**, which is **compact, affordable**, and capable of delivering **large language models directly to the edge**, allowing for **open-source models to run independently anywhere**.
* 🏡 In a casual demonstration, Nvidia's CEO showcased the Jetson Orin, highlighting its **impressive performance of nearly 70 trillion operations per second** at just **$249**, making it accessible for various applications.
* 🤖 This micro supercomputer can operate **without a cloud connection**, enabling artificial intelligence to be integrated into robots, Internet of Things devices, and vehicles, thereby **enhancing their capabilities significantly**.
* 🔍 Emergence AI partnered with Nvidia to demonstrate their enterprise-grade multi-agent orchestrator, which allows multiple agents to **automate complex web interactions**, increasing efficiency and **reducing the need for human intervention**.
* 🔒 Emergence AI emphasizes **privacy and security**, offering a fully hosted solution accessible via **API or self-hosted on a virtual private cloud**, making it suitable for **enterprise automation needs**.
* 💡 The Jetson Orin's specifications include 1,284 CUDA cores, 32 Tensor cores, and a six-core ARM Cortex CPU, providing **high performance with efficient power consumption of only 25 watts**.
* 📈 With its ability to run **275 trillion operations per second** and support for a variety of models, the Jetson Orin represents a **significant leap in technology**, promising to **transform the future of robotics and AI applications**.
* 🎉 Nvidia's innovation could lead to a future where most use cases can be handled by **models of 7 billion parameters or less**, all while maintaining **control, security, and privacy on-device**.

Nvidia just released an absolutely game-changing supercomputer, and here's the thing: it's tiny, very cheap, and has the potential to deliver large language models to the edge more than anything else that we've seen. This means open-source models running independently anywhere. Let me just have

the Nvidia CEO explain. Watch this. "Hi, welcome to my house. I'm living in a different house now. We're fixing the house that you guys saw the last time you were in my kitchen. And let's see, what were we doing? My hair was a lot longer, and I lifted a brand new HGX out

of my oven while I'm cooking something up for you again today. And let me show it to you. Okay, here we go, ladies and gentlemen, our brand new AI computer. Look at this. I think I might have cooked it a little bit too long. It shrunk! The little tiny Jetson Nano,

little Orin computer. This thing that's really amazing is that, um, a long time ago, starting with Xavier, you guys might know that we created a brand new type of processor. It was a robotics processor. Nobody understood what we were building at the time, and we imagined

that someday these deep learning models would evolve and we would have, you know, robots of all kinds. Everything that moves would be robotic. And, and now here we are. Uh, we're seeing all kinds of amazing robots: robots on wheels, robots on legs, two legs, three legs, and of

course, general humanoid robotics are nearly upon us. This is a brand new Jetson Nano super... almost 70 trillion operations per second, 25 watts, and $249. It runs everything that the HGX does, uh, even runs large language models, and, uh, can't wait for all of you to try it."

It's available everywhere. Go get it, enjoy. Robotics, and it runs CUDA and cuDNN and TRT. You could create an agentic AI that reasons and plans, so you can use it for a robot, you can use it for a workstation. It's an incredible computer. What do you

guys think? So this is the NVIDIA Jetson Orin. This is a micro supercomputer. It is blazingly fast at running inference on the edge. The edge means that it's not connecting to the cloud. You know that I'm very bullish on edge compute. I've

talked about it in the past, as chips in our phones, chips on our computers, and chips everywhere get better and better, and large language models continue to get better and smaller, we're going to be able to run artificial intelligence pretty much anywhere. And now, with the

Jetson Orin, this tiny supercomputer that is incredibly inexpensive, we are going to be able to put large language models into essentially anything: robots, Internet of Things devices, cars, literally anything. This little device can just plug into and run inference using local models. This tiny

supercomputer can run 275 trillion operations per second. That is an insane number. Thanks to Emergence AI for partnering with me on this video. If you've watched this channel at all, you know I'm bullish on agents, and especially agents that can accomplish real-world tasks for you. That's why I'm

excited to tell you about Emergence AI. Emergence AI just launched their enterprise-grade multi-agent orchestrator, and they showed off the first demo of a real-world use case where these agents can actually browse the web on your behalf. This is enhanced web automation. This means multiple

agents can go out and dynamically interact with different elements on the web under this intelligent orchestration to bring human-like interaction and navigation, but at machine-level scale. And what's really cool about this is these agents can actually perform complex and sophisticated web interactions that previously required a

human. They can navigate dynamically loading menus, fill out forms, adjust settings, process embedded files, and extract relevant data from PDFs and HTML. Emergent AI's orchestrator offers a combination of design-time flexibility and runtime determinism, which basically means that these agents can heal

themselves. So if it makes a mistake along the way, it can figure it out and then succeed on the next try. And Emergent AI has put a huge emphasis on privacy and security. They offer a fully hosted solution where you can access it via API,

or you can host it on your own virtual private cloud. So if you're an enterprise business and you're looking to automate a lot of your processes, Emergent AI is a great solution. Integrate their agent API and seamlessly orchestrate multiple agents to accomplish tasks for you and

your business. This includes interacting with both modern and legacy enterprise applications. So, Emergent AI is just starting to invite developers to try out their platform. Definitely check them out; tell them I sent you, of course! Go to their website, emergent.ai, or simply email them: contact@emergent.ai.

I'll drop all of the links in the description below. So, thank you again to Emergent AI for partnering with me on this video. Now back to the video: this tiny supercomputer has the potential to power the robots of the future, and those robots won't need to actually connect to

the cloud to do their inference, everything is going to be on-device. I could not be more excited about that future where we have more control over the large language models that we're running: more security, more privacy, everything on-device. And look, there's always going to be a time where we need

massive models like Gemini or GPT-4, but if you've watched my channel at all, you know I believe that 99% of use cases can be accomplished by models that are 7 billion parameters or less. That is a small model and an extremely capable

model. And now we have the device to run it, and it's kind of like the Raspberry Pi in a lot of ways. It is a very small, very modular device that can be built on and customized any which way you want. This device can run almost 70 trillion

operations per second. And look at the form factor, it is tiny! This is a nearly 2x improvement over its predecessor, and here's the killer feature: it's only $249. That is extremely inexpensive. Imagine stacking four of these up for $1,000, then you're running nearly 300

trillion operations per second, and it only draws up to 25 watts, which is so efficient. It uses the NVIDIA Ampere architecture with 1,284 CUDA cores and 32 Tensor cores. It has a six-core ARM Cortex CPU; this is 64-bit, 8GB

of memory, but very, very fast memory: 102GB per second. It has an SD card slot, so you just load up the operating system you want easily. And again, all of this under $250. That is the killer innovation in my mind. Let's take a look at the

performance. So, first, let's look at the large language models. We have the baseline of the previous generation, and we have all of these open-source models from Llama 3.1, Llama 3.2, Guanaco 2.5, and 3.5.

Even though 54 is out now, all of them perform extremely well. It can power vision models. It can power vision transformers as well. This is the stuff of the future. Throw one of these into your car; all of a sudden, it can autonomously drive. Throw it into any of

your internet of things devices that you have at your house, and all of a sudden, they are powered by a large language model and become insanely intelligent. Put it into any robot, and all of a sudden, the robot becomes powered by a large language model or a vision model

or anything. These devices just churn through the math required to run intelligence. I am so impressed with this release. I plan on getting one myself. Should I create a tutorial showing how to set it up and get it running? Let me know in the comments. Super impressive.

Congrats to Nvidia on this release! If you enjoyed this video, please consider giving a like and subscribing, and I'll see you in the next one.

NVIDIA Unveils STUNNING Nano Super Computer for Only $249

Artificial Intelligence (AI), Open Source, Generative Art, AI Art, Futurism, ChatGPT, Large Language Models (LLM), Machine Learning, Technology, Coding, Tutorials, AI News, and more

** Exclusive Pinecone Partner **

Matthew Berman

Nvidia's new Jetson Orin Nano supercomputer is small, affordable, and powerful, bringing large language models (LLMs) to edge devices.

Nvidia CEO Jensen Huang introduces the Jetson Orin Nano, highlighting its small size and powerful capabilities.

The Jetson Orin Nano was developed with the future of robotics and AI in mind.

The device boasts 70 trillion operations per second, 25 watts power consumption, and a $249 price point, capable of running LLMs.

The Jetson Orin is versatile and can be used for robotics, workstations, and creating agentic AI.

The Jetson Orin enables edge computing for LLMs, allowing AI to run anywhere without cloud connection.

The increasing power and decreasing size of chips, combined with the Jetson Orin, allows LLMs to be integrated into various devices.

The device's affordability and performance make it ideal for integrating LLMs into robots, IoT devices, cars, and more.

Emergence AI's multi-agent orchestrator allows agents to perform complex web tasks autonomously.

The orchestrator enables enhanced web automation with multiple agents interacting dynamically.

Agents can handle complex web interactions, form filling, data extraction, and dynamic menu navigation.

The agents are self-correcting and prioritize privacy and security.

The Jetson Orin empowers future robots with on-device inference, enhancing privacy and security.

On-device processing provides more control over LLMs, while still acknowledging the need for larger models in certain cases.

Most use cases can be handled by smaller, efficient models, which the Jetson Orin can run effectively.

The Jetson Orin is small, modular, and customizable, similar to a Raspberry Pi.