Mark Zuckerberg's Vision for Generative AI and Meta AI Studio

Mark Zuckerberg discusses the evolution of AI, open source philosophy, and the future of AI Studio and smart glasses.

### Mark Zuckerberg Discusses the Future of AI and Meta at SIGGRAPH 2024
* 🤖  Zuckerberg envisions a future where **AI-generated content becomes the norm**, with tools creating content on the fly or **synthesizing existing information**.
* 🧠  He highlights the importance of **building more general recommendation models** that can span different types of content, leading to **significant quality improvements**.
* 🎨  The concept of "**Creator AI**" is introduced, allowing creators to build AI versions of themselves to interact with their audience, potentially **revolutionizing content creation**.
* 🌐  Zuckerberg emphasizes the importance of **open source philosophy**, making AI accessible and ensuring Meta's future technology is not limited by **closed platforms**.
* 👓  The discussion also touches on the development of smart glasses, seen as the **next major computing platform**, with Zuckerberg predicting **over a billion users upgrading to smart glasses**.
* 🏢  The conversation concludes with Jensen Huang praising the impact of Meta's AI technology, particularly in computer vision applications like "**segment anything**," and discussing future AI projects beyond **ray tracing**.

Jensen hang sat down with Mark Zuckerberg at sigraph 2024 and discussed a range of artificial intelligence topics. I made a super cut of it so let's check it out. I guess my first question for you is, you know, how do you see how do you see the uh uh the advances of generative AI ad meta today and how do you apply it to either enhance your operations or introduce new capabilities that you're offering with generative AI. Um, I think we're going to quickly move into this Zone where not only is is the majority of the content that you see today on Instagram you know just recommended to you from kind of stuff that's out there in the world that matches your interests whether or not you follow the people. I think in the future a lot of the stuff is going to be created with these tools too. Some of that is going to be creators using the tools to create new content. Some of it, I think, eventually is going to be content that's either created on the Fly for you.

um, or, or, or kind of pull together and synthesize through different things that out there, so that that's just one example of how kind of the core part of what we're doing is just going to evolve, and it's been evolving for for 20 years already, but I think very few people realize that that, uh, one of the largest Computing systems the world has ever conceived of is a recommender system. Yeah, I mean it's this whole, yeah, it's this whole different path, right? It's, it's not quite the kind of gen hotness that people talk about, but I think it's, it's like, as I mean, it's all the Transformer architectures, and it's a similar thing of just building up more and more General models, embedding unstructured data into features, and yeah, I mean one of the big things that just drives quality improvements is, you know, it used to be that you'd have a different model for each type of content, right? So, a recent example is, you know, we had, you know, one model for ranking and

recommending reels and another model for ranking and recommending more long-form videos. And then, you know, take some product work to basically make it so that the system can display, you know, anything in line. But, you know, the more you kind of just create more general recommendations models that can span everything, it just gets better and better. So, I mean, part of it, I think, is just like economics and liquidity of content. And the broader of a pool that you can pull from, you're just not having these weird inefficiencies of pulling from different pools. But, yeah, I mean, as the models get bigger and more general, that gets better and better. So, I kind of dream of one day like you can almost imagine all of Facebook or Instagram being, you know, like a single AI model that has unified all these different content types and systems together that actually have objectives over different time frames, right? Because some of it is just showing you, you know, what's the interesting content that you're going to be that you want to see today. But some of it is helping you build out your

network over the long term. Right people you may know or accounts you might want to follow. And these multimodal models tend to be, yeah, tend to be much better at recognizing patterns, weak signals, and such. A lot of the Gen stuff is going to, on the one hand, it's, I think, going to just be this big upgrade for all of the workflows and products that we've had for a long time, but on the other hand, there's going to be all these completely new things that can now get created. So Meta AI, um, you know, the idea of having, you know, just an AI assistant that can help you with different tasks and, um, in our world, is going to be, you know, very creatively oriented like you're saying, but I mean they're very general, so I, you don't need to just constrain it to that. It'll be able to answer any question, um, over time. I think, you know, when we move from like the Llama 3 class of models to Llama 4 and beyond, it's going to, I think, feel

less like a chatbot, where it's like you give it a prompt and it just responds, then you give it a prompt and it responds, and it's just like back and forth. I think it's gonna pretty quickly evolve to you give it an intent and it actually can go away on multiple time frames. And I mean it probably should acknowledge that you gave it an intent upfront, but I mean, you know, some of the stuff I think will end up, you know, it'll spin up, you know, compute jobs that take, you know, weeks or months or something, and then just come back to you. And like something happens in the world, I think that that's going to be really powerful. So I mean, today's AI, as you know, is kind of term-based. You say something, it says something back to you. Um, but obviously when we think, when we're given a mission or we're giving a problem, you know, we'll contemplate multiple options or maybe we come up with a, you know, a tree of options, a decision tree.

and we walk down to the decision tree, simulating in our mind, you know, what are the different outcomes of each decision that we could potentially make. And so we were doing planning, and so in the future AIS will kind of do the same. One of the things that I was super excited about when you talked about your vision of Creator AI, I just think that's a home run idea, frankly. Tell everybody about the Creator AI and AI Studio that's going to enable you to do that. Yeah, so, so we actually, I mean this is something that we're, we're, you know, we've talked about it a bit, but we're rolling it out a lot wider today. You know, a lot of our vision is that I don't think that there's just going to be like one AI model, right. I mean this is something that some of the other companies in the industry, they're like, you know, it's like they're building like one Central agent and and yeah, we, we'll have the meta AI assistant that you can

use but a lot of our vision is that we want to empower all the people who use our products to basically create agents for themselves. So, whether that's, you know, all the many, many millions of creators that are on the platform or, you know, hundreds of millions of small businesses, um, we eventually want to just be able to pull in all your content and very quickly stand up a business agent and, um, be able to interact with your customers and, you know, do sales and customer support and all that. So, the one that we're, that just starting to roll out more now is, um, we call it AI studio, and it basically is, um, a set of tools that eventually is going to make it that every Creator can build sort of an AI version of themselves, um, as, as sort of an, an agent or an assistant that their Community can interact with, and know it's, it's going to be very clear that it's not engaging with the Creator themselves, but I think it'll be another interesting way just like how creators put out content on on these, um, social

systems to be able to have agents that do that. I think that there's going to be a thing where people basically create their own agents for all different kinds of uses. Some will be sort of customized utility things that they're trying to get done, that they want to kind of fine-tune and train an agent for. Some of them will be entertainment, and some of the things that people create are just funny, you know, and just kind of silly in different ways, or kind of have a funny attitude about things that, um, you know, we probably couldn't, we probably wouldn't build into meta AI as an assistant. But I think people, um, people are kind of pretty interested to see and interact with. And then one of the interesting use cases that we're seeing is people kind of using these agents for support. This was one thing that was a little bit surprising to me. One of the top use cases for meta AI already is people basically using it to roleplay difficult social situations that they're going to be in. So whether

it's a professional situation. It's like, All right, I want to ask my manager, like, how do I get a promotion or raise? Or I'm having this fight with my friend, or I'm having this difficult situation with my girlfriend. Like, how, like, how can this conversation go? And basically having a completely judgment-free zone where you can basically role-play that and see how the conversation will go and get feedback on it. Um, but a lot of people, they don't just want to interact with the same kind of, you know, agent, whether it's met AI or chat GPT or whatever it is that everyone else is using. They want to kind of create their own things. So that's roughly where we're going with AI Studio, but it's all part of this bigger, I guess, view that we have that there shouldn't just be one big AI that people interact with. We just think that the world will be better and more interesting if there's a diversity of these different things. I just think it's so cool that if

you're an artist, and you have a style. You could take your style, all of your body of work. You could fine-tune, yeah, one of your models. Yeah, and now this becomes an AI model that you can come, and you could prompt it. You could ask me to, uh, you know, create something along the lines of the art style that I have, and you might even give me a piece of art as a drawing, a sketch as an inspiration, and I can generate something for you. It could be, it could be a, uh, every single, uh, every single restaurant, every single website will probably in the future have these AIS, yeah. I mean, I, I kind of think that in the future, just like every business has, you know, an email address and a website and a social media account or several, I think in the future every business is going to have an AI agent that interfaces with their customer. So can I, can I use AI Studio to F tune with my images, my collection of images, yeah, you're, yeah, we're going to get there.

okay, and then I could, can I give it, load it all the things that I've written so that use it, use it as my rag, yeah, yeah, yeah, basically. Okay, and then every time I come back to it, it loads up the its memory again so it remembers where it left off last time, yep, and we carry on our conversation as if nothing ever happened. Tell me about your open source philosophy, where did I come from, and you know you open source py porch, yeah, and that it is now the framework by which AI is done, and now you've open source llama 3.1 or llama, there's a whole ecosystem built around it, and so I think it's terrific, but where did that all come from, yeah. So there's a bunch of history on a lot of this, I mean we've done a lot of Open Source work over time, um, I think part of it, you know, just bluntly is you know we got started after some of the other tech companies right in building.

out stuff like the distributed computing infrastructure and the data centers. And you know, because of that, by the time that we built that stuff, it wasn't a competitive advantage. So we're like, All right, we might as well make this open, and then we'll benefit from the ecosystem around that. So we had a bunch of projects like that. I think the biggest one was probably Open Compute, where we took our server designs, the network designs, and eventually the data center designs, and published all of that. And by having that become somewhat of an industry standard, all the supply chains basically got organized around it, which had this benefit of saving money for everyone. So by making it public and open, we basically saved billions of dollars from doing that work. Open Compute was also what made it possible for NVIDIA HGX. Is that we designed for one data center all of a sudden works in every data center. Awesome. So I'd say by the time that llama came around, we were sort of positively predisposed towards doing

this for for AI models specifically. I guess there's a few ways that I look at this. I mean, one is, you know, it's been really fun building stuff over the last 20 years at the company. Um, one of the things that has been sort of the most difficult has been kind of having to navigate the fact that we ship our apps through our competitors' mobile platforms. So, on the one hand, the mobile platforms have been this huge boon to the industry. That's been awesome. Um, on the other hand, having to deliver your products through your competitors is challenging. When you look at these generations of computing, there's this big recency bias where everyone just looks at mobile and thinks, Okay, because the closed ecosystem, because Apple basically won and set the terms of that. And like, yeah, I know that there's more Android phones out there technically, but like, Apple basically has the whole market. I think in general for the computing platforms that the whole industry is building on, there's a lot of value for that if the software

especially is open, so that's really shaped my philosophy on this. And um, both AI with Llama and with the work that we're doing in AR and VR, where we're basically making the Horizon OS that we're building for mixed reality, um, an open operating system in the sense of what Android or Windows was. And basically making it so that um, like we're going to be able to work with lots of different hardware companies to make all different kinds of devices. We basically just want to return the ecosystem to that level where that's going to be the open one. And I, I, I'm pretty optimistic that the next generation, the open ones are going to win for us specifically. Um, you know, I just want to make sure that we have access to. I mean, this is sort of selfish, but you, I mean, it's, you know, after building this company for a while, um, one of my things for the next 10 or 15 years is like I just want to make sure that we can build.

the fundamental technology that we're going to be building social experiences on because there have just been too many things that I've tried to build, and then have just been told, nah, you can't really build that by the platform provider. That at some level I'm just like, nah, that for the next generation. Um, like we're going to go build like all the way down, and make sure that we there goes our broadcast opportunity. Yeah, no, sorry, give me, give me talking about closed platforms, and I get angry. Um, so, um, you know, I do think there's this alignment where we're building it because we want the thing to exist, and we want to not get cut off from some closed model, right? And um, but it this isn't just like a piece of software that you can build. It's, you know, you need an ecosystem around it. And so it's almost like it kind of almost wouldn't even work that well if we didn't open source it, right? It's, it's

not we're not doing this because we're kind of altruistic people um even though I I I think that this is going to be helpful for the ecosystem. And we're doing it because we think that this is going to make the thing that we're building the best by by kind of having a robust ecos around. Look how many people contributed to PyTorch ecosystem. Yeah, totally. And I recognize an important thing. I recognize an important thing. And and I I think the Llama is genuinely important. We built this concept to call an AI Factory AI Foundry around it uh so that we can help everybody build take you know a lot of people they they they have a desire to um uh uh build AI. And it's very important for them to own the AI because once they put that into their their flywheel their data flywheel that's how their company's institutional knowledge is encoded and embedded into an an AI. So they can't afford to have that AI flywheel the data flywheel that experienced flywheel somewhere else so.

and and so open source allows them to do that. But they don't really know how to turn this whole thing into an AI. So we created this thing called an AI Foundry. We provide the tooling, we provide the expertise, uh llama, uh technology. We have the ability to help them, uh turn this whole thing, uh into an AI service. And then when, when we're done with that, they take it, they own it. We, the output of it, what we call a Nim, and this Nim, this, this neurom micro Nvidia inference microservice, uh they just download it, they take it and they run it anywhere they like, including on Prem. And we have a whole ecosystem of Partners, uh from oems that can run the NIMS to, uh gsis like Accenture that, that we've trained and work with to create llama based Nims and, and, uh and pipelines and, and now we're, we're off helping Enterprises all over the world do this. I mean it's really quite exciting thing. It's really all triggered off of, uh the Llama open

sourcing you know one of the things that I really love about the work that you guys do computer vision um uh one of the models that we use a lot internally uh is segment everything. The segment anything model that that you're talking about we're actually presenting I think the next version of that here at at sigraph segment anything to um and it is it now works it's faster it works with um oh here we go um it works in video now as well. Yeah so it's um a lot of fun effects will be able to be made with this and because it'll be open a lot of more serious applications across the industry too so I mean scientists use stuff to you know study um like coral reefs and natural habitats and um and kind of evolution of Landscapes and things like that but I mean it's uh being able to do this in video and having it be zero shot and be able to kind of interact with it and tell it what you want to track is um it's it's

it's pretty cool research. Now what else, what else are you guys going to work on Beyond uh Ray talk talk to me about. Yeah, so there's all the smart glasses, right? So I think when we think about the next Computing platform, you know, we kind of break it down into mixed reality, the headsets, and the smart glasses. And the smart glasses, I think it's easier for people to wrap their head around that and wearing it because it's, you know, pretty much everyone who's wearing a pair of glasses today will end up that'll get upgraded to smart glasses. And that's like more than a billion people in the world. So that's going to be a pretty big thing. The VMR headsets, I think some people find it interesting for gaming or different uses, some don't yet. My view is that they're going to be both in the world. I think the smart glasses are going to be sort of the mobile phone kind of always on version of the next Computing platform. And the mixed reality headsets are going to be

more like your workstation or your game console, where when you're sitting down for a more immersive session and you want access to more compute. I mean, look, I mean the glasses are just very small form factor. Um, there're going to be a lot of constraints on that, just like you can't do the same level computing on a phone. It came at exactly the time when all of these breakthroughs in generative AI happen, yeah. So we, we basically, for smart classes, we've been, we've been going at the problem from two different directions. On the one hand, we've been building what we think is sort of the technology that you need for the kind of ideal holographic AR glasses, and we're doing all the custom silicon work, all the custom display stack work, like all the stuff that you would need to do to make that work. And they're glasses, right? It's not a headset, it's not like a VR or MR headset. They look like glasses, but um, there's still quite a bit far off from the glasses that you're wearing now.

mean those are very thin, but even the Ray-Bans that we make, you couldn't quite fit all the tech that you need to into that yet for kind of full holographic AR, though we're getting close. The other angle that we've come at this is let's start with good-looking glasses. We have camera sensors, so you can take photos and videos. You can actually live stream to Instagram. You can take video calls on WhatsApp and stream to the other person. You know what you're seeing. I mean, it has a microphone and speaker. So, I mean, the speaker is actually really good. Like it's open ear, so a lot of people find it more comfortable than earbuds. You can listen to music, and it's just like this private experience that's pretty neat. People love that. You can take phone calls on it, but then it just turned out that that sensor package was exactly what you needed to be able to talk to AI too. It's great that we

have this but um but in the future, we're like not that many years away from being able to have a virtual meeting where like, you know, it's like I'm not here physically, it's just my hologram, and like, it just feels like we're there and we're physically present. We can work on something and collaborate on something together, but I think this is going to be especially important with application. I could live with a device that I'm not wearing all the time. Oh, yeah, but I think we're going to get to the point where it actually is, yeah. I could. It'll be, I mean, there's with within glasses, there's like thinner frames and there's thicker frames and there's like all these Styles, but um, so I don't I think we're we're a while away from having full holographic glasses in the form factor of your glasses, but I think having it in a pair of stylish, kind of chunkier framed glasses is not that far off. These sunglasses are face size these days. I could see that, yeah.

and and you know what that's that's a very helpful style TR exactly that's a very helpful St whoever you know it's like like I'm I'm trying to you know I'm trying to like make my way into becoming like a style influencer so I can like influence this before um you know before the glasses come to the market but you know well I can see you attempting it how's your style influencing work out for you you know it's early yeah but I don't know I feel like if if if a big part of the future of the business is going to be building um kind of stylish glasses that people wear um this is something I should probably start paying a little more attention to ladies and gentlemen Mark Zuckerberg thank you

Zuckerberg and Jensen Discuss The Future of AI

Artificial Intelligence (AI), Open Source, Generative Art, AI Art, Futurism, ChatGPT, Large Language Models (LLM), Machine Learning, Technology, Coding, Tutorials, AI News, and more

** Exclusive Pinecone Partner **

Matthew Berman

Mark Zuckerberg's Vision for Generative AI at Siggraph 2024

Mark Zuckerberg discusses the future of generative AI and its application in creating content for Instagram.

He talks about the evolution of recommender systems and the drive for quality improvement through more general recommendation models.

Zuckerberg envisions a unified AI model for Facebook and Instagram, spanning different content types and systems.

Meta AI is expected to revolutionize workflows and product creation with its advanced multimodal models.

The future of AI involves evolving from prompt-based interactions to long-term planning and simulating different outcomes.

Creator AI and AI Studio are set to be widely implemented, offering a vision of multiple AI models working together.

Meta's AI Studio and the Future of User-created Agents

Meta's AI Studio aims to empower millions of creators and small businesses to build their own AI agents for customer interaction.

The studio will allow users to create customized utility agents, entertainment agents, and even agents for roleplaying social situations.

One surprising use case for these AI agents is their role in helping people role-play difficult social scenarios, providing a judgment-free zone for users to practice and receive feedback.

The Impact of Open Source Philosophy on AI Development

Open source philosophy in AI development has its roots in the decision to make previously developed technology open to the public.

This approach was driven by the realization that making their work open and public could benefit from the ecosystem around it and save billions of dollars.

The Open Compute project, which published server designs and network designs, eventually became an industry standard, leading to cost savings for everyone.

The Impact of Mobile Platforms on App Delivery and the Shift to Open Systems

The speaker discusses the challenge of delivering apps through competitors' mobile platforms, despite the benefits they bring to the industry.

They emphasize the importance of creating an open operating system for AI, AR, and VR technology, in contrast to the current closed ecosystems.

The goal is to build a technology ecosystem that is open and allows for the development of a wide range of devices, driven by the desire to avoid getting cut off from closed models.

Llama is crucial for building AI and encoding institutional knowledge, with the AI Foundry providing tooling and expertise.

The output of this process is a Nim, a neuromicro Nvidia inference microservice, which can be run anywhere.

Llama's open source model, "Segment Anything," is being upgraded to work faster and with videos, expanding its potential for diverse applications in studying natural habitats and landscapes.

Smart glasses are projected to have a massive adoption, potentially reaching over a billion users worldwide, serving as a major computing platform.

These glasses are being developed to offer a wide range of functions, from live streaming and taking calls to interacting with AI, all in a small and stylish form factor.

The research is also focused on advancing the technology for ideal holographic AR glasses, although it is still a work in progress to fit all the necessary tech into a thin, stylish design.

The Future of Virtual Meetings: Holographic Glasses

In the near future, virtual meetings using holographic glasses may become a reality, allowing for a sense of physical presence without actually being there.

The development of stylish holographic glasses is anticipated, potentially changing the way people interact and collaborate.

There is a growing recognition of the importance of incorporating style into the design of futuristic glasses, indicating a potential shift in the business towards more fashionable eyewear.