last year a paper out of Stanford put thousands of agents in a fully simulated environment and let them live their lives. The agents formed relationships, built up memories, developed their own personalities. It was stunning, and this paper allowed us to have a vision of what could be possible with simulating

entire societies or even what the future of video games might be like. Imagine a video game world where NPCs had actual personalities, backstories, they lived their lives in real time. That was the promise of that paper, but now there is a new paper by the same author out of

Stanford, and it shows that it's possible to actually get real human personalities into these agents to then live their lives in these simulated environments. It is mind-blowing. Let me break it all down for you. So, this was the previous paper, generative agents: interactive simulacre

of human behavior. And what they found is by putting all of these AI agents powered by chatGPT into this simulated environment with a little bit of backstory, they would develop their own personalities, their own relationships. They would form friendships, they would develop plans. For example, one of the

agents threw a birthday party and invited all of their friends. But not only that, those friends invited some of their friends, and they coordinated and they showed up to the birthday party. It's really incredible to think about, but imagine this: you could actually put your own personality into these

agents now. Let me show you the new paper. So, by the same lead author, Jun-Sung Park, we have this new paper, generative agent simulations of 1,000 people. And so, the gist of what they've done is essentially taken that other paper but interviewed a

thousand people, two hourlong interviews of various questions trying to extract the personality of a real human and then place them in a simulated environment. Listen to this: we present a novel agent architecture that simulates the attitudes and behaviors of 1,528 real

individuals. They not only replicated the personalities of these real individuals, but they actually were able to test and prove that those agents behaved and had the same thoughts and personalities as their human counterparts. And the results? The generative agents replicate participants' responses on the General

Social Survey 85% as accurately as participants replicate their own answers two weeks later. So what does all of this mean? Let me break it all down. So they use very common social science tests, such as the General Social Survey, the Big Five personality inventory, well-known behavioral economic games, and

other social science experiments to try to extract the essence of what makes up somebody's personality. They then took that 2 hours worth of interview, converted it into memories for these agents to base their own answers on. They retested them against the General Social

Survey, the Big Five personality test, and the social science test, and they found that those agents behaved essentially 85% as accurately as the humans did when asked those same questions two weeks later. So that is extremely accurate, but why would they do this? What is the point?

According to the paper, these simulations could help pilot interventions, develop complex theories capturing nuanced causal and contextual interactions. And in my opinion, the most important: expand our understanding of structures like institutions and networks across domains such as economics, sociology, organizations, and political science. Essentially, we can start to predict how

people and organizations and societies will behave without actually implementing something extreme first so if we have this idea about a completely new tax plan for example and we want to see how people will behave based on this new tax plan rather than actually having to go implement it and then seeing or

predicting based on not as accurate methods we could actually set up entire societies of AI agents based on real people and then see how they might react to these massive tax changes today's video is brought to you by mamut mamut AI brings all of the best models

together in one place for one price Claude llama GPT 4, MRA, Gemini Pro and even GPT-1 and rather than having to pay for each of these AI separately you pay $10 to mamut and they bring it all together in one place plus they have image

generation Midjourney, Flux Pro, Dolly and Stable Diffusion again all for $10 models are frequently updated as soon as they're released so be sure to check out mamut for access to all the best models for one low price mamut doai that is M A

M M O U T H. AI thanks again to mamut let me tell you how it works we present a generative agent architecture that simulates more than 1,000 real individuals using 2-hour qualitative interviews the architecture combines these interviews with a large language

model to replicate individual's attitudes and behaviors by anchoring on individuals we can measure accuracy by comparing simulated attitudes and behaviors to the actual attitudes and behaviors they tested it against the General Social Survey Big Five personality inventory five well-known

behavioral economic games like the dictator game and the public goods game, and then the prisoner's dilemma experiment, which is funnily enough, out of Stanford's five social science experiments with control and treatment conditions that we sampled from a recent large-scale replication effort. And so how did they create these

agents that essentially replicated people's thoughts and behaviors well? They turned to in-depth interviews, not surveys, not simple question answers. They actually gave them real, long-form, and at times, dynamic interviews combining pre-specified questions with adaptive follow-up questions based on respondents' answers. These are a foundational social

science method with several advantages over more structured data collection techniques. And they were semi-structured interviews, meaning there's a set of questions, but then they allowed for dynamic and not predetermined follow-up questions. And the interesting part, they actually used AI to do all of the

interviews. This freedom to answer questions and really dynamic follow-ups give interviewees more freedom to highlight what they find important, ultimately shaping what is measured. So here's how it works. They have the human participants. They give a 2-hour voice-to-voice interview that is both sides are using

voice. The interview script is then transcribed, and those are given to the generative agents to serve as the agents' memory. Now, if we remember back to this previous paper, they basically started the agents with just a very brief description of their background, and each agent as they interacted with the world

they would develop these memories, long-term memory, short-term memory. They had some really cool techniques using RAG, essentially, that allowed them to draw from the memory to determine what actions that agent would take next, whether that action is where to go, what to do, or even how to interact with other

agents within the simulation, but now in the new paper they took that two hours of interview that essentially gets the essence of the thoughts and behaviors of a real human and they use that as the memory for the agents. So technically those agents should behave how their

real human counterpart would behave. Then they had the actual participant responses two weeks later, and then they tested what the simulated responses from those agents would be. So basically they gave the test once, used that to give memory to the agents, then two weeks

later they gave those same questions, those interview style questions to the humans and then they also gave those interview style questions to the agents and compared the results. And it turns out they were really accurate. The interview script explored a wide range of topics of interest to social

scientists from participants' life stories. Tell me the story of your life from your childhood to education to family and relationships and to any major life events you may have had, to their views on current societal issues. How have you responded to the increased focus on race and or racism and policing?

Those are just some examples. Then the AI interviewer dynamically generated follow-up questions tailored to each participant's responses. Then they took all those responses and gave it to the agent as memory. So when an agent is queried in the simulated environment, that is the entire interview transcript

is injected into the model prompt instructing the model to imitate the person based on their interview data. For experiments requiring multiple decision-making steps, agents were given memory of previous stimuli and the responses to those stimuli through short text descriptions. The resulting agents can respond to any textual stimulus

including Force Choice prompts, surveys, and multi-stage interactional settings. So let's look at the actual results for the GSS, the General Social Survey. The generative agents predicted participants' responses with an average normalized accuracy of 85%. Stunning! These interview-based agents significantly outperformed both

demographic-based and persona-based agents. So what does that mean? They basically took a bunch of information or a bunch of knowledge about what a demographic might respond with, rather than interviewing individuals. And then they use that as the memory. And what they found is when they use that more

generic knowledge rather than interviewing individuals, they didn't perform nearly as well. And that actually shows bias in that data, in that generic data. For the Big Five questions, the generative agents achieved a normalized correlation of 0.80. Again, stunning, and once again outperforming the demographic

based and persona-based versions of those agents. On the five well-known economic games designed to elicit participants' behavior in decision-making contexts with real stakes, they have the dictator game, the trust game, the public goods game, the prisoner's dilemma. The generative agents achieved a

normalized correlation of 66. And just to make sure that the agents didn't just answer the questions right or accurately just based on their existing knowledge before all of this additional knowledge was given to them, they randomly removed 80% of the interview transcript. Basically, 96 of the 120-minute interview

and the interview-based agents still outperformed the composite agent, achieving an average normalized accuracy of 79 on the GSS, with similar results observed for the Big Five. Second, to investigate whether the predictive power of interview stems from linguistic cues or the richness of the knowledge gained

we created interview summary generative agents by prompting GPT-4 to convert interview transcripts into bullet-pointed summaries of key response pairs capturing the factual content while removing the original linguistic features. They are just reaffirming that the knowledge gained from these

interviews is real and not just from different language cues. And once again, it outperformed. These findings suggest that when informing language models about human behavior, interviews are much more effective and efficient than survey-based methods. And they also talk a lot about bias in artificial intelligence,

which I know a lot of you have asked me to comment on and make videos about. I've done it a little bit in the past, so we're going to talk about that a little bit right now. There is concern about AI systems underperforming or

misrepresenting underrepresented populations. Basically, if there's not enough training data for underrepresented populations, how will the model know how to respond based on what those populations would have responded with? To address this concern, we conducted a subgroup analysis focusing on political ideology, race, and

gender dimensions of particular interest in relevant literature. So what did they find? Interview-based agents consistently reduce biases across tasks compared to demographic-based agents. So for political ideology, the bias dropped from 12.35 for demographic-based generative agents to 7.85 for

interview-based generative agents, and similar results across the other benchmarks that we talked about. So one thing that we've talked about on this channel, essentially all public data has been used to train models. Now imagine if AI agents were able to interview humans across the world, different questions

interview style with follow-up questions dynamically, and all of that data was then used to train different models for different purposes. That is a huge amount of additional data. And then if all of those agents were responding how their human counterparts would, or at least

highly accurately compared to their human counterparts, then imagine we can create all of this synthetic data by interviewing the actual AI counterparts. It's really cool to think about, but that's the gist of this paper. I hope you enjoyed it. I thought it was fascinating. If

you enjoyed this video, please consider giving a like and subscribe, and I'll see you in the next one.

Stanford Researchers Create an Entire AI Agent Society ("Personality Clones")

Artificial Intelligence (AI), Open Source, Generative Art, AI Art, Futurism, ChatGPT, Large Language Models (LLM), Machine Learning, Technology, Coding, Tutorials, AI News, and more

** Exclusive Pinecone Partner **

Matthew Berman

Stanford's research on simulating human behavior in AI agents.

A Stanford paper simulated thousands of AI agents, observing relationship formation, memory development, and personality emergence.

A new Stanford paper explores integrating real human personalities into these simulated agents.

Previous research showed AI agents developing personalities and relationships in a simulated environment, even organizing events like birthday parties.

The new study involves interviewing 1000 individuals for two hours to capture their personalities for integration into the simulated environment.

Methodology and accuracy of the personality replication.

The study replicated the personalities of 1528 individuals, achieving 85% accuracy in replicating responses on the General Social Survey compared to the individuals' own responses two weeks later.

Common social science tests like the General Social Survey, Big Five personality inventory, and behavioral economic games were used to extract personality traits.

The agents' responses on these tests matched human responses with 85% accuracy after two weeks.

Applications and implications of the research.

Simulations can help pilot interventions and develop complex theories in various fields like economics, sociology, and political science.

The simulations allow for predicting societal reactions to policies without real-world implementation.

The architecture combines 2-hour qualitative interviews with a large language model to replicate individual attitudes and behaviors.

Testing included the General Social Survey, Big Five personality inventory, and five behavioral economic games.

Details of the interview process and agent memory.

In-depth, semi-structured interviews were conducted, allowing for dynamic follow-up questions.

The interview transcripts served as the agents' memory in the simulation.

The agents used RAG techniques to draw from their memory to determine actions.

Results and comparison with other methods.

Interview-based agents achieved 85% accuracy on the GSS, significantly outperforming demographic-based and persona-based agents.

Similar outperformance was observed in the Big Five personality test and economic games.

Even with 80% of the interview transcript removed, interview-based agents still outperformed composite agents.

The study confirmed that the predictive power comes from the knowledge gained, not just linguistic cues.

The study addressed concerns about AI bias by conducting a subgroup analysis focusing on political ideology, race, and gender.

Interview-based agents consistently reduced bias compared to demographic-based agents.

The potential for using AI-conducted interviews to gather vast amounts of data for training models is discussed.

Stanford Researchers Create an Entire AI Agent Society ("Personality Clones")

Stanford Researchers Create an Entire AI Agent Society ("Personality Clones")

스크립트