what we're going to go into now is um I'm a technical founder so so I'm someone who uh builds a lot uh you know I try not to deliver uh PRs too often these days if my team's here uh but but I I build a lot in my hands on on the

tech a lot um and what it means is that um is that we've been we kind of we've been building assistance we've been using our own tools to build assistance and agents inside of our product so I'm going to spend some time talking to you

about our challenges what it looks like how we built it and really go a level deeper in like how how this stuff works so our our journey we we have a a co-pilot assistant in product and we started off with um like a lot of people

do this this little window to the side uh and and I think there's a huge opportunity for just better UX uh UX design of these and UX integration to product we're going to show you some of the changes we made and how we've integrated in but this has just been

really a platform for to deliver AI into the product what it means what what where we we've gone is the the assistant itself has 30 plus skills at this point I'm going to show you some of them some of them around creating dashboards and

graphs some some are around like debugging your data some are around searching across your data for for issues so these skills are things that like the kind of function calls or stuff below the surface that um that that our assistant can do in product and we the

the way we start off a lot of different architectures I would say this is a really common approach I see team after team after team after team and what this is is there's two LLM calls there's a router call which is determining what intent what your

what your skill what skill to to to take advantage of it's a tool call and there's a a skill which might include an LLM too and and the reason we only have two is because you know two is a lot to string together you're going to see it

takes you know some of these it takes a little bit of time um but what we do is is is most assistance I see in products look something like this it's kind of a router determining intent and then and some amount of work down here um so what

it means is each each red box here has a prompt template associated with it too typically now you can you can have a user uh user in the loop like you you'll see that's what we do we have a user in loop of our assistant where we come back

to the user interface after two LLM calls and get more data and so I'll show you I'll show you in a second what this looks like in practice with the demo um but this is a very common approach which is a router a a tool call which might

include an LLM and then a user in the user um in the loop so this is you know one layer router very very common architecture where we blew up in in kind of where we went to in arise over time is we ended up with with routers and LLM

calls of an our assistant that that broke out different things in different user interface entry points so in one of these cases I'm entering from a dashboard and I'm I'm creating a graph and another case I'm entering from from a search portion and uh doing doing

a bit of search so the complexity grows over time but you still see a similar pattern there's a a routing thing that's determining what skill to take advantage of maybe based upon my my uh user interface location and there's a skill that leverages LLM sometimes most of the

time it is to do something special uh and I'm going to spend some time on on AI search which is pretty fun it goes across the data and helps you analyze and debug it but I also do some graphs and some other things for you too um we also use our software every day to

build this agent and assistant which is a great way to build software you we dog food our our tools we we build with our tools and a lot of the stuff that you'll see is is that our platform uh kind of integrates across production and development I think all platforms that

do what we do have to do this the evals and evaluations that we use in development we also use in production I'll pause how many people know what an evaluation is these days is evals mean something to the room share hands okay quite a lot these days

well um when we say evals we typically mean LM as a judge or using AI to help evaluate your system uh kind of meta but but that's you know that's that's what's happening here is we're using AI to evaluate AI um you're annotating your

data you're using data sets um and so this is kind of a and I'm going to show you a bit of a demo of what what this looks like in in practice um so give me a second here so let's try the live demo hit it

do you uh can you do the live demo okay that worked wow uh High-tech so so first off this is what this we have a this this is an example of what assistant looks like in our system um each one of these is a a chat back and forth where

there's some some input um in this case it's asking a question about an SDK there's some response going on here there's some potentially retriever uh retrieval system um retrieving chunks of docs maybe from a we8 who's going to

speak here um there's some synthesis in the LLM um and response and what we're doing I mean some of these will be hundreds deep in calls you know a single call if you're using a framework can be 100 deep in in time in the amount of calls that occur

so the number first thing I would do first thing we all recommend doing is kind of tracing your agent tracing your assistant you have an entry point you have some things that's doing in this case it's one LM call so you trace so you get your data and then you evaluate

uh so so you'll see evaluations kind of attached here zoom in a little bit and what what these are what these are is essentially AI evaluating the the question and answer here in this case AI kind of said um the SDK you know there's no mention of the SDK in the reference

Okay, what's up? Um, this, the response is problematic, and, and so this is kind of, this is kind of the nuts and bolts of this looks like. Now, what I wanted to do was actually show you where does our, what you know, what does our assistant

look like, and where those integration points and what, what, what challenges have we had in building it? And then I'll tie it back to, to the product we use. Um, so one way we have an assistant here is we do have an AI search capability. Um,

sometimes you have questions and answers you might want to, you know, find all frustrated users, it's a pretty common one. Uh, so you can really type in anything you want to search across. And the challenge of this one, like I mentioned, there's two LLMs

calls, there's a first call here to figure out what, what to go do. There's a second one that pulls in like all the data and then searches across it. In this case, I'm finding people kind of annoyed or, or, or frustrated with, with our platform. Um, and you can build filters

from these. So, the first one is just AI search being able to kind of search across your, your, your, your system in certain ways. Um, but I can do other magic, so I don't, it's not just search, we, we decided over time we wanted to, like, people might want to

just understand the type of question. So, I might want to do some analytics before I filter. So, what I showed you was a search and filter, I can do things like, what are the top, you know, top questions being asked? So, I can do analytics on my data.

before I go filter, which is a different skill, so as a router it's realizing my question wasn't about filtering and so it was really about analytics. My questions about the top questions being asked from the user, so it kind of knows what columns are in the data, what the traces

data is, it knows what to go grab it and then pulls it in and and looks at so there's you know the top questions around SDKs or certain metrics being being asked, so so you can enter through a certain spot and call different skills based upon what you're what you're

trying to do. Um, I can also do things like you know a common thing here is we we have those evals we were talking about. Customers, you know customers have hallucinations and and evals around those, and evals around correctness. Well, I might want to

know what are the top reasons for hallucinations you know and and so you can you can kind of use this for uh for finding problems and building data sets uh and you can spell things wrong like hallucinations and typically we'll get it right too. Um so so there's quite a

lot you can build on top of you know one entry point, so this is kind of one entry point we have in the system with with a kind of a router around it. Um and in this case it did find hallucination spans, evals, uh information's not present

in the reference text, is using those explanations to figure out like why it hallucinated and giving you reasons. Um some quite powerful uh abilities to kind of build filters and do analysis. I also you know might grab data off and build data sets for for uh for my testing.

workflows which I do a lot of um but I want to give you some other examples of where the assistant actually gets used um there's examples in in dashboards uh we have an dashboard assistant that helps you create graphs so if I want to

create you know a time series widget or something and I just you know um create a Time series you know for token counts total impressions prompt I can do this on the fly I can choose the data to to run across and this will kind of create um you know

create graphs we'll create graphs that add um here is asking it's not sure the metric I want so kind of asking me some questions can you do total count on tokens so if it if you don't get it right it can ask you questions and um

kind of build build that chart chart for you which is kind of a a nice interaction uh Paradigm and so this this entry point here is um you know and then I can show the kind of you know stuff that's created I can add it to the add

it to the dashboard um with kind of a click of a button and I kind of have my My Graph whatever it is um so there's lots of different entry points and things you can do um I can do search uh we also do evals so um Gemini uh for example

runs as data is ingested there's these evals that run on top of data as it comes in the platform so we have co-pilot and our assistant that helps you write these templates where you know Gemini as a model from Vertex is running

on your data and augmenting, seeing whether it's hallucinated or correct. So there's integration points here where we help you write templates um for an eval. So let me let me kind of just edit this and let's do a let's create a template

for um you create an eval to detect um maybe angry users you could do, or frustrated users. So long story short is there's is we've come up with different ways. I mean, as you think about building assistance and building, so this kind of rewrote the

template here. I can save it off um you know we it's the assistance and agents is a platform to be able to deliver AI into your products and these assistants are going to be built into SaaS platform after SaaS platform after SaaS platform powered by the data. In our case, we're

building a system to help you build AI, but every and and we're using our own tools, tools to do it, but every single SaaS platform out there is going to be delivering an assistant to every white collar job in the US to help them with

their day-to-day and you all are probably building a lot of those uh and and the idea is that you can integrate these in, you can use this concept of LM routers and school calls to make it work. Um hopefully that gives you an idea of

of a little bit of our platform, a little bit how it works, um a little bit of like what an assistant looks like. And I probably have time for like a question or two if someone wants to ask a question. Go ahead. Yeah. So I I think that you're assuming that everyone

AI is already in the process of building systems, yeah, and you're showing uh a way of making them work better, of debugging them and analyzing them. Where maybe you could just for some of us step back a bit because in my experience, I've used models for

specific tests, code translation, migrating to the cloud, but it hasn't been an agent system. It's been like setting something up, a tool like SageMaker or something like that, yeah, and just you know doing batch transforms from storage into the cloud and then producing output and checking

it. So I'm having a hard time understanding what the problem is you're solving. I mean, I can imagine this agent paradigm, but maybe if you could just back up a little bit and explain what it is I would be trying to do, maybe in the context of a RAG system. Because one of

the things we're we're thinking about is is having something which can use our corporate data to answer questions as if they were trained in a model, which they aren't. And we have so much data. I work for the Federal Reserve, so we got like

lots of data, right? Yeah, and uh and how we would go about doing that. So maybe if you could answer the question in that context, yeah. So so I think I think as you saw, every single feature I probably showed in the platform would not be accessible or executable with

traditional ML in any way, shape, or form. So so I think what you're trying to do, so what what we see is, yeah, with traditional ML, we're we're we're using models, we're using like Llama 3 for example, to do these things, so

It's not traditional ML, it's not an agentic framework. It's more of prompt engineering and having different phases where we process the data to do different things, then get an answer. We might do SQL natural language to SQL conversion, we may go to a data catalog,

find a relevant data, then we would have it create, you know, a Python program to display it. Those types of steps which I think are different from what you're talking about. Yeah, so so I would say so the real question here is, I think it's a

question you hear all the time is, what is an agent? And maybe that's what you're even getting at, like I'm doing prompts and templates right now. What what is an agent? And and the key areas which I typically look at is is there, you know,

and I've heard I've heard kind of people in the ecosystem say this, but what we see it is is there control flow and is there iterative fashion on data? And in this case, you're iterating with with user input. Control flow is an LLM in

front of it deciding what to go do, and then you're doing an LLM called, you know, or could be a function call below that. So all we mean by an agent is an iterative fashion making decisions, um, taking in taking in data iteratively with typically it's an LM router or tool

call in front and it's doing something below it. So you might even be doing it today, um, maybe you are, maybe you aren't. But in the first generation of what most people were building was a prompt response, you know, one call you're done. And and what we've kind of evolved

to in the last year is using tools, so controlling the flow of that and then doing something with those and then potentially iterating on it. I just ask question, yeah. You're getting towards the problem we're trying to solve, so we work iteratively, and one of the things we find

we're writing special to do, yeah, because there are things you want to keep and there are things you don't want to keep. Yeah, yeah. And and as you're going through generations and and doing your prompt engineering, you have to manage that because otherwise

you're going to like run run out of buffer, right? I I I think that's key. I think the number one thing actually in most systems though is tool calling. So do you use AR? Do you do tool calling and specialization and then prompts below that and calls and is there flow control

and iterative loops on top of your system? If you're just prompts and responses, I would say it's probably more of the traditional, you know, you're probably the the first, that's kind of the first generation of systems here. Um, I do want to give anyone else one one one

more here and sorry I'm a little over. One more question, so with what you were uh displaying earlier, um, the it seems like there would have to be a very deep integration between, you know, your tooling and the underlying systems, whether it's the the model calls, the

tools, the calls to weave and so forth. So how do you address, you know, things like um security and whatnot in the sense that, you know, I'm building my system which might be in my cloud, yeah, I'm using you as a managed service, am I installing

yours as an agent you know just how the integration part of that would work it's a good question so um if you want to get started tomorrow we have open source so we we do so instrumentation wise we believe in otel which is typically I

think where everyone's going right now from instrumentation perspective um you can get started with our open source as you scale the Enterprise most of the most of the most of the companies in my vertical um have something that deploys on your on your software so into your

VPC so it's like data bricks meets DataDog type type delivery um I I I don't think the data leaves your system system if you're you know most Enterprises typically right now

Observability & Evaluation Platform for AI Engineers

Arize AI is the watcher, troubleshooter and the guardrail on deployed AI.

