Oh yeah, we were able to recognize the coffee cup, 100%, that was easy. But um, we forgot to note that it doesn't belong in a medieval fantastical world. So I have always been into computer-related things, uh, trying to learn how to program on my

own as a as a kid, pre-college, and in college. Um, I went to the University of Virginia and focused on systems and information engineering for undergrad and grad school, which is it's it's a combination of operations research and kind of management consulting and and various other things.

And I I minored in computer science because I still wanted that kind of really hands-on application experience. And I went into a lot of government work with a company called MITRE, which is a which runs a number of federally funded research and development centers.

It's a it's a really interesting company if you ever uh if you ever want to check out MITRE, they do a lot of really fascinating work for the government. Um, and they're a non-profit entity, so they're they're kind of this this uh strategic advisor, implementation

um, all kinds of things for the government. So that was that was um the springboard for learning a lot of things at the same time. Right out of school, I worked for most of the the intel and defense agencies in the the DC area, and I got a

lot of really interesting experience with data and with coming up with platforms and tools to help analysts to facilitate taking lots of disparate kinds of data and fusing it together to help answer questions. I became the the chief of technology for an intel fusion cell, and um built some

Uh, so so it's a place where you have lots of different kinds of analysts and lots of different kinds of data come together in one place to work together on, uh, on a specific kind of problem. So so imagine, um, expertise in human intelligence understanding, um

uh, intelligence reports working with somebody that is an expert in imagery analysis and working with experts in other domains in the intel and defense world and even people like myself, technologists that kind of specialize in in creating um software um and manipulating data and helping facilitate the kinds of analytic

products that they need to get out. So so I would do things like um, this this range of, uh, if we see somebody click a button 30 times in one day, you know that's something we need to fix, so we'll write a tool to help them so

they only click that button one time a day so they don't have to focus on that button anymore. They can focus on the on the hard work and anything from as simple as that to, uh, I know maybe 15 or so years ago we were we were extracting all of the nouns and

verbs from every piece of intelligence that we could get our hands on. When we were extracting billions of records and creating tools to help analysts search and query and identify you know those those proverbial uh needles in the haystack. So that that was a that was a lot of

really interesting varied experience and it took me to Afghanistan a number of times building tools out in Afghanistan and and traveling around training analysts on how to use those tools and how to take advantage of it. I did the same thing in Iraq for for a

very short period of time um so that was uh that's kind of what set me off in the uh the technology direction that's really interesting a little while ago we had on a company called imo uh we had their CEO Ivana on and

they uh what they do is they take medical terminology and they have like these subject matter experts like medical doctors just like pouring through textbooks and pulling out these terms and they put the terminology into data sets and deploy that to hospitals so that their systems can understand the

notes that the doctors take during their uh like visits with patients and stuff um and that helps with like creating automated billing processes and stuff it's but like it was really interesting to me learning that there's so much work that has to go into just deploying

literal words to for systems to understand the words and you talking about having to pull all those nouns and verbs for uh the government work sounds like pretty similar and it's crazy that I just feel like I went so long not hearing about that at all and now here we are

twice yeah no no I mean you're absolutely right it really comes back down to that and and uh and probably the reason why people don't talk about it as much is because it it has become kind of the status quo and people are focused on higher level activities like

um as a data scientist how do I focus on building my neural network and using that to automatically extract data um which is you know the ultimate goal but I think a lot of times people forget how important that data preparation stage is um which is which is kind of the building blocks of

all of the really fancy AI/ML stuff that we do today whether it's building on top of that named entity recognition extracting those nouns, verbs or you know people, places, things or whatever you're interested in um or something more complex, those are

really all the stakes in the ground and um that's actually at Innodata um the company I'm I'm a part of now, one of the the major platforms that we're building is a data for AI platform that focuses on how you get that training data so

the it kind of it kind of comes full circle to to helping the analysts you know 15 years ago who were spending 80 percent of their time manipulating data and 20 percent of their time doing the analysis that that that's been kind of a common

thread throughout my career whether it's helping analysts in the intel and defense world or helping automotive companies identify emerging issues or and now at Innodata helping data scientists focus on training their models so flipping that 80 20 so that they're not spending 80

percent of their time tagging and identifying individual words so they can train their model and only 20 percent of their time refining things but rather really focus on expediting facilitating that high quality training data so that you can spend more time on that that that higher level

AI/ML that that we hear about so much more these days, that makes a lot of sense so what are some major differences that you've encountered, it sounds like you've been trying to solve this same problem um while serving the government sector and

serving the commercial sector what are what have been some major differences between the two yeah um that's a really interesting question because I don't know there there aren't significant differences from a technology perspective there are obviously significant differences in in the application and what you're using it

for and maybe the types of information but when it comes down to it the the the type of technology that you would use today to analyze an intelligence report is very similar you know under the hood to something that you would use to extract relevant information from an insurance

claim or an earnings report um a lot of those technologies are share share a lot a lot of things in common um I think really when it comes down to it the differences are really just about the data so uh the availability of the data the quality of the data

the capacity to turn that into something that is useful for training a more more sophisticated model and and and to prepare it so a lot most data is very unclean it needs filtering it's inconsistent and so you need a lot of tools to help

with that so I think you know when I think about platforms the for data engineers like myself I think one of the last things that we think of is design and user experience and so I I think that is something that is going to change that is changing now and I think we'll

see a much larger focus on design and user experience around um that that data preparation stage that data analytics stage really the whole data life cycle I think we're going to see more focus on what it means to users because at the end of the day you can you can eventually use machine

learning to bootstrap and self-learn essentially, but you still need humans to kick that off and you need humans in the loop to manage data drift over time. So yeah, I think the um one of the the bigger differences, kind of going back to your question, is

um in the ways that you might interact with systems might be a little bit different between applications, you know, if that's government versus commercial. Um, but really I think that the biggest difference we're gonna see in the next few years is is a focus on what that

means for end users that makes sense. Yeah, so before we get into all the nitty-gritty of cleaning data and uh training AI algorithms, let's take a step back because a little while ago we had on your chief product and marketing officer, Raul.

But for those that didn't listen to that episode, can you give me like a brief overview of what Innodata does? Oh yeah, absolutely. Innodata is a really fascinating company, so Innodata has been around for about 30 years now, publicly traded on the Nasdaq. Um

uh historically kind of rooted in publishing, the publishing domain um with things like ebooks and I think um, you know, the the company evolved by uh taking a lot of that kind of data engineering experience and expanding into uh scientific journals, uh legal journals, medical journals, and over time built up

this impressive team of subject matter experts, and today Innodata has something north of 3,500 subject matter experts on staff that are including doctors and lawyers that help not just in that publishing space anymore but also in in all sorts of

other industries and verticals to apply that subject matter expertise to data extraction, data annotation, um, and various kinds of processing. And in recent years, Innodata has taken significant leaps forward in machine learning and artificial intelligence to augment and supplement um, the the good work that the subject

matter experts do. Um, so there's a number of products that Innodata offers in addition to the managed services for for data um, uh for uh data processing. Um, automated platforms for data extraction, extracting data points according to to a set taxonomy for example in the in the financial or the

legal spaces and and also uh the data annotation platform um, which is one of the one of the key platforms that I focus on in Innodata, which really takes which really takes into consideration all the lessons learned over the history of the company and the subject matter experts and how

we build our own models leveraging all that experience and turning it into a customer-facing SaaS self-service platform that helps facilitate that data from end to end. So from from the raw data to uh powering an actual machine learning model and applying that to a

real problem. So what does it look like using the data annotation platform in practice? Like what's the input and what's the output? Yeah, it's a good question. So so it's actually uh a number of things uh depending on the use case. So the way I talk about the

annotation platform is as workbenches, workflows, and KPIs, which kind of encompass the the ecosystem of things that you need in a more end-to-end self-service platform where somebody even outside of Innodata can sign up and click and and start their own process to

facilitate getting to that final product and so when you first fire up the platform you'll configure a new project and you'll select a workbench that's based on a use case so if you think of workbenches as my

collection of tools that I might use for a particular kind of problem if I'm categorizing credit card transactions then I need a workbench that shows me records and I need certain tools to help me get through a list of records something that

you might be familiar with seeing in a spreadsheet often tools to help you get through labeling that in an efficient and high quality way uh that might be one kind of workbench another kind of workbench might be more geared towards uh annotating entities in line within a document so

any reference you see to Steve or Adam uh in a document you might want to highlight in relation to an event and connect all those things visually so to to dumb it down a little bit um you're talking about you relating all the records in a

spreadsheet to like a tool for helping you sort through those that tool could be something like when you're working in a spreadsheet you have like an autocomplete that can uh you can like just drag the cell down and it'll fill in the next line of data based off of what

you've been putting in for each record is that is that a good yeah no that that is spot on so it it's kind of the um the hidden side of data science it's um it's really preparing all of this data so that you

have a set of clean labels that you have high confidence in so that when you go and train your model to automatically recognize the next credit card transaction and see that even though it's clearly a gas station to identify whether that's actually a gas purchase or if it's more of a convenience store, maybe the person bought a candy bar instead. There's a lot of factors that you might take into account. You can only take those into account if you've had a human take a look at that

first and make a decision on how to treat those types of things that makes sense. Yeah, cool. So what are what are some of the more interesting use cases you've seen for the platform? Just like stuff that's kind of out there. Yeah, so uh so we we do a lot of

text-based annotation. Um there's a lot of expertise in the company around the legal, the healthcare domains, the financial services domains. We've also done a lot of work in image and video annotation, which I think is is fascinating. I I have I've built a lot of geospatial analytic

platforms in the past, so uh assigning latitude longitude coordinates to to to points on a map. But image annotation is a little bit different from that, and I think image annotation is a is a fascinating world where it ranges anywhere from looking at a shopping cart

where you might have occluded items, so you might have a box of cereal but sitting on top of it is a jar of peanut butter and you need to be able to account for that in your in an image recognition model. So those are really interesting use cases and and

whether the annotator is tasked to outline the full occluded, you know, hidden box of cereal or just the visible portion. Those are all kinds of things that, um, that have different implications for for what you're trying to do and and you can, as you can imagine, uh

those get those kinds of decisions are much more critical when you start thinking about autonomous vehicles and recognizing things on the road reliably. Yeah, that makes sense. That's life or death like, but um, that actually makes me think of like in a really specific use case of AI

that I was listening to this other podcast called HPE Tech Talk and it's a tech podcast put out by Hewlett Packard and they were interviewing someone from the Walt Disney Studio Lab about how they're applying AI in the filmmaking process

because at the end of creating a movie they have these quality control experts that go through and check out the entire film frame by frame and look for pixel level anomalies on each individual frame. Like, that's the level of detail they get into on this quality control and they

used to have people go like that were responsible for finding a single miscolored pixel on a single frame of a movie, but now they're able to train their AI, presumably with some really high quality image recognition that's able to find these anomalies and

just like you were talking about flipping the 80/20 ratio there, now the quality control experts are able to spend a lot more of their time focusing on what to do about the anomalies. Is it worth fixing, and if it is worth fixing

how do you fix it because the movie's done right it's yeah that that's fascinating I I wonder if that can help um resolve the the the coffee cup in the the Game of Thrones uh issue I don't know I'm not familiar can you tell me again

there's a big uh um article um I guess it's been a few years I have no sense of time these days but with uh with Game of Thrones there was apparently a I think it was maybe a Starbucks cup that was caught in one of

the one of the scenes I think kind of a climactic battle scene and uh and and so I I don't know if they they had to go back and reshoot or if they just left it in for posterity but uh but yeah that sounds like a really

interesting kind of problem yeah it sounds like an anomaly that should have been picked up how do you train uh an AI to recognize that that doesn't belong in a medieval atmosphere yeah I I don't know I guess you'd have to classify objects

to such detail that you can recognize the objects in in various frames but um I think we're probably a long way away from thinking through that unintended consequence yeah oh yeah we were able to recognize the coffee cup 100 percent that was easy but um

we forgot to to to note that it doesn't belong in a medieval uh fantastical world yeah I guess that that level of quality control is still up to the humans today so are have you seen any store interesting stories of like data labeling gone wrong

[Music] um you know I I there's nothing in particular uh that I that I can think of except uh just kind of the general theme of bias in data which I think is is important I think bias is um has always been a challenge

for uh for machine learning models I think we'll continue to see a lot of focus on that as we get more into things like synthetic data generation um so that's one of the things that we're that we're working on in Novartis as part of the full ecosystem of data

machine learning models and the application of those one of the things that we often see is that sometimes there just isn't an adequate level of data available for training and so I think synthetic data is a really interesting topic I think the challenges are going to be

how to how to avoid bias and kind of that echo chamber of uh of error that causes lots of bias issues which have social economic implications um it also just causes issues with data drift over time if you're if you're training all of your data on uh on a on a narrow slice of reality

um it's going to eventually drift away from from that reality as those errors propagate so that's that's kind of the importance of continuing to train data over time it's it's never a it's never done you know you always have to account for new things

even for something like if you're trying to recognize events geographic socio-political events or medical issues you know pre-COVID-19 a machine learning model might not even know what that is and today if you were able to continue training your models in

whatever domain you can keep up with with things that are unexpected like that so my base level of synthetic data generation is that it's a solution for when you have an incomplete data set right um so a lot of the time I feel like bias in AI as you mentioned just comes

from just not having the data whether it's due to historical inequities or any reason for just not having the right amount of data the model ends up coming out biased but how can you tell if you have an incomplete data set especially if the

model is functioning as you expect it to and it's it it seems like it's giving you the right answer how can how can you tell that the model is biased when your subconscious view of reality might be biased as well yeah that that's there's an important

role there for data scientists and for metrics around evaluating the models and I I wouldn't profess to be an expert in in that domain but um I know one of the things that that we focus on with our data extraction machine learning technologies is the not just the confidence of our overall

extraction but at a data point level how confident is the model that it was able to identify this thing correctly so you might have an earnings report you might have a a legal contract that's 500 pages and you want to extract the the parties

and the the dates of interest and all sorts of parameters around around that that complex contract and at a data point specific level you can say well I'm I'm 99 sure that I got the um the date of execution correct but I'm only 50 percent sure

that I got the interest rate that we agreed on, correct? Because maybe the model wasn't able um to to have enough examples, enough variants of those examples or enough context to be able to pull out that specific data point. So that's that's one of the things that

active learning also helps address. Active learning is is really about pushing things in reverse. So having the model tell you, prompt the user to say I need to know more information about this thing. Tell me more about this. So the example is kind of more of a question prompting

process where after some initial set of documents you train your your model, an initial set of documents and then the machine says uh tell me about XYZ and it and it kind of targets those specific questions because it knows that those are the areas where it is least

confident. That's smart. I feel like that's a like a hallmark of intelligence in a person, like being smart enough to be able to ask for help um and that seems really cool that you're implementing that in computer intelligence. Yeah, it's self-awareness. I I hadn't thought

of it. Yeah, that way um outside of uh you know the the um machines becoming self-aware for for better or for worse. Sounds like that's a that's a use case for better um So is that earlier you mentioned you also so we talked about the data annotation

platform, you also mentioned something about a data extraction platform. Is that like the example you just gave of pouring through a legal document and pulling out the data points? Is that what the data extraction platform is? Exactly. Yeah, that that's exactly right. So um

we take all that training data that maybe we use the annotation platform to identify, you feed that into our machine learning models, they apply that on a per document basis and they give you the source transparency so they tell you exactly what the value of a data point was

how it was extracted from the document and it points you to the exact location in the document so that as a user, as an SME, as a subject matter expert, you can review that and verify that it's correct and it gives you those confidence values so that you can have some level of

confidence around the model itself and then you can feed that back in to the overall process. So I think that's that's also something that we've been thinking a lot more about in data uh within Innodata recently is how all of these things are really

connected and how our our vision for this interconnected platform starting with initial data annotation to training and applying models feeding that back so that annotators have a starting point. So if you imagine not just extracting the data points for a business purpose but also extracting the data points to help auto

suggest to the next annotator, hey this is a this is a thing that that we think we should be annotating uh do you agree? And that just makes the life of the annotator uh easier, facilitates the process and and cuts down the time for the annotators as well.

So it's really important uh just kind of going back to that that user experience. We we want to cut down that 80 percent of of data manipulation time for data scientists. We also want to do the same for annotators so they can be more

productive, they can generate more high-quality work, which is also, which is also important. That's really cool. It sounds like a similar advantage that you're giving annotators that CRMs gave salespeople when they first came around, like you have multiple, say, with the advantage of a CRM

that has all the notes of every client in there, you can give multiple salespeople the same set of clients and they can all know what's going on with each of them. Sounds like you can give multiple annotators the same data set. I mean, yeah, same data set, they could all work

on the same data set and all be on the same page thanks to the suggestions and and stuff that you're talking about in there. That, that's exactly right. And as part of our, um, SaaS platform, you configure, you start with that data workbench, that annotation workbench

and you configure the workflow, so you say, I've got 30 annotators on my team, but I want every, every document to be seen by at least two annotators, and then you can figure how an arbitrator might operate on disagreements between the annotators.

You can, the annotators can leave comments if there's something that's a little ambiguous, or if they want to justify their rationale behind why they made a decision, they can leave those comments behind for the arbitrator to take into account. We do QA on that and, and eventually

those, those results get exported and, and train, um, incrementally improve the machine learning models that then auto-suggest in the system again. So, uh, so yeah, I think that's a good analogy, and this is making me feel better about the future because I know, like, they're a big concern with

AI going forward as it runs more and more facets of society is making sure that it's explainable so you can look at it and know exactly why it's making the decisions it's making and it sounds like you guys are working on building explainability in at every step of the

process starting with the data which is super important yeah that's exactly right so there's a lot of ways to get at that kind of transparency and one of the important ways to do that is to have a really good handle on how the model arrived there and if you

know what the data was that went into the model then you can start to investigate if you start to identify any sources of bias really cool so a little while ago we had on Mark Messina he's the CEO of a company called Geek Plus and that was a really fun

episode because they do robotics in the logistics industry and he was literally sitting in his office with a glass wall behind him of a bunch of robots running around carrying boxes which was pretty cool but so obviously in these warehouses that are running mostly off of robotics uh they have like

robots carrying the packages to the humans that are putting them on like a rack where they need to go to be shipped out to the end user of the product obviously that's all a lot of it is run by AI and I was wondering

Steven Davis, VP Technology at Innodata - Enabling Data Scientists to Do More

Scripts