Okay. So recently I came across this project called LlamaOCR. So this seems to be something that's been created by together.AI.

Now that they're serving the Llama 3.2 vision model, and specifically I think the 90 billion version of that model.

And see what they've basically done is make an NPM package that you can simply just import LlamaOCR, pass in a file path, pass in your together API key, and you will get a decent result out of this.

So, what I'll do in this video is take a look at how it actually works.

We'll look at recreating this in Python, so that could you use it in Python. We'll then have a look at like how you could do this fully locally with something like Ollama.

And finally I'll wrap it up by talking about how you could use this in an agent and give you some examples of how you could use this for helping to scrape websites, et cetera.

So let's jump in and have a look at what they built.

All right, to use this is pretty simple. We can just basically take a screenshot or something, drag it in there.

And automatically they will convert it and give us some kind of output on markdown. So you can see that here in this case, we've gone from texts that was not editable because it was a screenshot to having texts that's fully editable, and it's retained the title and stuff like that in here.

Now you can see if we take one of the receipts that I've done things with the vision Phi-3 with and stuff like that.

And have a look at this. We can see how does this actually come out? Okay, so this one's actually taking longer to process.

All right. So we've got in a Walmart, we've
got the different things out, and in this case, we've actually
gotten a pretty good run.

And what do I mean by that? Because this is a stochastic model,
we will get different versions. So you can see here early on
here's one that I did, where we got the item sold being really big.

We didn't get the Walmart being so big, and we didn't perhaps get all the sort
of formatting as good as we did this time, like here, we can see clearly that, okay, it looks like it's basically
gotten all the prices correct.

It's gotten the subtotal correct. It's basically put the tax
on one line, et cetera. And interestingly, it's put the
barcode number, this TC number, actually after what it's got in there.

So you're always going to have a hit
and miss with this kind of thing, but generally you're going to
find it can do a pretty good job.

Now, here, we can see that
it's putting stuff out of order that, okay, we've got this part, and then it's trying to do it as the
markdown part.

And then we've got in markdown hittings and notated with.

And then we've got some notes about
the markdown actually in there. All right, so let's jump into
the code and actually have a look at recreating this in Python.

All right. So if we come in here and actually have
a look at the code for this, we can see that on the whole, it's a pretty cool,
but reasonably simple sort of package.

So a nice example of just taking
something where you use a large language model endpoint or
in this case a VLM endpoint, and actually just turn it
into something really useful.

So they have the demo up,
which I showed you earlier on.

if we come in here and actually look at the code. Sure enough, this whole thing is just one file, one TypeScript file in here.

and we can see that basically it's just importing from Together AI. We can see there's choice of model in there.

And probably the most important part is that we can actually see what the prompt is that's being used for this.

and it's all being done in a single pass with this particular prompt in here. If we look at the prompt in here, actually, there's some kind of interesting things in here.

We can see that first off it tells it to only solely return markdown content without any additional explanations or comments.

that is not what we got all the time when we used the managed service, that sometimes it was giving comments about the content, et cetera.

And perhaps that really shows how good is it, how not good is it in here. All right. So if we want to recreate this with Python, I've just put together a simple little notebook that allows us to do the exact same thing in Python here.

So we're using the Together package because I'm going to stick to using their API since that's what the other one was already using.

And if we come in here, we can actually see the models that they've got in here.

So we can just come in and basically look at a list of the models and then I'm filtering it out for vision models.

Okay. So we've got the 11 billion model in there and we've also got the 90 billion. So we'll use the 90 billion.

I think that's what they're using on their site as well in here. Actually just look at the Together AI docs, we can see under vision we can see that, okay, they do have these two models supported.

They're also offering a free model endpoint if you want to use that.

So that you can actually, we could actually change it to this if we wanted to use a totally free one.

Though, I'm not sure how long that's actually going to be up here. And then we can just see, okay, they've got some examples in here of how to do it.

So there's two main ways that you can do it. You can either pass in an image URL where it will go and get the image from the URL.

and you can see that they're doing that here. You will need to change the streaming code out to, for it to work nicely in CoLab.

In the version I've got, I've already changed that. And then they've got another version which is doing it with a local image where you were going to basically need to base64 encode the image.

And in this case, you want to make sure that you've got something that if you're going to be using JPEG, you need to encode as JPEG.

So I've put in some code for taking care of all of these in the notebook. Lastly, a key thing that is important is just exactly how do they work out the number of tokens that you're using.

So they show that for vision models, images are converted from 1,600 to 6,400 tokens, depending on the image size.

So here is the actual formula for that if you want to actually calculate how many tokens that you're using each time.

So if we come over and look here, we can see that the 90 billion model is going to be a dollar 20 per million tokens and the 11 billion malls on the 18 cents per million tokens.

So definitely if you can get your tasks to work in the smaller model, it's going to be a lot cheaper here.

Okay. So jumping in here, we can basically look at the model lists. We can see that they've got a number of these models available.

Unfortunately, the one that I really would like to compare this against is the Qwen 2 VL model, which for some reason they don't seem to be serving on their service.

Then if we just take their code, we've got two ways of getting the image into the Together API here.

And this is pretty consistent with Gemini, pretty consistent with other models and stuff as well, is that you tend to either basically do it by sending it a web URL and putting it up there or you do it locally.

So this is the example of sending a web URL. You can see that this basically is this image which comes from this article, about Sam Altman being involved in this startup called Thrive AI.

And it's basically just their user interface in here of their product.

So we can see there's image credits and stuff like that, but we can also see that it's got UI in there.

We can see a bunch of different things. We've got morning report, noon report, evening report, et cetera as we go through this.

Now I fixed up their streaming a little bit in here. The other thing though, is we're using their particular prompt yet, which the prompt is quite different.

So this is, describe the attached screenshot or UI mock up. Give us information of headers, footers, use the exact text from the screenshot.

So while we can see that it is doing it, is getting the text out, and it is OCRing the text.

We're getting a lot of description about what is actually on the page as we go through this. And I guess the main goal here is not to have that.

We really want the OCR just to give us the actual text that came back and then we can use this. So one of the techniques that you can do though, is have one pass where you get a description out, and one pass where you get the OCR out, that can be quite useful.

Alright, the second way to do this is to load the local images here.

So here I've basically got our Walmart receipt that I've used in a number of videos now. And you can see for this, I'm actually just using a real simple prompt of what is the image.

And sure enough, we're basically now going to encode this. So when we encode it, we need to make sure that the mime type of the encoding matches what we actually encoded.

So this is a JPEG image. We can see the mime type down here is also a JPEG image in here.

If this is a PNG, we need to basically replace this. So you'll see that I've created my own class in a second for doing this.

We're passing in the model there. again, fixing up the streaming. And you can see that sure enough, the output that we get while it's consistent, we still have a bit more of that sort of description going on.

It looks like it's getting most of the information out, but we've got these sort of really clear descriptions that we don't necessarily want.

All right. So what I wanted to do was make my own sort of edited version of this, where we actually make this image process a class, and then we can pass in the model.

We can pass in the prompt. You can see the model's going to default to the 11 B version in here.

And that one of the other things too, is based on what we put in, it will basically work out what is the correct mime type for the image.

So it can support PNG, JPEG, GIF files and WebP files in here, and it will then handle the encoding for us.

It will then pass this up. And you can see the prompt that we've got and the model that we've got set in here.

And then you can see to use this, all we need to do is instantiate our image processor, pass in a new prompt, if you want to override the prompt.

And actually here, what I'm going to do is go with the system prompt that they have. So this is the system prompt, taken from what they actually used in their LlamaOCR.

So I basically just passing that in. And then I can pass in some photos. This one actually doesn't exist.

So I want to test that, will it fail gracefully, et cetera, as we go through this. And can see sure enough, if we take that receipt, but now a PNG version of it.

it has no problem of sort of, you know, handling that. And you can see that now because we're prompting it specifically for the OCR, we're getting much more of sort of the markdown output.

Okay. So just to test out the MIME types part.

I'm going to put in a WebP image and you can see that it handles this quite nicely. So this is the image that I used earlier on.

It handles quite nicely. It extracts it out, actually gives the link as a URL link in here. Although, if we look at it carefully, we can see that it's saying that this is a heading, whereas this is clearly not any more of a heading than the second line or the other lines in there.

And this kind of brings me to the point around some of the issues that you're going to face in doing this is that you can't always rely on the VLM to extract, the OCR and the structure perfectly.

Okay. So one of the ways that you can get structure is by creating what we call a regions of interest model.

So this is a picture from a project that I worked on about six years ago.

At the time Google didn't have a good OCR system for Thai. And this particular customer wanted to be able to scan ID cards and extract out the various information.

So the way this works is you basically train up an object detection model to find the relevant parts in there.

And then you pass each of them to an OCR system. So in this case, I trained up an OCR model for Thai.

and we trained a separate object detection, regions of interest model to actually find the different parts. And then we'd basically take those parts and run the OCR on those specific boxes.

Because of that, we knew exactly the structure of it. And so we knew what was what in each place kind of thing.

So, if you're trying to do like a hardcore OCR task, you can go for sort of regions of interest models.

The other trick though, that you can do, which works pretty well is that with the smaller model, if you don't have huge issues around latency, et cetera, you can actually get it to do the OCR multiple times, say three times, and then pass each of those three things into a larger language model for a language model as a judge and tell it, okay, here are three versions of this OCR.

Because one of the tricks that you find is that these things don't tend to hallucinate consistently. Often, when they hallucinate, they will get it right two out of three times, and then get it wrong one time.

So if you can give the things to another
large language model, that's one way that you can fix up some of the mistakes.

And then you can also do that with
playing around with the prompt to help it in that kind of style.

So since the model did so well
here, the other thing that you could do is you could run this locally. So you can convert this code
to basically run the Ollama version locally, quite easily.

And then if we come and look at Ollama,
we can see that Ollama has got both the 11 B and the 90B versions that you can run.

Now the 90B version, you're probably
going to need quite a lot of RAM for that.

So maybe that's not that practical,
but certainly the 11B version is something that could run quite easily
on a home computer to do this task.

So lastly, just to finish up. If you're going to be doing this
kind of thing with agents, where this can become really useful is
where you're scraping web pages that have information on them like this.

So here, we've got this article and
then it's got the actual UI in here.

LlamaOCR - Building your Own Private OCR System

HI my name is Sam Witteveen, I have worked with Deep Learning for 9 years and with Transformers and LLM for 5+ years.  I was appointed a Google Developer Expert for Machine Learning in 2017 and I currently work on LLMs and and since earlier in 2023 on Autonomous Agents. 