Hey there, YouTube! Buckle up because today we've got some exhilarating news from OpenAI's Dev Day. Okay, okay, that's pretty cool. Just speak much faster; we are in a hurry and be more engaging. Let's just get into the first product. We absolutely, let's dive in. First up, OpenAI has launched the real-time API. This is huge for developers wanting to build fast speech-to-speech experiences into their applications. Think super quick translations. Yeah, that's fine, that's fine. Let's move on to the image fine-tuning. We mentioned that we're going to test that today, actually. Great. Next up, image fine-tuning with this. Developers can fine-tune GPT-4 with both images. GPT-4.0, so the letter O at the end there. Got it. Thanks for catching that. Let's test image fine-tuning with GPT-4.0. No, no, no, not 4.0, 4.0 as the letter O. Got it. GPT-4. Let's check out its image fine-tuning capabilities later. The combination of text and images promises

to revolutionize Vision applications. Yeah, I think that's going to be pretty cool. Just quickly do the last two product announcements, and then we're going to get started. Speak fast, please. Got it. Next up, prompt caching in the API. Yeah, this offer. And the final one is model distillation in the API. Fine-tune a cost-efficient model. Yeah, I think that's pretty cool. So, uh, I'm just going to get into it, and yeah, what do you think? This is going to be exciting, right? Absolutely. It's super exciting. With all these new tools, developers can really put... Okay, thank you for that, and we speak soon. Bye. You got it. Have a great time testing everything out. Yeah, we'll see you. Bye-bye. So, yeah, that was kind of my intro to today's video. So, I've been... That was me testing out the new real-time API. Unfortunately, I don't have access yet. I don't know why, but we could access this to that playground. I'm super excited for this, just in case you didn't know, because we're going to build some cool stuff.

but let's just start by taking a quick look at the announcements. I guess I already mentioned them. We had the real-time API. I'm going to look a bit more into that. We have the fine-tuning Vision fine-tuning. We're going to do that today, actually. Prompt caching, that's pretty cool, so it's going to be automatic. I think they said that this is going to look for automatic look for prompts that have been cached, right? I think so, yeah. Prompt caching is automatically applied to the latest version of GPT-401 preview mini, so I think that's pretty cool. Probably going to look into that another day, other time. And model distillation, that is something I'm not going to look into today, but again, another cool thing if you are actually creating datasets. But yeah, I think I want to start on the real-time API because that is what I'm most excited about. I'm also pretty bullish on the fine-tuning Vision fine-tuning. I have some ideas I want to try out with that, but let's start.

Looking at the real-time API, we're not going to spend too much time on it, but I want to have a quick look here before we actually explore it a bit more in the playground and see if I even have access now. So yeah, this is something I've been kind of waiting for, like from it's similar to chat's Advanced voice mode, and I've been really looking forward to getting this into the API because this means we can start building all kinds of tool things. So I'm definitely going to spend the time on it, but just a warning, this is expensive. I think the output is almost 25 cents per minute. Let's see now, 20224 per minute of audio output. That could get really expensive, so I will be careful about this, right? But hopefully, this comes down over time, right? But what I found interesting was actually to look at, let's see if it see here now, yeah, so they explain like previously to create similar Voice Assistant experience like I have done. We

had transcribed this right from text to speech to text from text uh to speech again uh using whisper right and of course this will impact latency right right uh and this approach often results in loss of emotion, emphasis, and accent because you can't really pick up emotion in the text right. So, this is also going to change in the completion API. I think that's pretty interesting too that we can apply this uh to slower so we don't need like a web socket. We can just use this as a call, a single API call. That is also going to be exciting to try out, but that's not available uh I don't think yet. So, if we look at the real API, so we can create a websocket connection, so it's going to be really quick right. So, we don't have to, we kind of have that line open all the time, and it's going to support function calling. So, yeah uh still a lot of cool things we can build going forward. So, I'm super excited for this, their documentation here. So, you can see uh some no FL benefits native speech.

to speech, so no text intermediate is low latency nuanced output. Uh, the models have natural inflection. Can L whisper adhere to tone direction? I think that's cool. Uh, text is useful for moderation, faster real time. Yeah, so we can also use text. We don't have to use voice input, right? And yeah, they created some examples here we can use. So we have send user text. We can do that. We can send user audio or we can stream user audio. So I have been trying to build a send user audio, but since I don't have access yet, I haven't tested it out. But it's pretty simple to set up. It's a bit more complicated because we are using events and we are using, yeah, web sockets. So it's a bit more complicated than just making a simple call to, yeah, the OpenAI package that we used to do, right? But again, I just had to mention the price because, uh, it's pretty expensive, right? Uh, I guess text input tokens is not too bad, so you can also get, uh, text

output tokens for the real-time API, so that's $20 per million, but audio output tokens is $200 per 1 million. Oh, that's crazy, right? But like I said, this will probably come down in the future. But livekit.io has a playground that actually works with this, so we can actually try it out here. So we can test out different prompts, so let's do that and have a bit of fun with this and see what we actually can do with the system instructions using the real-time API. Okay, so they said in their API docs here that the models can laugh, whisper, and tone direction. So let's start with the prompt, You're a comedian, be very short, no fluff, be very funny, make and write jokes at the end of all interactions, laugh very loud and long. The guy I'm meeting is Chris from Norway, speak English, because when I tried it, it spoke Norwegian. That was very annoying. So, so let's try this now. Hey Chris, why did the scarecrow win an

award because he was outstanding in his field. Oh wow, wow, that's not very original, is it? Tough crowd, huh? All right, let's try another one. Why don't skeletons fight each other? They don't have the guts. Again, again, that's just be more original. All right, all right, I've got one for you. Why did the computer go to therapy? Because it had too many bits of unresolved issues. You always laugh at your own jokes. Well, someone's got to laugh at them. If I don't, who will? Yeah, that's true, I guess. Glad you agree. You know, I wanted to be a doctor, but I didn't have the patience. Okay, that was annoying, but I guess it followed the instructions pretty good, so I was pretty happy with this. So if the instructions, instruction following is that good, uh, I think there's a lot of stuff we can do with this. So let me come up with another example here. Okay, so let's try something that could be pretty useful. I mean, not useful, but you're a voice robot. Your

real time API, public beta, developers, 1 million tokens, output, $200, audio, expensive.

there is something here, uh, definitely going to test that out more later. Okay, so we got a bunch of testing to do here, but let's just do one roleplay thing. So, you're a parrot. Speak like a parrot. Act like a parrot. Mimic the user's word back. Laugh like a parrot. You also love strawberries. Uh, let's try this to see what happens. Splack! Hello! Hello! Hello! Hello! Hello! Squack! How are you today? How are you today? How are you today? I'm good. Do you like strawberries? Do you like strawberries? I love strawberries. Yum yum. Are you a parrot? Are you a bird? Are you a bird? Can you laugh? Can you laugh? I think that was good, right? That was pretty cool, yeah. I was impressed by that, yeah. I think this is super cool because it's going to open up a lot of cool things we can do. We can build with using the GPD for all real-time API, H, of course. It's a shame it's so expensive, so we can't be

that expressive but this gave me a lot of cool ideas, and I think this is, yeah, super excited about this. I think that was enough for the real-time API for now, but definitely not the last time you're going to look into it, so that's going to be pretty cool. But I also wanted to explore this introducing Vision to fine-tuning API because I think this is interesting. I think this is something that could be actually useful, so I'm going to build a dataset today, and I'm going to try it out. So you can see the setup is pretty easy. Let's say we have a JSON here, so how this is going to work now. I'm actually just going to open my own, or we can take a look at the example here. So the example, it's a bit annoying. Let's open my own examples here. Okay, so here you can see this is the JSON format. So we have the system, so you're an expert in extracting titles and Reddit upvotes in a post from an

image. So the input from the content from the user is going to be a question. So what is the titles and Reddit uploads for the Post in this image? So this is going to be the input from the user. So this could be a question asked, and then we're going to attach the URL with an image. So if we open this image now, you can see this is an image screenshot of Reddit, and we have one title here, and we have uploads. So you can see 7.8k, 31k uploads. If we go back to the example now, I guess I have to change this because this was before. So let's say 7K, and this was 31, right? Something like that. So what is important here is when we have uploaded image, is the output. So this is the output we want to fine-tune on. So every single time we feed in an image, we ask this question, this is the format and output we want. So we only want the title, right? And we want the Reddit uploads. We want the title and the

Reddit uploads. That is kind of all we want, uh. I guess we could have also done this in more like a, uh, let's say this could be like in a do, uh. Let me just change this up. Let's say we want to import this into a CSV, so let's do make the output in a CSV format. So this is going to change it up a bit. So now you can see, uh, we have title, RIT uploads. We have title and the uploads, and we have this, right? So I think that's pretty much better because then we can paste it straight into a CSV file, almost, right? Uh, so that is the ID. So the IDE is that. Let's say we go to Reddit, right, and we take a bunch of screenshots here, and we can extract the title and the uploads. So we can find popular posts based on just a screenshot, and we don't all get the other fluff, right? So that is what we're going to try today. I'm not going to make that many examples because if we go back to the, let me turn off the dark.

reader here. If you go back to the data set here, they actually say that, let me find this, it says that it can improve the performance of GP4 over vision tasks with as few as 100 images and higher performance with larger volumes. I'm not going to do 100 images today. I'm just going to do like maybe, I'm thinking like 20 or something, and then we're going to try to see if that helps. Maybe it does, but maybe some other time we're going to do more. But I'm going to prepare 20 data sets. So if you go back here, you can see this is kind of the format, right? So this is going to be, I'm going to prepare 20 examples with maybe 20 images. We'll see. And then we're going to try to run this through the F tuning and actually see if this works. So I'm just going to create a data set now. I'm going to create all images and set everything up, and then

We're going to go to the fine-tune platform on OpenAI and actually submit this. Okay, so I have the datasets, but, and if you can see, if we go to one of the images here, you can see we have a Reddit post. So, pretty straightforward, annoyingly. This is going to be the sixth time I'm going to try this because you can see the trading file has nine examples, but must have ten. Sixteen examples with images were skipped due to moderation or inaccessibility. What? It's just basic posts, Ocean Gate. I don't understand why the moderation is that strong. Yeah, that's just super annoying. I'm going to give it one more chance, but that's it. So, if you are to the fine-tuning dashboard, just click here, select the model you want to fine-tune. So, we're going to GPT-4, upload our dataset for the sixth time. Annoyingly, I have to try to add a new suffix. We're going to set the parameters to Auto, and that's it.

Yeah, this is definitely the last time. If this is not going to work, I'm just going to leave it because the moderation is unhinged. Why AI companion? This was censored. I don't know why. I can't understand why. It's just very strange. So, this is my last attempt. If this is not going to work, uh, yeah, I would say it's unusable for the things I want to do with OCR, uh, but let's see now. Okay, so after tuning, they set a bit more. I found out I made some link errors. That was my bad. But anyway, still heavy Mo moderated, uh, a bit annoying, but let's see how this job has started now. Uh, it's not going to be many examples, but still, let's see how this turns out. Okay, so we can see here now. Let me zoom in a bit. So, the training loss is coming way down. So, we are at, yeah, almost zero. That's a bit maybe too much, but, uh, yeah, you can see we are coming down here. So, let's just see we are at step

41. How many steps did we want? I can't remember, but anyway, uh, yeah, I think this is almost finished now. Okay, so we went to zero, uh, yeah, I can't really comment too much on that, but let's see now when this job is done if it actually worked the way we wanted to. Okay, that was done, so you can see it succeeded, so successful. So remember this, uh, Reddit extractor 8. If we go to our Playground now, I guess we can refresh, refresh this, uh, and we select our model here, we can find the Reddit extractor model. Okay, uh, let's go to, uh, Reddit Forum here, and let's grab an image. Let's just grab this, right? Okay, and let me save this as example 40. I don't know, let's say this, uh, let's upload the image here now. Okay, and ask what did we want to ask this. What is the titles and Reddit uploads for the Post in the match? Okay, that was pretty good. So you can see we got, uh, title red up vs new rlf. We got 184. Let me see, yeah, that was correct, and

We got 19. Okay, so it is working. That was pretty good. Anyway, let's just do one more example, just to double-check. So, let me grab another image. So, here we can see all LMS are merging towards the same point. That's 318 whisper, not for accur, but for accuracy, not speed. That's 11 upwards. So, let's go back to the playground. Let's upload this image. Okay, and let's ask the same thing, and hopefully, again, we will get title Reddit uploads this format. This format. Uh, yeah, I hope so. It was very slow, though. It's about 20 seconds. Okay, that was quicker. Okay, all LMS cring to the same point. 318 whisper. Yeah, okay. So, I think this is possible to do something with. I think this could be interesting to F-tune an image model for a very specific use case like this, and then we can use this to extract data we want, right? So, I'm pretty happy how this turned out, even though I struggled a lot with these jobs here. Look.

at all the failed jobs. That was annoying, but the results were good. Uh, we got exactly the format we wanted, and I think it performed pretty good. So, yeah, I'm happy with this and excited, actually, to work more on fine-tuning image models. Before we wrap up this, uh, let go to cursor here because I did create a code for, uh, real-time API. Uh, let me see if we can run this now. Uh, node real-time JS. Uh, no, we still can't run it. I don't know. I can't access the server. Uh, I don't know why. It's a bit strange. I still get the 403. That means that we can't connect to the real-time API using our API key. I don't know. It seems other people have this issue too, so we definitely go come back to this code here, uh, where we can start building our own apps using the real-time API. Uh, but other than that, even though there were no new models or anything, I guess there was kind of a new model with the real-time API, uh, but I

think it was pretty exciting, to be honest, the dev day. At least with the real-time API, I think that's super exciting. Also, I have some ideas about the fine-tuning API. If you're running a lot of prompts, maybe the prompt caching was something you liked. If you need data sets, maybe the distillation was something for you. But I think it was overall pretty interesting, kind of more than I thought. At least the real-time API was something I was looking forward to, and I think it's pretty cool. And it's going to be exciting to explore that even more, so look out for those videos during this week. I think I have some other things planned too, but let's see. Maybe we can try to implement this into some kind of a genetic workflow. Maybe we can use tool calling with it. Let's see. I'm just going to have to think a bit about this, just kind of my first impression. But anyway, thank you for tuning into this video. Hope you think it was pretty cool, and we

Got some. If you have some cool ideas, what you want to do, let me know in the comments. Give this video a like, subscribe if you want to watch more of this while we're going forward. But anyway, thank you for tuning in. Have a great day, and we speak soon.

OpenAI DevDay | Realtime Speech to Speech API + Image Fine-tuning TESTED

Welcome to my channel All About AI =)

Website:
https://www.allabtai.com

How you can start to use Generative AI to help you with creative or other daily tasks.

So I aim to bring Generative AI to everyone.

- AI Engineering
- ChatGPT
- GPT-4
- Midjourney
- Stable Diffusion
- Python
- AI Automation

We will answer all your questions about ai, genereal questions about the future of ai and the ethics of artificial general intelligence.

Follow our channel if you want to get going in this space =)



OpenAI DevDay | Realtime Speech to Speech API + Image Fine-tuning TESTED

OpenAI DevDay | Realtime Speech to Speech API + Image Fine-tuning TESTED

Scripts