AI 웨이푸 브이튜버 만들기: 설정 및 기능 가이드

AI 와이푸 브이튜버를 제작하려면 코딩, 오디오 처리, 인터랙티브한 경험을 위한 다양한 API 통합이 필요합니다.

### 나만의 AI 와이푸 브이튜버 만들기: 단계별 가이드
* 🎤 시작하려면 코드를 실행하고 마이크 사용 또는 YouTube 실시간 스트림 채팅 중에서 선택하여 **마이크 옵션**부터 시작하여 **오른쪽 시프트 키**를 길게 눌러 오디오를 녹음합니다.
* 💻 코드는 깃허브에서 확인할 수 있으며, 다양한 라이브러리와 **VoiceVox 엔진**을 설치하고 OpenAI 및 DeepL**에서 **API 키를 받아야 하며, 후자는 **구글 번역도 사용할 수 있으므로 선택 사항입니다**.
* 📜 AI의 지식은 캐릭터의 **배경**을 설정하는 `lore.txt` 파일에 저장되며, 메인 코드는 Whisper AI를 사용하여 오디오 입력 및 녹취를 처리하여 **상황**에 맞게 번역합니다.
* 💬 대화는 변수에 저장되어 **단기 메모리**를 유지함으로써 AI가 **상황에 맞게** 응답할 수 있으며, OpenAI의 GPT 3.5 Turbo는 원하는 언어로 **번역**된 답변을 생성합니다.
* 🔊 오디오 응답을 생성한 후, 프로그램은 **VB 오디오 가상 케이블**을 통해 재생하여 VTuber 아바타가 오디오 출력과 입 움직임을 **동기화**할 수 있도록 합니다.
* 🖥 실시간 채팅 모드의 경우, 프로그램은 YouTube 실시간 채팅을 캡처하여 메시지를 처리하고 실시간으로 응답을 제공하여 **텍스트 파일**에 저장된 응답과 함께 **자막**을 OBS로 표시합니다.
* ⚙ 설정 성능은 컴퓨터 사양에 따라 달라질 수 있지만, 전반적인 프로세스를 통해 **참여형 인터랙티브 AI VTuber 경험**을 할 수 있습니다.
* 👋 영상은 감사의 인사와 함께 설명된 방법을 사용하여 나만의 AI 와이푸 가상 유튜버를 **만들어 보세요**라는 초대로 마무리됩니다.

okay guys, so today we will make AI waifu VTUBER like this. First, I will run the code, and the code will tell me to choose whether I use a microphone or YouTube live stream chat. For now, let's choose number one, and then I will press and hold right shift to record audio from my mic. Okay, what is your name? Tell me a good place in Japan, Tokyo. Why do you recommend that place? Okay, for the code, I already posted it on my GitHub repository. You can check it on the description link, and in this video, I will try to explain how to use this code, and I will explain the code. So, if you already cloned the code, you will see there are run.py. So, first, you need to install all the libraries that will be used in this code, and beside this, you will also need to install VoiceVox Engine, and you will need an API key from OpenAI and DeepL. The DeepL is optional; you can also use Google Translate. And of course, you will need to install VB Virtual Cable. What is this for?

I will explain it later. Let's way back to the code. So, in this code, you will need, like what I said before, you will need an API key from OpenAI. You can get it from the OpenAI website. And I save my API key on config.py. And then you will need lore.txt, where inside this, the txt will save the lore of your AI. So, for example, I write in here. So, I make the lore for my assistant. You are an AI Waifu Virtual YouTuber called Pina. Your creator is Ardha. He made you using blah blah blah blah blah. You answer with brief, to the point, and with no elaboration. And then the code will run from this section, the main section. So, you will need to input either 1 or 2. 1 if you want to use your microphone, and 2 if you want to get the AI to answer live chat, live stream from YouTube. So, for example, we choose one. If we choose one, you need to hold right Shift on your keyboard to make the program record your audio.

and then the program will call the record audio on the Record Audio. It will record your audio while you hold your right shift, and then it will save it in input.wav. After that, the input.wav will go to the transcribed transcribe_audio function. In this function, it will transcribe the audio into text using Whisper AI. So, for example, you say how are you, and then this function will transcribe the audio from your microphone into text after it translated into text music. So, in here, I translate the input from the user from Indonesia to English. Why? I translate it to English because it will be easier for the translator if we translate EN to JP compared with ID to JP. But if you are an English speaker, you don't need to use this code or this function. And then it will save the conversation into the variable conversation. This variable conversation will make your AI have a short-term memory, so it will be more human or be more contextual. And then after that, the code will call openai_answer in openai_answer.

first we will get the total number or total number of characters inside the conversation. Why we get the total character? Because OpenAI only can receive input in 4K character. I think 4K something. If the total character is more than 4000, it will remove the second dictionary from the list, so it will make sure your conversation is not more than 4000. After that, the conversation will get into the message and get an answer from OpenAI using GPT 3.5 Turbo, and then the response will also append to the conversation. And then the message that we get from the response from OpenAI, we will send it to the translate_text function. So in this function, we will translate. First, we will translate because the answer is in English. I translate it to Indonesia, and then I print it. I also translate it from. It must be EN. I forget to change it. So the text that we get is in English. I translate it into Indonesia so I know what it is, what is the answer in Indonesia. And then I also translate it to Japanese, where the Japanese text.

It will be used to convert the string into an audio in Japanese, and then we will send the result JP and result ID into speech_text. So, in this function, we will call katakana_converter, where this function will translate the English word into katakana. For example, if there is the word ORANGE in English, it will be translated to orenji. Because if it is written as ORANGE, your AI will spell orange O R A N G E, not orenji. So, we will need to convert it into katakana, and the katakana converter is on katakana.py. And also for the translate, there is in the translate function on translate.py. You can either use Google Translate or DeepL. If you want to use DeepL, you can use this function. And if you want to use Google Translate, you can use this function. And here I use DeepLX. DeepLX is a free and open-source DeepL API. And I use DeepLX because I don't have an authentication key from DeepL Pro, because DeepL is not available in my country.

If you want to use the DeepL, you can make the function in here, and then after you convert it into katakana, you will need to make requests to VoiceVox. Don't forget to run the VoiceVox Engine first before you run the script. To run the VoiceVox engine, you can use Docker. Okay, you can use Docker. You can run it on CPU or GPU. If you want to run it on GPU, you can run this 2 line of code, this and this. And don't forget to install, of course, to install the Docker. And after you run the VoiceVox Engine, you can request where the text is katakana, katakana_text, and the speaker is. I choose this 46. So where are you know, where do you know it's 46. You can try the audio that is available in VoiceVox in here. You can try, you can listen to the example of all the voice actors in this website or in this VoiceVox website. Can try and listen one by one if you get the interesting voice.

you can go to speaker.json in here, and you can find the ID number of the speaker in here. For example, this is the name of this speaker: I don't know how to read it, which is this one. So, she has 6 voice variations: 1, 2, 3, 4, 5, 6. You can choose one of the 6 variations of her voice in here. If you want to use this variation, you can choose 37 in the speaker section in here. So, after we request it, music, we will get the content of the request. We will receive the audio and write it in output.wav, and then after that, we will write on output.txt. We will write the result_id on output.txt. This will be used to display the subtitle on OBS. So, result_id is in here. The answer from OpenAI will be translated into Indonesia and then saved into result_id, and then it will be saved in output.txt. After it saves it in output.txt, it will appear in wait a minute in this. I use the output.txt in here.

so the output txt will appear on the OBS display. In here, you will see it later. And then we will also save the chat.txt, which we get it from the chat_now variable. The chat.txt will save what the user says or what the live chats say and save it into a txt file. The txt file will be used inside this section in OBS Chat.txt, and then it will appear on the upper side of the OBS display. We will use time sleep so the txt will be already written in txt format, and then we will play the sound. We'll play the sound that is already in output.txt. We set the is_speaking to true when it plays, and we set the is_speaking to false when it has already finished playing. The speaking is used to prevent the assistant from speaking more than one audio at a time. And then after the assistant is finished playing the sound or saying the audio, we use time sleep for one second, and then we delete the inside of output.txt and chat.txt. So it lets the subtitle appear when the assistant is

playing the audio or saying the answer, and then after one second, it's finished to say the answer. The subtitle will be gone (disappear) that if you choose the mode 1. If you choose the mode 2, it will run the preparation using thread. And also run the get_livechat function in the same time, so the get_livechat will get live chat from YouTube, get livechat from YouTube, blah blah blah blah blah. In here, if the chat author is the owner or the chatter is Nightbot, it will pass it or continue the code. Hmm, if you want to include the chat from the owner, you can delete this. And then we use this re sub to remove emoji from the chat because sometimes in live chat, there are emoji that will be like this. So I use this to ignore the emoji text like this. And then if the chat, the chat length is more than 5, I will save it into conversation. And I also save the author name Berkata: is Said: the author name said blah blah blah blah blah.

then I save it on the conversation. Why I save the author? Because it will make your AI know who is talking right now, and your AI will be answering to this username. I think you know what I mean. It will run the preparation also. So, in the preparation, if the assistant is not speaking and the chat is not empty, and the chat is not a command, and the chat is not the same as the previous chat, the assistant will answer the chat. So, I use this if the assistant is not speaking and the conversation line is more than one, and it's not start with exclamation mark, because sometimes another streamer uses exclamation mark as bot command, you know, something like !duel or something like that. So, the exclamation mark, I use it on the if condition, and if the chat now is not the same as chat previous, it will call openai_answer, and then it will be same as the mode 1. Okay, it's all for the explanation. Now, let's re-run the code for run the code. First, like what I said, you need to run the VoiceVox engine.

so because I already pulled the VoiceVox engine, I just need to run it. Open CMD, I run this, but I face an issue when I run this program. I need to delete this, so it will be just the port, enter. Okay, it's running, it's running, and we see on the Docker the VoiceVox Engine status is running. You can click it on this link, it will open a new tab on your browser. Okay, it's not running. Okay, it's already running. Now we refresh it. Now after this, if you want to see the docs of this API, you can /docs. If you're familiar with API, you can try to use this API. And then after the VoiceVox is running, we can run our code, our code, our code. You can use CMD or terminal from Visual Studio Code. For now, I will use terminal from Visual Studio Code. python run.py. Okay, my computer, my laptop is a little bit laggy because I open so many applications, so the program will need you to input 1 or 2 mode, 1 or mode 2.

let's now we try mode 1. OK, hold right shift to record audio. Let's try hello, how are you? OK, the bot is answering. It's generating the audio. It will be, it will be, need a time, so it will be like that. It needs a little bit of time because I open so many applications inside my laptop. Ah, I forget. Let's back at mode 1. Give me suggestions on what I must visit in Japan. OK, so the bot will have a short term of memory. We can just say why? instead of why I must go to Tokyo Tower? So I will just say why? The bot has a sort of memory, so it knows what the meaning of why? We can use the output from the code and use it inside of Vtube Studio. So you see, the Vtube Studio is responding to my mic as an input. So to use the code, the audio from the code as an input for this Avatar, for this Pina Avatar, or my Avatar from Visual Studio, you can go to

settings, and then you find where the mouth opens. You find the mouth open, and then change the input into voice volume. After you change it into voice volume, and then you need to change the microphone. Change the microphone and select the VB Audio Virtual Cable. VB Audio Virtual Cable, so it will use VB Audio Virtual Cable or cable output as the microphone, and then change your desktop audio into VB Audio Virtual Cable or the cable input. Okay, you can install it. You can find the tutorial on YouTube on how to install this. I will not explain how to install this in this video. So, the general logic is the code will play the audio inside the VB Audio Virtual Cable, and the VB Audio Virtual Cable will be used as an input inside the Vtube Studio, and then it will move the mouth of my character. So, let's try it. Like, for example, I say in this code, Do you have any suggestion besides Tokyo Tower? And then we see the Vtube Studio. It will be like that. And you use game capture to

to capture your Vtube Studio inside OBS, and then, like what I said before, you insert text like this question is from chat.txt and answer is from output.txt. The setting I use is like this, and then the setting of the transform for the answer is like this, and for the question is like this. Okay, let's try it once again. Do you have any suggestion of food in Japan? Let me see on OBS. It will make the subtitle. Please try Sushi, Ramen, Tempura. This after that will be like this. Okay, that's all for mode 1. So for mode 2, mode 2, mode 2, select mode 2. You will need to input Live Stream ID. So for the test, we can find another streamer that is live at this time. We can go to YouTube live music. Okay, for example, I use this one. Copy the link address. We place it in Notepad, and we just want to use this. Okay, I paste it here, and it will capture the live stream chat and answer it, and automatically answer it using OpenAI.

Okay, we just need to see our OBS. Okay, for the subtitle, I don't get the best result on editing the subtitle because sometimes the subtitle is bigger, sometimes smaller. I'm still finding the best setting for the subtitle. Okay, it's a little bit laggy. Okay, it will be the fast how it will respond. It will depend on the specification of your computer or your laptop. Okay, that's all for the explanation on how to make your own AI waifu virtual YouTuber. Thank you for watching and goodbye.

How to Make Your Own AI Waifu Virtual VTUBER or Assistant

Ryusui Kagaku Ch.

이 과정은 상호작용을 위해 마이크 또는 YouTube 라이브 스트림 채팅 중 하나를 선택하는 것으로 시작됩니다.

이 코드는 다양한 라이브러리, VoiceVox 엔진을 설치하고 OpenAI 및 DeepL(선택 사항)에서 API 키를 얻어야 합니다.

이 프로그램은 오디오를 녹음하고, Whisper AI를 사용하여 텍스트로 변환하며, 사용자의 입력을 번역하여 AI를 위한 단기 메모리를 생성하여 상황에 맞는 응답을 제공합니다.

대화의 총 글자 수가 OpenAI의 4000자 제한 이내로 유지되도록 모니터링됩니다.

5 터보의 응답은 영어에서 인도네시아어와 일본어로 번역되며, 일본어 텍스트는 오디오 변환에 사용됩니다.

스크립트에는 영어 단어에 대한 가타카나 변환기가 포함되어 있으며 DeepL 또는 Google 번역을 사용하여 번역할 수 있는 옵션도 제공합니다.

화자 음성 변형 및 실시간 채팅 통합 관리하기

화자의 ID와 음성 변형은 speaker.

json 파일에서 확인할 수 있으며 6가지 옵션을 사용할 수 있습니다.

오디오 요청이 처리되고 result_id가 output.

txt에 저장되어 OBS에 자막이 표시됩니다.

txt에 저장되며, 소유자의 채팅을 포함 또는 제외할 수 있는 옵션이 있으며, 명확성을 위해 채팅 텍스트에서 이모지가 제거됩니다.

작성자는 AI가 대화에서 적절하게 응답할 수 있도록 사용자 이름을 저장합니다.

VoiceVox 엔진은 CMD를 사용하여 실행하고, Docker에서 상태를 확인합니다.

엔진이 실행되면 작성자는 비주얼 스튜디오 코드 터미널을 사용하여 코드를 실행하고 원하는 모드를 입력할 계획입니다.

브이튜브 스튜디오에 VB 오디오 가상 케이블 사용

봇은 단기 기억력을 가지고 있으며 "왜?"라는 간단한 질문으로 질문에 응답할 수 있습니다.

VB 오디오 가상 케이블을 통해 봇의 오디오 출력으로 브이튜브 스튜디오를 제어할 수 있습니다.

설정은 마이크와 데스크톱 오디오 입력을 VB 오디오 가상 케이블로 변경하여 봇의 오디오가 Vtube Studio에서 캐릭터의 입을 애니메이션할 수 있도록 하는 것입니다.

이 과정은 OBS 내에서 Vtube Studio를 캡처하고 채팅 질문과 답변을 위한 텍스트를 삽입하는 것입니다.

모드 2는 라이브 스트림 ID가 필요하며, 설정은 채팅 캡처에는 라이브 유튜브 스트림을 사용하고 자동 응답에는 OpenAI를 사용할 수 있습니다.

OBS의 자막 편집은 컴퓨터 사양에 따라 응답 속도가 달라질 수 있으며, 아직 최적화 중입니다.