MacBook M1에 Kafka 설치 및 테스트하기

### MacBook M1에 아파치 카프카 설치 및 테스트하기
* 💻 오늘은 많은 회사에서 고성능 데이터 파이프라인 스트리밍 분석을 위해 사용하는 오픈 소스 분산 이벤트 스트리밍 플랫폼인 **Kafka에 대한 소개**부터 시작하여 **MacBook M1에 Apache Kafka를 설치하는 방법**을 안내해드리겠습니다.
* 🔗 시작하려면 Google에서 "Kafka Apache"를 검색하여 공식 Apache Kafka 사이트를 방문하면 **기본 버전 및 Confluent Kafka와 같은 엔터프라이즈 버전**을 포함하여 Kafka를 다운로드할 수 있는 리소스를 찾을 수 있습니다.
* 📥 설치하려면 **G 형식 등 원하는 버전**을 다운로드할 수 있으며, 다운로드가 완료되면 원하는 폴더에 압축을 풀고 바이너리 폴더로 이동하여 Kafka 에코시스템을 실행하는 데 필요한 스크립트에 액세스합니다.
* ⚙ 다운로드 후 특정 줄을 주석 처리하고 로컬 사용에 적합한 IP 주소를 설정하고 터미널에서 **Confluent의 홈 경로**를 설정하여 `server.properties` 파일을 구성해야 합니다.
* ☕ Java 15와 같은 상위 버전은 지원되지 않으므로 성공적인 작동을 위해 **Java 11로 전환**해야 할 수 있으므로 **JDK 버전 1.8 또는 11**을 사용 중인지 확인하세요.
* 🛠 Java가 올바르게 설정되면 **`confluent local services start` 명령**을 사용하여 Kafka 서비스를 시작할 수 있으며, 그러면 **Zookeeper, Kafka 및 기타 에코시스템의 구성 요소**가 시작됩니다.
* 📊 서비스를 시작한 후 **`localhost:9021`로 제어 센터에 접속**하여 테스트 주제를 생성하고 메시지를 생성 및 소비하여 기능을 확인하여 설치가 성공적으로 완료되었는지 확인할 수 있습니다.
* 마지막으로, 모든 서비스를 중지하고 생성된 토픽을 정리하여 깔끔한 환경을 유지하여 **MacBook M1에서 원활한 Kafka 환경**을 보장하세요.

Kafka는 효율적인 데이터 파이프라인과 분석을 지원하는 강력한 오픈 소스 이벤트 스트리밍 플랫폼입니다.

Today, I am going to install Kafka on a MacBook M1. Some of you might not know what Kafka is and its uses, so not to worry. I will show a video about Kafka coin details in the future, so let's get started. So, if you Google Kafka Apache, you will go to this link. This is the official Apache Kafka site. So, if you go, it is saying, What is Kafka? Apache Kafka is an open-source distributed event streaming platform. So, this is open-source. You can use it without a license, and it's used by thousands of companies for high-performance data pipeline streaming analytics. You can do so many things. So, this is a very popular messaging system. But in this, it's Apache Kafka license. If you are going, you will have basic Kafka like brokers, keepers, like you have limited assets. But if you go to enterprise Kafka, Confluent, then you will have so many components in it. So, if I can search Confluent Kafka, okay, so if you go to this website, you will see this page. So, you can

download from here, but here's what you need to do: get some license for this product. So, if I want to download, you need to go get started free, and you can go on this link, put your email address. Like, this is my email address, and you can download the manual version, okay? And you just click on agree, it will download for you. Format I need to choose, so here we have so many formats. If you are installing on Docker, you can get the Docker image, Debian, RPM package, entire, and chip. So, I'm going to download the G format, okay? So, it will take time to download it, but I have already downloaded it, so it's so don't worry about that. So, once you download it and unzip up to your choice, what folder you wanted to install. So, if I go to my terminal, let me screen big screen font, okay? So, if you go here, I already downloaded and copy-paste to my folder here, like Configurant 6.0.1. So, once you download, you will see inside. So, if I go to cd confluent

version 6. You will see this folder structure. So, in the binary folder structure, you have all the scripts you need to run the Kafka ecosystem. So, this is, um, if you wanted to run locally. So, this is the, like, um, CLI command. Confluent, you can start Kafka stock. So, you can, you can go with it. And you have Kafka producer, Kafka consumers, cells as well, like, script. So, if you want to produce and consume some Kafka message, so you can play around it. Okay, so now the question is, what were the steps we need to do to perform all those things? So, once you download it, you need to go to, uh, this folder, like, uh, if you do the LS, CD, go to CD, or you can do the edit. So, we have, I'm using the beam command. You can edit it. So, go to BIM, ITC server dot properties, ATC, sorry, Docker server dot properties. So, here, so this is nothing. This is just a configuration file. So, somebody, I already opened it here in VS Code. So, this is just, um, here it looks good to you. So, what you

need to do so once you download it. Some of this is uncommented, and so you need to uncomment it, and you need to put zero zero in this one. So, this is something like localhost kind. It will use your IP address to run the platform. So, and you need to uncomment this one and put here localhost. And also, you need to uncomment this one. So, this is the protocol how zookeeper and kafka communicate together. So, we are using plain text, like no SSL configuration at all. So, we need to go with this one. So, now once you install, what you need to do next step is you need to set the home path of the confluent cocktail. Okay, so if you are using MacBook M1, so sim, you need to go like if you go to parent folder here, put folder. So, you need to do bim zhrc, that shrc, and edit it. So, here you need to put the variable like export, give the variable name confluent home, and you need to give the path where your folder is present. Okay.

and you need to add this bin by library bin folder because in this bin folder you have all the binaries what you need to run while you're performing operations. Okay, and then also you need to include in the path so your confident home path is always this one but also we are including the binaries with the path. Okay, so once you done it and edit it and if you go to just source, source means you need to include in your path in this terminal. So if you run eco dollar confluent home then it will show you this your home part that mean your kafka is installed and you are able to access the path. Now I am using this terminal so this terminal is our apple architecture. Now if I am going to try to run so I have listed all the commands what you need to perform basic operation. So we have two constant one is you need to always use jdk 1.8 and jdk 11 version. Okay, you can't use java 14, java 15. So if I am going to run suppose to

start my Kafka. So, if I am going to run this Confluent architecture in my Apple terminal, I need to run with this command, extra command like architecture, and you need to put this hyphen x8664, then run in this Kafka. So, this is one way. So, suppose if I want to start this in my normal terminal, so it will going to work, but you need to put extra two lines, two commands, okay. So, if I am going to run here, so it will say, if you say trying to restart the Zookeeper and trying to start the Kafka, but I believe it's going to fail because they are saying this is the minimum requirement of version is JDK 1.8, 1.11, but what I currently install, the Java version 15. So, if I run it, check the Java version, what you have. So, here, if you see, I have installed Java 15, okay. So, this is the not right version. They are not supporting currently Confluent Kafka. So, what you need to do, you need to just use this one.

to switch over to use Java 11 version. So once you switch over Java version and check what is your Java version, so you will see Java 11 working. So if I start this command, it will going to work okay, but I don't want to you again to type these two extra, you know, architecture 1186 to start your Kafka. It's better you have to one more option you can go with that and it will, you know, need to put this extra two. What is that option is so Apple is supporting like if you go to terminal utilities. So here you have you are using the terminal. So if you go to edit this terminal and enable this extension like more info. If you click on the more info here, actually, it's already showing to you. So if you check out the checkbox, open using the row setup and enable it and click on the double click, it will work for you. But what I have done it, I enable this one and I clone one more terminal like I gave name is Rosetta terminal.

and I enabled this checkbox here. So for me, it's easy. If I'm doing some Intel architecture things, then I don't need to again do the checkbox and check. So I already installed. So if I'm going to open this terminal and run it, so let me do the font size figure so that you can see it properly. Okay, so now let's start all the steps. So first, I need to check what is the Java version I have currently here. So here, if you see JDK 15 is enabled. So I need to go to enable Java 11 to support it. And if I run it, so now our Java version is set back to 11. Now we are good with it. So next command, what you need to run, you don't need to run this, just start confluent local service start. Okay, confirm local services start. So what does this mean? This is going to start all the Kafka services you need in this ecosystem, convenient platform. So if you run it, so it is trying to install Zookeeper.

like zookeeper, then kafka. So, we have so many components, so don't worry about all these parts, how they are working together. We will go over there in future videos. So, if you see, zookeeper is started, kafka is started. Now, this is schema registry starting, kafka rest, and then kafka rest is up now, trying to connect. So, we have so many components. If you are going to work with confident kafka, but this is licensed version. So, if you are doing for your local development, that is good, but you can't use kafka in production. You need a license at least, okay? So, now it's starting the control center. So, control center is the last component of this kafka architecture. So, once it's done, then you'll be able to access the control center, okay? So, let's wait for a few seconds. It will going to be up, okay? So, it's taking time. Don't worry, it's going to start. So, now our kafka is started. So, how you see it's working or not, you need to go to one more time and just say local.

host and colon 9021. This is the port number where your contact center is running. So if you search it, then now your Kafka is working perfectly fine. So let's create one topic here and test is working or not. Okay, so if you see that we have brokers and this is internal partitions replicas topic. This is the Kafka ecosystem is creating for you, so you don't need to go worry about that. So let's test this. The basic test is working or not, or installation. So I'm going to click on the add topic and give the topic name is just test. Okay, so I'm clicking on the create default partition one. So don't worry about what is the more detail. So I'm just saying clicking on the create the defaults and now our topic is created. So if you don't see any message at all, right? So how we can put the message in this topic? You need to create, you need, you have to like produce something and consume something. That way messaging system works. Okay, so I'm going to here, I'm going to create

um like I'm going to run the producer, so you use this command. So I already told you in the bin folder, you have so many supporting, you know, binaries. So if you want to produce something, so you need to run binary Kafka console producer topic topic name. So if you see the hyphen hyphen topic, this is your topic name, and this is your broker list. So this is my Kafka broker, and it's running on the 9092 port. Okay, 9092. Okay, so if I'm going to run this one and enter here. Okay, so it will going to. So my Kafka producer is now available. Let me add one more terminal, new tab. Okay, basic. And I'm going to run the command which is going to consume that message. Okay, so suppose here I am going to run this command Kafka consumer. Okay, so now if you produce something higher snail, it's going to consume here. Okay, you see high sunny is coming very well instantly. Okay, similar say hi there. Okay, hi there. So if you see here is hi

there automatically coming to you that way you are using some messaging services you already know. So if I go to here in messaging service, so here his message should show you like consume, consume, and produce, but it's taking time. Let me repress it. Okay, so it's not saying here, but that's okay. So now is working fine for you. So your now Kafka setup is done. We are good here, and we can close Ctrl+C to command exit from here. Similar, you can exit from here. Okay, so now your producer and consumer are both closed. So what you need to do now, you can stop all your services. So both services are running. So you can go here and use this command to stop all your services. Now we are stopping for control center, then ksql. So all they are installing, like stopping in one by one. So once it's stopped, you will not be able to run this control center because this prompt is installed. Okay, so let me see it's stopped artillery. So you can, you see, this is not reachable. So stop so

now if you want to clean up everything, whatever your topic created, then you need to run this command to clean up your topic. You created the test topic, so it will clean everything from the hair. So, this way, you can install Kafka and test if it's working or not. Thank you for watching. Take care. Stay safe.

How to install Confluent Kafka in mac m1

Shri Krishna Sankirtanam

Kafka는 고성능 데이터 파이프라인 및 분석에 사용되는 오픈 소스 분산 이벤트 스트리밍 플랫폼입니다.

엔터프라이즈 버전인 Confluent Kafka는 추가 구성 요소를 제공하며 다운로드하려면 라이선스가 필요합니다.

Kafka를 다운로드하고 압축을 풀면 바이너리 폴더에 Kafka를 시작하고 메시지를 생성/소비하기 위한 명령을 포함하여 로컬에서 Kafka 에코시스템을 실행하는 스크립트가 포함되어 있습니다.

MacBook M1에서 Confluent Kafka 설정하기

구성에서 특정 줄의 주석을 해제하여 Zookeeper 및 Kafka의 IP 주소와 프로토콜을 설정해야 합니다.

Confluent Kafka의 홈 경로를 설정하고 경로에 바이너리를 포함하세요.

상위 버전은 지원되지 않으므로 JDK 1.

11을 사용하고 있는지 확인하고 올바른 아키텍처 플래그를 사용하여 Kafka 시작 명령을 실행하세요.

Java 11로 전환하고 Kafka 서비스 시작하기

이 프로세스에는 Java 11로 전환하고 터미널의 Intel 아키텍처용 Rosetta 설정을 사용하는 것이 포함됩니다.

Java 11을 설정한 후, "confluent 로컬 서비스 시작" 명령을 실행하여 Zookeeper 및 제어 센터를 포함한 모든 Kafka 서비스를 시작합니다.

이 설정은 로컬 개발용이며, 프로덕션 사용에는 라이선스가 필요합니다.

카프카 테스트하기: 토픽 및 메시징 만들기

Kafka는 포트 9021에서 실행 중이며, 에코시스템은 브로커, 파티션 및 복제본을 자동으로 생성합니다.

"test"라는 이름의 테스트 항목이 기본 파티션으로 생성됩니다.

메시지는 Kafka의 콘솔 생산자 및 소비자 명령을 사용하여 생성 및 소비되며, 메시지는 즉시 교환됩니다.

생산자 서비스와 소비자 서비스가 모두 실행되어 Kafka 설정이 완료되었습니다.

서비스를 중지하려면 명령을 사용하여 서비스를 하나씩 종료합니다.

마지막으로 생성된 토픽을 정리하는 명령이 제공됩니다.