Installing and Testing Kafka on MacBook M1

Kafka is a powerful open-source event streaming platform that enables efficient data pipeline and analytics.

### Installing and Testing Apache Kafka on MacBook M1
* 💻 Today, I will guide you through the installation of Apache Kafka on a MacBook M1, starting with an introduction to Kafka as an **open-source distributed event streaming platform** used by many companies for **high-performance data pipeline streaming analytics**.
* 🔗 To begin, you can visit the official Apache Kafka site by searching for "**Kafka Apache**" on Google, where you can find resources to download Kafka, including **basic and enterprise versions** like **Confluent Kafka**.
* 📥 For installation, you can download the desired version, such as the **G format**, and once downloaded, unzip it to your preferred folder, navigating to the **binary folder** to access the necessary scripts to run the **Kafka ecosystem**.
* ⚙ After downloading, you will need to configure the `server.properties` file by **uncommenting specific lines** and setting the appropriate **IP address for local use**, as well as setting the **home path for Confluent** in your terminal.
* ☕ Ensure you are using JDK versions **1.8 or 11**, as higher versions like Java 15 are **not supported**; switching to Java 11 may be necessary for **successful operation**.
* 🛠 Once Java is set correctly, you can start the Kafka services using the command `confluent local services start`, which will initiate **Zookeeper, Kafka, and other components of the ecosystem**.
* 📊 After starting the services, you can access the control center at `localhost:9021` to **create a test topic and verify functionality** by producing and consuming messages, confirming that your installation is **successful**.
* 🧹 Finally, remember to **stop all services and clean up any created topics** to maintain a tidy environment, ensuring a **smooth Kafka experience** on your MacBook M1.

Today, I am going to install Kafka on a MacBook M1. Some of you might not know what Kafka is and its uses, so not to worry. I will show a video about Kafka coin details in the future, so let's get started. So, if you Google Kafka Apache, you will go to this link. This is the official Apache Kafka site. So, if you go, it is saying, What is Kafka? Apache Kafka is an open-source distributed event streaming platform. So, this is open-source. You can use it without a license, and it's used by thousands of companies for high-performance data pipeline streaming analytics. You can do so many things. So, this is a very popular messaging system. But in this, it's Apache Kafka license. If you are going, you will have basic Kafka like brokers, keepers, like you have limited assets. But if you go to enterprise Kafka, Confluent, then you will have so many components in it. So, if I can search Confluent Kafka, okay, so if you go to this website, you will see this page. So, you can

download from here, but here's what you need to do: get some license for this product. So, if I want to download, you need to go get started free, and you can go on this link, put your email address. Like, this is my email address, and you can download the manual version, okay? And you just click on agree, it will download for you. Format I need to choose, so here we have so many formats. If you are installing on Docker, you can get the Docker image, Debian, RPM package, entire, and chip. So, I'm going to download the G format, okay? So, it will take time to download it, but I have already downloaded it, so it's so don't worry about that. So, once you download it and unzip up to your choice, what folder you wanted to install. So, if I go to my terminal, let me screen big screen font, okay? So, if you go here, I already downloaded and copy-paste to my folder here, like Configurant 6.0.1. So, once you download, you will see inside. So, if I go to cd confluent

version 6. You will see this folder structure. So, in the binary folder structure, you have all the scripts you need to run the Kafka ecosystem. So, this is, um, if you wanted to run locally. So, this is the, like, um, CLI command. Confluent, you can start Kafka stock. So, you can, you can go with it. And you have Kafka producer, Kafka consumers, cells as well, like, script. So, if you want to produce and consume some Kafka message, so you can play around it. Okay, so now the question is, what were the steps we need to do to perform all those things? So, once you download it, you need to go to, uh, this folder, like, uh, if you do the LS, CD, go to CD, or you can do the edit. So, we have, I'm using the beam command. You can edit it. So, go to BIM, ITC server dot properties, ATC, sorry, Docker server dot properties. So, here, so this is nothing. This is just a configuration file. So, somebody, I already opened it here in VS Code. So, this is just, um, here it looks good to you. So, what you

need to do so once you download it. Some of this is uncommented, and so you need to uncomment it, and you need to put zero zero in this one. So, this is something like localhost kind. It will use your IP address to run the platform. So, and you need to uncomment this one and put here localhost. And also, you need to uncomment this one. So, this is the protocol how zookeeper and kafka communicate together. So, we are using plain text, like no SSL configuration at all. So, we need to go with this one. So, now once you install, what you need to do next step is you need to set the home path of the confluent cocktail. Okay, so if you are using MacBook M1, so sim, you need to go like if you go to parent folder here, put folder. So, you need to do bim zhrc, that shrc, and edit it. So, here you need to put the variable like export, give the variable name confluent home, and you need to give the path where your folder is present. Okay.

and you need to add this bin by library bin folder because in this bin folder you have all the binaries what you need to run while you're performing operations. Okay, and then also you need to include in the path so your confident home path is always this one but also we are including the binaries with the path. Okay, so once you done it and edit it and if you go to just source, source means you need to include in your path in this terminal. So if you run eco dollar confluent home then it will show you this your home part that mean your kafka is installed and you are able to access the path. Now I am using this terminal so this terminal is our apple architecture. Now if I am going to try to run so I have listed all the commands what you need to perform basic operation. So we have two constant one is you need to always use jdk 1.8 and jdk 11 version. Okay, you can't use java 14, java 15. So if I am going to run suppose to

start my Kafka. So, if I am going to run this Confluent architecture in my Apple terminal, I need to run with this command, extra command like architecture, and you need to put this hyphen x8664, then run in this Kafka. So, this is one way. So, suppose if I want to start this in my normal terminal, so it will going to work, but you need to put extra two lines, two commands, okay. So, if I am going to run here, so it will say, if you say trying to restart the Zookeeper and trying to start the Kafka, but I believe it's going to fail because they are saying this is the minimum requirement of version is JDK 1.8, 1.11, but what I currently install, the Java version 15. So, if I run it, check the Java version, what you have. So, here, if you see, I have installed Java 15, okay. So, this is the not right version. They are not supporting currently Confluent Kafka. So, what you need to do, you need to just use this one.

to switch over to use Java 11 version. So once you switch over Java version and check what is your Java version, so you will see Java 11 working. So if I start this command, it will going to work okay, but I don't want to you again to type these two extra, you know, architecture 1186 to start your Kafka. It's better you have to one more option you can go with that and it will, you know, need to put this extra two. What is that option is so Apple is supporting like if you go to terminal utilities. So here you have you are using the terminal. So if you go to edit this terminal and enable this extension like more info. If you click on the more info here, actually, it's already showing to you. So if you check out the checkbox, open using the row setup and enable it and click on the double click, it will work for you. But what I have done it, I enable this one and I clone one more terminal like I gave name is Rosetta terminal.

and I enabled this checkbox here. So for me, it's easy. If I'm doing some Intel architecture things, then I don't need to again do the checkbox and check. So I already installed. So if I'm going to open this terminal and run it, so let me do the font size figure so that you can see it properly. Okay, so now let's start all the steps. So first, I need to check what is the Java version I have currently here. So here, if you see JDK 15 is enabled. So I need to go to enable Java 11 to support it. And if I run it, so now our Java version is set back to 11. Now we are good with it. So next command, what you need to run, you don't need to run this, just start confluent local service start. Okay, confirm local services start. So what does this mean? This is going to start all the Kafka services you need in this ecosystem, convenient platform. So if you run it, so it is trying to install Zookeeper.

like zookeeper, then kafka. So, we have so many components, so don't worry about all these parts, how they are working together. We will go over there in future videos. So, if you see, zookeeper is started, kafka is started. Now, this is schema registry starting, kafka rest, and then kafka rest is up now, trying to connect. So, we have so many components. If you are going to work with confident kafka, but this is licensed version. So, if you are doing for your local development, that is good, but you can't use kafka in production. You need a license at least, okay? So, now it's starting the control center. So, control center is the last component of this kafka architecture. So, once it's done, then you'll be able to access the control center, okay? So, let's wait for a few seconds. It will going to be up, okay? So, it's taking time. Don't worry, it's going to start. So, now our kafka is started. So, how you see it's working or not, you need to go to one more time and just say local.

host and colon 9021. This is the port number where your contact center is running. So if you search it, then now your Kafka is working perfectly fine. So let's create one topic here and test is working or not. Okay, so if you see that we have brokers and this is internal partitions replicas topic. This is the Kafka ecosystem is creating for you, so you don't need to go worry about that. So let's test this. The basic test is working or not, or installation. So I'm going to click on the add topic and give the topic name is just test. Okay, so I'm clicking on the create default partition one. So don't worry about what is the more detail. So I'm just saying clicking on the create the defaults and now our topic is created. So if you don't see any message at all, right? So how we can put the message in this topic? You need to create, you need, you have to like produce something and consume something. That way messaging system works. Okay, so I'm going to here, I'm going to create

um like I'm going to run the producer, so you use this command. So I already told you in the bin folder, you have so many supporting, you know, binaries. So if you want to produce something, so you need to run binary Kafka console producer topic topic name. So if you see the hyphen hyphen topic, this is your topic name, and this is your broker list. So this is my Kafka broker, and it's running on the 9092 port. Okay, 9092. Okay, so if I'm going to run this one and enter here. Okay, so it will going to. So my Kafka producer is now available. Let me add one more terminal, new tab. Okay, basic. And I'm going to run the command which is going to consume that message. Okay, so suppose here I am going to run this command Kafka consumer. Okay, so now if you produce something higher snail, it's going to consume here. Okay, you see high sunny is coming very well instantly. Okay, similar say hi there. Okay, hi there. So if you see here is hi

there automatically coming to you that way you are using some messaging services you already know. So if I go to here in messaging service, so here his message should show you like consume, consume, and produce, but it's taking time. Let me repress it. Okay, so it's not saying here, but that's okay. So now is working fine for you. So your now Kafka setup is done. We are good here, and we can close Ctrl+C to command exit from here. Similar, you can exit from here. Okay, so now your producer and consumer are both closed. So what you need to do now, you can stop all your services. So both services are running. So you can go here and use this command to stop all your services. Now we are stopping for control center, then ksql. So all they are installing, like stopping in one by one. So once it's stopped, you will not be able to run this control center because this prompt is installed. Okay, so let me see it's stopped artillery. So you can, you see, this is not reachable. So stop so

now if you want to clean up everything, whatever your topic created, then you need to run this command to clean up your topic. You created the test topic, so it will clean everything from the hair. So, this way, you can install Kafka and test if it's working or not. Thank you for watching. Take care. Stay safe.

How to install Confluent Kafka in mac m1

Shri Krishna Sankirtanam

Kafka is an open-source distributed event streaming platform used for high-performance data pipeline and analytics.

The enterprise version, Confluent Kafka, offers additional components and requires a license for download.

After downloading and unzipping Kafka, the binary folder contains scripts to run the Kafka ecosystem locally, including commands for starting Kafka and producing/consuming messages.

You need to uncomment specific lines in the configuration to set the IP address and protocol for Zookeeper and Kafka.

Set the home path for Confluent Kafka and include the binaries in the path.

11, as higher versions are not supported, and run the Kafka startup commands with the correct architecture flag.

Switching to Java 11 and Starting Kafka Services

The process involves switching to Java 11 and using the terminal's Rosetta setup for Intel architecture.

After setting Java 11, the command "confluent local services start" is run to initiate all Kafka services, including Zookeeper and the Control Center.

The setup is for local development, with a license required for production use.

Testing Kafka: Creating a Topic and Messaging

Kafka is running on port 9021, and the ecosystem automatically creates brokers, partitions, and replicas.

A test topic named "test" is created with a default partition.

Messages are produced and consumed using Kafka's console producer and consumer commands, with messages being exchanged instantly.

Kafka setup is complete, with both producer and consumer services running.

To stop the services, a command is used to shut them down one by one.

Finally, a command is provided to clean up any created topics.