Apache Kafka Tutorial: Build Your First Event-Driven Application

Learn how to easily set up Apache Kafka and build your very own real-time event-driven application from scratch.

kafka

Introduction to Event-Driven Architecture

Mac/Linux/Windows

What is Apache Kafka?

Apache Kafka is a powerful platform used for handling live data streams. Think of it as a massive, super-fast digital post office. In traditional systems, applications ask databases for information when they need it. In an event-driven system using Kafka, applications instantly send out a message (an "event") the moment something happens like a user clicking a button, a sensor detecting a temperature change, or a payment being made.

At its core, Kafka works by organizing these events into categories called "Topics," which are just like folders on your computer. Applications that create data are called "Producers," and they send their data to these folders. On the other side, applications that need to read the data are called "Consumers." The consumers constantly watch these folders and read the new messages the exact second they arrive. This allows different parts of a software system to communicate with each other instantly without being directly connected.

Kafka is incredibly popular because it is fast, reliable, and scalable. It can handle millions of events per second without slowing down or losing any data. Even if a computer in the network crashes, Kafka keeps your data safe because it saves copies across multiple machines. Large companies use it every day for things like live GPS tracking, fraud detection, and instant messaging.

Prerequisites

  • Java 17 or higher: Kafka requires Java to run. Ensure Java is installed and configured on your computer.
  • Terminal Access: You need basic knowledge of how to use a command-line interface (Terminal on Mac/Linux or Command Prompt/PowerShell on Windows).
  • Operating System: Linux, macOS, or Windows Subsystem for Linux (WSL) is highly recommended for running Kafka commands smoothly.
  • Internet Connection: You will need an active connection to download the Kafka files.

Step-by-Step Installation

Below are the exact commands you need to install and run Kafka using its modern, built-in KRaft mode (which does not require the older ZooKeeper tool).
1

Get Kafka

First, download the latest Kafka release and extract the files on your machine. Open your terminal and run:
BASH 
$ tar -xzf kafka_2.13-4.3.1.tgz
$ cd kafka_2.13-4.3.1
2

Start the Kafka Environment

Now, we need to format the storage and start the Kafka server.
First, generate a unique ID for your cluster:
BASH 
$ KAFKA_CLUSTER_ID="$(bin/kafka-storage.sh random-uuid)"
Next, format the log directories using the ID you just created:
BASH 
$ bin/kafka-storage.sh format --standalone -t $KAFKA_CLUSTER_ID -c config/server.properties
Finally, start the Kafka server:
BASH 
$ bin/kafka-server-start.sh config/server.properties
(Keep this terminal window open. Your Kafka server is now running!)
3

Create a Topic to Store Your Events

Before you can send messages, you need to create a Topic (a folder) to store them. Open a new terminal window and run:
BASH 
$ bin/kafka-topics.sh --create --topic quickstart-events --bootstrap-server localhost:9092
4

Write Events into the Topic (Producer)

Now, let's send some messages into the topic you just created. We will use the built-in producer tool. In your terminal, run:
BASH 
$ bin/kafka-console-producer.sh --topic quickstart-events --bootstrap-server localhost:9092
When the prompt opens, type a few sentences. Press Enter after each line to send the event to Kafka:
Plaintext 
>This is my first event
>This is my second event
5

Read the Events (Consumer)

Let's read the data you just pushed to Kafka. Open a third new terminal window and start the consumer tool:
BASH 
$ bin/kafka-console-consumer.sh --topic quickstart-events --from-beginning --bootstrap-server localhost:9092
You will instantly see the messages you typed earlier:
Plaintext 
This is my first event
This is my second event
(Tip: Try typing more messages in your Producer terminal and watch them appear in your Consumer terminal instantly!)
6

Stop the Environment

When you are done experimenting, you can safely shut everything down.
  1. Press Ctrl-C in the Producer and Consumer terminals to stop them.
  2. Press Ctrl-C in the Kafka Server terminal to stop the server.
  3. (Optional) If you want to delete the test data you created, run this command:
BASH 
$ rm -rf /tmp/kafka-logs /tmp/kraft-combined-logs

Final Thoughts

You have successfully set up your local Kafka environment and witnessed real-time event streaming in action.

  • Downloaded and extracted the Apache Kafka core files to your local machine.
  • Started a Kafka server using the modern, ZooKeeper-free KRaft mode.
  • Created a Kafka Topic to securely organize and store your data streams.
  • Ran a Producer client to write and send live messages into your topic.
  • Ran a Consumer client to instantly read and display those messages as they arrived.

With these fundamentals mastered, you are now ready to build more complex and highly scalable event-driven applications.

Explore Dedicated Servers

CTCservers Recommended Tutorials

Install-ROCm

Web, Network

Step-by-Step Guide: Install AMD ROCm on Ubuntu with RX 6600 GPU

Learn how to quickly and easily set up AMD ROCm on Ubuntu for your RX 6600 GPU, enabling powerful machine learning, AI workloads, and GPU-accelerated computing right on your system.
Install LAMP

Web, Network, Linux, Mysql, Ubuntu

LAMP Setup Guide 2026: Ubuntu & Debian | CTCservers

Install a secure LAMP stack on Debian or Ubuntu. Follow our step-by-step guide to configure Linux, Apache, MySQL, and PHP for your web server.
Deploy Phi-3

Web, Network, Ubuntu

Deploy Phi-3 with Ollama on Ubuntu GPU | CTCservers

Learn how to easily deploy the Phi-3 LLM on an Ubuntu 24.04 GPU server using Ollama and WebUI. Follow our step-by-step tutorial for seamless AI hosting.

Discover CTCservers Dedicated Server Locations

CTCservers servers are available around the world, providing diverse options for hosting websites. Each region offers unique advantages, making it easier to choose a location that best suits your specific hosting needs.