Introduction to Apache Kafka with a Node.js Example

Introduction

Apache Kafka is an open-source distributed messaging platform widely used for building real-time data pipelines and event-streaming systems. In this article, we'll explore the basics of Kafka, understand what it is used for, and create a mini project in Node.js to illustrate how it works.

What is Apache Kafka?

Apache Kafka was originally developed by LinkedIn and later donated to the Apache Software Foundation. It is designed to handle large volumes of real-time data distributed among multiple consumers.

Key Concepts:

1. Producer: Sends messages to Kafka.
2. Consumer: Reads messages from Kafka.
3. Topic: A category or feed name to which messages are sent and from which they are consumed.
4. Broker: A Kafka server that stores and distributes messages.
5. Partition: Topics are divided into partitions to allow scalability.
6. Offset: A unique identifier for each message within a partition.

Why Use Kafka?

Kafka is widely used for:

• Real-time analytics
• Event-driven architectures
• Log aggregation
• Building scalable microservices

Its ability to handle large streams of data with high throughput and low latency makes it ideal for modern applications.

Example: Building a Kafka Producer and Consumer with Node.js

We'll create a producer to send messages to a Kafka topic and a consumer to read them. We'll use Docker to spin up a Kafka environment quickly.

Prerequisites:

1. Docker installed
2. Node.js and npm installed
3. KafkaJS library

npm install kafkajs

Step 1: Set Up Kafka with Docker

Create a docker-compose.yml file in your project directory:

version: '3.8'
services:
  zookeeper:
    image: confluentinc/cp-zookeeper:7.5.0
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000
    ports:
      - '2181:2181'

  kafka:
    image: confluentinc/cp-kafka:7.5.0
    depends_on:
      - zookeeper
    ports:
      - '9092:9092'
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9093,PLAINTEXT_HOST://localhost:9092
      KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9093,PLAINTEXT_HOST://0.0.0.0:9092
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1

Run the containers:

docker-compose up -d

This starts Zookeeper and a Kafka broker. The broker will be accessible at localhost:9092.

Step 2: Create the Project

Initialize a Node.js project and install dependencies:

mkdir kafka-nodejs-docker-demo
cd kafka-nodejs-docker-demo
npm init -y
npm install kafkajs

Step 3: Write the Producer

Create a file producer.js:

const { Kafka } = require('kafkajs')

const kafka = new Kafka({
  clientId: 'demo-producer',
  brokers: ['localhost:9092'],
})

const topic = 'test-topic'

async function ensureTopic() {
  const admin = kafka.admin()
  await admin.connect()
  await admin.createTopics({ topics: [{ topic, numPartitions: 1, replicationFactor: 1 }] })
  await admin.disconnect()
}

async function run() {
  const producer = kafka.producer()
  await ensureTopic()
  await producer.connect()
  for (let i = 1; i <= 5; i++) {
    await producer.send({
      topic,
      messages: [{ key: `key-${i}`, value: `Message ${i} at ${new Date().toISOString()}` }],
    })
    console.log(`Sent message ${i}`)
  }
  await producer.disconnect()
}

run().catch((err) => {
  console.error('Producer error', err)
  process.exit(1)
})

Run the producer:

node producer.js

Step 4: Write the Consumer

Create a file consumer.js:

const { Kafka } = require('kafkajs')

const kafka = new Kafka({
  clientId: 'demo-consumer',
  brokers: ['localhost:9092'],
})

const topic = 'test-topic'

async function run() {
  const consumer = kafka.consumer({ groupId: 'demo-group' })
  await consumer.connect()
  await consumer.subscribe({ topic, fromBeginning: true })
  await consumer.run({
    eachMessage: async ({ topic, partition, message }) => {
      const key = message.key?.toString()
      const value = message.value?.toString()
      console.log(`Partition ${partition} | ${key} => ${value} | offset ${message.offset}`)
    },
  })
}

run().catch((err) => {
  console.error('Consumer error', err)
  process.exit(1)
})

Run the consumer:

node consumer.js

Step 5: Test It Out

1. Start the consumer first to begin listening for messages:

node consumer.js

2. In another terminal, run the producer to send messages to test-topic:

node producer.js

3. Watch the consumer log the messages in real-time.

Step 6: Stop the Kafka Containers

When you're done, stop Kafka and Zookeeper:

docker-compose down

Conclusion

Using Docker, we simplified the process of running Kafka locally without manual installation. In this project, we demonstrated how to use Kafka with Node.js to produce and consume messages. You can now expand this setup for more complex use cases, such as streaming data pipelines or event-driven architectures.