Setting Up a Local Azure Event Hub with Docker

When developing and testing applications that rely on Azure Event Hubs, it’s incredibly convenient to have a local setup. While Azure Event Hubs itself cannot run locally, we can emulate it using a combination of tools like Azurite for Azure Storage and Kafka for event streaming. This setup allows us to test our event-driven applications without incurring cloud costs or facing connectivity issues.

Thank me by sharing on Twitter 🙏

In this post, I’ll walk you through how I created a local Azure Event Hub alternative using Docker. Along the way, I’ll share code snippets, tips, and the reasoning behind each step to ensure the process is clear and straightforward.

Getting Started: The Tools You’ll Need

To emulate Azure Event Hubs locally, we’ll use the following tools:

1. Azurite: A lightweight Azure Storage emulator.
2. Kafka: A distributed event streaming platform that mimics Event Hub functionality.
3. Docker Compose: To orchestrate our services.

With these tools, you’ll have a complete setup that mimics an Azure Event Hub environment, enabling you to produce and consume messages just like in a real cloud scenario.

Step 1: Install Docker and Set Up a Workspace

Before diving into the setup, ensure Docker is installed on your system. If it isn’t, you can download it from the official Docker website. Once Docker is installed and running, create a new directory to house your project files. I named mine local-event-hub.

The Nvidia Way: Jensen Huang and the Making of a Tech Giant

(40)

$26.16 (as of December 21, 2024 19:39 GMT +00:00 - More infoProduct prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on [relevant Amazon Site(s), as applicable] at the time of purchase will apply to the purchase of this product.)

HP 67 Black/Tri-color Ink Cartridges (2 Pack) | Works with HP DeskJet 1255, 2700, 4100 Series, HP ENVY 6000, 6400 Series | Eligible for Instant Ink | 3YP29AN

(54044)

$36.89 (as of December 21, 2024 08:38 GMT +00:00 - More infoProduct prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on [relevant Amazon Site(s), as applicable] at the time of purchase will apply to the purchase of this product.)

Apple USB-C Woven Charge Cable (1 m) ​​​​​​​

(3130)

$15.00 (as of December 21, 2024 08:38 GMT +00:00 - More infoProduct prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on [relevant Amazon Site(s), as applicable] at the time of purchase will apply to the purchase of this product.)

Step 2: Create a Docker Compose File

The backbone of our setup is a docker-compose.yml file that defines and orchestrates the Azurite, Kafka, and Zookeeper containers. This file simplifies the process by spinning up all the required services with a single command.

Here’s the docker-compose.yml file I used:

services:
  azurite:
    image: mcr.microsoft.com/azure-storage/azurite
    container_name: azurite
    ports:
      - "10000:10000" # Blob service
      - "10001:10001" # Queue service
      - "10002:10002" # Table service
    command: "azurite-blob --blobHost 0.0.0.0 --blobPort 10000 && \
      azurite-queue --queueHost 0.0.0.0 --queuePort 10001 && \
      azurite-table --tableHost 0.0.0.0 --tablePort 10002"
    volumes:
      - azurite_data:/data
    networks:
      - eventhub-net

  zookeeper:
    image: confluentinc/cp-zookeeper:latest
    container_name: zookeeper
    ports:
      - "2181:2181"
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000
    networks:
      - eventhub-net

  kafka:
    image: confluentinc/cp-kafka:latest
    container_name: kafka
    depends_on:
      - zookeeper
    ports:
      - "9092:9092"
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: "zookeeper:2181"
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
    networks:
      - eventhub-net

volumes:
  azurite_data:

networks:
  eventhub-net:
    driver: bridge
services:
  azurite:
    image: mcr.microsoft.com/azure-storage/azurite
    container_name: azurite
    ports:
      - "10000:10000" # Blob service
      - "10001:10001" # Queue service
      - "10002:10002" # Table service
    volumes:
      - azurite_data:/data
    networks:
      - eventhub-net

  zookeeper:
    image: confluentinc/cp-zookeeper:latest
    container_name: zookeeper
    ports:
      - "2181:2181"
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000
    networks:
      - eventhub-net

  kafka:
    image: confluentinc/cp-kafka:latest
    container_name: kafka
    depends_on:
      - zookeeper
    ports:
      - "9092:9092"
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: "zookeeper:2181"
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
    networks:
      - eventhub-net

volumes:
  azurite_data:

networks:
  eventhub-net:
    driver: bridge

Step 3: Spin Up the Containers

With the docker-compose.yml file in place, navigate to the directory where it’s stored and run the following command:

docker-compose up

This command starts Azurite, Kafka, and Zookeeper containers in the background. You can verify that everything is running correctly by listing the active containers:

docker ps

If all goes well, you’ll see three containers running: one for Azurite, one for Kafka, and one for Zookeeper.

Step 4: Create a Kafka Topic

In Azure Event Hubs, a “namespace” typically contains multiple event hubs. In Kafka, the equivalent concept is a “topic.” Each topic represents a channel for sending and receiving messages.

To create a topic, access the Kafka container:

docker exec -it kafka bash

Once inside, use the following command to create a topic:

docker exec -it kafka kafka-topics --create --topic chat-messages --bootstrap-server localhost:9092 --partitions 1 --replication-factor 1

This command sets up a topic called chat-messages with one partition and a replication factor of one.

Step 5: Producing and Consuming Chat Messages

Now that the infrastructure is ready, it’s time to send and receive messages. In my setup, chat messages include a userId, timestamp, and content. To handle these, I chose JSON as the message format for simplicity and compatibility.

Producing Messages

Use Python to produce chat messages. Here’s the script I wrote:

from kafka import KafkaProducer
import json
from datetime import datetime

producer = KafkaProducer(
    bootstrap_servers='localhost:9092',
    value_serializer=lambda v: json.dumps(v).encode('utf-8')
)

# Example chat messages
messages = [
    {"userId": "user123", "timestamp": datetime.utcnow().isoformat(), "content": "Hello, this is a chat message!"},
    {"userId": "user456", "timestamp": datetime.utcnow().isoformat(), "content": "How are you?"},
]

# Send messages to Kafka
for message in messages:
    producer.send('chat-messages', value=message)

print("Messages sent successfully!")
producer.flush()
producer.close()

This script connects to the Kafka broker running on localhost:9092, serializes messages as JSON, and sends them to the chat-messages topic.

Consuming Messages

To consume messages from Kafka, I used the following Python script:

from kafka import KafkaConsumer
import json

consumer = KafkaConsumer(
    'chat-messages',
    bootstrap_servers='localhost:9092',
    auto_offset_reset='earliest',
    value_deserializer=lambda v: json.loads(v.decode('utf-8'))
)

print("Listening for messages...")
for message in consumer:
    print(f"Received message: {message.value}")

This script listens to the chat-messages topic and deserializes each message back into a Python dictionary.

Step 6: Testing the Workflow

With both the producer and consumer scripts ready, you can now test the entire workflow:

1. Run the consumer script first to start listening for messages.
2. Run the producer script to send a batch of chat messages.
3. Observe the messages being printed by the consumer in real time.

This process ensures that your local setup is working as expected and that messages are successfully flowing through your Kafka-based Event Hub alternative.

Step 7: Cleaning Up

When you’re done testing, you can stop and remove the Docker containers by running:

docker-compose down

This command stops and deletes all containers defined in the docker-compose.yml file. If you want to remove the associated volumes, add the --volumes flag.

Wrapping Up

Building a local Azure Event Hub alternative using Docker, Kafka, and Azurite is a practical solution for development and testing. It allows you to simulate real-world scenarios without relying on cloud infrastructure. By following these steps, you’ll have a fully functional local setup that can handle event-driven applications, from producing and consuming chat messages to testing advanced features like partitioning and offsets.

I hope this guide helps you get started with your own local environment. Whether you’re working on a new project or testing an existing application, having this setup in your toolbox is invaluable.

Related Posts:

Managing Files in Azure Blob Storage with Python

Choosing the Right Azure Key-Value Store: A…

Uploading Files to Azure Blob Storage Using SAS…