Administering Kafka Topics & Partitions

Timeflow

In this guide, we will aim to demonstrate the following:

  • Build familiarity with the Kafka command line tools
  • List, create and interact with Kafka topics
  • View metadata about your Kafka topics
  • Publish messages to your a Kafka topic
  • Build an understanding of Kafka partitions

Command Line Tools

Kafka comes with a number of scripts which can be used to interact with and manage the broker.  These are stored in the $KAFKA_HOME/bin directory within your Kafka download.  

    cd $KAFKA_HOME
    ls -la ./bin/*

Note that for each of the scripts we execute, we have to specify the endpoint to our Kafka server via the –bootstrap-server command line argument.  

    ./bin/kafka-topics.sh --bootstrap-server localhost:9092 --list

Creating And Listing Topics

We will begin by listing the topics on the Kafka broker using the kafka-topics.sh script.  If this is a brand new broker, you should find that no topics have been created at this time.  

    ./bin/kafka-topics.sh --bootstrap-server localhost:9092 --list

Out of the box, Kafka will create topics dynamically as publishers send messages to the topic.  However, here we will create one explicitly to demonstrate the process. This is done with the –create flag followed by a topic name.  

    ./bin/kafka-topics.sh --create orders --bootstrap-server localhost:9092

If we then list topics again we will see that the orders topic has been created.

    ./bin/kafka-topics.sh --bootstrap-server localhost:9092 --list
    orders

Producing And Consuming Messages

To test the topic, we can use the console producer script to send a message to the orders topic:

    ./bin/kafka-console-producer.sh --bootstrap-server localhost:9092 --topic orders
    >hello!

And receive the message with the kafka-console-consumer.sh script.

    ./bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic orders --from-beginning
    hello!

Now we have a message flowing through the broker, we can interrogate the status of the broker to understand what is happening more.  In a new terminal window, executing the *kafka-consumer-groups.sh *script will show us details about the connected console consumer, which topic they are up to, and where abouts in the topic they have ran.  We can also see that they are connected to a partition 0.  

    ./bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --all-groups --describe
    
   
    GROUP                  TOPIC           PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID                                                            HOST            CLIENT-ID
    console-consumer-72405 orders          0          -               1               -               consumer-console-consumer-72405-1-d936ff44-7f68-46e7-bb88-7cd54c0cfea3 /192.168.68.113 consumer-console-consumer-72405-1

Adding A Partition

Imagine we need to add more throughput to our broker to process more messages faster on our orders topic.  One way to do this is to partition the topic.  This is done with the –alter –partitions command line flags to –kafka-topics.  

    ./bin/kafka-topics.sh --bootstrap-server localhost:9092 --topic orders --alter --partitions 3

When this returns, the orders topic has now been changed into partitions.

In new shell windows, open more consumers.  

    ./bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic orders --from-beginning

And we will observe that each consumer has connected to each partition:

    media:kafka_2.13-2.7.0 benjaminwootton$ ./bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --all-groups --describe
    
    GROUP                  TOPIC           PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID                                                            HOST            CLIENT-ID
    console-consumer-23478 orders          0          -               1               -               consumer-console-consumer-23478-1-f0513dd6-2a42-45ff-a9a4-9cbb9eee5386 /192.168.68.113 consumer-console-consumer-23478-1
    console-consumer-23478 orders          1          -               0               -               consumer-console-consumer-23478-1-f0513dd6-2a42-45ff-a9a4-9cbb9eee5386 /192.168.68.113 consumer-console-consumer-23478-1
    console-consumer-23478 orders          2          -               0               -               consumer-console-consumer-23478-1-f0513dd6-2a42-45ff-a9a4-9cbb9eee5386 /192.168.68.113 consumer-console-consumer-23478-1
    
    GROUP                  TOPIC           PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID                                                            HOST            CLIENT-ID
    console-consumer-23756 orders          0          -               1               -               consumer-console-consumer-23756-1-6883fafd-a0a6-4467-8478-5cea9362a0c3 /192.168.68.113 consumer-console-consumer-23756-1
    console-consumer-23756 orders          1          -               0               -               consumer-console-consumer-23756-1-6883fafd-a0a6-4467-8478-5cea9362a0c3 /192.168.68.113 consumer-console-consumer-23756-1
    console-consumer-23756 orders          2          -               0               -               consumer-console-consumer-23756-1-6883fafd-a0a6-4467-8478-5cea9362a0c3 /192.168.68.113 consumer-console-consumer-23756-1
    
    GROUP                  TOPIC           PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID                                                            HOST            CLIENT-ID
    console-consumer-72405 orders          0          -               1               -               consumer-console-consumer-72405-1-d936ff44-7f68-46e7-bb88-7cd54c0cfea3 /192.168.68.113 consumer-console-consumer-72405-1
    console-consumer-72405 orders          1          -               0               -               consumer-console-consumer-72405-1-d936ff44-7f68-46e7-bb88-7cd54c0cfea3 /192.168.68.113 consumer-console-consumer-72405-1
    console-consumer-72405 orders          2          -               0               -               consumer-console-consumer-72405-1-d936ff44-7f68-46e7-bb88-7cd54c0cfea3 /192.168.68.113 consumer-console-consumer-72405-1
 


About Timeflow

A Low Code Platform For Working With Real-Time Data

Timeflow helps businesses improve their customer experience and business performance using streaming real-time data and analytics.