WWCode Dublin - Kafka Workshop

This guide will provide you a quick overview on how to get started with Kafka via CLI and with Python client library for Kafka

What is Kafka ?

Kafka is a highly distributed and scalable system that offers high throughput and allows you publish messages to a particular location called Topic, from the other side other Services and Systems will be able to consume those published messages by subscribing to the topic(s) that are interested to consume from and do its processing as they would like to.

Kafka as a Unified Log

Kafka, was created by Jay Krepps when Working on LinkedIn in early January 2011 and progressively overtime it's been defined as:

• Publish/subscribe messaging system

• A distributed commit log

• Distributing streaming platform

Kafka Use Cases

There is many use cases for Kafka, but as stated on its documentation let's stick to the 2 main use cases:

• Building real-time streaming data pipelines that reliably get data between systems or applications

• Building real-time streaming applications that transform or react to the streams of data

What Kafka is not for

• Synchronous communication

• Messages delivery to a particular location, i.e server X

• Long term storage

On the following sections we will go through the basics of Kafka and its components by using:

• CLI ( Command Line Interface ) embedded with the Kafka binaries

• Python client for the Apache Kafka

But first, Let's get the environment setup if you haven't done it yet :-)