This post walks you through the process of Streaming Data from Kafka to Postgres with Kafka Connect AVRO, Schema Registry and Python.

What you'll need

Stack

  • Python 3
  • Pipenv
  • Flake8
  • Docker Compose
  • Postgres
  • Kafka
  • Kafka Connect
  • AVRO
  • Confluent Schema Registry

Project structure

├── Pipfile
├── Pipfile.lock
├── avro_producer.py
├── docker-compose.yml
├── setup.cfg
└── sink-postgres.json

Project dependencies

Kafka AVRO Producer

Kafka Connect Sink Postgres Config

Docker Compose

Let's Start

Start Confluent Platform

confluent start

You would see this

connect is [UP]  
kafka-rest is [UP]  
schema-registry is [UP]  
kafka is [UP]  
zookeeper is [UP]

ZooKeeper, Kafka, Schema Registry and Kafka Connect should be start listening connections on port 2181, 9092, 8081, 8083 respectively.

Start PostgreSQL Database

docker-compose up

PostgreSQL Database Server should be start listening connections on port 5432

Install project's dependencies

pipenv install  
pipenv shell

Sending data to Kafka topic with AVRO Producer

python avro_producer.py

Load Kafka Connector

confluent load sink-1 -d sink-postgres.json

Postgres should have a table page_1 with data streaming from the consumer.

Source code

git@github.com:hellokoding/kafka-connect-sink-postgres-with-avro-schemaregistry-python.git

https://github.com/hellokoding/kafka-connect-sink-postgres-with-avro-schemaregistry-python

Kafka