This post walks you through the process of Streaming Data from Kafka to Postgres with Kafka Connect AVRO, Schema Registry and Python.

What you'll need


  • Python 3
  • Pipenv
  • Flake8
  • Docker Compose
  • Postgres
  • Kafka
  • Kafka Connect
  • AVRO
  • Confluent Schema Registry

Project structure

├── Pipfile
├── Pipfile.lock
├── docker-compose.yml
├── setup.cfg
└── sink-postgres.json

Project dependencies

Kafka AVRO Producer

Kafka Connect Sink Postgres Config

Docker Compose

Let's Start

Start Confluent Platform

confluent start

You would see this

connect is [UP]  
kafka-rest is [UP]  
schema-registry is [UP]  
kafka is [UP]  
zookeeper is [UP]

ZooKeeper, Kafka, Schema Registry and Kafka Connect should be start listening connections on port 2181, 9092, 8081, 8083 respectively.

Start PostgreSQL Database

docker-compose up

PostgreSQL Database Server should be start listening connections on port 5432

Install project's dependencies

pipenv install  
pipenv shell

Sending data to Kafka topic with AVRO Producer


Load Kafka Connector

confluent load sink-1 -d sink-postgres.json

Postgres should have a table page_1 with data streaming from the consumer.

Source code