High-performance real-time data processing: managing data using Debezium, Postgres, Kafka, and Redis

Abstract

This research focuses on monitoring and transferring logs of operations performed on a relational database, specifically PostgreSQL, in real-time using an event-driven approach. The logs generated from database operations are transferred using Apache Kafka, an open-source message queuing system, and Debezium running on Kafka, to Redis, a non-relational (No-SQL) key-value database. Time-consuming query operations and read operations are performed on Redis, which operates on memory (in-memory), instead of on the primary database, PostgreSQL. This approach has significantly improved query execution performance, data processing time, and backend service performance. The study showcases the practical application of an event-driven approach using Debezium, Kafka, Redis, and relational databases for real-time data processing and querying.