Dear Apache Kafka Community, I am an independent developer building *Debo Data Studio*, a unified visual ETL platform recently discussed on the Apache Incubator mailing list (thread: https://lists.apache.org/thread/ctw3g5v4mf8o5dko1w1xvmwgsq97f6kt <https://mail.google.com/mail/u/0/>).
Following the community-building advice I received from the Incubator members, I am reaching out to the Kafka community to discuss a specific architectural direction I am exploring regarding event-driven data pipelines. *The Proposal: PostgreSQL LISTEN/NOTIFY for Kafka-like Functionality* While Apache Kafka is the industry standard for high-throughput event streaming, I have observed that for many ETL and small-to-medium scale synchronization tasks, the operational overhead of managing a Kafka cluster can be a significant barrier. I am implementing a feature in Debo Data Studio that uses *PostgreSQL SQL scripts and LISTEN/NOTIFY functionality* to mimic core Kafka-like behaviors for smaller environments: - *Real-time Event Triggers:* Using database-level notifications to trigger visual ETL jobs immediately upon data changes. - *Lightweight Pub/Sub:* Leveraging the internal PostgreSQL bus to handle message distribution without spinning up external brokers. - *Reduced Complexity:* Allowing users to achieve basic event-driven workflows using their existing database infrastructure. *Seeking Feedback & Building a Community* As experts in the event-streaming space, I would highly value your perspective on a few questions: 1. In your experience, at what scale or under what specific conditions does a PostgreSQL-based notification system typically hit its limits, making a transition to Kafka mandatory? 2. Would Kafka users see value in a visual ETL tool that allows them to "start small" with PostgreSQL NOTIFY pipelines and seamlessly migrate or bridge those streams into Kafka as their data needs grow? 3. I am looking to build an open community around this "database-first" approach to streaming. If you are interested in collaborating, contributing, or offering architectural guidance, I would love to connect. My goal is not to replace the power of Kafka, but to provide an accessible, visual entry point for event-driven data movement. I look forward to your technical insights and feedback. Regards, Surafel
