Hi Kafka users, I'd like to announce a new open source project, called "Bottled Water", for getting data from PostgreSQL into Kafka: http://blog.confluent.io/2015/04/23/bottled-water-real-time-integration-of-postgresql-and-kafka/ https://github.com/confluentinc/bottledwater-pg/
Bottled Water combines a consistent snapshot with logical replication, so you can get a full dump of your database in Kafka, plus low-latency messages whenever a row is inserted, updated or deleted in the database. This allows you to write Kafka consumers which maintain a copy of a database in a downstream system, such as a full-text search index, caches, data warehouse, HDFS, etc. Bottled Water uses the new logical decoding feature in Postgres 9.4, and the log compaction feature in Kafka. Each table in Postgres becomes a topic in Kafka, and the table schema is automatically converted into an Avro schema for export. It's an alpha release that is not yet fit for production use, but it's ready for experimentation. Feedback and contributions welcome! Martin
