Hi James, Thanks for the kind words. I will definitely work on the persitence of binlog position, with a couple of persistence options.
The trickier part is figuring out the way to correctly figure out a key for the topic. Not all events indicate which database/table/entity they operate on. Getting the correct recipe for this will take a bit more time. Cheers, - pyr On 03/16/2015 06:29 PM, James Cheng wrote: > Super cool, and super simple. > > I like how it is pretty much a pure translation of the binlog into Kafka, > with no interpretation of the events. That means people can layer whatever > they want on top of it. They would have to understand what the mysql binary > events mean, but they would just have to interact with kafka, and not with > mysql. > > Will you be working on the reconnection strategy, so that you resume from the > last binlog position that you left off at? Do you anticipate that there will > be duplicate events in the output stream, or are you going to go for > exactly-once? > > -James > > On Mar 16, 2015, at 7:18 AM, Pierre-Yves Ritschard <p...@spootnik.org> wrote: > >> Hi kafka, >> >> I just wanted to mention I published a very simple project which can >> connect as MySQL replication client and stream replication events to >> kafka: https://github.com/pyr/sqlstream >> >> When you don't have control over an application, it can provide a simple >> way of consolidating SQL data in kafka. >> >> This is an early release and there are a few caveats (mentionned in the >> README), mostly the poor partitioning which I'm going to evolve quickly >> and the reconnection strategy which doesn't try to keep track of binlog >> position, other than that, it should work as advertised. >> >> Cheers, >> - pyr >
