well I guess this is what I need https://cwiki.apache.org/confluence/display/KAFKA/Transactional+Messaging+in+Kafka
anyone knows how far this effort has gone? thanks Yang On Mon, Jan 26, 2015 at 12:06 PM, Yang <teddyyyy...@gmail.com> wrote: > we are pulling from a DB, maybe 100k rows at a time. for each row we send > it to kafka. > > now the problem is , if any errors happen during the db pulling (such as > an earthquake), we stop. > next time we wake up. we can't write the same record again into kafka, > otherwise there might be over counting. > > > does kafka support a "BATCH mode" with commit ? so basically at the start > of my session, I declare "start transaction", and after all records have > been pulled and sent to kafka, I declare "commit" . > > Thanks > Yang >
