AFAIK , linkedin uses databus to do the same. Aesop is built on top of databus , extending its beautiful capabilities to mysql n hbase On Mar 18, 2015 7:37 AM, "Xiao" <lixiao1...@gmail.com> wrote:
> Hi, all, > > Do you know how Linkedin team publishes changed rows in Oracle to Kafka? I > believe they already knew the whole problem very well. > > Using triggers? or directly parsing the log? or using any Oracle > GoldenGate interfaces? > > Any lesson or any standard message format? Could the Linkedin people share > it with us? I believe it can help us a lot. > > Thanks, > > Xiao Li > > > On Mar 17, 2015, at 12:26 PM, James Cheng <jch...@tivo.com> wrote: > > > This is a great set of projects! > > > > We should put this list of projects on a site somewhere so people can > more easily see and refer to it. These aren't Kafka-specific, but most seem > to be "MySQL CDC." Does anyone have a place where they can host a page? > Preferably a wiki, so we can keep it up to date easily. > > > > -James > > > > On Mar 17, 2015, at 8:21 AM, Hisham Mardam-Bey < > hisham.mardam...@gmail.com> wrote: > > > >> Pretty much a hijack / plug as well (= > >> > >> https://github.com/mardambey/mypipe > >> > >> "MySQL binary log consumer with the ability to act on changed rows and > >> publish changes to different systems with emphasis on Apache Kafka." > >> > >> Mypipe currently encodes events using Avro before pushing them into > Kafka > >> and is Avro schema repository aware. The project is young; and patches > for > >> improvements are appreciated (= > >> > >> On Mon, Mar 16, 2015 at 10:35 PM, Arya Ketan <ketan.a...@gmail.com> > wrote: > >> > >>> Great work. > >>> Sorry for kinda hijacking this thread, but I though that we had built > >>> some-thing on mysql bin log event propagator and wanted to share it . > >>> You guys can also look into Aesop ( https://github.com/Flipkart/aesop > ). > >>> Its > >>> a change propagation frame-work. It has relays which listens to bin > logs of > >>> Mysql, keeps track of SCNs and has consumers which can then > (transform/map > >>> or interpret as is) the bin log-event to a destination. Consumers also > keep > >>> track of SCNs and a slow consumer can go back to a previous SCN if it > wants > >>> to re-listen to events ( similar to kafka's consumer view ). > >>> > >>> All the producers/consumers are extensible and you can write your own > >>> custom consumer and feed off the data to it. > >>> > >>> Common use-cases: > >>> a) Archive mysql based data into say hbase > >>> b) Move mysql based data to say a search store for serving reads. > >>> > >>> It has a decent ( not an awesome :) ) console too which gives a nice > human > >>> readable view of where the producers and consumers are. > >>> > >>> Current supported producers are mysql bin logs, hbase wall-edits. > >>> > >>> > >>> Further insights/reviews/feature reqs/pull reqs/advices are all > welcome. > >>> > >>> -- > >>> Arya > >>> > >>> Arya > >>> > >>> On Tue, Mar 17, 2015 at 1:48 AM, Gwen Shapira <gshap...@cloudera.com> > >>> wrote: > >>> > >>>> Really really nice! > >>>> > >>>> Thank you. > >>>> > >>>> On Mon, Mar 16, 2015 at 7:18 AM, Pierre-Yves Ritschard < > p...@spootnik.org > >>>> > >>>> wrote: > >>>>> Hi kafka, > >>>>> > >>>>> I just wanted to mention I published a very simple project which can > >>>>> connect as MySQL replication client and stream replication events to > >>>>> kafka: https://github.com/pyr/sqlstream > >>>>> > >>>>> When you don't have control over an application, it can provide a > >>> simple > >>>>> way of consolidating SQL data in kafka. > >>>>> > >>>>> This is an early release and there are a few caveats (mentionned in > the > >>>>> README), mostly the poor partitioning which I'm going to evolve > quickly > >>>>> and the reconnection strategy which doesn't try to keep track of > binlog > >>>>> position, other than that, it should work as advertised. > >>>>> > >>>>> Cheers, > >>>>> - pyr > >>>> > >>> > >> > >> > >> > >> -- > >> Hisham Mardam-Bey > >> http://hisham.cc/ > > > >
