Re: open source projects based on kafka
You could try implementing your own client. https://cwiki.apache.org/confluence/display/KAFKA/Clients On Sun, Sep 13, 2015 at 7:46 PM, Stevo Slavić wrote: > Have a look at https://github.com/allegro/hermes > > On Mon, Sep 14, 2015, 01:28 David Luu wrote: > > > The toy project idea is good. Another option I think could be to look at > > the various Kafka client langague bindings and/or utilities (like > > kafkacat). And from there, another option is to build a client language > > binding for a language that's kind of lacking Kafka support, some have > > better support than others. > > > > On Sun, Sep 13, 2015 at 11:46 AM, Hisham Mardam-Bey > > > wrote: > > > > > Hi Li, > > > > > > You can take a look at mypipe. > > > > > > "MySQL binary log consumer with the ability to act on changed rows and > > > publish changes to different systems with emphasis on Apache Kafka." > > > > > > https://github.com/mardambey/mypipe > > > > > > It uses Kafka as well as some related concepts and technologies like > > Avro, > > > a schema repository for data, and Akka. > > > > > > Hope this helps (= Happy hacking! > > > > > > hmb. > > > > > > > > > On Sun, Sep 13, 2015 at 4:45 AM, Manasvi Gupta > > > wrote: > > > > > > > Here's a toy project - analyzing twitter stream. > > > > > > > > 1) Create dev. account on twitter > > > > 2) Using your dev credentials, connect to twitter stream api to > > retrieve > > > > stream of tweets > > > > 3) Store tweets in Kafka (using Kafka producer) > > > > 4) Retrieve tweets (using Kafka consumer) > > > > 5) For each tweet (or group of tweets), compute some analysis either > > > using > > > > custom java OR use storm/samza/spark. e.g. country of origin of > tweet, > > > > sentiment analysis etc. > > > > > > > > Its very simple to do this and should not take you more than 1-2 days > > to > > > > implement. > > > > > > > > Thanks > > > > Manasvi > > > > > > > > > > > > On Sun, Sep 13, 2015 at 1:11 PM, Li Tao > > wrote: > > > > > > > > > Hi Roger, > > > > > > > > > > Thanks for your recommendation. I just got to know Samza. and > checked > > > its > > > > > code base. It is a little too huge for me. > > > > > > > > > > Maybe for now, I need to start a small project/application which > > > utilize > > > > > kafka as its infrastructure, so that I can use Kafka's API a lot > and > > > know > > > > > Kafka better. > > > > > > > > > > It's hard for me to initiate such project(small, useful/meaningful, > > > kafka > > > > > based). Anyone has better idea? > > > > > > > > > > On Sun, Sep 13, 2015 at 2:21 PM, Roger Hoover < > > roger.hoo...@gmail.com> > > > > > wrote: > > > > > > > > > > > Hi Li, > > > > > > > > > > > > You might take a look at Apache Samza. It's conceptually simple > > but > > > > > > powerful and makes heavy use of Kafka. > > > > > > > > > > > > Best, > > > > > > > > > > > > Roger > > > > > > > > > > > > Sent from my iPhone > > > > > > > > > > > > > On Sep 12, 2015, at 10:34 PM, Li Tao > > > wrote: > > > > > > > > > > > > > > Hi Hackers, > > > > > > > > > > > > > > This is Lee, a learner of kafka, i have read the original paper > > on > > > > > kafka, > > > > > > > and walked through the document. > > > > > > > > > > > > > > I think the best way to learn sth is to write and read code > about > > > > it. I > > > > > > am > > > > > > > wondering is there any open source code / system which is based > > on > > > > > kafka > > > > > > so > > > > > > > that i can read or contribute to? Not too complex, not too > > simple. > > > > > > > > > > > > > > Thanks a lot! > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > Hisham Mardam-Bey > > > -=[ CTO ]-=-[ Mate1 Inc. ]=- > > > > > > A: Because it messes up the order in which people normally read text. > > > Q: Why is top-posting such a bad thing? > > > A: Top-posting. > > > Q: What is the most annoying thing in e-mail? > > > > > > -=[ Codito Ergo Sum ]=- > > > > > > -- Jeff Holoman Systems Engineer
Re: open source projects based on kafka
Have a look at https://github.com/allegro/hermes On Mon, Sep 14, 2015, 01:28 David Luu wrote: > The toy project idea is good. Another option I think could be to look at > the various Kafka client langague bindings and/or utilities (like > kafkacat). And from there, another option is to build a client language > binding for a language that's kind of lacking Kafka support, some have > better support than others. > > On Sun, Sep 13, 2015 at 11:46 AM, Hisham Mardam-Bey > wrote: > > > Hi Li, > > > > You can take a look at mypipe. > > > > "MySQL binary log consumer with the ability to act on changed rows and > > publish changes to different systems with emphasis on Apache Kafka." > > > > https://github.com/mardambey/mypipe > > > > It uses Kafka as well as some related concepts and technologies like > Avro, > > a schema repository for data, and Akka. > > > > Hope this helps (= Happy hacking! > > > > hmb. > > > > > > On Sun, Sep 13, 2015 at 4:45 AM, Manasvi Gupta > > wrote: > > > > > Here's a toy project - analyzing twitter stream. > > > > > > 1) Create dev. account on twitter > > > 2) Using your dev credentials, connect to twitter stream api to > retrieve > > > stream of tweets > > > 3) Store tweets in Kafka (using Kafka producer) > > > 4) Retrieve tweets (using Kafka consumer) > > > 5) For each tweet (or group of tweets), compute some analysis either > > using > > > custom java OR use storm/samza/spark. e.g. country of origin of tweet, > > > sentiment analysis etc. > > > > > > Its very simple to do this and should not take you more than 1-2 days > to > > > implement. > > > > > > Thanks > > > Manasvi > > > > > > > > > On Sun, Sep 13, 2015 at 1:11 PM, Li Tao > wrote: > > > > > > > Hi Roger, > > > > > > > > Thanks for your recommendation. I just got to know Samza. and checked > > its > > > > code base. It is a little too huge for me. > > > > > > > > Maybe for now, I need to start a small project/application which > > utilize > > > > kafka as its infrastructure, so that I can use Kafka's API a lot and > > know > > > > Kafka better. > > > > > > > > It's hard for me to initiate such project(small, useful/meaningful, > > kafka > > > > based). Anyone has better idea? > > > > > > > > On Sun, Sep 13, 2015 at 2:21 PM, Roger Hoover < > roger.hoo...@gmail.com> > > > > wrote: > > > > > > > > > Hi Li, > > > > > > > > > > You might take a look at Apache Samza. It's conceptually simple > but > > > > > powerful and makes heavy use of Kafka. > > > > > > > > > > Best, > > > > > > > > > > Roger > > > > > > > > > > Sent from my iPhone > > > > > > > > > > > On Sep 12, 2015, at 10:34 PM, Li Tao > > wrote: > > > > > > > > > > > > Hi Hackers, > > > > > > > > > > > > This is Lee, a learner of kafka, i have read the original paper > on > > > > kafka, > > > > > > and walked through the document. > > > > > > > > > > > > I think the best way to learn sth is to write and read code about > > > it. I > > > > > am > > > > > > wondering is there any open source code / system which is based > on > > > > kafka > > > > > so > > > > > > that i can read or contribute to? Not too complex, not too > simple. > > > > > > > > > > > > Thanks a lot! > > > > > > > > > > > > > > > > > > > > -- > > Hisham Mardam-Bey > > -=[ CTO ]-=-[ Mate1 Inc. ]=- > > > > A: Because it messes up the order in which people normally read text. > > Q: Why is top-posting such a bad thing? > > A: Top-posting. > > Q: What is the most annoying thing in e-mail? > > > > -=[ Codito Ergo Sum ]=- > > >
Re: open source projects based on kafka
The toy project idea is good. Another option I think could be to look at the various Kafka client langague bindings and/or utilities (like kafkacat). And from there, another option is to build a client language binding for a language that's kind of lacking Kafka support, some have better support than others. On Sun, Sep 13, 2015 at 11:46 AM, Hisham Mardam-Bey wrote: > Hi Li, > > You can take a look at mypipe. > > "MySQL binary log consumer with the ability to act on changed rows and > publish changes to different systems with emphasis on Apache Kafka." > > https://github.com/mardambey/mypipe > > It uses Kafka as well as some related concepts and technologies like Avro, > a schema repository for data, and Akka. > > Hope this helps (= Happy hacking! > > hmb. > > > On Sun, Sep 13, 2015 at 4:45 AM, Manasvi Gupta > wrote: > > > Here's a toy project - analyzing twitter stream. > > > > 1) Create dev. account on twitter > > 2) Using your dev credentials, connect to twitter stream api to retrieve > > stream of tweets > > 3) Store tweets in Kafka (using Kafka producer) > > 4) Retrieve tweets (using Kafka consumer) > > 5) For each tweet (or group of tweets), compute some analysis either > using > > custom java OR use storm/samza/spark. e.g. country of origin of tweet, > > sentiment analysis etc. > > > > Its very simple to do this and should not take you more than 1-2 days to > > implement. > > > > Thanks > > Manasvi > > > > > > On Sun, Sep 13, 2015 at 1:11 PM, Li Tao wrote: > > > > > Hi Roger, > > > > > > Thanks for your recommendation. I just got to know Samza. and checked > its > > > code base. It is a little too huge for me. > > > > > > Maybe for now, I need to start a small project/application which > utilize > > > kafka as its infrastructure, so that I can use Kafka's API a lot and > know > > > Kafka better. > > > > > > It's hard for me to initiate such project(small, useful/meaningful, > kafka > > > based). Anyone has better idea? > > > > > > On Sun, Sep 13, 2015 at 2:21 PM, Roger Hoover > > > wrote: > > > > > > > Hi Li, > > > > > > > > You might take a look at Apache Samza. It's conceptually simple but > > > > powerful and makes heavy use of Kafka. > > > > > > > > Best, > > > > > > > > Roger > > > > > > > > Sent from my iPhone > > > > > > > > > On Sep 12, 2015, at 10:34 PM, Li Tao > wrote: > > > > > > > > > > Hi Hackers, > > > > > > > > > > This is Lee, a learner of kafka, i have read the original paper on > > > kafka, > > > > > and walked through the document. > > > > > > > > > > I think the best way to learn sth is to write and read code about > > it. I > > > > am > > > > > wondering is there any open source code / system which is based on > > > kafka > > > > so > > > > > that i can read or contribute to? Not too complex, not too simple. > > > > > > > > > > Thanks a lot! > > > > > > > > > > > > > -- > Hisham Mardam-Bey > -=[ CTO ]-=-[ Mate1 Inc. ]=- > > A: Because it messes up the order in which people normally read text. > Q: Why is top-posting such a bad thing? > A: Top-posting. > Q: What is the most annoying thing in e-mail? > > -=[ Codito Ergo Sum ]=- >
Re: open source projects based on kafka
Hi Li, You can take a look at mypipe. "MySQL binary log consumer with the ability to act on changed rows and publish changes to different systems with emphasis on Apache Kafka." https://github.com/mardambey/mypipe It uses Kafka as well as some related concepts and technologies like Avro, a schema repository for data, and Akka. Hope this helps (= Happy hacking! hmb. On Sun, Sep 13, 2015 at 4:45 AM, Manasvi Gupta wrote: > Here's a toy project - analyzing twitter stream. > > 1) Create dev. account on twitter > 2) Using your dev credentials, connect to twitter stream api to retrieve > stream of tweets > 3) Store tweets in Kafka (using Kafka producer) > 4) Retrieve tweets (using Kafka consumer) > 5) For each tweet (or group of tweets), compute some analysis either using > custom java OR use storm/samza/spark. e.g. country of origin of tweet, > sentiment analysis etc. > > Its very simple to do this and should not take you more than 1-2 days to > implement. > > Thanks > Manasvi > > > On Sun, Sep 13, 2015 at 1:11 PM, Li Tao wrote: > > > Hi Roger, > > > > Thanks for your recommendation. I just got to know Samza. and checked its > > code base. It is a little too huge for me. > > > > Maybe for now, I need to start a small project/application which utilize > > kafka as its infrastructure, so that I can use Kafka's API a lot and know > > Kafka better. > > > > It's hard for me to initiate such project(small, useful/meaningful, kafka > > based). Anyone has better idea? > > > > On Sun, Sep 13, 2015 at 2:21 PM, Roger Hoover > > wrote: > > > > > Hi Li, > > > > > > You might take a look at Apache Samza. It's conceptually simple but > > > powerful and makes heavy use of Kafka. > > > > > > Best, > > > > > > Roger > > > > > > Sent from my iPhone > > > > > > > On Sep 12, 2015, at 10:34 PM, Li Tao wrote: > > > > > > > > Hi Hackers, > > > > > > > > This is Lee, a learner of kafka, i have read the original paper on > > kafka, > > > > and walked through the document. > > > > > > > > I think the best way to learn sth is to write and read code about > it. I > > > am > > > > wondering is there any open source code / system which is based on > > kafka > > > so > > > > that i can read or contribute to? Not too complex, not too simple. > > > > > > > > Thanks a lot! > > > > > > -- Hisham Mardam-Bey -=[ CTO ]-=-[ Mate1 Inc. ]=- A: Because it messes up the order in which people normally read text. Q: Why is top-posting such a bad thing? A: Top-posting. Q: What is the most annoying thing in e-mail? -=[ Codito Ergo Sum ]=-
Re: open source projects based on kafka
Here's a toy project - analyzing twitter stream. 1) Create dev. account on twitter 2) Using your dev credentials, connect to twitter stream api to retrieve stream of tweets 3) Store tweets in Kafka (using Kafka producer) 4) Retrieve tweets (using Kafka consumer) 5) For each tweet (or group of tweets), compute some analysis either using custom java OR use storm/samza/spark. e.g. country of origin of tweet, sentiment analysis etc. Its very simple to do this and should not take you more than 1-2 days to implement. Thanks Manasvi On Sun, Sep 13, 2015 at 1:11 PM, Li Tao wrote: > Hi Roger, > > Thanks for your recommendation. I just got to know Samza. and checked its > code base. It is a little too huge for me. > > Maybe for now, I need to start a small project/application which utilize > kafka as its infrastructure, so that I can use Kafka's API a lot and know > Kafka better. > > It's hard for me to initiate such project(small, useful/meaningful, kafka > based). Anyone has better idea? > > On Sun, Sep 13, 2015 at 2:21 PM, Roger Hoover > wrote: > > > Hi Li, > > > > You might take a look at Apache Samza. It's conceptually simple but > > powerful and makes heavy use of Kafka. > > > > Best, > > > > Roger > > > > Sent from my iPhone > > > > > On Sep 12, 2015, at 10:34 PM, Li Tao wrote: > > > > > > Hi Hackers, > > > > > > This is Lee, a learner of kafka, i have read the original paper on > kafka, > > > and walked through the document. > > > > > > I think the best way to learn sth is to write and read code about it. I > > am > > > wondering is there any open source code / system which is based on > kafka > > so > > > that i can read or contribute to? Not too complex, not too simple. > > > > > > Thanks a lot! > > >
Re: open source projects based on kafka
Hi Roger, Thanks for your recommendation. I just got to know Samza. and checked its code base. It is a little too huge for me. Maybe for now, I need to start a small project/application which utilize kafka as its infrastructure, so that I can use Kafka's API a lot and know Kafka better. It's hard for me to initiate such project(small, useful/meaningful, kafka based). Anyone has better idea? On Sun, Sep 13, 2015 at 2:21 PM, Roger Hoover wrote: > Hi Li, > > You might take a look at Apache Samza. It's conceptually simple but > powerful and makes heavy use of Kafka. > > Best, > > Roger > > Sent from my iPhone > > > On Sep 12, 2015, at 10:34 PM, Li Tao wrote: > > > > Hi Hackers, > > > > This is Lee, a learner of kafka, i have read the original paper on kafka, > > and walked through the document. > > > > I think the best way to learn sth is to write and read code about it. I > am > > wondering is there any open source code / system which is based on kafka > so > > that i can read or contribute to? Not too complex, not too simple. > > > > Thanks a lot! >
Re: open source projects based on kafka
Hi Li, You might take a look at Apache Samza. It's conceptually simple but powerful and makes heavy use of Kafka. Best, Roger Sent from my iPhone > On Sep 12, 2015, at 10:34 PM, Li Tao wrote: > > Hi Hackers, > > This is Lee, a learner of kafka, i have read the original paper on kafka, > and walked through the document. > > I think the best way to learn sth is to write and read code about it. I am > wondering is there any open source code / system which is based on kafka so > that i can read or contribute to? Not too complex, not too simple. > > Thanks a lot!
