is kafka the right choice
Hey Kafka experts, After having read Jay Kreps awesome Kafka reading( https://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying) I have a doubt. For communication between browsers (lets say collaborative editing, chat etc.) is Kafka the right choice ? Especially given that Kafka consumers are designed to pull , rather than a callback style push. For low latency possibly ephemeral data/events is Kafka a good choice ? Can I have a browser open a socket into a webserver and each request initiate a consumer to consume from kafka (by polling?) OR is Kafka designed and/or meant to be used for a separate usecase ? Any feedback is appreciated. Let the bashing begin! Many Thanks, pranay
Re: is kafka the right choice
See Keyhole Software blog and particularly John Boardman's presentation of sample app with responsive web client using WebSockets connecting to a netty embedded web server that itself uses producer and consumer clients with a Kafka infrastructure (@johnwboardman). On first look, it seems like a valid approach. Behind the web server are services that are Kafka apps interacting with external web APIs. Anecdotally quite a few companies post jobs with Kafka playing a role in a micro architecture solution. I'll now let experts speak... On 23 Jun 2016 11:47 a.m., "Pranay Suresh" wrote: > Hey Kafka experts, > > After having read Jay Kreps awesome Kafka reading( > > https://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying > ) > I have a doubt. > > For communication between browsers (lets say collaborative editing, chat > etc.) is Kafka the right choice ? Especially given that Kafka consumers are > designed to pull , rather than a callback style push. For low latency > possibly ephemeral data/events is Kafka a good choice ? Can I have a > browser open a socket into a webserver and each request initiate a consumer > to consume from kafka (by polling?) OR is Kafka designed and/or meant to be > used for a separate usecase ? > > Any feedback is appreciated. Let the bashing begin! > > Many Thanks, > pranay >
Re: is kafka the right choice
Kafka uses a long poll <http://kafka.apache.org/documentation.html#design_pull>. So requests effectively block on the server, if there is insufficient data available. This elevates many of the issues associated with traditional polling approaches. Service-based applications often require directed channels to do request response. People do use Kafka in this way, Philippe gave a good example below. You just need to be aware that, should you have a lot of services that need to interact, it could involve creating a lot of topics. Kafka topics are persistent and generally long lived. They shouldn’t be considered ephemeral imho. B > On 23 Jun 2016, at 17:35, Philippe Derome wrote: > > See Keyhole Software blog and particularly John Boardman's presentation of > sample app with responsive web client using WebSockets connecting to a > netty embedded web server that itself uses producer and consumer clients > with a Kafka infrastructure (@johnwboardman). On first look, it seems like > a valid approach. Behind the web server are services that are Kafka apps > interacting with external web APIs. > > Anecdotally quite a few companies post jobs with Kafka playing a role in a > micro architecture solution. > > I'll now let experts speak... > On 23 Jun 2016 11:47 a.m., "Pranay Suresh" wrote: > >> Hey Kafka experts, >> >> After having read Jay Kreps awesome Kafka reading( >> >> https://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying >> ) >> I have a doubt. >> >> For communication between browsers (lets say collaborative editing, chat >> etc.) is Kafka the right choice ? Especially given that Kafka consumers are >> designed to pull , rather than a callback style push. For low latency >> possibly ephemeral data/events is Kafka a good choice ? Can I have a >> browser open a socket into a webserver and each request initiate a consumer >> to consume from kafka (by polling?) OR is Kafka designed and/or meant to be >> used for a separate usecase ? >> >> Any feedback is appreciated. Let the bashing begin! >> >> Many Thanks, >> pranay >>
Re: is kafka the right choice
correction: elevates => alleviates > On 24 Jun 2016, at 11:13, Ben Stopford wrote: > > Kafka uses a long poll > <http://kafka.apache.org/documentation.html#design_pull>. So requests > effectively block on the server, if there is insufficient data available. > This elevates many of the issues associated with traditional polling > approaches. > > Service-based applications often require directed channels to do request > response. People do use Kafka in this way, Philippe gave a good example > below. You just need to be aware that, should you have a lot of services > that need to interact, it could involve creating a lot of topics. Kafka > topics are persistent and generally long lived. They shouldn’t be considered > ephemeral imho. > > B > > >> On 23 Jun 2016, at 17:35, Philippe Derome > <mailto:phder...@gmail.com>> wrote: >> >> See Keyhole Software blog and particularly John Boardman's presentation of >> sample app with responsive web client using WebSockets connecting to a >> netty embedded web server that itself uses producer and consumer clients >> with a Kafka infrastructure (@johnwboardman). On first look, it seems like >> a valid approach. Behind the web server are services that are Kafka apps >> interacting with external web APIs. >> >> Anecdotally quite a few companies post jobs with Kafka playing a role in a >> micro architecture solution. >> >> I'll now let experts speak... >> On 23 Jun 2016 11:47 a.m., "Pranay Suresh" > <mailto:pranay.sur...@gmail.com>> wrote: >> >>> Hey Kafka experts, >>> >>> After having read Jay Kreps awesome Kafka reading( >>> >>> https://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying >>> >>> <https://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying> >>> ) >>> I have a doubt. >>> >>> For communication between browsers (lets say collaborative editing, chat >>> etc.) is Kafka the right choice ? Especially given that Kafka consumers are >>> designed to pull , rather than a callback style push. For low latency >>> possibly ephemeral data/events is Kafka a good choice ? Can I have a >>> browser open a socket into a webserver and each request initiate a consumer >>> to consume from kafka (by polling?) OR is Kafka designed and/or meant to be >>> used for a separate usecase ? >>> >>> Any feedback is appreciated. Let the bashing begin! >>> >>> Many Thanks, >>> pranay >>> >
