Re: Using Kafka for "data" messages

2013-06-15 Thread Milind Parikh
similar results? > > Thanks > Josh > > > > > > From: Mahendra M > To: users@kafka.apache.org; Josh Foure > Sent: Friday, June 14, 2013 8:03 AM > Subject: Re: Using Kafka for "data" messages > > > > Hi Josh

Re: Using Kafka for "data" messages

2013-06-14 Thread Josh Foure
__ From: Mahendra M To: users@kafka.apache.org; Josh Foure Sent: Friday, June 14, 2013 8:03 AM Subject: Re: Using Kafka for "data" messages Hi Josh, Thanks for clarifying the use case. The idea is good, but I see the following three issues 1. Creating a queue

Re: Using Kafka for "data" messages

2013-06-14 Thread Archie Cowan
t;> Web will be flooded with a ton of messages that it will promptly drop >> but I >>> don't want to create a new "response" or "recommendation" topic because >>> then I feel like I am tightly coupling the message to the functionality >>

Re: Using Kafka for "data" messages

2013-06-14 Thread Mahendra M
gt; > > > > > > From: Mahendra M > To: users@kafka.apache.org; Josh Foure > Sent: Thursday, June 13, 2013 12:56 PM > Subject: Re: Using Kafka for "data" messages > > > Hi Josh, > > The idea looks very interesting.

Re: Using Kafka for "data" messages

2013-06-13 Thread Taylor Gautier
ion" topic because > > then I feel like I am tightly coupling the message to the functionality > and > > in the future different systems may want to consume those messages as > well. > > > > Does that make sense? > > Josh > > > > > > > > > > &

Re: Using Kafka for "data" messages

2013-06-13 Thread Josh Foure
nt: Thursday, June 13, 2013 2:13 PM Subject: Re: Using Kafka for "data" messages Also since you're going to be creating a topic per user, the number of concurrent users will also be a concern to Kafka as it doesn't like massive amounts of topics. Tim On Thu, Jun 13, 2013 at 10

Re: Using Kafka for "data" messages

2013-06-13 Thread Timothy Chen
s well. > > Does that make sense? > Josh > > > > > > > > From: Mahendra M > To: users@kafka.apache.org; Josh Foure > Sent: Thursday, June 13, 2013 12:56 PM > Subject: Re: Using Kafka for "data" messages > > >

Re: Using Kafka for "data" messages

2013-06-13 Thread Josh Foure
make sense? Josh From: Mahendra M To: users@kafka.apache.org; Josh Foure Sent: Thursday, June 13, 2013 12:56 PM Subject: Re: Using Kafka for "data" messages Hi Josh, The idea looks very interesting. I just had one doubt. 1. A user logs in. H

Re: Using Kafka for "data" messages

2013-06-13 Thread Mahendra M
Hi Josh, The idea looks very interesting. I just had one doubt. 1. A user logs in. His login id is sent on a topic 2. Other systems (consumers on this topic) consumer this message and publish their results to another topic This will be happening without any particular order for hundreds of users

Re: Using Kafka for "data" messages

2013-06-13 Thread Taylor Gautier
I've been talking about this kind of architecture for years. As you said it's an EDA architecture. You might also want to have a look at Esper if you haven't already - it's a perfect complement to this strategy. At my last job I built a relatively low latency site wide pub sub system that showed

Using Kafka for "data" messages

2013-06-13 Thread Josh Foure
  Hi all, my team is proposing a novel way of using Kafka and I am hoping someone can help do a sanity check on this:   1.  When a user logs into our website, we will create a “logged in” event message in Kafka containing the user id.  2.  30+ systems (consumers each in their own consumer groups)