Sorry didn't realize the mailing list wasn't copied...
---------- Forwarded message ---------- From: Jonathan Hodges <hodg...@gmail.com> Date: Wed, Jun 4, 2014 at 10:56 AM Subject: Re: Hadoop Summit Meetups To: Neha Narkhede <neha.narkh...@gmail.com> We have a number of customer facing online learning applications. These applications are using heterogeneous technologies with different data models in underlying data stores such as RDBMS, Cassandra, MongoDB, etc. We would like to run offline analysis on the data contained in these learning applications with tools like Hadoop and Spark. One thought is to use Kafka as a way for these learning applications to emit data in near real-time for analytics. We developed a common model represented as Avro records in HDFS that spans these learning applications so that we can accept the same structured message from them. This allows for comparing apples to apples across these apps as opposed to messy transformations. So this all sounds good until you dig into the details. One pattern is for these applications to update state locally in their data stores first and then publish to Kafka. The problem with this is these two operations aren't atomic so the local persist can succeed and the publish to Kafka fail leaving the application and HDFS out of sync. You can try to add some retry logic to the clients, but this quickly becomes very complicated and still doesn't solve the underlying problem. Another pattern is to publish to Kafka first with -1 and wait for the ack from leader and replicas before persisting locally. This is probably better than the other pattern but does add some complexity to the client. The clients must now generate unique entity IDs/UUID for persistence when they typically rely on the data store for creating these. Also the publish to Kafka can succeed and persist locally can fail leaving the stores out of sync. In this case the learning application needs to determine how to get itself in sync. It can rely on getting this back from Kafka, but it is possible the local store failure can't be fixed in a timely manner e.g. hardware failure, constraint, etc. In this case the application needs to show an error to the user and likely need to do something like send a delete message to Kafka to remove the earlier published message. A third last resort pattern might be go the CDC route with something like Databus. This would require implementing additional fetchers and relays to support Cassandra and MongoDB. Also the data will need to be transformed on the Hadoop/Spark side for virtually every learning application since they have different data models. I hope this gives enough detail to start discussing transactional messaging in Kafka. We are willing to help in this effort if it makes sense for our use cases. Thanks Jonathan On Wed, Jun 4, 2014 at 9:44 AM, Neha Narkhede <neha.narkh...@gmail.com> wrote: > If you are comfortable, share it on the mailing list. If not, I'm happy to > have this discussion privately. > > Thanks, > Neha > On Jun 4, 2014 9:42 AM, "Neha Narkhede" <neha.narkh...@gmail.com> wrote: > >> Glad it was useful. It will be great if you can share your requirements >> on atomicity. A couple of us are very interested in thinking about >> transactional messaging in Kafka. >> >> Thanks, >> Neha >> On Jun 4, 2014 6:57 AM, "Jonathan Hodges" <hodg...@gmail.com> wrote: >> >>> Hi Neha, >>> >>> Thanks so much to you and the Kafka team for putting together the meetup. >>> It was very nice and gave people from out of town like us the ability to >>> join in person. >>> >>> We are the guys from Pearson Education and we talked a little about >>> supplying some details on some of our use cases with respect to atomicity >>> of source systems eventing data and persisting locally. Should we just >>> post to the list or is there somewhere else we should send these details? >>> >>> Thanks again! >>> Jonathan >>> >>> >>> >>> On Fri, Apr 11, 2014 at 9:31 AM, Neha Narkhede <neha.narkh...@gmail.com> >>> wrote: >>> >>> > Yes, that's a great idea. I can help organize the meetup at LinkedIn. >>> > >>> > Thanks, >>> > Neha >>> > >>> > >>> > On Fri, Apr 11, 2014 at 8:44 AM, Saurabh Agarwal (BLOOMBERG/ 731 >>> LEXIN) < >>> > sagarwal...@bloomberg.net> wrote: >>> > >>> > > great idea. I am interested in attending as well.... >>> > > >>> > > ----- Original Message ----- >>> > > From: users@kafka.apache.org >>> > > To: users@kafka.apache.org >>> > > At: Apr 11 2014 11:40:56 >>> > > >>> > > With the Hadoop Summit in San Jose 6/3 - 6/5 I wondered if any of the >>> > > LinkedIn geniuses were thinking of putting together a meet-up on any >>> of >>> > the >>> > > associated technologies like Kafka, Samza, Databus, etc. For us poor >>> > souls >>> > > that don't live on the West Coast it was a great experience >>> attending the >>> > > Kafka meetup last year. >>> > > >>> > > Jonathan >>> > > >>> > > >>> > > >>> > > >>> > > >>> > >>> ------------------------------------------------------------------------------- >>> > > >>> > >>> >>
