Thats already done. On Dec 13, 2013 9:28 AM, "Steve Morin" <st...@stevemorin.com> wrote:
> Shafaq, > That's pretty cool, have you already connected Kafka to spark > RRD/DStream or is that something you have to figure out? > -Steve > > > On Tue, Dec 10, 2013 at 7:10 PM, Shafaq <s.abdullah...@gmail.com> wrote: > >> Hi Steve, >> >> The first phase would be pretty simple, essentially hooking up the >> Kafka-DStream-consumer to perform KPI aggregation over the streamed data >> from Kafka Broker cluster in real-time. >> >> We would like to maximize the throughput by choosing the right message >> payload size, correct kafka topic/partition mapped to spark RRD/DStream >> and minimizing I/O. >> >> Next, we would be using featurizing the stream to be able to develop >> machine learning models using SVMs (Support Vector Machines) etc to provide >> rich insights. >> >> Would be soon giving talk on this one, so keep tuned. >> >> Regards, >> S.Abdullah >> >> >> On Tue, Dec 10, 2013 at 10:22 AM, Steve Morin <st...@stevemorin.com>wrote: >> >>> Shafaq, >>> What does the architecture of what your building look like? >>> -Steve >>> >>> >>> On Tue, Dec 10, 2013 at 10:19 AM, Shafaq <s.abdullah...@gmail.com>wrote: >>> >>>> Hey Guys, >>>> I would love to contribute to the book specially in the portion of >>>> Kafka-Spark integration or parts of kafka in general. >>>> Am building a Kafka-Spark Real-time framework here at Gree Intl Inc >>>> processing order of MBs of data per second. >>>> >>>> My profile: >>>> www.linkedin.com/in/shafaqabdullah/ >>>> >>>> >>>> Let me know, Im open in whatever ways. >>>> >>>> >>>> Regards, >>>> S.Abdullah >>>> >>>> >>>> >>>> On Tue, Dec 10, 2013 at 9:15 AM, chetan conikee <coni...@gmail.com>wrote: >>>> >>>>> Hey Guys >>>>> >>>>> Yes, Ben Lorica (Oreilly) and I are planning to pen a "Beginning Kafka" >>>>> book. >>>>> We only finalized this late October are hoping to start this mid-month >>>>> >>>>> Chetan >>>>> >>>>> >>>>> On Tue, Dec 10, 2013 at 8:45 AM, Steve Morin <st...@stevemorin.com> >>>>> wrote: >>>>> >>>>> > I'll let chetan comment if he's up for it. >>>>> > -Steve >>>>> > >>>>> > >>>>> > >>>>> > On Tue, Dec 10, 2013 at 8:40 AM, David Arthur <mum...@gmail.com> >>>>> wrote: >>>>> > >>>>> >> There was some talk a few months ago, not sure what the current >>>>> status is. >>>>> >> >>>>> >> >>>>> >> On 12/10/13 10:01 AM, S Ahmed wrote: >>>>> >> >>>>> >>> Is there a book or this was just an idea? >>>>> >>> >>>>> >>> >>>>> >>> On Mon, Mar 25, 2013 at 12:42 PM, Chris Curtin < >>>>> curtin.ch...@gmail.com >>>>> >>> >wrote: >>>>> >>> >>>>> >>> Thanks Jun, >>>>> >>>> >>>>> >>>> I've updated the example with this information. >>>>> >>>> >>>>> >>>> I've also removed some of the unnecessary newlines. >>>>> >>>> >>>>> >>>> Thanks, >>>>> >>>> >>>>> >>>> Chris >>>>> >>>> >>>>> >>>> >>>>> >>>> On Mon, Mar 25, 2013 at 12:04 PM, Jun Rao <jun...@gmail.com> >>>>> wrote: >>>>> >>>> >>>>> >>>> Chris, >>>>> >>>>> >>>>> >>>>> This looks good. One thing about partitioning. Currently, if a >>>>> message >>>>> >>>>> doesn't have a key, we always use the random partitioner >>>>> (regardless of >>>>> >>>>> what "partitioner.class" is set to). >>>>> >>>>> >>>>> >>>>> Thanks, >>>>> >>>>> >>>>> >>>>> Jun >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >> >>>>> > >>>>> >>>> >>>> >>>> >>>> -- >>>> Kind Regards, >>>> Shafaq >>>> >>>> >>> >> >> >> -- >> Kind Regards, >> Shafaq >> >> >
