Would you mind sharing your connection setup?
> On Dec 13, 2013, at 10:36, Shafaq <s.abdullah...@gmail.com> wrote: > > Thats already done. > >> On Dec 13, 2013 9:28 AM, "Steve Morin" <st...@stevemorin.com> wrote: >> Shafaq, >> That's pretty cool, have you already connected Kafka to spark RRD/DStream >> or is that something you have to figure out? >> -Steve >> >> >>> On Tue, Dec 10, 2013 at 7:10 PM, Shafaq <s.abdullah...@gmail.com> wrote: >>> Hi Steve, >>> >>> The first phase would be pretty simple, essentially hooking up the >>> Kafka-DStream-consumer to perform KPI aggregation over the streamed data >>> from Kafka Broker cluster in real-time. >>> >>> We would like to maximize the throughput by choosing the right message >>> payload size, correct kafka topic/partition mapped to spark RRD/DStream >>> and minimizing I/O. >>> >>> Next, we would be using featurizing the stream to be able to develop >>> machine learning models using SVMs (Support Vector Machines) etc to provide >>> rich insights. >>> >>> Would be soon giving talk on this one, so keep tuned. >>> >>> Regards, >>> S.Abdullah >>> >>> >>>> On Tue, Dec 10, 2013 at 10:22 AM, Steve Morin <st...@stevemorin.com> wrote: >>>> Shafaq, >>>> What does the architecture of what your building look like? >>>> -Steve >>>> >>>> >>>>> On Tue, Dec 10, 2013 at 10:19 AM, Shafaq <s.abdullah...@gmail.com> wrote: >>>> >>>>> Hey Guys, >>>>> I would love to contribute to the book specially in the portion of >>>>> Kafka-Spark integration or parts of kafka in general. >>>>> Am building a Kafka-Spark Real-time framework here at Gree Intl Inc >>>>> processing order of MBs of data per second. >>>>> >>>>> My profile: >>>>> www.linkedin.com/in/shafaqabdullah/ >>>>> >>>>> >>>>> Let me know, Im open in whatever ways. >>>>> >>>>> >>>>> Regards, >>>>> S.Abdullah >>>>> >>>>> >>>>> >>>>>> On Tue, Dec 10, 2013 at 9:15 AM, chetan conikee <coni...@gmail.com> >>>>>> wrote: >>>>>> Hey Guys >>>>>> >>>>>> Yes, Ben Lorica (Oreilly) and I are planning to pen a "Beginning Kafka" >>>>>> book. >>>>>> We only finalized this late October are hoping to start this mid-month >>>>>> >>>>>> Chetan >>>>>> >>>>>> >>>>>> On Tue, Dec 10, 2013 at 8:45 AM, Steve Morin <st...@stevemorin.com> >>>>>> wrote: >>>>>> >>>>>> > I'll let chetan comment if he's up for it. >>>>>> > -Steve >>>>>> > >>>>>> > >>>>>> > >>>>>> > On Tue, Dec 10, 2013 at 8:40 AM, David Arthur <mum...@gmail.com> wrote: >>>>>> > >>>>>> >> There was some talk a few months ago, not sure what the current >>>>>> >> status is. >>>>>> >> >>>>>> >> >>>>>> >> On 12/10/13 10:01 AM, S Ahmed wrote: >>>>>> >> >>>>>> >>> Is there a book or this was just an idea? >>>>>> >>> >>>>>> >>> >>>>>> >>> On Mon, Mar 25, 2013 at 12:42 PM, Chris Curtin >>>>>> >>> <curtin.ch...@gmail.com >>>>>> >>> >wrote: >>>>>> >>> >>>>>> >>> Thanks Jun, >>>>>> >>>> >>>>>> >>>> I've updated the example with this information. >>>>>> >>>> >>>>>> >>>> I've also removed some of the unnecessary newlines. >>>>>> >>>> >>>>>> >>>> Thanks, >>>>>> >>>> >>>>>> >>>> Chris >>>>>> >>>> >>>>>> >>>> >>>>>> >>>> On Mon, Mar 25, 2013 at 12:04 PM, Jun Rao <jun...@gmail.com> wrote: >>>>>> >>>> >>>>>> >>>> Chris, >>>>>> >>>>> >>>>>> >>>>> This looks good. One thing about partitioning. Currently, if a >>>>>> >>>>> message >>>>>> >>>>> doesn't have a key, we always use the random partitioner >>>>>> >>>>> (regardless of >>>>>> >>>>> what "partitioner.class" is set to). >>>>>> >>>>> >>>>>> >>>>> Thanks, >>>>>> >>>>> >>>>>> >>>>> Jun >>>>>> >>>>> >>>>>> >>>>> >>>>>> >>>>> >>>>>> >>>>> >>>>>> >> >>>>>> > >>>>> >>>>> >>>>> >>>>> -- >>>>> Kind Regards, >>>>> Shafaq >>> >>> >>> >>> -- >>> Kind Regards, >>> Shafaq
