Shafaq, That's pretty cool, have you already connected Kafka to spark RRD/DStream or is that something you have to figure out? -Steve
On Tue, Dec 10, 2013 at 7:10 PM, Shafaq <s.abdullah...@gmail.com> wrote: > Hi Steve, > > The first phase would be pretty simple, essentially hooking up the > Kafka-DStream-consumer to perform KPI aggregation over the streamed data > from Kafka Broker cluster in real-time. > > We would like to maximize the throughput by choosing the right message > payload size, correct kafka topic/partition mapped to spark RRD/DStream > and minimizing I/O. > > Next, we would be using featurizing the stream to be able to develop > machine learning models using SVMs (Support Vector Machines) etc to provide > rich insights. > > Would be soon giving talk on this one, so keep tuned. > > Regards, > S.Abdullah > > > On Tue, Dec 10, 2013 at 10:22 AM, Steve Morin <st...@stevemorin.com>wrote: > >> Shafaq, >> What does the architecture of what your building look like? >> -Steve >> >> >> On Tue, Dec 10, 2013 at 10:19 AM, Shafaq <s.abdullah...@gmail.com> wrote: >> >>> Hey Guys, >>> I would love to contribute to the book specially in the portion of >>> Kafka-Spark integration or parts of kafka in general. >>> Am building a Kafka-Spark Real-time framework here at Gree Intl Inc >>> processing order of MBs of data per second. >>> >>> My profile: >>> www.linkedin.com/in/shafaqabdullah/ >>> >>> >>> Let me know, Im open in whatever ways. >>> >>> >>> Regards, >>> S.Abdullah >>> >>> >>> >>> On Tue, Dec 10, 2013 at 9:15 AM, chetan conikee <coni...@gmail.com>wrote: >>> >>>> Hey Guys >>>> >>>> Yes, Ben Lorica (Oreilly) and I are planning to pen a "Beginning Kafka" >>>> book. >>>> We only finalized this late October are hoping to start this mid-month >>>> >>>> Chetan >>>> >>>> >>>> On Tue, Dec 10, 2013 at 8:45 AM, Steve Morin <st...@stevemorin.com> >>>> wrote: >>>> >>>> > I'll let chetan comment if he's up for it. >>>> > -Steve >>>> > >>>> > >>>> > >>>> > On Tue, Dec 10, 2013 at 8:40 AM, David Arthur <mum...@gmail.com> >>>> wrote: >>>> > >>>> >> There was some talk a few months ago, not sure what the current >>>> status is. >>>> >> >>>> >> >>>> >> On 12/10/13 10:01 AM, S Ahmed wrote: >>>> >> >>>> >>> Is there a book or this was just an idea? >>>> >>> >>>> >>> >>>> >>> On Mon, Mar 25, 2013 at 12:42 PM, Chris Curtin < >>>> curtin.ch...@gmail.com >>>> >>> >wrote: >>>> >>> >>>> >>> Thanks Jun, >>>> >>>> >>>> >>>> I've updated the example with this information. >>>> >>>> >>>> >>>> I've also removed some of the unnecessary newlines. >>>> >>>> >>>> >>>> Thanks, >>>> >>>> >>>> >>>> Chris >>>> >>>> >>>> >>>> >>>> >>>> On Mon, Mar 25, 2013 at 12:04 PM, Jun Rao <jun...@gmail.com> >>>> wrote: >>>> >>>> >>>> >>>> Chris, >>>> >>>>> >>>> >>>>> This looks good. One thing about partitioning. Currently, if a >>>> message >>>> >>>>> doesn't have a key, we always use the random partitioner >>>> (regardless of >>>> >>>>> what "partitioner.class" is set to). >>>> >>>>> >>>> >>>>> Thanks, >>>> >>>>> >>>> >>>>> Jun >>>> >>>>> >>>> >>>>> >>>> >>>>> >>>> >>>>> >>>> >> >>>> > >>>> >>> >>> >>> >>> -- >>> Kind Regards, >>> Shafaq >>> >>> >> > > > -- > Kind Regards, > Shafaq > >
