Hi Steve, The first phase would be pretty simple, essentially hooking up the Kafka-DStream-consumer to perform KPI aggregation over the streamed data from Kafka Broker cluster in real-time.
We would like to maximize the throughput by choosing the right message payload size, correct kafka topic/partition mapped to spark RRD/DStream and minimizing I/O. Next, we would be using featurizing the stream to be able to develop machine learning models using SVMs (Support Vector Machines) etc to provide rich insights. Would be soon giving talk on this one, so keep tuned. Regards, S.Abdullah On Tue, Dec 10, 2013 at 10:22 AM, Steve Morin <st...@stevemorin.com> wrote: > Shafaq, > What does the architecture of what your building look like? > -Steve > > > On Tue, Dec 10, 2013 at 10:19 AM, Shafaq <s.abdullah...@gmail.com> wrote: > >> Hey Guys, >> I would love to contribute to the book specially in the portion of >> Kafka-Spark integration or parts of kafka in general. >> Am building a Kafka-Spark Real-time framework here at Gree Intl Inc >> processing order of MBs of data per second. >> >> My profile: >> www.linkedin.com/in/shafaqabdullah/ >> >> >> Let me know, Im open in whatever ways. >> >> >> Regards, >> S.Abdullah >> >> >> >> On Tue, Dec 10, 2013 at 9:15 AM, chetan conikee <coni...@gmail.com>wrote: >> >>> Hey Guys >>> >>> Yes, Ben Lorica (Oreilly) and I are planning to pen a "Beginning Kafka" >>> book. >>> We only finalized this late October are hoping to start this mid-month >>> >>> Chetan >>> >>> >>> On Tue, Dec 10, 2013 at 8:45 AM, Steve Morin <st...@stevemorin.com> >>> wrote: >>> >>> > I'll let chetan comment if he's up for it. >>> > -Steve >>> > >>> > >>> > >>> > On Tue, Dec 10, 2013 at 8:40 AM, David Arthur <mum...@gmail.com> >>> wrote: >>> > >>> >> There was some talk a few months ago, not sure what the current >>> status is. >>> >> >>> >> >>> >> On 12/10/13 10:01 AM, S Ahmed wrote: >>> >> >>> >>> Is there a book or this was just an idea? >>> >>> >>> >>> >>> >>> On Mon, Mar 25, 2013 at 12:42 PM, Chris Curtin < >>> curtin.ch...@gmail.com >>> >>> >wrote: >>> >>> >>> >>> Thanks Jun, >>> >>>> >>> >>>> I've updated the example with this information. >>> >>>> >>> >>>> I've also removed some of the unnecessary newlines. >>> >>>> >>> >>>> Thanks, >>> >>>> >>> >>>> Chris >>> >>>> >>> >>>> >>> >>>> On Mon, Mar 25, 2013 at 12:04 PM, Jun Rao <jun...@gmail.com> wrote: >>> >>>> >>> >>>> Chris, >>> >>>>> >>> >>>>> This looks good. One thing about partitioning. Currently, if a >>> message >>> >>>>> doesn't have a key, we always use the random partitioner >>> (regardless of >>> >>>>> what "partitioner.class" is set to). >>> >>>>> >>> >>>>> Thanks, >>> >>>>> >>> >>>>> Jun >>> >>>>> >>> >>>>> >>> >>>>> >>> >>>>> >>> >> >>> > >>> >> >> >> >> -- >> Kind Regards, >> Shafaq >> >> > -- Kind Regards, Shafaq
