Re: RE: Spark or Storm

2015-06-19 Thread Tathagata Das
; Ashish Soni asoni.le...@gmail.com; ayan guha guha.a...@gmail.com; user@spark.apache.org; Sateesh Kavuri sateesh.kav...@gmail.com; Spark Enthusiast sparkenthusi...@yahoo.in; Sabarish Sasidharan sabarish.sasidha...@manthan.com *Subject:* RE: RE: Spark or Storm My question is not directly related

Re: RE: Spark or Storm

2015-06-19 Thread Enno Shioji
; sparkenthusi...@yahoo.in; sabarish.sasidha...@manthan.com *Subject:* Re: RE: Spark or Storm That general description is accurate, but not really a specific issue of the direct steam. It applies to anything consuming from kafka (or, as Matei already said, any streaming system really). You can't

Re: RE: Spark or Storm

2015-06-19 Thread Ashish Soni
...@gmail.com; ayan guha guha.a...@gmail.com; user@spark.apache.org; Sateesh Kavuri sateesh.kav...@gmail.com; Spark Enthusiast sparkenthusi...@yahoo.in; Sabarish Sasidharan sabarish.sasidha...@manthan.com *Subject:* RE: RE: Spark or Storm My question is not directly related: about the exactly-once

RE: RE: Spark or Storm

2015-06-19 Thread Haopu Wang
: prajod.vettiyat...@wipro.com; Cody Koeninger; bit1...@163.com; Jordan Pilat; Will Briggs; Ashish Soni; ayan guha; user@spark.apache.org; Sateesh Kavuri; Spark Enthusiast; Sabarish Sasidharan Subject: Re: RE: Spark or Storm Fair enough, on second thought, just saying that it should be idempotent is indeed

Re: RE: Spark or Storm

2015-06-19 Thread bit1...@163.com
; Sabarish Sasidharan Subject: RE: RE: Spark or Storm My question is not directly related: about the exactly-once semantic, the document (copied below) said spark streaming gives exactly-once semantic, but actually from my test result, with check-point enabled, the application always re-process

Re: RE: Spark or Storm

2015-06-19 Thread Tathagata Das
; sabarish.sasidha...@manthan.com *Subject:* Re: RE: Spark or Storm That general description is accurate, but not really a specific issue of the direct steam. It applies to anything consuming from kafka (or, as Matei already said, any streaming system really). You can't have exactly once semantics

Re: RE: Spark or Storm

2015-06-19 Thread Enno Shioji
...@yahoo.in; sabarish.sasidha...@manthan.com *Subject:* Re: RE: Spark or Storm That general description is accurate, but not really a specific issue of the direct steam. It applies to anything consuming from kafka (or, as Matei already said, any streaming system really). You can't have exactly

Re: RE: Spark or Storm

2015-06-19 Thread Cody Koeninger
...@gmail.com; Ashish Soni asoni.le...@gmail.com; ayan guha guha.a...@gmail.com; user@spark.apache.org; Sateesh Kavuri sateesh.kav...@gmail.com; Spark Enthusiast sparkenthusi...@yahoo.in; Sabarish Sasidharan sabarish.sasidha...@manthan.com *Subject:* RE: RE: Spark or Storm My question is not directly

Re: RE: Spark or Storm

2015-06-19 Thread Cody Koeninger
asoni.le...@gmail.com; ayan guha guha.a...@gmail.com; user@spark.apache.org; Sateesh Kavuri sateesh.kav...@gmail.com; Spark Enthusiast sparkenthusi...@yahoo.in; Sabarish Sasidharan sabarish.sasidha...@manthan.com *Subject:* RE: RE: Spark or Storm My question is not directly related: about

Re: RE: Spark or Storm

2015-06-18 Thread bit1...@163.com
...@gmail.com; eshi...@gmail.com CC: wrbri...@gmail.com; asoni.le...@gmail.com; guha.a...@gmail.com; user@spark.apache.org; sateesh.kav...@gmail.com; sparkenthusi...@yahoo.in; sabarish.sasidha...@manthan.com Subject: RE: Spark or Storm not being able to read from Kafka using multiple nodes Kafka is plenty

Re: RE: Spark or Storm

2015-06-18 Thread Cody Koeninger
...@wipro.com *Date:* 2015-06-18 16:56 *To:* jrpi...@gmail.com; eshi...@gmail.com *CC:* wrbri...@gmail.com; asoni.le...@gmail.com; guha.a...@gmail.com; user@spark.apache.org; sateesh.kav...@gmail.com; sparkenthusi...@yahoo.in; sabarish.sasidha...@manthan.com *Subject:* RE: Spark or Storm not being able

RE: RE: Spark or Storm

2015-06-18 Thread prajod.vettiyattil
(WT01 - BAS); jrpi...@gmail.com; eshi...@gmail.com; wrbri...@gmail.com; asoni.le...@gmail.com; ayan guha; user; sateesh.kav...@gmail.com; sparkenthusi...@yahoo.in; sabarish.sasidha...@manthan.com Subject: Re: RE: Spark or Storm That general description is accurate, but not really a specific