RDD for Storm Streaming in Spark
Hi, Can we use Storm Streaming as RDD in Spark? Or any way to get Spark work with Storm? Thanks Ajay
Re: RDD for Storm Streaming in Spark
Hi, I'm not sure what you are asking: Whether we can use spouts and bolts in Spark (= no) or whether we can do streaming in Spark: http://spark.apache.org/docs/latest/streaming-programming-guide.html -kr, Gerard. On Tue, Dec 23, 2014 at 9:03 AM, Ajay ajay.ga...@gmail.com wrote: Hi, Can we use Storm Streaming as RDD in Spark? Or any way to get Spark work with Storm? Thanks Ajay
Re: RDD for Storm Streaming in Spark
Hi, The question is to do streaming in Spark with Storm (not using Spark Streaming). The idea is to use Spark as a in-memory computation engine and static data coming from Cassandra/Hbase and streaming data from Storm. Thanks Ajay On Tue, Dec 23, 2014 at 2:03 PM, Gerard Maas gerard.m...@gmail.com wrote: Hi, I'm not sure what you are asking: Whether we can use spouts and bolts in Spark (= no) or whether we can do streaming in Spark: http://spark.apache.org/docs/latest/streaming-programming-guide.html -kr, Gerard. On Tue, Dec 23, 2014 at 9:03 AM, Ajay ajay.ga...@gmail.com wrote: Hi, Can we use Storm Streaming as RDD in Spark? Or any way to get Spark work with Storm? Thanks Ajay
Re: RDD for Storm Streaming in Spark
I'm not aware of a project trying to achieve this integration. At some point Summingbird had the intention of adding an Spark port, and that could potentially bridge Storm and Spark. Not sure if that evolved into something concrete. In any case, an attempt to bring Storm and Spark together will consist in Storm collecting the data to construct an RDD and provide that to Spark for processing, basically what Spark Streaming is doing. -kr, Gerard. On Tue, Dec 23, 2014 at 9:49 AM, Ajay ajay.ga...@gmail.com wrote: Hi, The question is to do streaming in Spark with Storm (not using Spark Streaming). The idea is to use Spark as a in-memory computation engine and static data coming from Cassandra/Hbase and streaming data from Storm. Thanks Ajay On Tue, Dec 23, 2014 at 2:03 PM, Gerard Maas gerard.m...@gmail.com wrote: Hi, I'm not sure what you are asking: Whether we can use spouts and bolts in Spark (= no) or whether we can do streaming in Spark: http://spark.apache.org/docs/latest/streaming-programming-guide.html -kr, Gerard. On Tue, Dec 23, 2014 at 9:03 AM, Ajay ajay.ga...@gmail.com wrote: Hi, Can we use Storm Streaming as RDD in Spark? Or any way to get Spark work with Storm? Thanks Ajay
Re: RDD for Storm Streaming in Spark
Right. I contacted the SummingBird users as well. It doesn't support Spark streaming currently. We are heading towards Storm as it is mostly widely used. Is Spark streaming production ready? Thanks Ajay On Tue, Dec 23, 2014 at 3:47 PM, Gerard Maas gerard.m...@gmail.com wrote: I'm not aware of a project trying to achieve this integration. At some point Summingbird had the intention of adding an Spark port, and that could potentially bridge Storm and Spark. Not sure if that evolved into something concrete. In any case, an attempt to bring Storm and Spark together will consist in Storm collecting the data to construct an RDD and provide that to Spark for processing, basically what Spark Streaming is doing. -kr, Gerard. On Tue, Dec 23, 2014 at 9:49 AM, Ajay ajay.ga...@gmail.com wrote: Hi, The question is to do streaming in Spark with Storm (not using Spark Streaming). The idea is to use Spark as a in-memory computation engine and static data coming from Cassandra/Hbase and streaming data from Storm. Thanks Ajay On Tue, Dec 23, 2014 at 2:03 PM, Gerard Maas gerard.m...@gmail.com wrote: Hi, I'm not sure what you are asking: Whether we can use spouts and bolts in Spark (= no) or whether we can do streaming in Spark: http://spark.apache.org/docs/latest/streaming-programming-guide.html -kr, Gerard. On Tue, Dec 23, 2014 at 9:03 AM, Ajay ajay.ga...@gmail.com wrote: Hi, Can we use Storm Streaming as RDD in Spark? Or any way to get Spark work with Storm? Thanks Ajay
