Re: Breaking lineage and reducing stages in Spark Streaming

2015-07-10 Thread Anand Nalya
image with the charts. What product? *From:* Anand Nalya [mailto:anand.na...@gmail.com] *Sent:* donderdag 9 juli 2015 11:48 *To:* spark users *Subject:* Breaking lineage and reducing stages in Spark Streaming Hi, I've an application in which an rdd is being updated with tuples coming

Re: Breaking lineage and reducing stages in Spark Streaming

2015-07-09 Thread Tathagata Das
product? *From:* Anand Nalya [mailto:anand.na...@gmail.com] *Sent:* donderdag 9 juli 2015 11:48 *To:* spark users *Subject:* Breaking lineage and reducing stages in Spark Streaming Hi, I've an application in which an rdd is being updated with tuples coming from RDDs in a DStream

Re: Breaking lineage and reducing stages in Spark Streaming

2015-07-09 Thread Anand Nalya
? *From:* Anand Nalya [mailto:anand.na...@gmail.com] *Sent:* donderdag 9 juli 2015 11:48 *To:* spark users *Subject:* Breaking lineage and reducing stages in Spark Streaming Hi, I've an application in which an rdd is being updated with tuples coming from RDDs in a DStream with following

Breaking lineage and reducing stages in Spark Streaming

2015-07-09 Thread Anand Nalya
Hi, I've an application in which an rdd is being updated with tuples coming from RDDs in a DStream with following pattern. dstream.foreachRDD(rdd = { myRDD = myRDD.union(rdd.filter(myfilter)).reduceByKey(_+_) }) I'm using cache() and checkpointin to cache results. Over the time, the lineage

RE: Breaking lineage and reducing stages in Spark Streaming

2015-07-09 Thread Michel Hubert
Hi, I was just wondering how you generated to second image with the charts. What product? From: Anand Nalya [mailto:anand.na...@gmail.com] Sent: donderdag 9 juli 2015 11:48 To: spark users Subject: Breaking lineage and reducing stages in Spark Streaming Hi, I've an application in which an rdd

Re: Breaking lineage and reducing stages in Spark Streaming

2015-07-09 Thread Anand Nalya
juli 2015 11:48 *To:* spark users *Subject:* Breaking lineage and reducing stages in Spark Streaming Hi, I've an application in which an rdd is being updated with tuples coming from RDDs in a DStream with following pattern. dstream.foreachRDD(rdd = { myRDD = myRDD.union(rdd.filter

Re: Breaking lineage and reducing stages in Spark Streaming

2015-07-09 Thread Dean Wampler
...@vsnsystemen.nl wrote: Hi, I was just wondering how you generated to second image with the charts. What product? *From:* Anand Nalya [mailto:anand.na...@gmail.com] *Sent:* donderdag 9 juli 2015 11:48 *To:* spark users *Subject:* Breaking lineage and reducing stages in Spark

Re: Breaking lineage and reducing stages in Spark Streaming

2015-07-09 Thread Anand Nalya
*To:* spark users *Subject:* Breaking lineage and reducing stages in Spark Streaming Hi, I've an application in which an rdd is being updated with tuples coming from RDDs in a DStream with following pattern. dstream.foreachRDD(rdd = { myRDD = myRDD.union(rdd.filter(myfilter

Re: Breaking lineage and reducing stages in Spark Streaming

2015-07-09 Thread Dean Wampler
generated to second image with the charts. What product? *From:* Anand Nalya [mailto:anand.na...@gmail.com] *Sent:* donderdag 9 juli 2015 11:48 *To:* spark users *Subject:* Breaking lineage and reducing stages in Spark Streaming Hi, I've an application in which an rdd is being

Re: Breaking lineage and reducing stages in Spark Streaming

2015-07-09 Thread Tathagata Das
[mailto:anand.na...@gmail.com] *Sent:* donderdag 9 juli 2015 11:48 *To:* spark users *Subject:* Breaking lineage and reducing stages in Spark Streaming Hi, I've an application in which an rdd is being updated with tuples coming from RDDs in a DStream with following pattern. dstream.foreachRDD