Hi, I was just wondering how you generated to second image with the charts. What product?
From: Anand Nalya [mailto:anand.na...@gmail.com] Sent: donderdag 9 juli 2015 11:48 To: spark users Subject: Breaking lineage and reducing stages in Spark Streaming Hi, I've an application in which an rdd is being updated with tuples coming from RDDs in a DStream with following pattern. dstream.foreachRDD(rdd => { myRDD = myRDD.union(rdd.filter(myfilter)).reduceByKey(_+_) }) I'm using cache() and checkpointin to cache results. Over the time, the lineage of myRDD keeps increasing and stages in each batch of dstream keeps increasing, even though all the earlier stages are skipped. When the number of stages grow big enough, the overall delay due to scheduling delay starts increasing. The processing time for each batch is still fixed. Following figures illustrate the problem: Job execution: https://i.imgur.com/GVHeXH3.png?1 [Image removed by sender.] Delays: https://i.imgur.com/1DZHydw.png?1 [Image removed by sender.] Is there some pattern that I can use to avoid this? Regards, Anand
