We have some logic that we need to apply while we are processing the events in the first minibatch only. For the second, third, etc. minibatches we don't need to do this special logic. I can't just do it as a one time thing - I need to modify a field on the events in the first minibatch.
One approach: 1- create inp = inputDStream 2- call inp.foreachRdd(rdd -> rdd.foreachPartition( isFirst = true ) ) 3- do m1 = inp.map( if (isFirst) setStuffOnEvent ) 4- do m2 = m1.mapPartition( isFirst = false ) 5- do m2.foreachRdd ( rest of my stuff) since foreachRdd in (2) is an action that just happens once right at the beginning? Or does this happen for every minibatch just like the foreachRdd() in (5) does? What is another way that we accomplish this need of setting something on the events in the first minibatch and not others? -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Conditionally-do-things-different-on-the-first-minibatch-vs-subsequent-minibatches-in-a-dstream-tp24547.html Sent from the Apache Spark User List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
