union eatch streaming window into a static rdd and use the static rdd periodicity

[lisendong]

the pseudo code :

object myApp {
  var myStaticRDD: RDD[Int]
  def main() {
  ...  //init streaming context, and get two DStream (streamA and streamB)
from two hdfs path

  //complex transformation using the two DStream
  val new_stream = streamA.transformWith(StreamB, (a, b, t) => {
      a.join(b).map(...)
    }
  )
  
  //join the new_stream's rdd with myStaticRDD
  new_stream.foreachRDD(rdd =>
    myStaticRDD = myStaticRDD.join(cur_stream)
  )

  // do complex model training every two hours.
  if (hour is 0, 2, 4, 6...) {
     model_training(myStaticRDD)   //will take 1 hour
  }
  }
}


I don't know how to write the code to realize training model every two hours
using that moment's myStaticRDD.
And when the model-training is running, the streaming task could also run
normally simultaneously...




--
View this message in context: 
http://apache-spark-user-list.1001560.n3.nabble.com/union-eatch-streaming-window-into-a-static-rdd-and-use-the-static-rdd-periodicity-tp22783.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org