I want to start a child-thread in foreachRDD. My situation is:
the job is reading from a hdfs dir continuously, and every 100 batches, I want to launch a model training task (I will make a snapshot of the rdds at that time and start the training task. the training task takes a very long time(2 hours), and I don't want the training task influence reading new batch of data. Is starting a new child thread a good solution? Could the child thread use SparkContext in the main thread and use the rdd in main thread? -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/can-we-start-a-new-thread-in-foreachRDD-in-spark-streaming-tp22845.html Sent from the Apache Spark User List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
