Is it possible to run a job that assigns work to every worker in the system? My bootleg right now is to have a spark listener hear whenever a block manager is added and to increase a split count by 1. It runs a spark job with that split count and hopes that it will at least run on the newest worker. There's some weirdness with block managers being removed sometimes allowing my count to go negative, so I just keep monotonically increasing my split count. Anyone have a way that doesn't suck?
-- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/How-to-run-a-job-on-all-workers-tp9163.html Sent from the Apache Spark User List mailing list archive at Nabble.com.
