Running a custom setup task on all workers

ideas on how to do this differently? Much obliged, *Noam Barcay* Developer // *Kenshoo* *Office* +972 3 746-6500 *427 // *Mobile* +972 54 475-3142 __ *www.Kenshoo.com* http://kenshoo.com/ -- This e-mail, as well as any attached document, may contain material

Re: sparkcontext.objectFile return thousands of partitions

maybe each of the file parts has many blocks? did you try SparkContext.coalesce to reduce the number of partitions? can be done w/ or w/o data-shuffle. *Noam Barcay* Developer // *Kenshoo* *Office* +972 3 746-6500 *427 // *Mobile* +972 54 475-3142

possible memory leak when re-creating SparkContext's in the same JVM

- for example, Ooyala's spark-jobserver - so I would imagine this is possible, no? https://github.com/ooyala/spark-jobserver/blob/master/job-server/src/spark.jobserver/JobManagerActor.scala#L104 Thanks, *Noam Barcay* Developer // *Kenshoo* *Office* +972 3 746-6500 *427 // *Mobile* +972 54 475-3142