When you call rdd.take() or rdd.first(), it may[1] executor the job locally (in driver), otherwise, all the jobs are executed in cluster.
There is config called `spark.localExecution.enabled` (since 1.1+) to change this, it's not enabled by default, so all the functions will be executed in cluster. If you change set this to `true`, then you get the same behavior as 1.0. [1] If it did not get enough items from the first partitions, it will try multiple partitions in a time, so they will be executed in cluster. On Thu, Oct 9, 2014 at 12:14 PM, esamanas <evan.sama...@gmail.com> wrote: > Hi, > > I am using pyspark and I'm trying to support both Spark 1.0.2 and 1.1.0 with > my app, which will run in yarn-client mode. However, it appears when I use > 'map' to run a python lambda function over an RDD, they appear to be run on > different machines, and this is causing problems. > > In both cases, I am using a Hadoop cluster that runs linux on all of its > nodes. I am submitting my jobs with a machine running Mac OS X 10.9. As a > reproducer, here is my script: > > import platform > print sc.parallelize([1]).map(lambda x: platform.system()).take(1)[0] > > The answer in Spark 1.1.0: > 'Linux' > > The answer in Spark 1.0.2: > 'Darwin' > > In other experiments I changed the size of the list that gets parallelized, > thinking maybe 1.0.2 just runs jobs on the driver node if they're small > enough. I got the same answer (with only 1 million numbers). > > This is a troubling difference. I would expect all functions run on an RDD > to be executed on my worker nodes in the Hadoop cluster, but this is clearly > not the case for 1.0.2. Why does this difference exist? How can I > accurately detect which jobs will run where? > > Thank you, > > Evan > > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/where-are-my-python-lambda-functions-run-in-yarn-client-mode-tp16059.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > --------------------------------------------------------------------- > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
