Hi Greg, SPARK_EXECUTOR_INSTANCES is the total number of workers in the cluster. The equivalent "spark.executor.instances" is just another way to set the same thing in your spark-defaults.conf. Maybe this should be documented. :)
"spark.yarn.executor.memoryOverhead" is just an additional margin added to "spark.executor.memory" for the container. In addition to the executor's memory, the container in which the executor is launched needs some extra memory for system processes, and this is what this "overhead" (somewhat of a misnomer) is for. The same goes for the driver equivalent. "spark.driver.memory" behaves differently depending on which version of Spark you are using. If you are using Spark 1.1+ (this was released very recently), you can directly set "spark.driver.memory" and this will take effect. Otherwise, setting this doesn't actually do anything for client deploy mode, and you have two alternatives: (1) set the environment variable equivalent SPARK_DRIVER_MEMORY in spark-env.sh, and (2) if you are using Spark submit (or bin/spark-shell, or bin/pyspark, which go through bin/spark-submit), pass the "--driver-memory" command line argument. If you want your PySpark application (driver) to pick up extra class path, you can pass the "--driver-class-path" to Spark submit. If you are using Spark 1.1+, you may set "spark.driver.extraClassPath" in your spark-defaults.conf. There is also an environment variable you could set (SPARK_CLASSPATH), though this is now deprecated. Let me know if you have more questions about these options, -Andrew 2014-09-08 6:59 GMT-07:00 Greg Hill <[email protected]>: > Is SPARK_EXECUTOR_INSTANCES the total number of workers in the cluster > or the workers per slave node? > > Is spark.executor.instances an actual config option? I found that in a > commit, but it's not in the docs. > > What is the difference between spark.yarn.executor.memoryOverhead and > spark.executor.memory > ? Same question for the 'driver' variant, but I assume it's the same > answer. > > Is there a spark.driver.memory option that's undocumented or do you have > to use the environment variable SPARK_DRIVER_MEMORY? > > What config option or environment variable do I need to set to get > pyspark interactive to pick up the yarn class path? The ones that work for > spark-shell and spark-submit don't seem to work for pyspark. > > Thanks in advance. > > Greg >
