Hi All, I have a Spark job for which I need to increase the amount of memory allocated to the driver to collect a large-ish (>200M) data structure. Formerly, I accomplished this by setting SPARK_MEM before invoking my job (which effectively set memory on the driver) and then setting spark.executor.memory before creating my spark context. This was a bit awkward since it wasn't clear exactly what SPARK_MEM was meant to do (although in practice it affected only the driver).
Since the release of 1.0.0, I've started receiving messages saying to set spark.executor.memory or SPARK_DRIVER_MEMORY. This definitely helps clear things up, but still feels a bit awkward since it seems that most configuration can now be done from within the program (indeed there are very few environment variables now listed on the Spark configuration page). Furthermore, SPARK_DRIVER_MEMORY doesn't seem to appear anywhere in the web documentation. Is there a better way to set SPARK_DRIVER_MEMORY, or some documentation that I'm missing? Is there a guiding principle that would help in figuring out which configuration parameters are set through environment variables and which are set programmatically, or somewhere to look in the source for an exhaustive list of environment variable configuration options? best, -Brad --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
