Re: Re: Spark Configuration with Python
You can use the SPARK_MEM environment variable instead of setting the system property. If you need to set other properties that can't be controlled by environment variables (which is why I wrote that patch), you can just apply that patch directly to your binary package -- it only patches a Python file. -Ewen On Wed, Nov 20, 2013 at 2:00 PM, Michal Romaniuk wrote: > Patrick: It looks to me like this configures the cluster before startup. > The setting that I want to change is the amount of memory available to > each task (by default it's 512m). It appears that this is a property of > the job itself rather than the cluster. > > Josh: I'm not sure about getting the latest version from Github because > I'm new to Spark. I didn't even manage to build the package from sources > and had to download the binaries. > > Thanks, > Michal > >> A recent pull request added a classmethod to PySpark's SparkContext that >> allows you to configure the Java system properties from Python: >> >> https://github.com/apache/incubator-spark/pull/97 >> >> >> On Wed, Nov 20, 2013 at 10:34 AM, Patrick Wendell wrote: >> >> You can add java options in SPARK_JAVA_OPTS inside of conf/spark-env.sh >> >> >> http://spark.incubator.apache.org/docs/latest/python-programming-guide.html#installing-and-configuring-pyspark >> >> - Patrick >> >> On Wed, Nov 20, 2013 at 8:52 AM, Michal Romaniuk >> wrote: >> > The info about configuration options is available at the link below, >> but >> > this seems to only work with Java. How can those options be set from >> Python? >> > >> > >> http://spark.incubator.apache.org/docs/latest/configuration.html#system-properties >> > >> > Thanks, >> > Michal > > > > >
Fwd: Re: Spark Configuration with Python
Patrick: It looks to me like this configures the cluster before startup. The setting that I want to change is the amount of memory available to each task (by default it's 512m). It appears that this is a property of the job itself rather than the cluster. Josh: I'm not sure about getting the latest version from Github because I'm new to Spark. I didn't even manage to build the package from sources and had to download the binaries. Thanks, Michal > A recent pull request added a classmethod to PySpark's SparkContext that > allows you to configure the Java system properties from Python: > > https://github.com/apache/incubator-spark/pull/97 > > > On Wed, Nov 20, 2013 at 10:34 AM, Patrick Wendell wrote: > > You can add java options in SPARK_JAVA_OPTS inside of conf/spark-env.sh > > > http://spark.incubator.apache.org/docs/latest/python-programming-guide.html#installing-and-configuring-pyspark > > - Patrick > > On Wed, Nov 20, 2013 at 8:52 AM, Michal Romaniuk > wrote: > > The info about configuration options is available at the link below, but > > this seems to only work with Java. How can those options be set from > Python? > > > > > http://spark.incubator.apache.org/docs/latest/configuration.html#system-properties > > > > Thanks, > > Michal
Re: Spark Configuration with Python
A recent pull request added a classmethod to PySpark's SparkContext that allows you to configure the Java system properties from Python: https://github.com/apache/incubator-spark/pull/97 On Wed, Nov 20, 2013 at 10:34 AM, Patrick Wendell wrote: > You can add java options in SPARK_JAVA_OPTS inside of conf/spark-env.sh > > > http://spark.incubator.apache.org/docs/latest/python-programming-guide.html#installing-and-configuring-pyspark > > - Patrick > > On Wed, Nov 20, 2013 at 8:52 AM, Michal Romaniuk > wrote: > > The info about configuration options is available at the link below, but > > this seems to only work with Java. How can those options be set from > Python? > > > > > http://spark.incubator.apache.org/docs/latest/configuration.html#system-properties > > > > Thanks, > > Michal >
Re: Spark Configuration with Python
You can add java options in SPARK_JAVA_OPTS inside of conf/spark-env.sh http://spark.incubator.apache.org/docs/latest/python-programming-guide.html#installing-and-configuring-pyspark - Patrick On Wed, Nov 20, 2013 at 8:52 AM, Michal Romaniuk wrote: > The info about configuration options is available at the link below, but > this seems to only work with Java. How can those options be set from Python? > > http://spark.incubator.apache.org/docs/latest/configuration.html#system-properties > > Thanks, > Michal
Spark Configuration with Python
The info about configuration options is available at the link below, but this seems to only work with Java. How can those options be set from Python? http://spark.incubator.apache.org/docs/latest/configuration.html#system-properties Thanks, Michal
