Because that's not how you launch apps in cluster mode; you have to do
it through the command line, or by calling directly the respective
backend code to launch it.
(That being said, it would be nice to have a programmatic way of
launching apps that handled all this - this has been brought up in a
few different contexts, but I don't think there's an "official"
solution yet.)
On Wed, Oct 1, 2014 at 9:59 AM, Tamas Jambor <jambo...@gmail.com> wrote:
> thanks Marcelo.
>
> What's the reason it is not possible in cluster mode, either?
>
> On Wed, Oct 1, 2014 at 5:42 PM, Marcelo Vanzin <van...@cloudera.com> wrote:
>> You can't set up the driver memory programatically in client mode. In
>> that mode, the same JVM is running the driver, so you can't modify
>> command line options anymore when initializing the SparkContext.
>>
>> (And you can't really start cluster mode apps that way, so the only
>> way to set this is through the command line / config files.)
>>
>> On Wed, Oct 1, 2014 at 9:26 AM, jamborta <jambo...@gmail.com> wrote:
>>> Hi all,
>>>
>>> I cannot figure out why this command is not setting the driver memory (it is
>>> setting the executor memory):
>>>
>>> conf = (SparkConf()
>>> .setMaster("yarn-client")
>>> .setAppName("test")
>>> .set("spark.driver.memory", "1G")
>>> .set("spark.executor.memory", "1G")
>>> .set("spark.executor.instances", 2)
>>> .set("spark.executor.cores", 4))
>>> sc = SparkContext(conf=conf)
>>>
>>> whereas if I run the spark console:
>>> ./bin/pyspark --driver-memory 1G
>>>
>>> it sets it correctly. Seemingly they both generate the same commands in the
>>> logs.
>>>
>>> thanks a lot,
>>>
>>>
>>>
>>>
>>>
>>> --
>>> View this message in context:
>>> http://apache-spark-user-list.1001560.n3.nabble.com/spark-driver-memory-is-not-set-pyspark-1-1-0-tp15498.html
>>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
>>> For additional commands, e-mail: user-h...@spark.apache.org
>>>
>>
>>
>>
>> --
>> Marcelo
--
Marcelo
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org
