Yes, we have just modified the configuration, and every thing works fine.
Thanks very much for the help.
On Thu, Mar 19, 2015 at 5:24 PM, Ted Yu yuzhih...@gmail.com wrote:
For YARN, possibly this one ?
property
nameyarn.nodemanager.local-dirs/name
Is it possible that `spark.local.dir` is overriden by others? The docs say:
NOTE: In Spark 1.0 and later this will be overriden by
SPARK_LOCAL_DIRS (Standalone, Mesos) or LOCAL_DIRS (YARN)
On Sat, Mar 14, 2015 at 5:29 PM, Peng Xia sparkpeng...@gmail.com wrote:
Hi Sean,
Thank very much for
IIRC you have to set that configuration on the Worker processes (for
standalone). The app can't override it (only for a client-mode
driver). YARN has a similar configuration, but I don't know the name
(shouldn't be hard to find, though).
On Thu, Mar 19, 2015 at 11:56 AM, Davies Liu
For YARN, possibly this one ?
property
nameyarn.nodemanager.local-dirs/name
value/hadoop/yarn/local/value
/property
Cheers
On Thu, Mar 19, 2015 at 2:21 PM, Marcelo Vanzin van...@cloudera.com wrote:
IIRC you have to set that configuration on the Worker processes (for
Hi Sean,
Thank very much for your reply.
I tried to config it from below code:
sf = SparkConf().setAppName(test).set(spark.executor.memory,
45g).set(spark.cores.max, 62),set(spark.local.dir, C:\\tmp)
But still get the error.
Do you know how I can config this?
Thanks,
Best,
Peng
On Sat, Mar
And I have 2 TB free space on C driver.
On Sat, Mar 14, 2015 at 8:29 PM, Peng Xia sparkpeng...@gmail.com wrote:
Hi Sean,
Thank very much for your reply.
I tried to config it from below code:
sf = SparkConf().setAppName(test).set(spark.executor.memory,
45g).set(spark.cores.max,
It means pretty much what it says. You ran out of space on an executor
(not driver), because the dir used for serialization temp files is
full (not all volumes). Set spark.local.dirs to something more
appropriate and larger.
On Sat, Mar 14, 2015 at 2:10 AM, Peng Xia sparkpeng...@gmail.com wrote:
Hi
I was running a logistic regression algorithm on a 8 nodes spark cluster,
each node has 8 cores and 56 GB Ram (each node is running a windows
system). And the spark installation driver has 1.9 TB capacity. The dataset
I was training on are has around 40 million records with around 6600
