bq. that 0.1 is always enough?
The answer is: it depends (on use cases).
The value of 0.1 has been validated by several users. I think it is a
reasonable default.
Cheers
On Mon, Mar 2, 2015 at 8:36 AM, Ryan Williams ryan.blake.willi...@gmail.com
wrote:
For reference, the initial version of
The problem is, you're left with two competing options then. You can
go through the process of deprecating the absolute one and removing it
eventually. You take away ability to set this value directly though,
meaning you'd have to set absolute values by depending on a % of what
you set your app
hey,
running my first map-red like (meaning disk-to-disk, avoiding in memory
RDDs) computation in spark on yarn i immediately got bitten by a too low
spark.yarn.executor.memoryOverhead. however it took me about an hour to
find out this was the cause. at first i observed failing shuffles leading
to
I have created SPARK-6085 with pull request:
https://github.com/apache/spark/pull/4836
Cheers
On Sat, Feb 28, 2015 at 12:08 PM, Corey Nolet cjno...@gmail.com wrote:
+1 to a better default as well.
We were working find until we ran against a real dataset which was much
larger than the test
Thanks for taking this on Ted!
On Sat, Feb 28, 2015 at 4:17 PM, Ted Yu yuzhih...@gmail.com wrote:
I have created SPARK-6085 with pull request:
https://github.com/apache/spark/pull/4836
Cheers
On Sat, Feb 28, 2015 at 12:08 PM, Corey Nolet cjno...@gmail.com wrote:
+1 to a better default as
There was a recent discussion about whether to increase or indeed make
configurable this kind of default fraction. I believe the suggestion
there too was that 9-10% is a safer default.
Advanced users can lower the resulting overhead value; it may still
have to be increased in some cases, but a
Having good out-of-box experience is desirable.
+1 on increasing the default.
On Sat, Feb 28, 2015 at 8:27 AM, Sean Owen so...@cloudera.com wrote:
There was a recent discussion about whether to increase or indeed make
configurable this kind of default fraction. I believe the suggestion
+1 to a better default as well.
We were working find until we ran against a real dataset which was much
larger than the test dataset we were using locally. It took me a couple
days and digging through many logs to figure out this value was what was
causing the problem.
On Sat, Feb 28, 2015 at
