Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/22252#discussion_r213783925
--- Diff: docs/configuration.md ---
@@ -152,7 +152,7 @@ of the most common options to set are:
<td><code>spark.driver.memory</code></td>
<td>1g</td>
<td>
- Amount of memory to use for the driver process, i.e. where
SparkContext is initialized, in MiB
+ Amount of memory to use for the driver process, i.e. where
SparkContext is initialized, in bytes
--- End diff --
I took a look at the history of this code and the only constant here is how
all places are really confused about the units over time.
It seems to me that things shifted back and forth a bit, and that now the
behavior actually differs depending on cluster manager: YARN and k8s default to
Mb, while others default to bytes.
I think it's just safer to not say what the default unit is and encourage
people to explicitly identify it; I think that's what's being done for a lot of
new time- and size-based configs. The "no unit" approach was always meant to be
only for backwards compatibility anyway.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
