[ https://issues.apache.org/jira/browse/SPARK-32293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Attila Zsolt Piros updated SPARK-32293: --------------------------------------- Target Version/s: 3.1.0 > Inconsistent default unit between Spark memory configs and JVM option > --------------------------------------------------------------------- > > Key: SPARK-32293 > URL: https://issues.apache.org/jira/browse/SPARK-32293 > Project: Spark > Issue Type: Bug > Components: Documentation, Spark Core > Affects Versions: 2.2.1, 2.2.2, 2.2.3, 2.3.0, 2.3.1, 2.3.2, 2.3.3, 2.3.4, > 2.4.0, 2.4.1, 2.4.2, 2.4.3, 2.4.4, 2.4.5, 2.4.6, 3.0.0, 3.0.1, 3.1.0 > Reporter: Attila Zsolt Piros > Priority: Major > > Spark's maximum memory can be configured in several ways: > - via Spark config > - command line argument > - environment variables > Both for executors and for the driver the memory can be configured > separately. All of these are following the format of JVM memory > configurations in a way they are using the very same size unit suffixes ("k", > "m", "g" or "t") but there is an inconsistency regarding the default unit. > When no suffix is given then the given amount is passed as it is to the JVM > (to the -Xmx and -Xms options) where this memory options are using bytes as a > default unit, for this please see the example > [here|https://docs.oracle.com/javase/8/docs/technotes/tools/windows/java.html]: > {noformat} > The following examples show how to set the maximum allowed size of allocated > memory to 80 MB using various units: > -Xmx83886080 > -Xmx81920k > -Xmx80m > {noformat} > Although the Spark memory config default suffix unit is "m". -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org