Attila Zsolt Piros created SPARK-32293: ------------------------------------------
Summary: Inconsistent default units for configuring Spark memory Key: SPARK-32293 URL: https://issues.apache.org/jira/browse/SPARK-32293 Project: Spark Issue Type: Bug Components: Documentation, Spark Core Affects Versions: 3.0.0, 2.4.6, 2.4.5, 2.4.4, 2.4.3, 2.4.2, 2.4.1, 2.4.0, 2.3.4, 2.3.3, 2.3.2, 2.3.1, 2.3.0, 2.2.3, 2.2.2, 2.2.1, 3.0.1, 3.1.0 Reporter: Attila Zsolt Piros Spark's maximum memory can be configured in several ways: - via Spark config - command line argument - environment variables Both for executors and for the driver the memory can be configured separately. All of these are following the format of JVM memory configurations in a way they are using the very same size unit suffixes ("k", "m", "g" or "t") but there is an inconsistency regarding the default unit. When no suffix is given then the given amount is passed as it is to the JVM (to the -Xmx and -Xms options) where this memory options are using bytes as a default unit, for this please see the example [here|https://docs.oracle.com/javase/8/docs/technotes/tools/windows/java.html]: {noformat} The following examples show how to set the maximum allowed size of allocated memory to 80 MB using various units: -Xmx83886080 -Xmx81920k -Xmx80m {noformat} Although the Spark memory config default suffix unit is "m". -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org