[ https://issues.apache.org/jira/browse/SPARK-31265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dongjoon Hyun resolved SPARK-31265. ----------------------------------- Resolution: Invalid > Add -XX:MaxDirectMemorySize jvm options in yarn mode > ---------------------------------------------------- > > Key: SPARK-31265 > URL: https://issues.apache.org/jira/browse/SPARK-31265 > Project: Spark > Issue Type: Improvement > Components: YARN > Affects Versions: 3.0.0 > Reporter: wangzhun > Priority: Minor > > Current memory composition `amMemory` + `amMemoryOverhead` > {code:java} > val capability = Records.newRecord(classOf[Resource]) > capability.setMemory(amMemory + amMemoryOverhead) > capability.setVirtualCores(amCores) > if (amResources.nonEmpty) { > ResourceRequestHelper.setResourceRequests(amResources, capability) > } > logDebug(s"Created resource capability for AM request: $capability") > {code} > {code:java} > // Add Xmx for AM memory > javaOpts += "-Xmx" + amMemory + "m" > {code} > It is possible that the physical memory of the container exceeds the limit > and is killed by yarn. > I suggest setting `-XX:MaxDirectMemorySize` here > {code:java} > // Add Xmx for AM memory > javaOpts += "-Xmx" + amMemory + "m" > javaOpts += s"-XX:MaxDirectMemorySize=${amMemoryOverhead}m"{code} -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org