Ruochen Zou created SPARK-50186:
-----------------------------------
Summary: Remove Hardcoded OnOutOfMemoryError Setting in Executor
Startup Script
Key: SPARK-50186
URL: https://issues.apache.org/jira/browse/SPARK-50186
Project: Spark
Issue Type: Improvement
Components: Spark Core
Affects Versions: 3.3.1, 4.0.0
Reporter: Ruochen Zou
Currently, the Executor startup script hardcodes the addition of
{{{}-XX:OnOutOfMemoryError='kill %p'{}}}, which causes the process to be killed
when the Executor encounters an OOM (Out Of Memory) error. As a result, the
YarnAllocator receives an exit code of 143 and is unable to accurately
determine the reason for the Executor's termination based on this exit code.
Moreover, the CoarseGrainedExecutorBackend cannot guarantee that StatusUpdate
messages are sent to the Driver before the process is killed.
Could we remove this setting, since users can set it via the
{{spark.executor.extraJavaOptions}} parameter if necessary?
Executor log:
!image-2024-10-31-14-02-10-261.png!
Driver log:
!image-2024-10-31-14-11-22-952.png!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
