liupengcheng created SPARK-26660:
------------------------------------
Summary: Add warning logs for large taskBinary size
Key: SPARK-26660
URL: https://issues.apache.org/jira/browse/SPARK-26660
Project: Spark
Issue Type: Improvement
Components: Spark Core
Affects Versions: 2.4.0, 2.1.0
Environment: Spark2.1
Reporter: liupengcheng
In our production environment, we encountered a OOM problem in ML application.
After carefully investigation, it was found that some ML library may generate
large ml model, thus causing executor may not able to deserialize it and result
in OOM failures.
!screenshot-1.png!
In order to facilitate the debuging of memory problem caused by large
taskBinary broadcast, we may should add same warning logs for it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
