GitHub user yaooqinn opened a pull request: https://github.com/apache/spark/pull/22180
[SPARK-25174][YARN]Limit the size of diagnostic message for am to unregister itself from rm ## What changes were proposed in this pull request? When using older versions of spark releases, a use case generated a huge code-gen file which hit the limitation `Constant pool has grown past JVM limit of 0xFFFF`. In this situation, it should fail immediately. But the diagnosis message sent to RM is too large, the ApplicationMaster suspended and RM's ZKStateStore was crashed. For 2.3 or later spark releases the limitation of code-gen has been removed, but maybe there are still some uncaught exceptions that contain oversized error message will cause such a problem. This PR is aim to cut down the diagnosis message size. ## How was this patch tested? Please review http://spark.apache.org/contributing.html before opening a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/yaooqinn/spark SPARK-25174 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/22180.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #22180 ---- commit 8f5b67a57f6f8e9237fbfcfd9f80a02ee73cfe5d Author: Kent Yao <yaooqinn@...> Date: 2018-08-22T02:01:28Z limit the size for am to unregister itself from rm ---- --- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org