GitHub user mukulmurthy opened a pull request: https://github.com/apache/spark/pull/22473
[SPARK-25449][CORE] Heartbeat shouldn't include accumulators for zero metrics ## What changes were proposed in this pull request? Heartbeat shouldn't include accumulators for zero metrics. Heartbeats sent from executors to the driver every 10 seconds contain metrics and are generally on the order of a few KBs. However, for large jobs with lots of tasks, heartbeats can be on the order of tens of MBs, causing tasks to die with heartbeat failures. We can mitigate this by not sending zero metrics to the driver. ## How was this patch tested? (Please explain how this patch was tested. E.g. unit tests, integration tests, manual tests) (If this patch involves UI changes, please attach a screenshot; otherwise, remove this) Please review http://spark.apache.org/contributing.html before opening a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/mukulmurthy/oss-spark 25449-heartbeat Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/22473.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #22473 ---- commit 3e0d9536512300d27201e1d5cc4d9b5755a47871 Author: Mukul Murthy <mukul.murthy@...> Date: 2018-09-17T21:55:21Z Don't send zero accumulators for metrics in heartbeat commit 3cf88a4ab34064074d42f5daa3a448e8f9def649 Author: Mukul Murthy <mukul.murthy@...> Date: 2018-09-19T18:40:47Z add tests ---- --- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org