Dhruve Ashar created SPARK-21243:
------------------------------------
Summary: Limit the number of maps in a single shuffle fetch
Key: SPARK-21243
URL: https://issues.apache.org/jira/browse/SPARK-21243
Project: Spark
Issue Type: Improvement
Components: Spark Core
Affects Versions: 2.1.1, 2.1.0
Reporter: Dhruve Ashar
Priority: Minor
Right now spark can limit the # of parallel fetches and also limits the amount
of data in one fetch, but one fetch to a host could be for 100's of blocks. In
one instance we saw 450+ blocks. When you have 100's of those and 1000's of
reducers fetching that becomes a lot of metadata and can run the Node Manager
out of memory. We should add a config to limit the # of maps per fetch to
reduce the load on the NM.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
