yiming.xu created SPARK-21424:
---------------------------------
Summary: Blockmanger has memory problems in high concurrent scenes
Key: SPARK-21424
URL: https://issues.apache.org/jira/browse/SPARK-21424
Project: Spark
Issue Type: Improvement
Components: Block Manager
Affects Versions: 2.1.1
Reporter: yiming.xu
Blockmanger removeBroadcast call entries in high concurrent scenes will copy a
lot of hashmap, producing many tuple2, we should redesign the
blockmanager.infos.
In the use of spark to do the application, will also submit a lot of the job,
this time to produce a lot of boardcost, but after the completion of the task,
clean up the boardcost need to copy the infos traverse to get their own need to
clear the block, which is very Unreasonable settings, resulting in the
generation of a large number of tuple2. The more tasks, the higher the memory
occupied, seriously affected the jvm performance.We should design a new
program, can make the block clean up do not need to copy so much data, but also
To ensure that multi-threaded at the same time.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
