[
https://issues.apache.org/jira/browse/SPARK-21424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16088869#comment-16088869
]
Sean Owen commented on SPARK-21424:
-----------------------------------
This is pretty vague. Can you provide a clearer description of the problem,
like, even a draft pull request? can you provide any numbers to show the
performance issue?
> Blockmanger has memory problems in high concurrent scenes
> ---------------------------------------------------------
>
> Key: SPARK-21424
> URL: https://issues.apache.org/jira/browse/SPARK-21424
> Project: Spark
> Issue Type: Improvement
> Components: Block Manager
> Affects Versions: 2.1.1
> Reporter: yiming.xu
>
> * Blockmanger removeBroadcast call entries in high concurrent scenes will
> copy a lot of hashmap, producing many tuple2, we should redesign the
> blockmanager.infos.
> * In the use of spark to do the application, will also submit a lot of the
> job, this time to produce a lot of boardcost, but after the completion of the
> task, clean up the boardcost need to copy the BlockInfoManager.infos traverse
> to get their own need to clear the block, which is very Unreasonable design,
> resulting in the generation of a large number of tuple2. The more tasks, the
> higher the memory occupied, seriously affected the jvm performance.We should
> design a new program, can make the block clean up do not need to copy so much
> data, but also To ensure that multi-threaded at the same time.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
