[
https://issues.apache.org/jira/browse/SPARK-21424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
yiming.xu updated SPARK-21424:
------------------------------
Description:
* Blockmanger removeBroadcast call entries in high concurrent scenes will copy
a lot of hashmap, producing many tuple2, we should redesign the
blockmanager.infos.
* In the use of spark to do the application, will also submit a lot of the job,
this time to produce a lot of boardcost, but after the completion of the task,
clean up the boardcost need to copy the infos traverse to get their own need to
clear the block, which is very Unreasonable design, resulting in the generation
of a large number of tuple2. The more tasks, the higher the memory occupied,
seriously affected the jvm performance.We should design a new program, can make
the block clean up do not need to copy so much data, but also To ensure that
multi-threaded at the same time.
was:
* Blockmanger removeBroadcast call entries in high concurrent scenes will copy
a lot of hashmap, producing many tuple2, we should redesign the
blockmanager.infos.
In the use of spark to do the application, will also submit a lot of the job,
this time to produce a lot of boardcost, but after the completion of the task,
clean up the boardcost need to copy the infos traverse to get their own need to
clear the block, which is very Unreasonable design, resulting in the generation
of a large number of tuple2. The more tasks, the higher the memory occupied,
seriously affected the jvm performance.We should design a new program, can make
the block clean up do not need to copy so much data, but also To ensure that
multi-threaded at the same time.
> Blockmanger has memory problems in high concurrent scenes
> ---------------------------------------------------------
>
> Key: SPARK-21424
> URL: https://issues.apache.org/jira/browse/SPARK-21424
> Project: Spark
> Issue Type: Improvement
> Components: Block Manager
> Affects Versions: 2.1.1
> Reporter: yiming.xu
>
> * Blockmanger removeBroadcast call entries in high concurrent scenes will
> copy a lot of hashmap, producing many tuple2, we should redesign the
> blockmanager.infos.
> * In the use of spark to do the application, will also submit a lot of the
> job, this time to produce a lot of boardcost, but after the completion of the
> task, clean up the boardcost need to copy the infos traverse to get their own
> need to clear the block, which is very Unreasonable design, resulting in the
> generation of a large number of tuple2. The more tasks, the higher the memory
> occupied, seriously affected the jvm performance.We should design a new
> program, can make the block clean up do not need to copy so much data, but
> also To ensure that multi-threaded at the same time.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
