[ https://issues.apache.org/jira/browse/SPARK-23129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
zhoukang updated SPARK-23129: ----------------------------- Description: Currently,the deserializeStream in ExternalAppendOnlyMap#DiskMapIterator init when DiskMapIterator instance created.This will cause memory use overhead when ExternalAppendOnlyMap spill too many times. We can avoid this by making deserializeStream init when it is used the first time. was: Currently,the deserializeStream in ExternalAppendOnlyMap#DiskMapIterator init when DiskMapIterator instance created.This will cause memory use overhead when ExternalAppendOnlyMap spill too much times. We can avoid this by making deserializeStream init when it is used the first time. > Lazy init DiskMapIterator#deserializeStream to reduce memory usage when > ExternalAppendOnlyMap spill too many times > ------------------------------------------------------------------------------------------------------------------- > > Key: SPARK-23129 > URL: https://issues.apache.org/jira/browse/SPARK-23129 > Project: Spark > Issue Type: Improvement > Components: Spark Core > Affects Versions: 2.1.0 > Reporter: zhoukang > Assignee: zhoukang > Priority: Major > Fix For: 2.3.0 > > > Currently,the deserializeStream in ExternalAppendOnlyMap#DiskMapIterator init > when DiskMapIterator instance created.This will cause memory use overhead > when ExternalAppendOnlyMap spill too many times. > We can avoid this by making deserializeStream init when it is used the first > time. -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org