[ https://issues.apache.org/jira/browse/YARN-3491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
zhihai xu updated YARN-3491: ---------------------------- Attachment: YARN-3491.000.patch > PublicLocalizer#addResource is too slow. > ---------------------------------------- > > Key: YARN-3491 > URL: https://issues.apache.org/jira/browse/YARN-3491 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager > Affects Versions: 2.7.0 > Reporter: zhihai xu > Assignee: zhihai xu > Priority: Critical > Attachments: YARN-3491.000.patch > > > Improve the public resource localization to do both FSDownload submission to > the thread pool and completed localization handling in one thread > (PublicLocalizer). > Currently FSDownload submission to the thread pool is done in > PublicLocalizer#addResource which is running in Dispatcher thread and > completed localization handling is done in PublicLocalizer#run which is > running in PublicLocalizer thread. > Because PublicLocalizer#addResource is time consuming, the thread pool can't > be fully utilized. Instead of doing public resource localization in > parallel(multithreading), public resource localization is serialized most of > the time. > Also there are two more benefits with this change: > 1. The Dispatcher thread won't be blocked by PublicLocalizer#addResource . > Dispatcher thread handles most of time critical events at Node manager. > 2. don't need synchronization on HashMap (pending). > Because pending will be only accessed in PublicLocalizer thread. -- This message was sent by Atlassian JIRA (v6.3.4#6332)