[ https://issues.apache.org/jira/browse/STORM-3476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Stig Rohde Døssing reopened STORM-3476: --------------------------------------- > LocalizedResourceRetentionSet cleanup causing excessive load on Hadoop > namenode > ------------------------------------------------------------------------------- > > Key: STORM-3476 > URL: https://issues.apache.org/jira/browse/STORM-3476 > Project: Apache Storm > Issue Type: Improvement > Affects Versions: 2.0.0 > Reporter: Aaron Gresch > Assignee: Aaron Gresch > Priority: Major > Labels: pull-request-available > Fix For: 2.1.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > One of our local dev Hadoop devs noticed our storm user was by far creating > the heaviest load on our production Hadoop cluster. Looking at one of the > heaviest supervisor nodes, and comparing debug logs to the Hadoop audit log, > it looks like LocalizedResourceRetentionSet cleanup was constantly doing > opens and never deleting any files. > > The frequency can be addressed by supervisor.localizer.cleanup.interval.ms, > but even so, it seems we will continually look for files to delete even when > the target size is acceptable, resulting in unnecessary calls to Hadoop. > > -- This message was sent by Atlassian Jira (v8.3.2#803003)