[
https://issues.apache.org/jira/browse/HDFS-14419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
HuangTao updated HDFS-14419:
----------------------------
Description:
`hdfs dfsadmin -listOpenFiles -path /any/path` will request all opened files.
In the NameNode side, the function LeaseManager.java#getUnderConstructionFiles
will be called.
When there are only N(<maxListOpenFilesResponses, default is 100) files meet
the conditions, but the leaseManager contains M(>maxListOpenFilesResponses)
files, we will scan all leases. Finally, the hasMore will be set true and the
openFileEntries contains N files, the scenario will cause listOpenFiles() be
called again.
If M is greater more than N, the two calls to getUnderConstructionFiles will
impact the NameNode performance.
was:
`hdfs dfsadmin -listOpenFiles -path /any/path` will request all opened files.
In the NameNode side, the function LeaseManager.java#getUnderConstructionFiles
will be called.
When there are only N(<maxListOpenFilesResponses, default is 100) files meet
the conditions, but the leaseManager contains M(>maxListOpenFilesResponses)
files, we will scan all leases. Finally, the hasMore will be set true and the
openFileEntries contains N files, the scenario will cause listOpenFiles() will
be called again.
If M is greater more than N, the two calls to getUnderConstructionFiles will
impact the NameNode performance.
> Avoid repeated calls to the listOpenFiles function
> --------------------------------------------------
>
> Key: HDFS-14419
> URL: https://issues.apache.org/jira/browse/HDFS-14419
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode, performance
> Affects Versions: 3.3.0
> Reporter: HuangTao
> Assignee: HuangTao
> Priority: Minor
> Fix For: 3.3.0
>
> Attachments: HDFS-14419.001.patch
>
>
> `hdfs dfsadmin -listOpenFiles -path /any/path` will request all opened files.
> In the NameNode side, the function
> LeaseManager.java#getUnderConstructionFiles will be called.
> When there are only N(<maxListOpenFilesResponses, default is 100) files meet
> the conditions, but the leaseManager contains M(>maxListOpenFilesResponses)
> files, we will scan all leases. Finally, the hasMore will be set true and the
> openFileEntries contains N files, the scenario will cause listOpenFiles() be
> called again.
> If M is greater more than N, the two calls to getUnderConstructionFiles will
> impact the NameNode performance.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
