[
https://issues.apache.org/jira/browse/HDFS-12502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16217777#comment-16217777
]
Zhe Zhang edited comment on HDFS-12502 at 10/24/17 9:59 PM:
------------------------------------------------------------
For some reason we were getting over 600k~700k FilesInGetListing per second
during a few days, causing spikes in GC time. Single op processing time (inside
the FSNLock, measured via {{FSNReadLockOpNameNanosAvgTime}}) increased by over
50%. And we don't have any tool find the abusing workload. Yes we are using
fair call queue but similar to NNTop it only considers number of ops; and each
large listing is 100 times as expensive as a getFileInfo. We should probably
also extend fair call queue to consider the cost of each op.
I'll work on reverting the patch now.
was (Author: zhz):
For some reason we were getting over 600k~700k FilesInGetListing per second
during a few days, causing spikes in GC time. Single op processing time (inside
the FSNLock, measured via {{FSNReadLockOpNameNanosAvgTime}}) increased by over
50%. And we don't have any tool find the abusing workload. Yes we are using
fair call queue but similar to NNTop it only considers number of ops; and each
large listing is 100 times as expensive as a getFileInfo. We should probably
also extend fair call queue to consider the cost of each op.
> nntop should support a category based on FilesInGetListingOps
> -------------------------------------------------------------
>
> Key: HDFS-12502
> URL: https://issues.apache.org/jira/browse/HDFS-12502
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: metrics
> Reporter: Zhe Zhang
> Assignee: Zhe Zhang
> Fix For: 2.9.0, 2.8.3, 2.7.5, 3.0.0, 3.1.0
>
> Attachments: HDFS-12502.00.patch, HDFS-12502.01.patch,
> HDFS-12502.02.patch, HDFS-12502.03.patch, HDFS-12502.04.patch
>
>
> Large listing ops can oftentimes be the main contributor to NameNode
> slowness. The aggregate cost of listing ops is proportional to the
> {{FilesInGetListingOps}} rather than the number of listing ops. Therefore
> it'd be very useful for nntop to support this category.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
