[ https://issues.apache.org/jira/browse/HDFS-12502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16217777#comment-16217777 ]
Zhe Zhang edited comment on HDFS-12502 at 10/24/17 9:59 PM: ------------------------------------------------------------ For some reason we were getting over 600k~700k FilesInGetListing per second during a few days, causing spikes in GC time. Single op processing time (inside the FSNLock, measured via {{FSNReadLockOpNameNanosAvgTime}}) increased by over 50%. And we don't have any tool find the abusing workload. Yes we are using fair call queue but similar to NNTop it only considers number of ops; and each large listing is 100 times as expensive as a getFileInfo. We should probably also extend fair call queue to consider the cost of each op. I'll work on reverting the patch now. was (Author: zhz): For some reason we were getting over 600k~700k FilesInGetListing per second during a few days, causing spikes in GC time. Single op processing time (inside the FSNLock, measured via {{FSNReadLockOpNameNanosAvgTime}}) increased by over 50%. And we don't have any tool find the abusing workload. Yes we are using fair call queue but similar to NNTop it only considers number of ops; and each large listing is 100 times as expensive as a getFileInfo. We should probably also extend fair call queue to consider the cost of each op. > nntop should support a category based on FilesInGetListingOps > ------------------------------------------------------------- > > Key: HDFS-12502 > URL: https://issues.apache.org/jira/browse/HDFS-12502 > Project: Hadoop HDFS > Issue Type: Improvement > Components: metrics > Reporter: Zhe Zhang > Assignee: Zhe Zhang > Fix For: 2.9.0, 2.8.3, 2.7.5, 3.0.0, 3.1.0 > > Attachments: HDFS-12502.00.patch, HDFS-12502.01.patch, > HDFS-12502.02.patch, HDFS-12502.03.patch, HDFS-12502.04.patch > > > Large listing ops can oftentimes be the main contributor to NameNode > slowness. The aggregate cost of listing ops is proportional to the > {{FilesInGetListingOps}} rather than the number of listing ops. Therefore > it'd be very useful for nntop to support this category. -- This message was sent by Atlassian JIRA (v6.4.14#64029) --------------------------------------------------------------------- To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org