[ https://issues.apache.org/jira/browse/HDFS-16044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
ludun updated HDFS-16044: ------------------------- Attachment: (was: HDFS-16044.03.patch) > getListing call getLocatedBlocks even source is a directory > ----------------------------------------------------------- > > Key: HDFS-16044 > URL: https://issues.apache.org/jira/browse/HDFS-16044 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode > Affects Versions: 3.1.1 > Reporter: ludun > Assignee: ludun > Priority: Major > Attachments: HDFS-16044.00.patch, HDFS-16044.01.patch, > HDFS-16044.02.patch > > > In production cluster when call getListing very frequent. The processing > time of rpc request is very high. we try to optimize the performance of > getListing request. > After some check, we found that, even the source and child is dir, the > getListing request also call getLocatedBlocks. > the request is and needLocation is false > {code:java} > 2021-05-27 15:16:07,093 TRACE ipc.ProtobufRpcEngine: 1: Call -> > 8-5-231-4/8.5.231.4:25000: getListing {src: > "/data/connector/test/topics/102test" startAfter: "" needLocation: false} > {code} > but getListing request 1000 times getLocatedBlocks which not needed. > {code:java} > `---ts=2021-05-27 14:19:15;thread_name=IPC Server handler 86 on > 25000;id=e6;is_daemon=true;priority=5;TCCL=sun.misc.Launcher$AppClassLoader@5fcfe4b2 > `---[35.068532ms] > org.apache.hadoop.hdfs.server.namenode.FSDirStatAndListingOp:getListing() > +---[0.003542ms] > org.apache.hadoop.hdfs.server.namenode.INodesInPath:getPathComponents() #214 > +---[0.003053ms] > org.apache.hadoop.hdfs.server.namenode.FSDirectory:isExactReservedName() #95 > +---[0.002938ms] > org.apache.hadoop.hdfs.server.namenode.FSDirectory:readLock() #218 > +---[0.00252ms] > org.apache.hadoop.hdfs.server.namenode.INodesInPath:isDotSnapshotDir() #220 > +---[0.002788ms] > org.apache.hadoop.hdfs.server.namenode.INodesInPath:getPathSnapshotId() #223 > +---[0.002905ms] > org.apache.hadoop.hdfs.server.namenode.INodesInPath:getLastINode() #224 > +---[0.002785ms] > org.apache.hadoop.hdfs.server.namenode.INode:getStoragePolicyID() #230 > +---[0.002236ms] > org.apache.hadoop.hdfs.server.namenode.INode:isDirectory() #233 > +---[0.002919ms] > org.apache.hadoop.hdfs.server.namenode.INode:asDirectory() #242 > +---[0.003408ms] > org.apache.hadoop.hdfs.server.namenode.INodeDirectory:getChildrenList() #243 > +---[0.005942ms] > org.apache.hadoop.hdfs.server.namenode.INodeDirectory:nextChild() #244 > +---[0.002467ms] org.apache.hadoop.hdfs.util.ReadOnlyList:size() #245 > +---[0.005481ms] > org.apache.hadoop.hdfs.server.namenode.FSDirectory:getLsLimit() #247 > +---[0.002176ms] > org.apache.hadoop.hdfs.server.namenode.FSDirectory:getLsLimit() #248 > +---[min=0.00211ms,max=0.005157ms,total=2.247572ms,count=1000] > org.apache.hadoop.hdfs.util.ReadOnlyList:get() #252 > +---[min=0.001946ms,max=0.005411ms,total=2.041715ms,count=1000] > org.apache.hadoop.hdfs.server.namenode.INode:isSymlink() #253 > +---[min=0.002176ms,max=0.005426ms,total=2.264472ms,count=1000] > org.apache.hadoop.hdfs.server.namenode.INode:getLocalStoragePolicyID() #254 > +---[min=0.002251ms,max=0.006849ms,total=2.351935ms,count=1000] > org.apache.hadoop.hdfs.server.namenode.FSDirStatAndListingOp:getStoragePolicyID() > #95 > +---[min=0.006091ms,max=0.012333ms,total=6.439434ms,count=1000] > org.apache.hadoop.hdfs.server.namenode.FSDirStatAndListingOp:createFileStatus() > #257 > +---[min=0.00269ms,max=0.004995ms,total=2.788194ms,count=1000] > org.apache.hadoop.hdfs.protocol.HdfsLocatedFileStatus:getLocatedBlocks() #265 > +---[0.003234ms] > org.apache.hadoop.hdfs.protocol.DirectoryListing:<init>() #274 > `---[0.002457ms] > org.apache.hadoop.hdfs.server.namenode.FSDirectory:readUnlock() #277 > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org