[ https://issues.apache.org/jira/browse/HIVE-21225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17036574#comment-17036574 ]
Gopal Vijayaraghavan commented on HIVE-21225: --------------------------------------------- bq. The overhead compared to the benefit where we use the statuses once per delta directory (After HIVE-21177) to determine RawFormat seems very high. This was also a correctness fix even for HDFS, the caching is necessary for the listing to be exactly once as the older compacted dirs can be deleted while a query is halfway through this loop (& there's a file-not-found [~aditya-shah]: Since your numbers seem way off from mine, I suspect you might be hitting another set of issues here & need a svg from an async-profiler attached to the AM. My builds do have HADOOP-16801, which makes a significant difference here. > ACID: getAcidState() should cache a recursive dir listing locally > ----------------------------------------------------------------- > > Key: HIVE-21225 > URL: https://issues.apache.org/jira/browse/HIVE-21225 > Project: Hive > Issue Type: Improvement > Components: Transactions > Reporter: Gopal Vijayaraghavan > Assignee: Vaibhav Gumashta > Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-21225.1.patch, HIVE-21225.10.patch, > HIVE-21225.11.patch, HIVE-21225.12.patch, HIVE-21225.13.patch, > HIVE-21225.14.patch, HIVE-21225.15.patch, HIVE-21225.15.patch, > HIVE-21225.16.patch, HIVE-21225.17.patch, HIVE-21225.2.patch, > HIVE-21225.3.patch, HIVE-21225.4.patch, HIVE-21225.4.patch, > HIVE-21225.5.patch, HIVE-21225.6.patch, HIVE-21225.7.patch, > HIVE-21225.7.patch, HIVE-21225.8.patch, HIVE-21225.9.patch, async-pid-44-2.svg > > > Currently getAcidState() makes 3 calls into the FS api which could be > answered by making a single recursive listDir call and reusing the same data > to check for isRawFormat() and isValidBase(). > All delta operations for a single partition can go against a single listed > directory snapshot instead of interacting with the NameNode or ObjectStore > within the inner loop. -- This message was sent by Atlassian Jira (v8.3.4#803005)