[ https://issues.apache.org/jira/browse/HIVE-21225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17038066#comment-17038066 ]
Gopal Vijayaraghavan commented on HIVE-21225: --------------------------------------------- bq. I have reopened the issue as I couldn't attach new files Since this is a task ticket titled "cache file listing" and the fix is to do exactly that, this shouldn't be reopened. If we have a performance issue from this patch, that should be a different issue - it is too late to layer on an addendum to this. Also bq. I'm not using S3guard which might have resulted in a significant amount of time for S3 calls. ACID doesn't work without S3guard on S3 - compactions will break the world for you. > ACID: getAcidState() should cache a recursive dir listing locally > ----------------------------------------------------------------- > > Key: HIVE-21225 > URL: https://issues.apache.org/jira/browse/HIVE-21225 > Project: Hive > Issue Type: Improvement > Components: Transactions > Reporter: Gopal Vijayaraghavan > Assignee: Vaibhav Gumashta > Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-21225.1.patch, HIVE-21225.10.patch, > HIVE-21225.11.patch, HIVE-21225.12.patch, HIVE-21225.13.patch, > HIVE-21225.14.patch, HIVE-21225.15.patch, HIVE-21225.15.patch, > HIVE-21225.16.patch, HIVE-21225.17.patch, HIVE-21225.2.patch, > HIVE-21225.3.patch, HIVE-21225.4.patch, HIVE-21225.4.patch, > HIVE-21225.5.patch, HIVE-21225.6.patch, HIVE-21225.7.patch, > HIVE-21225.7.patch, HIVE-21225.8.patch, HIVE-21225.9.patch, async-pid-44-2.svg > > > Currently getAcidState() makes 3 calls into the FS api which could be > answered by making a single recursive listDir call and reusing the same data > to check for isRawFormat() and isValidBase(). > All delta operations for a single partition can go against a single listed > directory snapshot instead of interacting with the NameNode or ObjectStore > within the inner loop. -- This message was sent by Atlassian Jira (v8.3.4#803005)