[
https://issues.apache.org/jira/browse/HIVE-21225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16881610#comment-16881610
]
Vineet Garg commented on HIVE-21225:
------------------------------------
The patch is building directory snapshot (dir path to file lists). For example
in this case snapshot keys look like as following:
* union_mm/delta_0000001_0000001_0002/HIVE_UNION_SUBDIR_2
* union_mm/delta_0000001_0000001_0001/HIVE_UNION_SUBDIR_1
This breaks the logic where base and delta directories are identified using
directory name prefix. Since none of these begin with base_ or delta_ they are
skipped.
We could update this logic to look for base_ or delta_ in the whole path but I
don't know if this is correct and safe.
> ACID: getAcidState() should cache a recursive dir listing locally
> -----------------------------------------------------------------
>
> Key: HIVE-21225
> URL: https://issues.apache.org/jira/browse/HIVE-21225
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
> Reporter: Gopal V
> Assignee: Vaibhav Gumashta
> Priority: Major
> Attachments: HIVE-21225.1.patch, HIVE-21225.2.patch,
> HIVE-21225.3.patch, HIVE-21225.4.patch, HIVE-21225.4.patch,
> HIVE-21225.5.patch, HIVE-21225.6.patch, HIVE-21225.7.patch,
> HIVE-21225.7.patch, HIVE-21225.8.patch, HIVE-21225.9.patch, async-pid-44-2.svg
>
>
> Currently getAcidState() makes 3 calls into the FS api which could be
> answered by making a single recursive listDir call and reusing the same data
> to check for isRawFormat() and isValidBase().
> All delta operations for a single partition can go against a single listed
> directory snapshot instead of interacting with the NameNode or ObjectStore
> within the inner loop.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
