[
https://issues.apache.org/jira/browse/HIVE-21225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16881575#comment-16881575
]
Vineet Garg commented on HIVE-21225:
------------------------------------
Went over the patch once again and I am struggling to understand how this patch
solves the problem cited in the jira. This patch is still calling FS in
{{isValidBase}}, {{isCompactedBase}} and {{isRawFormated}}. May be I am
misunderstanding something here?
Currently file/dir listing is done to retrieve top level directories/files only
(and then a recursive call within original (non-acid) directories if
necessary). Multiple calls are done for {{isCompactedBase}} which makes a call
to FS to read metadata file and for {{isRawFormatted}} which beside reading
metadata file also makes FS call to look for files matching particular pattern.
This seems to be done multiple times for the same base. I believe the low
hanging fruit here is to cache this information and reuse it. This should
reduce unnecessary calls to FS.
Doing a recursive call instead of only looking for top level
subdirectories/files are causing issue where directory layout is different
(.e.g. CTAS with union). I am not sure how should the current logic be
tuned/fixed to cater to this case.
> ACID: getAcidState() should cache a recursive dir listing locally
> -----------------------------------------------------------------
>
> Key: HIVE-21225
> URL: https://issues.apache.org/jira/browse/HIVE-21225
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
> Reporter: Gopal V
> Assignee: Vaibhav Gumashta
> Priority: Major
> Attachments: HIVE-21225.1.patch, HIVE-21225.2.patch,
> HIVE-21225.3.patch, HIVE-21225.4.patch, HIVE-21225.4.patch,
> HIVE-21225.5.patch, HIVE-21225.6.patch, HIVE-21225.7.patch,
> HIVE-21225.7.patch, HIVE-21225.8.patch, HIVE-21225.9.patch, async-pid-44-2.svg
>
>
> Currently getAcidState() makes 3 calls into the FS api which could be
> answered by making a single recursive listDir call and reusing the same data
> to check for isRawFormat() and isValidBase().
> All delta operations for a single partition can go against a single listed
> directory snapshot instead of interacting with the NameNode or ObjectStore
> within the inner loop.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
