[
https://issues.apache.org/jira/browse/HIVE-24535?focusedWorklogId=527239&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-527239
]
ASF GitHub Bot logged work on HIVE-24535:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 22/Dec/20 15:25
Start Date: 22/Dec/20 15:25
Worklog Time Spent: 10m
Work Description: pvargacl commented on a change in pull request #1779:
URL: https://github.com/apache/hive/pull/1779#discussion_r547339344
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java
##########
@@ -1316,7 +1282,7 @@ private Directory getAcidState() throws IOException {
}
- private AcidDirInfo callInternal() throws IOException {
+ private AcidDirectory callInternal() throws IOException {
if (context.acidOperationalProperties != null
Review comment:
I played around this a bit and I think will do a follow up ticket for
mm, because this again is very messy.
For mm tables HiveInputformat will take the partition/table directory, find
the delta directories in it and validate them against the validwriteid list.
This is neccessary to handle every other file format. But the consequence of
this, that the OrcInputFormat#getSplits will be called with the delta
directories directly, not with the partition/table dir as it is done normally.
So this bypass actually will do a listing in the delta dir itself. And
"findOriginals" not only finds the original for MM tables it finds all of the
files in the table, and it is necessary otherwise we would not generate split
for normal bucketfiles. So the sort answer is that these originals is not the
same originals as the context of mm tables ...
The most disgusting part is, when you create an orc table that is not
transactional, this method will be called, and we will create a full
AcidDirectory just to list all the files in the table as originals.
OrcInputformat is very much interweaved with acid code.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 527239)
Time Spent: 2h (was: 1h 50m)
> Cleanup AcidUtils.Directory and remove unnecessary filesystem listings
> ----------------------------------------------------------------------
>
> Key: HIVE-24535
> URL: https://issues.apache.org/jira/browse/HIVE-24535
> Project: Hive
> Issue Type: Improvement
> Reporter: Peter Varga
> Assignee: Peter Varga
> Priority: Major
> Labels: pull-request-available
> Time Spent: 2h
> Remaining Estimate: 0h
>
> * AcidUtils.getAcidState is doing a recursive listing on S3 FileSystem, it
> already knows the content of each delta and base directory, this could be
> returned to OrcInputFormat, to avoid listing each delta directory again there.
> * AcidUtils.getAcidstate submethods are collecting more and more infos about
> the state of the data directory. This could be done directly to the final
> Directory object to avoid 10+ parameters in methods.
> * AcidUtils.Directory, OrcInputFormat.AcidDirInfo and AcidUtils.TxnBase can
> be merged to one class, to clean up duplications.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
