[
https://issues.apache.org/jira/browse/HIVE-24535?focusedWorklogId=527134&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-527134
]
ASF GitHub Bot logged work on HIVE-24535:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 22/Dec/20 09:43
Start Date: 22/Dec/20 09:43
Worklog Time Spent: 10m
Work Description: pvargacl commented on a change in pull request #1779:
URL: https://github.com/apache/hive/pull/1779#discussion_r547173740
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java
##########
@@ -1490,49 +1432,31 @@ else if (prev != null && next.maxWriteId ==
prev.maxWriteId
prev = next;
}
else {
- obsolete.add(next.path);
+ directory.getObsolete().add(next.path);
}
}
+ directory.getCurrentDirectories().clear();
+ directory.getCurrentDirectories().addAll(deltas);
+ }
- if(bestBase.oldestBase != null && bestBase.basePath == null &&
- isCompactedBase(ParsedBase.parseBase(bestBase.oldestBase), fs,
dirSnapshots)) {
+ private static ValidTxnList getValidTxnList(Configuration conf) {
+ ValidTxnList validTxnList = null;
+ String s = conf.get(ValidTxnList.VALID_TXNS_KEY);
+ if(!Strings.isNullOrEmpty(s)) {
/*
- * If here, it means there was a base_x (> 1 perhaps) but none were
suitable for given
- * {@link writeIdList}. Note that 'original' files are logically a
base_Long.MIN_VALUE and thus
- * cannot have any data for an open txn. We could check {@link deltas}
has files to cover
- * [1,n] w/o gaps but this would almost never happen...
+ * getAcidState() is sometimes called on non-transactional tables, e.g.
+ * OrcInputFileFormat.FileGenerator.callInternal(). e.g. orc_merge3.q
In that case
+ * writeIdList is bogus - doesn't even have a table name.
+ * see https://issues.apache.org/jira/browse/HIVE-20856.
*
- * We only throw for base_x produced by Compactor since that base erases
all history and
- * cannot be used for a client that has a snapshot in which something
inside this base is
- * open. (Nor can we ignore this base of course) But base_x which is a
result of IOW,
- * contains all history so we treat it just like delta wrt visibility.
Imagine, IOW which
- * aborts. It creates a base_x, which can and should just be ignored.*/
- long[] exceptions = writeIdList.getInvalidWriteIds();
- String minOpenWriteId = exceptions != null && exceptions.length > 0 ?
- Long.toString(exceptions[0]) : "x";
- throw new IOException(ErrorMsg.ACID_NOT_ENOUGH_HISTORY.format(
- Long.toString(writeIdList.getHighWatermark()),
- minOpenWriteId, bestBase.oldestBase.toString()));
- }
-
- Path base = null;
- boolean isBaseInRawFormat = false;
- if (bestBase.basePath != null) {
- base = bestBase.basePath;
- isBaseInRawFormat = MetaDataFile.isRawFormat(base, fs, dirSnapshots !=
null ? dirSnapshots.get(base) : null);
+ * For now, assert that ValidTxnList.VALID_TXNS_KEY is set only if this
is really a read
+ * of a transactional table.
+ * see {@link #getChildState(FileStatus, HdfsFileStatusWithId,
ValidWriteIdList, List, List, List, List, TxnBase, boolean, List, Map,
FileSystem, ValidTxnList)}
Review comment:
Removed this link, because it is outdated, i don't see anything related
to this in getChildState.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 527134)
Time Spent: 1h 40m (was: 1.5h)
> Cleanup AcidUtils.Directory and remove unnecessary filesystem listings
> ----------------------------------------------------------------------
>
> Key: HIVE-24535
> URL: https://issues.apache.org/jira/browse/HIVE-24535
> Project: Hive
> Issue Type: Improvement
> Reporter: Peter Varga
> Assignee: Peter Varga
> Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> * AcidUtils.getAcidState is doing a recursive listing on S3 FileSystem, it
> already knows the content of each delta and base directory, this could be
> returned to OrcInputFormat, to avoid listing each delta directory again there.
> * AcidUtils.getAcidstate submethods are collecting more and more infos about
> the state of the data directory. This could be done directly to the final
> Directory object to avoid 10+ parameters in methods.
> * AcidUtils.Directory, OrcInputFormat.AcidDirInfo and AcidUtils.TxnBase can
> be merged to one class, to clean up duplications.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
