[
https://issues.apache.org/jira/browse/HIVE-24581?focusedWorklogId=533598&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-533598
]
ASF GitHub Bot logged work on HIVE-24581:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 10/Jan/21 03:59
Start Date: 10/Jan/21 03:59
Worklog Time Spent: 10m
Work Description: pvargacl commented on a change in pull request #1826:
URL: https://github.com/apache/hive/pull/1826#discussion_r553982610
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java
##########
@@ -1866,92 +1870,6 @@ private static boolean isDirUsable(Path child, long
visibilityTxnId, List<Path>
return true;
}
- public static class HdfsFileStatusWithoutId implements HdfsFileStatusWithId {
- private final FileStatus fs;
-
- public HdfsFileStatusWithoutId(FileStatus fs) {
- this.fs = fs;
- }
-
- @Override
- public FileStatus getFileStatus() {
- return fs;
- }
-
- @Override
- public Long getFileId() {
- return null;
- }
- }
-
- /**
- * Find the original files (non-ACID layout) recursively under the partition
directory.
- * @param fs the file system
- * @param dir the directory to add
- * @return the list of original files
- * @throws IOException
- */
- public static List<HdfsFileStatusWithId> findOriginals(FileSystem fs, Path
dir, Ref<Boolean> useFileIds,
- boolean ignoreEmptyFiles, boolean recursive) throws IOException {
- List<HdfsFileStatusWithId> originals = new ArrayList<>();
- List<HdfsFileStatusWithId> childrenWithId =
tryListLocatedHdfsStatus(useFileIds, fs, dir, hiddenFileFilter);
- if (childrenWithId != null) {
- for (HdfsFileStatusWithId child : childrenWithId) {
- if (child.getFileStatus().isDirectory()) {
- if (recursive) {
- originals.addAll(findOriginals(fs,
child.getFileStatus().getPath(), useFileIds,
- ignoreEmptyFiles, true));
- }
- } else {
- if (!ignoreEmptyFiles || child.getFileStatus().getLen() > 0) {
- originals.add(child);
- }
- }
- }
- } else {
- List<FileStatus> children = HdfsUtils.listLocatedStatus(fs, dir,
hiddenFileFilter);
- for (FileStatus child : children) {
- if (child.isDirectory()) {
- if (recursive) {
- originals.addAll(findOriginals(fs, child.getPath(), useFileIds,
ignoreEmptyFiles, true));
- }
- } else {
- if (!ignoreEmptyFiles || child.getLen() > 0) {
Review comment:
The findoriginals was called with ignoreEmpty = true always, so i
removed it. If I understand correctly this parameter could be removed from
every other acidutils call. It was there to handle MR, which created empty
files for every bucket.
https://issues.apache.org/jira/browse/HIVE-13040?focusedCommentId=15159223&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-15159223
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 533598)
Time Spent: 50m (was: 40m)
> Remove AcidUtils call from OrcInputformat for non transactional tables
> ----------------------------------------------------------------------
>
> Key: HIVE-24581
> URL: https://issues.apache.org/jira/browse/HIVE-24581
> Project: Hive
> Issue Type: Improvement
> Reporter: Peter Varga
> Assignee: Peter Varga
> Priority: Minor
> Labels: pull-request-available
> Time Spent: 50m
> Remaining Estimate: 0h
>
> Currently the split generation in OrcInputformat is tightly coupled with acid
> and AcidUtils.getAcidState is called even if the table is not transactional.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
