mdayakar commented on code in PR #4114:
URL: https://github.com/apache/hive/pull/4114#discussion_r1143305701
##########
ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java:
##########
@@ -1538,32 +1538,36 @@ private static HdfsDirSnapshot addToSnapshot(Map<Path,
HdfsDirSnapshot> dirToSna
public static Map<Path, HdfsDirSnapshot> getHdfsDirSnapshots(final
FileSystem fs, final Path path)
throws IOException {
Map<Path, HdfsDirSnapshot> dirToSnapshots = new HashMap<>();
- RemoteIterator<LocatedFileStatus> itr = FileUtils.listFiles(fs, path,
true, acidHiddenFileFilter);
- while (itr.hasNext()) {
- FileStatus fStatus = itr.next();
- Path fPath = fStatus.getPath();
- if (fStatus.isDirectory() && acidTempDirFilter.accept(fPath)) {
- addToSnapshot(dirToSnapshots, fPath);
- } else {
- Path parentDirPath = fPath.getParent();
- if (acidTempDirFilter.accept(parentDirPath)) {
- while (isChildOfDelta(parentDirPath, path)) {
- // Some cases there are other directory layers between the delta
and the datafiles
- // (export-import mm table, insert with union all to mm table,
skewed tables).
- // But it does not matter for the AcidState, we just need the
deltas and the data files
- // So build the snapshot with the files inside the delta directory
- parentDirPath = parentDirPath.getParent();
- }
- HdfsDirSnapshot dirSnapshot = addToSnapshot(dirToSnapshots,
parentDirPath);
- // We're not filtering out the metadata file and acid format file,
- // as they represent parts of a valid snapshot
- // We're not using the cached values downstream, but we can
potentially optimize more in a follow-up task
- if
(fStatus.getPath().toString().contains(MetaDataFile.METADATA_FILE)) {
- dirSnapshot.addMetadataFile(fStatus);
- } else if
(fStatus.getPath().toString().contains(OrcAcidVersion.ACID_FORMAT)) {
- dirSnapshot.addOrcAcidFormatFile(fStatus);
- } else {
- dirSnapshot.addFile(fStatus);
+ Deque<RemoteIterator<LocatedFileStatus>> stack = new ArrayDeque<>();
Review Comment:
@deniskuzZ I just followed the same approach implemented in
_getHdfsDirSnapshotsForCleaner()_ API. Here the main problem with the HADOOP
API. Already @ayushtkn raised an issue
[HADOOP-18662](https://issues.apache.org/jira/browse/HADOOP-18662) but the fix
is not yet merged. If Hadoop fix is merged then no need to do any fix from Hive
side.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For additional commands, e-mail: gitbox-h...@hive.apache.org
