deniskuzZ commented on code in PR #4114:
URL: https://github.com/apache/hive/pull/4114#discussion_r1145984918
##########
ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java:
##########
@@ -1538,32 +1538,36 @@ private static HdfsDirSnapshot addToSnapshot(Map<Path,
HdfsDirSnapshot> dirToSna
public static Map<Path, HdfsDirSnapshot> getHdfsDirSnapshots(final
FileSystem fs, final Path path)
throws IOException {
Map<Path, HdfsDirSnapshot> dirToSnapshots = new HashMap<>();
- RemoteIterator<LocatedFileStatus> itr = FileUtils.listFiles(fs, path,
true, acidHiddenFileFilter);
- while (itr.hasNext()) {
- FileStatus fStatus = itr.next();
- Path fPath = fStatus.getPath();
- if (fStatus.isDirectory() && acidTempDirFilter.accept(fPath)) {
- addToSnapshot(dirToSnapshots, fPath);
- } else {
- Path parentDirPath = fPath.getParent();
- if (acidTempDirFilter.accept(parentDirPath)) {
- while (isChildOfDelta(parentDirPath, path)) {
- // Some cases there are other directory layers between the delta
and the datafiles
- // (export-import mm table, insert with union all to mm table,
skewed tables).
- // But it does not matter for the AcidState, we just need the
deltas and the data files
- // So build the snapshot with the files inside the delta directory
- parentDirPath = parentDirPath.getParent();
- }
- HdfsDirSnapshot dirSnapshot = addToSnapshot(dirToSnapshots,
parentDirPath);
- // We're not filtering out the metadata file and acid format file,
- // as they represent parts of a valid snapshot
- // We're not using the cached values downstream, but we can
potentially optimize more in a follow-up task
- if
(fStatus.getPath().toString().contains(MetaDataFile.METADATA_FILE)) {
- dirSnapshot.addMetadataFile(fStatus);
- } else if
(fStatus.getPath().toString().contains(OrcAcidVersion.ACID_FORMAT)) {
- dirSnapshot.addOrcAcidFormatFile(fStatus);
- } else {
- dirSnapshot.addFile(fStatus);
+ Deque<RemoteIterator<LocatedFileStatus>> stack = new ArrayDeque<>();
Review Comment:
back to Sourabh's concern, what are we fixing here, if that method is not
used by Cleaner?
cc @SourabhBadhya , @veghlaci05
##########
ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java:
##########
@@ -1538,32 +1538,36 @@ private static HdfsDirSnapshot addToSnapshot(Map<Path,
HdfsDirSnapshot> dirToSna
public static Map<Path, HdfsDirSnapshot> getHdfsDirSnapshots(final
FileSystem fs, final Path path)
throws IOException {
Map<Path, HdfsDirSnapshot> dirToSnapshots = new HashMap<>();
- RemoteIterator<LocatedFileStatus> itr = FileUtils.listFiles(fs, path,
true, acidHiddenFileFilter);
- while (itr.hasNext()) {
- FileStatus fStatus = itr.next();
- Path fPath = fStatus.getPath();
- if (fStatus.isDirectory() && acidTempDirFilter.accept(fPath)) {
- addToSnapshot(dirToSnapshots, fPath);
- } else {
- Path parentDirPath = fPath.getParent();
- if (acidTempDirFilter.accept(parentDirPath)) {
- while (isChildOfDelta(parentDirPath, path)) {
- // Some cases there are other directory layers between the delta
and the datafiles
- // (export-import mm table, insert with union all to mm table,
skewed tables).
- // But it does not matter for the AcidState, we just need the
deltas and the data files
- // So build the snapshot with the files inside the delta directory
- parentDirPath = parentDirPath.getParent();
- }
- HdfsDirSnapshot dirSnapshot = addToSnapshot(dirToSnapshots,
parentDirPath);
- // We're not filtering out the metadata file and acid format file,
- // as they represent parts of a valid snapshot
- // We're not using the cached values downstream, but we can
potentially optimize more in a follow-up task
- if
(fStatus.getPath().toString().contains(MetaDataFile.METADATA_FILE)) {
- dirSnapshot.addMetadataFile(fStatus);
- } else if
(fStatus.getPath().toString().contains(OrcAcidVersion.ACID_FORMAT)) {
- dirSnapshot.addOrcAcidFormatFile(fStatus);
- } else {
- dirSnapshot.addFile(fStatus);
+ Deque<RemoteIterator<LocatedFileStatus>> stack = new ArrayDeque<>();
Review Comment:
👍
@mdayakar, could we document the side effects of using `listFiles` methods
(add a javadoc with the link to
[HADOOP-18662](https://issues.apache.org/jira/browse/HADOOP-18662))
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For additional commands, e-mail: gitbox-h...@hive.apache.org
