aokolnychyi commented on code in PR #6655:
URL: https://github.com/apache/iceberg/pull/6655#discussion_r1096373389
##########
core/src/main/java/org/apache/iceberg/hadoop/Util.java:
##########
@@ -84,11 +88,44 @@ public static String[] blockLocations(FileIO io,
ScanTaskGroup<?> taskGroup) {
return locations.toArray(HadoopInputFile.NO_LOCATION_PREFERENCE);
}
+ public static boolean mayHaveBlockLocations(FileIO io, String location) {
+ if (usesHadoopFileIO(io, location)) {
+ InputFile inputFile = io.newInputFile(location);
+ if (inputFile instanceof HadoopInputFile) {
+ String scheme = ((HadoopInputFile)
inputFile).getFileSystem().getScheme();
+ return LOCALITY_WHITELIST_FS.contains(scheme);
+
+ } else {
+ return false;
+ }
+ }
+
+ return false;
+ }
+
+ public static boolean usesHadoopFileIO(FileIO io, String location) {
+ if (io instanceof HadoopFileIO) {
+ return true;
+
+ } else if (io instanceof ResolvingFileIO) {
+ ResolvingFileIO resolvingFileIO = (ResolvingFileIO) io;
+ return
HadoopFileIO.class.isAssignableFrom(resolvingFileIO.ioClass(location));
+
+ } else {
+ return false;
+ }
+ }
+
private static String[] blockLocations(FileIO io, ContentScanTask<?> task) {
Review Comment:
Oops, this is the third overloaded `blockLocations`. Let's keep all three
next to each other.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
