ayushtkn commented on code in PR #4910:
URL: https://github.com/apache/hive/pull/4910#discussion_r1412379169
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergInputFormat.java:
##########
@@ -176,6 +179,20 @@ public RecordReader<Void, Container<Record>>
getRecordReader(InputSplit split, J
}
}
+ private static void validateFilesWithinTableDirectory(InputSplit split,
JobConf job) throws IOException {
+ boolean dataFilesWithingTableLocationOnly =
+
job.getBoolean(HiveConf.ConfVars.HIVE_ICEBERG_ALLOW_DATA_IN_TABLE_LOCATION_ONLY.varname,
+
HiveConf.ConfVars.HIVE_ICEBERG_ALLOW_DATA_IN_TABLE_LOCATION_ONLY.defaultBoolVal);
+ if (dataFilesWithingTableLocationOnly) {
+ Path tableLocation = new Path(job.get(InputFormatConfig.TABLE_LOCATION));
+ for (String location : split.getLocations()) {
Review Comment:
That wasn't a cool thing I believe, I changed to use the FileScanTasks, that
is what contains the actual files to be read.
I even added a test case where a file is actually readable, so the path
would be extracted here & then for normal read, to ensure some only one time
read stuff doesn't break the normal reads
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
