szehon-ho commented on a change in pull request #3400:
URL: https://github.com/apache/iceberg/pull/3400#discussion_r740456967
##########
File path:
spark/v3.2/spark/src/main/java/org/apache/iceberg/spark/source/SparkBatchQueryScan.java
##########
@@ -104,30 +144,100 @@
}
}
- if (splitSize != null) {
- scan = scan.option(TableProperties.SPLIT_SIZE, splitSize.toString());
+ for (Expression filter : filterExpressions()) {
+ scan = scan.filter(filter);
}
- if (splitLookback != null) {
- scan = scan.option(TableProperties.SPLIT_LOOKBACK,
splitLookback.toString());
+ try (CloseableIterable<FileScanTask> filesIterable = scan.planFiles()) {
+ this.files = Lists.newArrayList(filesIterable);
+ } catch (IOException e) {
+ throw new UncheckedIOException("Failed to close table scan: " + scan,
e);
}
+ }
- if (splitOpenFileCost != null) {
- scan = scan.option(TableProperties.SPLIT_OPEN_FILE_COST,
splitOpenFileCost.toString());
- }
+ return files;
+ }
- for (Expression filter : filterExpressions()) {
- scan = scan.filter(filter);
+ @Override
+ protected List<CombinedScanTask> tasks() {
+ if (tasks == null) {
+ CloseableIterable<FileScanTask> splitFiles = TableScanUtil.splitFiles(
+ CloseableIterable.withNoopClose(files()),
+ splitSize);
+ CloseableIterable<CombinedScanTask> scanTasks = TableScanUtil.planTasks(
+ splitFiles, splitSize,
+ splitLookback, splitOpenFileCost);
+ tasks = Lists.newArrayList(scanTasks);
+ }
+
+ return tasks;
+ }
+
+ @Override
+ public NamedReference[] filterAttributes() {
+ Set<Integer> partitionFieldSourceIds = Sets.newHashSet();
+
+ for (Integer specId : specIds()) {
+ PartitionSpec spec = table().specs().get(specId);
+ for (PartitionField field : spec.fields()) {
+ partitionFieldSourceIds.add(field.sourceId());
}
+ }
- try (CloseableIterable<CombinedScanTask> tasksIterable =
scan.planTasks()) {
- this.tasks = Lists.newArrayList(tasksIterable);
- } catch (IOException e) {
- throw new RuntimeIOException(e, "Failed to close table scan: %s",
scan);
+ Map<Integer, String> nameById =
TypeUtil.indexQuotedNameById(table().schema().asStruct());
Review comment:
Initially I was fearing a perverse case where you rename partition
columns to be each other (then. you would filter on the wrong column), but
Iceberg seems to prevent it:
* Caused by: java.lang.IllegalArgumentException: Cannot create identity
partition sourced from different field in schema: b
* java.lang.IllegalArgumentException: Cannot use partition name more than
once: b
Then, something simple like:
* Have a table partitioned by a, insert data.
* Rename the partition column to b (table.schema() will now be different
schema)
* Run the query
In this case, the namedAttribute returned will be different than what's on
the data. I guess here it's just a missed opportunity than a correctness
issue. Could be taken advantage of by checking all values for table.schemas(),
but probably it's worth a separate pr as may be something to do carefully.
But I may be missing something, let me know if I am.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
