rdblue commented on a change in pull request #3292:
URL: https://github.com/apache/iceberg/pull/3292#discussion_r780817560
##########
File path: core/src/main/java/org/apache/iceberg/BaseFileScanTask.java
##########
@@ -224,5 +217,51 @@ public Expression residual() {
public Iterable<FileScanTask> split(long splitSize) {
throw new UnsupportedOperationException("Cannot split a task which is
already split");
}
+
+ public boolean isAdjacent(SplitScanTask other) {
+ return (other != null) &&
+ (this.file().equals(other.file())) &&
+ (this.offset + this.len == other.offset);
+ }
+ }
+
+ static FileScanTask[] combineSimilarTasks(List<FileScanTask> tasks) {
+ if (tasks.isEmpty()) {
+ return new FileScanTask[0];
+ }
+
+ List<FileScanTask> combinedScans = Lists.newArrayList();
+ SplitScanTask lastSplit = null;
+
+ for (FileScanTask fileScanTask : tasks) {
+ if (!(fileScanTask instanceof SplitScanTask)) {
+ // We do not know how to combine anything but SplitScanTasks
+ combinedScans.add(fileScanTask);
+ } else {
+ SplitScanTask split = (SplitScanTask) fileScanTask;
+ if (lastSplit != null) {
+ if (lastSplit.isAdjacent(split)) {
+ // We can merge with the last split
+ lastSplit = new SplitScanTask(
+ lastSplit.offset,
+ lastSplit.len + split.len,
+ lastSplit.fileScanTask);
+ } else {
+ // We cannot merge with the last split, add it to the set of tasks
+ combinedScans.add(lastSplit);
+ lastSplit = split;
+ }
+ } else {
+ // We haven't seen another split yet, save this split in-case we can
combine it later
+ lastSplit = split;
+ }
+ }
+ }
+
+ if (lastSplit != null) {
+ combinedScans.add(lastSplit);
+ }
+
+ return combinedScans.stream().toArray(FileScanTask[]::new);
Review comment:
Since this method accepts `List<FileScanTask>`, I think it makes more
sense to return a list as well. The caller can always convert to an array if
that's appropriate.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
