hsiang-c commented on PR #9335:
URL: https://github.com/apache/iceberg/pull/9335#issuecomment-1897829796
Hello @RussellSpitzer,
@szehon-ho and I had a discussion about adopting the `reachableManifests`
method.
If I understand #8856 correctly, once we associate `as_of_snapshot` to
manifest files while querying "all_*" versions of metadata tables (e.g.
`all_entries`), each (as_of_snapshot, manifest_file) entry becomes unique and
de-duplication is no longer a must.
Please let us know if that's what you're looking for, thank you.
```java
protected CloseableIterable<ManifestFile> reachableManifests(
Function<Snapshot, Iterable<ManifestFile>> toManifests) {
Iterable<Snapshot> snapshots = table().snapshots();
Iterable<Iterable<ManifestFile>> manifestIterables =
Iterables.transform(snapshots, toManifests);
try (CloseableIterable<ManifestFile> iterable =
new ParallelIterable<>(manifestIterables, planExecutor())) {
return CloseableIterable.withNoopClose(Sets.newHashSet(iterable)); //
de-dup `ManifestFile`
} catch (IOException e) {
throw new UncheckedIOException("Failed to close parallel iterable", e);
}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
