rdblue commented on code in PR #18021:
URL: https://github.com/apache/iceberg/pull/18021#discussion_r3983371356
##########
core/src/main/java/org/apache/iceberg/V4ManifestReader.java:
##########
@@ -123,10 +132,13 @@ private void incrementSkipCount(FileContent content) {
}
private CloseableIterable<TrackedFile> open() {
+ InputFile file = ManifestFiles.newInputFile(io, manifest);
Review Comment:
I think we need one more change here. I checked with Dan and Russell about
the caching that this call triggers and we think that it's a good idea to
bypass it for now. We don't think that this caching is widely used because
there are many cases where the JVM isn't shared and caching just wastes memory.
In addition, we think that there are a couple of reasons why it is going to be
less useful. First, the recent work to read small files into memory is going to
reduce seeks by reading the entire file. And second, the design of v4 means we
have a lot fewer small manifests to worry about.
I think for now it's best if we bypass the cache.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
