dbtsai opened a new pull request, #17236: URL: https://github.com/apache/iceberg/pull/17236
## Problem FileIO input streams (`S3InputStream`, `GCSInputStream`, `ADLSInputStream`, and the GCS analytics-core wrapper) only incremented `READ_BYTES`/`READ_OPERATIONS` in the sequential `read()` paths. The positioned-read methods `readFully()`/`readTail()` — and the vectored path that Parquet layers on top of them since 1.11.0 via `ParquetRangeReadableInputStreamAdapter` — bypassed instrumentation entirely. Because `RangeReadable.readVectored()`'s default implementation loops over `readFully()`, and Parquet now routes most column-chunk data through vectored reads, nearly all Parquet data bytes disappeared from the read metrics that Spark consumes for task input metrics. This under-reports Hadoop read metrics significantly. Closes #17208 ## Fix Instrument `readFully()`/`readTail()` in all three object-store streams (`S3InputStream`, `ADLSInputStream`, `GCSInputStream`). The default `RangeReadable.readVectored()` loops over `readFully()`, so the vectored Parquet path is covered automatically for S3/ADLS/GCS. The GCS analytics-core wrapper (`AnalyticsCoreUtil`) overrides `readVectored()` and delegates to the analytics-core stream, so it is instrumented explicitly — recording bytes per range as each range future completes successfully (failed ranges are not counted). Empty-tail reads that return `-1` are guarded so they never decrement the byte counter. Additionally, the S3 analytics-accelerator path (`AnalyticsAcceleratorInputStreamWrapper`, used when `s3.analytics-accelerator` is enabled) previously tracked **no** read metrics at all. It is now threaded the `MetricsContext` already available on `S3InputFile` and increments the counters from its `read()` methods; it is not `RangeReadable`, so Parquet reads flow through `read(byte[], int, int)` and counting there captures all data bytes. ## Tests - `TestS3InputStream` — `testReadFullyTracksMetrics`, `testReadTailTracksMetrics` - `TestADLSInputStream` — `testReadFullyTracksMetrics`, `testReadTailTracksMetrics` - `TestGCSInputStream` — `testRangeReadMetrics`, `testReadTailEmptyObjectDoesNotDecrementMetrics` - `TestAnalyticsCoreUtil` — `readVectoredCountsOnlyCompletedRanges` (no eager counting; successful ranges counted; failed ranges not counted) - `TestAnalyticsAcceleratorInputStreamWrapper` — `testReadTracksMetrics` ## Scope notes - `OSSInputStream` (Aliyun) is unaffected today because it does not implement `RangeReadable`; it would need the same treatment if `RangeReadable` support is added there. - This covers the FileIO streams called out in the issue plus the accelerated S3/GCS paths; it is not a full audit of every custom input-stream wrapper. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
