dossett opened a new pull request, #3674: URL: https://github.com/apache/parquet-java/pull/3674
Parquet's client-side split planning already obtains a `FileStatus` for every input while listing files. It later discards that status and calls `getFileStatus()` again for each file before calculating block-aware splits. That adds one sequential NameNode RPC or remote metadata lookup per file, increasing job-startup latency and metadata-service load without changing the result. This change carries the status obtained during listing internally through footer processing and reuses it during split creation. It leaves `Footer`'s public API unchanged, does not retain `FileStatus` in the footer cache, and preserves the existing lookup as a fallback for caller-created footers. The optimized path covers direct footer reads, footer-cache hits, summary metadata, and deprecated split planning. This addresses the same underlying problem as #3395 while covering the cache and summary-file paths and avoiding a public nullable `FileStatus` API. Closes #3394. ## Implementation - Associate footer results with the `FileStatus` objects already supplied to footer reading using a private internal carrier. - Attach the current listed status when returning a cached footer, while storing only ordinary footer metadata in the cache. - Reuse the associated status during split generation and fall back to `getFileStatus()` when none is available. ## Testing - `mvn -pl parquet-hadoop -Dtest=TestInputFormat,TestInputFormatFileStatusReuse,DeprecatedInputFormatTest,org.apache.parquet.hadoop.example.TestInputOutputFormat test` (35 tests) - `mvn -pl parquet-hadoop test` (740 tests, 24 skipped) - `mvn -pl parquet-hadoop -DskipTests spotless:check` ## Development provenance This change was developed with assistance from Codex. I reviewed the implementation, tests, and conclusions and own the change. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
