limadog9 opened a new pull request, #25858: URL: https://github.com/apache/datafusion/pull/25858
## Which issue does this PR close? Closes #25841. ## Rationale for this change An overwrite can keep the same file size and modification time while changing its statistics and Parquet footer. Different object stores can also contain the same relative path with identical metadata. Reusing either cache entry can produce incorrect query results. ## What changes are included in this PR? - Validate ETags and versions independently when both cached and current metadata provide them, retaining the existing size, modification time, and schema checks. - Include the object store URL in statistics, listing, and Parquet metadata cache keys, and carry it through schema inference, statistics inference, and execution readers. - Keep anonymous listing-cache invalidation within the affected object store. - Document the affected public APIs in the 56.0.0 upgrade guide. - Remove a redundant semicolon flagged by strict Windows linting. Validation still depends on current object metadata; cached listings and changes between planning and execution retain the consistency limitations described in the issue. ## What is the testing strategy for this PR? The regression tests cover all combinations of matching, differing, and unavailable ETags/versions; equal-size overwrites at an unchanged timestamp; and separate stores with identical object metadata. DataFrame tests assert correct aggregates and filtered scans, while cache tests verify isolation, reuse, and listing invalidation. Passed locally: - `cargo test --profile ci -p datafusion-execution --lib cache::` (43 tests) - `cargo test --profile ci -p datafusion-datasource --lib cache` (5 tests) - `cargo test --profile ci -p datafusion-datasource-parquet --lib cache` (4 tests) - `cargo test --profile ci -p datafusion --test core_integration datasource::` (26 tests) - `cargo fmt --all -- --check` - `./ci/scripts/doc_prettier_check.sh --write --allow-dirty` - Strict Clippy (`--lib --tests --all-features -- -D warnings`) for `datafusion`, `datafusion-execution`, `datafusion-catalog-listing`, and all six data-source crates. ## Are there any user-facing changes? File caches no longer reuse entries from another object store or from a changed ETag/version when fresh metadata is available. API changes: `FileFormat` inference methods now take an object store URL; `TableScopedPath` includes it; `FileMetadataCache` uses `ObjectStorePath`; and the cached Parquet reader factory/cache builders require the URL. See `docs/source/library-user-guide/upgrading/56.0.0.md` for migration details. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
