abstractdog opened a new pull request, #27580: URL: https://github.com/apache/datafusion/pull/27580
## Which issue does this PR close? - Closes #27579. ## Rationale for this change This change is needed before [Ballista](https://github.com/apache/datafusion-ballista) executors can keep a **persistent** file metadata cache. Today a Ballista executor builds a fresh `RuntimeEnv`, with its own cache, for every session. Every new query therefore re-reads every Parquet footer and page index from object storage. At TPC-DS scale factor 10000 that is tens of thousands of footer reads per query. Ballista wants each executor to keep one `FileMetadataCache` for its whole lifetime and share it between the runtimes of all sessions, through `CacheManagerConfig::with_file_metadata_cache`. Footers cached by one query would then serve the next. An executor serves many sessions, and those sessions can read different buckets and endpoints. The cache key must therefore identify the object store, and today it does not. `FileMetadataCache` is keyed by the file's `object_store::path::Path`. That path is only unique within one object store. A cached entry is only checked against the file's size and last-modified time, so two files with the same in-store path, size and modification time in different stores share an entry. For example, `s3://bucket-a/t/x.parquet` and `s3://bucket-b/t/x.parquet` both have the path `t/x.parquet`. The second file read is then served the first file's metadata: its schema, statistics and row-group offsets. Depending on the data, that leads to wrong row-group pruning (silently wrong results) or decode errors. The collision already exists on `main` within a single `RuntimeEnv`, because one cache is shared by all its stores. It can be reproduced there: - write two different single-row Parquet files of equal size to two `InMemory` stores at the same path; - give them the same `last_modified`; - read both through `DFParquetMetadata` with one cache. Store B's read returns store A's statistics (`min = 1` instead of `2`). Inside one session this needs one query to read two stores with matching layouts, so it is rare. An executor-lifetime cache turns it into a cross-session, cross-user problem that lasts as long as the executor runs. For example, two users querying copies of the same table in different buckets, or per-environment buckets with the same layout, would be served each other's footers. A persistent cache is only safe once the key includes the store. ## What changes are included in this PR? - **New cache key.** `FileMetadataCacheKey` (in `datafusion-execution`) holds the store's `ObjectStoreUrl` (scheme and authority, e.g. `s3://bucket-a`) and the path. `FileMetadataCache` is now `dyn Cache<FileMetadataCacheKey, CachedFileMetadataEntry>`. The key contains nothing format-specific. - Credentials are removed from the URL, so they never appear in cache listings. - The key is a plain value, so runtimes built separately for the same store, such as one per session in a Ballista executor, produce equal keys and share hits through a shared cache. - **Scans pass the URL from their config.** `FileScanConfig::object_store_url` goes through `CachedParquetFileReaderFactory::new` (new second argument) and `ParquetFileReader::with_object_store_url` to `DFParquetMetadata::with_object_store_url`. Scans that are serialized and deserialized, as in Ballista, carry the URL in their config, so the executor's scan path always has it. - **`FileFormat` methods look the URL up.** `infer_schema`, `infer_stats` and the ordering methods only receive an `ObjectStore`, so they ask the registry for its URL with a new provided method, `ObjectStoreRegistry::get_store_url`, the reverse of `get_store`. - Its default returns `None`. `DefaultObjectStoreRegistry` implements it by matching the registered `Arc`; the pointer is only used for that lookup and is never part of the key. - Doing it this way keeps the `FileFormat` trait signatures unchanged. - **No URL means no caching.** When the URL is unknown, `DFParquetMetadata` skips the cache for that read. That can cost a cache hit, but never returns another file's entry. - **datafusion-cli.** The instrumented object store registry forwards `get_store_url`, and the `metadata_cache()` table function gains a first column, `object_store_url`. - **Docs.** An upgrade guide entry in `upgrading/57.0.0.md`, and the CLI functions docs are updated. ## What is the testing strategy for this PR? New unit tests: - `default_cache.rs`: two files with the same path, size and modification time in two stores get separate entries and never serve each other; the same store still hits; keys with and without credentials are equal. - `metadata.rs`: `DFParquetMetadata` caches per store, and does not cache without an object store URL. - `object_store.rs`: `DefaultObjectStoreRegistry::get_store_url` returns the registered URL, `None` for unregistered stores, and maps a store from `get_store` back to its URL. A second test covers removing credentials from `ObjectStoreUrl`. Existing tests: - Six tests in `datafusion/core/src/datasource/file_format/parquet.rs` count object store requests through an unregistered store. They now register the store or set the URL, and still assert that the cache hits. - `datafusion-cli`'s `metadata_cache()` snapshot test is updated for the new column. Results: - `cargo test -p datafusion-execution` (125 passed) and `-p datafusion-datasource-parquet` (315 passed) pass, as do the parquet file-format tests in `datafusion` (31 passed). - Clippy is clean with `-D warnings` on the changed crates. ## Are there any user-facing changes? Yes, these are breaking public API changes (`api change`), described in the 57.0.0 upgrade guide: - `FileMetadataCache` is keyed by `FileMetadataCacheKey` instead of `Path`. Custom `FileMetadataCache` implementations and code calling `get`, `put` or `list_entries` need to use the new key. - `CachedParquetFileReaderFactory::new` takes the store's `ObjectStoreUrl` as a second argument. - `DFParquetMetadata` and `ParquetFileReader` only use the file metadata cache when `with_object_store_url` is set as well. Callers that set only the cache now read the footer from storage on every call, without an error. - `ObjectStoreRegistry::get_store_url` is a new provided method. Custom registries should implement it, or forward it to the registry they wrap. Otherwise metadata read while inferring schemas and statistics is no longer cached; scans are unaffected. Ballista will forward it from its registry when it adopts the shared executor cache. - `datafusion-cli`'s `metadata_cache()` has a new first column, `object_store_url`. One limitation remains: two different endpoints registered under the same `ObjectStoreUrl` in runtimes that share one cache can still collide. The URL is what `RuntimeEnv` uses to select a store, so it is the identity this PR keys on. A deployment that gives one executor several endpoints for the same bucket name would need to register them under distinct URLs. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
