abstractdog opened a new pull request, #27580:
URL: https://github.com/apache/datafusion/pull/27580

   ## Which issue does this PR close?
   
   - Closes #27579.
   
   ## Rationale for this change
   
   This change is needed before 
[Ballista](https://github.com/apache/datafusion-ballista) executors can keep a 
**persistent** file metadata cache. Today a Ballista executor builds a fresh 
`RuntimeEnv`, with its own cache, for every session. Every new query therefore 
re-reads every Parquet footer and page index from object storage. At TPC-DS 
scale factor 10000 that is tens of thousands of footer reads per query.
   
   Ballista wants each executor to keep one `FileMetadataCache` for its whole 
lifetime and share it between the runtimes of all sessions, through 
`CacheManagerConfig::with_file_metadata_cache`. Footers cached by one query 
would then serve the next. An executor serves many sessions, and those sessions 
can read different buckets and endpoints. The cache key must therefore identify 
the object store, and today it does not.
   
   `FileMetadataCache` is keyed by the file's `object_store::path::Path`. That 
path is only unique within one object store. A cached entry is only checked 
against the file's size and last-modified time, so two files with the same 
in-store path, size and modification time in different stores share an entry. 
For example, `s3://bucket-a/t/x.parquet` and `s3://bucket-b/t/x.parquet` both 
have the path `t/x.parquet`. The second file read is then served the first 
file's metadata: its schema, statistics and row-group offsets. Depending on the 
data, that leads to wrong row-group pruning (silently wrong results) or decode 
errors.
   
   The collision already exists on `main` within a single `RuntimeEnv`, because 
one cache is shared by all its stores. It can be reproduced there:
   - write two different single-row Parquet files of equal size to two 
`InMemory` stores at the same path;
   - give them the same `last_modified`;
   - read both through `DFParquetMetadata` with one cache.
   
   Store B's read returns store A's statistics (`min = 1` instead of `2`).
   
   Inside one session this needs one query to read two stores with matching 
layouts, so it is rare. An executor-lifetime cache turns it into a 
cross-session, cross-user problem that lasts as long as the executor runs. For 
example, two users querying copies of the same table in different buckets, or 
per-environment buckets with the same layout, would be served each other's 
footers. A persistent cache is only safe once the key includes the store.
   
   ## What changes are included in this PR?
   
   - **New cache key.** `FileMetadataCacheKey` (in `datafusion-execution`) 
holds the store's `ObjectStoreUrl` (scheme and authority, e.g. `s3://bucket-a`) 
and the path. `FileMetadataCache` is now `dyn Cache<FileMetadataCacheKey, 
CachedFileMetadataEntry>`. The key contains nothing format-specific.
     - Credentials are removed from the URL, so they never appear in cache 
listings.
     - The key is a plain value, so runtimes built separately for the same 
store, such as one per session in a Ballista executor, produce equal keys and 
share hits through a shared cache.
   - **Scans pass the URL from their config.** 
`FileScanConfig::object_store_url` goes through 
`CachedParquetFileReaderFactory::new` (new second argument) and 
`ParquetFileReader::with_object_store_url` to 
`DFParquetMetadata::with_object_store_url`. Scans that are serialized and 
deserialized, as in Ballista, carry the URL in their config, so the executor's 
scan path always has it.
   - **`FileFormat` methods look the URL up.** `infer_schema`, `infer_stats` 
and the ordering methods only receive an `ObjectStore`, so they ask the 
registry for its URL with a new provided method, 
`ObjectStoreRegistry::get_store_url`, the reverse of `get_store`.
     - Its default returns `None`. `DefaultObjectStoreRegistry` implements it 
by matching the registered `Arc`; the pointer is only used for that lookup and 
is never part of the key.
     - Doing it this way keeps the `FileFormat` trait signatures unchanged.
   - **No URL means no caching.** When the URL is unknown, `DFParquetMetadata` 
skips the cache for that read. That can cost a cache hit, but never returns 
another file's entry.
   - **datafusion-cli.** The instrumented object store registry forwards 
`get_store_url`, and the `metadata_cache()` table function gains a first 
column, `object_store_url`.
   - **Docs.** An upgrade guide entry in `upgrading/57.0.0.md`, and the CLI 
functions docs are updated.
   
   ## What is the testing strategy for this PR?
   
   New unit tests:
   - `default_cache.rs`: two files with the same path, size and modification 
time in two stores get separate entries and never serve each other; the same 
store still hits; keys with and without credentials are equal.
   - `metadata.rs`: `DFParquetMetadata` caches per store, and does not cache 
without an object store URL.
   - `object_store.rs`: `DefaultObjectStoreRegistry::get_store_url` returns the 
registered URL, `None` for unregistered stores, and maps a store from 
`get_store` back to its URL. A second test covers removing credentials from 
`ObjectStoreUrl`.
   
   Existing tests:
   - Six tests in `datafusion/core/src/datasource/file_format/parquet.rs` count 
object store requests through an unregistered store. They now register the 
store or set the URL, and still assert that the cache hits.
   - `datafusion-cli`'s `metadata_cache()` snapshot test is updated for the new 
column.
   
   Results:
   - `cargo test -p datafusion-execution` (125 passed) and `-p 
datafusion-datasource-parquet` (315 passed) pass, as do the parquet file-format 
tests in `datafusion` (31 passed).
   - Clippy is clean with `-D warnings` on the changed crates.
   
   ## Are there any user-facing changes?
   
   Yes, these are breaking public API changes (`api change`), described in the 
57.0.0 upgrade guide:
   
   - `FileMetadataCache` is keyed by `FileMetadataCacheKey` instead of `Path`. 
Custom `FileMetadataCache` implementations and code calling `get`, `put` or 
`list_entries` need to use the new key.
   - `CachedParquetFileReaderFactory::new` takes the store's `ObjectStoreUrl` 
as a second argument.
   - `DFParquetMetadata` and `ParquetFileReader` only use the file metadata 
cache when `with_object_store_url` is set as well. Callers that set only the 
cache now read the footer from storage on every call, without an error.
   - `ObjectStoreRegistry::get_store_url` is a new provided method. Custom 
registries should implement it, or forward it to the registry they wrap. 
Otherwise metadata read while inferring schemas and statistics is no longer 
cached; scans are unaffected. Ballista will forward it from its registry when 
it adopts the shared executor cache.
   - `datafusion-cli`'s `metadata_cache()` has a new first column, 
`object_store_url`.
   
   One limitation remains: two different endpoints registered under the same 
`ObjectStoreUrl` in runtimes that share one cache can still collide. The URL is 
what `RuntimeEnv` uses to select a store, so it is the identity this PR keys 
on. A deployment that gives one executor several endpoints for the same bucket 
name would need to register them under distinct URLs.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to