autoCAT6 opened a new pull request, #26552: URL: https://github.com/apache/datafusion/pull/26552
## Which issue does this PR close? Closes #25662. ## Rationale for this change External catalogs and table providers need to reuse credentials and allow applications to supply them outside SQL. Passing each credential through connection options exposes values to statement and plan formatting and leaves each integration to manage its own store. ## What changes are included in this PR? - Add a pluggable asynchronous `SecretsStore`, an atomic in-memory default, and session/builder access for providers and their factories. - Add `CREATE [OR REPLACE] SECRET [IF NOT EXISTS]` and `DROP SECRET [IF EXISTS]`, using the same store as programmatic operations. Secrets are generic string option collections; providers interpret their fields and named references. - Redact secret statement/plan formatting, reject unsupported native secret wrappers without printing values, and refuse to serialize credential-bearing CREATE SECRET DDL. - Add a runnable two-provider example and binary/JSON connection-plan tests. The receiver resolves a name against its own store; the producer's credential is not transported with the plan. ## What is the testing strategy for this PR? Added 27 Rust tests plus `secrets.slt`, covering the store, SQL parser/planner, session execution, and protobuf/JSON connection-plan boundaries. Validation on Rust 1.99.1: - Extended workspace lib/tests/bins: **12,817 passed, 8 ignored**; **537/537** SQL logic files completed. Excluded `datafusion-examples`, `datafusion-benchmarks`, and `datafusion-cli`, as in the contributor instructions; enabled `avro,json,backtrace,extended_tests,recursive_protection,parquet_encryption`. - Focused SQL/store/serialization tests, six public-API provider tests, and the binary/JSON reference roundtrip passed. The six store unit tests were rerun against the final test source. - `cargo run --locked --profile ci -p datafusion-examples --example data_io -- secrets` passed. - `cargo fmt --all -- --check` and `cargo clippy --locked --profile ci --all-targets --all-features --keep-going -- -D warnings` passed. - Generated the example index and ran the pinned Markdown formatter; TOML, spelling, workflow, file-size and unused-dependency checks passed. Existing `sql_planner` benchmarks compared against `f9bc5a3fc60cc7f4319cd2c156cb3d024a8c1b21`, with independent target directories and the normal optimized release/LTO profile. Initial baseline/fixed/baseline/fixed runs used 20 samples, 1 s warm-up and 3 s measurement per case. Times below are microseconds per iteration; TPC-H is 21 schema-only logical plans. | Case | Baseline runs (µs) | New implementation runs (µs) | | --- | --- | --- | | `logical_select_one_from_700` | 130.053 / 126.554 | 121.485 / 125.088 | | `logical_wide_filter_200_predicates` | 784.394 / 703.979 | 915.057 / 732.920 | | `logical_plan_tpch_all` | 2324.694 / 2332.044 | 2276.311 / 2355.350 | The initial wide-filter difference warranted a longer check with the same binaries, 5 s warm-up, 8 s measurement and 30 samples. Other task builds were paused for both sets of measurements: | Order | Variant | Point estimate (µs) | Criterion CI (µs) | | --- | --- | --- | --- | | 1 | New | 684.074 | 681.104–687.737 | | 2 | Baseline | 688.689 | 683.534–696.559 | | 3 | Baseline | 678.172 | 675.996–680.448 | | 4 | New | 685.348 | 682.462–688.701 | The initial large gap did not reproduce; this local check establishes neither a performance improvement nor universal absence of regressions. Compiled production sources were unchanged during measurement; subsequent edits to the session unit-test iterator and a separate protobuf integration test were excluded from this benchmark target. Tests include duplicate/replacement/removal semantics, concurrent creates/removes, session sharing/isolation, custom stores, wrong/missing credentials, DDL restrictions, canary redaction, native parser compatibility, and reference-only plan roundtrips. ## Are there any user-facing changes? Yes: new session APIs and CREATE/DROP SECRET SQL, documented with an example and migration notes. `DdlStatement` and SQL parser `Statement` gain `CreateSecret` and `DropSecret` variants; downstream exhaustive matches must handle them. The `Session` trait method has a default implementation so existing implementors continue to compile. The default store is session-scoped plaintext memory; persistence, authorization, external secret-manager integrations, and automatic object-store configuration are outside this initial implementation. Names/references remain visible, raw SQL can still be retained by callers, and removing a secret does not revoke existing provider connections or snapshots. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
