dbtsai opened a new pull request, #120: URL: https://github.com/apache/spark-connect-rust/pull/120
### What changes were proposed in this pull request? Cache successfully analyzed schemas on the Rust `DataFrame`. `schema`, `columns`, and `dtypes` share the resolved schema, so repeated metadata reads no longer serialize the plan or issue `AnalyzePlan` RPCs. Exact Rust clones share a synchronized cache, including clones created before the first analysis. Every new logical plan gets its own empty cache; `zip` now uses the common constructor. RPC failures, missing schema responses, and type decoding errors are not cached. Rust returns an independent schema value and Python continues constructing a fresh `StructType`, preserving the existing mutation behavior. Derived plans, including schema-preserving transformations, resolve independently. No session-version caching or dependency on other open PRs. ### Why are the changes needed? Repeated metadata access currently performs a full analysis RPC each time. Its cost grows with plan depth despite an immutable logical plan. Reference PySpark caches the resolved schema on a DataFrame and returns independent schema copies. Client-only loopback fixture measurements: five fresh processes per engine/case, rotating order, Apache master `d884e347452c` versus this patch: | Public operation | Baseline Rust | Candidate Rust | Schema RPCs per measured loop | |---|---:|---:|---:| | Repeated one-field schema | 184.33 µs | 0.44 µs | 50 → 0 | | Repeated columns | 183.47 µs | 0.28 µs | 50 → 0 | | Repeated dtypes | 177.32 µs | 0.36 µs | 50 → 0 | | Repeated schema, prepared depth-256 chain | 2,834.10 µs | 0.66 µs | 20 → 0 | | Repeated schema, width-512 projection / one-field response | 691.35 µs | 0.90 µs | 20 → 0 | | Repeated 512-field schema response | 447.47 µs | 48.88 µs | 50 → 0 | Wide schemas still pay for deep cloning and Python object construction. First-access schema/columns/dtypes, deep/wide first analysis, transformation, tuple/pandas input, nested collection, Arrow result, and analysis-error guards were included. Their baseline/candidate timing ranges overlap; no first-access or unrelated speedup is claimed. All 360 samples across 24 cases and three clients validated. These measure client overhead against scripted protocol fixtures, not Spark execution time. The unchanged reference was pinned PySpark 4.2; Rust baseline/candidate used identical NumPy, pandas, PyArrow and psutil versions. ### Does this PR introduce _any_ user-facing change? Yes. After the first successful schema lookup on a DataFrame, subsequent schema/columns/dtypes reads use the cached schema. As with reference PySpark, later changes to external table/view metadata do not invalidate that cached schema. New/derived DataFrames resolve independently. Mutating a returned Python schema or metadata list does not affect subsequent results. Failed lookups remain retryable. ### How was this patch tested? - `cargo fmt --all -- --check` and `git diff --check` passed. - `cargo test --locked -p apache-spark-connect --lib`: 393 passed, including exact-clone sharing and derived-plan cache ownership. - Built a release wheel using `maturin build --release --locked` and installed it into a separate candidate environment. - `python -m pytest python/tests/test_schema_cache.py -q` with grpcio installed: 10 passed. A separate-process analysis-only gRPC fixture exercises the actual Rust transport. Tests cover first access through each accessor, repeated zero-RPC reads, independent StructType/list results, changed server schemas on new/derived plans, session isolation, concurrent first access, RPC errors, absent results, missing schemas and malformed types. - The new repeated-access test fails on the unchanged baseline: 10 RPCs instead of one. - Existing `python/tests/test_dropin_offline.py`: candidate and baseline both passed 273 tests and failed the same `test_udt_protocol_roundtrip` with `ModuleNotFoundError: tests.test_dropin_offline` under the installed-wheel invocation. - Frozen public workload harness: five fresh samples per engine/case, rotating order, actual RPC/byte counters, raw wall/CPU timings and externally sampled RSS. Heavy work was serialized through the shared CPU lock. Reference/baseline environments were unchanged. No live Spark server validation. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: OpenAI Codex (GPT-6) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
