DB Tsai created SPARK-60677:
-------------------------------
Summary: Cache immutable DataFrame schemas in Spark Connect Rust
Key: SPARK-60677
URL: https://issues.apache.org/jira/browse/SPARK-60677
Project: Spark
Issue Type: Improvement
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: DB Tsai
Assignee: DB Tsai
Fix For: connect-rust-4.3.0
Repeated DataFrame.schema/columns/dtypes accesses currently perform repeated
AnalyzePlan RPCs in the Rust Python client, whereas official PySpark Connect
caches the schema on the DataFrame. The frozen benchmark reports repeated
schema access at about 0.192 ms versus 0.003 ms for Python; on a 256-operation
prepared chain Rust takes about 2.61 ms per access. Cache schema metadata
consistently for the same immutable DataFrame plan, preserve isolation for
derived plans and clones/session boundaries, and avoid hiding errors or
changing public types. Add tests proving RPC counts and cache lifecycle, and
benchmark first/repeated schema, columns, dtypes and deep-plan cases plus
unrelated guards. Baseline Rust: d884e347452c. Session version caching is
separate scope.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
