DB Tsai created SPARK-60698:
-------------------------------
Summary: Batch Arrow input normalization configuration reads in
Spark Connect Rust
Key: SPARK-60698
URL: https://issues.apache.org/jira/browse/SPARK-60698
Project: Spark
Issue Type: Improvement
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: DB Tsai
Assignee: DB Tsai
Fix For: connect-rust-4.3.0
Follow-up to Arrow input PR118/SPARK-60675. Nonempty PyArrow/pandas columnar
input currently reads spark.sql.timestampType, spark.sql.session.timeZone and
spark.sql.execution.arrow.useLargeVarTypes in three sequential Config RPCs,
then reads five local-relation transport options in a fourth already-batched
RPC. Frozen combined benchmark input-cache1MiB-30000x3-construct shows4Config
calls/no artifact calls and8.574ms vsPython2.243ms; general10kx3 pandas3.44ms
vsPython1.28ms. Batch the three normalization reads using existing core
get_configs support, retaining fresh reads on every construction and correct
missing/error semantics; do not cache mutable session settings or redesign
transport. Measure paired candidate/baseline sharing identical PR118
dependency, unchanged frozen benchmark, IPC correctness and
changed-config/error/retry behavior. Target normalization ConfigRPC3to1 and
total4to2 on nonempty input, zero empty-input RPCs. No JVM.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
