[
https://issues.apache.org/jira/browse/SPARK-60698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-60698:
-----------------------------------
Labels: pull-request-available (was: )
> Batch Arrow input normalization configuration reads in Spark Connect Rust
> -------------------------------------------------------------------------
>
> Key: SPARK-60698
> URL: https://issues.apache.org/jira/browse/SPARK-60698
> Project: Spark
> Issue Type: Improvement
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: DB Tsai
> Assignee: DB Tsai
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.3.0
>
>
> Follow-up to Arrow input PR118/SPARK-60675. Nonempty PyArrow/pandas columnar
> input currently reads spark.sql.timestampType, spark.sql.session.timeZone and
> spark.sql.execution.arrow.useLargeVarTypes in three sequential Config RPCs,
> then reads five local-relation transport options in a fourth already-batched
> RPC. Frozen combined benchmark input-cache1MiB-30000x3-construct shows4Config
> calls/no artifact calls and8.574ms vsPython2.243ms; general10kx3 pandas3.44ms
> vsPython1.28ms. Batch the three normalization reads using existing core
> get_configs support, retaining fresh reads on every construction and correct
> missing/error semantics; do not cache mutable session settings or redesign
> transport. Measure paired candidate/baseline sharing identical PR118
> dependency, unchanged frozen benchmark, IPC correctness and
> changed-config/error/retry behavior. Target normalization ConfigRPC3to1 and
> total4to2 on nonempty input, zero empty-input RPCs. No JVM.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
