DB Tsai created SPARK-60683:
-------------------------------
Summary: Defer SQL function error imports in Spark Connect Rust
Key: SPARK-60683
URL: https://issues.apache.org/jira/browse/SPARK-60683
Project: Spark
Issue Type: Improvement
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: DB Tsai
Assignee: DB Tsai
Fix For: connect-rust-4.3.0
Fresh whole-plan construction is dominated by the first pyspark.sql.functions
import. A local cProfile of the frozen Rust master attributes nearly all
first-call time to importing pyspark.errors and its JSON error catalog, reached
by the module-level higher-order function PySparkValueError import. Ordinary
warm union construction takes microseconds. Defer error-only imports in the
custom SQL functions facade while preserving higher-order function exception
classes, conditions and messages. Validate unchanged fresh-process construction
benchmarks and both valid and invalid higher-order function guards. Do not
alter vendored error modules or simply move the import cost to startup.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
