DB Tsai created SPARK-60742:
-------------------------------
Summary: Build SQL function exports in linear time in Spark
Connect Rust
Key: SPARK-60742
URL: https://issues.apache.org/jira/browse/SPARK-60742
Project: Spark
Issue Type: Improvement
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: DB Tsai
Fix For: connect-rust-4.3.0
pyspark.sql.functions constructs __all__ by checking callable names against a
growing list. The current facade exports 568 unique names, making this
bookkeeping quadratic in the number of exported names.
Use a membership set while retaining the public list, exact export order,
callable/alias identities, signatures and
functions.builtin/functions.partitioning compatibility. Keep the change
confined to export construction. SPARK-60683 / PR126 already defers
SQL-function error imports; its export loop remains unchanged, so this is a
separate improvement.
Read-only attribution on the current published-change stack measured about
0.650 ms for the unchanged export assignment/loop across five warm batches of
100 iterations. Fresh import-only samples after library prerequisites were
1.820–2.433 ms. These measurements have different boundaries from whole-plan
benchmarks and do not establish a candidate speedup.
Validate matched, frozen baseline/candidate environments and the current
combined stack. Preserve all 568 exports in the same list order, builtin's
separate export list, aliases, generated signatures, star imports and
serialized plans. Compare the unchanged cold-plan benchmark and warm guards
with correctness/RPC validators, and report source/wheel provenance and
cold-import boundaries. No JVM is required for these client/mock measurements.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
