Hyukjin Kwon created SPARK-59167:
------------------------------------
Summary: Add an end-to-end Spark Connect client benchmark harness
(Rust drop-in vs reference PySpark)
Key: SPARK-59167
URL: https://issues.apache.org/jira/browse/SPARK-59167
Project: Spark
Issue Type: Improvement
Components: Connect
Affects Versions: 5.0.0
Reporter: Hyukjin Kwon
Add a reproducible end-to-end benchmark harness that compares the Rust-native
Spark Connect client (pyspark-client-rust) against the reference Python Spark
Connect client (pyspark-client). Both clients drive the same Spark Connect
server, so the only variable is the client implementation (plan building, gRPC
transport, Arrow IPC decoding, and Python result construction).
scripts/benchmark_suite.py measures, across multiple dimensions:
- transport payload-size sweep (64 KB - 16 MB)
- row-count sweep
- concurrent client-process counts
- UDF complexity classes (identity, arithmetic, strings, expensive pure Python,
NumPy, pandas_udf)
- cold-start latency
reporting rows/sec, GB/sec, CPU/GB, p50/p99/p999 latency, client CPU (Rust vs
Python), a local server-JVM CPU proxy, and peak RSS. Results are written as
JSON.
scripts/bench_report.py renders a Markdown comparison table from two result
JSON files.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
