[
https://issues.apache.org/jira/browse/SPARK-61581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Szehon Ho updated SPARK-61581:
------------------------------
Description:
h3. Pull request
[apache/spark#60408|https://github.com/apache/spark/pull/60408]
h3. Motivation
Three runtime-filter tests used ten separate single-row inserts only to prepare
their fixtures.
Batching each fixture into one multi-row insert reduces setup from 30 insert
jobs and append
commits to three.
h3. Coverage traceability
* All 116 suite tests remain.
* Each fixture retains the same rows and partition values. The in-memory table
still groups them
into the same ten single-row partitions in each affected test.
* All runtime-filter, result, and partition-count assertions remain unchanged.
* The intentional coverage difference is 30 append operations becoming three.
Other tests
continue to cover repeated appends to partitioned in-memory tables.
h3. Benchmark
Two local runs of the full suite on each commit produced:
|| Commit || Run 1 || Run 2 || Average ||
| Parent ({{0d52f01d106}}) | 17.740 s | 17.569 s | 17.655 s |
| PR ({{55000f019f3}}) | 17.547 s | 17.767 s | 17.657 s |
The 0.003-second (0.01%) difference is within run-to-run noise, so the
reduction from 30 to three
insert jobs does not produce a measurable change in full-suite wall time.
> Batch setup inserts in DSv2 runtime filter tests
> ------------------------------------------------
>
> Key: SPARK-61581
> URL: https://issues.apache.org/jira/browse/SPARK-61581
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 5.0.0
> Reporter: Szehon Ho
> Priority: Major
> Labels: pull-request-available
>
> h3. Pull request
> [apache/spark#60408|https://github.com/apache/spark/pull/60408]
> h3. Motivation
> Three runtime-filter tests used ten separate single-row inserts only to
> prepare their fixtures.
> Batching each fixture into one multi-row insert reduces setup from 30 insert
> jobs and append
> commits to three.
> h3. Coverage traceability
> * All 116 suite tests remain.
> * Each fixture retains the same rows and partition values. The in-memory
> table still groups them
> into the same ten single-row partitions in each affected test.
> * All runtime-filter, result, and partition-count assertions remain unchanged.
> * The intentional coverage difference is 30 append operations becoming three.
> Other tests
> continue to cover repeated appends to partitioned in-memory tables.
> h3. Benchmark
> Two local runs of the full suite on each commit produced:
> || Commit || Run 1 || Run 2 || Average ||
> | Parent ({{0d52f01d106}}) | 17.740 s | 17.569 s | 17.655 s |
> | PR ({{55000f019f3}}) | 17.547 s | 17.767 s | 17.657 s |
> The 0.003-second (0.01%) difference is within run-to-run noise, so the
> reduction from 30 to three
> insert jobs does not produce a measurable change in full-suite wall time.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]