[
https://issues.apache.org/jira/browse/SPARK-61580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Szehon Ho updated SPARK-61580:
------------------------------
Description:
h3. Pull request
[apache/spark#60416|https://github.com/apache/spark/pull/60416]
h3. Motivation
Each setup insert incurs SQL planning and execution overhead. The affected
fixtures exercise
Catalyst runtime filtering rather than repeated append behavior, so batching
reduces setup from
68 write jobs and append commits to 14.
h3. Coverage traceability
* All 29 tests remain.
* Every fixture retains the same rows and partition values.
* All runtime-filter queries, plan checks, partition checks, result checks, and
negative-path
assertions remain unchanged.
* The intentional coverage difference is 68 append operations becoming 14.
h3. Benchmark
Two local runs of the full suite on each commit produced:
|| Commit || Run 1 || Run 2 || Average ||
| Parent ({{b33faf5d24b}}) | 9.177 s | 9.343 s | 9.260 s |
| PR ({{2297095400e}}) | 8.438 s | 8.466 s | 8.452 s |
This saves 0.808 seconds, or 8.7%, on average.
> Batch setup inserts in Catalyst runtime filter tests
> ----------------------------------------------------
>
> Key: SPARK-61580
> URL: https://issues.apache.org/jira/browse/SPARK-61580
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 5.0.0
> Reporter: Szehon Ho
> Priority: Major
> Labels: pull-request-available
>
> h3. Pull request
> [apache/spark#60416|https://github.com/apache/spark/pull/60416]
> h3. Motivation
> Each setup insert incurs SQL planning and execution overhead. The affected
> fixtures exercise
> Catalyst runtime filtering rather than repeated append behavior, so batching
> reduces setup from
> 68 write jobs and append commits to 14.
> h3. Coverage traceability
> * All 29 tests remain.
> * Every fixture retains the same rows and partition values.
> * All runtime-filter queries, plan checks, partition checks, result checks,
> and negative-path
> assertions remain unchanged.
> * The intentional coverage difference is 68 append operations becoming 14.
> h3. Benchmark
> Two local runs of the full suite on each commit produced:
> || Commit || Run 1 || Run 2 || Average ||
> | Parent ({{b33faf5d24b}}) | 9.177 s | 9.343 s | 9.260 s |
> | PR ({{2297095400e}}) | 8.438 s | 8.466 s | 8.452 s |
> This saves 0.808 seconds, or 8.7%, on average.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]