[
https://issues.apache.org/jira/browse/SPARK-61580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun reassigned SPARK-61580:
-------------------------------------
Assignee: Szehon Ho
> Batch setup inserts in Catalyst runtime filter tests
> ----------------------------------------------------
>
> Key: SPARK-61580
> URL: https://issues.apache.org/jira/browse/SPARK-61580
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 5.0.0
> Reporter: Szehon Ho
> Assignee: Szehon Ho
> Priority: Major
> Labels: pull-request-available
>
> h3. Pull request
> [apache/spark#60416|https://github.com/apache/spark/pull/60416]
> h3. Motivation
> Each setup insert incurs SQL planning and execution overhead. The affected
> fixtures exercise
> Catalyst runtime filtering rather than repeated append behavior, so batching
> reduces setup from
> 68 write jobs and append commits to 14.
> h3. Coverage traceability
> * All 29 tests remain.
> * Every fixture retains the same rows and partition values.
> * All runtime-filter queries, plan checks, partition checks, result checks,
> and negative-path
> assertions remain unchanged.
> * The intentional coverage difference is 68 append operations becoming 14.
> h3. Benchmark
> Two local runs of the full suite on each commit produced:
> || Commit || Run 1 || Run 2 || Average ||
> | Parent ({{b33faf5d24b}}) | 9.177 s | 9.343 s | 9.260 s |
> | PR ({{2297095400e}}) | 8.438 s | 8.466 s | 8.452 s |
> This saves 0.808 seconds, or 8.7%, on average.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]