[ 
https://issues.apache.org/jira/browse/SPARK-61578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Szehon Ho updated SPARK-61578:
------------------------------
    Description: 
h3. Pull request

[apache/spark#60407|https://github.com/apache/spark/pull/60407]

h3. Motivation

{{checkAnswer(actual, expectedDataFrame)}} collects the expected DataFrame 
before delegating to
the row-based result check. In this suite that DataFrame is also the input 
already executed by
the write command, so collecting it again duplicates execution without adding 
an assertion.

h3. Coverage traceability

* All 99 test cases and configuration combinations remain.
* The write-source execution remains.
* The written-table result comparison remains, using expected rows derived from 
the same input
  tuples.
* The actual table's {{materializedRdd}} deserialization check performed by 
{{checkAnswer}}
  remains.
* The only removed execution is the redundant collection of the expected 
DataFrame.

h3. Benchmark

Two local runs of the full suite on each commit produced:

|| Commit || Run 1 || Run 2 || Average ||
| Parent ({{0d52f01d106}}) | 47.601 s | 48.217 s | 47.909 s |
| PR ({{19944049a7a}}) | 44.982 s | 44.947 s | 44.965 s |

This saves 2.945 seconds, or 6.1%, on average.


> Avoid collecting expected DataFrames in DSv2 write tests
> --------------------------------------------------------
>
>                 Key: SPARK-61578
>                 URL: https://issues.apache.org/jira/browse/SPARK-61578
>             Project: Spark
>          Issue Type: Improvement
>          Components: SQL
>    Affects Versions: 5.0.0
>            Reporter: Szehon Ho
>            Priority: Major
>              Labels: pull-request-available
>
> h3. Pull request
> [apache/spark#60407|https://github.com/apache/spark/pull/60407]
> h3. Motivation
> {{checkAnswer(actual, expectedDataFrame)}} collects the expected DataFrame 
> before delegating to
> the row-based result check. In this suite that DataFrame is also the input 
> already executed by
> the write command, so collecting it again duplicates execution without adding 
> an assertion.
> h3. Coverage traceability
> * All 99 test cases and configuration combinations remain.
> * The write-source execution remains.
> * The written-table result comparison remains, using expected rows derived 
> from the same input
>   tuples.
> * The actual table's {{materializedRdd}} deserialization check performed by 
> {{checkAnswer}}
>   remains.
> * The only removed execution is the redundant collection of the expected 
> DataFrame.
> h3. Benchmark
> Two local runs of the full suite on each commit produced:
> || Commit || Run 1 || Run 2 || Average ||
> | Parent ({{0d52f01d106}}) | 47.601 s | 48.217 s | 47.909 s |
> | PR ({{19944049a7a}}) | 44.982 s | 44.947 s | 44.965 s |
> This saves 2.945 seconds, or 6.1%, on average.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to