[ 
https://issues.apache.org/jira/browse/SPARK-61578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Dongjoon Hyun updated SPARK-61578:
----------------------------------
    Affects Version/s: 4.4.0
                           (was: 5.0.0)

> Avoid collecting expected DataFrames in DSv2 write tests
> --------------------------------------------------------
>
>                 Key: SPARK-61578
>                 URL: https://issues.apache.org/jira/browse/SPARK-61578
>             Project: Spark
>          Issue Type: Improvement
>          Components: SQL, Tests
>    Affects Versions: 4.4.0
>            Reporter: Szehon Ho
>            Assignee: Szehon Ho
>            Priority: Major
>              Labels: pull-request-available
>
> h3. Pull request
> [apache/spark#60407|https://github.com/apache/spark/pull/60407]
> h3. Motivation
> {{checkAnswer(actual, expectedDataFrame)}} collects the expected DataFrame 
> before delegating to
> the row-based result check. In this suite that DataFrame is also the input 
> already executed by
> the write command, so collecting it again duplicates execution without adding 
> an assertion.
> h3. Coverage traceability
> * All 99 test cases and configuration combinations remain.
> * The write-source execution remains.
> * The written-table result comparison remains, using expected rows derived 
> from the same input
>   tuples.
> * The actual table's {{materializedRdd}} deserialization check performed by 
> {{checkAnswer}}
>   remains.
> * The only removed execution is the redundant collection of the expected 
> DataFrame.
> h3. Benchmark
> Two local runs of the full suite on each commit produced:
> || Commit || Run 1 || Run 2 || Average ||
> | Parent ({{0d52f01d106}}) | 47.601 s | 48.217 s | 47.909 s |
> | PR ({{19944049a7a}}) | 44.982 s | 44.947 s | 44.965 s |
> This saves 2.945 seconds, or 6.1%, on average.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to