[
https://issues.apache.org/jira/browse/SPARK-58089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18099006#comment-18099006
]
Qiegang Long commented on SPARK-58089:
--------------------------------------
Performance test result with [variant conformance
benchmark|https://github.com/cloudera-labs/variant-conformance-benchmark].
tpc-ds (SF=5) queries are about 4 times faster after the fix.
{code:java}
ompare: tpcds-flat-pullout-20260714 vs tpcds-flat-no-pullout-20260714 (metric:
query_median)
Run A:
/Users/qlong/opensources/variant-conformance-benchmark/results/tpcds-flat-pullout-20260714/tpcds-flat/timings-variant.csv
Run B:
/Users/qlong/opensources/variant-conformance-benchmark/results/tpcds-flat-no-pullout-20260714/tpcds-flat/timings-variant.csv
query median_A median_B delta_s delta_%
------ -------- -------- ------- -------
q07 1.40 3.28 -1.88 -57.2%
q12 0.07 0.07 -0.00 -2.8%
q19 0.06 0.07 -0.01 -9.9%
q26 1.14 3.40 -2.26 -66.4%
q42 0.56 2.59 -2.03 -78.2%
q52 0.61 3.04 -2.43 -80.0%
q55 0.54 2.68 -2.14 -79.9%
q63 0.58 2.95 -2.37 -80.4%
q68 1.00 3.42 -2.42 -70.8%
q73 0.57 2.88 -2.31 -80.1%
q79 0.85 3.26 -2.41 -73.9%
q98 0.65 3.32 -2.67 -80.5%
Summary: 12 queries, 12 comparable
Geo-mean delta: -69.5% (Run A faster)
Total (query_median): 8.0s vs 31.0s {code}
> Variant extraction pushdown does not work for Aggregate, Sort, and Join
> -----------------------------------------------------------------------
>
> Key: SPARK-58089
> URL: https://issues.apache.org/jira/browse/SPARK-58089
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 4.1.2
> Reporter: Qiegang Long
> Priority: Major
> Labels: pull-request-available
>
> While running performance test, noticed that variant extraction pushdown does
> does not extend to aggregate and join, losing performance benefits of
> shreddred columns.
> Extaction pushdown not working for these examples:
> Example 1:
> {code:sql}
> SELECT AVG(variant_get(data, '$.qty', 'double'))
> FROM store_sales
> GROUP BY variant_get(data, '$.id', 'string')
> {code}
> Example 2:
> {code:sql}
> SELECT AVG(variant_get(ss.data, '$.qty', 'double'))
> FROM store_sales ss
> JOIN date_dim d ON ss.date_sk = d.date_sk
> JOIN store st ON ss.store_sk = st.store_sk
> GROUP BY variant_get(ss.data, '$.id', 'string')
> {code}
> Example 3:
> {code:sql}
> SELECT ss.k
> FROM store_sales ss
> JOIN date_dim d ON ss.date_sk = variant_get(d.data, '$.sk', 'int')
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
