[
https://issues.apache.org/jira/browse/SPARK-60765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jing Zhan updated SPARK-60765:
------------------------------
Labels: SPIP (was: )
> SPIP: Extend DSv2 push-down with join, set operation, and window function
> support
> ---------------------------------------------------------------------------------
>
> Key: SPARK-60765
> URL: https://issues.apache.org/jira/browse/SPARK-60765
> Project: Spark
> Issue Type: Epic
> Components: SQL
> Affects Versions: 4.4.0
> Reporter: Jing Zhan
> Priority: Major
> Labels: SPIP
>
> DSv2 sources can already push filters, aggregates, and a few join types, but
> a query that joins, aggregates, or unions tables from the same database still
> pulls the raw rows into Spark. The database has an engine that could run that
> work, and the network transfer is often the expensive part.
> For example, a full outer join of orders and customers, a GROUP BY with
> COUNT(DISTINCT), and a UNION with a third table from that same database
> should go to the source as one query. Spark should get back only the result
> rows. SPARK-52187 added the first join push-down. This SPIP extends that to
> the missing join types, set operations, and window functions. A source can
> decline any push, and Spark runs that part locally.
> SPIP doc:
> [https://docs.google.com/document/d/1ah8u-1vxPvgqpjw0qzRNXz0Oo-X8tEzlK9ngmg7EPXU|https://docs.google.com/document/d/1ah8u-1vxPvgqpjw0qzRNXz0Oo-X8tEzlK9ngmg7EPXU/edit]
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
