Hi all, I'd like to start a discussion on a SPIP to extend DSv2 operator push-down.
JIRA: https://issues.apache.org/jira/browse/SPARK-60765 Design: https://docs.google.com/document/d/1ah8u-1vxPvgqpjw0qzRNXz0Oo-X8tEzlK9ngmg7EPXU/edit Builds on SPARK-52187 (DSv2 join push-down). Shepherd: Holden Karau Today DSv2 can push filters, column pruning, aggregates, limit/TopN, sample, and INNER/LEFT/RIGHT joins. Those joins are off unless spark.sql.optimizer.datasourceV2JoinPushdown is set. A query that uses other join types, a set operation, or a window function over the same database still pulls the raw rows into Spark. The proposal adds opt-in interfaces, each of which a source can decline: 1. FULL OUTER / SEMI / ANTI join types 2. Alias tolerance in join push-down 3. Make the join push-down config public, and evaluate enabling it by default 4. Join + aggregate composition 5. Set operations (UNION / INTERSECT / EXCEPT) 6. Window functions JDBC/H2 is the reference implementation. This does not cover cross-source push-down or file-based sources. Please reply to this thread. A [VOTE] will follow once the discussion settles. Thanks, Jing Zhan
