XdithyX commented on code in PR #56486:
URL: https://github.com/apache/spark/pull/56486#discussion_r3476898289
##########
sql/catalyst/src/main/java/org/apache/spark/sql/connector/read/SupportsPushDownJoin.java:
##########
@@ -63,6 +65,61 @@ boolean pushDownJoin(
Predicate condition
);
+ /**
+ * Pushes down the join of the current {@code SupportsPushDownJoin} and the
other side of join
+ * {@code SupportsPushDownJoin}, with pushed samples from either side.
+ *
+ * @param other {@code SupportsPushDownJoin} that this {@code
SupportsPushDownJoin}
+ * gets joined with.
+ * @param joinType the type of join.
+ * @param leftSideRequiredColumnsWithAliases required output of the
+ * left side {@code
SupportsPushDownJoin}
+ * @param rightSideRequiredColumnsWithAliases required output of the
+ * right side {@code
SupportsPushDownJoin}
+ * @param condition join condition. Columns are named after the specified
aliases in
+ * {@code leftSideRequiredColumnWithAliases} and {@code
rightSideRequiredColumnWithAliases}
+ * @param leftSample pushed sample from the left side, or null if there is
no pushed sample.
+ * @param rightSample pushed sample from the right side, or null if there is
no pushed sample.
+ * @return True if join has been successfully pushed down.
+ *
+ * @since 4.3.0
+ */
+ default boolean pushDownJoin(
Review Comment:
Thanks @cloud-fan for the detailed review, this makes sense. I agree that
adding a new pushDownJoin overload for every pushed operator would not scale
well.
I’ll rework this to use a single join pushdown context object instead of
sample-specific parameters. The context can carry pushed state for the left and
right sides, starting with table sample information. If the context contains
result-affecting pushed operators that the connector has not explicitly
acknowledged/supports for join composition, Spark should reject join pushdown
rather than allowing the connector to silently drop them.
For JDBC, the implementation can continue using its existing pushed state
(`tableSampleClause`) to build the SQL.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
