vranes commented on code in PR #59016:
URL: https://github.com/apache/spark/pull/59016#discussion_r4124420485
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/joins/SortMergeAsOfJoinExec.scala:
##########
@@ -424,10 +405,38 @@ private[joins] class SortMergeAsOfJoinScanner(
}
/**
- * Forward scan for Forward/Nearest joins: distance-based termination.
- * Stop when distance starts increasing past the minimum found so far.
+ * Forward joins: returns the first row that passes both conditions. It
picks by buffer
+ * order, not by distance: the distance can rank STRUCT and ARRAY values
wrongly, for
Review Comment:
Again, we don't need historical / iterative style comments. And
`findFirstForward` and `findLastBackward` comments should be aligned.
##########
docs/sql-ref-syntax-qry-select-asof-join.md:
##########
@@ -121,7 +121,9 @@ comparison_operator
* **NULL.** A `NULL` in either operand never satisfies the `MATCH_CONDITION`
comparison. Left rows whose operand is `NULL` are dropped under `INNER ASOF`
and
- retained with `NULL` right-side columns under `LEFT ASOF`.
+ retained with `NULL` right-side columns under `LEFT ASOF`. A `NULL` element
of an
+ `ARRAY` operand, or a `NULL` field of a `STRUCT` operand, does not block a
match. It
+ sorts before any non-`NULL` value, so `ARRAY(NULL) <= ARRAY(5)` is true.
Review Comment:
Is this true in Spark generally?
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/basicLogicalOperators.scala:
##########
@@ -2772,8 +2772,7 @@ object AsOfJoin {
*/
private[catalyst] object MatchConditionTypes {
- def isValidOperandType(dataType: DataType): Boolean =
- RowOrdering.isOrderable(dataType) && !containsEmptyStructType(dataType)
+ def isValidOperandType(dataType: DataType): Boolean =
RowOrdering.isOrderable(dataType)
Review Comment:
Now for STRUCT<e: STRUCT<>>, a NULL value becomes a non-NULL struct with e =
NULL. As a result, two NULL values can match with >= . Is that behaviour
desired here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
