LukaZdravic commented on code in PR #59016:
URL: https://github.com/apache/spark/pull/59016#discussion_r4131635089
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/basicLogicalOperators.scala:
##########
@@ -2772,8 +2772,7 @@ object AsOfJoin {
*/
private[catalyst] object MatchConditionTypes {
- def isValidOperandType(dataType: DataType): Boolean =
- RowOrdering.isOrderable(dataType) && !containsEmptyStructType(dataType)
+ def isValidOperandType(dataType: DataType): Boolean =
RowOrdering.isOrderable(dataType)
Review Comment:
So it's an asof specific problem and it's about how the match condition
values are re-written. Both for nested and non-nested structs problem is the
same they get re-written as a new struct built from their fields, so a NULL
struct stops being NULL.
With this PR, structs that contain an empty struct can now reach this:
- `STRUCT<e: STRUCT<>>`: NULL becomes `{e: NULL}`
- `STRUCT<a: INT, e: STRUCT<>>`: NULL becomes `{a: NULL, e: NULL}`
This is not new with this PR. Master already does the same for normal
(non-empty) structs:
- `STRUCT<a: INT>`: NULL becomes `{a: NULL}`
- `STRUCT<e: STRUCT<a: INT>>`: NULL becomes `{col1: {a: NULL}}`
Because of that, `NULL >= NULL` matches in ASOF, while a plain `WHERE` with
the same comparison does not.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
