sunchao commented on code in PR #4715:
URL: https://github.com/apache/datafusion-comet/pull/4715#discussion_r4127872858
##########
native/core/src/execution/planner.rs:
##########
@@ -4922,8 +4932,14 @@ fn literal_to_array_ref(
let child_refs: Vec<&dyn Array> = child_arrays.iter().map(|a|
a.as_ref()).collect();
arrow::compute::concat(&child_refs)?
} else {
- // All entries are null or the list is empty
- new_empty_array(&dt)
+ // All entries are null or empty. The recursive call returns
the values of a `dt`
+ // list, so the empty values array takes `dt`'s element type,
not `dt` itself;
+ // otherwise this level gains an extra list and cannot be
concatenated with a
+ // populated sibling, e.g. [[[]], [[1]]].
+ match &dt {
+ DataType::List(element) =>
new_empty_array(element.data_type()),
Review Comment:
[P2] Preserve consistent nested field nullability in this empty-array
branch. With default constant folding and a nonempty Parquet table `t`, `SELECT
array(array(array()), array(array(array(INTERVAL '1' MONTH)))) FROM t` produces
a folded literal whose array elements are non-nullable. Here,
`new_empty_array(element.data_type())` preserves those nested flags, while
populated siblings are recursively rebuilt using nullable fields at line 4947.
Arrow rejects their concatenation, so a valid Spark query fails native
planning. The base declines this interval literal and falls back successfully.
Use the same nested field types for both branches and add a regression with
constant folding enabled.
Evidence: Spark 4.1.3 returned `[[[]],[[[P1M]]]]`, with optimized literal
`[[[]],[[[1]]]]` and `containsNull=false` at all four array levels. An isolated
probe using the unchanged exact-head `literal_to_array_ref`, matching
ListLiteral values and Arrow 59.3.0 reproduced: `It is not possible to
concatenate arrays of different data types (List(List(non-null
Interval(YearMonth))), List(List(Interval(YearMonth)))).` The all-nullable
control and the previous review’s exact reproducer passed. Validation was at
the decoder boundary, not full Comet execution.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
