sunchao commented on code in PR #5603:
URL: https://github.com/apache/datafusion-comet/pull/5603#discussion_r3904269703
##########
spark/src/main/scala/org/apache/comet/serde/structs.scala:
##########
@@ -31,12 +31,16 @@ import
org.apache.comet.CometSparkSessionExtensions.withFallbackReason
import org.apache.comet.DataTypeSupport
import org.apache.comet.serde.QueryPlanSerde.{exprToProtoInternal,
serializeDataType}
-object CometCreateNamedStruct extends CometExpressionSerde[CreateNamedStruct] {
+object CometCreateNamedStruct
+ extends CometExpressionSerde[CreateNamedStruct]
+ with CodegenDispatchFallback {
Review Comment:
[P2] Preserve duplicate children at both remaining Arrow readers
Could this dispatcher enrollment also cover the readers that still use
Arrow's default vector factory? With nonempty, native-compatible integer
Parquet input and project/codegen conversion enabled, retaining
`named_struct('x', a, 'x', b)` as a broadcast integer-equijoin payload now
reaches `coalesceBroadcastBatches` and its plain `ArrowStreamReader` when AQE
is off and broadcast/join conversion is enabled. Under Arrow 18.3.0's default
`CONFLICT_REPLACE`, the two `x` children become one. `VectorLoader` then
rejects the remaining nodes/buffers.
For the same inputs, `SELECT to_json(s), s FROM (SELECT named_struct('x', a,
'x', b) AS s FROM t) q` retains the producer when
`collapseProjectAlwaysInline=false`. With JVM JSON dispatch enabled and the
small schema within the configured kernel field limit, `CometUdfBridge` still
calls `Data.importVector`. Its default allocation leaves one child, so
`ArrayImporter` rejects the two-child argument before JSON evaluation. A
directly inlined `to_json(named_struct(...))` is not this path.
Both source-traced Spark 3.5 shapes retain Spark fallback at BASE.
Preserving duplicate children at both import boundaries, or retaining fallback
for affected producers, would cover these cases. Fixing only one reader leaves
the other path exposed. These query shapes were not executed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
