sunchao commented on issue #5544: URL: https://github.com/apache/datafusion-comet/issues/5544#issuecomment-5464226832
Two additional **P2 follow-up cases** from the re-review of #5452 at [`ec9832366`](https://github.com/apache/datafusion-comet/commit/ec9832366bf05b2c645f72c8fabeacfb7af48925), recorded here under the agreed follow-up scope. Both were reproduced on Spark 4.1.3 / JDK 17 with normal constant folding and a Parquet table `t(id INT)` containing `1, 2, 3` in one file. **1. Duplicate struct field names inside folded map values** ```sql SELECT id, array(map(1, named_struct('x', 10, 'x', 20))) AS a FROM t; ``` Spark preserves both struct values, `10` and `20`. Comet selects `CometProject` and fails with `NullPointerException` when the generated writer writes the second field. The [literal reconstruction](https://github.com/apache/datafusion-comet/blob/ec9832366bf05b2c645f72c8fabeacfb7af48925/spark/src/main/scala/org/apache/comet/serde/literals.scala#L273-L281) sends the struct value through whole-map JVM codegen, bypassing the duplicate-name guard on `CreateNamedStruct`. Arrow initializes struct children by name, collapsing the two `x` fields into one child; the writer's ordinal-1 vector is null. The direct `map(...)` variant also fails. Distinct `x`/`y` field-name controls pass. A narrow follow-up can retain Spark fallback for these nested shapes until the output path preserves duplicate field positions. **2. Calendar interval map values outside Arrow's nanosecond range** ```sql SELECT id, array(map(1, make_interval(0, 0, 0, 0, 3000000, 0, 0))) AS a FROM t; ``` Spark accepts the `3000000 hours` value, while Comet throws `ArithmeticException: long overflow`. The [new admission check](https://github.com/apache/datafusion-comet/blob/ec9832366bf05b2c645f72c8fabeacfb7af48925/spark/src/main/scala/org/apache/comet/serde/literals.scala#L295-L302) allows this map value type; the calendar-interval restriction applies to map keys. The [generated output writer](https://github.com/apache/datafusion-comet/blob/ec9832366bf05b2c645f72c8fabeacfb7af48925/spark/src/main/scala/org/apache/comet/codegen/CometBatchKernelCodegenOutput.scala#L222-L229) multiplies the interval's microseconds by `1000L` using `Math.multiplyExact`, exceeding the nanosecond field's range. The direct `map(...)` variant also fails; a 24-hour control passes natively. A narrow follow-up can retain Spark fallback for out-of-range calendar interval values. **Validation:** all four failing variants (direct map and array of maps for each case) were reproduced with a freshly built native library, current JVM classes, and `CometProject` asserted. Ordinary Spark succeeds. All eight reproduction/control tests pass after substituting the exact-base [`215ab706a` literal serializer](https://github.com/apache/datafusion-comet/blob/215ab706aa9af074e589f400cd6843dee9938030/spark/src/main/scala/org/apache/comet/serde/literals.scala), restoring Spark projection for these folded literals. That control replaces only the literal serializer on the current native/JVM runtime; it is not a full base build. Local execution was limited to Spark 4.1.3. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
