andygrove opened a new pull request, #6689: URL: https://github.com/apache/datafusion-comet/pull/6689
## Which issue does this PR close? Closes #6685. ## Rationale for this change A native shuffle whose child is a Spark-to-Arrow conversion, such as `CometSparkToColumnarExec` or `CometLocalTableScanExec`, fails on a struct column from the second batch of a partition with `no more field nodes for field ...`. #6685 has the details and reproductions. The fix was part of #6607, and is split out here at review so that it can land, and be backported, on its own. ## What changes are included in this PR? Native shuffle reads a `CometNativeArrowSource` child through `doExecuteAsArrowStream()`, as a native operator already does with such an input. It used to wrap the child's `executeColumnar()` batches in a `ColumnarBatchArrowReader`, which closes each batch once native has it. The conversion writes every batch into the same vectors, and closing a struct vector drops its children, so the next batch had none. The new `prepareArrowStreamShuffleDependency` builds the shuffle dependency around the stream. `prepareShuffleDependency`, which `CometCollectLimitExec` and `CometTakeOrderedAndProjectExec` use, now wraps its batches in a stream and calls it. Range partitioning still samples rows from the child's `executeColumnar()`, which does not close the batches. The code path dates from #4572, so 1.0.0 and `branch-1.1` have the bug too. It is not a 1.1.0 regression, and every conversion that reaches it is off by default. ## How are these changes tested? A new test in `CometNativeShuffleSuite` hash- and range-partitions a struct column that fills several batches per partition, read from an RDD scan that `spark.comet.convert.rdd.enabled` converts and from a local table scan. Both inputs fail without the change. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
