zhangfengcdt commented on code in PR #7536:
URL: https://github.com/apache/datafusion-comet/pull/7536#discussion_r4633491050
##########
spark/src/main/scala/org/apache/spark/sql/comet/CometIcebergWriteExec.scala:
##########
@@ -230,29 +231,41 @@ case class CometIcebergWriteExec(
override def doExecuteColumnar(): RDD[ColumnarBatch] = {
val schedulerProbe = IcebergSchedulerTestProbe.current
- val inputRDD = if (child.supportsColumnar) {
- child.executeColumnar()
- } else {
- throw new UnsupportedOperationException(
- "CometIcebergWriteExec requires a columnar (Comet native) child; got "
+
- child.getClass.getName)
+ // One Arrow stream per partition, which native consumes through the C
Stream Interface.
+ val streamRDD: RDD[ArrowArrayStream] = child match {
+ // Native reads the source's Arrow stream, as it does for a native
operator's input. The
+ // source reuses its vectors for the next batch, and wrapping its
`executeColumnar` batches
+ // below closes each batch once native has it; closing a struct vector
drops its children,
+ // so the next batch would lose them (#7239). The scheduler probe counts
rows as batches.
+ case source: CometNativeArrowSource if schedulerProbe.isEmpty =>
Review Comment:
Done. The comment now explains why the stream route is still better on
`main` (no extra copy per batch) and points to #7239 for the release branches.
##########
spark/src/main/scala/org/apache/spark/sql/comet/CometIcebergWriteExec.scala:
##########
@@ -230,29 +231,41 @@ case class CometIcebergWriteExec(
override def doExecuteColumnar(): RDD[ColumnarBatch] = {
val schedulerProbe = IcebergSchedulerTestProbe.current
- val inputRDD = if (child.supportsColumnar) {
- child.executeColumnar()
- } else {
- throw new UnsupportedOperationException(
- "CometIcebergWriteExec requires a columnar (Comet native) child; got "
+
- child.getClass.getName)
+ // One Arrow stream per partition, which native consumes through the C
Stream Interface.
+ val streamRDD: RDD[ArrowArrayStream] = child match {
+ // Native reads the source's Arrow stream, as it does for a native
operator's input. The
+ // source reuses its vectors for the next batch, and wrapping its
`executeColumnar` batches
+ // below closes each batch once native has it; closing a struct vector
drops its children,
+ // so the next batch would lose them (#7239). The scheduler probe counts
rows as batches.
+ case source: CometNativeArrowSource if schedulerProbe.isEmpty =>
Review Comment:
Fixed. The probe no longer changes the route: a `CometNativeArrowSource`
child always reads the stream, and the probe now hooks into that stream. So the
scheduler tests run the production route.
##########
spark/src/test/scala/org/apache/comet/CometIcebergWriteActionSuite.scala:
##########
@@ -412,6 +413,45 @@ class CometIcebergWriteActionSuite
}
}
+ // https://github.com/apache/datafusion-comet/issues/7239. A conversion of
Spark rows writes
+ // every batch into the same vectors, and closing a struct vector drops its
children. The write
+ // reads the conversion's Arrow stream, as native shuffle does, rather than
batches it would
+ // close once native had them, which left the second batch of a partition
without them.
+ test("native write reads a struct column that a conversion writes over
several batches") {
Review Comment:
Right, it guards the release branches; the backport PRs will show it failing
there. The test now covers both an RDD conversion and a local table scan.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]