HyukjinKwon commented on code in PR #38613:
URL: https://github.com/apache/spark/pull/38613#discussion_r1020130165
##########
connector/connect/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectStreamHandler.scala:
##########
@@ -144,36 +144,10 @@ class SparkConnectStreamHandler(responseObserver:
StreamObserver[Response]) exte
.toArrowBatchIterator(iter, schema, maxRecordsPerBatch, timeZoneId)
}
- val signal = new Object
- val partitions = collection.mutable.Map.empty[Int, Array[Batch]]
-
- val processPartition = (iter: Iterator[Batch]) => iter.toArray
-
// This callback is executed by the DAGScheduler thread.
- // After fetching a partition, it inserts the partition into the Map,
and then
- // wakes up the main thread.
- val resultHandler = (partitionId: Int, partition: Array[Batch]) => {
- signal.synchronized {
- partitions(partitionId) = partition
- signal.notify()
- }
- ()
- }
-
- spark.sparkContext.runJob(batches, processPartition, resultHandler)
-
- // The man thread will wait until 0-th partition is available,
- // then send it to client and wait for next partition.
- var currentPartitionId = 0
- while (currentPartitionId < numPartitions) {
- val partition = signal.synchronized {
- while (!partitions.contains(currentPartitionId)) {
- signal.wait()
- }
- partitions.remove(currentPartitionId).get
- }
-
- partition.foreach { case (bytes, count) =>
+ def writeBatches(arrowBatches: Array[Batch]): Unit = {
Review Comment:
Sorry I missed your comment when I opened this PR. BTW, this is actually
what PySpark's implementation doing, and was thinking that it's better to match
how they work, dedup, and improve together. It should work fine most cases -
PySpark implementation has been running in production many years, and I haven't
yet heard complaints related to this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
