ueshin commented on code in PR #40692: URL: https://github.com/apache/spark/pull/40692#discussion_r1160985748
########## connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/connect/client/SparkResult.scala: ########## @@ -60,13 +61,19 @@ private[sql] class SparkResult[T]( private def processResponses(stopOnFirstNonEmptyResponse: Boolean): Boolean = { while (responses.hasNext) { val response = responses.next() + if (response.hasSchema) { + structType = Review Comment: Now that the original schema arrives earlier than arrow batches, we should use it if it's available; otherwise fallback to the schema from arrow batch. Yes, the response schema and arrow schema could be inconsistent in terms of the nested field names if there are duplicates, but it's not problem while encoder is handling the `ColumnarBatch` as long as the data structure is consistent. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org