c21 commented on a change in pull request #32198:
URL: https://github.com/apache/spark/pull/32198#discussion_r619979939
##########
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileFormatDataWriter.scala
##########
@@ -243,6 +273,52 @@ class DynamicPartitionDataWriter(
statsTrackers.foreach(_.newFile(currentPath))
}
+ /**
+ * Increase the file counter and open a new OutputWriter.
+ * This is used when number of records records exceeding limit.
+ *
+ * @param partitionValues the partition which all tuples being written by
this `OutputWriter`
+ * belong to
+ * @param bucketId the bucket which all tuples being written by this
`OutputWriter` belong to
+ */
+ protected def increaseFileCounter(
Review comment:
In `DynamicPartitionDataConcurrentWriter.write`, I need this condition
check to update writer in map as well (as the current writer is renewed). If we
put the condition inside here, we need double check in
`DynamicPartitionDataConcurrentWriter.write`. Shall we keep it as it is?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
