Github user vackosar commented on a diff in the pull request: https://github.com/apache/spark/pull/21919#discussion_r208901343 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/WriteToDataSourceV2.scala --- @@ -58,6 +61,7 @@ case class WriteToDataSourceV2Exec(writer: DataSourceWriter, query: SparkPlan) e val useCommitCoordinator = writer.useCommitCoordinator val rdd = query.execute() val messages = new Array[WriterCommitMessage](rdd.partitions.length) + val totalNumRowsAccumulator = new LongAccumulator() --- End diff -- I am using here threadsafe java native LongAccumulator. Spark cluster accumulator is not needed here and would be also a bit out of the ordinary in the code base area. Note that StreamingQueryProgress is reported as an event object. That is why I am not using the cluster accumulator here.
--- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org