ZiyaZa commented on code in PR #55586:
URL: https://github.com/apache/spark/pull/55586#discussion_r3232927799
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/WriteToDataSourceV2Exec.scala:
##########
@@ -577,19 +614,22 @@ trait V2TableWriteExec
(index, result: DataWritingSparkTaskResult) => {
val commitMessage = result.writerCommitMessage
messages(index) = commitMessage
- totalNumRowsAccumulator.add(result.numRows)
+ numOutputRowsMetric.add(result.numRows)
batchWrite.onDataWriterCommit(commitMessage)
}
)
+ val executionId =
sparkContext.getLocalProperty(SQLExecution.EXECUTION_ID_KEY)
Review Comment:
No, the old `totalNumRowsAccumulator` was not attached to the plan node, so
it wasn't visible on the UI. I moved that accumulator to a SQLMetric attached
to this node, so now it should be visible for all write nodes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
