szehon-ho commented on code in PR #51091:
URL: https://github.com/apache/spark/pull/51091#discussion_r2152866689
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/MergeRowsExec.scala:
##########
@@ -233,6 +246,7 @@ case class MergeRowsExec(
}
}
+ longMetric("numTargetRowsCopied") += 1
Review Comment:
Yea, I think we are just interpreting it a bit differently, initially I
thought its more like 'rows that are processed but not matching the filters'.
In Delta it is indeed not written, but it's because its not a delta.
Maybe you are right, the name of the metric is 'copied', cc @aokolnychyi .
Maybe there is some value in my interpretation, as another metric (rows dropped
in delta..), but can do a subsequent pr.
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/MergeRowsExec.scala:
##########
@@ -233,6 +246,7 @@ case class MergeRowsExec(
}
}
+ longMetric("numTargetRowsCopied") += 1
Review Comment:
Yea, I think we are just interpreting it a bit differently, initially I
thought its more like 'rows that are processed but not matching the filters'.
In Delta it is indeed not written, but it's because its not a delta.
Maybe you are right, the name of the metric is 'copied', cc @aokolnychyi .
Maybe there is some value in my interpretation, as another metric, but can do a
subsequent pr.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
