Github user jose-torres commented on a diff in the pull request:
https://github.com/apache/spark/pull/20445#discussion_r165222931
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/MicroBatchExecution.scala
---
@@ -270,16 +270,17 @@ class MicroBatchExecution(
}
case s: MicroBatchReader =>
updateStatusMessage(s"Getting offsets from $s")
- reportTimeTaken("getOffset") {
- // Once v1 streaming source execution is gone, we can refactor
this away.
- // For now, we set the range here to get the source to infer
the available end offset,
- // get that offset, and then set the range again when we later
execute.
- s.setOffsetRange(
- toJava(availableOffsets.get(s).map(off =>
s.deserializeOffset(off.json))),
- Optional.empty())
-
- (s, Some(s.getEndOffset))
+ reportTimeTaken("setOffsetRange") {
--- End diff --
I agree that the old metric names don't make much sense anymore, but I
worry about changing external-facing behavior as part of an API migration.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
