Github user jose-torres commented on a diff in the pull request:
https://github.com/apache/spark/pull/21721#discussion_r200538008
--- Diff:
external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/JsonUtils.scala
---
@@ -95,4 +95,25 @@ private object JsonUtils {
}
Serialization.write(result)
}
+
+ /**
+ * Write per-topic partition lag as json string
--- End diff --
Is "lag" here just the difference (at the time a batch ends) between the
last offset Spark knows about and the last offset Spark has processed? I'm not
sure this is super useful to know. If maxOffsets isn't set it's always going to
be 0, no matter how far Spark gets behind the Kafka cluster.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
