[jira] [Commented] (SPARK-33833) Allow Spark Structured Streaming report Kafka Lag through Burrow

[ https://issues.apache.org/jira/browse/SPARK-33833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17259280#comment-17259280 ] L. C. Hsieh commented on SPARK-33833: - Thanks [~kabhwan]. > Allow Spark Structured Streaming

[jira] [Commented] (SPARK-33833) Allow Spark Structured Streaming report Kafka Lag through Burrow

[ https://issues.apache.org/jira/browse/SPARK-33833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17259263#comment-17259263 ] Jungtaek Lim commented on SPARK-33833: -- (Just to make clear, I was talking about setting the group

[jira] [Commented] (SPARK-33833) Allow Spark Structured Streaming report Kafka Lag through Burrow

[ https://issues.apache.org/jira/browse/SPARK-33833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17259094#comment-17259094 ] L. C. Hsieh commented on SPARK-33833: - I will close for now as duplicated. > Allow Spark Structured

[jira] [Commented] (SPARK-33833) Allow Spark Structured Streaming report Kafka Lag through Burrow

[ https://issues.apache.org/jira/browse/SPARK-33833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17258726#comment-17258726 ] L. C. Hsieh commented on SPARK-33833: - Yea, but this can be easily overcome here. We just need to

[jira] [Commented] (SPARK-33833) Allow Spark Structured Streaming report Kafka Lag through Burrow

[ https://issues.apache.org/jira/browse/SPARK-33833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17258717#comment-17258717 ] Jungtaek Lim commented on SPARK-33833: -- That’s available with serious caution. Spark has to have

[jira] [Commented] (SPARK-33833) Allow Spark Structured Streaming report Kafka Lag through Burrow

[ https://issues.apache.org/jira/browse/SPARK-33833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17258716#comment-17258716 ] L. C. Hsieh commented on SPARK-33833: - I read though the comments in the previous PR. The approach

[jira] [Commented] (SPARK-33833) Allow Spark Structured Streaming report Kafka Lag through Burrow

[ https://issues.apache.org/jira/browse/SPARK-33833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17258707#comment-17258707 ] L. C. Hsieh commented on SPARK-33833: - Btw, thanks for providing the useful link to previous

[jira] [Commented] (SPARK-33833) Allow Spark Structured Streaming report Kafka Lag through Burrow

[ https://issues.apache.org/jira/browse/SPARK-33833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17258705#comment-17258705 ] L. C. Hsieh commented on SPARK-33833: - I think SS allows users to specify custom group id, isn't it?

[jira] [Commented] (SPARK-33833) Allow Spark Structured Streaming report Kafka Lag through Burrow

[ https://issues.apache.org/jira/browse/SPARK-33833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17258695#comment-17258695 ] Jungtaek Lim commented on SPARK-33833: -- For SS, consumer group is randomly generated by intention,

[jira] [Commented] (SPARK-33833) Allow Spark Structured Streaming report Kafka Lag through Burrow

[ https://issues.apache.org/jira/browse/SPARK-33833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17258693#comment-17258693 ] L. C. Hsieh commented on SPARK-33833: - [~samdvr] Can you help elaborate the question above? Thanks.

[jira] [Commented] (SPARK-33833) Allow Spark Structured Streaming report Kafka Lag through Burrow

[ https://issues.apache.org/jira/browse/SPARK-33833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17258692#comment-17258692 ] L. C. Hsieh commented on SPARK-33833: - Hmm, I did a few test locally. Does Burrow work only if Spark

[jira] [Commented] (SPARK-33833) Allow Spark Structured Streaming report Kafka Lag through Burrow

[ https://issues.apache.org/jira/browse/SPARK-33833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17251926#comment-17251926 ] L. C. Hsieh commented on SPARK-33833: - I'm also in need of tracking the lag, but I maybe don't need

[jira] [Commented] (SPARK-33833) Allow Spark Structured Streaming report Kafka Lag through Burrow

[ https://issues.apache.org/jira/browse/SPARK-33833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17251925#comment-17251925 ] L. C. Hsieh commented on SPARK-33833: - Hmm, as SS commits source offsets in external storage by SS

[jira] [Commented] (SPARK-33833) Allow Spark Structured Streaming report Kafka Lag through Burrow

[ https://issues.apache.org/jira/browse/SPARK-33833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17251512#comment-17251512 ] Hyukjin Kwon commented on SPARK-33833: -- Looks like it leverages listenters. Can you use