Hello Brian, I think your main question is to distinguish 1) broker is alive but there's no new data coming into the source topics to process, and 2) broker is not alive and hence nothing is readable, in your monitoring system. I agree that currently process-rate / last-record-timestamp cannot successfully distinguish the two.
I've put a bit more thoughts on https://issues.apache.org/jira/browse/KAFKA-6520 and updated the description: originally we want to add a new State into Streams but later on we realized that 1) it is a bit overkill to complicate the FSM for this transient state and 2) it is actually a general issue that should be tackled on the lower level rather than on Streams only. Guozhang On Mon, Jul 8, 2019 at 5:15 PM Brian Putt <puttbr...@gmail.com> wrote: > Hello, > > We have multiple stream services that we're looking to monitor when they've > been disconnected from the broker so that we can restart the services. > > I've looked at https://issues.apache.org/jira/browse/KAFKA-6520 and am > wondering if anyone has suggestions on what we can do today to help ensure > our services don't go idle. > > As an example, we'll have our streaming services running and we'll > stop/start the kafka brokers. The services will remain running, but they're > not actually pulling any data. > > We could look at time since last record received, but that's not a > guarantee as there's always a possibility that data was legitimately turned > off upstream. > > Thanks, > > Brian > -- -- Guozhang
