Re: kafka streams consumer group reporting lag even on source topics removed from topology

2023-08-16 Thread Matthias J. Sax
Well, it's kinda expected behavior. It's a split brain problem. In the end, you use the same `application.id / group.id` and thus the committed offsets for the removed topics are still in `__consumer_offsets` topics and associated with the consumer group. If a tool inspects lags and compares

Re: Kafka connect Graceful stop of task failed

2023-08-16 Thread Greg Harris
Hi Robson, Thank you for the detailed bug report. I believe the behavior that you're describing is caused by this flaw: https://issues.apache.org/jira/browse/KAFKA-15090 which is still under discussion. Since the above flaw was introduced in 3.0, source connectors need to return from poll()

Kafka connect Graceful stop of task failed

2023-08-16 Thread Robson Hermes
Hello I'm using kafka connect 7.4.0 to read data from Postgres views and write to another Postgres tables. So using JDBC source and sink connectors. All works good, but whenever I stop the source connectors via the rest api: DEL http://kafka-connect:8083/connectors/connector_name_here The

list offsets of compacted topics

2023-08-16 Thread Shahar Cizer Kobrinsky
Hey Folks, I have a process which aims to measure time lag (in Scala). When the process bootstraps it looks at the history of offsets and collect the offset that existed for different timestamps (7 days ago, 6 days ago... etc in more frequency as it gets closer to *now*). In order to do that it

Re: kafka streams consumer group reporting lag even on source topics removed from topology

2023-08-16 Thread Pushkar Deole
Hi streams Dev community @matthias, @bruno Any inputs on above issue? Is this a bug in the streams library wherein the input topic removed from streams processor topology, the underlying consumer group still reporting lag against those? On Wed, Aug 9, 2023 at 4:38 PM Pushkar Deole wrote: > Hi