Hi Andrew, Thank you for your response.
I understand your explanation, but in both cases, I am using an "isolation level" of READ_COMMITTED. I believe the issue lies in the AdminClient.listOffsets method, as it may not be considering the isolation level, where as the consumer of AdminClient.listConsumerGroupOffsets does consider it. What are your thoughts on this? Additionally, would it be more suitable to implement a solution that reverts the offsets in case of transaction rollbacks? It's possible that there's a logic aspect I'm not fully grasping. Perhaps I need to utilize the internal control records and their offsets. Could you point me in the right direction for their documentation? Thank you, Henry ________________________________ De: Andrew Schofield <andrew_schofield_j...@outlook.com> Enviado: martes, 27 de junio de 2023 13:22 Para: dev@kafka.apache.org <dev@kafka.apache.org> Asunto: Fwd: Offsets: consumption and production in rollback Hi Henry, Thanks for your message. Kafka transactions are a bit unusual. If you produce a message inside a transaction, it is assigned an offset on a topic-partition before the transaction even commits. That offset is not “revoked” if the transaction rolls back. This is why the consumer has the concept of “isolation level”. It essentially controls whether the consumer can “see” the uncommitted or even rolled back messages. A consumer using the committed isolation level only consumes committed messages, but the offsets that it observes do reflect the uncommitted messages. So, if you observe the progress of the offsets of the records consumed, you see that they skip the messages that were produced but then rolled back. There are also invisible control records that are used to achieve transactional behaviour, and those also have offsets. I’m not sure that this is really “bogus lag” but, when you’re using transactions, there’s not a one-to-one relationship between offsets and consumable records. Hope this helps, Andrew Begin forwarded message: From: Henry GALVEZ <henry.gal...@intm.fr> Subject: Offsets: consumption and production in rollback Date: 27 June 2023 at 10:48:31 BST To: "us...@kafka.apache.org" <us...@kafka.apache.org>, "dev@kafka.apache.org" <dev@kafka.apache.org> Reply-To: dev@kafka.apache.org I have some doubts regarding message consumption and production, as well as transactional capabilities. I am using a Kafka template to produce a message within a transaction. After that, I execute another transaction that produces a message and intentionally throws a runtime exception to simulate a transaction rollback. Next, I use the Kafka AdminClient to retrieve the latest offset for the topic partition and the consumer group's offsets for the same topic partition. However, when I compare the offset numbers, I notice a difference. In this example, the consumer has 4 offsets, while the topic has only 2. I have come across references to this issue in a Spring-Kafka report, specifically in the Kafka-10683 report, where developers describe it as either Bogus or Pseudo Lag. I am keen on resolving this problem, and I would greatly appreciate hearing about your experiences and knowledge regarding this matter. Thank you very much Henry
