[jira] [Created] (KAFKA-17093) KafkaConsumer.seekToEnd should return LSO
Tom Kalmijn created KAFKA-17093: --- Summary: KafkaConsumer.seekToEnd should return LSO Key: KAFKA-17093 URL: https://issues.apache.org/jira/browse/KAFKA-17093 Project: Kafka Issue Type: Bug Components: consumer Affects Versions: 3.6.1 Environment: Ubuntu, IntelliJ, Scala "org.apache.kafka" % "kafka-clients" % "3.6.1" Reporter: Tom Kalmijn Expected When using a transactional producer then the method KafkaConsumer.seekToEnd(...) of a consumer configured with isolation level "read_committed" should return the LSO. Observed The offset returned is always the actual last offset of the partition, which is not the LSO if the latest offsets are occupied by transaction markers. Also see this Slack thread: https://confluentcommunity.slack.com/archives/C499EFQS0/p1720088282557559 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-17093) KafkaConsumer.seekToEnd should return LSO
[ https://issues.apache.org/jira/browse/KAFKA-17093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17863689#comment-17863689 ] Tom Kalmijn commented on KAFKA-17093: - kangning, thank you so much for your time and effort! I do apologise if my issue makes you double things which are actually ok. Maybe I am mistaken about the meaning of LSO. I understand "read_committed" instructs a consumer to look at committed messages only. If so I find it surprising that seekToEnd() takes me to an offset where there is nothing to be read? I am after the offset of the last existing message in a partition. If seekToEnd does not take me there then I have to resort to back tracking an polling to figure out where this message is at? > KafkaConsumer.seekToEnd should return LSO > -- > > Key: KAFKA-17093 > URL: https://issues.apache.org/jira/browse/KAFKA-17093 > Project: Kafka > Issue Type: Bug > Components: consumer >Affects Versions: 3.6.1 > Environment: Ubuntu, IntelliJ, Scala "org.apache.kafka" % > "kafka-clients" % "3.6.1" >Reporter: Tom Kalmijn >Priority: Major > > > Expected > When using a transactional producer then the method > KafkaConsumer.seekToEnd(...) of a consumer configured with isolation level > "read_committed" should return the LSO. > Observed > The offset returned is always the actual last offset of the partition, which > is not the LSO if the latest offsets are occupied by transaction markers. > Also see this Slack thread: > https://confluentcommunity.slack.com/archives/C499EFQS0/p1720088282557559 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-17093) KafkaConsumer.seekToEnd should return LSO
[ https://issues.apache.org/jira/browse/KAFKA-17093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17863690#comment-17863690 ] Tom Kalmijn commented on KAFKA-17093: - The thing you mention about apiVersion, is this something I could check on my end? > KafkaConsumer.seekToEnd should return LSO > -- > > Key: KAFKA-17093 > URL: https://issues.apache.org/jira/browse/KAFKA-17093 > Project: Kafka > Issue Type: Bug > Components: consumer >Affects Versions: 3.6.1 > Environment: Ubuntu, IntelliJ, Scala "org.apache.kafka" % > "kafka-clients" % "3.6.1" >Reporter: Tom Kalmijn >Priority: Major > > > Expected > When using a transactional producer then the method > KafkaConsumer.seekToEnd(...) of a consumer configured with isolation level > "read_committed" should return the LSO. > Observed > The offset returned is always the actual last offset of the partition, which > is not the LSO if the latest offsets are occupied by transaction markers. > Also see this Slack thread: > https://confluentcommunity.slack.com/archives/C499EFQS0/p1720088282557559 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-17093) KafkaConsumer.seekToEnd should return LSO
[ https://issues.apache.org/jira/browse/KAFKA-17093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17863713#comment-17863713 ] Tom Kalmijn commented on KAFKA-17093: - According to the documentation "If isolation. level=read_committed, the end offset will be the Last Stable Offset, i.e., the offset of the first message with an open transaction." In my case I dont think there are any open transactions as no new messages are coming in. However, there are trailing transaction markers from previous (possibly aborted) transactional messages. So the documentation may not be clear about which offset should be returned in this scenario? > KafkaConsumer.seekToEnd should return LSO > -- > > Key: KAFKA-17093 > URL: https://issues.apache.org/jira/browse/KAFKA-17093 > Project: Kafka > Issue Type: Bug > Components: consumer >Affects Versions: 3.6.1 > Environment: Ubuntu, IntelliJ, Scala "org.apache.kafka" % > "kafka-clients" % "3.6.1" >Reporter: Tom Kalmijn >Priority: Major > > > Expected > When using a transactional producer then the method > KafkaConsumer.seekToEnd(...) of a consumer configured with isolation level > "read_committed" should return the LSO. > Observed > The offset returned is always the actual last offset of the partition, which > is not the LSO if the latest offsets are occupied by transaction markers. > Also see this Slack thread: > https://confluentcommunity.slack.com/archives/C499EFQS0/p1720088282557559 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-17093) KafkaConsumer.seekToEnd should return LSO
[ https://issues.apache.org/jira/browse/KAFKA-17093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom Kalmijn updated KAFKA-17093: Attachment: Main.java > KafkaConsumer.seekToEnd should return LSO > -- > > Key: KAFKA-17093 > URL: https://issues.apache.org/jira/browse/KAFKA-17093 > Project: Kafka > Issue Type: Bug > Components: clients, consumer >Affects Versions: 3.6.1 > Environment: Ubuntu, IntelliJ, Scala "org.apache.kafka" % > "kafka-clients" % "3.6.1" >Reporter: Tom Kalmijn >Assignee: Andrew Schofield >Priority: Major > Attachments: Kafka17093.java, Main.java > > > > Expected > When using a transactional producer then the method > KafkaConsumer.seekToEnd(...) of a consumer configured with isolation level > "read_committed" should return the LSO. > Observed > The offset returned is always the actual last offset of the partition, which > is not the LSO if the latest offsets are occupied by transaction markers. > Also see this Slack thread: > https://confluentcommunity.slack.com/archives/C499EFQS0/p1720088282557559 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-17093) KafkaConsumer.seekToEnd should return LSO
[ https://issues.apache.org/jira/browse/KAFKA-17093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom Kalmijn updated KAFKA-17093: Attachment: (was: Main.java) > KafkaConsumer.seekToEnd should return LSO > -- > > Key: KAFKA-17093 > URL: https://issues.apache.org/jira/browse/KAFKA-17093 > Project: Kafka > Issue Type: Bug > Components: clients, consumer >Affects Versions: 3.6.1 > Environment: Ubuntu, IntelliJ, Scala "org.apache.kafka" % > "kafka-clients" % "3.6.1" >Reporter: Tom Kalmijn >Assignee: Andrew Schofield >Priority: Major > Attachments: Kafka17093.java > > > > Expected > When using a transactional producer then the method > KafkaConsumer.seekToEnd(...) of a consumer configured with isolation level > "read_committed" should return the LSO. > Observed > The offset returned is always the actual last offset of the partition, which > is not the LSO if the latest offsets are occupied by transaction markers. > Also see this Slack thread: > https://confluentcommunity.slack.com/archives/C499EFQS0/p1720088282557559 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-17093) KafkaConsumer.seekToEnd should return LSO
[ https://issues.apache.org/jira/browse/KAFKA-17093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom Kalmijn updated KAFKA-17093: Attachment: Kafka17093-v2.java > KafkaConsumer.seekToEnd should return LSO > -- > > Key: KAFKA-17093 > URL: https://issues.apache.org/jira/browse/KAFKA-17093 > Project: Kafka > Issue Type: Bug > Components: clients, consumer >Affects Versions: 3.6.1 > Environment: Ubuntu, IntelliJ, Scala "org.apache.kafka" % > "kafka-clients" % "3.6.1" >Reporter: Tom Kalmijn >Assignee: Andrew Schofield >Priority: Major > Attachments: Kafka17093-v2.java, Kafka17093.java > > > > Expected > When using a transactional producer then the method > KafkaConsumer.seekToEnd(...) of a consumer configured with isolation level > "read_committed" should return the LSO. > Observed > The offset returned is always the actual last offset of the partition, which > is not the LSO if the latest offsets are occupied by transaction markers. > Also see this Slack thread: > https://confluentcommunity.slack.com/archives/C499EFQS0/p1720088282557559 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-17093) KafkaConsumer.seekToEnd should return LSO
[ https://issues.apache.org/jira/browse/KAFKA-17093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17864234#comment-17864234 ] Tom Kalmijn commented on KAFKA-17093: - [~schofielaj] added now version of your test app with my additions and comments. > KafkaConsumer.seekToEnd should return LSO > -- > > Key: KAFKA-17093 > URL: https://issues.apache.org/jira/browse/KAFKA-17093 > Project: Kafka > Issue Type: Bug > Components: clients, consumer >Affects Versions: 3.6.1 > Environment: Ubuntu, IntelliJ, Scala "org.apache.kafka" % > "kafka-clients" % "3.6.1" >Reporter: Tom Kalmijn >Assignee: Andrew Schofield >Priority: Major > Attachments: Kafka17093-v2.java, Kafka17093.java > > > > Expected > When using a transactional producer then the method > KafkaConsumer.seekToEnd(...) of a consumer configured with isolation level > "read_committed" should return the LSO. > Observed > The offset returned is always the actual last offset of the partition, which > is not the LSO if the latest offsets are occupied by transaction markers. > Also see this Slack thread: > https://confluentcommunity.slack.com/archives/C499EFQS0/p1720088282557559 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (KAFKA-17093) KafkaConsumer.seekToEnd should return LSO
[ https://issues.apache.org/jira/browse/KAFKA-17093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17864234#comment-17864234 ] Tom Kalmijn edited comment on KAFKA-17093 at 7/9/24 2:31 PM: - [~schofielaj] added new version of your test app with my additions and comments. (removed poll statements as they seem redundant) was (Author: JIRAUSER306103): [~schofielaj] added new version of your test app with my additions and comments. > KafkaConsumer.seekToEnd should return LSO > -- > > Key: KAFKA-17093 > URL: https://issues.apache.org/jira/browse/KAFKA-17093 > Project: Kafka > Issue Type: Bug > Components: clients, consumer >Affects Versions: 3.6.1 > Environment: Ubuntu, IntelliJ, Scala "org.apache.kafka" % > "kafka-clients" % "3.6.1" >Reporter: Tom Kalmijn >Assignee: Andrew Schofield >Priority: Major > Attachments: Kafka17093-v2.java, Kafka17093.java > > > > Expected > When using a transactional producer then the method > KafkaConsumer.seekToEnd(...) of a consumer configured with isolation level > "read_committed" should return the LSO. > Observed > The offset returned is always the actual last offset of the partition, which > is not the LSO if the latest offsets are occupied by transaction markers. > Also see this Slack thread: > https://confluentcommunity.slack.com/archives/C499EFQS0/p1720088282557559 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (KAFKA-17093) KafkaConsumer.seekToEnd should return LSO
[ https://issues.apache.org/jira/browse/KAFKA-17093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17864234#comment-17864234 ] Tom Kalmijn edited comment on KAFKA-17093 at 7/9/24 2:31 PM: - [~schofielaj] added new version of your test app with my additions and comments. was (Author: JIRAUSER306103): [~schofielaj] added now version of your test app with my additions and comments. > KafkaConsumer.seekToEnd should return LSO > -- > > Key: KAFKA-17093 > URL: https://issues.apache.org/jira/browse/KAFKA-17093 > Project: Kafka > Issue Type: Bug > Components: clients, consumer >Affects Versions: 3.6.1 > Environment: Ubuntu, IntelliJ, Scala "org.apache.kafka" % > "kafka-clients" % "3.6.1" >Reporter: Tom Kalmijn >Assignee: Andrew Schofield >Priority: Major > Attachments: Kafka17093-v2.java, Kafka17093.java > > > > Expected > When using a transactional producer then the method > KafkaConsumer.seekToEnd(...) of a consumer configured with isolation level > "read_committed" should return the LSO. > Observed > The offset returned is always the actual last offset of the partition, which > is not the LSO if the latest offsets are occupied by transaction markers. > Also see this Slack thread: > https://confluentcommunity.slack.com/archives/C499EFQS0/p1720088282557559 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-17093) KafkaConsumer.seekToEnd should return LSO
[
https://issues.apache.org/jira/browse/KAFKA-17093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17864315#comment-17864315
]
Tom Kalmijn commented on KAFKA-17093:
-
Hi Andrew,
Great efforts.
My service starts up in a 'loading' state, reading all messages from the start
(earliest) of a (compacted) topic. After all current messages have been loaded
it should transition into its 'ready' state but it must keep on listening for
new messages.
In speudo code this is what I am trying to accomplish:
var lastOffset = determine offset off last message
while (true) {
read record
if record.offset == lastOffset then transition service from 'loading' into
'ready' state
}
Now the challenge is: if lastOffset is more advanced then the offset of the
last record (at start up) then it will never transition into 'ready' unless new
messages come in. But new message cannot come in unless the service is in the
'ready' state. Catch 22.
Also, I have a solution based on back tracking and polling, which works,
however it takes about 500 millis per offset to get the last offset this way.
This delay is slightly problematic for me since there are thousands of such
loading processes in a fairly short time frame.
If you tell me this is the only way to go, so be it.
But it does beg the question: why does seekToEnd make a distinction between
read_committed and read_uncommitted? read_committed consumers will skip the
control records anyway? Unless seekToEnd has more meaning, actually telling me
where the last message is.
> KafkaConsumer.seekToEnd should return LSO
> --
>
> Key: KAFKA-17093
> URL: https://issues.apache.org/jira/browse/KAFKA-17093
> Project: Kafka
> Issue Type: Bug
> Components: clients, consumer
>Affects Versions: 3.6.1
> Environment: Ubuntu, IntelliJ, Scala "org.apache.kafka" %
> "kafka-clients" % "3.6.1"
>Reporter: Tom Kalmijn
>Assignee: Andrew Schofield
>Priority: Major
> Attachments: Kafka17093-v2.java, Kafka17093-v3.java, Kafka17093.java
>
>
>
> Expected
> When using a transactional producer then the method
> KafkaConsumer.seekToEnd(...) of a consumer configured with isolation level
> "read_committed" should return the LSO.
> Observed
> The offset returned is always the actual last offset of the partition, which
> is not the LSO if the latest offsets are occupied by transaction markers.
> Also see this Slack thread:
> https://confluentcommunity.slack.com/archives/C499EFQS0/p1720088282557559
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (KAFKA-17093) KafkaConsumer.seekToEnd should return LSO
[
https://issues.apache.org/jira/browse/KAFKA-17093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17864315#comment-17864315
]
Tom Kalmijn edited comment on KAFKA-17093 at 7/9/24 5:46 PM:
-
Hi Andrew,
Great efforts.
My service starts up in a 'loading' state, reading all messages from the start
(earliest) of a (compacted) topic. After all current messages have been loaded
it should transition into its 'ready' state but it must keep on listening for
new messages.
In speudo code this is what I am trying to accomplish:
{quote}var lastOffset = determine offset off last message
while (true)
{
read record
if record.offset == lastOffset then transition 'loading' into 'ready' state
}
{quote}
Now the challenge is: if lastOffset is more advanced then the offset of the
last record (at start up) then it will never transition into 'ready' unless new
messages come in. But new message cannot come in unless the service is in the
'ready' state. Catch 22.
Also, I have a solution based on back tracking and polling, which works,
however it takes about 500 millis per offset to get the last offset this way.
This delay is slightly problematic for me since there are thousands of such
loading processes in a fairly short time frame.
If you tell me this is the only way to go, so be it.
But it does beg the question: why does seekToEnd make a distinction between
read_committed and read_uncommitted? read_committed consumers will skip the
control records anyway? Unless seekToEnd has more meaning, actually telling me
where the last message is.
was (Author: JIRAUSER306103):
Hi Andrew,
Great efforts.
My service starts up in a 'loading' state, reading all messages from the start
(earliest) of a (compacted) topic. After all current messages have been loaded
it should transition into its 'ready' state but it must keep on listening for
new messages.
In speudo code this is what I am trying to accomplish:
var lastOffset = determine offset off last message
while (true) {
read record
if record.offset == lastOffset then transition service from 'loading' into
'ready' state
}
Now the challenge is: if lastOffset is more advanced then the offset of the
last record (at start up) then it will never transition into 'ready' unless new
messages come in. But new message cannot come in unless the service is in the
'ready' state. Catch 22.
Also, I have a solution based on back tracking and polling, which works,
however it takes about 500 millis per offset to get the last offset this way.
This delay is slightly problematic for me since there are thousands of such
loading processes in a fairly short time frame.
If you tell me this is the only way to go, so be it.
But it does beg the question: why does seekToEnd make a distinction between
read_committed and read_uncommitted? read_committed consumers will skip the
control records anyway? Unless seekToEnd has more meaning, actually telling me
where the last message is.
> KafkaConsumer.seekToEnd should return LSO
> --
>
> Key: KAFKA-17093
> URL: https://issues.apache.org/jira/browse/KAFKA-17093
> Project: Kafka
> Issue Type: Bug
> Components: clients, consumer
>Affects Versions: 3.6.1
> Environment: Ubuntu, IntelliJ, Scala "org.apache.kafka" %
> "kafka-clients" % "3.6.1"
>Reporter: Tom Kalmijn
>Assignee: Andrew Schofield
>Priority: Major
> Attachments: Kafka17093-v2.java, Kafka17093-v3.java, Kafka17093.java
>
>
>
> Expected
> When using a transactional producer then the method
> KafkaConsumer.seekToEnd(...) of a consumer configured with isolation level
> "read_committed" should return the LSO.
> Observed
> The offset returned is always the actual last offset of the partition, which
> is not the LSO if the latest offsets are occupied by transaction markers.
> Also see this Slack thread:
> https://confluentcommunity.slack.com/archives/C499EFQS0/p1720088282557559
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (KAFKA-17093) KafkaConsumer.seekToEnd should return LSO
[
https://issues.apache.org/jira/browse/KAFKA-17093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17864315#comment-17864315
]
Tom Kalmijn edited comment on KAFKA-17093 at 7/9/24 5:47 PM:
-
Hi [~schofielaj],
Great efforts.
My service starts up in a 'loading' state, reading all messages from the start
(earliest) of a (compacted) topic. After all current messages have been loaded
it should transition into its 'ready' state but it must keep on listening for
new messages.
In speudo code this is what I am trying to accomplish:
{quote}var lastOffset = determine offset off last message
while (true) {
read record
if record.offset == lastOffset then transition 'loading' into'ready'
state
}{quote}
Now the challenge is: if lastOffset is more advanced then the offset of the
last record (at start up) then it will never transition into 'ready' unless new
messages come in. But new message cannot come in unless the service is in the
'ready' state. Catch 22.
Also, I have a solution based on back tracking and polling, which works,
however it takes about 500 millis per offset to get the last offset this way.
This delay is slightly problematic for me since there are thousands of such
loading processes in a fairly short time frame.
If you tell me this is the only way to go, so be it.
But it does beg the question: why does seekToEnd make a distinction between
read_committed and read_uncommitted? read_committed consumers will skip the
control records anyway? Unless seekToEnd has more meaning, actually telling me
where the last message is.
was (Author: JIRAUSER306103):
Hi Andrew,
Great efforts.
My service starts up in a 'loading' state, reading all messages from the start
(earliest) of a (compacted) topic. After all current messages have been loaded
it should transition into its 'ready' state but it must keep on listening for
new messages.
In speudo code this is what I am trying to accomplish:
{quote}var lastOffset = determine offset off last message
while (true)
{
read record
if record.offset == lastOffset then transition 'loading' into 'ready' state
}
{quote}
Now the challenge is: if lastOffset is more advanced then the offset of the
last record (at start up) then it will never transition into 'ready' unless new
messages come in. But new message cannot come in unless the service is in the
'ready' state. Catch 22.
Also, I have a solution based on back tracking and polling, which works,
however it takes about 500 millis per offset to get the last offset this way.
This delay is slightly problematic for me since there are thousands of such
loading processes in a fairly short time frame.
If you tell me this is the only way to go, so be it.
But it does beg the question: why does seekToEnd make a distinction between
read_committed and read_uncommitted? read_committed consumers will skip the
control records anyway? Unless seekToEnd has more meaning, actually telling me
where the last message is.
> KafkaConsumer.seekToEnd should return LSO
> --
>
> Key: KAFKA-17093
> URL: https://issues.apache.org/jira/browse/KAFKA-17093
> Project: Kafka
> Issue Type: Bug
> Components: clients, consumer
>Affects Versions: 3.6.1
> Environment: Ubuntu, IntelliJ, Scala "org.apache.kafka" %
> "kafka-clients" % "3.6.1"
>Reporter: Tom Kalmijn
>Assignee: Andrew Schofield
>Priority: Major
> Attachments: Kafka17093-v2.java, Kafka17093-v3.java, Kafka17093.java
>
>
>
> Expected
> When using a transactional producer then the method
> KafkaConsumer.seekToEnd(...) of a consumer configured with isolation level
> "read_committed" should return the LSO.
> Observed
> The offset returned is always the actual last offset of the partition, which
> is not the LSO if the latest offsets are occupied by transaction markers.
> Also see this Slack thread:
> https://confluentcommunity.slack.com/archives/C499EFQS0/p1720088282557559
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (KAFKA-17093) KafkaConsumer.seekToEnd should return LSO
[
https://issues.apache.org/jira/browse/KAFKA-17093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17864315#comment-17864315
]
Tom Kalmijn edited comment on KAFKA-17093 at 7/9/24 5:48 PM:
-
Hi [~schofielaj],
Great efforts.
My service starts up in a 'loading' state, reading all messages from the start
(earliest) of a (compacted) topic. After all current messages have been loaded
it should transition into its 'ready' state but it must keep on listening for
new messages.
In speudo code this is what I am trying to accomplish:
{quote}
{code:java}
var lastOffset = determine offset off last message
while (true) {
read record
if record.offset == lastOffset then
transition from 'loading' into'ready' state
}
{code}
{quote}
Now the challenge is: if lastOffset is more advanced then the offset of the
last record (at start up) then it will never transition into 'ready' unless new
messages come in. But new message cannot come in unless the service is in the
'ready' state. Catch 22.
Also, I have a solution based on back tracking and polling, which works,
however it takes about 500 millis per offset to get the last offset this way.
This delay is slightly problematic for me since there are thousands of such
loading processes in a fairly short time frame.
If you tell me this is the only way to go, so be it.
But it does beg the question: why does seekToEnd make a distinction between
read_committed and read_uncommitted? read_committed consumers will skip the
control records anyway? Unless seekToEnd has more meaning, actually telling me
where the last message is.
was (Author: JIRAUSER306103):
Hi [~schofielaj],
Great efforts.
My service starts up in a 'loading' state, reading all messages from the start
(earliest) of a (compacted) topic. After all current messages have been loaded
it should transition into its 'ready' state but it must keep on listening for
new messages.
In speudo code this is what I am trying to accomplish:
{quote}var lastOffset = determine offset off last message
while (true) {
read record
if record.offset == lastOffset then transition 'loading' into'ready'
state
}{quote}
Now the challenge is: if lastOffset is more advanced then the offset of the
last record (at start up) then it will never transition into 'ready' unless new
messages come in. But new message cannot come in unless the service is in the
'ready' state. Catch 22.
Also, I have a solution based on back tracking and polling, which works,
however it takes about 500 millis per offset to get the last offset this way.
This delay is slightly problematic for me since there are thousands of such
loading processes in a fairly short time frame.
If you tell me this is the only way to go, so be it.
But it does beg the question: why does seekToEnd make a distinction between
read_committed and read_uncommitted? read_committed consumers will skip the
control records anyway? Unless seekToEnd has more meaning, actually telling me
where the last message is.
> KafkaConsumer.seekToEnd should return LSO
> --
>
> Key: KAFKA-17093
> URL: https://issues.apache.org/jira/browse/KAFKA-17093
> Project: Kafka
> Issue Type: Bug
> Components: clients, consumer
>Affects Versions: 3.6.1
> Environment: Ubuntu, IntelliJ, Scala "org.apache.kafka" %
> "kafka-clients" % "3.6.1"
>Reporter: Tom Kalmijn
>Assignee: Andrew Schofield
>Priority: Major
> Attachments: Kafka17093-v2.java, Kafka17093-v3.java, Kafka17093.java
>
>
>
> Expected
> When using a transactional producer then the method
> KafkaConsumer.seekToEnd(...) of a consumer configured with isolation level
> "read_committed" should return the LSO.
> Observed
> The offset returned is always the actual last offset of the partition, which
> is not the LSO if the latest offsets are occupied by transaction markers.
> Also see this Slack thread:
> https://confluentcommunity.slack.com/archives/C499EFQS0/p1720088282557559
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (KAFKA-17093) KafkaConsumer.seekToEnd should return LSO
[
https://issues.apache.org/jira/browse/KAFKA-17093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17864315#comment-17864315
]
Tom Kalmijn edited comment on KAFKA-17093 at 7/9/24 5:49 PM:
-
Hi [~schofielaj],
Great efforts.
My service starts up in a 'loading' state, reading all messages from the start
(earliest) of a (compacted) topic. After all current messages have been loaded
it should transition into its 'ready' state but it must keep on listening for
new messages.
In speudo code this is what I am trying to accomplish:
{quote}
{code:java}
var lastOffset = determine offset off last message
while (true) {
read record
if (record.offset == lastOffset) {
transition from 'loading' into 'ready' state
}
}
{code}
{quote}
Now the challenge is: if lastOffset is more advanced then the offset of the
last record (at start up) then it will never transition into 'ready' unless new
messages come in. But new message cannot come in unless the service is in the
'ready' state. Catch 22.
Also, I have a solution based on back tracking and polling, which works,
however it takes about 500 millis per offset to get the last offset this way.
This delay is slightly problematic for me since there are thousands of such
loading processes in a fairly short time frame.
If you tell me this is the only way to go, so be it.
But it does beg the question: why does seekToEnd make a distinction between
read_committed and read_uncommitted? read_committed consumers will skip the
control records anyway? Unless seekToEnd has more meaning, actually telling me
where the last message is.
was (Author: JIRAUSER306103):
Hi [~schofielaj],
Great efforts.
My service starts up in a 'loading' state, reading all messages from the start
(earliest) of a (compacted) topic. After all current messages have been loaded
it should transition into its 'ready' state but it must keep on listening for
new messages.
In speudo code this is what I am trying to accomplish:
{quote}
{code:java}
var lastOffset = determine offset off last message
while (true) {
read record
if record.offset == lastOffset then
transition from 'loading' into'ready' state
}
{code}
{quote}
Now the challenge is: if lastOffset is more advanced then the offset of the
last record (at start up) then it will never transition into 'ready' unless new
messages come in. But new message cannot come in unless the service is in the
'ready' state. Catch 22.
Also, I have a solution based on back tracking and polling, which works,
however it takes about 500 millis per offset to get the last offset this way.
This delay is slightly problematic for me since there are thousands of such
loading processes in a fairly short time frame.
If you tell me this is the only way to go, so be it.
But it does beg the question: why does seekToEnd make a distinction between
read_committed and read_uncommitted? read_committed consumers will skip the
control records anyway? Unless seekToEnd has more meaning, actually telling me
where the last message is.
> KafkaConsumer.seekToEnd should return LSO
> --
>
> Key: KAFKA-17093
> URL: https://issues.apache.org/jira/browse/KAFKA-17093
> Project: Kafka
> Issue Type: Bug
> Components: clients, consumer
>Affects Versions: 3.6.1
> Environment: Ubuntu, IntelliJ, Scala "org.apache.kafka" %
> "kafka-clients" % "3.6.1"
>Reporter: Tom Kalmijn
>Assignee: Andrew Schofield
>Priority: Major
> Attachments: Kafka17093-v2.java, Kafka17093-v3.java, Kafka17093.java
>
>
>
> Expected
> When using a transactional producer then the method
> KafkaConsumer.seekToEnd(...) of a consumer configured with isolation level
> "read_committed" should return the LSO.
> Observed
> The offset returned is always the actual last offset of the partition, which
> is not the LSO if the latest offsets are occupied by transaction markers.
> Also see this Slack thread:
> https://confluentcommunity.slack.com/archives/C499EFQS0/p1720088282557559
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (KAFKA-17093) KafkaConsumer.seekToEnd should return LSO
[ https://issues.apache.org/jira/browse/KAFKA-17093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17864316#comment-17864316 ] Tom Kalmijn commented on KAFKA-17093: - By the way, with the addtion of your fix for adding an extra trailing control record the code now resembles my situation quite accurately, thanks. > KafkaConsumer.seekToEnd should return LSO > -- > > Key: KAFKA-17093 > URL: https://issues.apache.org/jira/browse/KAFKA-17093 > Project: Kafka > Issue Type: Bug > Components: clients, consumer >Affects Versions: 3.6.1 > Environment: Ubuntu, IntelliJ, Scala "org.apache.kafka" % > "kafka-clients" % "3.6.1" >Reporter: Tom Kalmijn >Assignee: Andrew Schofield >Priority: Major > Attachments: Kafka17093-v2.java, Kafka17093-v3.java, Kafka17093.java > > > > Expected > When using a transactional producer then the method > KafkaConsumer.seekToEnd(...) of a consumer configured with isolation level > "read_committed" should return the LSO. > Observed > The offset returned is always the actual last offset of the partition, which > is not the LSO if the latest offsets are occupied by transaction markers. > Also see this Slack thread: > https://confluentcommunity.slack.com/archives/C499EFQS0/p1720088282557559 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (KAFKA-17093) KafkaConsumer.seekToEnd should return LSO
[ https://issues.apache.org/jira/browse/KAFKA-17093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17864316#comment-17864316 ] Tom Kalmijn edited comment on KAFKA-17093 at 7/9/24 5:53 PM: - By the way, with the addition of your fix for adding an extra trailing control record the code now resembles my situation quite accurately, thanks. was (Author: JIRAUSER306103): By the way, with the addtion of your fix for adding an extra trailing control record the code now resembles my situation quite accurately, thanks. > KafkaConsumer.seekToEnd should return LSO > -- > > Key: KAFKA-17093 > URL: https://issues.apache.org/jira/browse/KAFKA-17093 > Project: Kafka > Issue Type: Bug > Components: clients, consumer >Affects Versions: 3.6.1 > Environment: Ubuntu, IntelliJ, Scala "org.apache.kafka" % > "kafka-clients" % "3.6.1" >Reporter: Tom Kalmijn >Assignee: Andrew Schofield >Priority: Major > Attachments: Kafka17093-v2.java, Kafka17093-v3.java, Kafka17093.java > > > > Expected > When using a transactional producer then the method > KafkaConsumer.seekToEnd(...) of a consumer configured with isolation level > "read_committed" should return the LSO. > Observed > The offset returned is always the actual last offset of the partition, which > is not the LSO if the latest offsets are occupied by transaction markers. > Also see this Slack thread: > https://confluentcommunity.slack.com/archives/C499EFQS0/p1720088282557559 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (KAFKA-17093) KafkaConsumer.seekToEnd should return LSO
[
https://issues.apache.org/jira/browse/KAFKA-17093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17864508#comment-17864508
]
Tom Kalmijn edited comment on KAFKA-17093 at 7/10/24 7:13 AM:
--
I think at least I (too) now understand why there is a read_committed flag for
seekToEnd. If not specified seekToEnd could possibly scroll passed an ongoing
uncommitted transaction and this can result in a bug where a consumer misses a
record.
*Let's conclude that seekToEnd works as designed and as documented?*
But is there any method at all that tells us the offset of the last message in
a topic/partition? I find it mind boggling if that really does not exist. I
have seen so many threads where people are throwing each other this "last
offset minus one" logic to get to this information. It is clearly something
more people need in their designs.
There is OffsetSpec for the adminClient:
{code:java}
request = Map(topicPartition -> OffsetSpec.maxTimestamp)
adminClient.listOffsets(request, options = new
ListOffsetsOptions(READ_COMMITTED)
{code}
This yields the same result as discussed above (NOT the offset of the last
message).
OffsetSpec currently supports: 'latest', 'earliest', 'forTimestamp' and
'maxTimestamp'.
If only it would include another option like 'latestCommittedMessage', that
would be very helpful.
was (Author: JIRAUSER306103):
I think at least I (too) now understand why there is a read_committed flag for
seekToEnd. If not specified seekToEnd could possibly scroll passed an ongoing
uncommitted transaction and this can result in a bug where a consumer misses a
record.
*Let's conclude that seekToEnd works as designed and as documented?*
But is there any method at all that tells us the offset of the last message in
a topic/partition? I find it mind boggling if that really does not exist. I
have seen so many threads where people are throwing each other this "last
offset minus one" logic to get to this information. It is clearly something
more people need in their designs.
There is OffsetSpec for the adminClient:
{code:java}
request = Map(topicPartition -> OffsetSpec.maxTimestamp)
adminClient.listOffsets(request, options = new
ListOffsetsOptions(READ_COMMITTED)
{code}
This yields the same result as discussed above (NOT the offset of the last
message).
OffsetSpec currently supports: 'latest', 'earliest', 'forTimestamp' and
'maxTimestamp'.
If only it would include another option like 'latestCommittedMessage', that
would be very helpful.
> KafkaConsumer.seekToEnd should return LSO
> --
>
> Key: KAFKA-17093
> URL: https://issues.apache.org/jira/browse/KAFKA-17093
> Project: Kafka
> Issue Type: Bug
> Components: clients, consumer
>Affects Versions: 3.6.1
> Environment: Ubuntu, IntelliJ, Scala "org.apache.kafka" %
> "kafka-clients" % "3.6.1"
>Reporter: Tom Kalmijn
>Assignee: Andrew Schofield
>Priority: Major
> Attachments: Kafka17093-v2.java, Kafka17093-v3.java, Kafka17093.java
>
>
>
> Expected
> When using a transactional producer then the method
> KafkaConsumer.seekToEnd(...) of a consumer configured with isolation level
> "read_committed" should return the LSO.
> Observed
> The offset returned is always the actual last offset of the partition, which
> is not the LSO if the latest offsets are occupied by transaction markers.
> Also see this Slack thread:
> https://confluentcommunity.slack.com/archives/C499EFQS0/p1720088282557559
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (KAFKA-17093) KafkaConsumer.seekToEnd should return LSO
[
https://issues.apache.org/jira/browse/KAFKA-17093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17864508#comment-17864508
]
Tom Kalmijn edited comment on KAFKA-17093 at 7/10/24 7:14 AM:
--
I think at least I (too) now understand why there is a read_committed flag for
seekToEnd. If not specified seekToEnd could possibly scroll passed an ongoing
uncommitted transaction and this can result in a bug where a consumer misses a
record.
*Let's conclude that seekToEnd works as designed and as documented?*
But is there any method at all that tells us the offset of the last message in
a topic/partition? I find it mind boggling if that really does not exist. I
have seen so many threads where people are throwing each other this "last
offset minus one" logic to get to this information. It is clearly something
more people need in their designs.
There is OffsetSpec for the adminClient:
{code:java}
request = Map(topicPartition -> OffsetSpec.maxTimestamp)
adminClient.listOffsets(request, options = new
ListOffsetsOptions(READ_COMMITTED))
{code}
This yields the same result as discussed above (NOT the offset of the last
message).
OffsetSpec currently supports: 'latest', 'earliest', 'forTimestamp' and
'maxTimestamp'.
If only it would include another option like 'latestCommittedMessage', that
would be very helpful.
was (Author: JIRAUSER306103):
I think at least I (too) now understand why there is a read_committed flag for
seekToEnd. If not specified seekToEnd could possibly scroll passed an ongoing
uncommitted transaction and this can result in a bug where a consumer misses a
record.
*Let's conclude that seekToEnd works as designed and as documented?*
But is there any method at all that tells us the offset of the last message in
a topic/partition? I find it mind boggling if that really does not exist. I
have seen so many threads where people are throwing each other this "last
offset minus one" logic to get to this information. It is clearly something
more people need in their designs.
There is OffsetSpec for the adminClient:
{code:java}
request = Map(topicPartition -> OffsetSpec.maxTimestamp)
adminClient.listOffsets(request, options = new
ListOffsetsOptions(READ_COMMITTED)
{code}
This yields the same result as discussed above (NOT the offset of the last
message).
OffsetSpec currently supports: 'latest', 'earliest', 'forTimestamp' and
'maxTimestamp'.
If only it would include another option like 'latestCommittedMessage', that
would be very helpful.
> KafkaConsumer.seekToEnd should return LSO
> --
>
> Key: KAFKA-17093
> URL: https://issues.apache.org/jira/browse/KAFKA-17093
> Project: Kafka
> Issue Type: Bug
> Components: clients, consumer
>Affects Versions: 3.6.1
> Environment: Ubuntu, IntelliJ, Scala "org.apache.kafka" %
> "kafka-clients" % "3.6.1"
>Reporter: Tom Kalmijn
>Assignee: Andrew Schofield
>Priority: Major
> Attachments: Kafka17093-v2.java, Kafka17093-v3.java, Kafka17093.java
>
>
>
> Expected
> When using a transactional producer then the method
> KafkaConsumer.seekToEnd(...) of a consumer configured with isolation level
> "read_committed" should return the LSO.
> Observed
> The offset returned is always the actual last offset of the partition, which
> is not the LSO if the latest offsets are occupied by transaction markers.
> Also see this Slack thread:
> https://confluentcommunity.slack.com/archives/C499EFQS0/p1720088282557559
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (KAFKA-17093) KafkaConsumer.seekToEnd should return LSO
[
https://issues.apache.org/jira/browse/KAFKA-17093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17864508#comment-17864508
]
Tom Kalmijn commented on KAFKA-17093:
-
I think at least I (too) now understand why there is a read_committed flag for
seekToEnd. If not specified seekToEnd could possibly scroll passed an ongoing
uncommitted transaction and this can result in a bug where a consumer misses a
record.
*Let's conclude that seekToEnd works as designed and as documented?*
But is there any method at all that tells us the offset of the last message in
a topic/partition? I find it mind boggling if that really does not exist. I
have seen so many threads where people are throwing each other this "last
offset minus one" logic to get to this information. It is clearly something
more people need in their designs.
There is OffsetSpec for the adminClient:
{code:java}
request = Map(topicPartition -> OffsetSpec.maxTimestamp)
adminClient.listOffsets(request, options = new
ListOffsetsOptions(READ_COMMITTED)
{code}
This yields the same result as discussed above (NOT the offset of the last
message).
OffsetSpec currently supports: 'latest', 'earliest', 'forTimestamp' and
'maxTimestamp'.
If only it would include another option like 'latestCommittedMessage', that
would be very helpful.
> KafkaConsumer.seekToEnd should return LSO
> --
>
> Key: KAFKA-17093
> URL: https://issues.apache.org/jira/browse/KAFKA-17093
> Project: Kafka
> Issue Type: Bug
> Components: clients, consumer
>Affects Versions: 3.6.1
> Environment: Ubuntu, IntelliJ, Scala "org.apache.kafka" %
> "kafka-clients" % "3.6.1"
>Reporter: Tom Kalmijn
>Assignee: Andrew Schofield
>Priority: Major
> Attachments: Kafka17093-v2.java, Kafka17093-v3.java, Kafka17093.java
>
>
>
> Expected
> When using a transactional producer then the method
> KafkaConsumer.seekToEnd(...) of a consumer configured with isolation level
> "read_committed" should return the LSO.
> Observed
> The offset returned is always the actual last offset of the partition, which
> is not the LSO if the latest offsets are occupied by transaction markers.
> Also see this Slack thread:
> https://confluentcommunity.slack.com/archives/C499EFQS0/p1720088282557559
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (KAFKA-17093) KafkaConsumer.seekToEnd should return LSO
[
https://issues.apache.org/jira/browse/KAFKA-17093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17864508#comment-17864508
]
Tom Kalmijn edited comment on KAFKA-17093 at 7/10/24 7:27 AM:
--
[~schofielaj] I think at least I (too) now understand why there is a
read_committed flag for seekToEnd. If not specified seekToEnd could possibly
scroll passed an ongoing uncommitted transaction and this can result in a bug
where a consumer misses a record.
*Let's conclude that seekToEnd works as designed and as documented?*
But is there any method at all that tells us the offset of the last message in
a topic/partition? I find it mind boggling if that really does not exist. I
have seen so many threads where people are throwing each other this "last
offset minus one" logic to get to this information. It is clearly something
more people need in their designs.
There is OffsetSpec for the adminClient:
{code:java}
request = Map(topicPartition -> OffsetSpec.maxTimestamp)
adminClient.listOffsets(request, options = new
ListOffsetsOptions(READ_COMMITTED))
{code}
This yields the same result as discussed above (NOT the offset of the last
message).
OffsetSpec currently supports: 'latest', 'earliest', 'forTimestamp' and
'maxTimestamp'.
If only it would include another option like 'latestCommittedMessage', that
would be very helpful.
was (Author: JIRAUSER306103):
I think at least I (too) now understand why there is a read_committed flag for
seekToEnd. If not specified seekToEnd could possibly scroll passed an ongoing
uncommitted transaction and this can result in a bug where a consumer misses a
record.
*Let's conclude that seekToEnd works as designed and as documented?*
But is there any method at all that tells us the offset of the last message in
a topic/partition? I find it mind boggling if that really does not exist. I
have seen so many threads where people are throwing each other this "last
offset minus one" logic to get to this information. It is clearly something
more people need in their designs.
There is OffsetSpec for the adminClient:
{code:java}
request = Map(topicPartition -> OffsetSpec.maxTimestamp)
adminClient.listOffsets(request, options = new
ListOffsetsOptions(READ_COMMITTED))
{code}
This yields the same result as discussed above (NOT the offset of the last
message).
OffsetSpec currently supports: 'latest', 'earliest', 'forTimestamp' and
'maxTimestamp'.
If only it would include another option like 'latestCommittedMessage', that
would be very helpful.
> KafkaConsumer.seekToEnd should return LSO
> --
>
> Key: KAFKA-17093
> URL: https://issues.apache.org/jira/browse/KAFKA-17093
> Project: Kafka
> Issue Type: Bug
> Components: clients, consumer
>Affects Versions: 3.6.1
> Environment: Ubuntu, IntelliJ, Scala "org.apache.kafka" %
> "kafka-clients" % "3.6.1"
>Reporter: Tom Kalmijn
>Assignee: Andrew Schofield
>Priority: Major
> Attachments: Kafka17093-v2.java, Kafka17093-v3.java, Kafka17093.java
>
>
>
> Expected
> When using a transactional producer then the method
> KafkaConsumer.seekToEnd(...) of a consumer configured with isolation level
> "read_committed" should return the LSO.
> Observed
> The offset returned is always the actual last offset of the partition, which
> is not the LSO if the latest offsets are occupied by transaction markers.
> Also see this Slack thread:
> https://confluentcommunity.slack.com/archives/C499EFQS0/p1720088282557559
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (KAFKA-17093) KafkaConsumer.seekToEnd should return LSO
[ https://issues.apache.org/jira/browse/KAFKA-17093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17864609#comment-17864609 ] Tom Kalmijn commented on KAFKA-17093: - [~schofielaj] I am for in your debt for spending so much time on something that works perfectly fine (still mind boggled though). switching over to Slack > KafkaConsumer.seekToEnd should return LSO > -- > > Key: KAFKA-17093 > URL: https://issues.apache.org/jira/browse/KAFKA-17093 > Project: Kafka > Issue Type: Bug > Components: clients, consumer >Affects Versions: 3.6.1 > Environment: Ubuntu, IntelliJ, Scala "org.apache.kafka" % > "kafka-clients" % "3.6.1" >Reporter: Tom Kalmijn >Assignee: Andrew Schofield >Priority: Major > Attachments: Kafka17093-v2.java, Kafka17093-v3.java, Kafka17093.java > > > > Expected > When using a transactional producer then the method > KafkaConsumer.seekToEnd(...) of a consumer configured with isolation level > "read_committed" should return the LSO. > Observed > The offset returned is always the actual last offset of the partition, which > is not the LSO if the latest offsets are occupied by transaction markers. > Also see this Slack thread: > https://confluentcommunity.slack.com/archives/C499EFQS0/p1720088282557559 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (KAFKA-17093) KafkaConsumer.seekToEnd should return LSO
[ https://issues.apache.org/jira/browse/KAFKA-17093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17864609#comment-17864609 ] Tom Kalmijn edited comment on KAFKA-17093 at 7/10/24 11:04 AM: --- [~schofielaj] I am for ever in your debt for spending so much time on something that works perfectly fine (still mind boggled though). switching over to Slack was (Author: JIRAUSER306103): [~schofielaj] I am for in your debt for spending so much time on something that works perfectly fine (still mind boggled though). switching over to Slack > KafkaConsumer.seekToEnd should return LSO > -- > > Key: KAFKA-17093 > URL: https://issues.apache.org/jira/browse/KAFKA-17093 > Project: Kafka > Issue Type: Bug > Components: clients, consumer >Affects Versions: 3.6.1 > Environment: Ubuntu, IntelliJ, Scala "org.apache.kafka" % > "kafka-clients" % "3.6.1" >Reporter: Tom Kalmijn >Assignee: Andrew Schofield >Priority: Major > Attachments: Kafka17093-v2.java, Kafka17093-v3.java, Kafka17093.java > > > > Expected > When using a transactional producer then the method > KafkaConsumer.seekToEnd(...) of a consumer configured with isolation level > "read_committed" should return the LSO. > Observed > The offset returned is always the actual last offset of the partition, which > is not the LSO if the latest offsets are occupied by transaction markers. > Also see this Slack thread: > https://confluentcommunity.slack.com/archives/C499EFQS0/p1720088282557559 -- This message was sent by Atlassian Jira (v8.20.10#820010)
