Hello Kafka users,
I am using exactly once semantics provided by kafka. Therefore, my producer
writes a message within a transaction. While my producer was sending 100th
message, right between send() and commitTransaction(); I killed the producer
process.
I read last few uncommitted messages in my topic.
Consumer record offset - message number
0 - 1
2 - 2
196 - 99 <- Last committed message by producer
198 - 100 <- 100th message was sent but not committed
Now, when I run consumer with read_committed isolation level. It exactly reads
from 1-99 messages. But for that I have read entire topic. Eventually, I am
going to store millions of messages in a topic. So, reading entire topic is not
preferable.
I used other methods i.e.
seekToEnd() - took me to offset#200
endOffsets() - took me to offset#200
Is there a way to get message that was committed by the Kafka producer? (In my
case, Offset#196)
Why do I care about last committed message?
I am using a single-partitioned topic. Each producer sends meta-data to
{P_NAME}_{TOPIC_NAME}_INTERNAL topic. I am writing a restartable producer. To
make this possible, they need to read last committed message
{P_NAME}_{TOPIC_NAME}_INTERNAL topic and use the message.
I have also asked the same question on Stackoverflow to reach out to other
people:
https://stackoverflow.com/questions/55342872/get-last-committed-message-in-a-partition-kafka/55344057?noredirect=1#comment97414434_55344057
--Thanks
Jigar Rathod
