Hi Team,
I understand that Kafka is meant for sequential access, but I have use case of
accessing the messages from Kafka based on random offsets and partitions.
For Example
There is topic called “topic-A” some consumers are listening from this topic,
it may happen that processing of these messages gets failed inside consumer so
in that case we will have to reprocess those messages.
We store the offset and partition details for the failed messages.
Another consumer will process these failed messages and put again to the topic,
so to reprocess those messages again we will be fetching those messages from
respective
topics based on offsets and partitions. So in this case we will have lots of
random offsets to access from different topics (our application should support
1000 messages/second including all topics).
So now my doubt comes here
Will accessing messages using random offsets and partitions invalidate the
cache for the topic that was used for sequential access (other consumers was
reading sequentially).
I want to understand the impact of this random offset, partition access on
kafka,
1. Will it slow down Kafka ?
2. Will it read messages from cache or from disk ?
3. If messages is not in cache then will it load from disk and store in
cache … in that case what will happen to the existing data that was in cache
and was being used for sequential access ?
I went through Kafka documentation but couldn’t answer to all my questions.
Need your help here.
Please let me know if my question is not clear or you need more info.
Regards
Rupesh
