Hello Kafka Developers , Users ,
We are exploring the SearchMessageByTimestamp feature in Kafka for our
use case .
Use Case : Kafka will be realtime message bus , users should be able to
pull Logs by specifying start_date and end_date or Pull me last five
minutes data etc ...
I did POC on SearchMessageByTimestamp , here is the code
https://gist.github.com/senthilec566/16e8e28b32834666fea132afc3a4e2f9 . And
i observed that Searching Messages is slow ..
Here is small test i did :
Query :Fetch Logs of Last *5 minutes*:
Result:
No of Records fetched : *30*
Fetch Time *6210* ms
Above test performed in a topic which has 4 partitions. In each partition
search & query processing happened .. in other words
consumer.offsetsForTimes()
consumer.assign(Arrays.asList(partition))
consumer.seek(this.partition, offsetTimestamp.offset())
consumer.poll(100)
are the API calls of each partition.. I realized that , this was the reason
for Kafka taking more time..
What is efficient way of implementing SerachMessageByTimeStamp ? Is Kafka
right candidate for our Use Case ?
Pls add your thoughts here ...
Cheers,
Senthil
