Hello Kafka Developers , Users , We are exploring the SearchMessageByTimestamp feature in Kafka for our use case .
Use Case : Kafka will be realtime message bus , users should be able to pull Logs by specifying start_date and end_date or Pull me last five minutes data etc ... I did POC on SearchMessageByTimestamp , here is the code https://gist.github.com/senthilec566/16e8e28b32834666fea132afc3a4e2f9 . And i observed that Searching Messages is slow .. Here is small test i did : Query :Fetch Logs of Last *5 minutes*: Result: No of Records fetched : *30* Fetch Time *6210* ms Above test performed in a topic which has 4 partitions. In each partition search & query processing happened .. in other words consumer.offsetsForTimes() consumer.assign(Arrays.asList(partition)) consumer.seek(this.partition, offsetTimestamp.offset()) consumer.poll(100) are the API calls of each partition.. I realized that , this was the reason for Kafka taking more time.. What is efficient way of implementing SerachMessageByTimeStamp ? Is Kafka right candidate for our Use Case ? Pls add your thoughts here ... Cheers, Senthil