Re: Efficient way of Searching Messages By Timestamp - Kafka

[SenthilKumar K]

Hi Dev, It would be great if anybody share your experience on Search
Message by Timestamp ..

Cheer's,
Senthil

On May 28, 2017 2:08 AM, "SenthilKumar K"  wrote:

> Hi Team , Any help here Pls ?
>
> Cheers,
> Senthil
>
> On Sat, May 27, 2017 at 8:25 PM, SenthilKumar K 
> wrote:
>
>> Hello Kafka Developers , Users ,
>>
>> We are exploring the SearchMessageByTimestamp feature in Kafka for
>> our use case .
>>
>> Use Case : Kafka will be realtime message bus , users should be able
>> to pull Logs by specifying start_date and end_date or  Pull me last five
>> minutes data etc ...
>>
>> I did POC on SearchMessageByTimestamp , here is the code
>> https://gist.github.com/senthilec566/16e8e28b32834666fea132afc3a4e2f9 .
>> And i observed that Searching Messages is slow ..
>>
>> Here is small test i did :
>> Query :Fetch Logs of Last *5 minutes*:
>> Result:
>> No of Records fetched : *30*
>> Fetch Time *6210* ms
>>
>> Above test performed in a topic which has 4 partitions. In each partition
>> search & query processing happened .. in other words
>> consumer.offsetsForTimes()
>> consumer.assign(Arrays.asList(partition))
>> consumer.seek(this.partition, offsetTimestamp.offset())
>> consumer.poll(100)
>>
>> are the API calls of each partition.. I realized that , this was the
>> reason for Kafka taking more time..
>>
>> What is efficient way of implementing SerachMessageByTimeStamp ?  Is
>> Kafka right candidate for our Use Case ?
>>
>> Pls add your thoughts here ...
>>
>>
>> Cheers,
>> Senthil
>>
>
>

Re: Efficient way of Searching Messages By Timestamp - Kafka

[SenthilKumar K]

Hi Team , Any help here Pls ?

Cheers,
Senthil

On Sat, May 27, 2017 at 8:25 PM, SenthilKumar K 
wrote:

> Hello Kafka Developers , Users ,
>
> We are exploring the SearchMessageByTimestamp feature in Kafka for our
> use case .
>
> Use Case : Kafka will be realtime message bus , users should be able
> to pull Logs by specifying start_date and end_date or  Pull me last five
> minutes data etc ...
>
> I did POC on SearchMessageByTimestamp , here is the code
> https://gist.github.com/senthilec566/16e8e28b32834666fea132afc3a4e2f9 .
> And i observed that Searching Messages is slow ..
>
> Here is small test i did :
> Query :Fetch Logs of Last *5 minutes*:
> Result:
> No of Records fetched : *30*
> Fetch Time *6210* ms
>
> Above test performed in a topic which has 4 partitions. In each partition
> search & query processing happened .. in other words
> consumer.offsetsForTimes()
> consumer.assign(Arrays.asList(partition))
> consumer.seek(this.partition, offsetTimestamp.offset())
> consumer.poll(100)
>
> are the API calls of each partition.. I realized that , this was the
> reason for Kafka taking more time..
>
> What is efficient way of implementing SerachMessageByTimeStamp ?  Is Kafka
> right candidate for our Use Case ?
>
> Pls add your thoughts here ...
>
>
> Cheers,
> Senthil
>