Jorge Esteban Quilcate Otoya created KAFKA-9929:
---------------------------------------------------
Summary: Support reverse iterator on WindowStore
Key: KAFKA-9929
URL: https://issues.apache.org/jira/browse/KAFKA-9929
Project: Kafka
Issue Type: Improvement
Components: streams
Reporter: Jorge Esteban Quilcate Otoya
Currently, WindowStore fetch operations return an iterator sorted from earliest
to latest result:
```
* For each key, the iterator guarantees ordering of windows, starting from the
oldest/earliest
* available window to the newest/latest window.
```
We have a use-case where traces are stored in a WindowStore<String, Trace> and
use Kafka Streams to create a materialized view of traces. A query request
comes with a time range (e.g. now-1h, now) and want to return the most recent
results, i.e. fetch from this period of time, iterate and pattern match
latest/most recent traces, and if enough results, then reply without moving
further on the iterator.
Same store is used to search for previous traces. In this case, it search a key
for the last day, if found traces, we would also like to iterate from the most
recent.
RocksDb seems to support iterating backward and forward:
[https://github.com/facebook/rocksdb/wiki/Iterator#iterating-upper-bound-and-lower-bound]
For reference: This in some way extracts some bits from this previous issue:
https://issues.apache.org/jira/browse/KAFKA-4212:
> The {{RocksDBWindowsStore}}, a {{WindowsStore}}, can expire items via segment
> dropping, but it stores multiple items per key, based on their timestamp. But
> this store can be repurposed as a cache by fetching the items in reverse
> chronological order and returning the first item found.
Would like to know if there is any impediment on RocksDb or WindowStore to
support this.
Adding an argument to reverse in current fetch methods would be great:
```
WindowStore.fetch(from,to,Direction.BACKWARD|FORWARD)
```
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
