Guozhang Wang created KAFKA-9455:
------------------------------------
Summary: Consider using TreeMap for In-memory stores of Streams
Key: KAFKA-9455
URL: https://issues.apache.org/jira/browse/KAFKA-9455
Project: Kafka
Issue Type: Improvement
Components: streams
Reporter: Guozhang Wang
>From [~ableegoldman]: It's worth noting that it might be a good idea to switch
>to TreeMap for different reasons. Right now the ConcurrentSkipListMap allows
>us to safely perform range queries without copying over the entire keyset, but
>the performance on point queries seems to scale noticeably worse with the
>number of unique keys. Point queries are used by aggregations while range
>queries are used by windowed joins, but of course both are available within
>the PAPI and for interactive queries so it's hard to say which we should
>prefer. Maybe rather than make that tradeoff we should have one version for
>efficient range queries (a "JoinWindowStore") and one for efficient point
>queries ("AggWindowStore") - or something. I know we've had similar thoughts
>for a different RocksDB store layout for Joins (although I can't find that
>ticket anywhere..), it seems like the in-memory stores could benefit from a
>special "Join" version as well cc/ Guozhang Wang
Here are some random thoughts:
1. For kafka streams processing logic (i.e. without IQ), it's better to make
all processing logic relying on point queries rather than range queries. Right
now the only processor that use range queries are, as mentioned above, windowed
stream-stream joins. I think we should consider using a different window
implementation for this (and as a result also get rid of the retainDuplicate
flags) to refactor the windowed stream-stream join operation.
2. With 1), range queries would only be exposed as IQ. Depending on its usage
frequency I think it makes lots of sense to optimize for single-point queries.
Of course, even without step 1) we should still consider using tree-map for
windowed in-memory stores to have a better scaling effect.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
