Hello Ara,
I'm wondering what this KTable is used for (e.g. join, aggregation, etc)?
And is the usage of TTL is because your computational logic can safely
ignore key-value pairs that has existed for 24 hours, or is it purely for
bounding the state store space usage (for this end, what is your current
state store space usage)?
Your rocksdb put / get inefficiency could be due to various reasons besides
storage space, e.g. your rocksDB configs. I'd like to help investigating
with more information on how the rocksdb was used (write frequency, read
frequency, key space per instance, etc).
Guozhang
On Mon, Feb 13, 2017 at 8:57 AM, Ara Ebrahimi
wrote:
> Hi,
>
> I have a ktable and I want to keep entries in it only for that past 24
> hours. How can I do that? I understand rocksdb has support for ttl. Should
> I set that? How? Should I use kafka-streams window functionality? Would it
> remove data from old windows?
>
> I want to do this because I’m seeing a huge lag forming in the repartition
> topic backing the ktable after a while and the whole thing slows down. In
> jmx I see values such as these for rocksdb metrics:
>
> all-get-avg-latency-ms=60123
> all-put-avg-latency-ms=40456
> all-put-qps=421
> all-get-qps=430
>
> What’s the best way to check to see how rocksdb is performing and where
> the bottleneck is? This is on an ssd.
>
> Thanks,
> Ara.
>
>
>
>
>
> This message is for the designated recipient only and may contain
> privileged, proprietary, or otherwise confidential information. If you have
> received it in error, please notify the sender immediately and delete the
> original. Any other use of the e-mail by you is prohibited. Thank you in
> advance for your cooperation.
>
>
>
--
-- Guozhang
