Re: Window Store

Thanks a lot Guozhang. I will try and let you know. Really appreciate all the help. This community has been amazing. Thanks On Tue, Feb 23, 2021 at 5:48 PM Guozhang Wang wrote: > Sorry I was not very clear before: by "WindowStore" I meant implementing > your own customized store based on a kvS

Object Serde

Hi All, Is there a widely used serde for java objects which provides both backward and forward compatibility? I have been using Kryo with a compatible field serializer but it seems to be very slow. Any suggestions would really help. We also have protos in some cases but moving everything proto wou

Re: Window Store

Sorry I was not very clear before: by "WindowStore" I meant implementing your own customized store based on a kvStore where the key is a combo . Note you put timestamp first then key in your serialization format, so that you can range-fetch with just the prefix on timestamp then. In fact `WindowSto

Re: Window Store

Thanks Guozhang. I don't see the remove method in window stores. Am I missing something? It would be very nice to implement the optimization you had mentioned. Thanks On Tue, Feb 23, 2021 at 11:11 AM Guozhang Wang wrote: > I see. In that case I think your design with a KVstore plus a book-keep

Re: NullPointerException after upgrading from 2.5.1 to 2.6.1 in my stream app

Is it possible that the flattened values contain `null` and hence `_.split` throws? On Tue, Feb 23, 2021 at 8:23 AM Nitay Kufert wrote: > Hey, missed your replay - but the code i've shared above the logs is the > code around those lines (removed some identifiers to make it a little bit > more ge

Re: Window Store

I see. In that case I think your design with a KVstore plus a book-keeping window store would work better. One minor optimization you can try though, is that instead of trying to check if the TTL has changed or not when expiring from the window store, you can try to delete from the window store whe

What metrics to use to diagnose performance degradation?

Hi, I have a Kafka cluster and two topics it, X and Y. I have two unrelated applications, one reads from topic X and one from topic Y. Those applications don’t share any resources except for Kafka and K8S clusters. In both application the processing is very basic and no changes were made to it.

Re: NullPointerException after upgrading from 2.5.1 to 2.6.1 in my stream app

Hey, missed your replay - but the code i've shared above the logs is the code around those lines (removed some identifiers to make it a little bit more generic): > inputStream.flatMapValues(_.split).peek((k, v) => {val _ = $k -> > ${v.printForDebug}")}) # return type KStream[Windowed[String], > Si