Re: Kafka Streams - Join synchronization issue

2017-04-18 Thread Damian Guy
Hi Marco, w.r.t your JoinWindows. I think you'll want to increase the until time as this is roughly how long the record will be kept in the local state store that is used for joins (based on event time). So if you have data arriving on different streams at varying rates, event time of one of your

Re: Kafka Streams - Join synchronization issue

2017-04-18 Thread Marco Abitabile
Hello Eno, yes it is 1-second width (I unintentionally wrote 1-minute, in my previous mail). Just to provide you more info about the data: location data have 1-second resolution, while user activity data arrives with varying speed, may happen to have 10-100 records as well as 0 records within 1 se

Re: Kafka Streams - Join synchronization issue

2017-04-18 Thread Eno Thereska
Hi Marco, I noticed your window is 1 second width, not 1 minute width. Is that intentional? Thanks Eno > On 17 Apr 2017, at 19:41, Marco Abitabile wrote: > > hello Eno, > thanks for your support. The two streams are both kstreams. The window is of > 1 minute-width until 5 minutes. This is the

Re: Kafka Streams - Join synchronization issue

2017-04-17 Thread Marco Abitabile
hello Eno, thanks for your support. The two streams are both kstreams. The window is of 1 minute-width until 5 minutes. This is the code: //Other Stream: User Location, is a string with the name of the city the//user is (like "San Francisco") KStreamBuilder builder = new KStreamBuilder(); KStre

Re: Kafka Streams - Join synchronization issue

2017-04-16 Thread Eno Thereska
Hi Marco, Could you share a bit of your code, or at a minimum provide some info on: - is userActivitiesStream and geoDataStream a KStream of KTable? - what is the length of "timewindow"? Thanks Eno > On 16 Apr 2017, at 19:44, Marco Abitabile wrote: > > Hi All! > > I need a little hint to unde

Kafka Streams - Join synchronization issue

2017-04-16 Thread Marco Abitabile
Hi All! I need a little hint to understand how join works, in regards of stream synchronization. This mail is a bit long, I need to explain the issue I'm facing. *TL-TR: * it seems that join synchonization between stream is not respected as explained here: https://cwiki.apache.org/confluence/dis

Kafka Streams - Join synchronization issue

2017-04-16 Thread Marco Abitabile
Hi All! I need a little hint to understand how join works, in regards of stream synchronization. This mail is a bit long, I need to explain the issue I'm facing. *TL-TR: * it seems that join synchonization between stream is not respected as explained here: https://cwiki.apache.org/confluence/dis