Re: Streaming join performance

Привет Артем! Are your tables backed by Kafka? If - yes, what if you use upsert-kafka connector from Table API , does it help to reduce the number of records in each subsequent join operator? I wrote a blog-p

Re: Streaming join performance

Hi, David Regarding the N-way join, this feature aims to address the issue of state simplification, it is on the roadmap. Technically there are no limitations, but we'll need some time to find a sensible solution. Best, Ron David Anderson 于2023年8月9日周三 10:38写道： > This join optimization sounds p

Re: Streaming join performance

This join optimization sounds promising, but I'm wondering why Flink SQL isn't taking advantage of the N-Ary Stream Operator introduced in FLIP-92 [1][2] to implement a n-way join in a single operator. Is there something that makes this impossible/impractical? [1] https://cwiki.apache.org/confluen

RE: Streaming join performance

Hi, we are also paying attention to this issue and have completed the validation of the minibatch join optimization including the intermediate message folding you mentioned. We plan to officially release it in Flink 1.19. This optimization could significantly improves the performance of join op

Re: join performance

Hi Henry, yes, with early firings you would have the problem of duplicate emission. I'm afraid I don't have a solution for that right now. For the "another question" I think you are right that this would be session windowing. Please have a look at this blog post that I wrote recently: http://data-

Re: join performance

So is the window defined as hour-window or second-window? If I am using hour-window, I guess I need to modify the trigger to fire early (e.g. every minute)? But I don't want to repeatedly emit the same joined records for every minute (i.e. on 2nd minute, I only want to emit the changes introduced

Re: join performance

Hi, you are right, everything will be emitted in a huge burst at the end of the hour. If you want to experiment a bit you can write a custom Trigger based on EventTimeTrigger that will delay firing of windows. You would change onEventTime() to not fire but instead register a processing-time timer a

Re: join performance

But the join requirement is to match the records from two streams occurring within one hour (besides the normal join key condition), if I use the second join window, those records wouldn't be in the same window any more. On Thu, Apr 28, 2016 at 11:47 PM, Ashutosh Kumar wrote: > Time unit can b

Re: join performance

Time unit can be in seconds as well. Is there specific need to get bursts hourly? On Fri, Apr 29, 2016 at 11:48 AM, Henry Cai wrote: > For the below standard stream/stream join, does flink store the results of > stream 1 and stream 2 into state store for the current hour and at the end > of the

join performance

For the below standard stream/stream join, does flink store the results of stream 1 and stream 2 into state store for the current hour and at the end of the hour window it will fire the window by iterating through all stored elements in the state store to find join matches? My concern is during mo