RE: Is it possible to preserve chaining for multi-input operators?

2023-03-24 Thread Schwalbe Matthias
Hi Viacheslav, … back from vacation … you are welcome, glad to hear it worked out  Thias From: Viacheslav Chernyshev Sent: Thursday, March 16, 2023 5:34 PM To: user@flink.apache.org Subject: Re: Is it possible to preserve chaining for multi-input operators? Hi Matthias, Just wanted to

Re: Is it possible to preserve chaining for multi-input operators?

2023-03-16 Thread Viacheslav Chernyshev
Hi Matthias, Just wanted to thank you for the hints! I've successfully developed a multi-stream operator that allows doing things like this: KeyedMultiInputStream.builder(environment, new UserDefinedFunction()) .addKeyedStream(fooSource, fooMapper, UserDefinedFunction::processFoo)

RE: Is it possible to preserve chaining for multi-input operators?

2023-02-28 Thread Schwalbe Matthias
Hi Viacheslav, Certainly I can … There is two parts to it, * setting up such MultipleInputStreamOperator, which is documented (sort of), but not quite complete * I can prepare some boiler-plate, not today, but in the next days (if you are interested) * Second part is about how

Re: Is it possible to preserve chaining for multi-input operators?

2023-02-28 Thread Viacheslav Chernyshev
Hi Matthias, Thank you for the reply. You are absolutely right, the first keyBy is unavoidable, but after that we fix the parallelism and maintain the same key throughout the pipeline. The MultipleInputStreamOperator approach that you've described looks very interesting! Unfortunately, I have

RE: Is it possible to preserve chaining for multi-input operators?

2023-02-28 Thread Schwalbe Matthias
Hi Viacheslav, These are two very interesting questions... You have found out about the chaining restriction to single input operators to be chained, it does also not help to union() multiple streams into a single input, they still count as multiple inputs. * The harder way to go would