Re: Migrate custom partitioner from Flink 1.7 to Flink 1.9

2020-01-08 Thread Arvid Heise
Hi Salva, I already answered on SO [1], but I'll replicate it here: With Flink 1.9, you cannot dynamically broadcast to all channels anymore. Your StreamPartitioner has to statically specify if it's a broadcast with isBroadcast. Then, selectChannel is never invoked. Do you have a specific use

Re: Migrate custom partitioner from Flink 1.7 to Flink 1.9

2020-01-03 Thread Salva Alcántara
Thanks Chesnay! Just to be clear, this how my current code looks like: ``` unionChannel = broadcastChannel.broadcast().union(singleChannel) result = new DataStream<>( unionChannel.getExecutionEnvironment(), new PartitionTransformation<>(unionChannel.getTransformation(), new

Re: Migrate custom partitioner from Flink 1.7 to Flink 1.9

2020-01-03 Thread Chesnay Schepler
You should be able to implement this on the DataStream API level using DataStream#broadcast and #union like this: input = ... singleChannel = input.filter(x -> !x.isBroadCastPartitioning); broadcastChannel = input.filter(x -> x.isBroadCastPartitioning); result =

Migrate custom partitioner from Flink 1.7 to Flink 1.9

2019-12-25 Thread Salva Alcántara
I am trying to migrate a custom dynamic partitioner from Flink 1.7 to Flink 1.9. The original partitioner implemented the `selectChannels` method within the `StreamPartitioner` interface like this: ```java // Original: working for Flink 1.7 //@Override public int[]