Hello,
I would like to upgrade the performance of my Apache Kudu Sink by using the new
“KuduPartitioner” of Kudu API to match Flink stream partitions with Kudu
partitions to lower the network shuffling.
For that, I would like to implement something like
stream.partitionCustom(new KuduFlinkPa
Hi Arnaud,
Maybe I don't fully understand the constraints, but what about
stream.map(new GetKuduPartitionMapper).keyBy(0).addSink(KuduSink());
The map(new GetKuduPartitionMapper) will be a regular RichMapFunction with
open() and close() where you can handle the connection with Kudu's
partitioning
“intermediate” technical info in the stream
payload.
Best,
Arnaud
De : Robert Metzger
Envoyé : vendredi 29 mai 2020 13:10
À : LINZ, Arnaud
Cc : user
Objet : Re: Best way to "emulate" a rich Partitioner with open() and close()
methods ?
Hi Arnaud,
Maybe I don't fully
; *De :* Robert Metzger
> *Envoyé :* vendredi 29 mai 2020 13:10
> *À :* LINZ, Arnaud
> *Cc :* user
> *Objet :* Re: Best way to "emulate" a rich Partitioner with open() and
> close() methods ?
>
>
>
> Hi Arnaud,
>
>
>
> Maybe I don't fully un
n
>> the stream payload.
>>
>> Best,
>>
>> Arnaud
>>
>>
>>
>> *De :* Robert Metzger
>> *Envoyé :* vendredi 29 mai 2020 13:10
>> *À :* LINZ, Arnaud
>> *Cc :* user
>> *Objet :* Re: Best way to "emulate" a ric
ioners
? That would avoid adding overhead and “intermediate” technical info in
the stream payload.
Best,
Arnaud
*De :* Robert Metzger
*Envoyé :* vendredi 29 mai 2020 13:10
*À :* LINZ, Arnaud
*Cc :* user
*Objet :* Re: Best way to "emulate" a rich Partitioner with open() and
close()