Best way to "emulate" a rich Partitioner with open() and close() methods ?

2020-05-28 Thread LINZ, Arnaud
Hello, I would like to upgrade the performance of my Apache Kudu Sink by using the new “KuduPartitioner” of Kudu API to match Flink stream partitions with Kudu partitions to lower the network shuffling. For that, I would like to implement something like stream.partitionCustom(new KuduFlinkPa

Re: Best way to "emulate" a rich Partitioner with open() and close() methods ?

2020-05-29 Thread Robert Metzger
Hi Arnaud, Maybe I don't fully understand the constraints, but what about stream.map(new GetKuduPartitionMapper).keyBy(0).addSink(KuduSink()); The map(new GetKuduPartitionMapper) will be a regular RichMapFunction with open() and close() where you can handle the connection with Kudu's partitioning

RE: Best way to "emulate" a rich Partitioner with open() and close() methods ?

2020-05-29 Thread LINZ, Arnaud
“intermediate” technical info in the stream payload. Best, Arnaud De : Robert Metzger Envoyé : vendredi 29 mai 2020 13:10 À : LINZ, Arnaud Cc : user Objet : Re: Best way to "emulate" a rich Partitioner with open() and close() methods ? Hi Arnaud, Maybe I don't fully

Re: Best way to "emulate" a rich Partitioner with open() and close() methods ?

2020-06-02 Thread Robert Metzger
; *De :* Robert Metzger > *Envoyé :* vendredi 29 mai 2020 13:10 > *À :* LINZ, Arnaud > *Cc :* user > *Objet :* Re: Best way to "emulate" a rich Partitioner with open() and > close() methods ? > > > > Hi Arnaud, > > > > Maybe I don't fully un

Re: Best way to "emulate" a rich Partitioner with open() and close() methods ?

2020-06-05 Thread Arvid Heise
n >> the stream payload. >> >> Best, >> >> Arnaud >> >> >> >> *De :* Robert Metzger >> *Envoyé :* vendredi 29 mai 2020 13:10 >> *À :* LINZ, Arnaud >> *Cc :* user >> *Objet :* Re: Best way to "emulate" a ric

Re: Best way to "emulate" a rich Partitioner with open() and close() methods ?

2020-06-09 Thread Aljoscha Krettek
ioners ? That would avoid adding overhead and “intermediate” technical info in the stream payload. Best, Arnaud *De :* Robert Metzger *Envoyé :* vendredi 29 mai 2020 13:10 *À :* LINZ, Arnaud *Cc :* user *Objet :* Re: Best way to "emulate" a rich Partitioner with open() and close()