Re: How Flink Kafka Consumer works when it restarts

2020-02-15 Thread Theo Diefenthal
I actually like to have both. As Robert states, storing it in state makes is more intrinsic and allows for exacly once processing. Having them in Kafka is especially good for monitoring. In our monitoring setup, its easy to visualize the committed offsets of kafka consumer groups whereas it

Re: BucketingSink capabilities for DataSet API

2020-02-15 Thread aj
Hi Rafi, I have a similar use case where I want to read parquet files in the dataset and want to perform some transformation and similarly want to write the result using year month day partitioned. I am stuck at first step only where how to read and write Parquet files using hadoop-Compatability.

Re: Flink 1.10 es sink exception

2020-02-15 Thread Leonard Xu
Hi, sunfulin I reproduce your case,this should be a bug in extracting unique key from plan and I create an issue[1] to trace this. CC: jark [1]https://issues.apache.org/jira/browse/FLINK-16070 > 在 2020年2月14日,23:39,sunfulin 写道: > > Hi,