About KafkaConsumer and WM'ing and EventTime charactersitics

2019-01-29 Thread Vishal Santoshi
In case where one needs t to use kafka event time ( ingestion time ) for watermark generation and timestamp extraction is setting EventTimeCharactersitic as EventTime enough ? Or is this explicit code required ? consumer.assignTimestampsAndWatermarks(new AssignerWithPunctuatedWatermarks() {

Re: About KafkaConsumer and WM'ing and EventTime charactersitics

2019-01-29 Thread Vishal Santoshi
It seems from https://ci.apache.org/projects/flink/flink-docs-stable/dev/event_time.html that iTimeCharacteristic.IngestionTime should do the trick. Just wanted to confirm that the ingestion time is the event time provided by the kafka producer. On Tue, Jan 29, 2019 at 3:21 PM Vishal Santoshi wr

Re: About KafkaConsumer and WM'ing and EventTime charactersitics

2019-01-29 Thread Congxian Qiu
Hi Vishal May this doc[1] be helpful for you. [1] https://ci.apache.org/projects/flink/flink-docs-release-1.7/dev/connectors/kafka.html#kafka-consumers-and-timestamp-extractionwatermark-emission Best, Congxian Vishal Santoshi 于2019年1月30日周三 上午4:36写道: > It seems from > https://ci.apache.org/pro

Re: About KafkaConsumer and WM'ing and EventTime charactersitics

2019-01-30 Thread Vishal Santoshi
Thank you. This though is a little different. The producer of the kafka message attaches a time stamp https://issues.apache.org/jira/browse/KAFKA-2511. I do not see how I can get to that timestamp through a any stream abstraction over FlinkKafkaConsumer API even though it is available here https

Re: About KafkaConsumer and WM'ing and EventTime charactersitics

2019-01-30 Thread Jamie Grier
Vishal, that answer to your question about IngestionTime is "no". Ingestion time in this context means the time the data was read by Flink not the time it was written to Kafka. To get the effect you're looking for you have to set TimeCharacteristic.EventTime and follow the instructions here: https