[jira] [Comment Edited] (FLINK-12675) Event time synchronization in Kafka consumer
[ https://issues.apache.org/jira/browse/FLINK-12675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16965425#comment-16965425 ] Thomas Weise edited comment on FLINK-12675 at 11/2/19 5:09 PM: --- [~wind_ljy] FLIP-27 has still ways to go and is going to result in brand new implementations for some of the existing connectors, while the old implementations that are used in production will need to be retained for a while until new things reach parity and maturity. The purpose of pointing you to FLIP-27 is not to arrive at compatible solutions (which is impossible) but rather as input for specific aspects such as the record and watermark emitter. It would be great if you continue your work for the existing consumer, or maybe me or someone else can pick it up as we will need this problem solved fairly soon. was (Author: thw): [~wind_ljy] FLIP-27 has still ways to go and is going to result in brand new implementations for some of the existing connectors, while the old implementations that are used in production will need to be retained for a while until new things reach parity and maturity. The purpose of pointing you to FLIP-27 is not to arrive at compatible solutions (which is impossible) but rather as input for specific aspects such as the record and watermark emitter. It would be great if you continue your work for the existing consumer, or else I can maybe pick it up as we will need this problem solved fairly soon. > Event time synchronization in Kafka consumer > > > Key: FLINK-12675 > URL: https://issues.apache.org/jira/browse/FLINK-12675 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Kafka >Reporter: Thomas Weise >Assignee: Thomas Weise >Priority: Major > > Integrate the source watermark tracking into the Kafka consumer and implement > the sync mechanism (different consumer model, compared to Kinesis). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (FLINK-12675) Event time synchronization in Kafka consumer
[ https://issues.apache.org/jira/browse/FLINK-12675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16965417#comment-16965417 ] Jiayi Liao edited comment on FLINK-12675 at 11/2/19 4:30 PM: - [~thw] Thanks for your review. I've improved some downsides of the design according to your comments. And It took me some time to look into FLIP-27 and Idle Partition related discussions. I'm concerned that this design may have conflicts with FLIP-27 because I can see FLIP-27 refactor may introduce completely new abstraction for the communication between data production (#KafkaConsumerThread) and data reads (#KafkaFetcher) in all connectors, which pretty much includes my whole design. IMO at least we should wait until a FLIP-27 MVP or maybe I can help to consider the feedback mechanism into the new #Reader implementation. But currently there're still some works worth doing like moving #WatermarkTracker into core you mentioned (we'll finally do it either way). What do you think? was (Author: wind_ljy): [~thw] Thanks for your review. I've addressed some downsides of the design according to your comments. And It took me some time to look into FLIP-27 and Idle Partition related discussions. I'm concerned that this design may have conflicts with FLIP-27 because I can see FLIP-27 refactor may introduce completely new abstraction for the communication between data production (#KafkaConsumerThread) and data reads (#KafkaFetcher) in all connectors, which pretty much includes my whole design. IMO at least we should wait until a FLIP-27 MVP or maybe I can help to consider the feedback mechanism into the new #Reader implementation. But currently there're still some works worth doing like moving #WatermarkTracker into core you mentioned (we'll finally do it either way). What do you think? > Event time synchronization in Kafka consumer > > > Key: FLINK-12675 > URL: https://issues.apache.org/jira/browse/FLINK-12675 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Kafka >Reporter: Thomas Weise >Assignee: Thomas Weise >Priority: Major > > Integrate the source watermark tracking into the Kafka consumer and implement > the sync mechanism (different consumer model, compared to Kinesis). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (FLINK-12675) Event time synchronization in Kafka consumer
[ https://issues.apache.org/jira/browse/FLINK-12675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16960301#comment-16960301 ] Jiayi Liao edited comment on FLINK-12675 at 10/28/19 12:03 AM: --- [~thw] I've drafted a design for this. It follows your design on Kinesis Source but there're still a lot of changes due to the difference between these two connectors processing model. Could you take a look if you can spare time? https://docs.google.com/document/d/1d4XhVK4BD9GGNqhfSk_U30nEYOXx8THCHhR77RNbIbA/edit?usp=sharing was (Author: wind_ljy): [~thw] I've drafted a design for this. It follows your design on Kinesis Source but there're still a lot of changes due to the difference between these two connectors processing model. Could you take a look if you can spare time? https://docs.google.com/document/d/1d4XhVK4BD9GGNqhfSk_U30nEYOXx8THCHhR77RNbIbA/edit# > Event time synchronization in Kafka consumer > > > Key: FLINK-12675 > URL: https://issues.apache.org/jira/browse/FLINK-12675 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Kafka >Reporter: Thomas Weise >Assignee: Thomas Weise >Priority: Major > > Integrate the source watermark tracking into the Kafka consumer and implement > the sync mechanism (different consumer model, compared to Kinesis). -- This message was sent by Atlassian Jira (v8.3.4#803005)
