pnowojski edited a comment on issue #7405: [FLINK-11249] FlinkKafkaProducer011 can not be migrated to FlinkKafkaProducer URL: https://github.com/apache/flink/pull/7405#issuecomment-456466692 Thanks for showing the interest @tvielgouarin in this topic! > But for migration tests of universal FlinkKafkaProducer between Flink versions : I read in the doc that this connector is supported since flink 1.7 . So currently, no migration test should be passing Both for the universal and `FlinkKafkaProducer011` the most important test to catch regressions is to test restoring from savepoint that was taken using the latest stable version (`1.7`). It would test if current code in the master (future `1.8` release) has state compatibility with `1.7` savepoints. > Also, do we just want to check that the connector can start from a previous version checkpoint ? I think at least for now only checking the compatibility with `1.7` is enough. > Most of the time the test fail with `The producer attempted to use a producer id which is not currently assigned to its transactional id` I guess your test is failing because you are restoring Flink's state from savepoint correctly, but you are ignoring the state of Kafka brokers. I haven't thought about this issue before and it looks like nobody else did it as well... By the way, thanks for discovering this issue. I guess this is the first migration test that must take care of some external state (besides Flink's internal state). Probably Kafka migration tests must store not only "savepoint" in the resources but also in one way or another store the state of kafka cluster as it was just after completing the savepoint from which we want to restore (transaction log? topics content?) Whatever that means... 😕Couple of random thoughts: 1. it might be possible to identify the list of files that define the internal state of Kafka that we need to archive, place it in the resources alongside of savepoint and use it during `KafkaTestBase` initialisation 2. maybe it will be better to express this logic in end to end tests? CC @tzulitai
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
