Hi Matthias, Very thanks for proposing the issue! Currently when releasing a new version, it is indeed required to manully generates new test data on the branch to release (namely release-1.16) for all the compatibility tests, and then pick the files back to the master branch so that the files could be used to check the compatibility during the development of the next release cycle.
Currently these tests requires RM to manully uncomment the @Ignore for the generation methods and generate the files manually, which is cumbersome and error-prone. Thus after releasing 1.15 I have create an issue for modifying these tests[1], but very sorry I have not finished the work yet. As a whole, I initially try to create a script to get the above action done automatically. I also think we could indeed move the declaration of current version into a common place so that we could modify it unifily, and we could also define a common structure and file patterns for the compatibility classes that explicitly defines generators, then we could have a class similar to `ConfigOptionsDocGenerator` to generates all the data, the script could then pick the generated files onto the specified branch. Best, Yun Gao [1] https://issues.apache.org/jira/browse/FLINK-27518 ------------------Original Mail ------------------ Sender:Matthias Pohl <matthias.p...@aiven.io.INVALID> Send Date:Tue Oct 18 16:35:47 2022 Recipients:Flink Dev <dev@flink.apache.org> CC:Yun Gao <yungao...@aliyun.com> Subject:Re: [DISCUSS] Aligning migration test data generation I created a draft PR to underline my proposal [1]. The PR doesn't cover all the relevant tests, yet. I'd like to get feedback from anyone who has done minor releases in the past (CC'ing Yun Gao since he did this task for Flink 1.15) before proceeding with the tests that are still not migrated (since they require a bit more work). I'm looking forward to feedback. Best, Matthias [1] https://github.com/apache/flink/pull/20954 On Wed, Oct 5, 2022 at 11:42 AM Matthias Pohl <matthias.p...@aiven.io> wrote: > Hi everyone, > I started looking into the migration test data generation after I got > confused about it in the context of the 1.16.0-rc1 voting process [1]. The > release manager needs to do some manual work around adding test data for > certain tests after the release is finalized (see item 15 in the "Checklist > to declare the process completed" section of the documentation for creating > a Flink release [2]): > - Multiple tests need to get an update to the version they run on. > - Tests will fail due to missing test data. The test data needs to be > created in a manual process for each of the tests as well. > > I'm playing around with aligning this a bit as part of FLINK-29485 [3]. > Essentially, I'd like to move the general Flink version range setup into > flink-test-utils. The release manager would have to update only one > location instead of each test. > Additionally, the junit test method implementations responsible for > generating the test data would be enabled by default. I want to introduce a > system property that can be used to specify the Flink version for which the > data shall be generated. We can use assume to enable/disable the test data > generation based on this system property. > > In the end, the release manager would have to change only code in a single > location (instead of multiple files) and run the test suite with this > system property being set once on the corresponding release branch to > generate the data. WDYT? I'm curious whether I'm missing something. > > Best, > Matthias > > [1] https://lists.apache.org/thread/b0295pq4qpwxg4gpnrr9rjk27zptgkcb > [2] > https://cwiki.apache.org/confluence/display/FLINK/Creating+a+Flink+Release > [3] https://issues.apache.org/jira/browse/FLINK-29485 >
