[jira] [Commented] (FLINK-26416) Release Testing: Sink V2 sanity checks
[ https://issues.apache.org/jira/browse/FLINK-26416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17504272#comment-17504272 ] Fabian Paul commented on FLINK-26416: - Good point, I forgot about that. Thanks for the investigation. I will add that to the release notes that users relying on the FileSink need to also update the user program. In general, this should not be the case for all other connectors since they are not bundled with Flink. I also checked an officially Flink does not offer binary compatibility meaning that users need to recompile the user program when upgrading. I will close this ticket as done because nothing critical appeared. Again thanks for all the effort :) > Release Testing: Sink V2 sanity checks > -- > > Key: FLINK-26416 > URL: https://issues.apache.org/jira/browse/FLINK-26416 > Project: Flink > Issue Type: Improvement > Components: Connectors / Common >Affects Versions: 1.15.0 >Reporter: Fabian Paul >Assignee: Liu >Priority: Blocker > Labels: release-testing > Fix For: 1.15.0 > > Attachments: image-2022-03-10-18-58-19-519.png, > image-2022-03-10-20-36-17-977.png, image-2022-03-10-20-36-45-974.png, > image-2022-03-10-20-48-19-744.png, image-2022-03-10-20-51-25-331.png, > image-2022-03-10-20-55-46-224.png > > > With the introduction of Sink V2, the operator model of the sink changed > slightly therefore it makes sense to test different upgrade/sanity scenarios. > > You can take any of the existing Sinks in the project. I would recommend the > FileSink. > > # Run a job with Flink 1.14 and take a savepoint and try to restore and > resume with 1.15 > # Run a job with Flink 1.15 and take a savepoint and try to restore and > resume with 1.15 > # Run a bounded job with Flink 1.15 > > In all cases, please verify that all records have been written at the end of > the scenario and there are no duplicates. > > > > > > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-26416) Release Testing: Sink V2 sanity checks
[ https://issues.apache.org/jira/browse/FLINK-26416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17504239#comment-17504239 ] Liu commented on FLINK-26416: - [~fpaul] flink-examples-streaming_2.11-1.14-SNAPSHOT.jar is built with flink 1.14 and can run with flink 1.14 env. But it can not be executed with flink 1.15 env. I find that there exist flink-connector-files-1.15-SNAPSHOT.jar in flink 1.15's lib by default. Maybe this is the reason. !image-2022-03-10-20-51-25-331.png|width=1576,height=222! !image-2022-03-10-20-48-19-744.png|width=1579,height=421! > Release Testing: Sink V2 sanity checks > -- > > Key: FLINK-26416 > URL: https://issues.apache.org/jira/browse/FLINK-26416 > Project: Flink > Issue Type: Improvement > Components: Connectors / Common >Affects Versions: 1.15.0 >Reporter: Fabian Paul >Assignee: Liu >Priority: Blocker > Labels: release-testing > Fix For: 1.15.0 > > Attachments: image-2022-03-10-18-58-19-519.png, > image-2022-03-10-20-36-17-977.png, image-2022-03-10-20-36-45-974.png, > image-2022-03-10-20-48-19-744.png, image-2022-03-10-20-51-25-331.png > > > With the introduction of Sink V2, the operator model of the sink changed > slightly therefore it makes sense to test different upgrade/sanity scenarios. > > You can take any of the existing Sinks in the project. I would recommend the > FileSink. > > # Run a job with Flink 1.14 and take a savepoint and try to restore and > resume with 1.15 > # Run a job with Flink 1.15 and take a savepoint and try to restore and > resume with 1.15 > # Run a bounded job with Flink 1.15 > > In all cases, please verify that all records have been written at the end of > the scenario and there are no duplicates. > > > > > > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-26416) Release Testing: Sink V2 sanity checks
[ https://issues.apache.org/jira/browse/FLINK-26416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17504209#comment-17504209 ] Fabian Paul commented on FLINK-26416: - I mean if you build a user program with 1.14 it should also be executable with 1.15. From the error messages, it looks like you have used the same user program but updated the connector version to 1.15. Can you verify that the wrong FileSink connector version is used? > Release Testing: Sink V2 sanity checks > -- > > Key: FLINK-26416 > URL: https://issues.apache.org/jira/browse/FLINK-26416 > Project: Flink > Issue Type: Improvement > Components: Connectors / Common >Affects Versions: 1.15.0 >Reporter: Fabian Paul >Assignee: Liu >Priority: Blocker > Labels: release-testing > Fix For: 1.15.0 > > Attachments: image-2022-03-10-18-58-19-519.png > > > With the introduction of Sink V2, the operator model of the sink changed > slightly therefore it makes sense to test different upgrade/sanity scenarios. > > You can take any of the existing Sinks in the project. I would recommend the > FileSink. > > # Run a job with Flink 1.14 and take a savepoint and try to restore and > resume with 1.15 > # Run a job with Flink 1.15 and take a savepoint and try to restore and > resume with 1.15 > # Run a bounded job with Flink 1.15 > > In all cases, please verify that all records have been written at the end of > the scenario and there are no duplicates. > > > > > > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-26416) Release Testing: Sink V2 sanity checks
[ https://issues.apache.org/jira/browse/FLINK-26416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17504184#comment-17504184 ] Liu commented on FLINK-26416: - [~fpaul] I build my java package with Flink 1.14. But it can not be executed in Flink 1.15. The exception is as following. So I rebuild it in Flink 1.15 and it can resume from savepoint. I think that it is reasonable and the test passes. What do you think? !image-2022-03-10-18-58-19-519.png|width=1465,height=327! > Release Testing: Sink V2 sanity checks > -- > > Key: FLINK-26416 > URL: https://issues.apache.org/jira/browse/FLINK-26416 > Project: Flink > Issue Type: Improvement > Components: Connectors / Common >Affects Versions: 1.15.0 >Reporter: Fabian Paul >Assignee: Liu >Priority: Blocker > Labels: release-testing > Fix For: 1.15.0 > > Attachments: image-2022-03-10-18-58-19-519.png > > > With the introduction of Sink V2, the operator model of the sink changed > slightly therefore it makes sense to test different upgrade/sanity scenarios. > > You can take any of the existing Sinks in the project. I would recommend the > FileSink. > > # Run a job with Flink 1.14 and take a savepoint and try to restore and > resume with 1.15 > # Run a job with Flink 1.15 and take a savepoint and try to restore and > resume with 1.15 > # Run a bounded job with Flink 1.15 > > In all cases, please verify that all records have been written at the end of > the scenario and there are no duplicates. > > > > > > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-26416) Release Testing: Sink V2 sanity checks
[ https://issues.apache.org/jira/browse/FLINK-26416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17504098#comment-17504098 ] Fabian Paul commented on FLINK-26416: - [~Jiangang] thanks for your findings. Can you clarify what you mean by "the interface is not compatible"? Does this mean you could not test the upgrade because of a problem? > Release Testing: Sink V2 sanity checks > -- > > Key: FLINK-26416 > URL: https://issues.apache.org/jira/browse/FLINK-26416 > Project: Flink > Issue Type: Improvement > Components: Connectors / Common >Affects Versions: 1.15.0 >Reporter: Fabian Paul >Assignee: Liu >Priority: Blocker > Labels: release-testing > Fix For: 1.15.0 > > > With the introduction of Sink V2, the operator model of the sink changed > slightly therefore it makes sense to test different upgrade/sanity scenarios. > > You can take any of the existing Sinks in the project. I would recommend the > FileSink. > > # Run a job with Flink 1.14 and take a savepoint and try to restore and > resume with 1.15 > # Run a job with Flink 1.15 and take a savepoint and try to restore and > resume with 1.15 > # Run a bounded job with Flink 1.15 > > In all cases, please verify that all records have been written at the end of > the scenario and there are no duplicates. > > > > > > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-26416) Release Testing: Sink V2 sanity checks
[ https://issues.apache.org/jira/browse/FLINK-26416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17503566#comment-17503566 ] Liu commented on FLINK-26416: - [~fpaul], thanks. I have test the three cases and all of them passed. I generate data in source and write them to files with FileSink. The result is the same as the source. The last two cases are easy to test. For the first case, I assign uids for operators and rebuild my java package in flink 1.15 for that the interface is not compatible. > Release Testing: Sink V2 sanity checks > -- > > Key: FLINK-26416 > URL: https://issues.apache.org/jira/browse/FLINK-26416 > Project: Flink > Issue Type: Improvement > Components: Connectors / Common >Affects Versions: 1.15.0 >Reporter: Fabian Paul >Assignee: Liu >Priority: Blocker > Labels: release-testing > Fix For: 1.15.0 > > > With the introduction of Sink V2, the operator model of the sink changed > slightly therefore it makes sense to test different upgrade/sanity scenarios. > > You can take any of the existing Sinks in the project. I would recommend the > FileSink. > > # Run a job with Flink 1.14 and take a savepoint and try to restore and > resume with 1.15 > # Run a job with Flink 1.15 and take a savepoint and try to restore and > resume with 1.15 > # Run a bounded job with Flink 1.15 > > In all cases, please verify that all records have been written at the end of > the scenario and there are no duplicates. > > > > > > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-26416) Release Testing: Sink V2 sanity checks
[ https://issues.apache.org/jira/browse/FLINK-26416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17503459#comment-17503459 ] Fabian Paul commented on FLINK-26416: - [~Jiangang] you should now be able to start the testing the blocker has been resolved. > Release Testing: Sink V2 sanity checks > -- > > Key: FLINK-26416 > URL: https://issues.apache.org/jira/browse/FLINK-26416 > Project: Flink > Issue Type: Improvement > Components: Connectors / Common >Affects Versions: 1.15.0 >Reporter: Fabian Paul >Assignee: Liu >Priority: Blocker > Labels: release-testing > Fix For: 1.15.0 > > > With the introduction of Sink V2, the operator model of the sink changed > slightly therefore it makes sense to test different upgrade/sanity scenarios. > > You can take any of the existing Sinks in the project. I would recommend the > FileSink. > > # Run a job with Flink 1.14 and take a savepoint and try to restore and > resume with 1.15 > # Run a job with Flink 1.15 and take a savepoint and try to restore and > resume with 1.15 > # Run a bounded job with Flink 1.15 > > In all cases, please verify that all records have been written at the end of > the scenario and there are no duplicates. > > > > > > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-26416) Release Testing: Sink V2 sanity checks
[ https://issues.apache.org/jira/browse/FLINK-26416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17500521#comment-17500521 ] Liu commented on FLINK-26416: - [~fpaul] Thanks. I would like to test it after the blocker finishes. > Release Testing: Sink V2 sanity checks > -- > > Key: FLINK-26416 > URL: https://issues.apache.org/jira/browse/FLINK-26416 > Project: Flink > Issue Type: Improvement > Components: Connectors / Common >Affects Versions: 1.15.0 >Reporter: Fabian Paul >Priority: Blocker > Labels: release-testing > Fix For: 1.15.0 > > > With the introduction of Sink V2, the operator model of the sink changed > slightly therefore it makes sense to test different upgrade/sanity scenarios. > > You can take any of the existing Sinks in the project. I would recommend the > FileSink. > > # Run a job with Flink 1.14 and take a savepoint and try to restore and > resume with 1.15 > # Run a job with Flink 1.15 and take a savepoint and try to restore and > resume with 1.15 > # Run a bounded job with Flink 1.15 > > In all cases, please verify that all records have been written at the end of > the scenario and there are no duplicates. > > > > > > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-26416) Release Testing: Sink V2 sanity checks
[ https://issues.apache.org/jira/browse/FLINK-26416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17500087#comment-17500087 ] Fabian Paul commented on FLINK-26416: - [~Jiangang] I can assign it to you but the testing effort is currently still blocked by FLINK-26173 > Release Testing: Sink V2 sanity checks > -- > > Key: FLINK-26416 > URL: https://issues.apache.org/jira/browse/FLINK-26416 > Project: Flink > Issue Type: Improvement > Components: Connectors / Common >Affects Versions: 1.15.0 >Reporter: Fabian Paul >Priority: Blocker > Labels: release-testing > Fix For: 1.15.0 > > > With the introduction of Sink V2, the operator model of the sink changed > slightly therefore it makes sense to test different upgrade/sanity scenarios. > > You can take any of the existing Sinks in the project. I would recommend the > FileSink. > > # Run a job with Flink 1.14 and take a savepoint and try to restore and > resume with 1.15 > # Run a job with Flink 1.15 and take a savepoint and try to restore and > resume with 1.15 > # Run a bounded job with Flink 1.15 > > In all cases, please verify that all records have been written at the end of > the scenario and there are no duplicates. > > > > > > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-26416) Release Testing: Sink V2 sanity checks
[ https://issues.apache.org/jira/browse/FLINK-26416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17500085#comment-17500085 ] Liu commented on FLINK-26416: - [~fpaul] Could you assign it to me? Thanks. > Release Testing: Sink V2 sanity checks > -- > > Key: FLINK-26416 > URL: https://issues.apache.org/jira/browse/FLINK-26416 > Project: Flink > Issue Type: Improvement > Components: Connectors / Common >Affects Versions: 1.15.0 >Reporter: Fabian Paul >Priority: Blocker > Labels: release-testing > Fix For: 1.15.0 > > > With the introduction of Sink V2, the operator model of the sink changed > slightly therefore it makes sense to test different upgrade/sanity scenarios. > > You can take any of the existing Sinks in the project. I would recommend the > FileSink. > > # Run a job with Flink 1.14 and take a savepoint and try to restore and > resume with 1.15 > # Run a job with Flink 1.15 and take a savepoint and try to restore and > resume with 1.15 > # Run a bounded job with Flink 1.15 > > In all cases, please verify that all records have been written at the end of > the scenario and there are no duplicates. > > > > > > -- This message was sent by Atlassian Jira (v8.20.1#820001)
