[GitHub] flink pull request: [FLINK-3651] Fix faulty RollingSink Restore
GitHub user aljoscha opened a pull request: https://github.com/apache/flink/pull/1830 [FLINK-3651] Fix faulty RollingSink Restore On restore the sink for subtask index i has to cleanup leftover files for subtask i. The pattern used for checking this was not properly terminated so the sink for subtask 1 would, for example, delete some files for sink i=11. This would lead to data loss. You can merge this pull request into a Git repository by running: $ git pull https://github.com/aljoscha/flink rolling-sink-fix Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/1830.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1830 commit a78d88f01a3d0da3bcfaad07c9abf419061a8025 Author: Aljoscha Krettek Date: 2016-03-07T16:15:20Z [FLINK-3651] Fix faulty RollingSink Restore On restore the sink for subtask index i has to cleanup leftover files for subtask i. The pattern used for checking this was not properly terminated so the sink for subtask 1 would, for example, delete some files for sink i=11. This would lead to data loss. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-3651] Fix faulty RollingSink Restore
Github user uce commented on the pull request: https://github.com/apache/flink/pull/1830#issuecomment-200795802 Good catch! The changes look good to me. I tried running the adjusted `RollingSinkFaultToleranceITCase` and `RollingSinkFaultTolerance2ITCase` w/o the fix in `RollingSink` and they still succeeded. If this expected? Is there a way we can test this differently? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-3651] Fix faulty RollingSink Restore
Github user rmetzger commented on the pull request: https://github.com/apache/flink/pull/1830#issuecomment-200796710 I think the test fails only sporadically, depending on the speed at which tasks are deleting the files. If you are lucky, everything is correct. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-3651] Fix faulty RollingSink Restore
Github user aljoscha commented on the pull request: https://github.com/apache/flink/pull/1830#issuecomment-200863307 Yes, what @rmetzger said is correct. Without the fix and the increase of parallelism it will be a flaky test that still succeeds sometimes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-3651] Fix faulty RollingSink Restore
Github user uce commented on the pull request: https://github.com/apache/flink/pull/1830#issuecomment-202794232 OK, depending on whether it is feasible to test this separately, I would go ahead and merge it as is or add a test and merge then. :+1: --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-3651] Fix faulty RollingSink Restore
Github user uce commented on the pull request: https://github.com/apache/flink/pull/1830#issuecomment-203004909 I'm going to merge this to `master` and `release-1.0`. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-3651] Fix faulty RollingSink Restore
Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/1830 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
