pvary commented on pull request #3030: URL: https://github.com/apache/iceberg/pull/3030#issuecomment-906998384
> How do we want to treat test failures in the future? Do we want to open a ticket when somebody runs into a non-related failing test + disable the test in the meantime until it's fixed? We have a smaller, more tighter knit community here, and we usually were able to discuss and fix the issues as they were surfaced. Usually if someone found an issue, then they tried to fix it, or wrote to the dev list / created an issue and we were able to fix those without a formal process. In Hive we have much bigger codebase and bigger number of contributors, so we have a more formal process: - We have a jenkins job which runs a specific test case 100 times, and if any of the runs fail we call the test flaky - we use it to confirm a flaky tests - If a test is confirmed flaky, we disable it (even without test runs) - We create a jira "Reenable flaky test: ...." and try to ping users who were working on the tests lately OTOH if we have a better codebase, like we have here in Iceberg, we might be better off identifying the commit which caused the flakiness and reverting it (or if the issue is not caused by the test, then fixing the bug). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
