Thanks for sharing the pain. That's the first step in improving something that is painful.
For the flaky tests GitHub Actions workflow pulsar-ci-flaky.yaml, the Codecov upload should be a separate job in the workflow so that the upload could be retried separately without running all tests. This type of approach is already used in the main GitHub Actions workflow, "Pulsar CI". Contributions are welcome! We could also consider disabling codecov for pull request builds until someone who cares about test code coverage metrics picks up the work. Code coverage is the first metric that most will ask about tests. It's not the only metric that matter, but it is something that helps understand what parts of the code isn't even run in our tests. It will also help plan improvements to tests. Codecov upload fails very frequently with errors such as https://github.com/codecov/codecov-action/issues/837 and https://github.com/codecov/codecov-action/issues/598 One possible resolution is https://community.codecov.com/t/upload-issues-unable-to-locate-build-via-github-actions-api/3954 . It's possible to make the codecov upload more stable by providing a token. This should be done for the master branch build so that the baseline code coverage metrics would succeed. For pull requests, the solution is to make the codecov upload retryable also in pulsar-ci-flaky.yaml. In addition, it could be made optional for builds in own forks. We should find a way as a development community to get code coverage metrics solution working. It is valuable even if an individual developer doesn't care about it at the moment. We need more Pulsar contributors to stand up that care about the quality aspects of our code base. Any volunteers? -Lari On 2023/03/21 10:50:17 tison wrote: > For example > https://github.com/apache/pulsar/actions/runs/4454158774/jobs/7867745340?pr=19842 > > I'm wondering if anyone cares about the report and if it helps you during > the coding or reviewing process? Now it generates a few of noise but I just > omit the report it gives ;-) > > For the issue itself, it seems some artifacts don't retain properly. > > Best, > tison. >
