The coverage issue is only with the Java builds in specific. Go abd Python have their coverage numbers codecov uploads done in GitHub Actions instead.
On Tue, Apr 11, 2023, 8:14 AM Moritz Mack <mm...@talend.com> wrote: > Thanks so much for looking into this! > > I’m absolutely +1 for removing Jenkins related friction and the proposed > changes sound legitimate. > > > > Also, considering the number of flaky tests in general [1], code coverage > might not be the pressing issue. Should it be disabled everywhere in favor > of more reliable / faster builds? Unless Devs here are willing to commit on > taking actions, it doesn’t seem to provide too much value recording these > numbers as part of the normal pre commit jobs? > > > > Kind regards, > > Moritz > > > > [1] > https://github.com/apache/beam/issues?q=is%3Aissue+is%3Aopen+label%3Aflake > > > > On 11.04.23, 16:24, "Danny McCormick via dev" <dev@beam.apache.org> wrote: > > > > ;tldr - I want to temporarily reduce the number of builds that we retain > to reduce pressure on Jenkins Hey everyone, over the past few days our > Jenkins runs have been particularly flaky across the board, with errors > like the following showing > > *;tldr - I want to temporarily reduce the number of builds that we retain > to reduce pressure on Jenkins* > > > > Hey everyone, over the past few days our Jenkins runs have been > particularly flaky across the board, with errors like the following showing > up all over the place [1]: > > > > java.nio.file.FileSystemException: > /home/jenkins/jenkins-home/jobs/beam_PreCommit_Python_Phrase/builds/3352/changelog.xml: > No space left on device [2] > > > > These errors indicate that we're out of space on the Jenkins master node. > After some digging (thanks @Yi Hu <ya...@google.com> @Ahmet Altay > <al...@google.com> and @Bruno Volpato <bvolp...@google.com> for > contributing), we've determined that at least one large contributing issue > is that some of our builds are eating up too much space. For example, our > beam_PreCommit_Java_Commit build is taking up 28GB of space by itself (this > is just one example). > > > > @Yi Hu <ya...@google.com> found one change around code coverage that is > likely heavily contributing to the problem and rolled that back [3]. We can > continue to find other contributing factors here. > > > > In the meantime, to get us back to healthy *I propose that we reduce the > number of builds that we are retaining to 40 for all jobs that are using a > large amount of storage (>5GB)*. This will hopefully allow us to return > Jenkins to a normal functioning state, though it will do so at the cost of > a significant amount of build history (right now, for example, > beam_PreCommit_Java_Commit is at 400 retained builds). We could restore the > normal retention limit once the underlying problem is resolved. Given that > this is irreversible (and not guaranteed to work), I wanted to gather > feedback before doing this. Personally, I rarely use builds that old, but > others may feel differently. > > > > Please let me know if you have any objections or support for this proposal. > > > > Thanks, > > Danny > > > > [1] Tracking issue: https://github.com/apache/beam/issues/26197 > <https://urldefense.com/v3/__https:/github.com/apache/beam/issues/26197__;!!CiXD_PY!T2IvNgoS_ZY4KOTpgTeZ000JosJwCZPhtfAsW6XuJwTrlb8ok4GFLYwv5Yp30wxLqKghq03f6F2F$> > > [2] Example run with this error: > https://ci-beam.apache.org/job/beam_PreCommit_Python_Phrase/3352/console > <https://urldefense.com/v3/__https:/ci-beam.apache.org/job/beam_PreCommit_Python_Phrase/3352/console__;!!CiXD_PY!T2IvNgoS_ZY4KOTpgTeZ000JosJwCZPhtfAsW6XuJwTrlb8ok4GFLYwv5Yp30wxLqKghqwigO9QA$> > > [3] Rollback PR: https://github.com/apache/beam/pull/26199 > <https://urldefense.com/v3/__https:/github.com/apache/beam/pull/26199__;!!CiXD_PY!T2IvNgoS_ZY4KOTpgTeZ000JosJwCZPhtfAsW6XuJwTrlb8ok4GFLYwv5Yp30wxLqKghq5Cv-ikZ$> > > *As a recipient of an email from the Talend Group, your personal data will > be processed by our systems. Please see our Privacy Notice > <https://www.talend.com/privacy-policy/>*for more information about our > collection and use of your personal information, our security practices, > and your data protection rights, including any rights you may have to > object to automated-decision making or profiling we use to analyze support > or marketing related communications. To manage or discontinue promotional > communications, use the communication preferences portal > <https://info.talend.com/emailpreferencesen.html>. To exercise your data > protection rights, use the privacy request form > <https://talend.my.onetrust.com/webform/ef906c5a-de41-4ea0-ba73-96c079cdd15a/b191c71d-f3cb-4a42-9815-0c3ca021704cl>. > Contact us here <https://www.talend.com/contact/>or by mail to either of > our co-headquarters: Talend, Inc.: 400 South El Camino Real, Ste 1400, San > Mateo, CA 94402; Talend SAS: 5/7 rue Salomon De Rothschild, 92150 Suresnes, > France >
