Re: continuous integration
Jeffrey Walton wrote: > CI tests should be catching these mistakes. (And problems like > _NoReturn on OS X). Yes, CI can catch some mistakes. Like, just last week, this one: [1]. Tim and I maintain a continuous integration for gnulib at [2]. More effort could be put in, in two directions: * Like Paul says, instead of only building testdirs, it could build some packages that use gnulib. I would estimate that this would catch 3x as many bugs as the current CI with just testdirs. * Like you suggest, it would also be useful to test macOS, FreeBSD, Cygwin, and mingw builds. > Is there any reasons services like Travis or Cirrus are not being used > to proactively detect problems on Linux, OS X and FreeBSD? For my part: * I have only limited time to work on this; that's why I limit myself to CI integrations for a couple of packages on gitlab. * I had not heard of Cirrus CI. Coverage of FreeBSD, additionally to Windows and macOS, sounds interesting. [3] * Travis and Cirrus CI are most easily used on Github [4][5]. I don't much like to work on Github, because it tends to become a closed environment. E.g. - You can fork someone else's repository only if you stay on Github. - Many developers' email addresses are not published, which prevents you from reporting issues by email. You have to use Github "issues" instead. But if someone wants to set it up and maintain it, I'm all for it! Bruno [1] https://lists.gnu.org/archive/html/bug-gnulib/2020-03/msg00041.html [2] https://gitlab.com/gnulib/gnulib-ci [3] https://cirrus-ci.org/features/#comparison-with-popular-ciaas [4] https://en.wikipedia.org/wiki/Travis_CI [5] https://cirrus-ci.org/faq/#only-github-support
Re: continuous integration
On 26.03.20 00:46, Bruno Haible wrote: > Jeffrey Walton wrote: >> CI tests should be catching these mistakes. (And problems like >> _NoReturn on OS X). > > Yes, CI can catch some mistakes. Like, just last week, this one: [1]. > > Tim and I maintain a continuous integration for gnulib at [2]. > > More effort could be put in, in two directions: > > * Like Paul says, instead of only building testdirs, it could build > some packages that use gnulib. I would estimate that this would catch > 3x as many bugs as the current CI with just testdirs. > > * Like you suggest, it would also be useful to test macOS, FreeBSD, > Cygwin, and mingw builds. > >> Is there any reasons services like Travis or Cirrus are not being used >> to proactively detect problems on Linux, OS X and FreeBSD? > > For my part: > > * I have only limited time to work on this; that's why I limit > myself to CI integrations for a couple of packages on gitlab. Same here. I really wish we could had more time to put into CI runners. I would like to point out that debugging using a CI like Travis is absolutely tedious and might take a lot of time. Docker-based CI (Linux only :-| ) are so much easier to debug as you can run the test environment (images + build scripts) locally. So while some errors are obvious and easy to fix, others are a nightmare as you can't 'log in' and just use a debugger. At least I don't have VMs with OSX or Windows for this purpose. Did anyone think about using the gcc build platform for automated testing ? I made up some scripts a while ago for Wget but then lost focus... if someone likes to take that up. > * I had not heard of Cirrus CI. Coverage of FreeBSD, additionally to > Windows and macOS, sounds interesting. [3] > > * Travis and Cirrus CI are most easily used on Github [4][5]. I don't > much like to work on Github, because it tends to become a closed > environment. E.g. > - You can fork someone else's repository only if you stay on Github. > - Many developers' email addresses are not published, which prevents > you from reporting issues by email. You have to use Github "issues" > instead. We just need a mirror / fork on Github that we push to (sync) from time to time. If someone cares for the initial Travis and/or Cirrus setup with OSX / FreeBSD / Windows in mind, that would be great ! > But if someone wants to set it up and maintain it, I'm all for it! > > Bruno > > [1] https://lists.gnu.org/archive/html/bug-gnulib/2020-03/msg00041.html > [2] https://gitlab.com/gnulib/gnulib-ci > [3] https://cirrus-ci.org/features/#comparison-with-popular-ciaas > [4] https://en.wikipedia.org/wiki/Travis_CI > [5] https://cirrus-ci.org/faq/#only-github-support > > Regards, Tim signature.asc Description: OpenPGP digital signature
Re: continuous integration
On Thu, 26 Mar 2020 at 05:51, Tim Rühsen wrote: > > We just need a mirror / fork on Github that we push to (sync) from time > to time. If someone cares for the initial Travis and/or Cirrus setup > with OSX / FreeBSD / Windows in mind, that would be great ! If such a fork/mirror is going to show up I'll add a .cirrus.yml to get at least FreeBSD coverage (and will add others if I manage to get them going easily).
Re: continuous integration
Hi Ed, On 11.06.20 21:23, Ed Maste wrote: > On Thu, 26 Mar 2020 at 05:51, Tim Rühsen wrote: >> >> We just need a mirror / fork on Github that we push to (sync) from time >> to time. If someone cares for the initial Travis and/or Cirrus setup >> with OSX / FreeBSD / Windows in mind, that would be great ! > > If such a fork/mirror is going to show up I'll add a .cirrus.yml to > get at least FreeBSD coverage (and will add others if I manage to get > them going easily). You can mirror gnulib on your own Github account to add/test Cirrus CI. I am not sure that gnulib will have an official account on Github, as Github is very proprietary. Another option is to get access to a FreeBSD machine and install the Gitlab runner [1]. With that (plus a token from our Gitlab account), the machine should appear on our list of CI runners and we can integrate it into the .gitlab-ci.yml. Regards, Tim [1] https://docs.gitlab.com/runner/install/ signature.asc Description: OpenPGP digital signature
help needed for continuous integration
Hi all, If you are familiar with Travis CI or Appveyor CI, please help keeping gnulib at a high quality! It is normal for a developer to push a buggy commit by mistake. But it should not be normal that such a mistake, that leads to a test failure on standard glibc systems, remains unnoticed for 10 days. It would be good to have a continuous integration (Travis or Appveyor, I don't mind) of gnulib, to detect test failures early. There is already a github mirror of the gnulib git repo, at https://github.com/coreutils/gnulib, and it is apparently kept up-to-date automatically. The continuous integration should probably run these commands: $ ./gnulib-tool --create-testdir --dir=/some/testdir --single-configure \ --with-c++-tests --without-privileged-tests $ cd /some/testdir $ make $ make check Please speak up and help! Bruno
continuous integration for many platforms
The continuous integration of Gnulib for many platforms is now operational. <https://github.com/gnu-gnulib/ci-testdir-check/actions> It tests a testdir of nearly all modules of Gnulib on the following platforms: - Ubuntu GNU/Linux 22.04 - CentOS 7 - Alpine Linux - macOS 11, 12, 13 (all x86_64) - macOS 14 (arm64) - FreeBSD 14.0 - NetBSD 10.0 - OpenBSD 7.5 - Solaris 11.4 - Solaris 11 OmniOS - Cygwin 3.3.6 and 3.5.3 - mingw (32 bit and 64 bit) - MSVC (32 bit and 64 bit) and, as a "goodie", also - on Ubuntu GNU/Linux 22.04 with clang's UBSAN and ASAN sanitizers. It will run once every week, plus it's also possible to trigger a run at any moment. Platforms that are not covered and that therefore continue to need occasional manual testing: - GNU/Hurd, - AIX, - Android, - other architectures (from Linux/alpha to Solaris/SPARC). Regarding AIX, I've been told that it's unlikely that there will ever be a GitHub runner. If you would like to get involved a) to be able to trigger a run, b) by receiving the failure reports and triaging failures, c) by maintaining the CI when things change on the GitHub site, please tell me and I can assign you the permissions. Btw., the older CI <https://gitlab.com/gnulib/gnulib-ci/-/pipelines> is still active. But it runs only on Debian machines and does therefore not report issues frequently. Bruno
Re: help needed for continuous integration
Am Sonntag, den 04.02.2018, 11:20 +0100 schrieb Bruno Haible: > Hi all, > > If you are familiar with Travis CI or Appveyor CI, please help > keeping gnulib > at a high quality! > > It is normal for a developer to push a buggy commit by mistake. But > it should > not be normal that such a mistake, that leads to a test failure on > standard > glibc systems, remains unnoticed for 10 days. > > It would be good to have a continuous integration (Travis or > Appveyor, I > don't mind) of gnulib, to detect test failures early. > > There is already a github mirror of the gnulib git repo, at > https://github.com/coreutils/gnulib, and it is apparently kept up-to- > date > automatically. > > The continuous integration should probably run these commands: > $ ./gnulib-tool --create-testdir --dir=/some/testdir --single- > configure \ > --with-c++-tests --without-privileged-tests > $ cd /some/testdir > $ make > $ make check Hi Bruno, I have CI experience with several projects and I am willing to help. >From my experience and knowledge, the Gitlab CI is much more configurable than e.g. Travis. It is docker based and thus limited to all kinds of Linux variants (including cross-platform builds, e.g. MinGW). Therefore I use Travis for OSX testing only. I have no experience with AppVeyor wich would be useful for native Windows testing. As a starter, I could set up a gnulib test CI as a subproject in https: //gitlab.com/gnuwget that syncs+tests gnulib e.g. once a day. WDYT ? Regards, Tim
Re: help needed for continuous integration
Hi Tim, > I have CI experience with several projects and I am willing to help. Cool! > From my experience and knowledge, the Gitlab CI is much more > configurable than e.g. Travis. It is docker based and thus limited to > all kinds of Linux variants (including cross-platform builds, e.g. > MinGW). Therefore I use Travis for OSX testing only. > > I have no experience with AppVeyor wich would be useful for native > Windows testing. Good, agree with Gitlab CI as a starter (since glibc systems are the most important platforms to test). > As a starter, I could set up a gnulib test CI as a subproject in https: > //gitlab.com/gnuwget that syncs+tests gnulib e.g. once a day. I think it would be better to register https://gitlab.com/gnulib as a new project. (In the long run, the intersection between wget maintainers and gnulib maintainers may be empty.) Would you be willing to do that, please? "syncs+tests gnulib once a day" sounds good. Bruno
Re: help needed for continuous integration
Am Sonntag, den 04.02.2018, 13:29 +0100 schrieb Bruno Haible: > Hi Tim, > > > I have CI experience with several projects and I am willing to > > help. > > Cool! > > > From my experience and knowledge, the Gitlab CI is much more > > configurable than e.g. Travis. It is docker based and thus limited > > to > > all kinds of Linux variants (including cross-platform builds, e.g. > > MinGW). Therefore I use Travis for OSX testing only. > > > > I have no experience with AppVeyor wich would be useful for native > > Windows testing. > > Good, agree with Gitlab CI as a starter (since glibc systems are the > most important platforms to test). > > > As a starter, I could set up a gnulib test CI as a subproject in > > https: > > //gitlab.com/gnuwget that syncs+tests gnulib e.g. once a day. > > I think it would be better to register https://gitlab.com/gnulib as a > new > project. (In the long run, the intersection between wget maintainers > and > gnulib maintainers may be empty.) Would you be willing to do that, > please? > > "syncs+tests gnulib once a day" sounds good. Hi Bruno, basically set up a minimalistic group and two projects. One project to build the docker images, another one to fetch gnulib and test it. The stretch CI image is already built, the testing CI still waiting to be processed. Meanwhile you should create a Gitlab account and tell me your nick. So I can invite you and give you the appropriate permissions. There is much to be tuned (permissions, build optimizations, different kind of builds/tests e.g. with/without sanitizers). And there is much to learn about the Gitlab UI. Maybe it's better to discuss those details via PM - feel free to contact me. Regards, Tim
Re: continuous integration for many platforms
On Thu, Jun 6, 2024 at 5:34 PM Bruno Haible wrote: > > The continuous integration of Gnulib for many platforms is now operational. > <https://github.com/gnu-gnulib/ci-testdir-check/actions> Congrats. That is a great accomplishment. Jeff
Re: continuous integration for many platforms
Hi Bruno, Bruno Haible writes: > The continuous integration of Gnulib for many platforms is now operational. > <https://github.com/gnu-gnulib/ci-testdir-check/actions> > > It tests a testdir of nearly all modules of Gnulib on the following platforms: Thanks! It already seems helpful from the tests I have worked on so far. Now we should be able to catch errors quickly as things update, etc. > Platforms that are not covered and that therefore continue to need occasional > manual testing: > - GNU/Hurd, > - AIX, > - Android, > - other architectures (from Linux/alpha to Solaris/SPARC). > > Regarding AIX, I've been told that it's unlikely that there will ever be a > GitHub runner. I assume the architecture problem is an issue for most free software projects. The compile farm should help with that and AIX, as long as they are available at least. > If you would like to get involved > a) to be able to trigger a run, > b) by receiving the failure reports and triaging failures, > c) by maintaining the CI when things change on the GitHub site, > please tell me and I can assign you the permissions. It would be nice to be able to trigger runs and help fix any failures. My GitHub is name is "collinfunk" [1]. I'm not sure if you have any "rules" for the repository, but I think it is best to follow the standard Gnulib procedure. Send any meaningful patches on the list so others can comment. Simple stuff like updating an 'apt' invocation to install necessary packages, for example, can be done silently. Collin [1] https://github.com/collinfunk
Re: continuous integration for many platforms
Collin Funk wrote: > Now we should be able to catch errors quickly as things update, etc. Right. With this multi-platform-CI is it now easier to add a unit test: Just make sure that it follows the specification and works on two or three platforms, commit it, and then we'll see (within a week at most) which other platforms need tweaks. > It would be nice to be able to trigger runs and help fix any failures. > My GitHub is name is "collinfunk" [1]. OK. Thanks for volunteering. > I'm not sure if you have any "rules" for the repository, but I think it > is best to follow the standard Gnulib procedure. Send any meaningful > patches on the list so others can comment. Simple stuff like updating an > 'apt' invocation to install necessary packages, for example, can be done > silently. Yes and no. This mailing list has quite a wide audience. I don't want to have lots of chatter about CI details on this list. Therefore, what belongs on this list is: - Discussion whether a certain failure is a platform bug, a Gnulib code bug, a Gnulib test bug, or a CI bug — unless it's obvious. - Major decisions, such as about adding another platform to the CI. But for the rest, such as messages like "I'm starting to investigate the NetBSD failure in the last run" or "do you know the package name of the locales on Alpine Linux", we should better use private email. If private email at some point does not work well, then we should better create a separate mailing list. Bruno
Re: continuous integration for many platforms
Hi Bruno, Bruno Haible writes: > Yes and no. This mailing list has quite a wide audience. I don't want to > have lots of chatter about CI details on this list. Therefore, what belongs > on this list is: > - Discussion whether a certain failure is a platform bug, a Gnulib code > bug, a Gnulib test bug, or a CI bug — unless it's obvious. > - Major decisions, such as about adding another platform to the CI. > But for the rest, such as messages like "I'm starting to investigate > the NetBSD failure in the last run" or "do you know the package name > of the locales on Alpine Linux", we should better use private email. > If private email at some point does not work well, then we should better > create a separate mailing list. Sounds good to me, thanks. Now that I think about it 99% of changes will probably be made while investigating some platform specific bug. Therefore your explanation makes more sense. Collin
Re: continuous integration for many platforms
I wrote: > The continuous integration of Gnulib for many platforms is now operational. > <https://github.com/gnu-gnulib/ci-testdir-check/actions> Let me document it in the HACKING file. 2024-06-07 Bruno Haible Update HACKING. * HACKING: Mention the new many-platforms continuous integration. diff --git a/HACKING b/HACKING index 34c3adf033..8ea5ae7791 100644 --- a/HACKING +++ b/HACKING @@ -131,9 +131,36 @@ and test this directory on various platforms: - Android, - and other platforms of your choice. -There is a continuous integration that regularly performs this testing -on a Linux/glibc system: https://gitlab.com/gnulib/gnulib-ci -But this will catch only the most blatant mistakes. +There are two continuous integrations that regularly perform this testing: +* On a Linux/glibc system only: + https://gitlab.com/gnulib/gnulib-ci + This one will catch only the most blatant mistakes. +* On many platforms: + https://github.com/gnu-gnulib/ci-testdir-check/actions + This one runs on many platforms, currently (as of June 2024): + - Ubuntu GNU/Linux 22.04 + - CentOS GNU/Linux 7 + - Alpine Linux + - macOS 11, 12, 13 (all x86_64) + - macOS 14 (arm64) + - FreeBSD 14.0 + - NetBSD 10.0 + - OpenBSD 7.5 + - Solaris 11.4 + - Solaris 11 OmniOS + - Cygwin 3.3.6 (32 bit) and 3.5.3 (64 bit) + - mingw (32 bit and 64 bit) + - MSVC (32 bit and 64 bit) + and also + - on Ubuntu GNU/Linux 22.04 with clang's UBSAN and ASAN sanitizers. + This one catches real portability problems. + Note that the following platforms are not covered and thus still require + occasional manual testing: + - AIX + - Solaris 10 + - Haiku + - Android + - and other platforms of your choice. Warning Options
