Re: coreutils CI

2023-05-13 Thread Pádraig Brady 

On 11/05/2023 15:31, Bruno Haible wrote:

Pádraig Brady wrote:

@Padraig: you're at least doing some pre-release tests on several platforms,
but that's no real CI, is it?


Right.
No real CI at present.
It would be good though, directly for coreutils
and indirectly for gnulib.


We have a CI for gnulib at Gitlab [1], but it detects only failures that are
covered by the unit tests, and — as you know — some of the complicated uses
of gnulib are only covered by coreutils tests, not by gnulib tests.

We have a CI also for a couple of GNU packages at Gitlab [2][3][4][5][6][7][8],
and they occasionally detect regressions. But with only 400 free CI minutes per
month [9], you couldn't have very many "expensive" test runs with coreutils
on that platform, even when using the coreutils mirror that someone already
has set up [10].


I think I'll have a look at setting up
some automated testing on the GCC compile farm.


Reading [11], it looks like this would be feasible, if you pick a machine that
has lots of CPU power and use 'ulimit' "to reduce the risk of a DOS attack".

I agree that it would be good.

Bruno

[1] https://gitlab.com/gnulib/gnulib-ci/-/pipelines
[2] https://gitlab.com/gnu-diffutils/ci-distcheck/-/pipelines
[3] https://gitlab.com/gnu-gettext/ci-distcheck/-/pipelines
[4] https://gitlab.com/gnu-grep/ci-distcheck/-/pipelines
[5] https://gitlab.com/gnu-gzip/ci-distcheck/-/pipelines
[6] https://gitlab.com/gnu-libunistring/ci-distcheck/-/pipelines
[7] https://gitlab.com/gnu-m4/ci-distcheck/-/pipelines
[8] https://gitlab.com/gnu-sed/ci-distcheck/-/pipelines
[9] https://about.gitlab.com/pricing/
[10] https://gitlab.com/saiwp/coreutils
[11] https://gcc.gnu.org/wiki/CompileFarm


OK, a daily coreutils build/test run is now in place
against the latest coreutils and gnulib,
on 1 machine in the compile farm.

Failures currently go to my desktop notifications.

I may get time to expand to multiple machines,
and make the results more consumable.

cheers,
Pádraig

Re: coreutils CI

2023-05-11 Thread Bruno Haible 
Pádraig Brady wrote:
> > @Padraig: you're at least doing some pre-release tests on several platforms,
> > but that's no real CI, is it?
> 
> Right.
> No real CI at present.
> It would be good though, directly for coreutils
> and indirectly for gnulib.

We have a CI for gnulib at Gitlab [1], but it detects only failures that are
covered by the unit tests, and — as you know — some of the complicated uses
of gnulib are only covered by coreutils tests, not by gnulib tests.

We have a CI also for a couple of GNU packages at Gitlab [2][3][4][5][6][7][8],
and they occasionally detect regressions. But with only 400 free CI minutes per
month [9], you couldn't have very many "expensive" test runs with coreutils
on that platform, even when using the coreutils mirror that someone already
has set up [10].

> I think I'll have a look at setting up
> some automated testing on the GCC compile farm.

Reading [11], it looks like this would be feasible, if you pick a machine that
has lots of CPU power and use 'ulimit' "to reduce the risk of a DOS attack".

I agree that it would be good.

Bruno

[1] https://gitlab.com/gnulib/gnulib-ci/-/pipelines
[2] https://gitlab.com/gnu-diffutils/ci-distcheck/-/pipelines
[3] https://gitlab.com/gnu-gettext/ci-distcheck/-/pipelines
[4] https://gitlab.com/gnu-grep/ci-distcheck/-/pipelines
[5] https://gitlab.com/gnu-gzip/ci-distcheck/-/pipelines
[6] https://gitlab.com/gnu-libunistring/ci-distcheck/-/pipelines
[7] https://gitlab.com/gnu-m4/ci-distcheck/-/pipelines
[8] https://gitlab.com/gnu-sed/ci-distcheck/-/pipelines
[9] https://about.gitlab.com/pricing/
[10] https://gitlab.com/saiwp/coreutils
[11] https://gcc.gnu.org/wiki/CompileFarm

Re: coreutils CI

2023-05-11 Thread Pádraig Brady 

On 11/05/2023 07:58, Bernhard Voelker wrote:

On 5/9/23 20:57, Bruno Haible wrote:

Bernhard Voelker wrote:

I noticed the issue because the following "very-expensive" tests failed
(which succeeded with coreutils-9.3):

 FAIL: tests/rm/ext3-perf
 FAIL: tests/rm/many-dir-entries-vs-OOM


Did you notice these failures by chance, or do you run coreutils continuous
integration on a fixed schedule?


I regularly run the 'check-very-expensive' tests on my local clone.


More generally, for which systems are coreutils continuous integrations being
run regularly? For which systems would it be useful to have it?

https://www.gnu.org/software/coreutils/ references
https://hydra.nixos.org/jobset/gnu/coreutils-master but it seems that this
CI stopped working in 2021.


I know the Hydra tests have stopped working, but some (small) attempts to get
support to get it working again failed.

@Padraig: you're at least doing some pre-release tests on several platforms,
but that's no real CI, is it?


Right.
No real CI at present.
It would be good though, directly for coreutils
and indirectly for gnulib.

I think I'll have a look at setting up
some automated testing on the GCC compile farm.

cheers,
Pádraig

Re: coreutils CI

2023-05-11 Thread Bernhard Voelker 

On 5/9/23 20:57, Bruno Haible wrote:

Bernhard Voelker wrote:

I noticed the issue because the following "very-expensive" tests failed
(which succeeded with coreutils-9.3):

FAIL: tests/rm/ext3-perf
FAIL: tests/rm/many-dir-entries-vs-OOM


Did you notice these failures by chance, or do you run coreutils continuous
integration on a fixed schedule?


I regularly run the 'check-very-expensive' tests on my local clone.


More generally, for which systems are coreutils continuous integrations being
run regularly? For which systems would it be useful to have it?

https://www.gnu.org/software/coreutils/ references
https://hydra.nixos.org/jobset/gnu/coreutils-master but it seems that this
CI stopped working in 2021.


I know the Hydra tests have stopped working, but some (small) attempts to get
support to get it working again failed.

@Padraig: you're at least doing some pre-release tests on several platforms,
but that's no real CI, is it?

Have a nice day,
Berny

Re: coreutils CI

2023-05-09 Thread Bruno Haible 
Bernhard Voelker wrote:
> Gnulib commit 3f0950f65abb (2023-04-26) not only lead to build time
> issues, but also made e.g. coreutils' rm(1) fail:
> ...
> I can confirm that this gnulib commit d4d8abb39eb0 fixes the issue again.

Yep, mistakes do happen. Gnulib has stable branches that can protect some
packages from these breakages

but AFAIU coreutils frequently needs new features from Gnulib and therefore
cannot use the stable branches.

> I noticed the issue because the following "very-expensive" tests failed
> (which succeeded with coreutils-9.3):
> 
>FAIL: tests/rm/ext3-perf
>FAIL: tests/rm/many-dir-entries-vs-OOM

Did you notice these failures by chance, or do you run coreutils continuous
integration on a fixed schedule?

More generally, for which systems are coreutils continuous integrations being
run regularly? For which systems would it be useful to have it?

https://www.gnu.org/software/coreutils/ references
https://hydra.nixos.org/jobset/gnu/coreutils-master but it seems that this
CI stopped working in 2021.

Bruno