Re: source tarballs potentially built for each derivation
On 2021-08-01, Vagrant Cascadian wrote: > Turning this conversation into a bug, original thread around here: > > https://lists.gnu.org/archive/html/guix-devel/2021-05/threads.html#00427 For reference: bug#49810: source tarballs potentially built for each derivation live well, vagrant signature.asc Description: PGP signature
source tarballs potentially built for each derivation
Turning this conversation into a bug, original thread around here:
https://lists.gnu.org/archive/html/guix-devel/2021-05/threads.html#00427
On 2021-05-29, Vagrant Cascadian wrote:
> On 2021-05-01, Leo Famulari wrote:
>> On Sat, May 01, 2021 at 06:45:32PM -0700, Vagrant Cascadian wrote:
>>> Pragmatically speaking, on slower platforms this is a huge resource
>>> overhead. So much so that ci.guix.gnu.org *usually* times out when
>>> generating the linux-libre aarch64 tarballs:
>>>
>>>
>>> https://ci.guix.gnu.org/search?query=system%3Aaarch64-linux+linux-libre-arm64-generic
>>
>> Thanks for letting me know. I didn't know this was happening.
>>
>> The immediate solution is for me to make sure the tarballs have built
>> before committing the updates. I already do this for x86_64 and I can
>> start doing it for aarch64 too.
>
> This has definitely helped sometimes, thanks! I even saw a substitute of
> linux-libre for aarch64 earlier today! :)
>
> Still, I'm noticing another problem with the way way the tarballs are
> generated on ci.guix.gnu.org ...
>
> When it generates a tarball, all the various packages independently try
> to recreate the source tarball; so you have at least fours jobs
> ("linux-libre", "linux-libre-arm64-generic", "linux-libre-headers",
> "linux-libre-bpf") all concurrently trying to build the very same
> very-slow-to-build tarball on ci.guix.gnu.org. Sometimes one of them
> might succeed, but the others may not, and even though one of them
> succeeded, none of the failing ones retry...
>
> Not knowing exactly how ci.guix.gnu.org works, would it make sense to
> create a tarball package instead of the ... computed origin(?) tarball,
> so it could be better represented in the package dependency graph, and
> the various linux-libre-* packages can wait till it is available rather
> than all trying to recreate the same thing?
>
> That still requires the tarball generation to not time out in the first
> place, but maybe it would help with the resource limitations a bit to
> only build the source tarball once per architecture?
This seems to still be an issue for ci.guix.gnu.org, but the
linux-libre* substitutes for aarch64 seem to be available on
https://bordeaux.guix.gnu.org ...
$ guix weather linux-libre linux-libre-arm64-generic
computing 2 package derivations for aarch64-linux...
looking for 2 store items on https://ci.guix.gnu.org...
https://ci.guix.gnu.org
0.0% substitutes available (0 out of 2)
unknown substitute sizes
0.0 MiB on disk (uncompressed)
0.740 seconds per request (0.7 seconds in total)
1.4 requests per second
0.0% (0 out of 2) of the missing items are queued
1 queued builds
aarch64-linux: 1 (100.0%)
build rate: .00 builds per hour
x86_64-linux: 0.00 builds per hour
aarch64-linux: 0.00 builds per hour
i686-linux: 0.00 builds per hour
looking for 2 store items on https://bordeaux.guix.gnu.org...
https://bordeaux.guix.gnu.org
100.0% substitutes available (2 out of 2)
83.9 MiB of nars (compressed)
202.2 MiB on disk (uncompressed)
(continuous integration information unavailable)
live well,
vagrant
signature.asc
Description: PGP signature
Re: Adding Substitute Mirrors page to installer
raid5atemyhomework writes: > In any case, it looks to me that bordeaux is already in > `%default-substitute-mirrors`, which this patch uses, so it should get > included anyway as a fallback in case the SJTU mirror is not available > or something. So maybe the patch is OK as-is? I haven't looked at the changes here, but given bordeaux.guix.gnu.org is a default substitute server, I don't think it'll need special handling. signature.asc Description: PGP signature
Project direction with testing changes (branches and patches)
Hey, This is sort of a followup to [1], at least I think that's the last main email I sent out about testing changes (although I didn't use that term). I did also send out some notes from the Guix Day event back in February 2021 though [2]. 1: https://lists.gnu.org/archive/html/guix-devel/2020-11/msg00583.html 2: https://lists.gnu.org/archive/html/guix-devel/2021-02/msg00125.html Back in early 2020, I managed to start work on the Guix Build Coordinator [3]. That was meant to enable running reliable and performant substitute servers, but also meant to enable the kind of testing and quality assurance work that I had been thinking about, mostly through the perspective of testing patches. 3: https://lists.gnu.org/archive/html/guix-devel/2020-04/msg00323.html Getting the benefits to users didn't go as smoothly as I'd hoped, but since bordeaux.guix.gnu.org [4] launched back in June, there's a chance that the work on the Guix Build Coordinator has benefited users of Guix through improved substitutes. 4: https://guix.gnu.org/en/blog/2021/substitutes-now-also-available-from-bordeauxguixgnuorg/ As I said in [1], I did do some work last year to use the Guix Build Coordinator for testing patches and branches. Unfortunately the setup I'm using is currently not operating, I was having issues with running out of disk space on the main server, and I haven't got around to spending the time/money to resolve that. I want to get another iteration of the patch testing setup working, but recent experiences with working on providing substitutes has made me think that discussing the direction with maintainers and as a project is almost more important. So, I think I've recently switched to thinking about the problem as one of testing changes, rather than just testing patches. Since both patch series, and branches are used to propose changes, I think this makes sense. In abstract, when testing a change, I would break down the problem as follows: - You need to work out what's affected by the change, so that you can assess the impact - Once you know what's effected, you can then build those packages/system tests/... and compare the build statuses and outputs against some baseline - Then there's the general UI component, ideally a first time contributor would be able to take advantage of automatic feedback about a patch they submit. There's multiple other groups of users though, like patch reviewers, and committers for example. I think the first two sub-problems are effectively solved. The Guix Data Service is able to determine the changes between two revisions (assuming it's processed them). The Guix Build Coordinator can then be used to build the relevant packages/system tests, and report that information back to the Guix Data Service. The UI part is much less certain, I've done some work with Patchwork, and I do have some ideas in mind, but there's still more thinking and work to do in this area. Before pressing on though, I think it would be good to know if this is a viable direction? Currently, there's no automated testing of patches, and testing of branches is limited to the information that Cuirass provides on failed builds. What I'm proposing for the future is: using the Guix Data Service together with the Guix Build Coordinator to analyse the effects of changes, whether that be from a patch series or a branch. I realise that I've already been experimenting with this, what I'm mostly referring to here is moving towards this being the documented approach, maintained by the project, not just me. So yes, is this something that people want, or don't want? If you're uncertain and have questions, it would be good to know what those questions are? Thanks, Chris signature.asc Description: PGP signature
Re: Adding Substitute Mirrors page to installer
BUMP
