Hi, l...@gnu.org (Ludovic Courtès) writes:
>> 2.1 The testing infrastructure >> ────────────────────────────── >> >> Currently Cuirass is providing a small test suite consisting only of >> unit tests. The integration tests are done manually with basic >> examples consisting in building small packages such as GNU Hello. >> This is not good, because actual packages take a consequent time to >> compile, and testing only basic examples leaves the interesting >> complex case out of the scope of testing. The worst is that the >> result depend on the phase of the moon since every commit in Guix can >> potentially break the build. >> >> The idea is to take inspiration from the Guix test suite by providing >> tiny mocked packages that will help having lightweight build >> specifications. Those tests require launching another instance of the >> `guix-daemon' with a local temporary store. > > Sounds like a good idea. > > It would be nice to refine what it is we’re trying to test and that’s > not addressed by unit tests. At one end of the spectrum, I imagine we > could be spawning a couple of GuixSD VMs connected together, one serving > a Git repo, and the other one running Cuirass pulling from that repo (a > system test). > > At the middle of the spectrum is something like you describe, I think: > run Cuirass directly upon “make check”, though there are complications: > what repo do we pull from, do we really want to build packages and if > not how to we mock builds, etc. The idea I had was to create fake vcs repositories programmatically. I imagine that it is possible to associate some dummy build script that we can deterministically make fail. > There are several components we’d like to test more closely IMO: SCM > polling, evaluation, and job scheduling. Perhaps some of that can also > be achieved through unit tests? Yeah, that would be better. >> 2.2 HTTP API >> ──────────── >> >> Since commit [12d71ee098c3eb84a9a5dc2419bd37a3b9e55dd2] Cuirass has a >> web server which is running in a separate thread of the `cuirass' >> process. However the resources it provides are pretty limited since >> you can only query the lists of specifications currently watched. >> This is done with the `/specifications' or `/jobsets' resource names. >> "Jobsets" is the terminology used by Hydra and is used here for >> compatibility with its HTTP API. > > For a start, I think we should implement all or a subset of the Hydra > API, as-is (it’s currently used by Emacs-Guix): > > https://github.com/NixOS/hydra/blob/master/src/lib/Hydra/Controller/API.pm > > A large part of it is about monitoring the status of Hydra: queued > builds, recently-completed builds, jobs, etc. That is the most pressing > need, I think. OK. >> This roadmap lacks details about the concrete tasks and its associated >> timeline. However I prefer having some feedback about this proposal >> first, before providing a complete roadmap. >> >> • From May 30 to June 26, I will work on the testing infrastructure,. >> • From June 27 to August 29, I will work on the HTTP API. > > You may find that you’ll want to interleave tasks. I don’t know about > you, but sometimes I get bored if I keep working on the same task or > domain for too long, so switching to something else for a while can be a > relief. :-) I don't know, I have been rarely confronted to that issue personnaly (more the opposite: too much context switching). But I will preventively adapt my roadmap. > There are a couple of isolated tasks that come to mind, which do not > really fit in the proposal but are worth looking into as time permits: > > • improved error reporting upon evaluation failures and so on; > > • improved error recovery; I totally forgot to precise that the error handling issue is exactly the reason I think we need a better test infrastructure for Cuirass. Being able to reproduce various errors or failues allows a better confidence in the error being properly handled. I find it hard to just imagine and write the appropriate handler. > • use of Guile-Git instead of Git. I Will add this to the plan. Thanks. -- Mathieu Lirzin GPG: F2A3 8D7E EB2B 6640 5761 070D 0ADE E100 9460 4D37