On Wed, Apr 17, 2019 at 12:32 PM E. Madison Bray <[email protected]> wrote: > > On Tue, Apr 16, 2019 at 2:21 PM E. Madison Bray <[email protected]> wrote: > > > > On Tue, Apr 16, 2019 at 2:11 PM Dima Pasechnik <[email protected]> wrote: > > > > > > > > > > > > On Tue, 16 Apr 2019 14:04 E. Madison Bray, <[email protected]> wrote: > > >> > > >> Hi Daniel, > > >> > > >> GitLab work is still going and any help would be appreciated. The > > >> biggest hurdle at the moment remains infrastructure. We both lack the > > >> amount of necessary physical infrastructure to keep builds going, as > > >> well as the human infrastructure to regularly monitor the builds and > > >> address problems. > > >> > > >> In fact, since you brought it up, I just realized that the gitlab > > >> runner I'm administering has been broken for a couple weeks. > > > > > > > > > > > > it could be as it was using VMs from my Google grant that expired, so I > > > had to shut them down. > > > > How much was the grant for? I thought we set that up like 2 months > > ago at the most. Disconcerting that it expired that quickly... > > > > > > I don't know what happened with my openstack-based runner, but I > > kicked it off again and it's doing a build-from-clean of 8.8.beta2 > > now: https://gitlab.com/sagemath/sage/-/jobs/197508827 > > > > I'll need to look into what I can do to improve monitoring so that I > > become aware of any problems sooner...
> I don't know what's going on. Meanwhile the VM that ran this build is > still hanging around, and my gitlab-runner master server isn't > spawning a new one... Oops this is just me forgetting / misunderstanding how gitlab-runner works again. I'm using the docker+machine executor [1] which runs docker-machine from the gitlab-runner master host to spawn VMs to host the actual builds in Docker containers. It doesn't spawn a VM per-build; just a container. This allows you to spawn build VMs as-needed, and destroy them when they go idle. Currently I have just IdleCount=1 so it only spawns one VM at a time anyways, and IdleTime=600 (10 minutes) which is never reached since currently my build runner is like, the only one dedicated to Sage that's actually working so it's always busy. And indeed it's busy right now on another job. Unfortunately that still doesn't explain why the last build-from-clean job failed randomly. I'll try bumping IdleCount=4 and see how that goes, since I think I can get away with that... [1] https://docs.gitlab.com/runner/executors/docker_machine.html -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout.
