On Tue, Apr 16, 2019 at 2:21 PM E. Madison Bray <[email protected]> wrote: > > On Tue, Apr 16, 2019 at 2:11 PM Dima Pasechnik <[email protected]> wrote: > > > > > > > > On Tue, 16 Apr 2019 14:04 E. Madison Bray, <[email protected]> wrote: > >> > >> Hi Daniel, > >> > >> GitLab work is still going and any help would be appreciated. The > >> biggest hurdle at the moment remains infrastructure. We both lack the > >> amount of necessary physical infrastructure to keep builds going, as > >> well as the human infrastructure to regularly monitor the builds and > >> address problems. > >> > >> In fact, since you brought it up, I just realized that the gitlab > >> runner I'm administering has been broken for a couple weeks. > > > > > > > > it could be as it was using VMs from my Google grant that expired, so I > > had to shut them down. > > How much was the grant for? I thought we set that up like 2 months > ago at the most. Disconcerting that it expired that quickly... > > > I don't know what happened with my openstack-based runner, but I > kicked it off again and it's doing a build-from-clean of 8.8.beta2 > now: https://gitlab.com/sagemath/sage/-/jobs/197508827 > > I'll need to look into what I can do to improve monitoring so that I > become aware of any problems sooner...
Ugh. The last build failed yet again for no apparent reason: https://gitlab.com/sagemath/sage/-/jobs/197508827 This seems to be a recurring problem. It gets almost all the way to the end and then tries to upload some build artifacts: Step 68/76 : RUN if [ x"$ARTIFACT_BASE" != x"source-clean" ]; then mkdir -p $HOME/patch && find . -type f > $HOME/make-fast-rebuild-clean.manifest && cat $HOME/make-fast-rebuild-clean.manifest $HOME/artifact-base.manifest | sort | uniq -u > $HOME/obsolete && find . -type f -cnewer $HOME/artifact-base.manifest > $HOME/modified && tar -cJf $HOME/patch/modified.tar.xz -T $HOME/modified && cat $HOME/obsolete $HOME/modified | xz > $HOME/patch/modified.xz && rm -rf $SAGE_ROOT && mkdir -p $SAGE_ROOT && mv $HOME/patch $SAGE_ROOT/; fi ---> Running in e14d3e44e129 Removing intermediate container e14d3e44e129 ---> 93a2aa0fe57f Step 69/76 : FROM $ARTIFACT_BASE as sagemath-dev ---> 6d284e9c7c71 Step 70/76 : ARG SAGE_ROOT=/home/sage/sage ---> Running in ed3efc505105 Removing intermediate container ed3efc505105 ---> 4e3f781d0f96 Step 71/76 : COPY --chown=sage:sage --from=sagemath-dev-patch $SAGE_ROOT $SAGE_ROOT Uploading artifacts... gitlab-build-docker.log: found 1 matching files Uploading artifacts to coordinator... ok id=197508827 responseStatus=201 Created token=8HA3zms- ERROR: Job failed: exit code 1 But then it fails here for no apparent reason. There is one artifact uploaded which is gitlab-build-docker.log. Every time I try to download the artifact directly from https://gitlab.com/sagemath/sage/-/jobs/197508827/artifacts/file/gitlab-build-docker.log I get a truncated file, which is very bizarre. Though if I download the zip archive and extract it it seems to be complete: The log again just ends at: Step 71/76 : COPY --chown=sage:sage --from=sagemath-dev-patch $SAGE_ROOT $SAGE_ROOT I don't know what's going on. Meanwhile the VM that ran this build is still hanging around, and my gitlab-runner master server isn't spawning a new one... > >> Why, I > >> don't know, but it started failing to create VM instances (the > >> infrastructure I'm running it on is not my own; it's hosted by the > >> university and has been disappointingly flaky in general). I just > >> gave it a kick and it's working again though. > >> > >> The last couple betas failed to build too, due to the lack of > >> available runners. The last successful build was of the 8.7 release > >> (so at least there's that): https://gitlab.com/sagemath/sage/pipelines > >> > >> So if anyone wants to contribute hardware to work as runners, I can > >> walk them through the setup process now that I've done it myself. > >> It's quite easy for the most part. > >> > >> We had some runners hosted on GCE as well that Dima had procured, and > >> I'm not sure why they don't seem to be working... > >> > >> Meanwhile over here you can see pipelines building for tickets on > >> Sage's Trac (including several of your own apparently): > >> https://gitlab.com/sagemath/dev/trac/pipelines > >> > >> Some of these are passing, but some are failing due to timeouts, > >> likely in part because the betas didn't build (and this is a problem > >> because there were some recent updates to OpenBLAS which make any > >> intermediate builds that don't start from one of the beta release > >> images take too long). > >> > >> So again, TL;DR the biggest hurdle to making progress with GitLab IMO > >> is just reliability of the CI infrastructure. > >> > >> Meanwhile, it's also still possible to submit change requests to Sage > >> in the form of Merge Requests: > >> https://gitlab.com/sagemath/sage/merge_requests > >> > >> It would be nice if more people would keep trying this out. > >> Experience so far is that it works quite well, and it would be nice to > >> advertise this as a way that people can submit changes to Sage without > >> worrying about having a Trac account, at least at first. One thing I > >> still want to do is figure out a scheme to map labels on GitLab to > >> Trac ticket fields (e.g. for setting priority and component). This > >> should be reasonably straightforward with some more tinkering on the > >> synchronization plugin, but I'd like to see more people trying it out > >> and providing concrete feedback before I spend much more time on it. > >> > >> That's all for now. And again anything you think you can do to help > >> just say so and I can point you in the right direction. > >> > >> On Tue, Apr 16, 2019 at 1:34 PM Daniel Krenn <[email protected]> wrote: > >> > > >> > Some time ago there was an attempt to use additionally Gitlab for our > >> > development process. What is the current status of this? > >> > > >> > (Back then, I (@dakrenn) asked to be included as user, so that I can > >> > contribute/test as well, as I frequently use Gitlab for many other > >> > projects.) > >> > > >> > Best, Daniel > >> > > >> > -- > >> > You received this message because you are subscribed to the Google > >> > Groups "sage-devel" group. > >> > To unsubscribe from this group and stop receiving emails from it, send > >> > an email to [email protected]. > >> > To post to this group, send email to [email protected]. > >> > Visit this group at https://groups.google.com/group/sage-devel. > >> > For more options, visit https://groups.google.com/d/optout. > >> > >> -- > >> You received this message because you are subscribed to the Google Groups > >> "sage-devel" group. > >> To unsubscribe from this group and stop receiving emails from it, send an > >> email to [email protected]. > >> To post to this group, send email to [email protected]. > >> Visit this group at https://groups.google.com/group/sage-devel. > >> For more options, visit https://groups.google.com/d/optout. > > > > -- > > You received this message because you are subscribed to the Google Groups > > "sage-devel" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [email protected]. > > To post to this group, send email to [email protected]. > > Visit this group at https://groups.google.com/group/sage-devel. > > For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout.
