Re: Renamed .gitlab-ci.yml to disable CI builds (at least for now)

[Wouter Verhelst]

Hi,

On Tue, Nov 27, 2018 at 11:00:34AM +, Damyan Ivanov wrote:
> -=| Wouter Verhelst, 27.11.2018 08:20:08 +0100 |=-
> > Hi Laura,
> > 
> > On Sun, Nov 25, 2018 at 09:27:19PM +0100, Laura Arjona Reina wrote:
> > > Hello all
> > > We've been told that the webwml CI builds in Salsa use too much
> > > resources, affecting Salsa's ability to serve other users.
> > 
> > That's not good, obviously.
> > 
> > How are we overdoing it? E.g., are we using too many runners, or are we
> > using too much space for artifacts?
> 
> Too many runners, I think. That's from #alioth:
> 
> webwml starts about 20 builds in parallel, 20 times docker pull of 
> a large image, 20 times git pull of the not too small repo, this 
> produces a lot of load
> 
> i asked them to disable the builds, as the result of those 
> builds are not used, not even to check for errors. and 
> breaking the concurrent build restriction to do nothing is 
> a bit bold
> 
> HTH to pinpoint the issue.

It does, thanks.

The reason we currently do it that way is that, while I know you can
tell buildbot to skip certain builds if they are not necessary based on
certain criteria, I couldn't find an obvious way to make "were files
under a certain directory touched in the most recent push" be part of
those criteria. I did add a small shell test to the start of every
translation that skips the build if the current translation did not see
any changes in the push for which we're building, but didn't consider
the fact that even the setup for running that really small script is
significant, too.

I'll go through the documentation and see if I can find a way to make
that work better, so that translation jobs aren't even started if they
don't need to do anything.

Regards,

-- 
To the thief who stole my anti-depressants: I hope you're happy

  -- seen somewhere on the Internet on a photo of a billboard

Re: Renamed .gitlab-ci.yml to disable CI builds (at least for now)

[Damyan Ivanov]

-=| Wouter Verhelst, 27.11.2018 08:20:08 +0100 |=-
> Hi Laura,
> 
> On Sun, Nov 25, 2018 at 09:27:19PM +0100, Laura Arjona Reina wrote:
> > Hello all
> > We've been told that the webwml CI builds in Salsa use too much
> > resources, affecting Salsa's ability to serve other users.
> 
> That's not good, obviously.
> 
> How are we overdoing it? E.g., are we using too many runners, or are we
> using too much space for artifacts?

Too many runners, I think. That's from #alioth:

webwml starts about 20 builds in parallel, 20 times docker pull of 
a large image, 20 times git pull of the not too small repo, this 
produces a lot of load

i asked them to disable the builds, as the result of those 
builds are not used, not even to check for errors. and 
breaking the concurrent build restriction to do nothing is 
a bit bold

HTH to pinpoint the issue.

-- dam

Re: Renamed .gitlab-ci.yml to disable CI builds (at least for now)

[Wouter Verhelst]

Hi Laura,

On Sun, Nov 25, 2018 at 09:27:19PM +0100, Laura Arjona Reina wrote:
> Hello all
> We've been told that the webwml CI builds in Salsa use too much
> resources, affecting Salsa's ability to serve other users.

That's not good, obviously.

How are we overdoing it? E.g., are we using too many runners, or are we
using too much space for artifacts?

> Currently the use of shared runners is disabled (see Settings > CI in
> Salsa webwml project), and I have renamed the .gitlab-ci.yml file to
> README_CI.gitlab-ci.yml and added an explanation inside the file (see
> the paragraph added below):
[...]
> We can also discuss in which cases it's reasonable to enable the CI
> setup for the team repo, and how (apart of renaming the file back and
> forth?).

Some things that spring to mind:

- We could install a project-specific runner on a machine that is owned
  by (a member of) the webmaster team. That way we don't overload the
  shared runners. This would not help much if the main problem is the
  overuse of artifacts, though.
- We could disable artifacts altogether, and use some custom deployment
  instead (e.g., on www-staging.debian.org). This would require that
  translations are able to be built when English is not built, however;
  I'm not sure if that's the case currently. It also ignores the shared
  runners issue (if that exists). Obviously the two methods could be
  combined if necessary.
- Please note that with gitlab CI, it's also possible to specify on
  which branches the runner should do nothing. That way, we can tell it
  to ignore the master branch, but build it everywhere else. It would
  then still be used for merge requests, without a need for renaming the
  file over and over again. I think this would be more useful as an
  intermediate solution than the current "disabled everywhere" thing.

Regards,

-- 
To the thief who stole my anti-depressants: I hope you're happy

  -- seen somewhere on the Internet on a photo of a billboard

Renamed .gitlab-ci.yml to disable CI builds (at least for now)

[Laura Arjona Reina]

Hello all
We've been told that the webwml CI builds in Salsa use too much
resources, affecting Salsa's ability to serve other users.

Currently the use of shared runners is disabled (see Settings > CI in
Salsa webwml project), and I have renamed the .gitlab-ci.yml file to
README_CI.gitlab-ci.yml and added an explanation inside the file (see
the paragraph added below):

.

--
+# Note about the name of this file: README_CI.gitlab-ci.yml
+#
+# Automatic builds of the Debian website via Gitlab CI were setup during
+# September-October 2018 (see <20180914134901.gc29...@grep.be>
+# and <20181011191247.ga2...@grep.be> (and the corresponding threads in
+# debian-www mailing list) for context.
+#
+# This file (when named .gitlab-ci.yml) builds the website both for English
+# and translations, and the result (the html files) can be browsed via
+# artifacts.
+#
+# On November 2018 Salsa (Gitlab) settings for webwml project were set
to keep
+# artifacts 1 week (default was 4 weeks) in order to save space.
+# Unfortunately the builds use too much resources, affecting salsa's
ability
+# to serve other users.
+#
+# So the settings were tweaked again to disable the use of shared runners,
+# and this file has been renamed from .gitlab-ci.yml to
README_CI.gitlab-ci.yml
+#
+# If you would like to enable Salsa's (GitLab's) CI builds for some test,
+# please contact the Debian web team, or you can rename this file
+# to its original name in your fork, and the CI build will run
(occasionally)
+# in your Salsa account.
+#
+

 stages:
 - english
 - translations
-

As I state in the file, I think the setup is still useful, for example
to test things in personal forks (e.g. prior to MR or commit to the team
repo). Thanks again Wouter for making it possible!

We can also discuss in which cases it's reasonable to enable the CI
setup for the team repo, and how (apart of renaming the file back and
forth?).

Kind regards,

Laura Arjona Reina
https://wiki.debian.org/LauraArjona