Just a cursory glance at some build artifacts showed just test output
which sometimes extended into the multiple megabytes.
So everyone else knows, I just chatted with ChrisL in Slack and he
confirmed that our disk utilization is down already (after HBASE-22563).
He thanked us for the quick response.
We should keep pulling on this thread now that we're looking at it :)
On 6/10/19 8:36 PM, 张铎(Duo Zhang) wrote:
Oh, it is the build artifacts, not the jars...
Most of our build artifacts are build logs, but maybe the problem is that
some of the logs are very large if the test hangs...
张铎(Duo Zhang) <palomino...@gmail.com> 于2019年6月11日周二 上午8:16写道：
For flakey we just need the commit id in the console output then we can
build the artifacts locally. +1 on removing artifacts caching.
Josh Elser <els...@apache.org> 于2019年6月11日周二 上午7:50写道：
Sure, Misty. No arguments here.
I think that might be a bigger untangling. Maybe Peter or Busbey know
better about how these could be de-coupled (e.g. I think flakies
actually look back at old artifacts), but I'm not sure off the top of my
head. I was just going for a quick fix to keep Infra from doing
something super-destructive.
For context, I've dropped them a note in Slack to make sure what I'm
doing is having a positive effect.
On 6/10/19 7:34 PM, Misty Linville wrote:
Keeping artifacts and keeping build logs are two separate things. I
don’t
see a need to keep any artifacts past the most recent green and most
recent
red builds. Alternately if we need the artifacts let’s have Jenkins put
them somewhere rather than keeping them there. You can get back to
whatever
hash you need within git to reproduce a build problem.
On Mon, Jun 10, 2019 at 2:26 PM Josh Elser <els...@apache.org> wrote:
https://issues.apache.org/jira/browse/HBASE-22563 for a quick bandaid
(I
hope).
On 6/10/19 4:31 PM, Josh Elser wrote:
Eyes on.
Looking at master, we already have the linked configuration, set to
retain 30 builds.
We have some extra branches which we can lop off (branch-1.2,
branch-2.0, maybe some feature branches too). A quick fix might be to
just pull back that 30 to 10.
Largely figuring out how this stuff works now, give me a shout in
Slack
if anyone else has cycles.
On 6/10/19 2:34 PM, Peter Somogyi wrote:
Hi,
HBase jobs are using more than 400GB based on this list.
Could someone take a look at the job configurations today?
Otherwise, I
will look into it tomorrow morning.
Thanks,
Peter
---------- Forwarded message ---------
From: Chris Lambertus <c...@apache.org>
Date: Mon, Jun 10, 2019 at 7:57 PM
Subject: ACTION REQUIRED: disk space on jenkins master nearly full
To: <bui...@apache.org>
Cc: <d...@mesos.apache.org>, <d...@pulsar.apache.org>
Hello,
The jenkins master is nearly full.
The workspaces listed below need significant size reduction within 24
hours
or Infra will need to perform some manual pruning of old builds to
keep the
jenkins system running. The Mesos “Packaging” job also needs to be
corrected to include the project name (mesos-packaging) please.
It appears that the typical ‘Discard Old Builds’ checkbox in the job
configuration may not be working for multibranch pipeline jobs.
Please
refer to these articles for information on discarding builds in
multibranch
jobs:
https://support.cloudbees.com/hc/en-us/articles/115000237071-How-do-I-set-discard-old-builds-for-a-Multi-Branch-Pipeline-Job-
https://issues.jenkins-ci.org/browse/JENKINS-35642
https://issues.jenkins-ci.org/browse/JENKINS-34738?focusedCommentId=263489&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-263489
NB: I have not fully vetted the above information, I just notice that
many
of these jobs have ‘Discard old builds’ checked, but it is clearly
not
working.
If you are unable to reduce your disk usage beyond what is listed,
please
let me know what the reasons are and we’ll see if we can find a
solution.
If you believe you’ve configured your job properly and the space
usage
is
more than you expect, please comment here and we’ll take a look at
what
might be going on.
I cut this list off arbitrarily at 40GB workspaces and larger. There
are
many which are between 20 and 30GB which also need to be addressed,
but
these are the current top contributors to the disk space situation.
594G Packaging
425G pulsar-website-build
274G pulsar-master
195G hadoop-multibranch
173G HBase Nightly
138G HBase-Flaky-Tests
119G netbeans-release
108G Any23-trunk
101G netbeans-linux-experiment
96G Jackrabbit-Oak-Windows
94G HBase-Find-Flaky-Tests
88G PreCommit-ZOOKEEPER-github-pr-build
74G netbeans-windows
71G stanbol-0.12
68G Sling
63G Atlas-master-NoTests
48G FlexJS Framework (maven)
45G HBase-PreCommit-GitHub-PR
42G pulsar-pull-request
40G Atlas-1.0-NoTests
Thanks,
Chris
ASF Infra
