Hello Chris,
could you please take a look at
https://builds.apache.org/view/M-R/view/OpenMeetings/job/openmeetings
It has rule to keep 3 last builds, but somehow it keeps 5
Manual delete fails with stacktrace:
ava.io.IOException: openmeetings #2860:
/x1/jenkins/jenkins-home/jobs/openmeetings/builds/2860 looks to have
already been deleted; siblings: [2864, 2869, lastSuccessfulBuild,
lastStableBuild, 1696, 1699, 2867, lastUnstableBuild, 2868, 1700,
legacyIds, lastFailedBuild, 1698, lastUnsuccessfulBuild, 2866]
at hudson.model.Run.delete(Run.java:1564)
at hudson.maven.MavenModuleSetBuild.delete(MavenModuleSetBuild.java:450)
at hudson.model.Run.doDoDelete(Run.java:2298)
at
java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:627)
at org.kohsuke.stapler.Function$MethodFunction.invoke(Function.java:396)
On Sat, 15 Jun 2019 at 09:19, Chris Lambertus <c...@apache.org> wrote:
> Hmm. Upon further investigation, I do see that the
> pulsar-master/modules/*/builds directories contain current builds between
> 13 and 15 June, and then hundreds of builds from 2018 and 2017. It looks
> like these are indeed “orphaned” builds, and your ‘discard old builds’
> configuration is working properly.
>
> I have manually removed all builds older than 14 days from the
> pulsar-master/modules directory, and your usage is looking good now:
>
> root@jenkins02:/x1/jenkins/jenkins-home/jobs# du -sh pulsar-master
> 23G pulsar-master
>
> I really appreciate your attention to this. I’ll check back again in a few
> weeks time to make sure that the builds are getting pruned as intended.
>
>
> I will also look for this pathology in other build directories too, in
> case the problem is with jenkins rather than the build config.
>
> -Chris
>
>
>
> > On Jun 14, 2019, at 6:15 PM, Matteo Merli <matteo.me...@gmail.com>
> wrote:
> >
> > Hi Chris,
> > sorry, I lost the updates on this thread.
> >
> > After applying the "discard old builds" check, I saw all the old stuff
> > going away. Even now I don't see any of the old builds, from the
> > Jenkins UI.
> > https://builds.apache.org/job/pulsar-master/
> >
> > Is it possible that maybe Jenkins failed to cleanup these, for some
> > reason? In any case, please go ahead and remove those directories.
> >
> > Matteo
> > --
> > Matteo Merli
> > <matteo.me...@gmail.com>
> >
> > On Mon, Jun 10, 2019 at 2:29 PM Chris Lambertus <c...@apache.org> wrote:
> >>
> >> Matteo,
> >>
> >> pulsar-website cleaned up nicely. pulsar-master is still problematic -
> despite having run a few minutes ago, there are still builds dating back to
> 2017 in the pulsar-master/modules/org.apache.pulsar* directories, so it
> also appears that maven module ‘discard old builds’ is not working either.
> I have not yet found any suggested solutions to this.
> >>
> >> -Chris
> >>
> >>
> >>
> >>> On Jun 10, 2019, at 11:14 AM, Chris Lambertus <c...@apache.org> wrote:
> >>>
> >>> Outstanding, thanks. I believe the job cleanup runs when the next
> build runs. You could manually trigger a build to test, or we can check
> next time the build runs automatically (presuming it runs nighty.)
> >>>
> >>> -Chris
> >>>
> >>>
> >>>> On Jun 10, 2019, at 11:10 AM, Matteo Merli <mme...@apache.org> wrote:
> >>>>
> >>>> For pulsar-website-build and pulsar-master, the "discard old builds"
> >>>> wasn't set unfortunately. I just enabled it now. Not sure if there's a
> >>>> way to quickly trigger a manual cleanup.
> >>>>
> >>>> Regarding "pulsar-pull-request": this was an old Jenkins job no longer
> >>>> used (since we switched to multiple smaller PR validation jobs a while
> >>>> ago). I have removed the Jenkins job. Hopefully that should take care
> >>>> of cleaning all the files.
> >>>>
> >>>>
> >>>> Thanks,
> >>>> Matteo
> >>>>
> >>>> --
> >>>> Matteo Merli
> >>>> <mme...@apache.org>
> >>>>
> >>>> On Mon, Jun 10, 2019 at 10:57 AM Chris Lambertus <c...@apache.org>
> wrote:
> >>>>>
> >>>>> Hello,
> >>>>>
> >>>>> The jenkins master is nearly full.
> >>>>>
> >>>>> The workspaces listed below need significant size reduction within
> 24 hours or Infra will need to perform some manual pruning of old builds to
> keep the jenkins system running. The Mesos “Packaging” job also needs to be
> corrected to include the project name (mesos-packaging) please.
> >>>>>
> >>>>> It appears that the typical ‘Discard Old Builds’ checkbox in the job
> configuration may not be working for multibranch pipeline jobs. Please
> refer to these articles for information on discarding builds in multibranch
> jobs:
> >>>>>
> >>>>>
> https://support.cloudbees.com/hc/en-us/articles/115000237071-How-do-I-set-discard-old-builds-for-a-Multi-Branch-Pipeline-Job-
> >>>>> https://issues.jenkins-ci.org/browse/JENKINS-35642
> >>>>>
> https://issues.jenkins-ci.org/browse/JENKINS-34738?focusedCommentId=263489&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-263489
> >>>>>
> >>>>>
> >>>>>
> >>>>> NB: I have not fully vetted the above information, I just notice
> that many of these jobs have ‘Discard old builds’ checked, but it is
> clearly not working.
> >>>>>
> >>>>>
> >>>>> If you are unable to reduce your disk usage beyond what is listed,
> please let me know what the reasons are and we’ll see if we can find a
> solution. If you believe you’ve configured your job properly and the space
> usage is more than you expect, please comment here and we’ll take a look at
> what might be going on.
> >>>>>
> >>>>> I cut this list off arbitrarily at 40GB workspaces and larger. There
> are many which are between 20 and 30GB which also need to be addressed, but
> these are the current top contributors to the disk space situation.
> >>>>>
> >>>>>
> >>>>> 594G Packaging
> >>>>> 425G pulsar-website-build
> >>>>> 274G pulsar-master
> >>>>> 195G hadoop-multibranch
> >>>>> 173G HBase Nightly
> >>>>> 138G HBase-Flaky-Tests
> >>>>> 119G netbeans-release
> >>>>> 108G Any23-trunk
> >>>>> 101G netbeans-linux-experiment
> >>>>> 96G Jackrabbit-Oak-Windows
> >>>>> 94G HBase-Find-Flaky-Tests
> >>>>> 88G PreCommit-ZOOKEEPER-github-pr-build
> >>>>> 74G netbeans-windows
> >>>>> 71G stanbol-0.12
> >>>>> 68G Sling
> >>>>> 63G Atlas-master-NoTests
> >>>>> 48G FlexJS Framework (maven)
> >>>>> 45G HBase-PreCommit-GitHub-PR
> >>>>> 42G pulsar-pull-request
> >>>>> 40G Atlas-1.0-NoTests
> >>>>>
> >>>>>
> >>>>>
> >>>>> Thanks,
> >>>>> Chris
> >>>>> ASF Infra
> >>>
> >>
>
>
--
WBR
Maxim aka solomax
