Does surefire have some options to truncate the test output if it is too large? Or jenkins has some options to truncate or compress a file when archiving?
Josh Elser <els...@apache.org> 于2019年6月11日周二 上午8:40写道： > Just a cursory glance at some build artifacts showed just test output > which sometimes extended into the multiple megabytes. > > So everyone else knows, I just chatted with ChrisL in Slack and he > confirmed that our disk utilization is down already (after HBASE-22563). > He thanked us for the quick response. > > We should keep pulling on this thread now that we're looking at it :) > > On 6/10/19 8:36 PM, 张铎(Duo Zhang) wrote: > > Oh, it is the build artifacts, not the jars... > > > > Most of our build artifacts are build logs, but maybe the problem is that > > some of the logs are very large if the test hangs... > > > > 张铎(Duo Zhang) <palomino...@gmail.com> 于2019年6月11日周二 上午8:16写道： > > > >> For flakey we just need the commit id in the console output then we can > >> build the artifacts locally. +1 on removing artifacts caching. > >> > >> Josh Elser <els...@apache.org> 于2019年6月11日周二 上午7:50写道： > >> > >>> Sure, Misty. No arguments here. > >>> > >>> I think that might be a bigger untangling. Maybe Peter or Busbey know > >>> better about how these could be de-coupled (e.g. I think flakies > >>> actually look back at old artifacts), but I'm not sure off the top of > my > >>> head. I was just going for a quick fix to keep Infra from doing > >>> something super-destructive. > >>> > >>> For context, I've dropped them a note in Slack to make sure what I'm > >>> doing is having a positive effect. > >>> > >>> On 6/10/19 7:34 PM, Misty Linville wrote: > >>>> Keeping artifacts and keeping build logs are two separate things. I > >>> don’t > >>>> see a need to keep any artifacts past the most recent green and most > >>> recent > >>>> red builds. Alternately if we need the artifacts let’s have Jenkins > put > >>>> them somewhere rather than keeping them there. You can get back to > >>> whatever > >>>> hash you need within git to reproduce a build problem. > >>>> > >>>> On Mon, Jun 10, 2019 at 2:26 PM Josh Elser <els...@apache.org> wrote: > >>>> > >>>>> https://issues.apache.org/jira/browse/HBASE-22563 for a quick > bandaid > >>> (I > >>>>> hope). > >>>>> > >>>>> On 6/10/19 4:31 PM, Josh Elser wrote: > >>>>>> Eyes on. > >>>>>> > >>>>>> Looking at master, we already have the linked configuration, set to > >>>>>> retain 30 builds. > >>>>>> > >>>>>> We have some extra branches which we can lop off (branch-1.2, > >>>>>> branch-2.0, maybe some feature branches too). A quick fix might be > to > >>>>>> just pull back that 30 to 10. > >>>>>> > >>>>>> Largely figuring out how this stuff works now, give me a shout in > >>> Slack > >>>>>> if anyone else has cycles. > >>>>>> > >>>>>> On 6/10/19 2:34 PM, Peter Somogyi wrote: > >>>>>>> Hi, > >>>>>>> > >>>>>>> HBase jobs are using more than 400GB based on this list. > >>>>>>> Could someone take a look at the job configurations today? > >>> Otherwise, I > >>>>>>> will look into it tomorrow morning. > >>>>>>> > >>>>>>> Thanks, > >>>>>>> Peter > >>>>>>> > >>>>>>> ---------- Forwarded message --------- > >>>>>>> From: Chris Lambertus <c...@apache.org> > >>>>>>> Date: Mon, Jun 10, 2019 at 7:57 PM > >>>>>>> Subject: ACTION REQUIRED: disk space on jenkins master nearly full > >>>>>>> To: <bui...@apache.org> > >>>>>>> Cc: <d...@mesos.apache.org>, <d...@pulsar.apache.org> > >>>>>>> > >>>>>>> > >>>>>>> Hello, > >>>>>>> > >>>>>>> The jenkins master is nearly full. > >>>>>>> > >>>>>>> The workspaces listed below need significant size reduction within > 24 > >>>>>>> hours > >>>>>>> or Infra will need to perform some manual pruning of old builds to > >>>>>>> keep the > >>>>>>> jenkins system running. The Mesos “Packaging” job also needs to be > >>>>>>> corrected to include the project name (mesos-packaging) please. > >>>>>>> > >>>>>>> It appears that the typical ‘Discard Old Builds’ checkbox in the > job > >>>>>>> configuration may not be working for multibranch pipeline jobs. > >>> Please > >>>>>>> refer to these articles for information on discarding builds in > >>>>>>> multibranch > >>>>>>> jobs: > >>>>>>> > >>>>>>> > >>>>> > >>> > https://support.cloudbees.com/hc/en-us/articles/115000237071-How-do-I-set-discard-old-builds-for-a-Multi-Branch-Pipeline-Job- > >>>>>>> > >>>>>>> https://issues.jenkins-ci.org/browse/JENKINS-35642 > >>>>>>> > >>>>> > >>> > https://issues.jenkins-ci.org/browse/JENKINS-34738?focusedCommentId=263489&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-263489 > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> NB: I have not fully vetted the above information, I just notice > that > >>>>>>> many > >>>>>>> of these jobs have ‘Discard old builds’ checked, but it is clearly > >>> not > >>>>>>> working. > >>>>>>> > >>>>>>> > >>>>>>> If you are unable to reduce your disk usage beyond what is listed, > >>>>> please > >>>>>>> let me know what the reasons are and we’ll see if we can find a > >>>>> solution. > >>>>>>> If you believe you’ve configured your job properly and the space > >>> usage > >>>>> is > >>>>>>> more than you expect, please comment here and we’ll take a look at > >>> what > >>>>>>> might be going on. > >>>>>>> > >>>>>>> I cut this list off arbitrarily at 40GB workspaces and larger. > There > >>> are > >>>>>>> many which are between 20 and 30GB which also need to be addressed, > >>> but > >>>>>>> these are the current top contributors to the disk space situation. > >>>>>>> > >>>>>>> > >>>>>>> 594G Packaging > >>>>>>> 425G pulsar-website-build > >>>>>>> 274G pulsar-master > >>>>>>> 195G hadoop-multibranch > >>>>>>> 173G HBase Nightly > >>>>>>> 138G HBase-Flaky-Tests > >>>>>>> 119G netbeans-release > >>>>>>> 108G Any23-trunk > >>>>>>> 101G netbeans-linux-experiment > >>>>>>> 96G Jackrabbit-Oak-Windows > >>>>>>> 94G HBase-Find-Flaky-Tests > >>>>>>> 88G PreCommit-ZOOKEEPER-github-pr-build > >>>>>>> 74G netbeans-windows > >>>>>>> 71G stanbol-0.12 > >>>>>>> 68G Sling > >>>>>>> 63G Atlas-master-NoTests > >>>>>>> 48G FlexJS Framework (maven) > >>>>>>> 45G HBase-PreCommit-GitHub-PR > >>>>>>> 42G pulsar-pull-request > >>>>>>> 40G Atlas-1.0-NoTests > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> Thanks, > >>>>>>> Chris > >>>>>>> ASF Infra > >>>>>>> > >>>>> > >>>> > >>> > >> > > >
