The additional slots (@3 directories) take up even more space now than before.
I'm testing out https://github.com/apache/beam/pull/12326 which could help by cleaning up workspaces after a run (just started a seed job). On Fri, Jul 24, 2020 at 12:13 PM Tyson Hamilton <tyso...@google.com> wrote: > 664M beam_PreCommit_JavaPortabilityApi_Commit > 656M beam_PreCommit_JavaPortabilityApi_Commit@2 > 611M beam_PreCommit_JavaPortabilityApi_Cron > 616M beam_PreCommit_JavaPortabilityApiJava11_Commit > 598M beam_PreCommit_JavaPortabilityApiJava11_Commit@2 > 662M beam_PreCommit_JavaPortabilityApiJava11_Cron > 2.9G beam_PreCommit_Portable_Python_Commit > 2.9G beam_PreCommit_Portable_Python_Commit@2 > 1.7G beam_PreCommit_Portable_Python_Commit@3 > 3.4G beam_PreCommit_Portable_Python_Cron > 1.9G beam_PreCommit_Python2_PVR_Flink_Commit > 1.4G beam_PreCommit_Python2_PVR_Flink_Cron > 1.3G beam_PreCommit_Python2_PVR_Flink_Phrase > 6.2G beam_PreCommit_Python_Commit > 7.5G beam_PreCommit_Python_Commit@2 > 7.5G beam_PreCommit_Python_Cron > 1012M beam_PreCommit_PythonDocker_Commit > 1011M beam_PreCommit_PythonDocker_Commit@2 > 1011M beam_PreCommit_PythonDocker_Commit@3 > 1002M beam_PreCommit_PythonDocker_Cron > 877M beam_PreCommit_PythonFormatter_Commit > 988M beam_PreCommit_PythonFormatter_Cron > 986M beam_PreCommit_PythonFormatter_Phrase > 1.7G beam_PreCommit_PythonLint_Commit > 2.1G beam_PreCommit_PythonLint_Cron > 7.5G beam_PreCommit_Python_Phrase > 346M beam_PreCommit_RAT_Commit > 341M beam_PreCommit_RAT_Cron > 338M beam_PreCommit_Spotless_Commit > 339M beam_PreCommit_Spotless_Cron > 5.5G beam_PreCommit_SQL_Commit > 5.5G beam_PreCommit_SQL_Cron > 5.5G beam_PreCommit_SQL_Java11_Commit > 750M beam_PreCommit_Website_Commit > 750M beam_PreCommit_Website_Commit@2 > 750M beam_PreCommit_Website_Cron > 764M beam_PreCommit_Website_Stage_GCS_Commit > 771M beam_PreCommit_Website_Stage_GCS_Cron > 336M beam_Prober_CommunityMetrics > 693M beam_python_mongoio_load_test > 339M beam_SeedJob > 333M beam_SeedJob_Standalone > 334M beam_sonarqube_report > 556M beam_SQLBigQueryIO_Batch_Performance_Test_Java > 175G total > > On Fri, Jul 24, 2020 at 12:04 PM Tyson Hamilton <tyso...@google.com> > wrote: > >> Ya looks like something in the workspaces is taking up room: >> >> @apache-ci-beam-jenkins-8:/home/jenkins$ sudo du -shc . >> 191G . >> 191G total >> >> >> On Fri, Jul 24, 2020 at 11:44 AM Tyson Hamilton <tyso...@google.com> >> wrote: >> >>> Node 8 is also full. The partition that /tmp is on is here: >>> >>> Filesystem Size Used Avail Use% Mounted on >>> /dev/sda1 485G 482G 2.9G 100% / >>> >>> however after cleaning up tmp with the crontab command, there is only 8G >>> usage yet it still remains 100% full: >>> >>> @apache-ci-beam-jenkins-8:/tmp$ sudo du -shc /tmp >>> 8.0G /tmp >>> 8.0G total >>> >>> The workspaces are in the /home/jenkins/jenkins-slave/workspace >>> directory. When I run a du on that, it takes really long. I'll let it keep >>> running for a while to see if it ever returns a result but so far this >>> seems suspect. >>> >>> >>> >>> >>> >>> On Fri, Jul 24, 2020 at 11:19 AM Tyson Hamilton <tyso...@google.com> >>> wrote: >>> >>>> Everything I've been looking at is in the /tmp dir. Where are the >>>> workspaces, or what are the named? >>>> >>>> >>>> >>>> >>>> On Fri, Jul 24, 2020 at 11:03 AM Udi Meiri <eh...@google.com> wrote: >>>> >>>>> I'm curious to what you find. Was it /tmp or the workspaces using up >>>>> the space? >>>>> >>>>> On Fri, Jul 24, 2020 at 10:57 AM Tyson Hamilton <tyso...@google.com> >>>>> wrote: >>>>> >>>>>> Bleck. I just realized that it is 'offline' so that won't work. I'll >>>>>> clean up manually on the machine using the cron command. >>>>>> >>>>>> On Fri, Jul 24, 2020 at 10:56 AM Tyson Hamilton <tyso...@google.com> >>>>>> wrote: >>>>>> >>>>>>> Something isn't working with the current set up because node 15 >>>>>>> appears to be out of space and is currently 'offline' according to >>>>>>> Jenkins. >>>>>>> Can someone run the cleanup job? The machine is full, >>>>>>> >>>>>>> @apache-ci-beam-jenkins-15:/tmp$ df -h >>>>>>> Filesystem Size Used Avail Use% Mounted on >>>>>>> udev 52G 0 52G 0% /dev >>>>>>> tmpfs 11G 265M 10G 3% /run >>>>>>> */dev/sda1 485G 484G 880M 100% /* >>>>>>> tmpfs 52G 0 52G 0% /dev/shm >>>>>>> tmpfs 5.0M 0 5.0M 0% /run/lock >>>>>>> tmpfs 52G 0 52G 0% /sys/fs/cgroup >>>>>>> tmpfs 11G 0 11G 0% /run/user/1017 >>>>>>> tmpfs 11G 0 11G 0% /run/user/1037 >>>>>>> >>>>>>> apache-ci-beam-jenkins-15:/tmp$ sudo du -ah --time . | sort -rhk 1,1 >>>>>>> | head -n 20 >>>>>>> 20G 2020-07-24 17:52 . >>>>>>> 580M 2020-07-22 17:31 ./junit1031982597110125586 >>>>>>> 517M 2020-07-22 17:31 >>>>>>> ./junit1031982597110125586/junit8739924829337821410/heap_dump.hprof >>>>>>> 517M 2020-07-22 17:31 >>>>>>> ./junit1031982597110125586/junit8739924829337821410 >>>>>>> 263M 2020-07-22 12:23 ./pip-install-2GUhO_ >>>>>>> 263M 2020-07-20 09:30 ./pip-install-sxgwqr >>>>>>> 263M 2020-07-17 13:56 ./pip-install-bWSKIV >>>>>>> 242M 2020-07-21 20:25 ./beam-pipeline-tempmByU6T >>>>>>> 242M 2020-07-21 20:21 ./beam-pipeline-tempV85xeK >>>>>>> 242M 2020-07-21 20:15 ./beam-pipeline-temp7dJROJ >>>>>>> 236M 2020-07-21 20:25 ./beam-pipeline-tempmByU6T/tmpOWj3Yr >>>>>>> 236M 2020-07-21 20:21 ./beam-pipeline-tempV85xeK/tmppbQHB3 >>>>>>> 236M 2020-07-21 20:15 ./beam-pipeline-temp7dJROJ/tmpgOXPKW >>>>>>> 111M 2020-07-23 00:57 ./pip-install-1JnyNE >>>>>>> 105M 2020-07-23 00:17 ./beam-artifact1374651823280819755 >>>>>>> 105M 2020-07-23 00:16 ./beam-artifact5050755582921936972 >>>>>>> 105M 2020-07-23 00:16 ./beam-artifact1834064452502646289 >>>>>>> 105M 2020-07-23 00:15 ./beam-artifact682561790267074916 >>>>>>> 105M 2020-07-23 00:15 ./beam-artifact4691304965824489394 >>>>>>> 105M 2020-07-23 00:14 ./beam-artifact4050383819822604421 >>>>>>> >>>>>>> On Wed, Jul 22, 2020 at 12:03 PM Robert Bradshaw < >>>>>>> rober...@google.com> wrote: >>>>>>> >>>>>>>> On Wed, Jul 22, 2020 at 11:57 AM Tyson Hamilton <tyso...@google.com> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Ah I see, thanks Kenn. I found some advice from the Apache infra >>>>>>>>> wiki that also suggests using a tmpdir inside the workspace [1]: >>>>>>>>> >>>>>>>>> Procedures Projects can take to clean up disk space >>>>>>>>> >>>>>>>>> Projects can help themselves and Infra by taking some basic steps >>>>>>>>> to help clean up their jobs after themselves on the build nodes. >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> 1. Use a ./tmp dir in your jobs workspace. That way it gets >>>>>>>>> cleaned up when job workspaces expire. >>>>>>>>> >>>>>>>>> >>>>>>>> Tests should be (able to be) written to use the standard temporary >>>>>>>> file mechanisms, and the environment set up on Jenkins such that that >>>>>>>> falls >>>>>>>> into the respective workspaces. Ideally this should be as simple as >>>>>>>> setting >>>>>>>> the TMPDIR (or similar) environment variable (and making sure it >>>>>>>> exists/is >>>>>>>> writable). >>>>>>>> >>>>>>>>> >>>>>>>>> 1. Configure your jobs to wipe workspaces on start or finish. >>>>>>>>> 2. Configure your jobs to only keep 5 or 10 previous builds. >>>>>>>>> 3. Configure your jobs to only keep 5 or 10 previous artifacts. >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> [1]: >>>>>>>>> https://cwiki.apache.org/confluence/display/INFRA/Disk+Space+cleanup+of+Jenkins+nodes >>>>>>>>> >>>>>>>>> On Wed, Jul 22, 2020 at 8:06 AM Kenneth Knowles <k...@apache.org> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Those file listings look like the result of using standard temp >>>>>>>>>> file APIs but with TMPDIR set to /tmp. >>>>>>>>>> >>>>>>>>>> On Mon, Jul 20, 2020 at 7:55 PM Tyson Hamilton < >>>>>>>>>> tyso...@google.com> wrote: >>>>>>>>>> >>>>>>>>>>> Jobs are hermetic as far as I can tell and use unique >>>>>>>>>>> subdirectories inside of /tmp. Here is a quick look into two >>>>>>>>>>> examples: >>>>>>>>>>> >>>>>>>>>>> @apache-ci-beam-jenkins-4:/tmp$ sudo du -ah --time . | sort -rhk >>>>>>>>>>> 1,1 | head -n 20 >>>>>>>>>>> 1.6G 2020-07-21 02:25 . >>>>>>>>>>> 242M 2020-07-17 18:48 ./beam-pipeline-temp3ybuY4 >>>>>>>>>>> 242M 2020-07-17 18:46 ./beam-pipeline-tempuxjiPT >>>>>>>>>>> 242M 2020-07-17 18:44 ./beam-pipeline-tempVpg1ME >>>>>>>>>>> 242M 2020-07-17 18:42 ./beam-pipeline-tempJ4EpyB >>>>>>>>>>> 242M 2020-07-17 18:39 ./beam-pipeline-tempepea7Q >>>>>>>>>>> 242M 2020-07-17 18:35 ./beam-pipeline-temp79qot2 >>>>>>>>>>> 236M 2020-07-17 18:48 >>>>>>>>>>> ./beam-pipeline-temp3ybuY4/tmpy_Ytzz >>>>>>>>>>> 236M 2020-07-17 18:46 >>>>>>>>>>> ./beam-pipeline-tempuxjiPT/tmpN5_UfJ >>>>>>>>>>> 236M 2020-07-17 18:44 >>>>>>>>>>> ./beam-pipeline-tempVpg1ME/tmpxSm8pX >>>>>>>>>>> 236M 2020-07-17 18:42 >>>>>>>>>>> ./beam-pipeline-tempJ4EpyB/tmpMZJU76 >>>>>>>>>>> 236M 2020-07-17 18:39 >>>>>>>>>>> ./beam-pipeline-tempepea7Q/tmpWy1vWX >>>>>>>>>>> 236M 2020-07-17 18:35 >>>>>>>>>>> ./beam-pipeline-temp79qot2/tmpvN7vWA >>>>>>>>>>> 3.7M 2020-07-17 18:48 >>>>>>>>>>> ./beam-pipeline-temp3ybuY4/tmprlh_di >>>>>>>>>>> 3.7M 2020-07-17 18:46 >>>>>>>>>>> ./beam-pipeline-tempuxjiPT/tmpLmVWfe >>>>>>>>>>> 3.7M 2020-07-17 18:44 >>>>>>>>>>> ./beam-pipeline-tempVpg1ME/tmpvrxbY7 >>>>>>>>>>> 3.7M 2020-07-17 18:42 >>>>>>>>>>> ./beam-pipeline-tempJ4EpyB/tmpLTb6Mj >>>>>>>>>>> 3.7M 2020-07-17 18:39 >>>>>>>>>>> ./beam-pipeline-tempepea7Q/tmptYF1v1 >>>>>>>>>>> 3.7M 2020-07-17 18:35 >>>>>>>>>>> ./beam-pipeline-temp79qot2/tmplfV0Rg >>>>>>>>>>> 2.7M 2020-07-17 20:10 ./pip-install-q9l227ef >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> @apache-ci-beam-jenkins-11:/tmp$ sudo du -ah --time . | sort >>>>>>>>>>> -rhk 1,1 | head -n 20 >>>>>>>>>>> 817M 2020-07-21 02:26 . >>>>>>>>>>> 242M 2020-07-19 12:14 ./beam-pipeline-tempUTXqlM >>>>>>>>>>> 242M 2020-07-19 12:11 ./beam-pipeline-tempx3Yno3 >>>>>>>>>>> 242M 2020-07-19 12:05 ./beam-pipeline-tempyCrMYq >>>>>>>>>>> 236M 2020-07-19 12:14 >>>>>>>>>>> ./beam-pipeline-tempUTXqlM/tmpstXoL0 >>>>>>>>>>> 236M 2020-07-19 12:11 >>>>>>>>>>> ./beam-pipeline-tempx3Yno3/tmpnnVn65 >>>>>>>>>>> 236M 2020-07-19 12:05 >>>>>>>>>>> ./beam-pipeline-tempyCrMYq/tmpRF0iNs >>>>>>>>>>> 3.7M 2020-07-19 12:14 >>>>>>>>>>> ./beam-pipeline-tempUTXqlM/tmpbJjUAQ >>>>>>>>>>> 3.7M 2020-07-19 12:11 >>>>>>>>>>> ./beam-pipeline-tempx3Yno3/tmpsmmzqe >>>>>>>>>>> 3.7M 2020-07-19 12:05 >>>>>>>>>>> ./beam-pipeline-tempyCrMYq/tmp5b3ZvY >>>>>>>>>>> 2.0M 2020-07-19 12:14 >>>>>>>>>>> ./beam-pipeline-tempUTXqlM/tmpoj3orz >>>>>>>>>>> 2.0M 2020-07-19 12:11 >>>>>>>>>>> ./beam-pipeline-tempx3Yno3/tmptng9sZ >>>>>>>>>>> 2.0M 2020-07-19 12:05 >>>>>>>>>>> ./beam-pipeline-tempyCrMYq/tmpWp6njc >>>>>>>>>>> 1.2M 2020-07-19 12:14 >>>>>>>>>>> ./beam-pipeline-tempUTXqlM/tmphgdj35 >>>>>>>>>>> 1.2M 2020-07-19 12:11 >>>>>>>>>>> ./beam-pipeline-tempx3Yno3/tmp8ySXpm >>>>>>>>>>> 1.2M 2020-07-19 12:05 >>>>>>>>>>> ./beam-pipeline-tempyCrMYq/tmpNVEJ4e >>>>>>>>>>> 992K 2020-07-12 12:00 ./junit642086915811430564 >>>>>>>>>>> 988K 2020-07-12 12:00 ./junit642086915811430564/beam >>>>>>>>>>> 984K 2020-07-12 12:00 >>>>>>>>>>> ./junit642086915811430564/beam/nodes >>>>>>>>>>> 980K 2020-07-12 12:00 >>>>>>>>>>> ./junit642086915811430564/beam/nodes/0 >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Mon, Jul 20, 2020 at 6:46 PM Udi Meiri <eh...@google.com> >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>>> You're right, job workspaces should be hermetic. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Mon, Jul 20, 2020 at 1:24 PM Kenneth Knowles < >>>>>>>>>>>> k...@apache.org> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> I'm probably late to this discussion and missing something, >>>>>>>>>>>>> but why are we writing to /tmp at all? I would expect TMPDIR to >>>>>>>>>>>>> point >>>>>>>>>>>>> somewhere inside the job directory that will be wiped by Jenkins, >>>>>>>>>>>>> and I >>>>>>>>>>>>> would expect code to always create temp files via APIs that >>>>>>>>>>>>> respect this. >>>>>>>>>>>>> Is Jenkins not cleaning up? Do we not have the ability to set >>>>>>>>>>>>> this up? Do >>>>>>>>>>>>> we have bugs in our code (that we could probably find by setting >>>>>>>>>>>>> TMPDIR to >>>>>>>>>>>>> somewhere not-/tmp and running the tests without write permission >>>>>>>>>>>>> to /tmp, >>>>>>>>>>>>> etc) >>>>>>>>>>>>> >>>>>>>>>>>>> Kenn >>>>>>>>>>>>> >>>>>>>>>>>>> On Mon, Jul 20, 2020 at 11:39 AM Ahmet Altay <al...@google.com> >>>>>>>>>>>>> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Related to workspace directory growth, +Udi Meiri >>>>>>>>>>>>>> <eh...@google.com> filed a relevant issue previously ( >>>>>>>>>>>>>> https://issues.apache.org/jira/browse/BEAM-9865) for >>>>>>>>>>>>>> cleaning up workspace directory after successful jobs. >>>>>>>>>>>>>> Alternatively, we >>>>>>>>>>>>>> can consider periodically cleaning up the /src directories. >>>>>>>>>>>>>> >>>>>>>>>>>>>> I would suggest moving the cron task from internal cron >>>>>>>>>>>>>> scripts to the inventory job ( >>>>>>>>>>>>>> https://github.com/apache/beam/blob/master/.test-infra/jenkins/job_Inventory.groovy#L51). >>>>>>>>>>>>>> That way, we can see all the cron jobs as part of the source >>>>>>>>>>>>>> tree, adjust >>>>>>>>>>>>>> frequencies and clean up codes with PRs. I do not know how >>>>>>>>>>>>>> internal cron >>>>>>>>>>>>>> scripts are created, maintained, and how would they be recreated >>>>>>>>>>>>>> for new >>>>>>>>>>>>>> worker instances. >>>>>>>>>>>>>> >>>>>>>>>>>>>> /cc +Tyson Hamilton <tyso...@google.com> >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Mon, Jul 20, 2020 at 4:50 AM Damian Gadomski < >>>>>>>>>>>>>> damian.gadom...@polidea.com> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hey, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I've recently created a solution for the growing /tmp >>>>>>>>>>>>>>> directory. Part of it is the job mentioned by Tyson: >>>>>>>>>>>>>>> *beam_Clean_tmp_directory*. It's intentionally not >>>>>>>>>>>>>>> triggered by cron and should be a last resort solution for some >>>>>>>>>>>>>>> strange >>>>>>>>>>>>>>> cases. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Along with that job, I've also updated every worker with an >>>>>>>>>>>>>>> internal cron script. It's being executed once a week and >>>>>>>>>>>>>>> deletes all the >>>>>>>>>>>>>>> files (and only files) that were not accessed for at least >>>>>>>>>>>>>>> three days. >>>>>>>>>>>>>>> That's designed to be as safe as possible for the running jobs >>>>>>>>>>>>>>> on the >>>>>>>>>>>>>>> worker (not to delete the files that are still in use), and >>>>>>>>>>>>>>> also to be >>>>>>>>>>>>>>> insensitive to the current workload on the machine. The cleanup >>>>>>>>>>>>>>> will always >>>>>>>>>>>>>>> happen, even if some long-running/stuck jobs are blocking the >>>>>>>>>>>>>>> machine. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I also think that currently the "No space left" errors may >>>>>>>>>>>>>>> be a consequence of growing workspace directory rather than >>>>>>>>>>>>>>> /tmp. I didn't >>>>>>>>>>>>>>> do any detailed analysis but e.g. currently, on >>>>>>>>>>>>>>> apache-beam-jenkins-7 the >>>>>>>>>>>>>>> workspace directory size is 158 GB while /tmp is only 16 GB. We >>>>>>>>>>>>>>> should >>>>>>>>>>>>>>> either guarantee the disk size to hold workspaces for all jobs >>>>>>>>>>>>>>> (because >>>>>>>>>>>>>>> eventually, every worker will execute each job) or clear also >>>>>>>>>>>>>>> the >>>>>>>>>>>>>>> workspaces in some way. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Regards, >>>>>>>>>>>>>>> Damian >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Mon, Jul 20, 2020 at 10:43 AM Maximilian Michels < >>>>>>>>>>>>>>> m...@apache.org> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> +1 for scheduling it via a cron job if it won't lead to >>>>>>>>>>>>>>>> test failures >>>>>>>>>>>>>>>> while running. Not a Jenkins expert but maybe there is the >>>>>>>>>>>>>>>> notion of >>>>>>>>>>>>>>>> running exclusively while no other tasks are running? >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> -Max >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On 17.07.20 21:49, Tyson Hamilton wrote: >>>>>>>>>>>>>>>> > FYI there was a job introduced to do this in Jenkins: >>>>>>>>>>>>>>>> beam_Clean_tmp_directory >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > Currently it needs to be run manually. I'm seeing some >>>>>>>>>>>>>>>> out of disk related errors in precommit tests currently, >>>>>>>>>>>>>>>> perhaps we should >>>>>>>>>>>>>>>> schedule this job with cron? >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > On 2020/03/11 19:31:13, Heejong Lee <heej...@google.com> >>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>> >> Still seeing no space left on device errors on jenkins-7 >>>>>>>>>>>>>>>> (for example: >>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>> https://builds.apache.org/job/beam_PreCommit_PythonLint_Commit/2754/ >>>>>>>>>>>>>>>> ) >>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>> >> On Fri, Mar 6, 2020 at 7:11 PM Alan Myrvold < >>>>>>>>>>>>>>>> amyrv...@google.com> wrote: >>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>> >>> Did a one time cleanup of tmp files owned by jenkins >>>>>>>>>>>>>>>> older than 3 days. >>>>>>>>>>>>>>>> >>> Agree that we need a longer term solution. >>>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>>> >>> Passing recent tests on all executors except >>>>>>>>>>>>>>>> jenkins-12, which has not >>>>>>>>>>>>>>>> >>> scheduled recent builds for the past 13 days. Not >>>>>>>>>>>>>>>> scheduling: >>>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>>> https://builds.apache.org/computer/apache-beam-jenkins-12/builds >>>>>>>>>>>>>>>> >>> < >>>>>>>>>>>>>>>> https://www.google.com/url?q=https://builds.apache.org/computer/apache-beam-jenkins-12/builds&sa=D >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> >>> Recent passing builds: >>>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>>> https://builds.apache.org/computer/apache-beam-jenkins-1/builds >>>>>>>>>>>>>>>> >>> < >>>>>>>>>>>>>>>> https://www.google.com/url?q=https://builds.apache.org/computer/apache-beam-jenkins-1/builds&sa=D >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>>> https://builds.apache.org/computer/apache-beam-jenkins-2/builds >>>>>>>>>>>>>>>> >>> < >>>>>>>>>>>>>>>> https://www.google.com/url?q=https://builds.apache.org/computer/apache-beam-jenkins-2/builds&sa=D >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>>> https://builds.apache.org/computer/apache-beam-jenkins-3/builds >>>>>>>>>>>>>>>> >>> < >>>>>>>>>>>>>>>> https://www.google.com/url?q=https://builds.apache.org/computer/apache-beam-jenkins-3/builds&sa=D >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>>> https://builds.apache.org/computer/apache-beam-jenkins-4/builds >>>>>>>>>>>>>>>> >>> < >>>>>>>>>>>>>>>> https://www.google.com/url?q=https://builds.apache.org/computer/apache-beam-jenkins-4/builds&sa=D >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>>> https://builds.apache.org/computer/apache-beam-jenkins-5/builds >>>>>>>>>>>>>>>> >>> < >>>>>>>>>>>>>>>> https://www.google.com/url?q=https://builds.apache.org/computer/apache-beam-jenkins-5/builds&sa=D >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>>> https://builds.apache.org/computer/apache-beam-jenkins-6/builds >>>>>>>>>>>>>>>> >>> < >>>>>>>>>>>>>>>> https://www.google.com/url?q=https://builds.apache.org/computer/apache-beam-jenkins-6/builds&sa=D >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>>> https://builds.apache.org/computer/apache-beam-jenkins-7/builds >>>>>>>>>>>>>>>> >>> < >>>>>>>>>>>>>>>> https://www.google.com/url?q=https://builds.apache.org/computer/apache-beam-jenkins-7/builds&sa=D >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>>> https://builds.apache.org/computer/apache-beam-jenkins-8/builds >>>>>>>>>>>>>>>> >>> < >>>>>>>>>>>>>>>> https://www.google.com/url?q=https://builds.apache.org/computer/apache-beam-jenkins-8/builds&sa=D >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>>> https://builds.apache.org/computer/apache-beam-jenkins-9/builds >>>>>>>>>>>>>>>> >>> < >>>>>>>>>>>>>>>> https://www.google.com/url?q=https://builds.apache.org/computer/apache-beam-jenkins-9/builds&sa=D >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>>> https://builds.apache.org/computer/apache-beam-jenkins-10/builds >>>>>>>>>>>>>>>> >>> < >>>>>>>>>>>>>>>> https://www.google.com/url?q=https://builds.apache.org/computer/apache-beam-jenkins-10/builds&sa=D >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>>> https://builds.apache.org/computer/apache-beam-jenkins-11/builds >>>>>>>>>>>>>>>> >>> < >>>>>>>>>>>>>>>> https://www.google.com/url?q=https://builds.apache.org/computer/apache-beam-jenkins-11/builds&sa=D >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>>> https://builds.apache.org/computer/apache-beam-jenkins-13/builds >>>>>>>>>>>>>>>> >>> < >>>>>>>>>>>>>>>> https://www.google.com/url?q=https://builds.apache.org/computer/apache-beam-jenkins-13/builds&sa=D >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>>> https://builds.apache.org/computer/apache-beam-jenkins-14/builds >>>>>>>>>>>>>>>> >>> < >>>>>>>>>>>>>>>> https://www.google.com/url?q=https://builds.apache.org/computer/apache-beam-jenkins-14/builds&sa=D >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>>> https://builds.apache.org/computer/apache-beam-jenkins-15/builds >>>>>>>>>>>>>>>> >>> < >>>>>>>>>>>>>>>> https://www.google.com/url?q=https://builds.apache.org/computer/apache-beam-jenkins-15/builds&sa=D >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>>> https://builds.apache.org/computer/apache-beam-jenkins-16/builds >>>>>>>>>>>>>>>> >>> < >>>>>>>>>>>>>>>> https://www.google.com/url?q=https://builds.apache.org/computer/apache-beam-jenkins-16/builds&sa=D >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>>> >>> On Fri, Mar 6, 2020 at 11:54 AM Ahmet Altay < >>>>>>>>>>>>>>>> al...@google.com> wrote: >>>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>>> >>>> +Alan Myrvold <amyrv...@google.com> is doing a one >>>>>>>>>>>>>>>> time cleanup. I agree >>>>>>>>>>>>>>>> >>>> that we need to have a solution to automate this task >>>>>>>>>>>>>>>> or address the root >>>>>>>>>>>>>>>> >>>> cause of the buildup. >>>>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>>>> >>>> On Thu, Mar 5, 2020 at 2:47 AM Michał Walenia < >>>>>>>>>>>>>>>> michal.wale...@polidea.com> >>>>>>>>>>>>>>>> >>>> wrote: >>>>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>>>> >>>>> Hi there, >>>>>>>>>>>>>>>> >>>>> it seems we have a problem with Jenkins workers >>>>>>>>>>>>>>>> again. Nodes 1 and 7 >>>>>>>>>>>>>>>> >>>>> both fail jobs with "No space left on device". >>>>>>>>>>>>>>>> >>>>> Who is the best person to contact in these cases >>>>>>>>>>>>>>>> (someone with access >>>>>>>>>>>>>>>> >>>>> permissions to the workers). >>>>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>>>> >>>>> I also noticed that such errors are becoming more and >>>>>>>>>>>>>>>> more frequent >>>>>>>>>>>>>>>> >>>>> recently and I'd like to discuss how can this be >>>>>>>>>>>>>>>> remedied. Can a cleanup >>>>>>>>>>>>>>>> >>>>> task be automated on Jenkins somehow? >>>>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>>>> >>>>> Regards >>>>>>>>>>>>>>>> >>>>> Michal >>>>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>>>> >>>>> -- >>>>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>>>> >>>>> Michał Walenia >>>>>>>>>>>>>>>> >>>>> Polidea <https://www.polidea.com/> | Software >>>>>>>>>>>>>>>> Engineer >>>>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>>>> >>>>> M: +48 791 432 002 <+48%20791%20432%20002> < >>>>>>>>>>>>>>>> +48791432002 <+48%20791%20432%20002>> >>>>>>>>>>>>>>>> >>>>> E: michal.wale...@polidea.com >>>>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>>>> >>>>> Unique Tech >>>>>>>>>>>>>>>> >>>>> Check out our projects! < >>>>>>>>>>>>>>>> https://www.polidea.com/our-work> >>>>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
smime.p7s
Description: S/MIME Cryptographic Signature
