Sorry, mistake while copying, [1] should be: [1] https://github.com/apache/beam/blob/8aca8ccc7f1a14516ad769b63845ddd4dc163d92/.test-infra/jenkins/CommonJobProperties.groovy#L63
On Tue, Jul 28, 2020 at 7:21 PM Damian Gadomski <damian.gadom...@polidea.com> wrote: > That's interesting. I didn't check that myself but all the Jenkins jobs > are configured to wipe the workspace just before the actual build happens > [1] > <https://github.com/apache/beam/blob/8aca8ccc7f1a14516ad769b63845ddd4dc163d92/.test-infra/jenkins/CommonJobProperties.groovy#L6>. > Git SCM plugin is used for that and it enables the option called "Wipe out > repository and force clone". Docs state that it "deletes the contents of > the workspace before build and before checkout" [2] > <https://plugins.jenkins.io/git/>. Therefore I assume that removing > workspace just after the build won't change anything. > > The ./.gradle/caches/modules-2/files-2.1 dir is indeed present on the > worker machines but it's rather in /home/jenkins dir. > > damgad@apache-ci-beam-jenkins-13:/home/jenkins/.gradle$ sudo du -sh > 11G . > damgad@apache-ci-beam-jenkins-13:/home/jenkins/.gradle$ sudo du -sh > caches/modules-2/files-2.1 > 2.3G caches/modules-2/files-2.1 > > I can't find that directory structure inside workspaces. > > damgad@apache-ci-beam-jenkins-13:/home/jenkins/jenkins-slave/workspace$ > sudo find -name "files-2.1" > damgad@apache-ci-beam-jenkins-13:/home/jenkins/jenkins-slave/workspace$ > > [1] > https://github.com/apache/beam/blob/8aca8ccc7f1a14516ad769b63845ddd4dc163d92/.test-infra/jenkins/CommonJobProperties.groovy#L6 > [2] https://plugins.jenkins.io/git/ > > On Tue, Jul 28, 2020 at 5:47 PM Kenneth Knowles <k...@apache.org> wrote: > >> Just checking - will this wipe out dependency cache? That will slow >> things down and significantly increase flakiness. If I recall correctly, >> the default Jenkins layout was: >> >> /home/jenkins/jenkins-slave/workspace/$jobname >> /home/jenkins/jenkins-slave/workspace/$jobname/.m2 >> /home/jenkins/jenkins-slave/workspace/$jobname/.git >> >> Where you can see that it did a `git clone` right into the root workspace >> directory, adjacent to .m2. This was not hygienic. One important thing was >> that `git clean` would wipe the maven cache with every build. So in >> https://github.com/apache/beam/pull/3976 we changed it to >> >> /home/jenkins/jenkins-slave/workspace/$jobname >> /home/jenkins/jenkins-slave/workspace/$jobname/.m2 >> /home/jenkins/jenkins-slave/workspace/$jobname/src/.git >> >> Now the .m2 directory survives and we do not constantly see flakes >> re-downloading deps that are immutable. This does, of course, use disk >> space. >> >> That was in the maven days. Gradle is the same except for $HOME/.m2 is >> replaced by $HOME/.gradle/caches/modules-2/files-2.1. Is Jenkins configured >> the same way so we will be wiping out the dependencies? If so, can you >> address this issue? Everything in that directory should be immutable and >> just a cache to avoid pointless re-download. >> >> Kenn >> >> On Tue, Jul 28, 2020 at 2:25 AM Damian Gadomski < >> damian.gadom...@polidea.com> wrote: >> >>> Agree with Udi, workspaces seem to be the third culprit, not yet >>> addressed in any way (until PR#12326 >>> <https://github.com/apache/beam/pull/12326> is merged). I feel that >>> it'll solve the issue of filling up the disks for a long time ;) >>> >>> I'm also OK with moving /tmp cleanup to option B, and will happily >>> investigate on proper TMPDIR config. >>> >>> >>> >>> On Tue, Jul 28, 2020 at 3:07 AM Udi Meiri <eh...@google.com> wrote: >>> >>>> What about the workspaces, which can take up 175GB in some cases (see >>>> above)? >>>> I'm working on getting them cleaned up automatically: >>>> https://github.com/apache/beam/pull/12326 >>>> >>>> My opinion is that we would get more mileage out of fixing the jobs >>>> that leave behind files in /tmp and images/containers in Docker. >>>> This would also help keep development machines clean. >>>> >>>> >>>> On Mon, Jul 27, 2020 at 5:31 PM Tyson Hamilton <tyso...@google.com> >>>> wrote: >>>> >>>>> Here is a summery of how I understand things, >>>>> >>>>> - /tmp and /var/lib/docker are the culprit for filling up disks >>>>> - inventory Jenkins job runs every 12 hours and runs a docker prune >>>>> to clean up images older than 24hr >>>>> - crontab on each machine cleans up /tmp files older than three days >>>>> weekly >>>>> >>>>> This doesn't seem to be working since we're still running out of disk >>>>> periodically and requiring manual intervention. Knobs and options we have >>>>> available: >>>>> >>>>> 1. increase frequency of deleting files >>>>> 2. decrease the number of days required to delete a file (e.g. older >>>>> than 2 days) >>>>> >>>>> The execution methods we have available are: >>>>> >>>>> A. cron >>>>> - pro: runs even if a job gets stuck in Jenkins due to full disk >>>>> - con: config baked into VM which is tough to update, not >>>>> discoverable or documented well >>>>> B. inventory job >>>>> - pro: easy to update, runs every 12h already >>>>> - con: could get stuck if Jenkins agent runs out of disk or is >>>>> otherwise stuck, tied to all other inventory job frequency >>>>> C. configure startup scripts for the VMs that set up the cron job >>>>> anytime the VM is restarted >>>>> - pro: similar to A. and easy to update >>>>> - con: similar to A. >>>>> >>>>> Between the three I prefer B. because it is consistent with other >>>>> inventory jobs. If it ends up that stuck jobs prohibit scheduling of the >>>>> inventory job often we could further investigate C to avoid having to >>>>> rebuild the VM images repeatedly. >>>>> >>>>> Any objections or comments? If not, we'll go forward with B. and >>>>> reduce the date check from 3 days to 2 days. >>>>> >>>>> >>>>> On 2020/07/24 20:13:29, Ahmet Altay <al...@google.com> wrote: >>>>> > Tests may not be doing docker cleanup. Inventory job runs a docker >>>>> prune >>>>> > every 12 hours for images older than 24 hrs [1]. Randomly looking at >>>>> one of >>>>> > the recent runs [2], it cleaned up a long list of containers >>>>> consuming >>>>> > 30+GB space. That should be just 12 hours worth of containers. >>>>> > >>>>> > [1] >>>>> > >>>>> https://github.com/apache/beam/blob/master/.test-infra/jenkins/job_Inventory.groovy#L69 >>>>> > [2] >>>>> > >>>>> https://ci-beam.apache.org/job/beam_Inventory_apache-beam-jenkins-14/501/console >>>>> > >>>>> > On Fri, Jul 24, 2020 at 1:07 PM Tyson Hamilton <tyso...@google.com> >>>>> wrote: >>>>> > >>>>> > > Yes, these are on the same volume in the /var/lib/docker >>>>> directory. I'm >>>>> > > unsure if they clean up leftover images. >>>>> > > >>>>> > > On Fri, Jul 24, 2020 at 12:52 PM Udi Meiri <eh...@google.com> >>>>> wrote: >>>>> > > >>>>> > >> I forgot Docker images: >>>>> > >> >>>>> > >> ehudm@apache-ci-beam-jenkins-3:~$ sudo docker system df >>>>> > >> TYPE TOTAL ACTIVE SIZE >>>>> > >> RECLAIMABLE >>>>> > >> Images 88 9 >>>>> 125.4GB >>>>> > >> 124.2GB (99%) >>>>> > >> Containers 40 4 >>>>> 7.927GB >>>>> > >> 7.871GB (99%) >>>>> > >> Local Volumes 47 0 >>>>> 3.165GB >>>>> > >> 3.165GB (100%) >>>>> > >> Build Cache 0 0 0B >>>>> > >> 0B >>>>> > >> >>>>> > >> There are about 90 images on that machine, with all but 1 less >>>>> than 48 >>>>> > >> hours old. >>>>> > >> I think the docker test jobs need to try harder at cleaning up >>>>> their >>>>> > >> leftover images. (assuming they're already doing it?) >>>>> > >> >>>>> > >> On Fri, Jul 24, 2020 at 12:31 PM Udi Meiri <eh...@google.com> >>>>> wrote: >>>>> > >> >>>>> > >>> The additional slots (@3 directories) take up even more space >>>>> now than >>>>> > >>> before. >>>>> > >>> >>>>> > >>> I'm testing out https://github.com/apache/beam/pull/12326 which >>>>> could >>>>> > >>> help by cleaning up workspaces after a run (just started a seed >>>>> job). >>>>> > >>> >>>>> > >>> On Fri, Jul 24, 2020 at 12:13 PM Tyson Hamilton < >>>>> tyso...@google.com> >>>>> > >>> wrote: >>>>> > >>> >>>>> > >>>> 664M beam_PreCommit_JavaPortabilityApi_Commit >>>>> > >>>> 656M beam_PreCommit_JavaPortabilityApi_Commit@2 >>>>> > >>>> 611M beam_PreCommit_JavaPortabilityApi_Cron >>>>> > >>>> 616M beam_PreCommit_JavaPortabilityApiJava11_Commit >>>>> > >>>> 598M beam_PreCommit_JavaPortabilityApiJava11_Commit@2 >>>>> > >>>> 662M beam_PreCommit_JavaPortabilityApiJava11_Cron >>>>> > >>>> 2.9G beam_PreCommit_Portable_Python_Commit >>>>> > >>>> 2.9G beam_PreCommit_Portable_Python_Commit@2 >>>>> > >>>> 1.7G beam_PreCommit_Portable_Python_Commit@3 >>>>> > >>>> 3.4G beam_PreCommit_Portable_Python_Cron >>>>> > >>>> 1.9G beam_PreCommit_Python2_PVR_Flink_Commit >>>>> > >>>> 1.4G beam_PreCommit_Python2_PVR_Flink_Cron >>>>> > >>>> 1.3G beam_PreCommit_Python2_PVR_Flink_Phrase >>>>> > >>>> 6.2G beam_PreCommit_Python_Commit >>>>> > >>>> 7.5G beam_PreCommit_Python_Commit@2 >>>>> > >>>> 7.5G beam_PreCommit_Python_Cron >>>>> > >>>> 1012M beam_PreCommit_PythonDocker_Commit >>>>> > >>>> 1011M beam_PreCommit_PythonDocker_Commit@2 >>>>> > >>>> 1011M beam_PreCommit_PythonDocker_Commit@3 >>>>> > >>>> 1002M beam_PreCommit_PythonDocker_Cron >>>>> > >>>> 877M beam_PreCommit_PythonFormatter_Commit >>>>> > >>>> 988M beam_PreCommit_PythonFormatter_Cron >>>>> > >>>> 986M beam_PreCommit_PythonFormatter_Phrase >>>>> > >>>> 1.7G beam_PreCommit_PythonLint_Commit >>>>> > >>>> 2.1G beam_PreCommit_PythonLint_Cron >>>>> > >>>> 7.5G beam_PreCommit_Python_Phrase >>>>> > >>>> 346M beam_PreCommit_RAT_Commit >>>>> > >>>> 341M beam_PreCommit_RAT_Cron >>>>> > >>>> 338M beam_PreCommit_Spotless_Commit >>>>> > >>>> 339M beam_PreCommit_Spotless_Cron >>>>> > >>>> 5.5G beam_PreCommit_SQL_Commit >>>>> > >>>> 5.5G beam_PreCommit_SQL_Cron >>>>> > >>>> 5.5G beam_PreCommit_SQL_Java11_Commit >>>>> > >>>> 750M beam_PreCommit_Website_Commit >>>>> > >>>> 750M beam_PreCommit_Website_Commit@2 >>>>> > >>>> 750M beam_PreCommit_Website_Cron >>>>> > >>>> 764M beam_PreCommit_Website_Stage_GCS_Commit >>>>> > >>>> 771M beam_PreCommit_Website_Stage_GCS_Cron >>>>> > >>>> 336M beam_Prober_CommunityMetrics >>>>> > >>>> 693M beam_python_mongoio_load_test >>>>> > >>>> 339M beam_SeedJob >>>>> > >>>> 333M beam_SeedJob_Standalone >>>>> > >>>> 334M beam_sonarqube_report >>>>> > >>>> 556M beam_SQLBigQueryIO_Batch_Performance_Test_Java >>>>> > >>>> 175G total >>>>> > >>>> >>>>> > >>>> On Fri, Jul 24, 2020 at 12:04 PM Tyson Hamilton < >>>>> tyso...@google.com> >>>>> > >>>> wrote: >>>>> > >>>> >>>>> > >>>>> Ya looks like something in the workspaces is taking up room: >>>>> > >>>>> >>>>> > >>>>> @apache-ci-beam-jenkins-8:/home/jenkins$ sudo du -shc . >>>>> > >>>>> 191G . >>>>> > >>>>> 191G total >>>>> > >>>>> >>>>> > >>>>> >>>>> > >>>>> On Fri, Jul 24, 2020 at 11:44 AM Tyson Hamilton < >>>>> tyso...@google.com> >>>>> > >>>>> wrote: >>>>> > >>>>> >>>>> > >>>>>> Node 8 is also full. The partition that /tmp is on is here: >>>>> > >>>>>> >>>>> > >>>>>> Filesystem Size Used Avail Use% Mounted on >>>>> > >>>>>> /dev/sda1 485G 482G 2.9G 100% / >>>>> > >>>>>> >>>>> > >>>>>> however after cleaning up tmp with the crontab command, there >>>>> is only >>>>> > >>>>>> 8G usage yet it still remains 100% full: >>>>> > >>>>>> >>>>> > >>>>>> @apache-ci-beam-jenkins-8:/tmp$ sudo du -shc /tmp >>>>> > >>>>>> 8.0G /tmp >>>>> > >>>>>> 8.0G total >>>>> > >>>>>> >>>>> > >>>>>> The workspaces are in the >>>>> /home/jenkins/jenkins-slave/workspace >>>>> > >>>>>> directory. When I run a du on that, it takes really long. >>>>> I'll let it keep >>>>> > >>>>>> running for a while to see if it ever returns a result but so >>>>> far this >>>>> > >>>>>> seems suspect. >>>>> > >>>>>> >>>>> > >>>>>> >>>>> > >>>>>> >>>>> > >>>>>> >>>>> > >>>>>> >>>>> > >>>>>> On Fri, Jul 24, 2020 at 11:19 AM Tyson Hamilton < >>>>> tyso...@google.com> >>>>> > >>>>>> wrote: >>>>> > >>>>>> >>>>> > >>>>>>> Everything I've been looking at is in the /tmp dir. Where >>>>> are the >>>>> > >>>>>>> workspaces, or what are the named? >>>>> > >>>>>>> >>>>> > >>>>>>> >>>>> > >>>>>>> >>>>> > >>>>>>> >>>>> > >>>>>>> On Fri, Jul 24, 2020 at 11:03 AM Udi Meiri <eh...@google.com> >>>>> wrote: >>>>> > >>>>>>> >>>>> > >>>>>>>> I'm curious to what you find. Was it /tmp or the workspaces >>>>> using >>>>> > >>>>>>>> up the space? >>>>> > >>>>>>>> >>>>> > >>>>>>>> On Fri, Jul 24, 2020 at 10:57 AM Tyson Hamilton < >>>>> tyso...@google.com> >>>>> > >>>>>>>> wrote: >>>>> > >>>>>>>> >>>>> > >>>>>>>>> Bleck. I just realized that it is 'offline' so that won't >>>>> work. >>>>> > >>>>>>>>> I'll clean up manually on the machine using the cron >>>>> command. >>>>> > >>>>>>>>> >>>>> > >>>>>>>>> On Fri, Jul 24, 2020 at 10:56 AM Tyson Hamilton < >>>>> > >>>>>>>>> tyso...@google.com> wrote: >>>>> > >>>>>>>>> >>>>> > >>>>>>>>>> Something isn't working with the current set up because >>>>> node 15 >>>>> > >>>>>>>>>> appears to be out of space and is currently 'offline' >>>>> according to Jenkins. >>>>> > >>>>>>>>>> Can someone run the cleanup job? The machine is full, >>>>> > >>>>>>>>>> >>>>> > >>>>>>>>>> @apache-ci-beam-jenkins-15:/tmp$ df -h >>>>> > >>>>>>>>>> Filesystem Size Used Avail Use% Mounted on >>>>> > >>>>>>>>>> udev 52G 0 52G 0% /dev >>>>> > >>>>>>>>>> tmpfs 11G 265M 10G 3% /run >>>>> > >>>>>>>>>> */dev/sda1 485G 484G 880M 100% /* >>>>> > >>>>>>>>>> tmpfs 52G 0 52G 0% /dev/shm >>>>> > >>>>>>>>>> tmpfs 5.0M 0 5.0M 0% /run/lock >>>>> > >>>>>>>>>> tmpfs 52G 0 52G 0% /sys/fs/cgroup >>>>> > >>>>>>>>>> tmpfs 11G 0 11G 0% /run/user/1017 >>>>> > >>>>>>>>>> tmpfs 11G 0 11G 0% /run/user/1037 >>>>> > >>>>>>>>>> >>>>> > >>>>>>>>>> apache-ci-beam-jenkins-15:/tmp$ sudo du -ah --time . | >>>>> sort -rhk >>>>> > >>>>>>>>>> 1,1 | head -n 20 >>>>> > >>>>>>>>>> 20G 2020-07-24 17:52 . >>>>> > >>>>>>>>>> 580M 2020-07-22 17:31 ./junit1031982597110125586 >>>>> > >>>>>>>>>> 517M 2020-07-22 17:31 >>>>> > >>>>>>>>>> >>>>> ./junit1031982597110125586/junit8739924829337821410/heap_dump.hprof >>>>> > >>>>>>>>>> 517M 2020-07-22 17:31 >>>>> > >>>>>>>>>> ./junit1031982597110125586/junit8739924829337821410 >>>>> > >>>>>>>>>> 263M 2020-07-22 12:23 ./pip-install-2GUhO_ >>>>> > >>>>>>>>>> 263M 2020-07-20 09:30 ./pip-install-sxgwqr >>>>> > >>>>>>>>>> 263M 2020-07-17 13:56 ./pip-install-bWSKIV >>>>> > >>>>>>>>>> 242M 2020-07-21 20:25 ./beam-pipeline-tempmByU6T >>>>> > >>>>>>>>>> 242M 2020-07-21 20:21 ./beam-pipeline-tempV85xeK >>>>> > >>>>>>>>>> 242M 2020-07-21 20:15 ./beam-pipeline-temp7dJROJ >>>>> > >>>>>>>>>> 236M 2020-07-21 20:25 >>>>> > >>>>>>>>>> ./beam-pipeline-tempmByU6T/tmpOWj3Yr >>>>> > >>>>>>>>>> 236M 2020-07-21 20:21 >>>>> > >>>>>>>>>> ./beam-pipeline-tempV85xeK/tmppbQHB3 >>>>> > >>>>>>>>>> 236M 2020-07-21 20:15 >>>>> > >>>>>>>>>> ./beam-pipeline-temp7dJROJ/tmpgOXPKW >>>>> > >>>>>>>>>> 111M 2020-07-23 00:57 ./pip-install-1JnyNE >>>>> > >>>>>>>>>> 105M 2020-07-23 00:17 >>>>> ./beam-artifact1374651823280819755 >>>>> > >>>>>>>>>> 105M 2020-07-23 00:16 >>>>> ./beam-artifact5050755582921936972 >>>>> > >>>>>>>>>> 105M 2020-07-23 00:16 >>>>> ./beam-artifact1834064452502646289 >>>>> > >>>>>>>>>> 105M 2020-07-23 00:15 >>>>> ./beam-artifact682561790267074916 >>>>> > >>>>>>>>>> 105M 2020-07-23 00:15 >>>>> ./beam-artifact4691304965824489394 >>>>> > >>>>>>>>>> 105M 2020-07-23 00:14 >>>>> ./beam-artifact4050383819822604421 >>>>> > >>>>>>>>>> >>>>> > >>>>>>>>>> On Wed, Jul 22, 2020 at 12:03 PM Robert Bradshaw < >>>>> > >>>>>>>>>> rober...@google.com> wrote: >>>>> > >>>>>>>>>> >>>>> > >>>>>>>>>>> On Wed, Jul 22, 2020 at 11:57 AM Tyson Hamilton < >>>>> > >>>>>>>>>>> tyso...@google.com> wrote: >>>>> > >>>>>>>>>>> >>>>> > >>>>>>>>>>>> Ah I see, thanks Kenn. I found some advice from the >>>>> Apache >>>>> > >>>>>>>>>>>> infra wiki that also suggests using a tmpdir inside the >>>>> workspace [1]: >>>>> > >>>>>>>>>>>> >>>>> > >>>>>>>>>>>> Procedures Projects can take to clean up disk space >>>>> > >>>>>>>>>>>> >>>>> > >>>>>>>>>>>> Projects can help themselves and Infra by taking some >>>>> basic >>>>> > >>>>>>>>>>>> steps to help clean up their jobs after themselves on >>>>> the build nodes. >>>>> > >>>>>>>>>>>> >>>>> > >>>>>>>>>>>> >>>>> > >>>>>>>>>>>> >>>>> > >>>>>>>>>>>> 1. Use a ./tmp dir in your jobs workspace. That way >>>>> it gets >>>>> > >>>>>>>>>>>> cleaned up when job workspaces expire. >>>>> > >>>>>>>>>>>> >>>>> > >>>>>>>>>>>> >>>>> > >>>>>>>>>>> Tests should be (able to be) written to use the standard >>>>> > >>>>>>>>>>> temporary file mechanisms, and the environment set up on >>>>> Jenkins such that >>>>> > >>>>>>>>>>> that falls into the respective workspaces. Ideally this >>>>> should be as simple >>>>> > >>>>>>>>>>> as setting the TMPDIR (or similar) environment variable >>>>> (and making sure it >>>>> > >>>>>>>>>>> exists/is writable). >>>>> > >>>>>>>>>>> >>>>> > >>>>>>>>>>>> >>>>> > >>>>>>>>>>>> 1. Configure your jobs to wipe workspaces on start or >>>>> > >>>>>>>>>>>> finish. >>>>> > >>>>>>>>>>>> 2. Configure your jobs to only keep 5 or 10 previous >>>>> builds. >>>>> > >>>>>>>>>>>> 3. Configure your jobs to only keep 5 or 10 previous >>>>> > >>>>>>>>>>>> artifacts. >>>>> > >>>>>>>>>>>> >>>>> > >>>>>>>>>>>> >>>>> > >>>>>>>>>>>> >>>>> > >>>>>>>>>>>> [1]: >>>>> > >>>>>>>>>>>> >>>>> https://cwiki.apache.org/confluence/display/INFRA/Disk+Space+cleanup+of+Jenkins+nodes >>>>> > >>>>>>>>>>>> >>>>> > >>>>>>>>>>>> On Wed, Jul 22, 2020 at 8:06 AM Kenneth Knowles < >>>>> > >>>>>>>>>>>> k...@apache.org> wrote: >>>>> > >>>>>>>>>>>> >>>>> > >>>>>>>>>>>>> Those file listings look like the result of using >>>>> standard >>>>> > >>>>>>>>>>>>> temp file APIs but with TMPDIR set to /tmp. >>>>> > >>>>>>>>>>>>> >>>>> > >>>>>>>>>>>>> On Mon, Jul 20, 2020 at 7:55 PM Tyson Hamilton < >>>>> > >>>>>>>>>>>>> tyso...@google.com> wrote: >>>>> > >>>>>>>>>>>>> >>>>> > >>>>>>>>>>>>>> Jobs are hermetic as far as I can tell and use unique >>>>> > >>>>>>>>>>>>>> subdirectories inside of /tmp. Here is a quick look >>>>> into two examples: >>>>> > >>>>>>>>>>>>>> >>>>> > >>>>>>>>>>>>>> @apache-ci-beam-jenkins-4:/tmp$ sudo du -ah --time . >>>>> | sort >>>>> > >>>>>>>>>>>>>> -rhk 1,1 | head -n 20 >>>>> > >>>>>>>>>>>>>> 1.6G 2020-07-21 02:25 . >>>>> > >>>>>>>>>>>>>> 242M 2020-07-17 18:48 >>>>> ./beam-pipeline-temp3ybuY4 >>>>> > >>>>>>>>>>>>>> 242M 2020-07-17 18:46 >>>>> ./beam-pipeline-tempuxjiPT >>>>> > >>>>>>>>>>>>>> 242M 2020-07-17 18:44 >>>>> ./beam-pipeline-tempVpg1ME >>>>> > >>>>>>>>>>>>>> 242M 2020-07-17 18:42 >>>>> ./beam-pipeline-tempJ4EpyB >>>>> > >>>>>>>>>>>>>> 242M 2020-07-17 18:39 >>>>> ./beam-pipeline-tempepea7Q >>>>> > >>>>>>>>>>>>>> 242M 2020-07-17 18:35 >>>>> ./beam-pipeline-temp79qot2 >>>>> > >>>>>>>>>>>>>> 236M 2020-07-17 18:48 >>>>> > >>>>>>>>>>>>>> ./beam-pipeline-temp3ybuY4/tmpy_Ytzz >>>>> > >>>>>>>>>>>>>> 236M 2020-07-17 18:46 >>>>> > >>>>>>>>>>>>>> ./beam-pipeline-tempuxjiPT/tmpN5_UfJ >>>>> > >>>>>>>>>>>>>> 236M 2020-07-17 18:44 >>>>> > >>>>>>>>>>>>>> ./beam-pipeline-tempVpg1ME/tmpxSm8pX >>>>> > >>>>>>>>>>>>>> 236M 2020-07-17 18:42 >>>>> > >>>>>>>>>>>>>> ./beam-pipeline-tempJ4EpyB/tmpMZJU76 >>>>> > >>>>>>>>>>>>>> 236M 2020-07-17 18:39 >>>>> > >>>>>>>>>>>>>> ./beam-pipeline-tempepea7Q/tmpWy1vWX >>>>> > >>>>>>>>>>>>>> 236M 2020-07-17 18:35 >>>>> > >>>>>>>>>>>>>> ./beam-pipeline-temp79qot2/tmpvN7vWA >>>>> > >>>>>>>>>>>>>> 3.7M 2020-07-17 18:48 >>>>> > >>>>>>>>>>>>>> ./beam-pipeline-temp3ybuY4/tmprlh_di >>>>> > >>>>>>>>>>>>>> 3.7M 2020-07-17 18:46 >>>>> > >>>>>>>>>>>>>> ./beam-pipeline-tempuxjiPT/tmpLmVWfe >>>>> > >>>>>>>>>>>>>> 3.7M 2020-07-17 18:44 >>>>> > >>>>>>>>>>>>>> ./beam-pipeline-tempVpg1ME/tmpvrxbY7 >>>>> > >>>>>>>>>>>>>> 3.7M 2020-07-17 18:42 >>>>> > >>>>>>>>>>>>>> ./beam-pipeline-tempJ4EpyB/tmpLTb6Mj >>>>> > >>>>>>>>>>>>>> 3.7M 2020-07-17 18:39 >>>>> > >>>>>>>>>>>>>> ./beam-pipeline-tempepea7Q/tmptYF1v1 >>>>> > >>>>>>>>>>>>>> 3.7M 2020-07-17 18:35 >>>>> > >>>>>>>>>>>>>> ./beam-pipeline-temp79qot2/tmplfV0Rg >>>>> > >>>>>>>>>>>>>> 2.7M 2020-07-17 20:10 ./pip-install-q9l227ef >>>>> > >>>>>>>>>>>>>> >>>>> > >>>>>>>>>>>>>> >>>>> > >>>>>>>>>>>>>> @apache-ci-beam-jenkins-11:/tmp$ sudo du -ah --time . >>>>> | sort >>>>> > >>>>>>>>>>>>>> -rhk 1,1 | head -n 20 >>>>> > >>>>>>>>>>>>>> 817M 2020-07-21 02:26 . >>>>> > >>>>>>>>>>>>>> 242M 2020-07-19 12:14 >>>>> ./beam-pipeline-tempUTXqlM >>>>> > >>>>>>>>>>>>>> 242M 2020-07-19 12:11 >>>>> ./beam-pipeline-tempx3Yno3 >>>>> > >>>>>>>>>>>>>> 242M 2020-07-19 12:05 >>>>> ./beam-pipeline-tempyCrMYq >>>>> > >>>>>>>>>>>>>> 236M 2020-07-19 12:14 >>>>> > >>>>>>>>>>>>>> ./beam-pipeline-tempUTXqlM/tmpstXoL0 >>>>> > >>>>>>>>>>>>>> 236M 2020-07-19 12:11 >>>>> > >>>>>>>>>>>>>> ./beam-pipeline-tempx3Yno3/tmpnnVn65 >>>>> > >>>>>>>>>>>>>> 236M 2020-07-19 12:05 >>>>> > >>>>>>>>>>>>>> ./beam-pipeline-tempyCrMYq/tmpRF0iNs >>>>> > >>>>>>>>>>>>>> 3.7M 2020-07-19 12:14 >>>>> > >>>>>>>>>>>>>> ./beam-pipeline-tempUTXqlM/tmpbJjUAQ >>>>> > >>>>>>>>>>>>>> 3.7M 2020-07-19 12:11 >>>>> > >>>>>>>>>>>>>> ./beam-pipeline-tempx3Yno3/tmpsmmzqe >>>>> > >>>>>>>>>>>>>> 3.7M 2020-07-19 12:05 >>>>> > >>>>>>>>>>>>>> ./beam-pipeline-tempyCrMYq/tmp5b3ZvY >>>>> > >>>>>>>>>>>>>> 2.0M 2020-07-19 12:14 >>>>> > >>>>>>>>>>>>>> ./beam-pipeline-tempUTXqlM/tmpoj3orz >>>>> > >>>>>>>>>>>>>> 2.0M 2020-07-19 12:11 >>>>> > >>>>>>>>>>>>>> ./beam-pipeline-tempx3Yno3/tmptng9sZ >>>>> > >>>>>>>>>>>>>> 2.0M 2020-07-19 12:05 >>>>> > >>>>>>>>>>>>>> ./beam-pipeline-tempyCrMYq/tmpWp6njc >>>>> > >>>>>>>>>>>>>> 1.2M 2020-07-19 12:14 >>>>> > >>>>>>>>>>>>>> ./beam-pipeline-tempUTXqlM/tmphgdj35 >>>>> > >>>>>>>>>>>>>> 1.2M 2020-07-19 12:11 >>>>> > >>>>>>>>>>>>>> ./beam-pipeline-tempx3Yno3/tmp8ySXpm >>>>> > >>>>>>>>>>>>>> 1.2M 2020-07-19 12:05 >>>>> > >>>>>>>>>>>>>> ./beam-pipeline-tempyCrMYq/tmpNVEJ4e >>>>> > >>>>>>>>>>>>>> 992K 2020-07-12 12:00 >>>>> ./junit642086915811430564 >>>>> > >>>>>>>>>>>>>> 988K 2020-07-12 12:00 >>>>> ./junit642086915811430564/beam >>>>> > >>>>>>>>>>>>>> 984K 2020-07-12 12:00 >>>>> > >>>>>>>>>>>>>> ./junit642086915811430564/beam/nodes >>>>> > >>>>>>>>>>>>>> 980K 2020-07-12 12:00 >>>>> > >>>>>>>>>>>>>> ./junit642086915811430564/beam/nodes/0 >>>>> > >>>>>>>>>>>>>> >>>>> > >>>>>>>>>>>>>> >>>>> > >>>>>>>>>>>>>> >>>>> > >>>>>>>>>>>>>> On Mon, Jul 20, 2020 at 6:46 PM Udi Meiri < >>>>> eh...@google.com> >>>>> > >>>>>>>>>>>>>> wrote: >>>>> > >>>>>>>>>>>>>> >>>>> > >>>>>>>>>>>>>>> You're right, job workspaces should be hermetic. >>>>> > >>>>>>>>>>>>>>> >>>>> > >>>>>>>>>>>>>>> >>>>> > >>>>>>>>>>>>>>> >>>>> > >>>>>>>>>>>>>>> On Mon, Jul 20, 2020 at 1:24 PM Kenneth Knowles < >>>>> > >>>>>>>>>>>>>>> k...@apache.org> wrote: >>>>> > >>>>>>>>>>>>>>> >>>>> > >>>>>>>>>>>>>>>> I'm probably late to this discussion and missing >>>>> something, >>>>> > >>>>>>>>>>>>>>>> but why are we writing to /tmp at all? I would >>>>> expect TMPDIR to point >>>>> > >>>>>>>>>>>>>>>> somewhere inside the job directory that will be >>>>> wiped by Jenkins, and I >>>>> > >>>>>>>>>>>>>>>> would expect code to always create temp files via >>>>> APIs that respect this. >>>>> > >>>>>>>>>>>>>>>> Is Jenkins not cleaning up? Do we not have the >>>>> ability to set this up? Do >>>>> > >>>>>>>>>>>>>>>> we have bugs in our code (that we could probably >>>>> find by setting TMPDIR to >>>>> > >>>>>>>>>>>>>>>> somewhere not-/tmp and running the tests without >>>>> write permission to /tmp, >>>>> > >>>>>>>>>>>>>>>> etc) >>>>> > >>>>>>>>>>>>>>>> >>>>> > >>>>>>>>>>>>>>>> Kenn >>>>> > >>>>>>>>>>>>>>>> >>>>> > >>>>>>>>>>>>>>>> On Mon, Jul 20, 2020 at 11:39 AM Ahmet Altay < >>>>> > >>>>>>>>>>>>>>>> al...@google.com> wrote: >>>>> > >>>>>>>>>>>>>>>> >>>>> > >>>>>>>>>>>>>>>>> Related to workspace directory growth, +Udi Meiri >>>>> > >>>>>>>>>>>>>>>>> <eh...@google.com> filed a relevant issue >>>>> previously ( >>>>> > >>>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/BEAM-9865) >>>>> for >>>>> > >>>>>>>>>>>>>>>>> cleaning up workspace directory after successful >>>>> jobs. Alternatively, we >>>>> > >>>>>>>>>>>>>>>>> can consider periodically cleaning up the /src >>>>> directories. >>>>> > >>>>>>>>>>>>>>>>> >>>>> > >>>>>>>>>>>>>>>>> I would suggest moving the cron task from internal >>>>> cron >>>>> > >>>>>>>>>>>>>>>>> scripts to the inventory job ( >>>>> > >>>>>>>>>>>>>>>>> >>>>> https://github.com/apache/beam/blob/master/.test-infra/jenkins/job_Inventory.groovy#L51 >>>>> ). >>>>> > >>>>>>>>>>>>>>>>> That way, we can see all the cron jobs as part of >>>>> the source tree, adjust >>>>> > >>>>>>>>>>>>>>>>> frequencies and clean up codes with PRs. I do not >>>>> know how internal cron >>>>> > >>>>>>>>>>>>>>>>> scripts are created, maintained, and how would >>>>> they be recreated for new >>>>> > >>>>>>>>>>>>>>>>> worker instances. >>>>> > >>>>>>>>>>>>>>>>> >>>>> > >>>>>>>>>>>>>>>>> /cc +Tyson Hamilton <tyso...@google.com> >>>>> > >>>>>>>>>>>>>>>>> >>>>> > >>>>>>>>>>>>>>>>> On Mon, Jul 20, 2020 at 4:50 AM Damian Gadomski < >>>>> > >>>>>>>>>>>>>>>>> damian.gadom...@polidea.com> wrote: >>>>> > >>>>>>>>>>>>>>>>> >>>>> > >>>>>>>>>>>>>>>>>> Hey, >>>>> > >>>>>>>>>>>>>>>>>> >>>>> > >>>>>>>>>>>>>>>>>> I've recently created a solution for the growing >>>>> /tmp >>>>> > >>>>>>>>>>>>>>>>>> directory. Part of it is the job mentioned by >>>>> Tyson: >>>>> > >>>>>>>>>>>>>>>>>> *beam_Clean_tmp_directory*. It's intentionally not >>>>> > >>>>>>>>>>>>>>>>>> triggered by cron and should be a last resort >>>>> solution for some strange >>>>> > >>>>>>>>>>>>>>>>>> cases. >>>>> > >>>>>>>>>>>>>>>>>> >>>>> > >>>>>>>>>>>>>>>>>> Along with that job, I've also updated every >>>>> worker with >>>>> > >>>>>>>>>>>>>>>>>> an internal cron script. It's being executed once >>>>> a week and deletes all >>>>> > >>>>>>>>>>>>>>>>>> the files (and only files) that were not accessed >>>>> for at least three days. >>>>> > >>>>>>>>>>>>>>>>>> That's designed to be as safe as possible for the >>>>> running jobs on the >>>>> > >>>>>>>>>>>>>>>>>> worker (not to delete the files that are still in >>>>> use), and also to be >>>>> > >>>>>>>>>>>>>>>>>> insensitive to the current workload on the >>>>> machine. The cleanup will always >>>>> > >>>>>>>>>>>>>>>>>> happen, even if some long-running/stuck jobs are >>>>> blocking the machine. >>>>> > >>>>>>>>>>>>>>>>>> >>>>> > >>>>>>>>>>>>>>>>>> I also think that currently the "No space left" >>>>> errors >>>>> > >>>>>>>>>>>>>>>>>> may be a consequence of growing workspace >>>>> directory rather than /tmp. I >>>>> > >>>>>>>>>>>>>>>>>> didn't do any detailed analysis but e.g. >>>>> currently, on >>>>> > >>>>>>>>>>>>>>>>>> apache-beam-jenkins-7 the workspace directory >>>>> size is 158 GB while /tmp is >>>>> > >>>>>>>>>>>>>>>>>> only 16 GB. We should either guarantee the disk >>>>> size to hold workspaces for >>>>> > >>>>>>>>>>>>>>>>>> all jobs (because eventually, every worker will >>>>> execute each job) or clear >>>>> > >>>>>>>>>>>>>>>>>> also the workspaces in some way. >>>>> > >>>>>>>>>>>>>>>>>> >>>>> > >>>>>>>>>>>>>>>>>> Regards, >>>>> > >>>>>>>>>>>>>>>>>> Damian >>>>> > >>>>>>>>>>>>>>>>>> >>>>> > >>>>>>>>>>>>>>>>>> >>>>> > >>>>>>>>>>>>>>>>>> On Mon, Jul 20, 2020 at 10:43 AM Maximilian >>>>> Michels < >>>>> > >>>>>>>>>>>>>>>>>> m...@apache.org> wrote: >>>>> > >>>>>>>>>>>>>>>>>> >>>>> > >>>>>>>>>>>>>>>>>>> +1 for scheduling it via a cron job if it won't >>>>> lead to >>>>> > >>>>>>>>>>>>>>>>>>> test failures >>>>> > >>>>>>>>>>>>>>>>>>> while running. Not a Jenkins expert but maybe >>>>> there is >>>>> > >>>>>>>>>>>>>>>>>>> the notion of >>>>> > >>>>>>>>>>>>>>>>>>> running exclusively while no other tasks are >>>>> running? >>>>> > >>>>>>>>>>>>>>>>>>> >>>>> > >>>>>>>>>>>>>>>>>>> -Max >>>>> > >>>>>>>>>>>>>>>>>>> >>>>> > >>>>>>>>>>>>>>>>>>> On 17.07.20 21:49, Tyson Hamilton wrote: >>>>> > >>>>>>>>>>>>>>>>>>> > FYI there was a job introduced to do this in >>>>> Jenkins: >>>>> > >>>>>>>>>>>>>>>>>>> beam_Clean_tmp_directory >>>>> > >>>>>>>>>>>>>>>>>>> > >>>>> > >>>>>>>>>>>>>>>>>>> > Currently it needs to be run manually. I'm >>>>> seeing some >>>>> > >>>>>>>>>>>>>>>>>>> out of disk related errors in precommit tests >>>>> currently, perhaps we should >>>>> > >>>>>>>>>>>>>>>>>>> schedule this job with cron? >>>>> > >>>>>>>>>>>>>>>>>>> > >>>>> > >>>>>>>>>>>>>>>>>>> > >>>>> > >>>>>>>>>>>>>>>>>>> > On 2020/03/11 19:31:13, Heejong Lee < >>>>> > >>>>>>>>>>>>>>>>>>> heej...@google.com> wrote: >>>>> > >>>>>>>>>>>>>>>>>>> >> Still seeing no space left on device errors on >>>>> > >>>>>>>>>>>>>>>>>>> jenkins-7 (for example: >>>>> > >>>>>>>>>>>>>>>>>>> >> >>>>> > >>>>>>>>>>>>>>>>>>> >>>>> https://builds.apache.org/job/beam_PreCommit_PythonLint_Commit/2754/ >>>>> > >>>>>>>>>>>>>>>>>>> ) >>>>> > >>>>>>>>>>>>>>>>>>> >> >>>>> > >>>>>>>>>>>>>>>>>>> >> >>>>> > >>>>>>>>>>>>>>>>>>> >> On Fri, Mar 6, 2020 at 7:11 PM Alan Myrvold < >>>>> > >>>>>>>>>>>>>>>>>>> amyrv...@google.com> wrote: >>>>> > >>>>>>>>>>>>>>>>>>> >> >>>>> > >>>>>>>>>>>>>>>>>>> >>> Did a one time cleanup of tmp files owned by >>>>> jenkins >>>>> > >>>>>>>>>>>>>>>>>>> older than 3 days. >>>>> > >>>>>>>>>>>>>>>>>>> >>> Agree that we need a longer term solution. >>>>> > >>>>>>>>>>>>>>>>>>> >>> >>>>> > >>>>>>>>>>>>>>>>>>> >>> Passing recent tests on all executors except >>>>> > >>>>>>>>>>>>>>>>>>> jenkins-12, which has not >>>>> > >>>>>>>>>>>>>>>>>>> >>> scheduled recent builds for the past 13 >>>>> days. Not >>>>> > >>>>>>>>>>>>>>>>>>> scheduling: >>>>> > >>>>>>>>>>>>>>>>>>> >>> >>>>> > >>>>>>>>>>>>>>>>>>> >>>>> https://builds.apache.org/computer/apache-beam-jenkins-12/builds >>>>> > >>>>>>>>>>>>>>>>>>> >>> < >>>>> > >>>>>>>>>>>>>>>>>>> >>>>> https://www.google.com/url?q=https://builds.apache.org/computer/apache-beam-jenkins-12/builds&sa=D >>>>> > >>>>>>>>>>>>>>>>>>> > >>>>> > >>>>>>>>>>>>>>>>>>> >>> Recent passing builds: >>>>> > >>>>>>>>>>>>>>>>>>> >>> >>>>> > >>>>>>>>>>>>>>>>>>> >>>>> https://builds.apache.org/computer/apache-beam-jenkins-1/builds >>>>> > >>>>>>>>>>>>>>>>>>> >>> < >>>>> > >>>>>>>>>>>>>>>>>>> >>>>> https://www.google.com/url?q=https://builds.apache.org/computer/apache-beam-jenkins-1/builds&sa=D >>>>> > >>>>>>>>>>>>>>>>>>> > >>>>> > >>>>>>>>>>>>>>>>>>> >>> >>>>> > >>>>>>>>>>>>>>>>>>> >>>>> https://builds.apache.org/computer/apache-beam-jenkins-2/builds >>>>> > >>>>>>>>>>>>>>>>>>> >>> < >>>>> > >>>>>>>>>>>>>>>>>>> >>>>> https://www.google.com/url?q=https://builds.apache.org/computer/apache-beam-jenkins-2/builds&sa=D >>>>> > >>>>>>>>>>>>>>>>>>> > >>>>> > >>>>>>>>>>>>>>>>>>> >>> >>>>> > >>>>>>>>>>>>>>>>>>> >>>>> https://builds.apache.org/computer/apache-beam-jenkins-3/builds >>>>> > >>>>>>>>>>>>>>>>>>> >>> < >>>>> > >>>>>>>>>>>>>>>>>>> >>>>> https://www.google.com/url?q=https://builds.apache.org/computer/apache-beam-jenkins-3/builds&sa=D >>>>> > >>>>>>>>>>>>>>>>>>> > >>>>> > >>>>>>>>>>>>>>>>>>> >>> >>>>> > >>>>>>>>>>>>>>>>>>> >>>>> https://builds.apache.org/computer/apache-beam-jenkins-4/builds >>>>> > >>>>>>>>>>>>>>>>>>> >>> < >>>>> > >>>>>>>>>>>>>>>>>>> >>>>> https://www.google.com/url?q=https://builds.apache.org/computer/apache-beam-jenkins-4/builds&sa=D >>>>> > >>>>>>>>>>>>>>>>>>> > >>>>> > >>>>>>>>>>>>>>>>>>> >>> >>>>> > >>>>>>>>>>>>>>>>>>> >>>>> https://builds.apache.org/computer/apache-beam-jenkins-5/builds >>>>> > >>>>>>>>>>>>>>>>>>> >>> < >>>>> > >>>>>>>>>>>>>>>>>>> >>>>> https://www.google.com/url?q=https://builds.apache.org/computer/apache-beam-jenkins-5/builds&sa=D >>>>> > >>>>>>>>>>>>>>>>>>> > >>>>> > >>>>>>>>>>>>>>>>>>> >>> >>>>> > >>>>>>>>>>>>>>>>>>> >>>>> https://builds.apache.org/computer/apache-beam-jenkins-6/builds >>>>> > >>>>>>>>>>>>>>>>>>> >>> < >>>>> > >>>>>>>>>>>>>>>>>>> >>>>> https://www.google.com/url?q=https://builds.apache.org/computer/apache-beam-jenkins-6/builds&sa=D >>>>> > >>>>>>>>>>>>>>>>>>> > >>>>> > >>>>>>>>>>>>>>>>>>> >>> >>>>> > >>>>>>>>>>>>>>>>>>> >>>>> https://builds.apache.org/computer/apache-beam-jenkins-7/builds >>>>> > >>>>>>>>>>>>>>>>>>> >>> < >>>>> > >>>>>>>>>>>>>>>>>>> >>>>> https://www.google.com/url?q=https://builds.apache.org/computer/apache-beam-jenkins-7/builds&sa=D >>>>> > >>>>>>>>>>>>>>>>>>> > >>>>> > >>>>>>>>>>>>>>>>>>> >>> >>>>> > >>>>>>>>>>>>>>>>>>> >>>>> https://builds.apache.org/computer/apache-beam-jenkins-8/builds >>>>> > >>>>>>>>>>>>>>>>>>> >>> < >>>>> > >>>>>>>>>>>>>>>>>>> >>>>> https://www.google.com/url?q=https://builds.apache.org/computer/apache-beam-jenkins-8/builds&sa=D >>>>> > >>>>>>>>>>>>>>>>>>> > >>>>> > >>>>>>>>>>>>>>>>>>> >>> >>>>> > >>>>>>>>>>>>>>>>>>> >>>>> https://builds.apache.org/computer/apache-beam-jenkins-9/builds >>>>> > >>>>>>>>>>>>>>>>>>> >>> < >>>>> > >>>>>>>>>>>>>>>>>>> >>>>> https://www.google.com/url?q=https://builds.apache.org/computer/apache-beam-jenkins-9/builds&sa=D >>>>> > >>>>>>>>>>>>>>>>>>> > >>>>> > >>>>>>>>>>>>>>>>>>> >>> >>>>> > >>>>>>>>>>>>>>>>>>> >>>>> https://builds.apache.org/computer/apache-beam-jenkins-10/builds >>>>> > >>>>>>>>>>>>>>>>>>> >>> < >>>>> > >>>>>>>>>>>>>>>>>>> >>>>> https://www.google.com/url?q=https://builds.apache.org/computer/apache-beam-jenkins-10/builds&sa=D >>>>> > >>>>>>>>>>>>>>>>>>> > >>>>> > >>>>>>>>>>>>>>>>>>> >>> >>>>> > >>>>>>>>>>>>>>>>>>> >>>>> https://builds.apache.org/computer/apache-beam-jenkins-11/builds >>>>> > >>>>>>>>>>>>>>>>>>> >>> < >>>>> > >>>>>>>>>>>>>>>>>>> >>>>> https://www.google.com/url?q=https://builds.apache.org/computer/apache-beam-jenkins-11/builds&sa=D >>>>> > >>>>>>>>>>>>>>>>>>> > >>>>> > >>>>>>>>>>>>>>>>>>> >>> >>>>> > >>>>>>>>>>>>>>>>>>> >>>>> https://builds.apache.org/computer/apache-beam-jenkins-13/builds >>>>> > >>>>>>>>>>>>>>>>>>> >>> < >>>>> > >>>>>>>>>>>>>>>>>>> >>>>> https://www.google.com/url?q=https://builds.apache.org/computer/apache-beam-jenkins-13/builds&sa=D >>>>> > >>>>>>>>>>>>>>>>>>> > >>>>> > >>>>>>>>>>>>>>>>>>> >>> >>>>> > >>>>>>>>>>>>>>>>>>> >>>>> https://builds.apache.org/computer/apache-beam-jenkins-14/builds >>>>> > >>>>>>>>>>>>>>>>>>> >>> < >>>>> > >>>>>>>>>>>>>>>>>>> >>>>> https://www.google.com/url?q=https://builds.apache.org/computer/apache-beam-jenkins-14/builds&sa=D >>>>> > >>>>>>>>>>>>>>>>>>> > >>>>> > >>>>>>>>>>>>>>>>>>> >>> >>>>> > >>>>>>>>>>>>>>>>>>> >>>>> https://builds.apache.org/computer/apache-beam-jenkins-15/builds >>>>> > >>>>>>>>>>>>>>>>>>> >>> < >>>>> > >>>>>>>>>>>>>>>>>>> >>>>> https://www.google.com/url?q=https://builds.apache.org/computer/apache-beam-jenkins-15/builds&sa=D >>>>> > >>>>>>>>>>>>>>>>>>> > >>>>> > >>>>>>>>>>>>>>>>>>> >>> >>>>> > >>>>>>>>>>>>>>>>>>> >>>>> https://builds.apache.org/computer/apache-beam-jenkins-16/builds >>>>> > >>>>>>>>>>>>>>>>>>> >>> < >>>>> > >>>>>>>>>>>>>>>>>>> >>>>> https://www.google.com/url?q=https://builds.apache.org/computer/apache-beam-jenkins-16/builds&sa=D >>>>> > >>>>>>>>>>>>>>>>>>> > >>>>> > >>>>>>>>>>>>>>>>>>> >>> >>>>> > >>>>>>>>>>>>>>>>>>> >>> On Fri, Mar 6, 2020 at 11:54 AM Ahmet Altay < >>>>> > >>>>>>>>>>>>>>>>>>> al...@google.com> wrote: >>>>> > >>>>>>>>>>>>>>>>>>> >>> >>>>> > >>>>>>>>>>>>>>>>>>> >>>> +Alan Myrvold <amyrv...@google.com> is >>>>> doing a one >>>>> > >>>>>>>>>>>>>>>>>>> time cleanup. I agree >>>>> > >>>>>>>>>>>>>>>>>>> >>>> that we need to have a solution to automate >>>>> this >>>>> > >>>>>>>>>>>>>>>>>>> task or address the root >>>>> > >>>>>>>>>>>>>>>>>>> >>>> cause of the buildup. >>>>> > >>>>>>>>>>>>>>>>>>> >>>> >>>>> > >>>>>>>>>>>>>>>>>>> >>>> On Thu, Mar 5, 2020 at 2:47 AM Michał >>>>> Walenia < >>>>> > >>>>>>>>>>>>>>>>>>> michal.wale...@polidea.com> >>>>> > >>>>>>>>>>>>>>>>>>> >>>> wrote: >>>>> > >>>>>>>>>>>>>>>>>>> >>>> >>>>> > >>>>>>>>>>>>>>>>>>> >>>>> Hi there, >>>>> > >>>>>>>>>>>>>>>>>>> >>>>> it seems we have a problem with Jenkins >>>>> workers >>>>> > >>>>>>>>>>>>>>>>>>> again. Nodes 1 and 7 >>>>> > >>>>>>>>>>>>>>>>>>> >>>>> both fail jobs with "No space left on >>>>> device". >>>>> > >>>>>>>>>>>>>>>>>>> >>>>> Who is the best person to contact in these >>>>> cases >>>>> > >>>>>>>>>>>>>>>>>>> (someone with access >>>>> > >>>>>>>>>>>>>>>>>>> >>>>> permissions to the workers). >>>>> > >>>>>>>>>>>>>>>>>>> >>>>> >>>>> > >>>>>>>>>>>>>>>>>>> >>>>> I also noticed that such errors are >>>>> becoming more >>>>> > >>>>>>>>>>>>>>>>>>> and more frequent >>>>> > >>>>>>>>>>>>>>>>>>> >>>>> recently and I'd like to discuss how can >>>>> this be >>>>> > >>>>>>>>>>>>>>>>>>> remedied. Can a cleanup >>>>> > >>>>>>>>>>>>>>>>>>> >>>>> task be automated on Jenkins somehow? >>>>> > >>>>>>>>>>>>>>>>>>> >>>>> >>>>> > >>>>>>>>>>>>>>>>>>> >>>>> Regards >>>>> > >>>>>>>>>>>>>>>>>>> >>>>> Michal >>>>> > >>>>>>>>>>>>>>>>>>> >>>>> >>>>> > >>>>>>>>>>>>>>>>>>> >>>>> -- >>>>> > >>>>>>>>>>>>>>>>>>> >>>>> >>>>> > >>>>>>>>>>>>>>>>>>> >>>>> Michał Walenia >>>>> > >>>>>>>>>>>>>>>>>>> >>>>> Polidea <https://www.polidea.com/> | >>>>> Software >>>>> > >>>>>>>>>>>>>>>>>>> Engineer >>>>> > >>>>>>>>>>>>>>>>>>> >>>>> >>>>> > >>>>>>>>>>>>>>>>>>> >>>>> M: +48 791 432 002 <+48%20791%20432%20002> >>>>> <+48%20791%20432%20002> < >>>>> > >>>>>>>>>>>>>>>>>>> +48791432002 <+48%20791%20432%20002> >>>>> <+48%20791%20432%20002>> >>>>> > >>>>>>>>>>>>>>>>>>> >>>>> E: michal.wale...@polidea.com >>>>> > >>>>>>>>>>>>>>>>>>> >>>>> >>>>> > >>>>>>>>>>>>>>>>>>> >>>>> Unique Tech >>>>> > >>>>>>>>>>>>>>>>>>> >>>>> Check out our projects! < >>>>> > >>>>>>>>>>>>>>>>>>> https://www.polidea.com/our-work> >>>>> > >>>>>>>>>>>>>>>>>>> >>>>> >>>>> > >>>>>>>>>>>>>>>>>>> >>>> >>>>> > >>>>>>>>>>>>>>>>>>> >> >>>>> > >>>>>>>>>>>>>>>>>>> >>>>> > >>>>>>>>>>>>>>>>>> >>>>> > >>>>> >>>>
