I forgot Docker images:
ehudm@apache-ci-beam-jenkins-3:~$ sudo docker system df
TYPE TOTAL ACTIVE SIZE
RECLAIMABLE
Images 88 9 125.4GB
124.2GB (99%)
Containers 40 4 7.927GB
7.871GB (99%)
Local Volumes 47 0 3.165GB
3.165GB (100%)
Build Cache 0 0 0B
0BThere are about 90 images on that machine, with all but 1 less than 48 hours old. I think the docker test jobs need to try harder at cleaning up their leftover images. (assuming they're already doing it?) On Fri, Jul 24, 2020 at 12:31 PM Udi Meiri <eh...@google.com> wrote: > The additional slots (@3 directories) take up even more space now than > before. > > I'm testing out https://github.com/apache/beam/pull/12326 which could > help by cleaning up workspaces after a run (just started a seed job). > > On Fri, Jul 24, 2020 at 12:13 PM Tyson Hamilton <tyso...@google.com> > wrote: > >> 664M beam_PreCommit_JavaPortabilityApi_Commit >> 656M beam_PreCommit_JavaPortabilityApi_Commit@2 >> 611M beam_PreCommit_JavaPortabilityApi_Cron >> 616M beam_PreCommit_JavaPortabilityApiJava11_Commit >> 598M beam_PreCommit_JavaPortabilityApiJava11_Commit@2 >> 662M beam_PreCommit_JavaPortabilityApiJava11_Cron >> 2.9G beam_PreCommit_Portable_Python_Commit >> 2.9G beam_PreCommit_Portable_Python_Commit@2 >> 1.7G beam_PreCommit_Portable_Python_Commit@3 >> 3.4G beam_PreCommit_Portable_Python_Cron >> 1.9G beam_PreCommit_Python2_PVR_Flink_Commit >> 1.4G beam_PreCommit_Python2_PVR_Flink_Cron >> 1.3G beam_PreCommit_Python2_PVR_Flink_Phrase >> 6.2G beam_PreCommit_Python_Commit >> 7.5G beam_PreCommit_Python_Commit@2 >> 7.5G beam_PreCommit_Python_Cron >> 1012M beam_PreCommit_PythonDocker_Commit >> 1011M beam_PreCommit_PythonDocker_Commit@2 >> 1011M beam_PreCommit_PythonDocker_Commit@3 >> 1002M beam_PreCommit_PythonDocker_Cron >> 877M beam_PreCommit_PythonFormatter_Commit >> 988M beam_PreCommit_PythonFormatter_Cron >> 986M beam_PreCommit_PythonFormatter_Phrase >> 1.7G beam_PreCommit_PythonLint_Commit >> 2.1G beam_PreCommit_PythonLint_Cron >> 7.5G beam_PreCommit_Python_Phrase >> 346M beam_PreCommit_RAT_Commit >> 341M beam_PreCommit_RAT_Cron >> 338M beam_PreCommit_Spotless_Commit >> 339M beam_PreCommit_Spotless_Cron >> 5.5G beam_PreCommit_SQL_Commit >> 5.5G beam_PreCommit_SQL_Cron >> 5.5G beam_PreCommit_SQL_Java11_Commit >> 750M beam_PreCommit_Website_Commit >> 750M beam_PreCommit_Website_Commit@2 >> 750M beam_PreCommit_Website_Cron >> 764M beam_PreCommit_Website_Stage_GCS_Commit >> 771M beam_PreCommit_Website_Stage_GCS_Cron >> 336M beam_Prober_CommunityMetrics >> 693M beam_python_mongoio_load_test >> 339M beam_SeedJob >> 333M beam_SeedJob_Standalone >> 334M beam_sonarqube_report >> 556M beam_SQLBigQueryIO_Batch_Performance_Test_Java >> 175G total >> >> On Fri, Jul 24, 2020 at 12:04 PM Tyson Hamilton <tyso...@google.com> >> wrote: >> >>> Ya looks like something in the workspaces is taking up room: >>> >>> @apache-ci-beam-jenkins-8:/home/jenkins$ sudo du -shc . >>> 191G . >>> 191G total >>> >>> >>> On Fri, Jul 24, 2020 at 11:44 AM Tyson Hamilton <tyso...@google.com> >>> wrote: >>> >>>> Node 8 is also full. The partition that /tmp is on is here: >>>> >>>> Filesystem Size Used Avail Use% Mounted on >>>> /dev/sda1 485G 482G 2.9G 100% / >>>> >>>> however after cleaning up tmp with the crontab command, there is only >>>> 8G usage yet it still remains 100% full: >>>> >>>> @apache-ci-beam-jenkins-8:/tmp$ sudo du -shc /tmp >>>> 8.0G /tmp >>>> 8.0G total >>>> >>>> The workspaces are in the /home/jenkins/jenkins-slave/workspace >>>> directory. When I run a du on that, it takes really long. I'll let it keep >>>> running for a while to see if it ever returns a result but so far this >>>> seems suspect. >>>> >>>> >>>> >>>> >>>> >>>> On Fri, Jul 24, 2020 at 11:19 AM Tyson Hamilton <tyso...@google.com> >>>> wrote: >>>> >>>>> Everything I've been looking at is in the /tmp dir. Where are the >>>>> workspaces, or what are the named? >>>>> >>>>> >>>>> >>>>> >>>>> On Fri, Jul 24, 2020 at 11:03 AM Udi Meiri <eh...@google.com> wrote: >>>>> >>>>>> I'm curious to what you find. Was it /tmp or the workspaces using up >>>>>> the space? >>>>>> >>>>>> On Fri, Jul 24, 2020 at 10:57 AM Tyson Hamilton <tyso...@google.com> >>>>>> wrote: >>>>>> >>>>>>> Bleck. I just realized that it is 'offline' so that won't work. I'll >>>>>>> clean up manually on the machine using the cron command. >>>>>>> >>>>>>> On Fri, Jul 24, 2020 at 10:56 AM Tyson Hamilton <tyso...@google.com> >>>>>>> wrote: >>>>>>> >>>>>>>> Something isn't working with the current set up because node 15 >>>>>>>> appears to be out of space and is currently 'offline' according to >>>>>>>> Jenkins. >>>>>>>> Can someone run the cleanup job? The machine is full, >>>>>>>> >>>>>>>> @apache-ci-beam-jenkins-15:/tmp$ df -h >>>>>>>> Filesystem Size Used Avail Use% Mounted on >>>>>>>> udev 52G 0 52G 0% /dev >>>>>>>> tmpfs 11G 265M 10G 3% /run >>>>>>>> */dev/sda1 485G 484G 880M 100% /* >>>>>>>> tmpfs 52G 0 52G 0% /dev/shm >>>>>>>> tmpfs 5.0M 0 5.0M 0% /run/lock >>>>>>>> tmpfs 52G 0 52G 0% /sys/fs/cgroup >>>>>>>> tmpfs 11G 0 11G 0% /run/user/1017 >>>>>>>> tmpfs 11G 0 11G 0% /run/user/1037 >>>>>>>> >>>>>>>> apache-ci-beam-jenkins-15:/tmp$ sudo du -ah --time . | sort -rhk >>>>>>>> 1,1 | head -n 20 >>>>>>>> 20G 2020-07-24 17:52 . >>>>>>>> 580M 2020-07-22 17:31 ./junit1031982597110125586 >>>>>>>> 517M 2020-07-22 17:31 >>>>>>>> ./junit1031982597110125586/junit8739924829337821410/heap_dump.hprof >>>>>>>> 517M 2020-07-22 17:31 >>>>>>>> ./junit1031982597110125586/junit8739924829337821410 >>>>>>>> 263M 2020-07-22 12:23 ./pip-install-2GUhO_ >>>>>>>> 263M 2020-07-20 09:30 ./pip-install-sxgwqr >>>>>>>> 263M 2020-07-17 13:56 ./pip-install-bWSKIV >>>>>>>> 242M 2020-07-21 20:25 ./beam-pipeline-tempmByU6T >>>>>>>> 242M 2020-07-21 20:21 ./beam-pipeline-tempV85xeK >>>>>>>> 242M 2020-07-21 20:15 ./beam-pipeline-temp7dJROJ >>>>>>>> 236M 2020-07-21 20:25 ./beam-pipeline-tempmByU6T/tmpOWj3Yr >>>>>>>> 236M 2020-07-21 20:21 ./beam-pipeline-tempV85xeK/tmppbQHB3 >>>>>>>> 236M 2020-07-21 20:15 ./beam-pipeline-temp7dJROJ/tmpgOXPKW >>>>>>>> 111M 2020-07-23 00:57 ./pip-install-1JnyNE >>>>>>>> 105M 2020-07-23 00:17 ./beam-artifact1374651823280819755 >>>>>>>> 105M 2020-07-23 00:16 ./beam-artifact5050755582921936972 >>>>>>>> 105M 2020-07-23 00:16 ./beam-artifact1834064452502646289 >>>>>>>> 105M 2020-07-23 00:15 ./beam-artifact682561790267074916 >>>>>>>> 105M 2020-07-23 00:15 ./beam-artifact4691304965824489394 >>>>>>>> 105M 2020-07-23 00:14 ./beam-artifact4050383819822604421 >>>>>>>> >>>>>>>> On Wed, Jul 22, 2020 at 12:03 PM Robert Bradshaw < >>>>>>>> rober...@google.com> wrote: >>>>>>>> >>>>>>>>> On Wed, Jul 22, 2020 at 11:57 AM Tyson Hamilton < >>>>>>>>> tyso...@google.com> wrote: >>>>>>>>> >>>>>>>>>> Ah I see, thanks Kenn. I found some advice from the Apache infra >>>>>>>>>> wiki that also suggests using a tmpdir inside the workspace [1]: >>>>>>>>>> >>>>>>>>>> Procedures Projects can take to clean up disk space >>>>>>>>>> >>>>>>>>>> Projects can help themselves and Infra by taking some basic steps >>>>>>>>>> to help clean up their jobs after themselves on the build nodes. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> 1. Use a ./tmp dir in your jobs workspace. That way it gets >>>>>>>>>> cleaned up when job workspaces expire. >>>>>>>>>> >>>>>>>>>> >>>>>>>>> Tests should be (able to be) written to use the standard temporary >>>>>>>>> file mechanisms, and the environment set up on Jenkins such that that >>>>>>>>> falls >>>>>>>>> into the respective workspaces. Ideally this should be as simple as >>>>>>>>> setting >>>>>>>>> the TMPDIR (or similar) environment variable (and making sure it >>>>>>>>> exists/is >>>>>>>>> writable). >>>>>>>>> >>>>>>>>>> >>>>>>>>>> 1. Configure your jobs to wipe workspaces on start or finish. >>>>>>>>>> 2. Configure your jobs to only keep 5 or 10 previous builds. >>>>>>>>>> 3. Configure your jobs to only keep 5 or 10 previous >>>>>>>>>> artifacts. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> [1]: >>>>>>>>>> https://cwiki.apache.org/confluence/display/INFRA/Disk+Space+cleanup+of+Jenkins+nodes >>>>>>>>>> >>>>>>>>>> On Wed, Jul 22, 2020 at 8:06 AM Kenneth Knowles <k...@apache.org> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> Those file listings look like the result of using standard temp >>>>>>>>>>> file APIs but with TMPDIR set to /tmp. >>>>>>>>>>> >>>>>>>>>>> On Mon, Jul 20, 2020 at 7:55 PM Tyson Hamilton < >>>>>>>>>>> tyso...@google.com> wrote: >>>>>>>>>>> >>>>>>>>>>>> Jobs are hermetic as far as I can tell and use unique >>>>>>>>>>>> subdirectories inside of /tmp. Here is a quick look into two >>>>>>>>>>>> examples: >>>>>>>>>>>> >>>>>>>>>>>> @apache-ci-beam-jenkins-4:/tmp$ sudo du -ah --time . | sort >>>>>>>>>>>> -rhk 1,1 | head -n 20 >>>>>>>>>>>> 1.6G 2020-07-21 02:25 . >>>>>>>>>>>> 242M 2020-07-17 18:48 ./beam-pipeline-temp3ybuY4 >>>>>>>>>>>> 242M 2020-07-17 18:46 ./beam-pipeline-tempuxjiPT >>>>>>>>>>>> 242M 2020-07-17 18:44 ./beam-pipeline-tempVpg1ME >>>>>>>>>>>> 242M 2020-07-17 18:42 ./beam-pipeline-tempJ4EpyB >>>>>>>>>>>> 242M 2020-07-17 18:39 ./beam-pipeline-tempepea7Q >>>>>>>>>>>> 242M 2020-07-17 18:35 ./beam-pipeline-temp79qot2 >>>>>>>>>>>> 236M 2020-07-17 18:48 >>>>>>>>>>>> ./beam-pipeline-temp3ybuY4/tmpy_Ytzz >>>>>>>>>>>> 236M 2020-07-17 18:46 >>>>>>>>>>>> ./beam-pipeline-tempuxjiPT/tmpN5_UfJ >>>>>>>>>>>> 236M 2020-07-17 18:44 >>>>>>>>>>>> ./beam-pipeline-tempVpg1ME/tmpxSm8pX >>>>>>>>>>>> 236M 2020-07-17 18:42 >>>>>>>>>>>> ./beam-pipeline-tempJ4EpyB/tmpMZJU76 >>>>>>>>>>>> 236M 2020-07-17 18:39 >>>>>>>>>>>> ./beam-pipeline-tempepea7Q/tmpWy1vWX >>>>>>>>>>>> 236M 2020-07-17 18:35 >>>>>>>>>>>> ./beam-pipeline-temp79qot2/tmpvN7vWA >>>>>>>>>>>> 3.7M 2020-07-17 18:48 >>>>>>>>>>>> ./beam-pipeline-temp3ybuY4/tmprlh_di >>>>>>>>>>>> 3.7M 2020-07-17 18:46 >>>>>>>>>>>> ./beam-pipeline-tempuxjiPT/tmpLmVWfe >>>>>>>>>>>> 3.7M 2020-07-17 18:44 >>>>>>>>>>>> ./beam-pipeline-tempVpg1ME/tmpvrxbY7 >>>>>>>>>>>> 3.7M 2020-07-17 18:42 >>>>>>>>>>>> ./beam-pipeline-tempJ4EpyB/tmpLTb6Mj >>>>>>>>>>>> 3.7M 2020-07-17 18:39 >>>>>>>>>>>> ./beam-pipeline-tempepea7Q/tmptYF1v1 >>>>>>>>>>>> 3.7M 2020-07-17 18:35 >>>>>>>>>>>> ./beam-pipeline-temp79qot2/tmplfV0Rg >>>>>>>>>>>> 2.7M 2020-07-17 20:10 ./pip-install-q9l227ef >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> @apache-ci-beam-jenkins-11:/tmp$ sudo du -ah --time . | sort >>>>>>>>>>>> -rhk 1,1 | head -n 20 >>>>>>>>>>>> 817M 2020-07-21 02:26 . >>>>>>>>>>>> 242M 2020-07-19 12:14 ./beam-pipeline-tempUTXqlM >>>>>>>>>>>> 242M 2020-07-19 12:11 ./beam-pipeline-tempx3Yno3 >>>>>>>>>>>> 242M 2020-07-19 12:05 ./beam-pipeline-tempyCrMYq >>>>>>>>>>>> 236M 2020-07-19 12:14 >>>>>>>>>>>> ./beam-pipeline-tempUTXqlM/tmpstXoL0 >>>>>>>>>>>> 236M 2020-07-19 12:11 >>>>>>>>>>>> ./beam-pipeline-tempx3Yno3/tmpnnVn65 >>>>>>>>>>>> 236M 2020-07-19 12:05 >>>>>>>>>>>> ./beam-pipeline-tempyCrMYq/tmpRF0iNs >>>>>>>>>>>> 3.7M 2020-07-19 12:14 >>>>>>>>>>>> ./beam-pipeline-tempUTXqlM/tmpbJjUAQ >>>>>>>>>>>> 3.7M 2020-07-19 12:11 >>>>>>>>>>>> ./beam-pipeline-tempx3Yno3/tmpsmmzqe >>>>>>>>>>>> 3.7M 2020-07-19 12:05 >>>>>>>>>>>> ./beam-pipeline-tempyCrMYq/tmp5b3ZvY >>>>>>>>>>>> 2.0M 2020-07-19 12:14 >>>>>>>>>>>> ./beam-pipeline-tempUTXqlM/tmpoj3orz >>>>>>>>>>>> 2.0M 2020-07-19 12:11 >>>>>>>>>>>> ./beam-pipeline-tempx3Yno3/tmptng9sZ >>>>>>>>>>>> 2.0M 2020-07-19 12:05 >>>>>>>>>>>> ./beam-pipeline-tempyCrMYq/tmpWp6njc >>>>>>>>>>>> 1.2M 2020-07-19 12:14 >>>>>>>>>>>> ./beam-pipeline-tempUTXqlM/tmphgdj35 >>>>>>>>>>>> 1.2M 2020-07-19 12:11 >>>>>>>>>>>> ./beam-pipeline-tempx3Yno3/tmp8ySXpm >>>>>>>>>>>> 1.2M 2020-07-19 12:05 >>>>>>>>>>>> ./beam-pipeline-tempyCrMYq/tmpNVEJ4e >>>>>>>>>>>> 992K 2020-07-12 12:00 ./junit642086915811430564 >>>>>>>>>>>> 988K 2020-07-12 12:00 ./junit642086915811430564/beam >>>>>>>>>>>> 984K 2020-07-12 12:00 >>>>>>>>>>>> ./junit642086915811430564/beam/nodes >>>>>>>>>>>> 980K 2020-07-12 12:00 >>>>>>>>>>>> ./junit642086915811430564/beam/nodes/0 >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Mon, Jul 20, 2020 at 6:46 PM Udi Meiri <eh...@google.com> >>>>>>>>>>>> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> You're right, job workspaces should be hermetic. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> On Mon, Jul 20, 2020 at 1:24 PM Kenneth Knowles < >>>>>>>>>>>>> k...@apache.org> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> I'm probably late to this discussion and missing something, >>>>>>>>>>>>>> but why are we writing to /tmp at all? I would expect TMPDIR to >>>>>>>>>>>>>> point >>>>>>>>>>>>>> somewhere inside the job directory that will be wiped by >>>>>>>>>>>>>> Jenkins, and I >>>>>>>>>>>>>> would expect code to always create temp files via APIs that >>>>>>>>>>>>>> respect this. >>>>>>>>>>>>>> Is Jenkins not cleaning up? Do we not have the ability to set >>>>>>>>>>>>>> this up? Do >>>>>>>>>>>>>> we have bugs in our code (that we could probably find by setting >>>>>>>>>>>>>> TMPDIR to >>>>>>>>>>>>>> somewhere not-/tmp and running the tests without write >>>>>>>>>>>>>> permission to /tmp, >>>>>>>>>>>>>> etc) >>>>>>>>>>>>>> >>>>>>>>>>>>>> Kenn >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Mon, Jul 20, 2020 at 11:39 AM Ahmet Altay < >>>>>>>>>>>>>> al...@google.com> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Related to workspace directory growth, +Udi Meiri >>>>>>>>>>>>>>> <eh...@google.com> filed a relevant issue previously ( >>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/BEAM-9865) for >>>>>>>>>>>>>>> cleaning up workspace directory after successful jobs. >>>>>>>>>>>>>>> Alternatively, we >>>>>>>>>>>>>>> can consider periodically cleaning up the /src directories. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I would suggest moving the cron task from internal cron >>>>>>>>>>>>>>> scripts to the inventory job ( >>>>>>>>>>>>>>> https://github.com/apache/beam/blob/master/.test-infra/jenkins/job_Inventory.groovy#L51). >>>>>>>>>>>>>>> That way, we can see all the cron jobs as part of the source >>>>>>>>>>>>>>> tree, adjust >>>>>>>>>>>>>>> frequencies and clean up codes with PRs. I do not know how >>>>>>>>>>>>>>> internal cron >>>>>>>>>>>>>>> scripts are created, maintained, and how would they be >>>>>>>>>>>>>>> recreated for new >>>>>>>>>>>>>>> worker instances. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> /cc +Tyson Hamilton <tyso...@google.com> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Mon, Jul 20, 2020 at 4:50 AM Damian Gadomski < >>>>>>>>>>>>>>> damian.gadom...@polidea.com> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Hey, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> I've recently created a solution for the growing /tmp >>>>>>>>>>>>>>>> directory. Part of it is the job mentioned by Tyson: >>>>>>>>>>>>>>>> *beam_Clean_tmp_directory*. It's intentionally not >>>>>>>>>>>>>>>> triggered by cron and should be a last resort solution for >>>>>>>>>>>>>>>> some strange >>>>>>>>>>>>>>>> cases. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Along with that job, I've also updated every worker with an >>>>>>>>>>>>>>>> internal cron script. It's being executed once a week and >>>>>>>>>>>>>>>> deletes all the >>>>>>>>>>>>>>>> files (and only files) that were not accessed for at least >>>>>>>>>>>>>>>> three days. >>>>>>>>>>>>>>>> That's designed to be as safe as possible for the running jobs >>>>>>>>>>>>>>>> on the >>>>>>>>>>>>>>>> worker (not to delete the files that are still in use), and >>>>>>>>>>>>>>>> also to be >>>>>>>>>>>>>>>> insensitive to the current workload on the machine. The >>>>>>>>>>>>>>>> cleanup will always >>>>>>>>>>>>>>>> happen, even if some long-running/stuck jobs are blocking the >>>>>>>>>>>>>>>> machine. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> I also think that currently the "No space left" errors may >>>>>>>>>>>>>>>> be a consequence of growing workspace directory rather than >>>>>>>>>>>>>>>> /tmp. I didn't >>>>>>>>>>>>>>>> do any detailed analysis but e.g. currently, on >>>>>>>>>>>>>>>> apache-beam-jenkins-7 the >>>>>>>>>>>>>>>> workspace directory size is 158 GB while /tmp is only 16 GB. >>>>>>>>>>>>>>>> We should >>>>>>>>>>>>>>>> either guarantee the disk size to hold workspaces for all jobs >>>>>>>>>>>>>>>> (because >>>>>>>>>>>>>>>> eventually, every worker will execute each job) or clear also >>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>> workspaces in some way. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Regards, >>>>>>>>>>>>>>>> Damian >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Mon, Jul 20, 2020 at 10:43 AM Maximilian Michels < >>>>>>>>>>>>>>>> m...@apache.org> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> +1 for scheduling it via a cron job if it won't lead to >>>>>>>>>>>>>>>>> test failures >>>>>>>>>>>>>>>>> while running. Not a Jenkins expert but maybe there is the >>>>>>>>>>>>>>>>> notion of >>>>>>>>>>>>>>>>> running exclusively while no other tasks are running? >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> -Max >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On 17.07.20 21:49, Tyson Hamilton wrote: >>>>>>>>>>>>>>>>> > FYI there was a job introduced to do this in Jenkins: >>>>>>>>>>>>>>>>> beam_Clean_tmp_directory >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > Currently it needs to be run manually. I'm seeing some >>>>>>>>>>>>>>>>> out of disk related errors in precommit tests currently, >>>>>>>>>>>>>>>>> perhaps we should >>>>>>>>>>>>>>>>> schedule this job with cron? >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > On 2020/03/11 19:31:13, Heejong Lee <heej...@google.com> >>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>> >> Still seeing no space left on device errors on >>>>>>>>>>>>>>>>> jenkins-7 (for example: >>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>> https://builds.apache.org/job/beam_PreCommit_PythonLint_Commit/2754/ >>>>>>>>>>>>>>>>> ) >>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>> >> On Fri, Mar 6, 2020 at 7:11 PM Alan Myrvold < >>>>>>>>>>>>>>>>> amyrv...@google.com> wrote: >>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>> >>> Did a one time cleanup of tmp files owned by jenkins >>>>>>>>>>>>>>>>> older than 3 days. >>>>>>>>>>>>>>>>> >>> Agree that we need a longer term solution. >>>>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>>>> >>> Passing recent tests on all executors except >>>>>>>>>>>>>>>>> jenkins-12, which has not >>>>>>>>>>>>>>>>> >>> scheduled recent builds for the past 13 days. Not >>>>>>>>>>>>>>>>> scheduling: >>>>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>>>> https://builds.apache.org/computer/apache-beam-jenkins-12/builds >>>>>>>>>>>>>>>>> >>> < >>>>>>>>>>>>>>>>> https://www.google.com/url?q=https://builds.apache.org/computer/apache-beam-jenkins-12/builds&sa=D >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> >>> Recent passing builds: >>>>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>>>> https://builds.apache.org/computer/apache-beam-jenkins-1/builds >>>>>>>>>>>>>>>>> >>> < >>>>>>>>>>>>>>>>> https://www.google.com/url?q=https://builds.apache.org/computer/apache-beam-jenkins-1/builds&sa=D >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>>>> https://builds.apache.org/computer/apache-beam-jenkins-2/builds >>>>>>>>>>>>>>>>> >>> < >>>>>>>>>>>>>>>>> https://www.google.com/url?q=https://builds.apache.org/computer/apache-beam-jenkins-2/builds&sa=D >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>>>> https://builds.apache.org/computer/apache-beam-jenkins-3/builds >>>>>>>>>>>>>>>>> >>> < >>>>>>>>>>>>>>>>> https://www.google.com/url?q=https://builds.apache.org/computer/apache-beam-jenkins-3/builds&sa=D >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>>>> https://builds.apache.org/computer/apache-beam-jenkins-4/builds >>>>>>>>>>>>>>>>> >>> < >>>>>>>>>>>>>>>>> https://www.google.com/url?q=https://builds.apache.org/computer/apache-beam-jenkins-4/builds&sa=D >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>>>> https://builds.apache.org/computer/apache-beam-jenkins-5/builds >>>>>>>>>>>>>>>>> >>> < >>>>>>>>>>>>>>>>> https://www.google.com/url?q=https://builds.apache.org/computer/apache-beam-jenkins-5/builds&sa=D >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>>>> https://builds.apache.org/computer/apache-beam-jenkins-6/builds >>>>>>>>>>>>>>>>> >>> < >>>>>>>>>>>>>>>>> https://www.google.com/url?q=https://builds.apache.org/computer/apache-beam-jenkins-6/builds&sa=D >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>>>> https://builds.apache.org/computer/apache-beam-jenkins-7/builds >>>>>>>>>>>>>>>>> >>> < >>>>>>>>>>>>>>>>> https://www.google.com/url?q=https://builds.apache.org/computer/apache-beam-jenkins-7/builds&sa=D >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>>>> https://builds.apache.org/computer/apache-beam-jenkins-8/builds >>>>>>>>>>>>>>>>> >>> < >>>>>>>>>>>>>>>>> https://www.google.com/url?q=https://builds.apache.org/computer/apache-beam-jenkins-8/builds&sa=D >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>>>> https://builds.apache.org/computer/apache-beam-jenkins-9/builds >>>>>>>>>>>>>>>>> >>> < >>>>>>>>>>>>>>>>> https://www.google.com/url?q=https://builds.apache.org/computer/apache-beam-jenkins-9/builds&sa=D >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>>>> https://builds.apache.org/computer/apache-beam-jenkins-10/builds >>>>>>>>>>>>>>>>> >>> < >>>>>>>>>>>>>>>>> https://www.google.com/url?q=https://builds.apache.org/computer/apache-beam-jenkins-10/builds&sa=D >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>>>> https://builds.apache.org/computer/apache-beam-jenkins-11/builds >>>>>>>>>>>>>>>>> >>> < >>>>>>>>>>>>>>>>> https://www.google.com/url?q=https://builds.apache.org/computer/apache-beam-jenkins-11/builds&sa=D >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>>>> https://builds.apache.org/computer/apache-beam-jenkins-13/builds >>>>>>>>>>>>>>>>> >>> < >>>>>>>>>>>>>>>>> https://www.google.com/url?q=https://builds.apache.org/computer/apache-beam-jenkins-13/builds&sa=D >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>>>> https://builds.apache.org/computer/apache-beam-jenkins-14/builds >>>>>>>>>>>>>>>>> >>> < >>>>>>>>>>>>>>>>> https://www.google.com/url?q=https://builds.apache.org/computer/apache-beam-jenkins-14/builds&sa=D >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>>>> https://builds.apache.org/computer/apache-beam-jenkins-15/builds >>>>>>>>>>>>>>>>> >>> < >>>>>>>>>>>>>>>>> https://www.google.com/url?q=https://builds.apache.org/computer/apache-beam-jenkins-15/builds&sa=D >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>>>> https://builds.apache.org/computer/apache-beam-jenkins-16/builds >>>>>>>>>>>>>>>>> >>> < >>>>>>>>>>>>>>>>> https://www.google.com/url?q=https://builds.apache.org/computer/apache-beam-jenkins-16/builds&sa=D >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>>>> >>> On Fri, Mar 6, 2020 at 11:54 AM Ahmet Altay < >>>>>>>>>>>>>>>>> al...@google.com> wrote: >>>>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>>>> >>>> +Alan Myrvold <amyrv...@google.com> is doing a one >>>>>>>>>>>>>>>>> time cleanup. I agree >>>>>>>>>>>>>>>>> >>>> that we need to have a solution to automate this task >>>>>>>>>>>>>>>>> or address the root >>>>>>>>>>>>>>>>> >>>> cause of the buildup. >>>>>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>>>>> >>>> On Thu, Mar 5, 2020 at 2:47 AM Michał Walenia < >>>>>>>>>>>>>>>>> michal.wale...@polidea.com> >>>>>>>>>>>>>>>>> >>>> wrote: >>>>>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>>>>> >>>>> Hi there, >>>>>>>>>>>>>>>>> >>>>> it seems we have a problem with Jenkins workers >>>>>>>>>>>>>>>>> again. Nodes 1 and 7 >>>>>>>>>>>>>>>>> >>>>> both fail jobs with "No space left on device". >>>>>>>>>>>>>>>>> >>>>> Who is the best person to contact in these cases >>>>>>>>>>>>>>>>> (someone with access >>>>>>>>>>>>>>>>> >>>>> permissions to the workers). >>>>>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>>>>> >>>>> I also noticed that such errors are becoming more >>>>>>>>>>>>>>>>> and more frequent >>>>>>>>>>>>>>>>> >>>>> recently and I'd like to discuss how can this be >>>>>>>>>>>>>>>>> remedied. Can a cleanup >>>>>>>>>>>>>>>>> >>>>> task be automated on Jenkins somehow? >>>>>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>>>>> >>>>> Regards >>>>>>>>>>>>>>>>> >>>>> Michal >>>>>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>>>>> >>>>> -- >>>>>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>>>>> >>>>> Michał Walenia >>>>>>>>>>>>>>>>> >>>>> Polidea <https://www.polidea.com/> | Software >>>>>>>>>>>>>>>>> Engineer >>>>>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>>>>> >>>>> M: +48 791 432 002 <+48%20791%20432%20002> < >>>>>>>>>>>>>>>>> +48791432002 <+48%20791%20432%20002>> >>>>>>>>>>>>>>>>> >>>>> E: michal.wale...@polidea.com >>>>>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>>>>> >>>>> Unique Tech >>>>>>>>>>>>>>>>> >>>>> Check out our projects! < >>>>>>>>>>>>>>>>> https://www.polidea.com/our-work> >>>>>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>
smime.p7s
Description: S/MIME Cryptographic Signature
