Is the issue that the workspace grows over time? Couldn't we delete it daily to ensure it does not grow too much? Always deleting it on successful runs may be too costly because we have to recreate the workspace every time.
Logs are stored separately. I suppose they could also add up over time. On 30.04.20 21:48, Hannah Jiang wrote: > Indeed, I can see the no space left on device in the following but > not in the log above: > > --info tag was passed to docker image build commands with PythonDocker > Precommit to capture more logs. Without the tag, errors from DockerFile > step are not printed out to the console. > > On Thu, Apr 30, 2020 at 11:19 AM Udi Meiri <eh...@google.com > <mailto:eh...@google.com>> wrote: > > I checked node 8 and it had over 40GB space available. Does your job > require more than that? > > Long term, I'm thinking we could clean up workspaces for successful > jobs. This should free up additional space (I guess at least 100GB). > https://plugins.jenkins.io/ws-cleanup/ - we already use this plugin > to clean workspaces at job start. > > > On Thu, Apr 30, 2020, 07:33 Maximilian Michels <m...@apache.org > <mailto:m...@apache.org>> wrote: > > *It's working again, probably because it's running on a different > machine now. > > Who can check the disk space of the Jenkins hosts? > > Thanks, > Max > > On 30.04.20 11:55, Maximilian Michels wrote: > > Sorry, I meant to include the Jenkins log: > > > > https://builds.apache.org/job/beam_LoadTests_Python_ParDo_Flink_Streaming_PR/5/console > > > > Thanks for investigating Hannah! Indeed, I can see the no > space left on > > device in the following but not in the log above: > > > > https://builds.apache.org/job/beam_PreCommit_PythonDocker_Commit/473/console > > > > I'm going to try running the build again. Do you think we > could add more > > storage to our Jenkins hosts or delete old build data? > > > > Thanks, > > Max > > > > On 30.04.20 08:43, Hannah Jiang wrote: > >> Max, I found a link from your PR and noticed below errors. > This would be > >> the true error. > >> > >> *07:57:03* >*Task :sdks:python:container:py37:docker* > >> *07:57:03* [91mERROR: Could not install packages due to an > EnvironmentError: [Errno 28] No space left on device > >> *07:57:03* > >> *07:57:03* [0m > >> *07:57:03* >*Task :sdks:python:container:py35:docker* > >> *07:57:03* [91mERROR: Could not install packages due to an > EnvironmentError: [Errno 28] No space left on device > >> > >> > >> > >> On Wed, Apr 29, 2020 at 5:59 PM Hannah Jiang > <hannahji...@google.com <mailto:hannahji...@google.com> > >> <mailto:hannahji...@google.com > <mailto:hannahji...@google.com>>> wrote: > >> > >> There is a PythonDocker Precommit test running for PRs > with Python > >> changes. It seems running well.[1] > >> Max, can you please give me a link so I can check more > details? Do > >> other images with different Python versions fail as well? > >> > >> > 1. https://builds.apache.org/job/beam_PreCommit_PythonDocker_Commit/ > >> > >> > >> On Wed, Apr 29, 2020 at 2:44 PM Ahmet Altay > <al...@google.com <mailto:al...@google.com> > >> <mailto:al...@google.com <mailto:al...@google.com>>> wrote: > >> > >> +Valentyn Tymofieiev <mailto:valen...@google.com > <mailto:valen...@google.com>> +Hannah Jiang > >> <mailto:hannahji...@google.com > <mailto:hannahji...@google.com>> -- in case they have relevant > >> information. > >> > >> On Wed, Apr 29, 2020 at 12:35 PM Maximilian Michels > >> <m...@apache.org <mailto:m...@apache.org> > <mailto:m...@apache.org <mailto:m...@apache.org>>> wrote: > >> > >> Hi, > >> > >> has anyone noticed the Python 3.7 Docker > container fails to > >> build? I > >> haven't been able to build the Python 3.7 > container, neither > >> locally nor > >> on Jenkins. > >> > >> I get: > >> > >> 17:48:10 > Task :sdks:python:container:py37:docker > >> 17:49:36 The command '/bin/sh -c pip install -r > >> /tmp/base_image_requirements.txt && python -c "from > >> google.protobuf.internal import > api_implementation; assert > >> api_implementation._default_implementation_type > == 'cpp'; print > >> ('Verified fast protobuf used.')" && rm -rf > >> /root/.cache/pip' returned a > >> non-zero code: 1 > >> 17:49:36 > >> 17:49:36 > Task > :sdks:python:container:py37:docker FAILED > >> > >> > >> Cheers, > >> Max > >> >
