Having looked through the ws-cleanup dirs currently on lucene3 and 4, it does not look to be a permissions issue, as those dirs and all files in them are owned by jenkins, so it's a different issue in this case - further investigation needed.
On 2026/07/14 12:56:59 Bob Thomson wrote: > Like I say, not my code as such, a Gen AI suggestion, but I would say it's an > easy way for the process running as the jenkins user, to remove files which > are not owned by the jenkins user, which seems to often be the reason for > these ws-cleanup directories not being removed (Often in the Camel project > case anyway). The other reason might be a restart of the server for example, > which doesn't happen on lucene but sometimes does happen on some other CI > servers we have due to Jenkins locking up (GC/RAM usage often). > > On 2026/07/14 12:49:48 David Smiley wrote: > > I modified one build, > > https://ci-builds.apache.org/job/Solr/job/Solr-Maven-Snapshots-main/ to > > have an additional shell execution step with that snippet, which performs > > the cleanup via Docker as you suggested, albeit not with the Jenkins/groovy > > wrapper all around it. It ran successfully this morning. I'm not sure why > > Docker is involved for what looks like a simple file removal; might you > > explain? > > > > On Mon, Jul 6, 2026 at 8:41 AM Bob Thomson <[email protected]> wrote: > > > > > Hi, > > > > > > In Infra, we don't work on job definitions as that is a project's domain. > > > The above is only a suggestion which may or may not help. > > > If it is easier, you could look at adding similar code to the jobs as a > > > shell script post build step in order to remove the files. Aurélien Pupier > > > of the Camel project has been looking at the same issue there, so worth > > > having a chat with them about potential solutions. > > > > > > In terms of the jobs which are leaving ws-cleanup directories behind, it > > > seems to be many of them. On lucene 3 today, there are already 113 such > > > directories, 213GB so far. > > > > > > the jobs that they are associated with, from per their directory names > > > are: > > > > > > OWASP-9.x > > > OWASP-main > > > Solr-Artifacts-10.x > > > Solr-Artifacts-9.11 > > > Solr-Artifacts-9.x > > > Solr-Artifacts-main > > > Solr-BadApples-Tests-main > > > Solr-Docker-Nightly-9.11 > > > Solr-Lint-10.x > > > Solr-Lint-9.11 > > > Solr-Lint-9.x > > > Solr-Lint-main > > > Solr-Maven-Snapshots-main > > > Solr-Test-10.0 > > > Solr-Test-10.x > > > Solr-Test-9.11 > > > Solr-Test-9.x > > > Solr-Test-main > > > Solr-TestIntegration-10.x > > > Solr-TestIntegration-9.11 > > > Solr-TestIntegration-9.x > > > Solr-TestIntegration-main > > > solr-desktop-ui-nightly > > > > > > On 4, there are 80 from these builds: > > > > > > OWASP-9.x > > > OWASP-main > > > Solr-Artifacts-10.x > > > Solr-Artifacts-9.11 > > > Solr-Artifacts-9.x > > > Solr-Docker-Nightly-9.11 > > > Solr-Lint-10.x > > > Solr-Lint-9.11 > > > Solr-Lint-9.x > > > Solr-Lint-main > > > Solr-Maven-Snapshots-main > > > Solr-Test-10.0 > > > Solr-Test-10.x > > > Solr-Test-9.11 > > > Solr-Test-9.x > > > Solr-Test-main > > > Solr-TestIntegration-9.11 > > > Solr-TestIntegration-9.x > > > > > > On 2026/07/03 20:32:09 David Smiley wrote: > > > > I know too little about Jenkins to know how to apply this solution, but > > > > I > > > > can try to help. The Solr jobs are defined in Jenkins, not in our repo > > > in > > > > Jenkinsfile. Maybe you could apply this to one of the jobs yourself? > > > Does > > > > this need to be done on all jobs or just some or will just one do? > > > > > > > > On Fri, Jul 3, 2026 at 8:20 AM Bob Thomson <[email protected]> wrote: > > > > > > > > > This has come up as an issue again today, necessitating intervention > > > > > to > > > > > keep the service available. > > > > > > > > > > On 2026/06/29 07:04:23 Robert Thomson wrote: > > > > > > Hi, > > > > > > > > > > > > We are regularly noticing a build-up of large ws-cleanup directories > > > > > > on lucene3 and 4, necessitating deletion. Camel also experiences > > > > > > this > > > > > > issue with builds and the root cause is that there are files created > > > > > > as part of the docker build that are not owned by the user used for > > > > > > the cleanup, so fail to be deleted by the workspace clean up plugin. > > > > > > > > > > > > Searching with GenAI came up with the following suggestion when > > > > > > discussing with them that you may like to try out in your pipelines > > > > > > that create docker images. > > > > > > > > > > > > Kind regards, > > > > > > -Bob Thomson, > > > > > > ASF Infrastructure > > > > > > > > > > > > post { > > > > > > always { > > > > > > // Fallback option: If standard cleanWs() stalls out on > > > > > permissions > > > > > > script { > > > > > > try { > > > > > > cleanWs deleteDirs: true, disableDeferredWipeout: > > > true > > > > > > } catch (Exception e) { > > > > > > echo “Standard cleanup failed, launching Docker Root > > > > > > Purge force-override...” > > > > > > sh “”" > > > > > > docker run --rm \ > > > > > > -v ${WORKSPACE}/..:/zap \ > > > > > > alpine:latest \ > > > > > > sh -c ‘rm -rf /zap/*-ws-cleanup*’ > > > > > > “”" > > > > > > } > > > > > > } > > > > > > } > > > > > > } > > > > > > > > > > > > --------------------------------------------------------------------- > > > > > > To unsubscribe, e-mail: [email protected] > > > > > > For additional commands, e-mail: [email protected] > > > > > > > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > > > To unsubscribe, e-mail: [email protected] > > > > > For additional commands, e-mail: [email protected] > > > > > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > To unsubscribe, e-mail: [email protected] > > > For additional commands, e-mail: [email protected] > > > > > > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > > --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
