One thing you could try, to see if it helps, is Disable Deferred Wipeout - this will cause a delay in your job completing, but it means the workspace cleanup happens before the job carries on, rather than being offloaded to a thread to do later, something which seems to be not working properly at the moment.
It's under: Build Environment, Delete workspace before build starts, Advanced: Disable deferred wipeout On lucene 3, there are several ws-cleanup directories for Solr-Test-main, so that would seem a good job to try this out on. If it makes the job fail, we will hopefully get info on why, if not, we can see what sort of delay is incurred. The explanation from Gemin Gen AI: "In a Freestyle job, when the "deferred wipeout" feature is active, Jenkins instantly renames your workspace directory to workspace_ws-cleanup_[timestamp] and tells the build that it's finished. It then delegates the actual deletion of that directory to a separate Java thread in the background. By checking Disable deferred wipeout, you are telling Jenkins: "Do not use background threads. Hold up the build execution for an extra second, delete the files synchronously right now, and only mark the build complete once they are 100% gone."" On 2026/07/14 12:49:48 David Smiley wrote: > I modified one build, > https://ci-builds.apache.org/job/Solr/job/Solr-Maven-Snapshots-main/ to > have an additional shell execution step with that snippet, which performs > the cleanup via Docker as you suggested, albeit not with the Jenkins/groovy > wrapper all around it. It ran successfully this morning. I'm not sure why > Docker is involved for what looks like a simple file removal; might you > explain? > > On Mon, Jul 6, 2026 at 8:41 AM Bob Thomson <[email protected]> wrote: > > > Hi, > > > > In Infra, we don't work on job definitions as that is a project's domain. > > The above is only a suggestion which may or may not help. > > If it is easier, you could look at adding similar code to the jobs as a > > shell script post build step in order to remove the files. Aurélien Pupier > > of the Camel project has been looking at the same issue there, so worth > > having a chat with them about potential solutions. > > > > In terms of the jobs which are leaving ws-cleanup directories behind, it > > seems to be many of them. On lucene 3 today, there are already 113 such > > directories, 213GB so far. > > > > the jobs that they are associated with, from per their directory names are: > > > > OWASP-9.x > > OWASP-main > > Solr-Artifacts-10.x > > Solr-Artifacts-9.11 > > Solr-Artifacts-9.x > > Solr-Artifacts-main > > Solr-BadApples-Tests-main > > Solr-Docker-Nightly-9.11 > > Solr-Lint-10.x > > Solr-Lint-9.11 > > Solr-Lint-9.x > > Solr-Lint-main > > Solr-Maven-Snapshots-main > > Solr-Test-10.0 > > Solr-Test-10.x > > Solr-Test-9.11 > > Solr-Test-9.x > > Solr-Test-main > > Solr-TestIntegration-10.x > > Solr-TestIntegration-9.11 > > Solr-TestIntegration-9.x > > Solr-TestIntegration-main > > solr-desktop-ui-nightly > > > > On 4, there are 80 from these builds: > > > > OWASP-9.x > > OWASP-main > > Solr-Artifacts-10.x > > Solr-Artifacts-9.11 > > Solr-Artifacts-9.x > > Solr-Docker-Nightly-9.11 > > Solr-Lint-10.x > > Solr-Lint-9.11 > > Solr-Lint-9.x > > Solr-Lint-main > > Solr-Maven-Snapshots-main > > Solr-Test-10.0 > > Solr-Test-10.x > > Solr-Test-9.11 > > Solr-Test-9.x > > Solr-Test-main > > Solr-TestIntegration-9.11 > > Solr-TestIntegration-9.x > > > > On 2026/07/03 20:32:09 David Smiley wrote: > > > I know too little about Jenkins to know how to apply this solution, but I > > > can try to help. The Solr jobs are defined in Jenkins, not in our repo > > in > > > Jenkinsfile. Maybe you could apply this to one of the jobs yourself? > > Does > > > this need to be done on all jobs or just some or will just one do? > > > > > > On Fri, Jul 3, 2026 at 8:20 AM Bob Thomson <[email protected]> wrote: > > > > > > > This has come up as an issue again today, necessitating intervention to > > > > keep the service available. > > > > > > > > On 2026/06/29 07:04:23 Robert Thomson wrote: > > > > > Hi, > > > > > > > > > > We are regularly noticing a build-up of large ws-cleanup directories > > > > > on lucene3 and 4, necessitating deletion. Camel also experiences this > > > > > issue with builds and the root cause is that there are files created > > > > > as part of the docker build that are not owned by the user used for > > > > > the cleanup, so fail to be deleted by the workspace clean up plugin. > > > > > > > > > > Searching with GenAI came up with the following suggestion when > > > > > discussing with them that you may like to try out in your pipelines > > > > > that create docker images. > > > > > > > > > > Kind regards, > > > > > -Bob Thomson, > > > > > ASF Infrastructure > > > > > > > > > > post { > > > > > always { > > > > > // Fallback option: If standard cleanWs() stalls out on > > > > permissions > > > > > script { > > > > > try { > > > > > cleanWs deleteDirs: true, disableDeferredWipeout: > > true > > > > > } catch (Exception e) { > > > > > echo “Standard cleanup failed, launching Docker Root > > > > > Purge force-override...” > > > > > sh “”" > > > > > docker run --rm \ > > > > > -v ${WORKSPACE}/..:/zap \ > > > > > alpine:latest \ > > > > > sh -c ‘rm -rf /zap/*-ws-cleanup*’ > > > > > “”" > > > > > } > > > > > } > > > > > } > > > > > } > > > > > > > > > > --------------------------------------------------------------------- > > > > > To unsubscribe, e-mail: [email protected] > > > > > For additional commands, e-mail: [email protected] > > > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > > To unsubscribe, e-mail: [email protected] > > > > For additional commands, e-mail: [email protected] > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: [email protected] > > For additional commands, e-mail: [email protected] > > > > > --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
