The cleanup setting on the job only applies to that job config. I think it can be set more globally, but not looked into the details as yet.
Absolutely no ws-cleanup dirs on lucene3, so looks great! On lucene4 there are a few: solr-desktop-ui-nightly_ws-cleanup_1784131113250 Solr-TestIntegration-9.11_ws-cleanup_1784154127477 Solr-Lint-10.x_ws-cleanup_1784162700403 Solr-Artifacts-9.11_ws-cleanup_1784170570055 Solr-Test-9.11_ws-cleanup_1784113968538 Solr-Test-9.x_ws-cleanup_1784118593979 Solr-Test-10.x_ws-cleanup_1784121394984 Solr-Test-10.0_ws-cleanup_1784125677574 Solr-Test-10.0_ws-cleanup_1784149989136 Solr-Test-10.0_ws-cleanup_1784172720758 Solr-Test-10.0_ws-cleanup_1784182295683 Kind regards, -Bob Thomson, ASF Infrastructure On 2026/07/16 01:23:48 David Smiley wrote: > ok cool. I enabled this flag on several more jobs -- all the bulk test > ones for the different Solr non-release branches. I assume that cleanup > needn't be tied to any particular job so much as the cleanup needs to > happen enough that it executes on the 2 build machines. > > On Wed, Jul 15, 2026 at 6:52 AM Bob Thomson <[email protected]> wrote: > > > That one claims to have cleared the workspace and it didn't seem to affect > > the job duration at all either which is good: > > > > [WS-CLEANUP] Deleting project workspace... > > [WS-CLEANUP] Deferred wipeout is disabled by the job configuration... > > [WS-CLEANUP] Done > > > > and that worked, as I don't see a ws-cleanup for Solr-Test-main after this > > one: > > > > Jul 14 02:12 Solr-Test-main_ws-cleanup_1784004182558 > > > > I'll remove all the other, olrder Solr-Test-main_ws-cleanup_1* dirs on > > lucene3/4, and we can monitor and confirm no new ones get created/left. If > > so, that seems a valid setting for all of the jobs that run on lucene3/4. > > We can then see if it's possible to enable it at a higher level so all the > > jobs run that way, provided you don't see any longer duration on job run > > time due to this. > > > > Kind regards, > > -Bob Thomson, > > ASF Infrastructure > > > > On 2026/07/15 01:09:34 David Smiley wrote: > > > I checked "Disable deferred wipeout" and scheduled a build. It ran a few > > > hours ago on Lucene-3 > > > https://ci-builds.apache.org/job/Solr/job/Solr-Test-main/14288/ > > > > > > On Tue, Jul 14, 2026 at 9:55 AM Bob Thomson <[email protected]> wrote: > > > > > > > One thing you could try, to see if it helps, is Disable Deferred > > Wipeout - > > > > this will cause a delay in your job completing, but it means the > > workspace > > > > cleanup happens before the job carries on, rather than being offloaded > > to a > > > > thread to do later, something which seems to be not working properly > > at the > > > > moment. > > > > > > > > It's under: > > > > Build Environment, Delete workspace before build starts, Advanced: > > > > > > > > Disable deferred wipeout > > > > > > > > On lucene 3, there are several ws-cleanup directories for > > Solr-Test-main, > > > > so that would seem a good job to try this out on. If it makes the job > > fail, > > > > we will hopefully get info on why, if not, we can see what sort of > > delay is > > > > incurred. > > > > > > > > The explanation from Gemin Gen AI: > > > > "In a Freestyle job, when the "deferred wipeout" feature is active, > > > > Jenkins instantly renames your workspace directory to > > > > workspace_ws-cleanup_[timestamp] and tells the build that it's > > finished. It > > > > then delegates the actual deletion of that directory to a separate Java > > > > thread in the background. > > > > > > > > By checking Disable deferred wipeout, you are telling Jenkins: "Do not > > use > > > > background threads. Hold up the build execution for an extra second, > > delete > > > > the files synchronously right now, and only mark the build complete > > once > > > > they are 100% gone."" > > > > > > > > > > > > On 2026/07/14 12:49:48 David Smiley wrote: > > > > > I modified one build, > > > > > https://ci-builds.apache.org/job/Solr/job/Solr-Maven-Snapshots-main/ > > to > > > > > have an additional shell execution step with that snippet, which > > performs > > > > > the cleanup via Docker as you suggested, albeit not with the > > > > Jenkins/groovy > > > > > wrapper all around it. It ran successfully this morning. I'm not > > sure > > > > why > > > > > Docker is involved for what looks like a simple file removal; might > > you > > > > > explain? > > > > > > > > > > On Mon, Jul 6, 2026 at 8:41 AM Bob Thomson <[email protected]> > > wrote: > > > > > > > > > > > Hi, > > > > > > > > > > > > In Infra, we don't work on job definitions as that is a project's > > > > domain. > > > > > > The above is only a suggestion which may or may not help. > > > > > > If it is easier, you could look at adding similar code to the jobs > > as a > > > > > > shell script post build step in order to remove the files. Aurélien > > > > Pupier > > > > > > of the Camel project has been looking at the same issue there, so > > worth > > > > > > having a chat with them about potential solutions. > > > > > > > > > > > > In terms of the jobs which are leaving ws-cleanup directories > > behind, > > > > it > > > > > > seems to be many of them. On lucene 3 today, there are already 113 > > such > > > > > > directories, 213GB so far. > > > > > > > > > > > > the jobs that they are associated with, from per their directory > > names > > > > are: > > > > > > > > > > > > OWASP-9.x > > > > > > OWASP-main > > > > > > Solr-Artifacts-10.x > > > > > > Solr-Artifacts-9.11 > > > > > > Solr-Artifacts-9.x > > > > > > Solr-Artifacts-main > > > > > > Solr-BadApples-Tests-main > > > > > > Solr-Docker-Nightly-9.11 > > > > > > Solr-Lint-10.x > > > > > > Solr-Lint-9.11 > > > > > > Solr-Lint-9.x > > > > > > Solr-Lint-main > > > > > > Solr-Maven-Snapshots-main > > > > > > Solr-Test-10.0 > > > > > > Solr-Test-10.x > > > > > > Solr-Test-9.11 > > > > > > Solr-Test-9.x > > > > > > Solr-Test-main > > > > > > Solr-TestIntegration-10.x > > > > > > Solr-TestIntegration-9.11 > > > > > > Solr-TestIntegration-9.x > > > > > > Solr-TestIntegration-main > > > > > > solr-desktop-ui-nightly > > > > > > > > > > > > On 4, there are 80 from these builds: > > > > > > > > > > > > OWASP-9.x > > > > > > OWASP-main > > > > > > Solr-Artifacts-10.x > > > > > > Solr-Artifacts-9.11 > > > > > > Solr-Artifacts-9.x > > > > > > Solr-Docker-Nightly-9.11 > > > > > > Solr-Lint-10.x > > > > > > Solr-Lint-9.11 > > > > > > Solr-Lint-9.x > > > > > > Solr-Lint-main > > > > > > Solr-Maven-Snapshots-main > > > > > > Solr-Test-10.0 > > > > > > Solr-Test-10.x > > > > > > Solr-Test-9.11 > > > > > > Solr-Test-9.x > > > > > > Solr-Test-main > > > > > > Solr-TestIntegration-9.11 > > > > > > Solr-TestIntegration-9.x > > > > > > > > > > > > On 2026/07/03 20:32:09 David Smiley wrote: > > > > > > > I know too little about Jenkins to know how to apply this > > solution, > > > > but I > > > > > > > can try to help. The Solr jobs are defined in Jenkins, not in > > our > > > > repo > > > > > > in > > > > > > > Jenkinsfile. Maybe you could apply this to one of the jobs > > yourself? > > > > > > Does > > > > > > > this need to be done on all jobs or just some or will just one > > do? > > > > > > > > > > > > > > On Fri, Jul 3, 2026 at 8:20 AM Bob Thomson <[email protected]> > > > > wrote: > > > > > > > > > > > > > > > This has come up as an issue again today, necessitating > > > > intervention to > > > > > > > > keep the service available. > > > > > > > > > > > > > > > > On 2026/06/29 07:04:23 Robert Thomson wrote: > > > > > > > > > Hi, > > > > > > > > > > > > > > > > > > We are regularly noticing a build-up of large ws-cleanup > > > > directories > > > > > > > > > on lucene3 and 4, necessitating deletion. Camel also > > experiences > > > > this > > > > > > > > > issue with builds and the root cause is that there are files > > > > created > > > > > > > > > as part of the docker build that are not owned by the user > > used > > > > for > > > > > > > > > the cleanup, so fail to be deleted by the workspace clean up > > > > plugin. > > > > > > > > > > > > > > > > > > Searching with GenAI came up with the following suggestion > > when > > > > > > > > > discussing with them that you may like to try out in your > > > > pipelines > > > > > > > > > that create docker images. > > > > > > > > > > > > > > > > > > Kind regards, > > > > > > > > > -Bob Thomson, > > > > > > > > > ASF Infrastructure > > > > > > > > > > > > > > > > > > post { > > > > > > > > > always { > > > > > > > > > // Fallback option: If standard cleanWs() stalls out > > on > > > > > > > > permissions > > > > > > > > > script { > > > > > > > > > try { > > > > > > > > > cleanWs deleteDirs: true, > > disableDeferredWipeout: > > > > > > true > > > > > > > > > } catch (Exception e) { > > > > > > > > > echo “Standard cleanup failed, launching > > Docker > > > > Root > > > > > > > > > Purge force-override...” > > > > > > > > > sh “”" > > > > > > > > > docker run --rm \ > > > > > > > > > -v ${WORKSPACE}/..:/zap \ > > > > > > > > > alpine:latest \ > > > > > > > > > sh -c ‘rm -rf /zap/*-ws-cleanup*’ > > > > > > > > > “”" > > > > > > > > > } > > > > > > > > > } > > > > > > > > > } > > > > > > > > > } > > > > > > > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > > > > > > > To unsubscribe, e-mail: [email protected] > > > > > > > > > For additional commands, e-mail: [email protected] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > > > > > > To unsubscribe, e-mail: [email protected] > > > > > > > > For additional commands, e-mail: [email protected] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > > > > To unsubscribe, e-mail: [email protected] > > > > > > For additional commands, e-mail: [email protected] > > > > > > > > > > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > > To unsubscribe, e-mail: [email protected] > > > > For additional commands, e-mail: [email protected] > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: [email protected] > > For additional commands, e-mail: [email protected] > > > > > --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
