hashar added a comment.
I have checked in Docker and it does a `os.removeAll` which is waited for
completion. If a container has huge amount of files that have to be deleted,
that would file the memory cache for writes which is is known in Linux as the
Dirty memory. From `proc(5)`:
> **Dirty %lu**
> Memory which is waiting to get written back to the disk.
That is tunable via system settings:
name="sudo sysctl -ar 'dirty'|egrep -v "= 0$""
vm.dirty_background_ratio = 10
vm.dirty_expire_centisecs = 3000
vm.dirty_ratio = 20
vm.dirty_writeback_centisecs = 500
vm.dirtytime_expire_seconds = 43200
And In Graphite looking at a random agent using the metrics:
|
secondYAxis(integration.integration-agent-docker-1002.iostat.vda.writes_per_second)
|
| integration.integration-agent-docker-1002.memory.Dirty
|
|
The Graphite graph
<https://graphite-labs.wikimedia.org/?width=592&height=386&from=-2hours&target=secondYAxis(integration.integration-agent-docker-1002.iostat.vda.writes_per_second)&target=integration.integration-agent-docker-1002.memory.Dirty>
(green memory to be written, blue writes per seconds):
F34743426: disk_write_dirty.png <https://phabricator.wikimedia.org/F34743426>
I would assume the `unlinkat` are piled up in memory fil it up and at some
point the cache is full and the unlinks are done synchronously? Maybe if we
raise the dirty cache limit (`vm.dirty_ratio`) we would get a larger buffer.
I don't know much about the arcane of Linux disk cache though :-\
TASK DETAIL
https://phabricator.wikimedia.org/T265615
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: hashar
Cc: Aklapper, dduvall, bd808, jeena, Daimona, thcipriani, hashar, awight,
kostajh, Lucas_Werkmeister_WMDE, 786, TheReadOnly, Suran38, Biggs657,
Invadibot, Lalamarie69, maantietaja, Juan90264, Alter-paule, Beast1978, Un1tY,
Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, NebulousIris,
Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan,
GoranSMilovanovic, QZanden, LawExplorer, Lewizho99, Maathavan, _jensen,
rosalieper, Liudvikas, Scott_WUaS, Wikidata-bugs, aude, Jdforrester-WMF,
Mbch331, Jay8g
_______________________________________________
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
