On 12 Dec 2017, at 20:10, Steven M. Bellovin wrote:
On 12 Dec 2017, at 17:48, Bill Cole wrote:
On 12 Dec 2017, at 1:10 (-0500), Steven M. Bellovin wrote:
On 11 Dec 2017, at 23:26, Bill Cole wrote:
On 10 Dec 2017, at 21:14 (-0500), Steven M. Bellovin wrote:
My suspicion is that the problem has to do with very large
directories on APFS file systems
This would be shocking. One of the rationales for APFS existing is
that the HFS foundation was played out for dealing with large
directories efficiently. I haven't looked into the details (life is
short...) but if APFS is *worse* than HFS{+,X} with large
directories then Apple is in a worse state than I had thought...
Yah. I have no other explanation, though. To give a current example,
on a machine -- an old one, to be sure -- a Time Machine backup
started almost 10 hours ago. It's dumped 77.5 MB -- out of a total
of 152.7 MB -- in that time, and it's been at about 77 MB for the
last ~7-8 hours. At some point, though, it will pass the expensive
point and run at a reasonable rate. This dump is to a directly
connected USB 2.0 drive. And the CPU is about 96% idle, according to
'top'.
Btw: by "big", I mean that I have one mailbox with 114K messages;
the directory itself is 3.6 MB. No other mailbox is more than half
that size, though I have four that are over 1 MB.
Oh my.
Yah. I knew some were large, but I didn't think *that* large. Worse
yet, one of the top few is my inbox, which I haven't been cleaning out
of late. I've been following the MailMate mantra: just create smart
folders...
Since the backup disk can't be APFS (Time Machine relies on
hard-linked directories, which APFS won't do) you're still dealing
with that huge directory in HFS+ on the write side. If that directory
has changes it is going to be spectacularly slow for TM to do 114k
file hard links and copy a handful of changed files into a new
directory.
Right, which explains older slowness, but not the sudden problem.
OK. I split up the really largest mailboxes, which has taken care of the
immediate problem: I can now do backups in finite time. I still suspect
an APFS issue which suddenly made the problem critical, but at least I'm
back on the air. (And I'm building a Python script to ease mailbox
splitting...)
--Steve Bellovin, https://www.cs.columbia.edu/~smb
_______________________________________________
mailmate mailing list
mailmate@lists.freron.com
https://lists.freron.com/listinfo/mailmate
