Re: [MlMt] Moving large IMAP folders
On 11 Jan 2014, at 23:20, Bill Cole wrote: The way to do that with a correctly-behaving IMAP server is to create a new folder, copy all of the existing messages into it, and delete them from the original folder. This seems to be what MM does and it is a less risky and more universally workable strategy, but in principle it should be possible to do a real tree rearrangement with one RENAME command instead. I am interested in *why* Benny chose not to use RENAME. That's a good question. I had completely forgotten this part of the IMAP RFC. I cannot recall any other reason for not using it than the current behavior means that I can do the same thing no matter what action the user has taken (renaming or dragging a mailbox to a different location or even a different account — it's all a variation of moving messages). With respect to performance. For MailMate it would not be more efficient to use RENAME. It would just be a different code-path. Some servers though probably handle RENAME much more efficiently — and for some servers it would also mean that quota limits would not come into play. In any case, the biggest performance problem with renaming IMAP mailboxes is not solved by RENAME. It's only the email client doing the renaming that can handle it efficiently. All other (offline) email clients are, at least in theory, going to clear their caches and then refetch all messages in renamed mailboxes. @Kee: Yes, MailMate should be smarter about renaming (and moving) a parent mailbox. I'll put IMAP RENAME on the list of potential performance improvements. -- Benny ___ mailmate mailing list mailmate@lists.freron.com http://lists.freron.com/listinfo/mailmate
Re: [MlMt] Moving large IMAP folders
[ Sorry about the double-post. The backstory involves the indirect
interaction of Little Snitch and a snowblower.]
On 11 Jan 2014, at 8:49, Kee Hinckley wrote:
On Jan 11, 2014, at 2:31 AM, "Bill Cole"
wrote:
On 3 Jan 2014, at 14:04, Benny Kjær Nielsen wrote:
On 3 Jan 2014, at 17:08, Kee Hinckley wrote:
Is there in fact an underlying IMAP command to move a folder
hierarchy (as opposed to rename)? Or if I tell MailMate to move a
folder, is it going to copy and delete?
It is going to be copy and delete. Many servers though do that just
as efficiently as if it was a move (but not all servers).
Does the IMAP RENAME command not actually work reliably to move
folders and their children?
http://tools.ietf.org/html/rfc3501#section-6.3.5 seems to say it
should.
I ran into an issue with this the other day and meant to comment on
it.
Apparently IMAP folders aren't really hierarchical in a true sense, so
if you rename a folder and expect it to do what a directory does when
you rename it (have all the same folders under it, not just the
immediate files (messages)) you have to change the path of each
folder.
That is addressed in the RFC at the section I gave the link to:
If the name has inferior hierarchical names, then the inferior
hierarchical names MUST also be renamed. For example, a rename of
"foo" to "zap" will rename "foo/bar" (assuming "/" is the
hierarchy delimiter character) to "zap/bar".
In other words: the protocol spec requires the *server* to do any
downstream renaming. A server that only renames an enclosing folder
while leaving its subfolders at their old names is broken.
Administering mail servers has been a substantial part of my work for
many years, which is why I know which RFC to go to for IMAP4rev1
details, but unlike Benny I don't maintain code that actually has to use
IMAP4rev1 as a client with a diverse jungle of servers. I could manually
test what a handful of different IMAP servers actually do, but that
would not answer the question of whether MM uses a copy+delete process
out of necessity because RENAME is broken often enough to make it
useless or out of a misunderstanding of what RENAME actually does.
It is useful to understand that for many years of the early life of
IMAP4, there was a widely deployed diversity of formal breakage in
servers and clients which created a culture of ad hoc pragmatic
workarounds that lives on in botches like GMail's faux-IMAP and the maze
of client adaptations to it. The hierarchical model is one area where
this has been focused, a consequence of the many edge and corner cases
of multiple uncoordinated clients (and often mutually oblivious
server-side processes) having simultaneous access with modification
powers to a directory tree.
Mailmate's rename is only renaming the top-level folder and not
telling IMAP to rename those under it. So you end up with the top
level folder having a new name, but all the folder children still
having the old path.
The way to do that with a correctly-behaving IMAP server is to create a
new folder, copy all of the existing messages into it, and delete them
from the original folder. This seems to be what MM does and it is a less
risky and more universally workable strategy, but in principle it should
be possible to do a real tree rearrangement with one RENAME command
instead. I am interested in *why* Benny chose not to use RENAME.
I can't think of any case where that's the preferred behavior. I know
the recursive rename is the default in my webmail. I ended up going
there to rename my folder.
(The efficiency issue is a separate one. I can imagine that if
MailMate issues the rename, it could be smart enough to know that all
it has to do locally is move the messages. I don't know I'd it does, I
imagine that logic could get tricky.)
Historically, IMAP servers have mostly mimicked a hierarchical
filesystem for mailboxes rather than actually using their real
filesystems as a directly mapped hierarchy. This solves some server-side
problems (performance, access control, concurrent acess, metadata
storage, etc.) at the cost of making logically atomic operations on the
directory tree non-atomic at the real filesystem level. Depending on
their specific storage models, servers using file-per-message schemes
("Maildir" and related schemes) may have as many distinct filesystem
operations as there are leaf nodes or branch points in a sub-tree being
moved with RENAME, which must seem atomic to clients.
___
mailmate mailing list
mailmate@lists.freron.com
http://lists.freron.com/listinfo/mailmate
Re: [MlMt] Moving large IMAP folders
> On Jan 11, 2014, at 2:31 AM, "Bill Cole" > wrote: > >> On 3 Jan 2014, at 14:04, Benny Kjær Nielsen wrote: >> >>> On 3 Jan 2014, at 17:08, Kee Hinckley wrote: >>> >>> Is there in fact an underlying IMAP command to move a folder hierarchy (as >>> opposed to rename)? Or if I tell MailMate to move a folder, is it going to >>> copy and delete? >> >> It is going to be copy and delete. Many servers though do that just as >> efficiently as if it was a move (but not all servers). > > Does the IMAP RENAME command not actually work reliably to move folders and > their children? http://tools.ietf.org/html/rfc3501#section-6.3.5 seems to say > it should. I ran into an issue with this the other day and meant to comment on it. Apparently IMAP folders aren't really hierarchical in a true sense, so if you rename a folder and expect it to do what a directory does when you rename it (have all the same folders under it, not just the immediate files (messages)) you have to change the path of each folder. Mailmate's rename is only renaming the top-level folder and not telling IMAP to rename those under it. So you end up with the top level folder having a new name, but all the folder children still having the old path. I can't think of any case where that's the preferred behavior. I know the recursive rename is the default in my webmail. I ended up going there to rename my folder. (The efficiency issue is a separate one. I can imagine that if MailMate issues the rename, it could be smart enough to know that all it has to do locally is move the messages. I don't know I'd it does, I imagine that logic could get tricky.) ___ mailmate mailing list mailmate@lists.freron.com http://lists.freron.com/listinfo/mailmate
Re: [MlMt] Moving large IMAP folders
On 3 Jan 2014, at 14:04, Benny Kjær Nielsen wrote: On 3 Jan 2014, at 17:08, Kee Hinckley wrote: Is there in fact an underlying IMAP command to move a folder hierarchy (as opposed to rename)? Or if I tell MailMate to move a folder, is it going to copy and delete? It is going to be copy and delete. Many servers though do that just as efficiently as if it was a move (but not all servers). Does the IMAP RENAME command not actually work reliably to move folders and their children? http://tools.ietf.org/html/rfc3501#section-6.3.5 seems to say it should. ___ mailmate mailing list mailmate@lists.freron.com http://lists.freron.com/listinfo/mailmate
Re: [MlMt] Moving large IMAP folders
On 3 Jan 2014, at 14:04, Benny Kjær Nielsen wrote: On 3 Jan 2014, at 17:08, Kee Hinckley wrote: Is there in fact an underlying IMAP command to move a folder hierarchy (as opposed to rename)? Or if I tell MailMate to move a folder, is it going to copy and delete? It is going to be copy and delete. Many servers though do that just as efficiently as if it was a move (but not all servers). Does the IMAP RENAME command not actually work reliably to move folders and their children? http://tools.ietf.org/html/rfc3501#section-6.3.5 seems to say it should. ___ mailmate mailing list mailmate@lists.freron.com http://lists.freron.com/listinfo/mailmate
Re: [MlMt] Moving large IMAP folders
On 3 Jan 2014, at 17:08, Kee Hinckley wrote: Is there in fact an underlying IMAP command to move a folder hierarchy (as opposed to rename)? Or if I tell MailMate to move a folder, is it going to copy and delete? It is going to be copy and delete. Many servers though do that just as efficiently as if it was a move (but not all servers). Having imported a few gig or so of local email, I now want to reorganize, and I'm trying to figure out the most optimal way. I can do it via MailMate, or I could do it via my webmail interface, where the move would be efficient, but then MailMate might download everything all over again. Is one way going to be better than another? If both ways are going to result in copying data, it seems like the server case might be the most efficient. Renaming a large mailbox might also result in quota issues on the server since some servers are going to measure each message twice until the originals are deleted. It's probably safer to do this on the server while it's more efficient (for MailMate) to do it locally. In any case, it'll at most be efficient for 1 instance of 1 email client using the account. All other email clients are going to delete and fetch just like if you reorganize on the server. -- Benny ___ mailmate mailing list mailmate@lists.freron.com http://lists.freron.com/listinfo/mailmate
[MlMt] Moving large IMAP folders
Is there in fact an underlying IMAP command to move a folder hierarchy (as opposed to rename)? Or if I tell MailMate to move a folder, is it going to copy and delete? Having imported a few gig or so of local email, I now want to reorganize, and I'm trying to figure out the most optimal way. I can do it via MailMate, or I could do it via my webmail interface, where the move would be efficient, but then MailMate might download everything all over again. Is one way going to be better than another? If both ways are going to result in copying data, it seems like the server case might be the most efficient. --iPhoned ___ mailmate mailing list mailmate@lists.freron.com http://lists.freron.com/listinfo/mailmate
