Timo Sirainen wrote:
On Aug 28, 2009, at 8:38 PM, Adam McDougall wrote:
Early next week I need to upload over 100,000 emails to an IMAP
server as quickly as possible from an Outlook client. I am looking
for any methods I can use to (temporarily?) speed up the rate at
which dovecot can accept and store IMAP uploads, whether it be
storing on local disk, ram disk, etc. I can setup a temporary server
on a laptop for example and once the upload has finished I can use
standard file copying methods to transfer the mail to stable,
permanent storage. I haven't been able to see over about 7 msgs/sec
upload speed from a local folder in any mail client to dovecot (only
NFS or ZFS backend tested so far with Maildir). Is there something
horribly wrong with the speed I am seeing or are there just tricks I
can try? Any tips? I'll be working on it all weekend until I find
something satisfactory. It seems like I can upload mails to an
Exchange server quicker. I'll setup just about anything that my
experience allows me to, I can be very resourceful with adhoc
hardware and software.
From Dovecot's side the only thing you can do is fsync_disable=yes.
The main problem is probably network latency, because Outlook doesn't
support MULTIAPPEND extension (and perhaps not even LITERAL+
extension?) Did you already try running Dovecot on the same computer
as Outlook (some virtual thingy or maybe it works in cygwin)?
I just tried fsync_disable=yes but with NFS and had to turn off
mail_nfs_index = yes as well but the speed was the same. Do you think
it would be different with a UFS or ZFS backend with fsync_disable? I
have not tried running dovecot on the same computer. When you mention
dovecot+cygwin I think of the reported issues in the past on the mailing
list and don't know if they were resolved. I could try dovecot in
virtualbox I suppose (I put it on my list to try).
Alternatively I'll take a fast way of converting Exchange email to a
tree of local mbox files which I can then run mb2md on.
If the mails are in Exchange, can't you connect to it using IMAP?
In theory yes, but I don't have access to the actual Exchange server
until Monday at the earliest, and the user is using "cached exchange
mode" which in past experience leaves the possibility of local mail
which is not actually on the server due to a desync. Unless I am sure
it is perfectly in sync, I've seen a second Outlook connect to Exchange
using the native protocols and it initiated a massive deletion of mail
which we had to toil to recover from obscure cache files on the original
client. I don't know if an IMAP connection might trigger the same
issue. For performance testing's sake, I'll see if I can upload some
mail to our own Exchange server and see how fast an mbox capable mail
client can download it. I can do some limited testing in the real
environment on Monday but I'm expected to do the real migration on
Tuesday unless I have to cancel. Thanks for the ideas.