Re: [Dovecot] Quick and dirty server optimized for IMAP upload speed?
Adam McDougall wrote: Timo Sirainen wrote: On Aug 28, 2009, at 8:38 PM, Adam McDougall wrote: Early next week I need to upload over 100,000 emails to an IMAP server as quickly as possible from an Outlook client. I am looking for any methods I can use to (temporarily?) speed up the rate at which dovecot can accept and store IMAP uploads, whether it be storing on local disk, ram disk, etc. I can setup a temporary server on a laptop for example and once the upload has finished I can use standard file copying methods to transfer the mail to stable, permanent storage. I haven't been able to see over about 7 msgs/sec upload speed from a local folder in any mail client to dovecot (only NFS or ZFS backend tested so far with Maildir). Is there something horribly wrong with the speed I am seeing or are there just tricks I can try? Any tips? I'll be working on it all weekend until I find something satisfactory. It seems like I can upload mails to an Exchange server quicker. I'll setup just about anything that my experience allows me to, I can be very resourceful with adhoc hardware and software. From Dovecot's side the only thing you can do is fsync_disable=yes. The main problem is probably network latency, because Outlook doesn't support MULTIAPPEND extension (and perhaps not even LITERAL+ extension?) Did you already try running Dovecot on the same computer as Outlook (some virtual thingy or maybe it works in cygwin)? I just tried fsync_disable=yes but with NFS and had to turn off mail_nfs_index = yes as well but the speed was the same. Do you think it would be different with a UFS or ZFS backend with fsync_disable? I have not tried running dovecot on the same computer. When you mention dovecot+cygwin I think of the reported issues in the past on the mailing list and don't know if they were resolved. I could try dovecot in virtualbox I suppose (I put it on my list to try). I did a lot of testing today and found some things. The two biggest real bottlenecks: - Thunderbird is just slow at uploading to IMAP. With a bunch of small msgs it only does a few per second and you can tell the server is waiting for something to do. Outlook is considerably faster. Other clients not tested. - Perdition (IMAP proxy), at least in my current setup, slows down the mail upload speed around 50%. Non-bottlenecks: - fsync (I can't measure the difference at the client, but on the server I can see the behavior change) - filesystem (nfs/ufs/zfs all performed about the same) - server cpu - imap server being over the local network as opposed to running inside virtualbox on the same pc I think I am satisfied with the speeds I am seeing now for the needs I have next week. Depending on the resulting speed across campus, I may run dovecot on a portable laptop for the upload; I'll just go around the perdition proxy (plan to retire that in a few weeks). During my testing I did notice an issue with Outlook 2003 on dovecot 1.2 that I don't have with 1.1, I cannot delete an IMAP folder (maybe after clicking on it first). I get an error about 'folder is open in another session'. It happens on a Maildir store on a local filesystem or NFS and I only have one client accessing it. I might have time to look into it properly tomorrow, but if not, probably not for a few days at least. Unrelated: Outlook 2003 running on Windows 7 seems to abort the upload after just a few hundred messages with an error message. Works on XP. Alternatively I'll take a fast way of converting Exchange email to a tree of local mbox files which I can then run mb2md on. If the mails are in Exchange, can't you connect to it using IMAP? In theory yes, but I don't have access to the actual Exchange server until Monday at the earliest, and the user is using cached exchange mode which in past experience leaves the possibility of local mail which is not actually on the server due to a desync. Unless I am sure it is perfectly in sync, I've seen a second Outlook connect to Exchange using the native protocols and it initiated a massive deletion of mail which we had to toil to recover from obscure cache files on the original client. I don't know if an IMAP connection might trigger the same issue. For performance testing's sake, I'll see if I can upload some mail to our own Exchange server and see how fast an mbox capable mail client can download it. I can do some limited testing in the real environment on Monday but I'm expected to do the real migration on Tuesday unless I have to cancel. Thanks for the ideas.
[Dovecot] Dovecot 1.0.14 + osx 10.6 and authentication PAM
Howdy, Today I installed the new version of Apple OSX 10.6. Besides that it deleted the dovecot user, things went quite smooth. As far as I can tell it is running as it used to. However, I can't login to my account. It used to work fine with PAM taking the username and password from my osx installation. Now it doesn't allow me to login anymore. I have read on a website that apple changed from Linux-Pam to Open-Pam. If this is true, how would I need to change the PAM entry file? Right now it is as follows: /etc/pam.d/dovecot: # dovecot: auth account password session auth required pam_nologin.so auth sufficient pam_securityserver.so auth sufficient pam_unix.so auth required pam_deny.so account required pam_permit.so password required pam_deny.so session required pam_uwtmp.so Here is a list with all dovecot processes Maarten-2:sbin maarten$ ps auxw|grep dovecot\|imap\|pop3 maarten249 1,1 0,0 2435032528 s000 S+3:21pm 0:00.00 grep dovecot\|imap\|pop3 dovecot218 0,0 0,0 602248660 ?? S 3:06pm 0:00.08 imap-login root 215 0,0 0,0 601144636 ?? S 3:06pm 0:00.08 dovecot-auth root 214 0,0 0,0 599940356 ?? Ss3:06pm 0:00.11 ./dovecot dovecot238 0,0 0,0 602248660 ?? S 3:17pm 0:00.03 imap-login dovecot221 0,0 0,0 602248660 ?? S 3:06pm 0:00.08 imap-login and here is my Dovecot configuration: # 1.0.14: /opt/etc/dovecot.conf Warning: mail_extra_groups setting was often used insecurely so it is now deprecated, use mail_access_groups or mail_privileged_group instead base_dir: /var/run/dovecot/ protocols: imap ssl_disable: yes login_dir: /var/run/dovecot//login login_executable: /opt/libexec/dovecot/imap-login mail_extra_groups: mail mail_access_groups: mail default_mail_env: maildir:~/Maildir mail_location: maildir:~/Maildir auth default: passdb: driver: pam userdb: driver: passwd Thanks in advance, Maarten
Re: [Dovecot] Dovecot 1.0.14 + osx 10.6 and authentication PAM
On Sun, Aug 30, 2009 at 03:33:03PM +0200, Maarten Koster wrote: snip I have read on a website that apple changed from Linux-Pam to Open-Pam. If this is true, how would I need to change the PAM entry file? Right now it is as follows: This was discussed just a few posts earlier: http://www.dovecot.org/list/dovecot/2009-August/042504.html Thomas
Re: [Dovecot] Outlook 2007 w/SPA, Active Directory (was NTLM failures with an interesting twist)
On Sat, 2009-08-29 at 21:55 -0600, Jason Gunthorpe wrote: On Sun, Aug 30, 2009 at 01:50:02AM +0100, Gavin Hamill wrote: Has anyone successfully configured the above to enable Single Sign-On? I would love to move away from Exchange but SSO is a corporate requirement. I looked at this in some detail and concluded that the NTLM support on Outlook 2007 was only for encryption, it was not using SPA. I couldn't find a hidden registry setting or whatnot to switch it. Heh, have just found you here: https://bugzilla.mozilla.org/show_bug.cgi?id=284538 You mention that you managed to get Thunderbird working with SSO; I've not achieved that - I'm still required to provide the password before the NTLM login is successful.. Is there any particular magic needed with Thunderbird 2.0.0.23 ? If you have a corporate support arrangement with MS, maybe ask them? Many people would love an answer. Even a trace of outlook using SPA with Exchange over IMAP would be interesting to see. Alas our MS support arrangement extends to the same 'Google + web forums' that most other places use. gdh
Re: [Dovecot] Postfix : lda problem
On Mon 17 Aug 2009 10:21:47 PM CEST, denis wrote denis a écrit : Ok, here is a configuration that works fine but without success to retrieve the correct gid in the database. In fact, aliases no longer work In trying with alias_maps or virtual_alias_maps and in both cases the following error: User unknown in virtual mailbox table Any Ideas ? Thanks Denis ## /etc/postfix/main.cf alias_maps =proxy:mysql:/etc/postfix/myalias.cf, hash:/etc/aliases virtual_mailbox_base = /var/alternc/mail virtual_mailbox_maps = proxy:mysql:/etc/postfix/myvirtual.cf virtual_mailbox_domains = mysql:/etc/postfix/mydomain.cf default_privs = www-data dont give web server full email read access outside dovecot auth virtual_uid_maps = static:33 virtual_gid_maps = static:33 virtual_transport = dovecot dovecot_destination_recipient_limit = 1 ## /etc/postfix/myalias.cf user = xyz password = xyz hosts = 10.0.112.1 dbname = alternc table = mail_alias select_field = alias where_field = mail now sync dovecot to use same db backend as postfix so virtual_mailbox in postfix is equal to dovecot mailbox, make sure any alias in postfix is delivered to a mailbox not just another alias that is delivered to a alias in dovecot, else you get mailbox does not exists -- xpoint
Re: [Dovecot] Outlook 2007 w/SPA, Active Directory (was NTLM failures with an interesting twist)
On Sun, Aug 30, 2009 at 08:38:20PM +0100, Gavin Hamill wrote: On Sat, 2009-08-29 at 21:55 -0600, Jason Gunthorpe wrote: On Sun, Aug 30, 2009 at 01:50:02AM +0100, Gavin Hamill wrote: Has anyone successfully configured the above to enable Single Sign-On? I would love to move away from Exchange but SSO is a corporate requirement. I looked at this in some detail and concluded that the NTLM support on Outlook 2007 was only for encryption, it was not using SPA. I couldn't find a hidden registry setting or whatnot to switch it. Heh, have just found you here: https://bugzilla.mozilla.org/show_bug.cgi?id=284538 You mention that you managed to get Thunderbird working with SSO; I've not achieved that - I'm still required to provide the password before the NTLM login is successful.. Is there any particular magic needed with Thunderbird 2.0.0.23 ? Yes, you can't use NTLM in Thunderbird either, you have to use Kerberos (GSSAPI). I run NTLM through winbind and GSSAPI through MIT Kerberos, and then run exim through dovecot-auth. This gives complete SSO using GSSAPI for Thunderbird on all platforms, and secure challenge/response NTLM hashed passwords for roaming users without Kerberos. The kerberos setup is pretty easy.. 'net ads join' your server, go into the adsi editor and provide a imap and smtp SPN for the host, use 'net ads keytab' to put the imap and smtp SPNs in the system keytab, and then you are good to go. I test it with mutt first as the error messages are somewhat better. Apparently if you direct the GSSAPI messages through winbind (like for NTLM) then you can omit the 'net ads keytab' steps and things work a bit smoother, but I have not attempted that configuration. Jason
Re: [Dovecot] Quick and dirty server optimized for IMAP upload speed?
On Sun, Aug 30, 2009 at 12:33:43PM -0400, Charles Marcus wrote: On 8/30/2009, Adam McDougall (mcdou...@egr.msu.edu) wrote: The two biggest real bottlenecks: - Thunderbird is just slow at uploading to IMAP. With a bunch of small msgs it only does a few per second and you can tell the server is waiting for something to do. Did you try TBird 3.0b3? It has many, many IMAP improvements... -- Best regards, Charles Wow it sure is faster at uploading, thanks for mentioning it!
