Re: load balancing at fastmail.fm
Hello, what do you think about moving the mailspool to a central SAN storage shared via NFS and having several blades to manage the mmapped files like seen state, quota etc.? So still only one server is responsible for a certain set of mailboxes, but these SAN boxes have nice backup and redundancy features which are hard to get with common servers and there shouldn't be mmap problems as long as all indices remain on the blade on a separate metadata-partition. Regards Marten Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: load balancing at fastmail.fm
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 2/12/07 5:41 AM, Marten Lehmann wrote: Hello, what do you think about moving the mailspool to a central SAN storage shared via NFS and having several blades to manage the mmapped files like seen state, quota etc.? So still only one server is responsible for a certain set of mailboxes, but these SAN boxes have nice backup and redundancy features which are hard to get with common servers and there shouldn't be mmap problems as long as all indices remain on the blade on a separate metadata-partition. Cyrus and NFS don't get along due to locking issues; I believe this is covered in the docs. I tried this about a year ago and it spewed errors at an impressive rate. Instead, you might want to check out the Cyrus IMAP Aggregator Design page, which allows you to distribute mailboxes across multiple servers: http://asg.web.cmu.edu/cyrus/ag.html dn -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.1 (Darwin) iD8DBQFF0HxpyPxGVjntI4IRAlA8AKCFPCJAFuXkoeyTqI4ofTgoPvxIxACg+spS sLbF5pLBdqaF64S9QnJZe9M= =oNKh -END PGP SIGNATURE- Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Mail cluster with Murder or Perdition and MySQL: what is best and how to do?
Hello! I want to build load-balanced mail system with LVS, Postfix, Cyrus IMAP, MySQL and Murder or Perdition. For now I plan to use two nodes for storing users' mailboxes (each node will store one half of mailboxes). First, incoming mail will come to Postfix. But where (at what node) Postfix should direct mail to (to what node?) Cyrus IMAP. I heard, there is LMTP-proxy (in Cyrus IMAP?) which looks at database and redirects LMTP connections from Postfix to needed IMAP backend. There are three more questions: can this proxy use MySQL database to figure out where mailboxes are? And can LMTP proxy be on another host i.e. contacted by TCP connection and not UNIX filesystem socket. (I mean, is it possible to access LMTP-proxy via network?) Can several LMTP-proxies be launched simultaneously on different host using the same database. Next, mail will be stored on needed host. Everything is ok for now. But we want not to send mail only, but to receive to by POP3 and IMAP. So, user will contact some IMAP-proxy (Perdition or Murder). IMAP-proxy will make MySQL database lookup and redrect user to needed backend. Now there are questions. Can both Perdition and Murder work with MySQL. What is best? Thanks. Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Cyrus + LDAP = death by 13
Hi list, I have a problem with my cyrus server that I managed to track to the presence of the LDAP on the system. The user and group information is obtained form the LDAP server. When this functionality is enabled, when I start cyrus I get the following error: Feb 12 14:58:12 pingo master[22999]: about to exec /usr/lib/cyrus/idled Feb 12 14:58:12 pingo master[22963]: ready for work Feb 12 14:58:12 pingo master[22963]: process 23054 exited, signaled to death by 13 Feb 12 14:58:12 pingo master[22963]: process 23055 exited, signaled to death by 13 Feb 12 14:58:12 pingo master[22963]: process 23056 exited, signaled to death by 13 Feb 12 14:58:14 pingo master[22963]: process 23057 exited, signaled to death by 13 Feb 12 14:58:14 pingo master[22963]: service imaps pid 23057 in READY If I change the nssswitch.conf to obtain the group information from files, cyrus starts up fine. passwd: files ldap #group: files ldap group: files When I shut down ldap server, leave the nsswitch.conf to obtain the info from files ldap and start cyrus, I get the following error for some time, and them cyrus starts up normally. Feb 12 15:13:07 pingo master[32551]: retrying with 1024 (current max) Feb 12 15:13:07 pingo master[32551]: process started Feb 12 15:13:07 pingo master[32554]: nss_ldap: failed to bind to LDAP server ldaps://localhost/: Can't contact LDAP server Feb 12 15:13:07 pingo master[32554]: nss_ldap: failed to bind to LDAP server ldaps://localhost/: Can't contact LDAP server Feb 12 15:13:07 pingo master[32554]: nss_ldap: reconnecting to LDAP server (sleeping 1 seconds)... Feb 12 15:13:08 pingo master[32554]: nss_ldap: failed to bind to LDAP server ldaps://localhost/: Can't contact LDAP server Feb 12 15:13:08 pingo master[32554]: nss_ldap: reconnecting to LDAP server (sleeping 2 seconds)... In both of last cases, I can turn on ldap after the cyrus has started, and it does not affect its later functionality and works fine. What can be the problem of initial start up of cyrus and signaled to death by 13 (broken pipe) ? Has anyone experienced the same problem ? -- Respectfully, Konstantin V. Gavrilenko Managing Director Arhont Ltd - Information Security web:http://www.arhont.com http://www.wi-foo.com e-mail: [EMAIL PROTECTED] tel: +44 (0) 870 44 31337 fax: +44 (0) 117 969 0141 PGP: Key ID - 0xE81824F4 PGP: Server - keyserver.pgp.com Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Mail cluster with Murder or Perdition and MySQL: what is best and how to do?
Igor Zhbanov wrote: Hello! I want to build load-balanced mail system with LVS, Postfix, Cyrus IMAP, MySQL and Murder or Perdition. A note on LVS with perdition (launched from xinetd): set the KEEPALIVE flag to true if you are using xinetd. For now I plan to use two nodes for storing users' mailboxes (each node will store one half of mailboxes). First, incoming mail will come to Postfix. But where (at what node) Postfix should direct mail to (to what node?) Cyrus IMAP. I heard, there is LMTP-proxy (in Cyrus IMAP?) which looks at database and redirects LMTP connections from Postfix to needed IMAP backend. You can configure postfix to redirect to the correct backend via lmtp over tcp. Postfix can look in mysql/ldap/text files/... There are three more questions: can this proxy use MySQL database to figure out where mailboxes are? And can LMTP proxy be on another host i.e. contacted by TCP connection and not UNIX filesystem socket. (I mean, is it possible to access LMTP-proxy via network?) Can several LMTP-proxies be launched simultaneously on different host using the same database. I'm not using lmtp proxy but I thought you should point postfix to deliver to the lmtp proxy and this will then redirect to the correct backend. I thought you could run several lmtp proxies. Next, mail will be stored on needed host. Everything is ok for now. But we want not to send mail only, but to receive to by POP3 and IMAP. So, user will contact some IMAP-proxy (Perdition or Murder). IMAP-proxy will make MySQL database lookup and redrect user to needed backend. Now there are questions. Can both Perdition and Murder work with MySQL. What If you had a look at the perdition site you would see it can look up in mysql databases. I'm not sure (I'm not using murder) but murder doesn't look up in mysql. Murder uses it own database server, the MUPDATE server. -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- Rudy Gevaert [EMAIL PROTECTED] tel:+32 9 264 4734 Directie ICT, afd. Infrastructuur ICT Department, Infrastructure office Groep SystemenSystems group Universiteit Gent Ghent University Krijgslaan 281, gebouw S9, 9000 Gent, Belgie www.UGent.be -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: load balancing at fastmail.fm
On Mon, 12 Feb 2007, Marten Lehmann wrote: what do you think about moving the mailspool to a central SAN storage shared via NFS and having several blades to manage the mmapped files like seen state, quota etc.? Why do you need NFS? The whole point of a SAN is distributed access to storage after all :). So still only one server is responsible for a certain set of mailboxes, but these SAN boxes have nice backup and redundancy features which are hard to get with common servers It depends how much you trust your SAN. Some of my colleagues who run a SAN have had no end of grief. At which point you are dependant on the abilities of the vendor to diagnose and fix problems. It was this experience that encouraged me to try application level replication with lots of small servers in the first place. At least that way I can keep a close eye on what the various copies are up to. A SAN doesn't protect you if your filesystem decides to explode: I believe that Fastmail have direct experience of this. Two independent copies of the data allows you to keep running a service for the hours that an fsck typically takes to complete with file per msg stores on large modern disks. It also means rather less stress if the fsck fails to complete. I've heard horror stories about all the common Linux filesystems and I've personally watched fsck.ext3 (supposedly the safest option) unravel a filesystem, with thousands of entries left in lost+found. ZFS looks nice. -- David Carter Email: [EMAIL PROTECTED] University Computing Service,Phone: (01223) 334502 New Museums Site, Pembroke Street, Fax: (01223) 334679 Cambridge UK. CB2 3QH. Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Thunderbird + Kerberos 5 + Cyrus SASL-and-IMAP?
If anyone wants to assist in testing, here is the bug report I filed just now: https://bugzilla.mozilla.org/show_bug.cgi?id=370178 Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
New messages stop showing up in Outlook 2000 SP3
Hi everyone, I have a user running Outlook 2000 and new messages stopped showing a few days ago. After running reconstruct once, two new messages showed up, but subsequent runs have not had any affect. I have about 50 users running a mix of various Outlooks, Thunderbird and Eudora and none one else is having a problem. Oddly, he has multiple imap mailbox profiles (multiple mailboxes) in his Outlook and none of the other ones are affected. So, I'm not sure if this is an Outlook issue or cyrus-imap issue -- I'm leaning towards it being something wrong with that profile on Outlook since if he switches to Tbird all of the messages show up just fine -- and I had him create a profile on my instance of Outlook and all the messages showed up as well. The version of Cyrus-imap I'm running is 2.2.10. Any suggestions would be greatly appreciated. --Jim James Miller - MCSE RHCE CISSP Sr Systems Network Administrator Simutronics Corp. www.play.net 636.946.4263 x113 Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: New messages stop showing up in Outlook 2000 SP3
On Monday 12 February 2007 10:56, James Miller wrote: So, I'm not sure if this is an Outlook issue or cyrus-imap issue -- I'm leaning towards it being something wrong with that profile on Outlook since if he switches to Tbird all of the messages show up just fine -- and I had him create a profile on my instance of Outlook and all the messages showed up as well. What version of Outlook do you use? Didn't 2000 have a limitation of folder size? Having said that, discussing Outlook is probably off-topic for this list. wt -- Warren Turkal Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: load balancing at fastmail.fm
Hello, Why do you need NFS? because NFS is the only standard network file protocol. I don't want to load a proprietary driver into the kernel to access a SAN device. The whole point of a SAN is distributed access to storage after all :). So where's the point? SANs usually have redundant network devices to access the redudant disk array behind it. It depends how much you trust your SAN. Sure, but at some level you always have to trust to something. A SAN doesn't protect you if your filesystem decides to explode: Well, there are inode based SANs and file based SANs. If I'm just splitting an inode based SAN, I could also use internal disks which give me more control. But with file based SANs I can actually store files (through NFS). And a lot of SANs offer the possibility to do snapshots or replicate their data filebased to another SAN. So you have a very high redundancy and availability. Me idea was, that Cyrus does lock and mmap indices and databases, but not the actual message-files. So these message files could be stored in the SAN with very high redundancy, whereas the metadata which needs to be mmaped remains on the blade with internal disks so in case of problems you could at least restore the messages from the SAN (and its snapshots if you accidentally deleted something) and rebuild the indices. I've heard horror stories about all the common Linux filesystems and I've personally watched fsck.ext3 (supposedly the safest option) unravel a filesystem, with thousands of entries left in lost+found. ext3 with journal? I have never experienced this. ZFS looks nice. Well, but you are on your own because this project for linux is pretty young. Regards Marten Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: load balancing at fastmail.fm
On Mon, 12 Feb 2007, Marten Lehmann wrote: because NFS is the only standard network file protocol. I don't want to load a proprietary driver into the kernel to access a SAN device. Fair enough, although NFS is likely to be really rather slow compared to a block device which just happens to be accessed via a fibre channel link. I would be surprised if NFS worked given that it is only a approximation to a real Unix filesystem. Cyrus really hammers the filesystem. I've heard horror stories about all the common Linux filesystems and I've personally watched fsck.ext3 (supposedly the safest option) unravel a filesystem, with thousands of entries left in lost+found. ext3 with journal? I have never experienced this. It was in a RAID set which had had a dodgy disk, but there was a definite urk moment when I saw what fsck had done. Fortunately not critical data. ZFS looks nice. Well, but you are on your own because this project for linux is pretty young. I don't have any problem with OpenSolaris, though it would be a little amusing given that we moved from Solaris to Linux about 4 years back. -- David Carter Email: [EMAIL PROTECTED] University Computing Service,Phone: (01223) 334502 New Museums Site, Pembroke Street, Fax: (01223) 334679 Cambridge UK. CB2 3QH. Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: load balancing at fastmail.fm
David Carter wrote: Why do you need NFS? The whole point of a SAN is distributed access to storage after all :). SAN distributes the disk, not the filesystem. I presume in this case hes not using the SAN for its multiple-client-access features but just because its fast/reliable. Some of my colleagues who run a SAN have had no end of grief. At which point you are dependant on the abilities of the vendor to diagnose and fix problems. It was this experience that encouraged me to try application level replication with lots of small servers in the first place. At least that way I can keep a close eye on what the various copies are up to. SAN really has nothing to do with replication. You have your data somewhere (local or external disks, local/ext raid, NAS, SAN, etc), and youve got your various replication options (file-level, block-level, via client, via server, etc). None of these are a replacement for backups. A SAN doesn't protect you if your filesystem decides to explode: I believe that Fastmail have direct experience of this. Two independent copies of the data allows you to keep running a service for the hours that an fsck typically takes to complete with file per msg stores on large modern disks. It also means rather less stress if the fsck fails to complete. Fastmail dont use SAN, as I understand they use external raid arrays. There are many ways to lose your data, one of these being filesystem error, others being software bugs and human error. Block-level replication (typically used in SANs) is very fast and uses few resources but doesnt protect from filesystem error (although it can offer instant recovery). File-level replication is somewhat more resilient and easier to monitor, but is just as prone to human errors, bugs, misconfigurations, etc. I've heard horror stories about all the common Linux filesystems and I've personally watched fsck.ext3 (supposedly the safest option) unravel a filesystem, with thousands of entries left in lost+found. ZFS looks nice. There will be horror stories for every given system in the world. Generally speaking ext3 is very reliable, but naturally no filesystem is going to remove the need for replication and no replication system is going to remove the need for backups. Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: load balancing at fastmail.fm
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 2/12/07 11:01 AM, David Carter wrote: I would be surprised if NFS worked given that it is only a approximation to a real Unix filesystem. Cyrus really hammers the filesystem. NFS does not work with cyrus. Been there, done that, didn't like the end that movie at all. dn -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.3 (Darwin) iD8DBQFF0NPdyPxGVjntI4IRAvbaAJ9oTmYaBCR4DvuZ0E0V2u8E1HTn9ACfW5bN 06ZaSbfhmp+Tv5ioG5Ra+Ys= =82sl -END PGP SIGNATURE- Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: load balancing at fastmail.fm
Fastmail dont use SAN, as I understand they use external raid arrays. There are many ways to lose your data, one of these being filesystem error, others being software bugs and human error. Block-level replication (typically used in SANs) is very fast and uses few resources but doesnt protect from filesystem error (although it can offer instant recovery). If it's using block level replication, how does it offer instant recovery on filesystem corruption? Does it track every block written to disk, and can thus roll back to effectively what was on disk at a particular instant in time, so you then just remount the filesystem and the replay of the journal should restore to a good state? File-level replication is somewhat more resilient and easier to monitor, but is just as prone to human errors, bugs, misconfigurations, etc. Any replication system is prone to human errors and bugs, the most common one being split brain syndrome which is pretty much possible with any replication system regardless of which approach it uses if you stuff up. Which is why good tools and automation that ensure you can't stuff it up are really important! :) There will be horror stories for every given system in the world. Generally speaking ext3 is very reliable, but naturally no filesystem is going to remove the need for replication and no replication system is going to remove the need for backups. Indeed. Which is what we have, a replicated setup with nightly incremental backups. And things like filesystem or LVM snapshots are NOT backups, they're still relying on the integrity of your filesystem, rather than being on completely separate storage. The main thing we were trying to avoid was single points of failure. With a SAN, you generally have a very reliable, though very expensive central data store, but it's still a single point of failure, and even better you're dealing with some closed system you have to rely on a vendor for support for. That may or may not be a good thing depending on your point of view. You still have the SAN as a single point of failure With block based replication, you get the hardware redundancy, but you still have the filesystem as a single point of failure. If master end gets corrupted (eg http://oss.sgi.com/projects/xfs/faq.html#dir2) the other end replicates the corruption. With file based replication, about your only way of failure is the replication software going crazy blowing both sides away somehow, which given that the protocol is strictly designed to be one way, seems extremely unlikely that anything will happen to the master side. Rob PS. As a separate observation, if you're looking to get performance out of cyrus with a large number of users in a significantly busy environment, don't use ext3. We've been using reiserfs for years, but after the SUSE announcement, decided to try ext3 again on a machine. We had to switch it back to reiserfs, the load difference and visible performance difference for our users was quite large. And yes we tried with dirindex and various journal options. None of them came close to matching the load and response times of our standard reiser mount options; noatime,nodiratime,notail,data=journal, but read these first: http://www.irbs.net/internet/info-cyrus/0412/0042.html http://lists.andrew.cmu.edu/pipermail/info-cyrus/2006-October/024119.html Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: load balancing at fastmail.fm
If it's using block level replication, how does it offer instant recovery on filesystem corruption? Does it track every block written to disk, and can thus roll back to effectively what was on disk at a particular instant in time, so you then just remount the filesystem and the replay of the journal should restore to a good state? Yes. I may be wrong but to my understanding at least NetApp has this capability. With file based replication, about your only way of failure is the replication software going crazy blowing both sides away somehow, which given that the protocol is strictly designed to be one way, seems extremely unlikely that anything will happen to the master side. I agree of course about avoiding SPOFs, but I do like a multi-tiered approach, I mean multiple lines of defense. I use SAN for its speed, reliability, and ease of administration, but naturally I replicate everything on the SAN and have true backups as well. Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: load balancing at fastmail.fm
On Mon, 12 Feb 2007, urgrue wrote: If it's using block level replication, how does it offer instant recovery on filesystem corruption? Does it track every block written to disk, and can thus roll back to effectively what was on disk at a particular instant in time, so you then just remount the filesystem and the replay of the journal should restore to a good state? Yes. I may be wrong but to my understanding at least NetApp has this capability. No, NetApp takes snapshots of the filesystems on a schedule (hourly, daily, weekly, etc), and you can read files off of those snapshots. you cannot getany more granular then that. David Lang Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Thunderbird + Kerberos 5 + Cyrus SASL-and-IMAP?
GSSAPI authentication from Thunderbird to Cyrus IMAP works! You MUST: 1. Specify a FQDN for your IMAP server in Thunderbird's account settings. I was specifying an IP address. Not good enough. 2. The FQDN must resolve somehow. For me, it was a matter of adding info to C:\WINDOWS\System32\drivers\etc\hosts 192.168.168.100 noodle.foo.com 3. Your domain, of course, must map to some Kerberos realm. This is done in your /etc/krb5.conf or krb5.ini for Windows. Here's how mine was setup when working: [realms] JBTEST = { kdc = 192.168.168.100 admin_server = 192.168.168.100 } [domain_realm] foo.com = JBTEST .foo.com = JBTEST 4. Obviously specify 'Secure Authentication' in the IMAP account's properties. 5. In Thunderbird: Tools | Options | Advanced, Config editor set network.auth.use-sspi to false. Jeff Blaine wrote: If anyone wants to assist in testing, here is the bug report I filed just now: https://bugzilla.mozilla.org/show_bug.cgi?id=370178 Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: load balancing at fastmail.fm
I agree of course about avoiding SPOFs, but I do like a multi-tiered approach, I mean multiple lines of defense. I use SAN for its speed, reliability, and ease of administration, but naturally I replicate everything on the SAN and have true backups as well. So you have multiple SAN's? Or your SAN is still a potential SPOF? Nice if you can afford it :) Rob Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html