Re: Several issues with 2.1.13
Wil Cooley wrote: The upgrade wasn't entirely smooth; I wrote up some notes about what I did: http://nakedape.cc/wiki/index.cgi/CyrusImapNotes in case someone else wanders along this path... The biggest issue was that ctl_cyrusdb wasn't able to read my old mailboxes.db file; I reverted to my old installation and dumped the database to a flat file and it went okay. For some reason that I don't understand, I had to remove the /var/lib/imap/db directory for the rebuilding of the database to work correctly. If the version of db is different, you can't just expect to use the binary database files and logs. Dumping the contents to a text file, wiping the transaction logs, and then reloading them is the safest way. 'rehash full' did very strange things; it only created directories of A-Z, none of a-z and my own mailbox information was under 'I/' in both the mailbox spool and the '/var/lib/imap/user' directory. As a result, I had to disable 'hashimapspool', which Simon's RPMs enabled by default. You should have had A-W, not Z (23 is a prime to give better distribution of the hash values between directories) -- full hashing chooses a hash directory based on the complete mailbox name rather than just the first character. Traditional hashing tends to overload some directories while leaving others almost empty. mbpath will give you the path for the mailbox, or just ls -ld /cyrus/*/user/wcooley (for example). If you have less than a thousand mailboxes, there is no need to worry about hashing. Otherwise, you will likely get poor performance with long namei lookups. Finally, I have a customer that's a small ISP that's currently running 2.0.16. I'm going to upgrade regardless, just so I can bounce messages delivered through LMTP to boxes that are over-quota. However, I recurrent problem we have is with POP3 users (which everyone is) who lose their connection (usually because of problems with dial-up). The pop3d stays running and locks the mailbox for 15 minutes or so, causing lots of support calls and grumbling. I'm guessing the connection stays in TIME_WAIT for this period, but 15 minutes seems like a long time for it to stay open. I see the 'poptimeout' setting that might help, but even 10 minutes might be too long. (This 15 minutes could be only 10 minutes already; I'm just being told this by the guy who does support.) Will anything that's changed between 2.0.16 and 2.1.13 help assuage this problem? In this day, is there really any good reason to continue using POP? There are so many problems with it (including support issues when people downloaded their mail to one computer and wonder why they can't access it from another) it seems best to retire it. When the user calls up complaining about this, that gives you a perfect opening to convince them to move to IMAP. -- John A. Tamplin Unix System Administrator Emory University, School of Public Health +1 404/727-9931
Re: Several issues with 2.1.13
On Mon, 9 Jun 2003, Wil Cooley wrote: 'rehash full' did very strange things; it only created directories of A-Z, none of a-z and my own mailbox information was under 'I/' in both the mailbox spool and the '/var/lib/imap/user' directory. As a result, I had to disable 'hashimapspool', which Simon's RPMs enabled by default. This is exactly what is expected in a fulldirhash system. I have several messages that are GPG-signed (using PGP/MIME format) that were sent through a Mailman 2.1 server that will only show the footer appended by Mailman. I've looked at the message in the spool, and it's complete there. I turned on telemetry logging and found out that the entire message is not being sent over the IMAP connection; so it's the server and not just my client. I can make a copy of the message available if someone wants to verify (it's not a private message). I had this problem with 2.0.16 and still with 2.1.13. I don't seem to have this problem with other GPG-signed messages going through Mailman lists. Sending the message and logs would be helpful, but I won't promise anything. Even better, create a bug on bugzilla.andrew.cmu.edu. However, I recurrent problem we have is with POP3 users (which everyone is) who lose their connection (usually because of problems with dial-up). The pop3d stays running and locks the mailbox for 15 minutes or so, causing lots of support calls and grumbling. I'm guessing the connection stays in TIME_WAIT for this period, but 15 minutes seems like a long time for it to stay open. I see the 'poptimeout' setting that might help, but even 10 minutes might be too long. (This 15 minutes could be only 10 minutes already; I'm just being told this by the guy who does support.) Will anything that's changed between 2.0.16 and 2.1.13 help assuage this problem? No thing between the versions affected this behavior. You can try turning down the poptimeout setting (since most likely if they are getting disconnected it is just the idle timeout that is waiting for them). You can always stop having your people rudely disconnect, or encourage them to use IMAP clients. Using Cyrus as a POP-only server is sort of like using a semi-trailer to haul a bag of groceries. ;) -Rob -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- Rob Siemborski * Andrew Systems Group * Cyert Hall 207 * 412-268-7456 Research Systems Programmer * /usr/contributed Gatekeeper
Re: Several issues with 2.1.13
On Mon, 2003-06-09 at 12:53, John Alton Tamplin wrote: If the version of db is different, you can't just expect to use the binary database files and logs. Dumping the contents to a text file, wiping the transaction logs, and then reloading them is the safest way. That was sort of what I figured. You should have had A-W, not Z (23 is a prime to give better distribution of the hash values between directories) -- full hashing chooses a hash directory based on the complete mailbox name rather than just the first character. Traditional hashing tends to overload some directories while leaving others almost empty. Yeah, you're probably right; I just saw a directory full of capital letters and assumed it was all of them. I do recall prime-based hash functions being better. However, after I did the hash, imapd seemed to still be using the traditional first-letter-hash. I had this in my logs and couldn't open my mailboxes: Jun 8 17:51:46 rheingold imapd[14729]: IOERROR: opening /home/imap/w/user/wcooley/cyrus.header: No such file or directory mbpath will give you the path for the mailbox, or just ls -ld /cyrus/*/user/wcooley (for example). If you have less than a thousand mailboxes, there is no need to worry about hashing. Otherwise, you will likely get poor performance with long namei lookups. In this day, is there really any good reason to continue using POP? There are so many problems with it (including support issues when people downloaded their mail to one computer and wonder why they can't access it from another) it seems best to retire it. When the user calls up complaining about this, that gives you a perfect opening to convince them to move to IMAP. Our customers have a 10MB mail quota which they can fill up pretty quickly. We don't want them leaving their stuff on our server, and they want to be able to have their mail even when they're off-line (I know there's IMAP sync, but it's questionable how well supported it is, and I don't really want to try to explain it to customers). Wil -- Wil Cooley [EMAIL PROTECTED] Naked Ape Consultinghttp://nakedape.cc * * * * Linux, UNIX, Networking and Security Solutions * * * * * Tired of spam and viruses in your e-mail? Get the * * Naked Ape Mail Defender! http://nakedape.cc/r/maildefender * signature.asc Description: This is a digitally signed message part
Re: Several issues with 2.1.13
On Mon, 2003-06-09 at 13:01, Rob Siemborski wrote: On Mon, 9 Jun 2003, Wil Cooley wrote: 'rehash full' did very strange things; it only created directories of A-Z, none of a-z and my own mailbox information was under 'I/' in both the mailbox spool and the '/var/lib/imap/user' directory. As a result, I had to disable 'hashimapspool', which Simon's RPMs enabled by default. This is exactly what is expected in a fulldirhash system. Okay, but see my other message about imapd not looking in the right hashed directory. Sending the message and logs would be helpful, but I won't promise anything. Even better, create a bug on bugzilla.andrew.cmu.edu. Will do. I hadn't noticed the Bugzilla link before. No thing between the versions affected this behavior. You can try turning down the poptimeout setting (since most likely if they are getting disconnected it is just the idle timeout that is waiting for them). Will it actually allow me to set it below 10 minutes? You can always stop having your people rudely disconnect, or encourage them to use IMAP clients. Using Cyrus as a POP-only server is sort of like using a semi-trailer to haul a bag of groceries. ;) There's no way to clean up the phone lines in rural areas or to force people to buy good modems (although when they have lots of problems we recommend that they do). As I said in my other response to John, I don't think IMAP would work very well for our customers, and user-visible changes always cause lots of suffering--the support guy has to spend hours on the phone when we could have him doing better things and I have to listen to his complaining. There are other good reasons to use Cyrus, though. Wil -- Wil Cooley [EMAIL PROTECTED] Naked Ape Consultinghttp://nakedape.cc * * * * Linux, UNIX, Networking and Security Solutions * * * * * Tired of spam and viruses in your e-mail? Get the * * Naked Ape Mail Defender! http://nakedape.cc/r/maildefender * signature.asc Description: This is a digitally signed message part
Re: Several issues with 2.1.13
On Mon, 9 Jun 2003, Wil Cooley wrote: On Mon, 2003-06-09 at 13:01, Rob Siemborski wrote: On Mon, 9 Jun 2003, Wil Cooley wrote: 'rehash full' did very strange things; it only created directories of A-Z, none of a-z and my own mailbox information was under 'I/' in both the mailbox spool and the '/var/lib/imap/user' directory. As a result, I had to disable 'hashimapspool', which Simon's RPMs enabled by default. This is exactly what is expected in a fulldirhash system. Okay, but see my other message about imapd not looking in the right hashed directory. Right, apparently your binaries weren't built to deal with a fulldirhash system, but you ran the rehash command to use a fulldirhash system. No thing between the versions affected this behavior. You can try turning down the poptimeout setting (since most likely if they are getting disconnected it is just the idle timeout that is waiting for them). Will it actually allow me to set it below 10 minutes? Well, not in the default code base. RFC 1939 states: A POP3 server MAY have an inactivity autologout timer. Such a timer MUST be of at least 10 minutes' duration. The receipt of any command from the client during that interval should suffice to reset the autologout timer. I'm not going to recommend you run a non-complient server, but finding the place to fix this in pop3d.c shouldn't be very hard (Hint: search for poptimeout). There's no way to clean up the phone lines in rural areas or to force people to buy good modems (although when they have lots of problems we recommend that they do). Right, I was joking about that part ;) -Rob -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- Rob Siemborski * Andrew Systems Group * Cyert Hall 207 * 412-268-7456 Research Systems Programmer * /usr/contributed Gatekeeper
Re: Several issues with 2.1.13
Wil Cooley wrote: Yeah, you're probably right; I just saw a directory full of capital letters and assumed it was all of them. I do recall prime-based hash functions being better. However, after I did the hash, imapd seemed to still be using the traditional first-letter-hash. I had this in my logs and couldn't open my mailboxes: Jun 8 17:51:46 rheingold imapd[14729]: IOERROR: opening /home/imap/w/user/wcooley/cyrus.header: No such file or directory Then it sounds like your imapd was not compiled with --enable-fulldirhash. The argument to rehash should match however your imapd was compiled (and using someone else's package may make it hard to tell). Our customers have a 10MB mail quota which they can fill up pretty quickly. We don't want them leaving their stuff on our server, and they want to be able to have their mail even when they're off-line (I know there's IMAP sync, but it's questionable how well supported it is, and I don't really want to try to explain it to customers). Your call, but even when I worked at an ISP we had far more problems with customers who used POP than those that didn't. Also, reducing the Cyrus timeout isn't going to help clear the half-connected sockets in the kernel if the customers just disconnect without dropping all their TCP connections first. -- John A. Tamplin Unix System Administrator Emory University, School of Public Health +1 404/727-9931
Re: Several issues with 2.1.13
On Mon, 2003-06-09 at 13:52, Rob Siemborski wrote: Right, apparently your binaries weren't built to deal with a fulldirhash system, but you ran the rehash command to use a fulldirhash system. Yeah, duh. Re-RTFMing, I see Simon has in README.RPM: 4) Please note that this RPM uses 'basic' hashed imap spool per default. Fulldirhash is supported as build time option but the RPM does not rehash your mailspool when upgrading. You have to run rehash on your existing mailspool if you have changed the hashing mode. Now I understand the difference between 'basic' and 'full'. No thing between the versions affected this behavior. You can try turning down the poptimeout setting (since most likely if they are getting disconnected it is just the idle timeout that is waiting for them). Will it actually allow me to set it below 10 minutes? Well, not in the default code base. RFC 1939 states: A POP3 server MAY have an inactivity autologout timer. Such a timer MUST be of at least 10 minutes' duration. The receipt of any command from the client during that interval should suffice to reset the autologout timer. I'm not going to recommend you run a non-complient server, but finding the place to fix this in pop3d.c shouldn't be very hard (Hint: search for poptimeout). Got it. But John's probably right about the kernel keeping the sockets open. But we never had this problem with cucipop (that's what I always have to listen to: We didn't have this problem before blah blah blah). Somehow, even though cucipop locked the mailboxes, it was able to operate concurrently on them (somewhat), without stupidly making a whole copy, the way UW's pop3d did. There's no way to clean up the phone lines in rural areas or to force people to buy good modems (although when they have lots of problems we recommend that they do). Right, I was joking about that part ;) Yeah, I know :) It'll be a beautiful day when customers don't have all those needless disconnections. But no thoughts on the squatter problem? I saw in the archives other people reporting the same problem, but no responses to indicate why. Anything more I could do to help debug it? Research Systems Programmer * /usr/contributed Gatekeeper You might want to look at the couple scripts I wrote to replace the TCL-based ones in Managing IMAP: http://nakedape.cc/wiki/index.cgi/CyrusImapNotes Wil -- Wil Cooley [EMAIL PROTECTED] Naked Ape Consultinghttp://nakedape.cc * * * * Linux, UNIX, Networking and Security Solutions * * * * * Tired of spam and viruses in your e-mail? Get the * * Naked Ape Mail Defender! http://nakedape.cc/r/maildefender * signature.asc Description: This is a digitally signed message part
Re: Several issues with 2.1.13
Wil Cooley wrote: Got it. But John's probably right about the kernel keeping the sockets open. But we never had this problem with cucipop (that's what I always have to listen to: We didn't have this problem before blah blah blah). Somehow, even though cucipop locked the mailboxes, it was able to operate concurrently on them (somewhat), without stupidly making a whole copy, the way UW's pop3d did. Yikes -- I know nothing about cucipop, but allowing multiple POP clients to operate on the same mailbox at the same time is begging for problems, since the protocol doesn't have any of the features that would be needed to support concurrent access. Maybe they assumed that when that happened that it was because the other connection was dropped and they automatically dropped it (releasing any locks held), but that still seems very dangerous. -- John A. Tamplin Unix System Administrator Emory University, School of Public Health +1 404/727-9931
RE: Several issues with 2.1.13
-Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Wil Cooley Sent: Tuesday, 10 June 2003 9:23 a.m. To: [EMAIL PROTECTED] Subject: Re: Several issues with 2.1.13 Got it. But John's probably right about the kernel keeping the sockets open. But we never had this problem with cucipop (that's what I always Do you really care about the kernel keeping the sockets open? You *probably* have plenty of them available, user who got disconnected will *probably* get new IP address assigned when he next time dials-in and he'll be able to download his e-mail, at least from the kernel TCP/IP stack perspective, assuming that pop3d is working ok. Or am I missing something here? Best regards, Bojan Zdrnja
Re: Several issues with 2.1.13
On Mon, 9 Jun 2003, Wil Cooley wrote: Research Systems Programmer * /usr/contributed Gatekeeper You might want to look at the couple scripts I wrote to replace the TCL-based ones in Managing IMAP: http://nakedape.cc/wiki/index.cgi/CyrusImapNotes I don't believe anything in Managing IMAP, since it's written about Cyrus v1, which is ancient ;) -Rob -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- Rob Siemborski * Andrew Systems Group * Cyert Hall 207 * 412-268-7456 Research Systems Programmer * /usr/contributed Gatekeeper
Re: Several issues with 2.1.13
On Mon, 2003-06-09 at 16:15, Rob Siemborski wrote: On Mon, 9 Jun 2003, Wil Cooley wrote: Research Systems Programmer * /usr/contributed Gatekeeper You might want to look at the couple scripts I wrote to replace the TCL-based ones in Managing IMAP: http://nakedape.cc/wiki/index.cgi/CyrusImapNotes I don't believe anything in Managing IMAP, since it's written about Cyrus v1, which is ancient ;) Well, that's why I wrote replacements :) The Perl scripts he provides still work and with my replacements, you can effectively convert a system from mbox mailboxes to Cyrus. At least, I did for 2.0.16. Oh, now that I look at the source, I see the scripts from the book aren't actually in contrib/; they're just packaged with Simon's RPMs. Guess I should be talking to him. Wil -- Wil Cooley [EMAIL PROTECTED] Naked Ape Consultinghttp://nakedape.cc * * * * Linux, UNIX, Networking and Security Solutions * * * * * Tired of spam and viruses in your e-mail? Get the * * Naked Ape Mail Defender! http://nakedape.cc/r/maildefender * signature.asc Description: This is a digitally signed message part
