Re: Cyrus support for multiple client access?
Ken Murchison wrote:
To be more specific, this means we have a mailbox where our customer
sends mail to and about 9 people should be able to be connected to
this mailbox seeing the incoming messages and those, that have been
answererd or seen). Of course, they should be able to answer, delete
or move mails.
Well, seen state is per user, so you won't know if someone else read
the message. If you allow all of these users to store message flags
('w' right), then you will know when someone else replied via the
\Answered flag. Similarly, allowing the 'd' right will allow the
messages to be marked as \Deleted.
You could have a convention of using user-defined labels for messages to
indicate that someone is taking care of it (which is what I am assuming
you are interested in for the Seen flags). If this is for something
like help desk support, you can make it work (I did at a previous job),
but setting up a real ticket system isn't that hard and is a lot easier
to use.
--
John A. Tamplin Unix System Administrator
Emory University, School of Public Health +1 404/727-9931
Re: APPEND vs RFC2822 vs STD0011
Edward Reid wrote: What RFC3501 says is The APPEND command appends the literal argument as a new message to the end of the specified destination mailbox. This argument SHOULD be in the format of an [RFC-2822] message. Since it's the client that constructs the literal, it appears to me that the RFC is giving the client the choice. The language appears to specify only the client/server combination. Obviously it would be better if the language were clearer, but it's not. In other places, the RFC explicitly refers to the server, for example, the lines immediately after the above: My interpretation is still that it is describing the protocol and that SHOULD in this case allows (but certainly does not require) the client or server to relax this restriction where they see fit. If the server is allowed to not relax this restriction, then clearly the client has to be prepared to deal with a response from a server that doesn't. I agree there should be clarification there, but as written I don't see how it can be interpreted as the client MAY choose not to send an RFC2822 message and the server MUST be prepared to accept a message that is not in RFC2822 format. Surely if such a strong requirement of the server was intended it would have been said, rather than SHOULD which is intended to give the implementation room to relax requirements if necessary. 8-bit characters are permitted in the message. A server implementation that is unable to preserve 8-bit data properly MUST be able to reversibly convert 8-bit APPEND data to 7-bit using a [MIME-IMB] content transfer encoding. Note: There MAY be exceptions, e.g., draft messages, in which required [RFC-2822] header lines are omitted in the message literal argument to APPEND. The full implications of doing so MUST be understood and carefully weighed. I had not previously noted the implications of that last paragraph. It implies that even the headers are not necessarily required to be RFC2822 compliant. But it's even less clear about whether the client or the server makes the choice. But again, I'm not personally concerned about headers. The formal syntax is no help in this case, because it includes no restriction beyond literal. The addition of this example seems to me to strengthen the SHOULD be an RFC2822 message statement, since it clearly suggests that even a reasonable exception (involving only missing header lines but maintaining the basic format) requires careful consideration of the implications. Regarding bare newlines, RFC3501 2.2 states: All interactions transmitted by client and server are in the form of lines, that is, strings that end with a CRLF. The protocol receiver of an IMAP4rev1 client or server is either reading a line, or is reading a sequence of octets with a known count followed by a line. This clearly does not mean that a literal cannot contain CR or LF -- in fact, in general the APPEND literal will always contain CRLFs which do not end the literal. The literal is prefix-coded with the octet count, and so the rule about lines ending in CRLF does not apply until the octet count is exhausted. The part of 2.2 that applies to this situation is the last line, reading a sequence of octets with a known count. Yes, sorry -- from RFC2822, 2.1: Messages are divided into lines of characters. A line is a series of characters that is delimited with the two characters carriage-return and line-feed; that is, the carriage return (CR) character (ASCII value 13) followed immediately by the line feed (LF) character (ASCII value 10). (The carriage-return/line-feed pair is usually written in this document as CRLF.) I guess there is still the debate of whether a client should expect the server to store something that isn't an RFC2822 message, and on that I guess we will have to disagree until the next IMAP RFC is written and clarifies it. -- John A. Tamplin Unix System Administrator Emory University, School of Public Health +1 404/727-9931
Re: APPEND vs RFC2822 vs STD0011
Edward Reid wrote: The mail provider (MX) for my domain, fastmail.fm, runs cyrus. I use Eudora (for Mac, v5.2), mostly in POP mode, but I use some IMAP features too. In particular, some of my filters copy incoming (POP) messages to an IMAP mailbox at fastmail.fm. That's where the problems start. Some of these incoming messages contain NULs or bare CR or LF. Yes, the sender is broken as far as RFC2822 is concerned, but the messages get through anyway. And the messages are valid RFC822/STD0011 format. When Eudora tries to copy these (APPEND them) to the IMAP mailbox, cyrus (at fastmail.fm) returns an error. I could live with an occasional copy failure, but the worst part is that when Eudora gets the server error, it thinks it's a terrible problem and throws up a dialog box and ceases all processing. Since I (like many people) depend on Eudora cleaning up my mailbox and doing other things with incoming mail automatically when I'm not at my desk, this gets to be a serious problem. That's what sieve is for -- do it in the server and you won't have to rely on a particular client doing it for you. So I started reading RFC3501 to find the reason. I assumed that I'd find a good reason that I could quote to Eudora support, telling them why Eudora has to clean up the message before storing it in an IMAP mailbox. But I didn't find that. What I found -- under the APPEND command (section 6.3.11) -- is : The APPEND command appends the literal argument as a new message : to the end of the specified destination mailbox. This argument SHOULD : be in the format of an [RFC-2822] message. Note well: that's SHOULD, not MUST. This is important. RFC2119 gives the meaning of SHOULD: : This word [...] mean[s] that there may exist valid reasons in : particular circumstances to ignore a particular item [...] So based on my reading of the RFC, it's the client's choice: it should normally append RFC2822 messages, but if it has a valid reason, it's allowed to append something that's not RFC2822. Now, IMAP mailboxes are intended for email -- Internet message format or Internet text messages in the RFC language -- and so it would be hard to make a case for storing anything that's not such a message. But RFC822 messages are still rampant on the Internet. In fact, as I understand it, although RFC2822 has obsoleted RFC822, STD0011 (which is identical to RFC822) is still a standard and has not yet been superseded. And it certainly seems to me that making a copy of an existing message is a valid reason for copying it intact, without the modifications needed to force it to conform to the stricter format of RFC2822. Since RFC3501 leaves this decision up to the client, it follows that cyrus is broken when it refuses the message. If RFC3501 said MUST, then I'd say it's Eudora's responsibility to fix the message before attempting an APPEND. But the RFC says SHOULD. Is there any good argument for cyrus' action? If there is, I'd be happy to take it to Eudora and push them to fix Eudora. Eudora's not exactly known for its stellar IMAP support, and I'd like them to improve this. I've shoved the RFCs in their face plenty of times in the past. But in this case, my reading of the RFCs is that Eudora's APPEND action is defensible and cyrus' action is incorrect. I disagree - it sounds like it would be defensible if Cyrus supported storing such messages even though it is clearly recommended against by the standard. If Eudora insists on storing such messages, it should be prepared to deal with a server that is unwilling to do so. If you allow the IMAP server to store arbitrary data, it makes the other functions much more difficult -- if it can't assume there is a message-id that is globally unique, it has to create its own unique key for a message. Searching based on header fields is more problematic since you can't assume there is even a header or that if there is one that it's value corresponds to anything you might expect (ie, character set, line length, and other issues). Allowing null characters in particular is problematic for any code that uses null-terminated strings for messages or parts of messages, and would require changing the code everywhere to use and pass the length of all the strings instead. I don't know if there is a technical reason behind not supporting bare newlines or not. As far as STD0011 not being obsoleted, there are plenty of RFCs etc. that are not obsoleted by something but are still not best current practice. Clearly if the RFC it has been based on has been obsoleted, the STD should be updated as well. -- John A. Tamplin Unix System Administrator Emory University, School of Public Health +1 404/727-9931
Re: quota warning problem - Is it a bug of cyrus imap?
Joe Rhett wrote: On Wed, Jul 02, 2003 at 12:29:41PM -0500, Paul M Fleming wrote: Rounding error would present a problem for us and our users. My 2 cents .. quotas 4GB are fairly rare. I know with a large student user base with 50-75Mb quotas rounding to the nearest K wouldn't be desirable. Okay, color me confused. Say your mailbox quota is 50mb. Isn't that exactly 51200k? Or do you really need to give some users 5,121,133 byte quotas? Do you really manage your quotas down to less than 1 kilobyte, when you are giving the users 50 megabyte boundaries on the low side? Or am I missing your point entirely? The problem is if you keep only the count of kBytes in a mailbox, what do you do when you receive a message of 512 bytes? Do you not count it at all, or do you count it as 1kB? If you keep actual bytes, you still have the problem of it fitting in a 32-bit number, and if you round it either way the quota will eventually be so wrong it is useless. -- John A. Tamplin Unix System Administrator Emory University, School of Public Health +1 404/727-9931
Re: 4.3.0 lmtpd: Virtual memory exhausted - problem :(
Pascal Gienger wrote: I am using Cyrus IMAP 2.1.12 (Suse Linux package, no source install allowed here), and we see some errors from lmtpd saying Virtual memory exhausted. ulimits are unlimited for every resource (no limit), and there are 4 Gigabytes of swap used by 5% and 2 Gigabytes of RAM. System is GNU/Linux 2.4.20-64GB. Our postfix mailer tries to deliver a message to a local lmtpd (Cyrus system) and here is the result: Jun 30 07:38:43 rauch postfix/lmtp[3129]: EC7FD4400088: to=[EMAIL PROTECTED], orig_to=[EMAIL PROTECTED], relay=/mail/cyrus/socket/lmtp[/mail/cyrus/socket/lmtp], delay=142730, status=deferred (host /mail/cyrus/socket/lmtp[/mail/cyrus/socket/lmtp] said: 421 4.3.0 lmtpd: Virtual memory exhausted (in reply to end of DATA command)) Has anybody seen this already? Which memory lmtpd claims to be exhausted? With no process limits at all? What version of DB are you running? Does this happen immediately or only after it has been running a while? On Solaris 9 with DB 4.1.24-25, we had a problem where DB functions would eventually return ENOSPC, which might be interpreted as Virtual memory exhausted. There has been a patch posted to Cyrus to fix this problem, but if you can't use a source package you might have to downgrade to DB 4.0.14 if this is your problem. -- John A. Tamplin Unix System Administrator Emory University, School of Public Health +1 404/727-9931
Re: cyradm erros when reconstructing a mailbox
Ezsra McDonald wrote: Let me go to another account with the same problem. All af the folders exist. Some are empty and reconstruct reports an IOERROR when it can't find cyrus.* in the folders. He has several folders whose names are made up of multiple words like for example one is named 'user.randy.Child Development'. It is also an empty folder. Permisions on all files and folders are setup for cyrus:mail. If the folders are empty (no cyrus.* files) but still in the mailbox list, it sounds like your copy did not move everything over properly or else the mailbox db doesn't correspond to the spool files (empty directories can be left around when deleting a folder if the folder is selected when it is deleted). Aside from these particular accounts, I would be concerned that other things were not copied correctly. I am spending too much time on this. What I would like to do is just remove all references to both users from the imap databases. Is there a way to do this. Then I can recreate a clean account and have the users move their mail over manually. With cyrus shut down, do ctl_mboxlist -d file, edit the file, the ctl_mboxlist -u file. -- John A. Tamplin Unix System Administrator Emory University, School of Public Health +1 404/727-9931
Re: Sieve failure on deliverdb corruption...
Rob Siemborski wrote: But its not even an issue since duplicate_init is called in service_init. (Before the lmtpd is handling a connection). So the MTA will just see connection reset by peer and defer it, right? If so, that behavior seems fine. I didn't have any problem noticing it before when I was having the problem with db 4.1.25, since all my mail that was getting sorted into folders started getting dumped in my inbox. Enough lists like this one, and the mean time to detection is vanishingly small :). -- John A. Tamplin Unix System Administrator Emory University, School of Public Health +1 404/727-9931
Re: Cyrus on Red Hat Enterprise Linux
Craig Ringer wrote: [OT: Yes, I'm aware that there other other Linuces beyond RH, but we're committed to HP hardware which is only certified for RH and SuSE (one of my colleagues has been told by an HP engineer that they support Debian but I've yet to see anything official). We have zero SuSE experience in-house, so RH kind of have us by the danglies...] That sounds depressingly like you're talking about MS or some custom software vendor, not Red Hat. The price of successful business, I guess. He is referring to HP certifying the software, which I am sure has more to do with demand than anything devious. HP isn't going to go to the trouble to support some version that not many of their customers are using. I am not sure why you would fault RedHat for HP not certifying other distributions besides RedHat and SuSE. -- John A. Tamplin Unix System Administrator Emory University, School of Public Health +1 404/727-9931
Re: Can't delete old folders from before ALTNAMESPACE
Lee wrote: You should be able to delete these from within cyradm as an admin, unless somebody deleted stuff by hand from the filesystem. I think that might be the problem. When i try to SAM the folders pre-deletion in cyradm, i get a: setaclmailbox: admin: lcp: System I/O error Is there a way to force remove cyrus' internal list of those folders? If you deleted them without Cyrus knowing about it, your best bet is to take Cyrus down, dump the mboxlist (ctl_mboxlist -d file.txt), edit the text version of the file to remove things that are no longer there, and then undump the mboxlist (ctl_mboxlist -u -f file.txt), and then bring everything up. You really should not be deleting things under Cyrus's control except through Cyrus. -- John A. Tamplin Unix System Administrator Emory University, School of Public Health +1 404/727-9931
Re: Can't delete old folders from before ALTNAMESPACE
Lee wrote: Will this cause any problems with seen / unseen flags (or anything for that matter)? Seen flags are kept in a different database, under /var/imap/user/X/user.seen. I presume Cyrus gracefully deals with nonexistent folders in the seen/sub files, but I haven't looked at that part of the code so conceivably that might be a problem. If you want to be sure, restore the missing directories from backup and delete them the proper way though Cyrus. -- John A. Tamplin Unix System Administrator Emory University, School of Public Health +1 404/727-9931
Re: migrating from sendmail/shadow to sasl on new box
blast wrote: As far as I've been able to find, the only way to migrate use info from an old box running sendmail authenticating against shadow to a new system running postfix authenticating against sasldb2 is to use the auto_transition feature. Is this true, or is there perhaps a manual_transition feature that I might employ? If you have their clear-text password, or are willing to reset the passwords, it would be trivial to write a script to do it. Otherwise, you do not have sufficient information (shadow only stores a one-way hash of the password, using crypt or md5 depending on your OS and configuration) to regenerate the clear-text password that is required. -- John A. Tamplin Unix System Administrator Emory University, School of Public Health +1 404/727-9931
Re: Cyrus Backup Strategy
pnelson wrote: My last thing to do prior to converting to production is a backup strategy. I have been doing this with tar something like: tar -C /var/lib-czf lib-date.tar.gzimap tar -C /var/spool -czf spool-date.tar.gz imap tar -cf cyrus-date.tar This is producing a pretty big file(s): lib-date.tar.gz- 2.2M spool-date.tar.gz - 11.0M and ultimately cyrus-date.tar - 13.5M and I was thinking maybe there is a better way that someone else has come up with. Anyone doing backups a different way? Are you then dumping these files to tape? Otherwise, just having another copy on disk doesn't protect against many potential causes of data loss. We just backup the mail files normally with Veritas Netbackup (be sure to disable true image restore with the millions of files), with the only custom bit being to dump a copy of the mailbox list to a text file before the backups run (this is a precaution since we are backing up structured database files without synchronization with the program writing them, so there is no guarantee the resulting backup is useful). Our full backup for all Cyrus files is 98G and 4.2M files, taking 5-8 hours depending on other data hitting the tape drives. Daily incremental backups average 1-3G and 10-30k files. -- John A. Tamplin Unix System Administrator Emory University, School of Public Health +1 404/727-9931
Re: Compiling Cyrus IMAPD on AMD Opteron
[EMAIL PROTECTED] wrote: make[2]: Entering directory `/root/src/cyrus-imapd-2.1.13/perl/imap' rm -f blib/arch/auto/Cyrus/IMAP/IMAP.so LD_RUN_PATH=/usr/lib64 cc -shared -L/usr/local/lib64 IMAP.o -o blib/arch/auto/Cyrus/IMAP/IMAP.so ../../lib/libcyrus.a -lssl -lcrypto /usr/lib64/gcc-lib/x86_64-suse-linux/3.2.2/../../../../x86_64-suse-linux/bin/ld: ../../lib/libcyrus.a(imclient.o): relocation R_X86_64_32 can not be used when making a shared object; recompile with -fPIC ../../lib/libcyrus.a: could not read symbols: Bad value collect2: ld returned 1 exit status make[2]: *** [blib/arch/auto/Cyrus/IMAP/IMAP.so] Error 1 make[2]: Leaving directory `/root/src/cyrus-imapd-2.1.13/perl/imap' make[1]: *** [all] Error 1 make[1]: Leaving directory `/root/src/cyrus-imapd-2.1.13/perl' make: *** [all] Error 1 [snippet] Can anyone help me with it? I wanted to get it working to test out the performance of the system. What you need to look at is where imclient.c was compiled. Either it wasn't compiled with -shared, or the gcc port is not properly choosing the right flags for -shared. Run the cc command yourself in that directory, adding the -v flag and see if -fPIC gets passed to cc1. If not, you can fix the problem just by adding -fPIC yourself in the compile commands. If it is getting passed, then it looks like the machine definition is improperly choosing a particular addressing mode when generating PIC code and that will need to get fixed by whoever maintains the x86_64 machine description for gcc. -- John A. Tamplin Unix System Administrator Emory University, School of Public Health +1 404/727-9931
Re: special imap server
Brasseur Valéry wrote: did someone know if there exist a IMAp server who would be able to use a POP3 server for the INBOX folder and his normal folders for all others ! note : i need something like this for a migration purpose ! The basic problem is that you cannot implement the IMAP protocol using the functionality provided by a POP server. Conceivably, you could have a proxy that did what you wanted and provided a very restricted subset of IMAP commands, but you would likely find clients would not work with it. You should just bite the bullet and convert them at once, offline. -- John A. Tamplin Unix System Administrator Emory University, School of Public Health +1 404/727-9931
Re: Weird pop3d hang problem (fd blocked?!)
foobar wrote: See word *theoretically* , didn't urandom gather some data from network-interfaces too so it may be affected. Nobody knows when it takes data from device nr X. My point was simply before you decide to link random to urandom for the sake of Cyrus, you should consider the impact that will have on other applications that need random numbers. If others have access to your machine and you are generating private keys, they could exhaust all the entropy from /dev/random, read enough of /dev/urandom to determine the position in the sequence, and then know what random numbers your key generation code used. Granted, it is far-fetched and a lot of work, but when you are building a key that will be used for years and could compromise other keys if revealed, it pays to be safe. /dev/urandom appeared in solaris since version8 (patch). random's device-number is 8 while urandom's is 9. What about if there is config-option for this device? When you build SASL, just define -DDEV_RANDOM=/dev/urandom. -- John A. Tamplin Unix System Administrator Emory University, School of Public Health +1 404/727-9931
Re: Several issues with 2.1.13
Wil Cooley wrote: The upgrade wasn't entirely smooth; I wrote up some notes about what I did: http://nakedape.cc/wiki/index.cgi/CyrusImapNotes in case someone else wanders along this path... The biggest issue was that ctl_cyrusdb wasn't able to read my old mailboxes.db file; I reverted to my old installation and dumped the database to a flat file and it went okay. For some reason that I don't understand, I had to remove the /var/lib/imap/db directory for the rebuilding of the database to work correctly. If the version of db is different, you can't just expect to use the binary database files and logs. Dumping the contents to a text file, wiping the transaction logs, and then reloading them is the safest way. 'rehash full' did very strange things; it only created directories of A-Z, none of a-z and my own mailbox information was under 'I/' in both the mailbox spool and the '/var/lib/imap/user' directory. As a result, I had to disable 'hashimapspool', which Simon's RPMs enabled by default. You should have had A-W, not Z (23 is a prime to give better distribution of the hash values between directories) -- full hashing chooses a hash directory based on the complete mailbox name rather than just the first character. Traditional hashing tends to overload some directories while leaving others almost empty. mbpath will give you the path for the mailbox, or just ls -ld /cyrus/*/user/wcooley (for example). If you have less than a thousand mailboxes, there is no need to worry about hashing. Otherwise, you will likely get poor performance with long namei lookups. Finally, I have a customer that's a small ISP that's currently running 2.0.16. I'm going to upgrade regardless, just so I can bounce messages delivered through LMTP to boxes that are over-quota. However, I recurrent problem we have is with POP3 users (which everyone is) who lose their connection (usually because of problems with dial-up). The pop3d stays running and locks the mailbox for 15 minutes or so, causing lots of support calls and grumbling. I'm guessing the connection stays in TIME_WAIT for this period, but 15 minutes seems like a long time for it to stay open. I see the 'poptimeout' setting that might help, but even 10 minutes might be too long. (This 15 minutes could be only 10 minutes already; I'm just being told this by the guy who does support.) Will anything that's changed between 2.0.16 and 2.1.13 help assuage this problem? In this day, is there really any good reason to continue using POP? There are so many problems with it (including support issues when people downloaded their mail to one computer and wonder why they can't access it from another) it seems best to retire it. When the user calls up complaining about this, that gives you a perfect opening to convince them to move to IMAP. -- John A. Tamplin Unix System Administrator Emory University, School of Public Health +1 404/727-9931
Re: Several issues with 2.1.13
Wil Cooley wrote: Yeah, you're probably right; I just saw a directory full of capital letters and assumed it was all of them. I do recall prime-based hash functions being better. However, after I did the hash, imapd seemed to still be using the traditional first-letter-hash. I had this in my logs and couldn't open my mailboxes: Jun 8 17:51:46 rheingold imapd[14729]: IOERROR: opening /home/imap/w/user/wcooley/cyrus.header: No such file or directory Then it sounds like your imapd was not compiled with --enable-fulldirhash. The argument to rehash should match however your imapd was compiled (and using someone else's package may make it hard to tell). Our customers have a 10MB mail quota which they can fill up pretty quickly. We don't want them leaving their stuff on our server, and they want to be able to have their mail even when they're off-line (I know there's IMAP sync, but it's questionable how well supported it is, and I don't really want to try to explain it to customers). Your call, but even when I worked at an ISP we had far more problems with customers who used POP than those that didn't. Also, reducing the Cyrus timeout isn't going to help clear the half-connected sockets in the kernel if the customers just disconnect without dropping all their TCP connections first. -- John A. Tamplin Unix System Administrator Emory University, School of Public Health +1 404/727-9931
Re: Several issues with 2.1.13
Wil Cooley wrote: Got it. But John's probably right about the kernel keeping the sockets open. But we never had this problem with cucipop (that's what I always have to listen to: We didn't have this problem before blah blah blah). Somehow, even though cucipop locked the mailboxes, it was able to operate concurrently on them (somewhat), without stupidly making a whole copy, the way UW's pop3d did. Yikes -- I know nothing about cucipop, but allowing multiple POP clients to operate on the same mailbox at the same time is begging for problems, since the protocol doesn't have any of the features that would be needed to support concurrent access. Maybe they assumed that when that happened that it was because the other connection was dropped and they automatically dropped it (releasing any locks held), but that still seems very dangerous. -- John A. Tamplin Unix System Administrator Emory University, School of Public Health +1 404/727-9931
Re: Subject field corruption
NOwlar wrote: I use Postfix 2.0.10 with TLS patch, cyrus-imapd 2.1.13, sasl 2.1.13 The problem is as follow: When a message with russian charset characters (KOI8-R, WIN1251) in Subject field arrives or is moved into the Sent Items folders by Outlook Express, each russian character is replaced by an X (ex: Re: mazda 626 ). I happens only when the Subject field is without charaterset specification. I dumped mailboxes with cyrdump, the Xes are in. What can be wrog? The problem is that 8-bit characters are not allowed in the header of an RFC822 message. There are extensions designed to allow 8-bit characters, by encoding the character set and a base64 or quoted-printable representation of them. The problem is that Cyrus cannot properly transform these messages to standards compliance since it cannot know what character set was used, only that it wasn't US-ASCII, so Cyrus chooses to either drop the message or replace all 8-bit characters with X depending on a configuration setting. There was some discussion of hacking Cyrus to allow an administrator-supplied default character set in the case of 8-bit data without a character set, but the consensus of the developers seemed to be that was more an MTA function than Cyrus, and nobody felt like doing a hack that would not be accepted into the official source. -- John A. Tamplin Unix System Administrator Emory University, School of Public Health +1 404/727-9931
Re: Weird pop3d hang problem (fd blocked?!)
Rob Siemborski wrote: You can symlink /dev/random to point at /dev/urandom, but keep in mind that will affect all applications, not just cyrus. And in particular you may not want to do this if you are generating RSA private keys or equivalent on a machine that anyone else may have shell access to. I would think that whoever packaged it chose poorly. -- John A. Tamplin Unix System Administrator Emory University, School of Public Health +1 404/727-9931
Re: Linking imapd with db-4.1.25
Lars Peterson wrote: Snippet from configure's output: === checking for db.h... yes checking for db_create in -ldb-4.1... no checking for db_create in -ldb4.1... no checking for db_create in -ldb41... no checking for db_create in -ldb-4.0... yes === When I do an ldd on the resultant imapd binary after the build finishes, I see that it been linked to the bdb libraries that came with the RedHat install (4.0.14-20). I've played around with LDFLAGS, CPPFLAGS, and LD_LIBRARY_PATH environment variables to try and force the directories of my non-standard libraries and headers to be loaded first, but that didn't change a thing. I'm reluctant to remove the RedHat db4 installation because I quite certain that it would break a large amount of dependencies... Any ideas? In the directory you passed for bdb-libdir, is there a libdb-4.1.so? Are permissions on it set properly? BTW, you may be happier using 4.0.14. On Solaris anyway, I consistently ran into problems after a week or two of uptime with 4.1.25, and that all went away after downgrading to 4.0.14. -- John A. Tamplin Unix System Administrator Emory University, School of Public Health +1 404/727-9931
Re: Linking imapd with db-4.1.25
Lars Peterson wrote: Yes libdb-4.1.so is in the same directory I passed to configure for bdb-libdir. Permissions are same user I'm building as, and mode is 775. I was able to link against these libraries successfully for an openldap-2.1.19 build. Then I suggest looking at the log generated by configure and see why it didn't think it passed the test. I'll take this into consideration. I've had no problems thus far with db-4.1.25 with respect to openldap / slapd... Right, I haven't had any problems in any other application either. I don't know if that is because they don't get as heavy usage or if the usage pattern is different or what. What kind of problems were you seeing on your Solaris box? After a week or two of uptime, I would get ENOSPC returns from dbopen, and sieve would stop working (since my duplicate delivery database is using db3_nosync). Other people reported similar problems with 4.1.24-25 in this list without any resolution, until someone suggested to me that going back to db 4.0.14 would fix the problem. -- John A. Tamplin Unix System Administrator Emory University, School of Public Health +1 404/727-9931
Re: Linking imapd with db-4.1.25
Igor Brezac wrote: This problem is fixed in cvs for both 2.1.x and 2.2. Ok, so it was a problem in Cyrus not db? Thanks. -- John A. Tamplin Unix System Administrator Emory University, School of Public Health +1 404/727-9931
Re: Linking imapd with db-4.1.25
Lars Peterson wrote: Here's some output from config.log: snip configure:5808: checking for db_create in -ldb-4.1 configure:5839: gcc -o conftest -Wall -g -O2 -I/usr/local/itech/iwaymail/include -L/usr/local/lib -Wl,-rpath,/usr/local/lib - L/usr/local/itech/iwaymail/lib conftest.c -ldb-4.1 -L/usr/local/itech/iwaymail/lib -lresolv -Wl,-rpath,/usr/local/itech/iw aymail/lib 5 /usr/local/itech/iwaymail/lib/libdb-4.1.so: undefined reference to `pthread_condattr_setpshared' /usr/local/itech/iwaymail/lib/libdb-4.1.so: undefined reference to `pthread_mutexattr_destroy' /usr/local/itech/iwaymail/lib/libdb-4.1.so: undefined reference to `pthread_mutexattr_setpshared' /usr/local/itech/iwaymail/lib/libdb-4.1.so: undefined reference to `pthread_mutexattr_init' /usr/local/itech/iwaymail/lib/libdb-4.1.so: undefined reference to `pthread_mutex_trylock' collect2: ld returned 1 exit status configure:5842: $? = 1 snip Any ideas where I might find some support for db-4.1.25? Can't find any mailing lists or news feeds on sleepycat's site. Ah, you are running into fun with RH9's NPTL (New Posix Threads Library). If you look at the archives for this list you will find some discussion about it. I believe the quick and dirty solution was to compile DB4 with a flag telling it to assume phtreads were not available, but Simon also posted a link to a discussion at RedHat about more general issues with NPTL. The particular errors you are getting are because it can't find the pthreads library when it does the test. You can fix that by telling configure to use -lpthreads on any compiles, but if I remember correctly you will then run into other issues when DB4 tries to do some operations on the threads. I am staying with RH8 for my production machines because of this and a few other minor issues. -- John A. Tamplin Unix System Administrator Emory University, School of Public Health +1 404/727-9931
Re: cyrus server and backup
Phil Chambers wrote: I have never used the Cyrus IMAP server before and have just intalled it on a test system to evaluate it, with a view to moving our service over to it. We have over 20,000 users and need to upgrade our current system. One of our fundamental requirements is to be able to take security backups. These are primarily for recovery in case of hardware failures. The backups are conventional periodic full backups with intermediate incrementals. I am concerned that because Cyrus is a black box system which keeps track of its own internal organisation we may have problems if we restore a disc from our backups. It will take hours to do a backup and the files within the Cyrus structure will be changing as we do it. Are there going to be problems with inconsistencies between files? There is a secondary, but important use of our current backup service, which is to dig users out of a hole when they make mistakes: Occasionally a user will accidentally delete a message or even a whole folder and then come and ask if I can recover it for them. With our current backup system it is ussually very easy because I have no problem identifying the relevant files to be recovered. I seems to that it will be impossible to recover deleted messages because I will not be able to identify the files which I need. If I can identify the files, presumably there is no way to get them back into the Cyrus system? The only information I have been able to find which says anything about recovery is in the Server Overview and Concepts document. However, this does not go into sufficient detail and includes statements like Many objects in the configuration directory are discussed in the Database Recover section. I cannot find any section anywhere with a title of Database Recover! Any advice will be much appreciated. Search the list archives for detailed discussion of this topic. What we do is folder-level restores (we haven't had to restore more than one at a time yet -- the same approach works but it would be a lot of work), and we create a top-level folder that will be the restore target, restore the contents of the directory containing the user's folder that needs to be restored into the directory containing the restore folder, reconstruct that folder, and grant access to the user. It is up to the user to do whatever they want, presumably copying messages that were deleted accidentally. After one week, the restore folder is deleted. We decided to use a top-level folder rather than put it under each user to avoid quota issues (we could set a quotaroot on the restore subtree, but it is easier this way) and to make it easier to clean up expired restore folders. If this were a non-user error (we haven't had one of those), we would probably just restore the whole cyrus directory structure (depending on the damage) if reconstruct couldn't fix it. -- John A. Tamplin Unix System Administrator Emory University, School of Public Health +1 404/727-9931
Re: message in /var/log/messages
Nicolas Gauvrit wrote: i post this message yesterday, but i got no answer... (is my question to stupid for you ?) i'd like to know what does mean those message, is i do something wrong ? is my Cyrus Imap will work fine, even if i got thoses Error messages ? i was searching several hours, but i do not found anything yet... Apr 1 07:04:29 ilemadame master: setrlimit: Unable to set file descriptors limit to -1: Operation not permitted Apr 1 07:04:29 ilemadame master: retrying with 1024 (current max) Apr 1 07:04:40 ilemadame lmtpd[1177]: DBERROR db4: 7 lockers Apr 1 07:04:40 ilemadame pop3d[1176]: DBERROR db4: 8 lockers These are just warnings about lock contention, but you should probably switch from Berkeley DB to skiplist for your mailboxdb. The setrlimit failure is just warning you that it was unable to set unlimited file descriptors, but unless you have 1000 simultaneous connections you don't need to worry about it. -- John A. Tamplin Unix System Administrator Emory University, School of Public Health +1 404/727-9931
Re: sieve: Too many levels of symbolic links
Ken Murchison wrote: FYI, Websieve does this automatically when no script is present, unless the user overrides it. The administrator can change that configuration in websieve.conf, variable $defaultscript. -- John A. Tamplin Unix System Administrator Emory University, School of Public Health +1 404/727-9931
Re: interesting limitation
Dave O wrote: 2 level hashing would work, but I don't know if Cyrus supports that. It would most likely be trivial to implement. eg spool/s/sm/user/smith Or in the case of full dir hashing, have a second hash function and hash the names that get assigned to one bin into an additional set of bins. As was previously mentioned, having multiple partitions also solves this problem but since the trend seems to be consolidating storage it would be nice to be able to handle the large numbers of users in a single partition rather than creating multiple partitions in the same filesystem with the associated administrative hassle. -- John A. Tamplin Unix System Administrator Emory University, School of Public Health +1 404/727-9931
Re: Bare newlines
Sebastian Konstanty Zdrojewski wrote: I was attempting to move an email message from a local account onto an IMAP directory located on Cyrus. The client I am using is Netscape Mail and usually it gives me no problems. This time, I am not able to move the message because the client gives the following error: Unable to perform operation. Message contains bare newlines. The message has an attached PDF file (2Mb size) and a signature with image integrated. Any idea of what can be my problem? In IMAP, messages are required to have \r\n as line terminators, not just \n. Many other IMAP servers (or mail clients' local folders) do not enforce strict RFC compliance, while Cyrus does -- when we converted from UW-IMAP to Cyrus I had to write code to munge messages into the proper format (8 bit or null data in headers, and bogus headers are the other two issues you will likely run into). -- John A. Tamplin Unix System Administrator Emory University, School of Public Health +1 404/727-9931
Re: Problem with cyradm crashing
Marco Colombo wrote: 2) the old cyradm binary, from a 2.0.16 installation, never crashes. (that made me turn to the client side instead of the server side) The old binaries were compiled vs. SASL v.1.5.26 (shared), which has been recently upgraded to 1.5.28. Cyrus 2.x requires SASL 2.x -- if you are running it with SASL 1 perhaps that is your problem. See install-prereq.html in the documentation. -- John A. Tamplin Unix System Administrator Emory University, School of Public Health +1 404/727-9931
Re: I tried chinese in cyrus's folder name and chinese IMAP client,but something wrong
Temp wrote: I tried chinese in postfix+cyrus-imap and in Outlook Express 5, the Inbox is still the one in chinese, but the Sent Items and Drafts folders which I saw in english IMAP client become normal folders, and there are another folder instead of them in chinese in chinese name. so, does IMAP server not provide a uniform interface? or just the problem of IMAP protocol? or just my fault to set up something wrong?? yes, the Inbox is good at all in chinese. Inbox is the only special folder in IMAP -- the others are just conventions used by the mail clients. If two different clients (OE/English and OE/Chinese) decide to name them differently, there is nothing the IMAP server can or should do about it. -- John A. Tamplin Unix System Administrator Emory University, School of Public Health +1 404/727-9931
Re: Outlook Express and Seen database - And Outlook Express and Cyrusin general
Steve Hanson wrote: There was some discussion on the list in the past about problems with Outlook Express not interoperating well with Cyrus due to it corrupting the seen database by using multiple concurrent connections, confusing the caching model in Cyrus. Our desktop support people are trying to stump up support for making Outlook Express on Windows be the preferred email client - which at the moment is more or less Eudora. Does anyone have opinions about whether Outlook Express 6 still causes the seen database corruption with Cyrus 2.1.10 or later? Also if anyone has any other bad (or good) experiences with Outlook Express as a client with Cyrus I'd like to hear about them. I don't know of any discussion of database corruption, but the problem is that Outlook would be confused about the seen flags since it uses two different IMAP connections to process them and Cyrus keeps all that in memory per-process. The only corruption issue I have seen is that twice we had a user being unable to delete messages from a folder that was perfectly fine (and other IMAP clients could delete messages without a problem), and after deleting the subscription and resubscribing the problem went away. Research suggested this was a race condition in OE that had nothing to do with the IMAP server other than perhaps participation in the timing aspects of the race condition. It was not reproducable and has not happened in over 2 months. (This is with a user base of 2300 users, ~250 connected at any one time, and 90G spool space). Most of our users use OE6, and there have been no issues I am aware of in the 3.5 months we have been running it since I applied my patch for flushing the in-memory seen flags state to disk whenever it changes and checking the seen flags on disk for updates before replying to the client. This is with 2.1.11 and 2.1.12, and if you need the patch I would be happy to send it to you. Note that this will increase I/O traffic so you may not want to apply the patch if you have little margin for I/O bandwidth. Our server is so lightly loaded (it was sized to handle the load when we were running UW-IMAP) we didn't see any noticable difference, but I know the traffic will be higher. -- John A. Tamplin Unix System Administrator Emory University, School of Public Health +1 404/727-9931
Re: pam_mysql and cyrus_sasl
Andreas Meyer wrote: eta saslauthd[982]: pam_mysql: where clause = eta saslauthd[982]: SELECT password FROM accountuser WHERE username='karl' eta saslauthd[982]: pam_mysql: select returned more than one result eta saslauthd[982]: returning 7 after db_checkpasswd. eta saslauthd[982]: AUTHFAIL: user=karl service=imap realm= [PAM auth error] It looks like you have multiple rows in your accountuser table that have username='karl' (you should probably have a unique index on it anyway), or else accountuser is a view that is matching more than one row. The error message is telling you exactly what the problem is -- when it looks up the username in your mysql table, it is getting more than one row so it doesn't know what to use to validate the login. -- John A. Tamplin Unix System Administrator Emory University, School of Public Health +1 404/727-9931
Re: libcyrus and the imap proto
Patrick Welche wrote:
Overall I see that you send commands to the imap server using imclient_send,
the last argument of which is essentially the text of the imap command.
You register callbacks based on keyword, so that when the server sends you
a reply, the function registered gets called. I suspect I am getting confused
between tagged and untagged. The following is fine, and I think that
. capability - sent with imclient_send
* CAPABILITY IMAP4 IMAP4rev1 ... - untagged? capability response
picked up by callback based on
keyword CAPABILITY
. OK Completed - tagged? ok response picked up by
imclient_send's finishproc
but now for the problem
. examine inbox- sent with imclient_send
* FLAGS (\Answered \Flagged ...- untaged flags response, callback_flags
* OK [PERMANENTFLAGS ()] - untagged ok response, callback_ok
* 13 EXISTS * What about these two?
* 1 RECENT * untagged responses with no keyword?
* OK [UNSEEN 13] - untagged ok response, callback_ok
* OK [UIDVALIDITY 1043953684] - untagged ok response, callback_ok
* OK [UIDNEXT 15] - untagged ok response, callback_ok
. OK [READ-ONLY] Completed - tagged? ok response picked up by
imclient_send's finishproc
same for eg
. fetch 1 (internaldate)
* 1 FETCH (INTERNALDATE 30-Jan-2003 19:23:41 +)
. OK Completed
How can I see the result of fetch?
I have not used the C library, but the perl interface is similar so I
suspect this approach will work for you. What I did was add a callback
with an empty string for a trigger. The callback gets called on every
line, and you can parse it as you please.
$imap-addcallback({-trigger='',-callback=sub {
my [EMAIL PROTECTED];
if($a{-text}=~ /RFC822\.SIZE\s+(\d+)/i) {
$totmsg++;
$totbytes+=$1;
if($110) {
$largemsg++;
$largebytes+=$1;
}
$largest=$1 if($1$largest);
}
}});
($rc,$text)=$imap-send('','','FETCH 1:* RFC822.SIZE',$mbx);
--
John A. Tamplin Unix System Administrator
Emory University, School of Public Health +1 404/727-9931
Re: seen flag for multiple users // cyrus and procmail
Kristian Rink wrote: (a) in this network, IMAP is used because the vast majority of mail traffic needs to be seen not only by a single user but by a group of persons. For what I have experienced and also read in the archive of this list, it is not possible in cyrus (in IMAP?) to have a general seen flag for mails or mailboxes so this way it is impossible for any user who is able to read mail in the shared mailboxes to find out whether or not anyone else already has read and possibly answered new mail. Is there a way to work around this condition? Reading the cyrus man pages I hoped that the s ACL setting would be doing right what I need here but obviously it didn't work. IMAP seen states are kept per-user, but the other flags are persistent and visible across any users viewing the mailbox. So, you should definitely see that a given message has been answered, and by convention you can use a different flag for meaning seen by the group. -- John A. Tamplin Unix System Administrator Emory University, School of Public Health +1 404/727-9931
Re: Vacation for non-existant users?
Tarjei Huse wrote: I got a few users whom I've removed, and now I'd like to set a auto-responder for the addresses. I'd like to do this so people emailing them get their new address. Any suggestions on how to set this using sieve? If they don't have an inbox, I don't believe you can use sieve. Why not just use the standard vacation program, with a separate aliases file for old accounts? -- John A. Tamplin Unix System Administrator Emory University, School of Public Health +1 404/727-9931
Re: Backup of Cryus IMAP
Michael Frank wrote: I have heard that if I backup the database and/or user files as they are being accessed I might corrupt the files. Is this true? If so how can I implement an automated backup without taking the service down? I need to have the service up 24/7 and am unable to take it down at any point. Can anyone point me in the correct direction? Simply backing up the files as they are actively being used will not corrupt them. However, it is possible that the file you backup is not usable since there is no synchronization between the updates to the database and the backup program. The best practice discussed here for restoring inidividual folders is to create an empty folder, restore the message files themselves, and then reconstruct that folder, so there is no problem there. For a full restore, to guard against corrupting important database files I do ctl_mboxlist -d to a text file just before the backup, and then backup that file. If a full restore is required, the database can be rebuilt from the text file if necessary. -- John A. Tamplin Unix System Administrator Emory University, School of Public Health +1 404/727-9931
Re: Backup of Cryus IMAP (restore?)
Thomas Hannan wrote: So to backup Cyrus, I should do the following?: ctl_mboxlist -d /var/backup/mydbbackupfile cp -dPr /var/imap /var/backup/imap cp -dPr /var/spool/imap /var/backup/spool/imap Would it be better to tar them first, or just copy and then tar later? And to do a full restore? How would I do so? (I assume I'd have to stop the service to restore)... I have a cron job that does ctl_mboxlist -d /var/imap/mailboxes.txt and then I just let our normal backup software (Veritas Netbackup DataCenter) backup that file as it would any other. That way I have if the recovered mailboxes.db file was corrupt. If you have the disk space to do your backup on disk, you are probably better doing snapshots (mirror the volume, then break the mirror) than cp. -- John A. Tamplin Unix System Administrator Emory University, School of Public Health +1 404/727-9931
Re: informing users about quotas
[EMAIL PROTECTED] wrote: What is your custom delivery program? I'm interested in that kind of solution but i didn't found yet a deliver program for postfix that use lmtp and can add the ignorequota option. It is a small perl program that uses Net::LMTP (hacked to support authentication and the ignorequota option), which also does things like strip nulls and reformat envelope from addresses that contain quotes. I'll be happy to send it and the patches to Net::LMTP if you are interested. Beware since it uses a TCP socket rather than a Unix socket authentication is required and performance is slightly impacted. It also has code to proxy the LMTP connection to multiple destination mail servers, which we used along with perdition to transition accounts one at a time. -- John A. Tamplin Unix System Administrator Emory University, School of Public Health +1 404/727-9931
Re: Move Emails From a unix format to cyrus
Brandon High wrote: On Thu, 2003-02-06 at 18:11, test s wrote: Does anyone know how to Move Emails From a unix format which is on server A to cyrus which is on server B We migrated from WU-imapd to Cyrus last year... The easiest way is to pipe the mail into deliver. It takes a little custom shell/perl scripting, but nothing too tricky. With that approach, you lose all flags and seen state. The mbxcvt program from UW imap-utils worked fine for us (with a perl wrapper to handle the fact that we were doing the transition one user at a time via a proxy rather than all at once and shutting down as well as going through all the folders automatically), with some minor hacking to handle the passwords. I have been told that it now supports SASL so you can use auth plain and proxy as the administrator account rather than logging in as each user. -- John A. Tamplin Unix System Administrator Emory University, School of Public Health +1 404/727-9931
Re: saslauthd getpwent
RJ45 wrote: hello I noticed that getpwent method of saslauthd fails on solaris, it works with pam or shadow but not with getpwent. IS it normal for Solaris 9 ? or is there some trick to make it work? During the transition here, I was running saslauthd -a getpwent because it was easier to hack the backdoor password into the getpwent method rather than pam, and it worked just fine feeding from NIS+ on Solaris 9. Is you name service switch setup properly? Are you running NIS, or are you using local shadow files? -- John A. Tamplin Unix System Administrator Emory University, School of Public Health +1 404/727-9931
Re: looking for Cyrus mail format documentation
Phil Howard wrote: That would result in doubling the bandwidth on the inside server connection since it would be dealing with the mail first coming in to the MX, then being replicated back out to the other server. By delivering outside mail to the outside server first, the only bandwidth usage is replicating to the inside server (reverse the scenario for mail originating inside). Is the cost of bandwidth to your inside server really so expensive as to justify the expense of complicated development, hosting an offsite server with that much bandwidth, and maintaining a remote system? It really sounds like you are overengineering the problem. If there was a way to track when the flags got changed. I feel it's OK to trust the clocks on the servers, and simply decide which flag state prevails based on which has the later timestamp. But I bet that metadata isn't in the current mailstore design. No, the time a flag was changed isn't kept. In fact for seen flags which are cached in memory while a mailbox is open, only a single bit is kept. -- John A. Tamplin Unix System Administrator Emory University, School of Public Health +1 404/727-9931
Re: mailinglists
Peter Burggraef wrote: I want cyrus to delete mails, older then sometime. How can I do this? man ipurge -- John A. Tamplin Unix System Administrator Emory University, School of Public Health +1 404/727-9931
Re: looking for Cyrus mail format documentation
Patrick Welche wrote: All this sounds remarkably similar to the postgres-r database replication problem cf nice paper by Bettina Kemme http://www.cs.mcgill.ca/~kemme/papers/vldb00.html Here it would be client connects to imap server A and says APPEND. Server A then sends APPEND to server A and server B using a group communciation protocol (cf spread) which guarantees the ordering of the commands. Server A and server B then receive the APPEND and do it. If server B received an APPEND at nearly the same time, that APPEND would still appear in the same place in the input queue of both servers = the UID would come out the same. You still have the hard problem of conflict resolution after network partitioning :( The issue is not when the two servers can talk -- that is easily solved with techniques such as two phase commit. The problem is when server A and B are not able to communicate and you want both of them to be able to continue taking updates yet build a consistent view of the database once they can communicate. -- John A. Tamplin Unix System Administrator Emory University, School of Public Health +1 404/727-9931
Re: creating user-mailboxes without cyradm
Hans Wilmer wrote: Sure, read RFC 2060. You'd do tag CREATE user/username How is this dealt with in respect to security and reliability? Just write a script that logs in and automatically creates mailboxes from randomly generated (user-) names until the storage is full. That's sort of making DOS attacks utterly easy. Obviously you have to be authenticated as a user with privileges to create the folders. This is no different than saying you should connect to an IMAP server and append millions of messages -- the answer is still proper authentication and access controls. -- John A. Tamplin Unix System Administrator Emory University, School of Public Health +1 404/727-9931
Re: looking for Cyrus mail format documentation
Henrique de Moraes Holschuh wrote: On Tue, 04 Feb 2003, Phil Howard wrote: I wonder how well that method of replication works when both nodes cannot reach each other, and both are doing updates. And I wonder They don't. If they cannot reach each other, at most one of them must allow updates. Which, as far as I know, is how all commercial databases also do replication. Only those servers involved in a quorum (even if it is because the primary is treated as having more weight) can accept updates -- the other replicas can only be read-only. Trying to allow arbitrary updates during disconnected operation and then merge the results into a consistent and deterministic state is not just hard, but impossible. For example, if you allow flags to be set differently on the same message on each node, there is no way to resolve the two updates without discarding at least some of the work of one of the sessions. If you carefully constrain the updates that can be performed during disconnected operation (and for consistency that means even not in disconnected operation), then you can transform the problem from impossible to merely extremely difficult. -- John A. Tamplin Unix System Administrator Emory University, School of Public Health +1 404/727-9931
Re: looking for Cyrus mail format documentation
Earl R Shannon wrote: But that does now beg the question. There must be some form of coordination between the various processes as they access the mail store. Can this not be abstracted out and put in an API to make it easier for people to write their own applications? I would venture a guess to say that the API already exists in some form, it just needs to be formalized and published. The point is if you expose the internal API for accessing the mailstore you are stuck with it and can't make changes. I can't imagine there is a big need for this or other people wanting to write code to implement that API, so if you really want to do this it is probably better as Rob suggested to just link to the Cyrus code that manipulates it (and watch for version skew between programs accessing the mail store). -- John A. Tamplin Unix System Administrator Emory University, School of Public Health +1 404/727-9931
deliver db problem
I am running Cyrus 2.1.11 (Solaris 9, mboxlist/seen=skiplist, sub=flat, duplicate/tls=db3_nosync), and periodically sieve stops working after the server has been up a couple of weeks. Upon further investigation, I find it is some problem with the deliver.db file, as running ctl_deliver -d results in ctl_deliver: unable to init duplicate delivery database. Stopping and restarting master fixes the problem (perhaps because it runs ctl_cyrusdb -r?), and ctl_deliver -d dumps the database properly after this. No error messages appear in the logs (even logging at debug level) during normal operation, but trying to run ctl_deliver -d logs DBERROR: opening /var/imap/deliver.db: Not enough space. There is 39G free swap space (out of 40G), and the machine has 8G RAM. Has anyone else seen this problem? What can I do to get more information about what is causing it? -- John A. Tamplin Unix System Administrator Emory University, School of Public Health +1 404/727-9931
Re: Conversion/Migration
Peter Lawler wrote: OK, thanks to those who replied. I've gone this far: http://batleth.sapienti-sat.org/projects/mb2md/ It's a pretty neato script, although I haven't tested the version released yesterday. It converts just as we want. Now, the trick after the conversion that the new files are in, say ~/Maildir/new/, so that they need to get moved into the cyrus store for the user. That's no problem. However I've got stuck at the next bit. OK, I can work out that the messages in the INBOX are individually numbered (eg, 52., 53., 54.). So, the converted mail files need to be renamed, one would guess. Can someone suggest the best method for renaming these files? Is the order of numbering of the files important (eg, datestamping), or is it irrelevant. Next thing is that only occasionally have I been able to rebuild the cyrus DB to have these converted messages appear in the INBOX. It was late last night, so I can't recall ALL the details. However, I do suspect that it *MAY* have something to do with the naming of the files mentioned above. Additionally, it *COULD* have something to do with the cyrus commands I'm issuing to try and rebuild the cyrus DB. Thirdly, when (on only two occasions, mind you) the converted messages did pop up, they are being displayed as 'read'. How can I set their status to 'unread'. I did suspect that *MAYBE* the fullstop (period) at the end of the filename might be the clue, but since I didn't manage to get the messages appearing again, I could work it out. Any comments, suggestions, flames, greatly appreciated. I think you would be much better off going through IMAP rather than trying to put the files in there correctly and then reconstructing it. You won't be able to maintain seen state (or any other flags for that matter) unless you build all the cyrus files correctly during the conversion. If you use mbxcvt from UW imap-utils, it will preserve the flags as part of the transfer. The only catch is you will have to be able to authenticate as the user when you convert. What I did was a temporary hack to saslauthd which allowed a backdoor password to work for all accounts and hacked mbxcvt to accept the password on the command line (no user accounts on this machine to watch ps). A wrapper script went through all the accounts to transfer (one by one) and updated the database used by perdition (imap/pop proxy) to show the account was being used and killed all active sessions. It then converted each mailbox using the hacked mbxcvt, connected as the admin account to set the quota, and connected as the user to set the subscription list. When the account was moved, update the perdition database so new connections go to the new server (and LMTP deliveries via proxy) and go to the next one. If you don't need to do the transfer while everything is up, obviously you don't have to fool with the database for the proxies. -- John A. Tamplin Unix System Administrator Emory University, School of Public Health +1 404/727-9931
Re: Working Vacation?? [solved?]
Mikael Brandström wrote: It should be possible to rewrite rulset Parse0 (and maybe Parse1) to make them leave the local adress qualified while delivering locally. There might even be something among the files in cf/cf which kan fix it, I have a vague memory of something such. Just my 2c. I made changes locally that amounted to always setting $@ whenever $#local was selected as a mailer (for mailers that don't use $h, it shouldn't matter at all). If you redefine the local mailer to be cyrusv2, then look for $#cyrusv2. The places that have to be changed are: Ruleset 0, change: R@$#local $: @ special case error msgs to R@$#local $@ $M $: @special case error msgs (if you aren't masquerading use $j instead of $M -- it could be cleaned up so it would work for more general configs but I was in a hurry to get something working for our setup). In Ruleset Parse1, change: R $+ + $* $+ $#local $: $1 + $2 plussed name? to R $+ + $* @ $+$#local $@ $3 $: $1 + $2plussed name? and then the last two rules in Parse1 change from: R$=L$#local $: @ $1 special local names R$+ $#local $: $1regular local names to R$=L$#local $@ $M $: @ $1special local names R$+ $#local $@ $M $: $1 regular local names Remember that tabs separate the columns, not spaces. Then in the mailer definition, you can pass $u@$h rather than just $u. I am currently using my own deliver program which then connects to LMTP -- this should work for the stock deliver program as well, but for direct LMTP delivery from sendmail you will have to get more creative. I haven't tried it, but it should be possible to copy ruleset EnvFromSMTP to something else and modify it to add @$h to the end of the address if $h is defined, and then change the cyrusv2 mailer to use that for the envelope sender rewrite ruleset. If anyone wants patches to the stock sendmail 8.12.7 proto.m4 file, I will be happy to send them with the usual disclaimers (I have not made any effort to make the changes more general, just working for our particular configuration -- YMMV). I also have changes (including 3 lines of source changes) to allow the w flag to be used with a local mailer yet pass through any users that aren't listed in getpwent (ie, I wanted to have user .forward files yet still allow delivery to shared mailboxes). This hack is even more likely to be a problem in the future, as they are likely to eventually use the flag letter 'y' for something else, but if you want them just ask. One day when I get time I will clean these up and try and submit them to see if they will get included. -- John A. Tamplin Unix System Administrator Emory University, School of Public Health +1 404/727-9931
Re: Problem with cyrus and deleting a message with a virus.
[EMAIL PROTECTED] wrote: Irrelevant question. The fact that it could happen is enough. I can't stop my users going to someone's computer (which has no virus protection) and connecting to my IMAP server. I have students who will no doubt use the IMAP server as a filestore when they run out of quota on the fileserver. The same arguments apply if you were talking about an Oracle database -- users could store viruses into the database and someone else could extract it from that database and execute it. However, you wouldn't run a virus scanner on Oracle databases that just deleted files if it didn't like them -- the Cyrus mailstore is no different, even if some of the parts are stored in a familiar format. The clean way would be to add a filtering layer wherever messages could be stored into Cyrus. It is easy enough to add a front-end to the delivery side using the various MTAs, but it would be more work to filter messages stored via IMAP. Until then, the correct way to do it would be to use IMAP to muck with the message store (even if you found which files you had a problem with by running directly on the filesystem, but of course there is no guarantee you are seeing a consistent state). If you insist on deleting the files out from under Cyrus, then be content with private hacks to work around the problem, reconstruct the mailboxes you tamper with, or just live with a partially broken mailstore. -- John A. Tamplin Unix System Administrator Emory University, School of Public Health +1 404/727-9931
Re: Vacation problem
[EMAIL PROTECTED] wrote: I'm wondering why cyrus setting sender to , not to [EMAIL PROTECTED], or even postmaster as I made configuration in my imapd.conf? May be I missed something? You don't want an auto-generated message generating a bounce message, since that could lead to an infinite loop. Thus, just like mailer-daemon messages in the MTA, a vacation message should have the envelope from set to . -- John A. Tamplin Unix System Administrator Emory University, School of Public Health +1 404/727-9931
Re: Vacation problem
[EMAIL PROTECTED] wrote: So what is you suggestion? How do I fix this problem? Why the mail server doesn't accept the message if the envelope from set right? It is hard to tell from the logs since you are using 4 different email addresses, but it looks like sieve tried to send mail from (which sendmail accepted, but gave an authentication warning since cyrus isn't listed as a trusted user in your sendmail.cf file) to [EMAIL PROTECTED] That gets sent, apparently to the same MTA, which then has a problem talking to the relay mail.montana-education.ab.ca. It then tries to send it to [EMAIL PROTECTED] (presumably an alias from postmaster), and again has a problem talking to mail.intellinetinc.com and panics since it has nowhere to send the bounce message. So, it looks like Cyrus is fine but your MTA is misconfigured. -- John A. Tamplin Unix System Administrator Emory University, School of Public Health +1 404/727-9931
Re: Pb with cyradmin and quotas...
Mr Bullier Erick wrote: I use quotas on mailbox with cyradmin. I use scripts which check (in the night) if user go out the warning limit (80%) of is mailbox quota. Today, i see that a user maibox is 85% full. So i use cyradm to check it, and cyradm write: localhost lq user.toto STORAGE 5996/1 (59%) I didn't understand and i made a check on the file systeme: # du -h /var/spool/cyrus/mail/user/toto 28k/var/spool/cyrus/mail/user/toto/Transmis 856k/var/spool/cyrus/mail/user/toto/Poubelle 16k/var/spool/cyrus/mail/user/toto/Brouillon 16k/var/spool/cyrus/mail/user/toto/arch1 16k/var/spool/cyrus/mail/user/toto/arch2 16k/var/spool/cyrus/mail/user/toto/arch3 8.5M/var/spool/cyrus/mail/user/totoe What's happend?.. I don't understand. Do you have an idea? The cyrus quota doesn't include administrative files, such as cyrus.index and cyrus.cache -- the user is only charged for files they have control over. Rather than checking the filesystem directly, use /usr/local/cyrus/bin/quota (or wherever you installed it) and process that output, or have a script connect to the IMAP server using an administrative account and do getquotaroot commands. -- John A. Tamplin Unix System Administrator Emory University, School of Public Health +1 404/727-9931
Re: Problem with cyrus and deleting a message with a virus.
Mark London wrote: I'm not messing with it, uvscan is doing it. Is there a better software alternative that will delete viruses on the server? Are we the only people using cyrus that are running virus scanning software on the server? I think most people scanning their mail do so before it is stored in the filesystem. Btw, I would think cyrus should be able to handle the simple case of a missing single file. I should be able to delete a message for which the message file is already missing. We're not talking about a complex database file structure here. It's a single file with a single message. How far should the server go assuming it knows the reason why some unexpected condition exists? Should it happily ignore a missing /etc/cyrus.conf and assume default settings? Should it assume /var/imap ran out of disk space because there were log files it should silently clean up for you? I imagine it wouldn't be very difficult to hack the source so that whenever it tried to open a message file that didn't exist, it could create a message that says it was removed by virus scanning and then open that file, but that would have to be something you want to run -- I wouldn't want that in the version I was running and I doubt such a hack would get accepted into the codebase. -- John A. Tamplin Unix System Administrator Emory University, School of Public Health +1 404/727-9931
Re: Problem with cyrus and deleting a message with a virus.
Ted Cabeen wrote: Right. Other than losing the flags data, are there any other downsides to this solution? (We were thinking of using this to delete old messages from users spam folders) If your users all have spam folders named the same or similar (presumably put there by spam filtering software), wouldn't ipurge do what you want already and without screwing with Cyrus's data structures? -- John A. Tamplin Unix System Administrator Emory University, School of Public Health +1 404/727-9931
Re: UW to Cyrus transition
[EMAIL PROTECTED] wrote: In the last week of december, I ported 1 ( c.a. ) users from UW to Cyrus. As I had a REALLY hard time finding ANY help whatsoever ( scripts etc ), I wrote some of my own. I must say, they DO take time. I was running them on a Sun Fire 880 with 4x900MHz CPUs and 8GB RAM, and all files local. It takes around 8-10 hours per 2500 users with around 20GB of data total. So for 1 users with around 80GB of data, up to 40 hours of conversion time is a fairly close estimate. I am sure that the scripts are not completely bug-free, but they did work for me. BEWARE: usernames with dots in them are NOT very well handled. Did you have unixhierarchysep set? If not, then period isn't a legal character in a username. I converted 2300 users with 80G of mail over the course of 3 days, using a proxy solution. The accounts were moved over one at a time, with mail delivery held up and imap/pop login blocked while each account was moved. The proxies were a hacked perdition (to keep track of sessions so they could be killed and to treat a servername beginning with ! as a reject message to show to the client attempting to login) and a custom perl delivery agent which consulted the same database used by perdition and connected via LMTP (to procmail on the UW-IMAP side) to the two hosts. Based on the tests I had done (same V880 configuration on both old and new servers) it would have taken over 60 hours to convert everyone at once (using a hacked mboxcvt and a bunch of custom perl scripts), and that amount of downtime was unacceptable. That was also putting /var/imap on a tmpfs filesystem and striping /cyrus across 14 FC-connected drives. -- John A. Tamplin Unix System Administrator Emory University, School of Public Health +1 404/727-9931
Re: Websieve and Cyrus 2.1.11
[EMAIL PROTECTED] wrote: I am currently trying to make Websieve run with my Cyrus 2.1.11 installation. Unfortunately this doesn't work as well as planned, I copied the files to my cgi-bin directory but now when I run http://mailserver/cgi-bin/websieve.pl I get the following error: Received a program error! Error: Can't locate IMAP/Admin.pm in @INC (@INC contains: /usr/perl5/5.6.1/lib/sun4-solaris-64int /usr/perl5/5.6.1/lib /usr/perl5/site_perl/5.6.1/sun4-solaris-64int /usr/perl5/site_perl/5.6.1 /usr/perl5/site_perl /usr/perl5/vendor_perl/5.6.1/sun4-solaris-64int /usr/perl5/vendor_perl/5.6.1 /usr/perl5/vendor_perl .) at ./funclib.pl line 5. BEGIN failed--compilation aborted at ./funclib.pl line 5. Compilation failed in require at /opt/wmapache/cgi-bin/websieve.pl line 53. BEGIN failed--compilation aborted at /opt/wmapache/cgi-bin/websieve.pl line 64. Any ideas ? I also remarked that websieve is a bit old, I think more than 1 year old, do I maybe need to patch something to make it work ? Or maybe do you have any alternatives for a web interface especially for vacation (must be LDAP capable). perl is telling you precisely what the problem is -- it is looking for the file IMAP/Admin.pm (ie, the perl module IMAP::Admin), and it can't find it in any of the directories listed. If you installed it somewhere else, you will need to get perl to look for it where you installed it. If you haven't installed the module, it won't work until you do. -- John A. Tamplin Unix System Administrator Emory University, School of Public Health +1 404/727-9931
Re: Websieve and Cyrus 2.1.11
[EMAIL PROTECTED] wrote: ok right I just saw that my Admin.pm perl module is in /usr/perl5/site_perl/5.6.1/sun4-solaris-64int/Cyrus/IMAP/Admin.pm so in the file funclib.pl from Websieve I've changed the following line: use IMAP::Admin; to use Cyrus::IMAP:Admin; and I got further but now it's complaining that it cannot find IMAP/Sieve.pm and I am 100% sure this file is NOT part of Cyrus IMAPD at least not 2.1.11. Do you know where I can find it maybe ? This Websieve thing really looks old :( If you want to use the Cyrus version of those modules, the name for that one is Cyrus::SIEVE::managesieve. It looks like you are using funclib.pl rather than funclib.cyrus -- the readme.txt file explains that either you need to use funclib.cyrus and use the internal Cyrus modules, or you need to install IMAP::Admin and IMAP::Sieve. The readme also tells you where to find those files if you don't want to use the Cyrus modules. As far as really old, there are many programs on my system which haven't been updated as recently as a September 2001, which is the date on websieve. -- John A. Tamplin Unix System Administrator Emory University, School of Public Health +1 404/727-9931
Re: presubscribing mailboxes
Earl Shannon wrote: An alternative is to authenticate as one user, probably the admin user, and be authorized to act on behalf of the account. That said, I don't know how to do this. If someone has an example of some perl code using the Perl IMAP Admin module that does this and is able to post it to the list I would appreciate it. Currently I do what you do when I need to subscribe a user to some mailboxes in their account. Edit their subscription files. Would definitely prefere being able to do this via the network instead of logging into the box. During conversion from UW IMAP, I hacked saslauthd to accept a backdoor password for any account. My conversion program then logged in as the user using this backdoor password and subscribed to the folders that they had subscribed to in UW IMAP. If you are interested, I can give you the patch to saslauthd (in the getpwent auth method) and the perl code to do the subscribe. -- John A. Tamplin Unix System Administrator Emory University, School of Public Health +1 404/727-9931
Re: [PATCH][CVS IMAPd 2.1] lmtp_downcase_rcpt implementation (2)
Lawrence Greenfield wrote: I'm kinda unhappy about the limited scope of the patch. It makes usernames case-insensitive during delivery, but they're case-sensitive everywhere else. I'm not sure this is a great idea. Maybe a global switch of case-sensitive mailbox names would make more sense (thus LeG+detail - user.leg.Detail, and LEG could log in). I'm not sure, though. I made that suggestion, and the only response at the time was negative. -- John A. Tamplin Unix System Administrator Emory University, School of Public Health +1 404/727-9931
Re: [PATCH][CVS IMAPd 2.1] lmtp_downcase_rcpt implementation (2)
Lawrence Greenfield wrote: Date: Mon, 13 Jan 2003 16:31:15 -0500 From: John Alton Tamplin [EMAIL PROTECTED] Lawrence Greenfield wrote: I'm kinda unhappy about the limited scope of the patch. It makes usernames case-insensitive during delivery, but they're case-sensitive everywhere else. I'm not sure this is a great idea. Maybe a global switch of case-sensitive mailbox names would make more sense (thus LeG+detail - user.leg.Detail, and LEG could log in). I'm not sure, though. I made that suggestion, and the only response at the time was negative. I'm very fickle. I think also a global case-sensitivity switch is considerably hard than the patch Henrique proposed, but somehow I feel that the patch Henrique proposed better belongs in the MTA. I dunno. Maybe we should just commit it, solve this problem, and go on with life. There seem to be 3 places that have to be dealt with: 1) login 2) message delivery 3) mailbox names It seems like 1 is very localized and a trivial change in auth_canonifyid (if it isn't appropriate for Kerberos, then the switch could just be ignored in auth_krb*.c) . 2 is easy enough and a small variant of Henrique's patch should do it. 3 is much harder, since you potentially have to handle select Shared Folders/user/JTampli/Test to resolve to user/jtampli/Test and similar situations. I think it is less of an issue because most clients are just going to show the user what LIST/etc return (and any running altnamespace won't see them except for other users that have granted permissions to one of their folders), so I would be happy with a patch that solves only 1 2 (and in fact I will probably implement it myself for local use if nobody is interested in getting into the official distribution). -- John A. Tamplin Unix System Administrator Emory University, School of Public Health +1 404/727-9931
Re: [PATCH][CVS IMAPd 2.1] lmtp_downcase_rcpt implementation (2)
Lawrence Greenfield wrote: Well, message delivery and mailbox names are tightly intertwined. How should I know that leg+detail should be delivered to user.leg.Detail? There is no historical usage that says imap folder names within a mailbox aren't case sensitive, so I would say leg+Detail should be delivered to user.leg.Detail and leg+detail should bounce (or actually get dropped into INBOX). The issue is that users have treated mail addresses as case insensitive and other implementations have enforced that. You also need to prevent two mailboxes from being created with the same name (except for case). If we are only talking about smashing the case on user names and therefore mailboxes, couldn't that be handled both for delivery and other uses in mboxname*_tointernal? There's the i18n problem: case mapping isn't trivial across lots of languages. True. Since mail addresses are defined in US-ASCII, where is the mapping from RFC2047 into the actual characters done? It seems the case mapping should be done there, at the point you know the character set. If, as in most cases, it is US-ASCII or a Latin1 character set, it is easy to implement. That will get the majority of the need with little effort. For those other character sets, it isn't clear there is as much of a need since they don't have historical implementations that smashed case in the user part of the email address to deal with. Finally, we need to deal with people toggling this option on and off. What happens if user.leg.foo and user.leg.Foo both exist, and then the administrator turns on case-insensitivity? Since what I am proposing only affects user names, that example wouldn't be a problem. If user.foo and user.Foo exist and the option is turned on, then user.Foo won't be able to receive any mail. I don't think Cyrus should protect sysadmins from that any more than it should protect against them deleting a mailbox they shouldn't. -- John A. Tamplin Unix System Administrator Emory University, School of Public Health +1 404/727-9931
Re: delete selected folder leaves empty dir on disk
Rob Siemborski wrote: Yeah, we know about this. There's no good way around it, since a totally separate imapd could be selected on the folder as well, and the directory would still be left around. So is it ok to sweep the filesystem and delete any empty directories? Would that leave a race condition with an imapd creating a new folder? It seems like it should be safe to rmdir() the directory even if some process is sitting on it or has it open -- there might need to be additional error checking to handle a directory disappearing unexpectedly, but that doesn't seem to be a big problem. -- John A. Tamplin Unix System Administrator Emory University, School of Public Health +1 404/727-9931
Re: delete selected folder leaves empty dir on disk
Rob Siemborski wrote: We do currently call rmdir() on the directory. The applicable code is in mailbox_delete() in mailbox.c if you'd like to suggest a patch. Since rmdir() doesn't delete it, I assume that means there is some entry in the directory which wasn't deleted. Since you have apparently looked at this before, what is the cause? Is unlink failing on some file for some reason, or is it a race condition of a file being created after the opendir() and before the rmdir() loop? This has been particularly annoying here during the conversion, because Mail::IMAPClient uses temporary folders and it deletes them while they are selected, leaving thousands of empty top-level directories. I shut the server down and cleaned them up, but that isn't something I would like to do often. -- John A. Tamplin Unix System Administrator Emory University, School of Public Health +1 404/727-9931
Re: lmtpd NUL rejection
[EMAIL PROTECTED] wrote: I've recently upgraded imapd from 1.5.19 to 2.1.11, and instead of having sendmail invoke deliver it now talks to lmtpd over a Unix socket. All is well, except that lmtpd is much more scrupulous about checking its input than deliver was - in the space of a week, it's detected three otherwise normal messages containing embedded NULs and has rejected them with DSN 554 5.6.0 Message contains NUL characters (status IMAP_MESSAGE_CONTAINSNULL in imap/lmtpengine.c). OK, fair enough, except that sendmail responds to the bounce by trying to copy the message to postmaster. Via lmtpd. Oops. Clearly the input is bad and lmtpd is justified in rejecting it. However, broken mail clients (or whatever - we haven't identified any common factor yet) are a fact of life, and having mail stuck in a non-delivery loop isn't very helpful for our users. What's the Right Thing to do here? Should sendmail (8.11.2) be configured to somehow report the failure without forwarding the message, or perhaps do NUL filtering on the fly? Or is there some way of configuring the lmtp mailer definition to get around this problem? And out of general curiosity, have other sites moving to lmtpd encountered this, or are we just particularly weird? I noticed it here as well after switching from UW IMAP to Cyrus 2.1.11, and I am currently still running everything through a perl delivery program which does a number of clean-ups including stripping NUL characters. The problem is that a number of large mailing lists (such as some on Yahoo Groups) include NULs, so just dropping the mail wasn't an option. Eventually, I want to get back to the stock deliver but I will need to patch it to include an option to strip NULs similarly (as well as other issues currently being handled in the perl script) before I can. -- John A. Tamplin Unix System Administrator Emory University, School of Public Health +1 404/727-9931
Re: saslauthd performance anxiety
Paul M Fleming wrote: Timing out the passwords is simple ( I think ) I would store the time when the entry is added and force a reauth if the password has been cached longer than a timeout (for example one hour ). That forces a reauth at least every timeout period of time. If an entry isn't in the cache (or if it is different the entry would be removed and ) a reauth would be forced. Every successfull auth would be added to the cache. Also, if the authentication against the cached entry fails it should be refreshed and tried against the new one, so that if the user changes their password the caching is transparent. -- John A. Tamplin Unix System Administrator Emory University, School of Public Health +1 404/727-9931
Re: [PATCH][CVS IMAPd 2.1] lmtp_downcase_rcpt implementation (2)
Henrique de Moraes Holschuh wrote: Here's the improved patch. It skips over the +Mailbox part of the recipient, in the most straightforward way possible ;-) Would it make sense to make the control more generic, such as downcase_account_name, and have it also apply to logins and other places besides LMTP? We have a number of users who were logging in with randomly capitalized account names in UW IMAP (and it happily accepted them). We have been contacting them to change it, but perhaps others with a similar problem might prefer to just accept it. -- John A. Tamplin Unix System Administrator Emory University, School of Public Health +1 404/727-9931
Re: Slow LIST command
On Wed, 11 Dec 2002, Rob Siemborski wrote: Having watched the activity, the thing which seems to consume most of the resources are the LIST * and LIST % commands. These return with a response like OK Completed (1.480 secs 28 calls) but can take a very long real time. Is there anything which can be done to speed up the performance of the LIST command? My imapd.conf is appended. Are you using the skiplist backend for your mboxlist? If not, that should tremendously increase LIST performance. Is the recommendation for skiplist compared just to flat or to db3 as well? If the latter, how is reliability compared to db3, and does that extend to the other uses besides mboxlist? -- John A. Tamplin Unix System Administrator Emory University, School of Public Health +1 404/727-9931
Re: cyrus 2.2 status
Rob Siemborski wrote: But unless the contents of the folders are backed up in this way too, you haven't really gained a significant amount, since the transactions that cyrus needs to make rely on the contents of the filesystem as well. True, although with the metadata secure you can politely tell the user their message is gone I agree that is of little benefit unless the messages are stored in the database as well. For a large installation, avoiding the downtime to reconstruct large databases after a crash might be a benefit worth the effort. -- John A. Tamplin Unix System Administrator Emory University, School of Public Health +1 404/727-9931
Re: using Murder for migration from UW IMAP
Ted Cabeen wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Content-Type: text/plain; charset=us-ascii In message [EMAIL PROTECTED], John A. Tamplin writ es: Quoting Rob Siemborski [EMAIL PROTECTED]: Do you really have so much data that it would take 2-3 days to move it? The tests I have done so far (using both imap-utils mbxcvt and custom perl code) it looks like it will take around 60 hours to convert everything (2300 users, 23000 folders, 70G total), and that is putting /var/imap on a tmpfs filesystem for the conversion (about a 2x speed improvement there) and striping /cyrus across 14 drives in an FC-AL array. Have you considered just doing a direct mailbox conversion? You can convert directly to Cyrus mailboxes and then reconstruct everything. That might be faster than dealing with the mbxcvt code. I need to get the flags and seen state transferred as well -- that wouldn't do it, right? It is important that the transition be as seemless as possible. Having to rename a few mailboxes that UW IMAP allows but Cyrus doesn't is enough hassle to the users. -- John A. Tamplin Unix System Administrator Emory University, School of Public Health +1 404/727-9931
using Murder for migration from UW IMAP
Would it be possible to use Murder to migrate from UW IMAP? I have Cyrus setup and running on a new machine, but the problem is that taking everything down and converting all the mailboxes would be too much downtime (2-3 days). What I was thinking of is setting up a frontend server with UW IMAP and Cyrus as the backend servers, with the initial mupdated database showing all the mailboxes were on the old IMAP server. UW IMAP isn't going to talk to mupdated, but if no mailboxes are being created there it seems like that won't matter. As each mailbox is moved to the new server, the master list of mailboxes is reflected to update the new location. From my inspection of the documentation and some of the code, it looks like this should work with a couple of caveats (during the transition, users have to be prevented from creating new mailboxes -- this can be hacked in the code, and connections that have been referred to the old server have to be broken when the mailbox is transferred). Does anyone see a reason why this won't work or any other gotchas to watch for? Thanks. -- John A. Tamplin Unix System Administrator Emory University, School of Public Health +1 404/727-9931
