Re: Question about squatter for Xapian
On Fri, 24 Jan 2020, at 4:38 AM, ego...@sarenet.es wrote: > - Does it regenerate all mailboxes indexes?. Just the non-indexed emails?. I > assume it should be extremely slow… so could this be launched?. Could you > advise me please, if another way is preferred? Normally, just the non-indexed emails. squatter -i (incremental) should be all you need to fill the gaps in your index. Obviously how long it takes depends on how much mail has arrived and how good your disks are, but for 12 hours worth I wouldn't expect more than a couple of hours to fill the gaps. > - I assume not, but as we move records between Xapian tiers nightly… if the > Squatter launched by me, by hand (for those non indexed emails), runs at the > same time as this between tiers movement of records or at the same time too > as the rolling mode squatter (-R) could one squatter process interfere in the > job of the other instance of squatter?. It's ok to run them all at the same time. Cyrus has appropriate locks to make sure that Xapian updates and repacks don't get in each others' way. Rob N. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Cyrus mail spool entirely on SSD - tiered storage
On Fri, 21 Oct 2016, at 08:46 PM, Vladislav Kurz via Info-cyrus wrote: > would you be so kind and point to some info (docs, howtos) about cyrus > tiered storage? All I found is > http://www.cyrusimap.org/~vanmeeuwen/imap/features/mail-spool-partitions.html > but that seem to be quite tedious to set up That doc[1] is describing splitting the mail store by folder. We don't use that. Instead, we use the "archive partition" feature, which splits the mail store by message age (and size and flags). The interesting imapd.conf[2] parameters are: archive_enabled archive_days archive_maxsize archive_keepflagged archivepartition-name. And then you'll want to look at the -A flag to cyr_expire[3], which is the thing that moves "old" stuff from the main partition down to the archive partition. The docs suggest this feature is in the currently-unreleased Cyrus 3.0 series. That's probably true, but I can't be sure (FastMail runs the current dev/master branch, so we're ahead of 3.0). It'll be well worth the upgrade when the time comes though if you've got large mail stores. Its a rock-solid feature, I love it. > How transparent it is for users? Entirely. The message indexes are still stored in the metadata area like normal; its only the actual messages that are on the archive partition. In the worst case loading an older archived message is slightly slower because the disk is slower but most people probably won't notice anyway. For reference, FastMail's config is very simple: defaultpartition: default partition-default: /mnt/ssd21d1/sloti21d1t01/store92/spool archivepartition-default: /mnt/i21d1t01/sloti21d1t01/store92/spool-archive archive_enabled: yes Our mount naming convention isn't hugely important; the main thing to note is that ssd21d1 is a SSD partition, while i21d1t01 is on conventional spindles. The other archive_* options aren't used because the defaults are sensible (which makes sense; we developed the feature so they're our defaults!). And then every day we run cyr_expire -A 7d from cron, and that's it. Its worth noting that tiered search indexes are possible, but are managed through a different set of config options. I'm not going to write about that right now because there's no docs for me to point back to and search setup is a bit more involved depending on your needs. What I will do is invoke the Cyrus docs people so they're aware of the gap in the docs and hopefully we can get a better guide put together sometime. Cheers, Rob N. Links: 1. http://cyrusimap.org/imap/concepts/features/mail-spool-partitions.html 2. http://cyrusimap.org/imap/reference/manpages/configs/imapd.conf.html 3. http://cyrusimap.org/imap/reference/manpages/systemcommands/cyr_expire.html Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Cyrus mail spool entirely on SSD
On Thu, 20 Oct 2016, at 05:28 AM, Michael Ulitskiy via Info-cyrus wrote: > My question is: assuming cost is not an issue, would it be advisable > to put an entire cyrus mail spool on SSD? > I'm thinking about combinining multiple SSDs into RAID10 to further > increase performance. At FastMail we have our mail spools on SSDs, in RAID10. We use Intel DC S3700 which, while on the expensive side, have been incredibly reliable and fast. They're all still 6G SATA; we're still looking at NVMe (with some enthusiasm). We actually have too much mail right now to use SSDs for the entire spool, so we use Cyrus' tiered storage facility to keep the most recent (last week) mail on the SSD, and the rest on spindles (WD Gold 4TBs). There's a cyr_expire job that runs periodically to move old stuff down to the spindles. Metadata and caches are exclusively on the SSDs. The search indexes are spread across disks; the fastest in ramdisks, the SSDs, then spindles again. Again, there's a regular job to eject older stuff down the stack. Ramdisks are used to make search response on recent messages ridiculously quick. There is an operational cost in that in order to cleanly shutdown the machine, you have to compact down the ramdisk search indexes and if you don't, you have to reindex the entire spool. It's pretty rare for that to happen though, and since the search index is not a primary data source, its an acceptable risk. So yes, SSDs are a great idea. I'd have _everything_ on SSDs if it were cost-effective :) Cheers, Rob N. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: De-duping attachments
How difficult or easy would it be to modify Cyrus to strip all attachments from emails and store them separately in files? In the message file, replace the attachment with a special tag which will point to the attachment file. Whenever the message is fetched for any reason, the original MIME-encoded message will be re-constructed and delivered. Like anything, doable, but quite a lot of work. cyrus likes to mmap the whole file so it can just offset into it to extract which ever part is requested. In IMAP, you can request any arbitrary byte range from the raw RFC822 message using the body[]start.length construct, so you have to be able to byte accurately reconstruct the original email if you remove attachments. Consider the problem of transfer encoding. Say you have a base64 encoded attachment (which basically all are). When storing and deduping, you'd want to base64 decode it to get the underlying binary data. But depending on the line length of the base64 encoded data, the same file can be encoded in a large number of different ways. When you reconstruct the base64 data, you have to be byte accurate in your reconstruction so your offsets are correct, and so any signing of the message (eg DKIM) isn't broken. Once you've solved those problems, the rest is pretty straight forward :) Rob Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: De-duping attachments
A 500-user company can easily acquire an email archive of 2-5TB. I don't care how much the IO load of that archive server increases, but I'd like to reduce disk space utilisation. If the customer can stick to 2TB of It would be interesting to measure the amount of duplication that is going on with attachments in emails. While we could do that with Fastmail data, I think because of the broad range of users, we'd be getting one data point, which might be quite different to a data point inside one company. Eg. An architectural firm might end up sending big blueprint documents back and forth between each other a lot, so they'd gain a lot from deduplication. Also even within deduplication, there's some interesting ideas as well. For instance, if you know the same file is being sent back and forth a lot with minor changes, you might want to store the most recent version, and store binary diffs between the most recent and old versions (eg xdelta). Yes accessing the older versions would be much slower (have to get most recent + apply N deltas), but the space savings could be huge. Can we just brainstorm with you and others in this thread... how do we re-create a byte-identical attachment from a disk file? One overall implementation issue. With the message file, do you: 1. Completely rewrite the message file removing the attachments and adding any extra meta data you want in it's place 2. Leave the message file as exactly the same size, just don't write out the attachment content and assume your filesystem supports sparse files (http://en.wikipedia.org/wiki/Sparse_file) The advantage of 2 is that it leaves the message file size correct, and all the offsets in the file are still correct. The downsides are that you must ensure your FS supports sparse files well, and there's the question of where do you actually store the information that links to the external file? - file name/reference - full MIME header of the attachment block I'd leave these intact in the actual message, and just add an extra X-Detached-File header or something like that includes some external reference to the file. Hmmm, that'll break signing though. Not so easy... - separator string (this will be retained in the message body anyway) - transfer encoding - if encoding = base64 then base64 line length Remember every line can actually be a different length! In most cases they will be the same length, but you can't assume it. And you do see messages that have lines in repeating groups like 76, 76, 76, 76, 74, 76, 76, 76, 76, 74, ... repeat ... or cases like that, a pain to deal with. - checksum of encoded attachment (as a sanity check in case the re-encoding fails to recreate exactly the same image as the original) This is seeming a bit more tricky... Rob Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: Log all actions
Also check out the Fastmail auditlog patches. http://cyrus.brong.fastmail.fm/ Rob - Original Message - From: Dr. Tilo Levante t...@levante.de To: Info-cyrus@lists.andrew.cmu.edu Sent: Friday, May 28, 2010 7:59 PM Subject: Log all actions Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Any interest to implement RFC4978 (IMAP COMPRESS)?
For low bandwith connections this could be useful but I don't know if that's a typical case nowadays. Together with the IMAP IDLE command it should be fine for mobile devices... [1] http://tools.ietf.org/html/rfc4978 I thought that this was supported in 2.3.16. It's definitely in CVS, not sure what released version it's in (check the changelog?). We use it at FastMail. http://blog.fastmail.fm/2009/05/01/help-test-proxy-to-improve-imap-performance/ Rob Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: cyrus and 500k to 1M mails per inbox
It's quite likely, that these mailboxes will grow to 50 or even more then 1M mails per mailbox. Does anybody have experience with such big mailboxes? Is the I/O cost of message adding relative to O(n), n being the number of msgs already in the mailbox, or is it independent thereof ? Stacks of stuff is O(n), sorry. It's not really an ideal way to do things. You'd be better off archiving per month for each mailbox and keeping the messages counts to a reasonable number. Well it's not quite as bad as that. If the mailbox is mostly just appended to, then it's not a problem. The append code just appends to cyrus.index and cyrus.cache I believe (Bron?), it doesn't have to read the cyrus.* files each time. We have actually seen a mailbox with 1M messages at FastMail, and I remember that I could view it ok. However, I'll add a few caveats around that. 1. Our web interface has been highly optimised over time to make sure it deals ok with large UID lists. I'm not sure how many email clients will handle viewing an IMAP folder with 1M messages 2. You need a good filesystem as well, because there will be 1M files in a directory. We use reiserfs3, and again, it has no problem with that many files in a directory, mileage with other filesystems might vary. Rob Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: what happens when cyrus.squat is old?
it seems that running squatter nightly on all mailboxes takes too long for us. I'm thinking of splitting the mailboxes over different nights or doing the job over the weekend. Are you using the new incremental mode david carter added? -i Incremental updates where squat indexes already exist. One question though, what happens when a cyrus.squat file is out of date (ie the mailbox has been changed due to mails added/removed from the mailbox). Does it - use the index for what it can and then look directly at the remaining emails? Yes, it does this. Which is nice, because it means if the index isn't up to date, your searches still work correctly, just slower. Rob Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: OT: Enhanced IMAP protocol
I would argue that it's out of scope -- credential management should be taken care of by your credential management system, be it through a web interface or whatever. Even if it were to be an accepted spec, the chances of all of the client-writers implementing it, and in a reasonable way, are slim to none. I'd bet more on clients providing a reasonable interface to the IMAP motd -- which, after authentication, would be a great way to let the user know that their password needs to be changed. On Jan 5, 2010, at 11:02 PM, ram wrote: Would it be practical to have common protocols like IMAP to support enhanced features For eg. IMAP protocol may possibly support change password. A IMAP server administrator may optionally configure a change-password hook on the server which would change the password on whatever backend he uses for eg ldap or a RDBMS or Active directory The advantage is that the MUA can support change password and the user experience will be a lot better having only a single familiar app to deal with. That will be a lot easier for admins to tell the users to change password regularly rather than giving them a new link and asking them to change password ( .. and that mail looks more like a phishing mail ) Thanks Ram Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: cyrus replication : master to replica and replica to master
i'm very surprised that there is not really official point from cyrus-imap dev team against using cyrus in cluster active/active mode I can't comment, but I guess they're busy. Since serverals years the messaging service become very important and the clustering system is the right way to provide a real fail over system After googlize i had read that the only best method is to using drbd but there no really wikis and documentations about implementing this kind of structure. drbd doesn't allow a master/master mode either, unless you use some clustered filesystem, and take the performance hit due to locked access to meta data (eg mailboxes.db) that would be happening. cyrus replication does what drbd does, but has the added advantage that it's content aware, so it only has to replicate the actual needed data, it can do checksumming of email data on both sides, and you won't lose everything if you get some os level file system corruption on your master. Basically they're similar solutions (master - slave replication) but at different levels (block level vs content level). Rob Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: cyrus replication : master to replica and replica to master
Client A: upload message to Inbox, gets UID 100 At the same time, Client B: upload message to Inbox, gets UID 100 You can't have two messages with the same UID. There's 3 solutions I can see: 1. Mysql solves this by having interleving id's on separate servers (eg. auto-increment column on server A is odd numbers, on server B it's even numbers). I guess you could in theory do the same with IMAP (though I'd have to double check the spec), but it would create really annoying UID lists because you basically lose the ability to use things like 30:50. One other option would be to alternate in 100's or something like that (eg. 1-100 on s1, 101-200 on s2, etc) I realised another potential problem with this. The IMAP spec says UIDs must be incremental. So if you upload a message to s1 and it gets UID 100, and upload on s2, and it gets UID 200, then when you upload the next message on s1, it has to get UID 300. So you have to make sure that if any UIDs are allocated in a higher range, you have to jump to the next range. This could cause you to run out of UIDs quite quickly in pathalogical back and forth cases (eg 2 IMAP clients connected separately to s1 s2 both uploading a bunch of messages to the same folder). 3. Use some conflict resolution strategy. If some client uploads UID 100 on s1, and another uploads UID 100 on s2, then when the conflict is noticed, both sides have to delete + expunge the message (because different IMAP clients might have different ideas on what message UID 100 is) and create new UIDs 101 and 102 with the two messages. This can be messy because if a POP client is connected, you can't alter the mailbox at all because the message list isn't allowed to change under the POP clients feet, so connected POP clients could cause nasty locking issues. As an FYI, this is basically what we currently do with active/passive replication if we have to do a hard failover, but we have to do it all by hand. In a controlled failover, we make sure all the sync logs are correctly played before switching roles, so we know master/replica are in sync when we change roles. However if a server hard fails (eg kernel panic or some other OS lockup), then we may have to switch roles, without knowing if the logs have been cleanly played. When we get the dead server back online, it may be that a message with the same UID exists on both ends. In the current cyrus code, the master wins, and overwrites the message on the replica. That's dangerous, and might end up destroying messages. We have a patch for cyrus that checks for this case, and compares the GUID of the messages (basically SHA1 of message content), and if they're different, it refuses to overwrite the message, and instead just logs a message to syslog. We have another script that notices those syslog messages, and emails us. We then have yet another script, that lets us inspect the offending message on both sides (eg master and replica), to see what the situation is. In most cases, it's a message that was clearly delivered to one side, but not replicated to the other before the machine crashed. We can then give a flag to the script that makes it delete the UID on both sides, and then re-appends both messages to the current master server, causing both messages to get new UIDs. Ideally, this whole process would be automated, and at some point we probably will make our scripts do it automatically, but it would probably be nice if this was eventually pushed down into the cyrus replication protocol somehow. As Bron noted, there would still be lots of other things to make active-active replication work properly (thinking about folder renames while someone else has a folder selected on the other server really scares me), but at least this would deal with the most common active-passive case where a non-graceful failover occured. Rob Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: cyrus replication : master to replica and replica to master
Well - it's theoretically possible. But I don't know anyone who's done it, and it has the potential to get ugly if you're delivering to the same mailboxes at each end. There's nothing I can see that would actually stop it working. I think Bron failed to put sufficiently large warning signs here :) The difference between in theory this would work and the practice of actually doing it are huge. Basically it works only if you are 100% sure that only one side is ever being accessed at a time. eg. IMAP/POP/LMTP/etc. If somehow you get an LTMP deliver to different ends, then one side will end up overwriting the other. In other words, DON'T DO THIS. Rob Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Cyrus-Imapd on Linux-iSCSI
- cyrus-imapd-2.3.7 (from RHEL5/CentOS-5) with some minor patches in the popd (UUID format and an enhancement to the authentication - both shouldn't have any impact on the storage part) As I'm sure others will mention, this is a quite old cyrus now with many known bugs. You chould definitely consider upgrading to Simon Matters RPMs http://www.invoca.ch/pub/packages/cyrus-imapd/ It happened now and then and is somewhat inconvenient. And it's not fun if the /var/log partition is full just because cyr_expire stumbles over a corrupted BDB. We used to see this every now and then as well, but it's stopped since we fixed this bug: http://www.mail-archive.com/cyrus-de...@lists.andrew.cmu.edu/msg01123.html I'm not sure if Bron has commited this back to cyrus CVS for the next release yet, but if not, it should definitely get included. We switched to iSCSI just recently - so that's the only really new thing in the setup. I think it's either a coincidence (eg the bug is not iSCSI related), or if it is iSCSI related, it's not directly related, and probably just caused by timing race changes created by a change in the storage layer. Rob Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: incremental squatter
Also, does anyone know what this means for searches on material that has changed since the last squatter run? I have assumed, and hope, that the search procedure is something like this: search in the squatter index remove results referring to deleted items do an unindexed search on items added since last index. Is that right? Or, for example, are new messages just ignored? The squatter index isn't a perfect index. What it does is given a search term, it returns a list of messages that might contain the term, and excludes messages that definitely do not contain the search term. For each message that squatter says might contain the search term, cyrus then opens the message and does a complete search on it to see if it definitely contains the search term. Because of that, if squatter sees a message id it hasn't indexed, it will always return that id, because that id might contain the term, it doesn't know. The net result is that things work as expected. New messages that haven't been squatter indexed are always searched, you never miss messages. Rob Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: imapd -U in cyrus.conf
We also wanted to use -U 1 so we could be sure changes to imapd.conf would be used more quickly since there wouldn't be imapd procs hanging around with old settings. FYI, another way of doing this without forcing -U 1 is to touch the imapd executable file. The cyrus master notices if the executable file mod time on disk has changed, and when the user logs off or disconnects from the imapd, it'll terminate it. Rob Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Registry activity log
Check out the auditlog patches we have here: http://cyrus.brong.fastmail.fm/ It syslogs every message/folder create/delete in a fairly regular format. eg things like: auditlog: create sessionid=... mailbox=... uniqueid=... auditlog: delete sessionid=... mailbox=... uniqueid=... auditlog: append sessionid=... mailbox=... uniqueid=... uid=... guid=... message-id=... auditlog: expunge sessionid=... mailbox=... uniqueid=... uid=... auditlog: copy sessionid=... srcmailbox=... srcuniqueid=... srcuid=... mailbox=... uniqueid=... uid=... guid=... message-id=... Rob - Original Message - From: Ana Ribas Roca ana.ri...@upcnet.es To: info-cyrus@lists.andrew.cmu.edu Sent: Tuesday, June 30, 2009 8:58 PM Subject: Registry activity log I'd like knowing if I can obtain more log information, than the habitual in cyrus, about the imap accesses. I need to register all the activity of every mailbox (deleted messages, folders creation ...) Is it possible? How? I've changed the syslog configuration file, adding local6.info and local6.debug, but i haven't seen any difference in the log file. Can someone help me, please? Thank's in advance and sorry for my bad english. -- Anna Ribas Roca Projectes Tecnològics UPCnet, Universitat Politècnica de Catalunya 08034 BARCELONA Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Problem with websieve and accents
I am blocked with websieve.pl vacation/out-of-office because when users are entering accents the script fails with an error... Does anyone has a suggestion on how to make accents work ? Yes, but I haven't committed it to CVS yet. I'm working on full UTF8 support in sieve scripts. Well it depends where there are accents in the sieve script. I believe the current state of things (2.3.14) is: 1. In folder names - should work (you need to use the utf-8 version of the folder name, not the modified utf-7 version) 2. In vacation messages - should work 3. In header/body checks - needs Bron's fixes And a reminder, the script must be in utf-8 encoding. Rob Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Cyrus APIs ?
- IMAP protocol extensions (most needed thing would be to idle on every folders, not just inbox) Yeah, good luck with that one. It's a pretty major protocol extention, and everything's very folder centric. It would be a rather large SMOP (small matter of programming) for this. If you have idled stupport, this shouldn't be too hard to implement. Basically the way idled currently works is that a process sends it a message about which mailbox it's interested in, and then idled sends it a USR1 sig if something changes on that mailbox. I think all you'd need to do is loop and send multiple mailbox names to idled, and it'll USR1 you when any of them change. Of course you then have to poll to find out which ones, which is a bit annoying, but would be fixable with a bit more work. Rob Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Does Cyrus benefit greatly from increased FS buffer cache?
Our plan is to throw 12-16GB at it, with the purpose of vastly increasing the FS buffer cache (and decreasing I/O). Or, will that just be a waste of RAM? Some indications are that, yes, it does improve performance notably: http://blog.fastmail.fm/2007/09/21/reiserfs-bugs-32-bit-vs-64-bit-kernels-cache-vs-inode-memory/ Anyone have any specific thoughts? Is there any other benefit we might see from large memory allocation in 64-bit architecture? Given that I wrote that blog post, I can only tell you that in our environment, 64-bit kernels made a big difference. Your environment sounds pretty small though, so you might want to see what your current load actually is. If it's already 1, then adding extra RAM + a 64-bit kernel isn't going to buy you that much more. On the other hand, RAM is pretty cheap, and it might be worth doing it and going to 64-bit so you don't have to deal with it one day in the future... Rob Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Intermittent mailbox not found
I have inherited a cyrus / postfix server, which is generally well-behaved. However, there is an intermittent error when delivering messages to a certain mailbox. It is infrequent, and generally the message is delievered on the user's next attempt in his MUA. The log is something like: What version of cyrus and what version of postfix? Can you should the logs for a successful delivery to that account. Rob Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Doing Cyrus *right* for a large installation
I would like to add a *lot* more storage so that we can increase our email quotas (currently 200MB per user). It seems like the proper way to scale up is to split the Cyrus metadata off and use some large SATA drives for the message files. I was considering adding a shelf of 1TB SATA drives to our SAN. I could store the metadata on existing FC drives on the SAN, or just use internal disks on the servers. We split our meta/data onto 10k/15k RPM RAID1 for meta vs 7.2k RPM RAID5 for data. The meta is 1/20th the size of the data. The meta drives get more data written to them, the data drives get more data read (we have lots of memory now, so probably lots of meta is cached). On average, utilisation for meta is still higher than data, but they're relatively well balanced with that split it seems. But then I started thinking about how I was going to backup all this new data... Our backup administrator isn't too excited about trying to backup 12TB of email data. We backup to a X4500 server. Bron built our custom backup system for cyrus. Each cyrus machine has a backup daemon that has a simple network protocol. The daemon knows where meta and data files are, and can read and understand cyrus.* files. A backup process on the X4500 runs each day and connects to the daemon on each cyrus machine and uses it to find out changes for each user for each folder and updates the backup on the X4500. All backups are stored in .tgz streams with a copy of every email, every cyrus.* file. Meta data is stored in an sqlite file. In general the backup process just appends to the .tgz stream. When it calculates that the ratio of old data in the .tgz is too high, it re-packs the whole thing removing the old data. The whole thing relies a lot on internal knowledge of our setup, so it's not something we can easily release unfortunately. What if we used Cyrus replication in combination with delayed expunge as a form of backup? We currently only keep 1 month of daily backups anyways... It's an option, but it's a bit scary still. What if there's a replication protocol error that blows away your replica? Unlikely, but possible. I think we might be a bit paranoid. We don't like relying on any one thing. Filesystems, software, hardware, etc. Net result is we've ended up with quite a few levels of redundancy. 1. All data on RAID so any HD failure is just a replacement HD no downtime at all 2. Delayed delete, so any user deletion error can be fixed by re-inserting the deleted messages 3. All data replicated, so any server/storage unit failure is just switching master/replica 4. Nightly backups to a completely separate server, with different OS and filesystem, and with no shared credentials or trust. Basically a last resort in case of major hardware/OS/security screw up that you absolutely hope you never have to use. Rob Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Elusive replication bug in 2.3.13
Since I upgraded to 2.3.13 (Invoca RPM rev 4) I've been running into a mysterious replication bug. In some circumstances, creating a user with a three letter long username causes the sync master process to choke, on either signal 11 or 6. Like this: Interestingly, we just encountered this as well after a new rebuild from CVS. I got a stack back trace, which showed realloc() was choking. I looked at the code around there, and couldn't see anything obviously wrong. Bron looked at it and saw the problem in 10 seconds. I think I've been out of practice with C programming for too long :) The bug occured in this commit where Ken changed a glob() call to a custom readdir() loop. http://github.com/brong/cyrus-imapd/commit/56d0823d3192ea13fa3afe397d625252b16252b9 There's a missing sizeof() in there meaning that if you have 25 quotas in a dir (or 12 on a 64-bit build), you'll have memory corruption occuring. Here's the patch to fix: http://github.com/brong/cyrus-imapd/commit/1de9d758aeb360714236388c4e1689db0522c21e All the fastmail patches are now on github, you can see them here: http://github.com/brong/cyrus-imapd/commits/fastmail/ http://github.com/brong/cyrus-sieve/commits/fastmail/ Rob FYI: We haven't changed to skiplist for quota db's because they get a lot of writes, and a skiplist db requires locking the entire db for the period of the write, so I'm worried about contention issues. To be honest, I haven't tested, so it might not be a problem at all, it's just something we'd need to test first. Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: choosing a file system
We've found that splitting the data up into more volumes + more cyrus instances seems to help as well because it seems to reduce overall contention points in the kernel + software (eg filesystem locks spread across multiple mounts, db locks are spread across multiple dbs, etc) Makes sense. Single cyrus env here, might consider that in the future. At that point though, I'd probably consider Murder or similar. That should work fine as well. I believe murder just does two main things. 1. It's merges the mailboxes.db from each instance into each other instance, so you end up with just one giant single namespace 2. It proxies everything (imap/pop/lmtp) as needed to the appropriate instance if it's not the local one We don't use murder as we don't really need (1), and we do (2) ourselves with a combination of nginx and custom lmtpproxy tool. Rob Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: choosing a file system
Running multiple cyrus instances with different dbs ? How do we do that. I have seen the ultimate io-contention point is the mailboxes.db file. And that has to be single. Do you mean dividing the users to different cyrus instances. That is a maintenance issue IMHO. As Bron said, yes it is, but if you have more than 1 machines worth of users anyway, you have maintenance issues anyway. So rather than just one instance per machine, we run multiple instances per machine. The only issue it really introduces is that folder sharing between arbitrary users isn't possible (unless you used murder to join all the instances together again, but we don't), only users within an instance can share. I had the feeling whatever optimizations done at the FS level would give us a max of 5-10% benefit. We migrated from ext3 to reiserfs on our cyrus servers with 30k mailboxes. I am not sure I saw a great benefit in terms of the iowait. At peak times I always see a iowait of 40-60% To be honest, that's not what we saw in our ext3 - reiserfs tests. What mount options are you using? Are you using the mount options I mentioned? noatime,nodiratime,notail,data=ordered But the new Solid-State-Disks seem very promising. They are claimed to give 30x the throughput of a 15k rpm disk. If IO improves by 30 times that should make all these optimizations unnecessary. As my boss used to tell me ... Good hardware always compensates for not-so-good software. What we've found is that the meta-data (eg mailbox.db, seen db's, quota files, cyrus.* files) use WAY more IO than the email data, but only use 1/20th the space. By separating the meta data onto RAID1 10k/15k RPM drives, and the email data onto RAID5/6 7.2k RPM drives, you can get a good balance of space/speed. Rob Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: choosing a file system
Ext4, I never tried. Nor reiser3. I may have to, we will build a brand new Cyrus spool (small, just 5K users) next month, and the XFS unlink [lack of] performance worries me. From what I can tell, all filesystems seem to have relatively poor unlink performance and unlinks often cause excessive contention and IO for what you think they should be doing. And it's not just filesystems. SQL deletes in MySQL InnoDB are way slower than you'd expect as well. Maybe deletes in general are just not as optimised a path, or there's something tricky about making atomic deletes work, I admit I've never really looked into it. Anyway, that's part of the reason we sponsored Ken to create the delayed expunge mode code for cyrus, which allows us to delay unlinks to the weekends when IO load due to other things is the lowest. --- Added support for delayed expunge, in which messages are removed from the mailbox index at the time of the EXPUNGE (hiding them from the client), but the message files and cache entries are left behind, to be purged at a later time by ttcyr_expire/tt. This reduces the amount of I/O that takes place at the time of EXPUNGE and should result in greater responsiveness for the client, especially when expunging a large number of messages. The new ttexpunge_mode/tt option in ttimapd.conf/tt controls whether expunges are immediate or delayed. iDevelopment sponsored by FastMail/i. --- Rob Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: choosing a file system
Now see, I've had almost exactly the opposite experience. Reiserfs seemed to start out well and work consistently until the filesystem reached a certain size (around 160GB, ~30m files) at which point backing it up would start to take too long and at around 180GB would take nearly a week. This forced us to move to ext3 and it doesn't seem to be degrade that way. We did, however, also move from a single partition to 8 of them, so that obviously has some effect as well. As you noted, changing two variables at once doesn't help you determine which was the problem! Multiple partitions will definitely allow more parallelism, which definitely helps speed things up, which is one of the other things we have done over time. Basically we went from a few large volumes to hundreds of 300G(data)/15G(meta) volumes. One of our machines has 40 data volumes + 40 meta data volumes + the standard FS mounts. $ mount | wc -l 92 We've found that splitting the data up into more volumes + more cyrus instances seems to help as well because it seems to reduce overall contention points in the kernel + software (eg filesystem locks spread across multiple mounts, db locks are spread across multiple dbs, etc) Also one thing I did fail to mention, was that for the data volumes, you should definitely be using the notail mount option. Unfortunately that's not the default, and I think it probably should be. Tails packing is neat for saving space, but it reduces the average meta-data density, which makes stating lots of files in a directory a lot slower. I think that's what you might have been seeing. Of course you also mounted noatime,nodiratime on both? I think that's another problem with a lot of filesystem benchmarks, not finding out what the right mount tuning options are for your benchmark. Arguing that the default should be fine is clearly wrong, because every sane person uses noatime, so you're already doing some tuning, so you should find out what's best for the filesystem you are trying. For the record, we use: noatime,nodiratime,notail,data=ordered On all our reiserfs volumes. Rob Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: choosing a file system
There are /lots/ of (comparative) tests done: The most recent I could find with a quick Google is here: http://www.phoronix.com/scan.php?page=articleitem=ext4_benchmarks Almost every filesystem benchmark I've ever seen is effectively useless for comparing what's best for a cyrus mail server. They try and show the maximums/minimums of a bunch of discrete operation types (eg streaming IO, creating files, deleting files, lots of small random reads, etc) running on near empty volumes. What none of them show is what happens to a filesystem when it's a real world cyrus mail spool/index: * 100,000's of directories * 10,000,000's of files * 1-1,000,000 files per directory * files continuously being created and deleted (emails) * data being appended to existing files (cyrus.* files) * lots of fsync calls all over the place (every lmtp append has multiple fsyncs, as well as various imap actions) * run over the course of multiple years of continuous operations * with a filesystem that's 60-90% full depending on your usage levels There's serious fragmentation issues going on here that no benchmark even comes close to simulating. Now from our experience, I can tell you that ext3 really does poorly on this workload compared to reiserfs. We had two exact same servers, one all reiserfs and one all ext3. The ext3 one started out ok, but over the course of a few weeks/months, it started getting worse and worse and was eventually being completely crushed by IO load. The machine running reiserfs had no problems at all even though it had more users on it as well and was growing at the same rate as the other machine. Yes we did have directory indexing enabled (we had it turned on from the start), and we tried different data modes like data=writeback and data=ordered but that didn't help either. To be honest, I don't know why exactly, and working out what's causing IO bottlenecks is not easy. We just went back to reiserfs. Some previous comments I've made. http://www.irbs.net/internet/info-cyrus/0412/0042.html http://lists.andrew.cmu.edu/pipermail/info-cyrus/2006-October/024119.html The problem with reiserfs is... well. The developers have explicitely stated that the development of v3 has come to its end, and there was the In this particular case, I'm really almost happy with this! Reiserfs has been very stable for us for at least 5 years, and I'm almost glad no-one is touching it because invariably people working on something will introduce new weird edge case bugs. This was a while back, but it demonstrates how apparently just adding 'some sparse endian annotations' caused a bug. http://oss.sgi.com/projects/xfs/faq.html#dir2 That one was really nasty, even the xfs_repair tool couldn't fix it for a while! Having said that, there have been some bugs over the last few years with reiserfs, however the kernel developers will still help with bug fixes if you find them and can trace them down. http://blog.fastmail.fm/2007/09/21/reiserfs-bugs-32-bit-vs-64-bit-kernels-cache-vs-inode-memory/ http://lkml.org/lkml/2005/7/12/396 http://lkml.org/lkml/2008/6/17/9 Rob Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Too many open files
Shelley Waltz wrote: Oddly enough I had the identical issue today while trying to remove a bunch of users. I am running cyrus-imapd-2.3.7-2.el5 on RHEL5. I used cyradm and did sam user.x anyone c dm user.x This worked fine for about 20 users, then I got the following in maillog ... Nov 3 10:35:28 rr imap[16539]: IOERROR: opening quota file /var/lib/imap/quota/x/user.x: Too many open files Nov 3 10:35:28 rr imap[16539]: DBERROR: error fetching user.x: cyrusdb error Nov 3 10:35:28 rr imap[16539]: Deleted mailbox user.x If I look in /var/spool/imap/x/user/ the directory and folders for user x are still there. If I try to remove the mailbox once again using cyradm, it claims the mailbox does not exist. How does one resolve this issue safely? I have the same problem with a few mailboxes myself, and don't know how to clean it up. To avoid the problem in the first place, if you have just a few (tens ? couple of hundred) users beginning with the same prefix, I think you can avoid the problem in the first place by putting e.g. imap cmd=imapd listen=imap prefork=0 maxfds=4096 in /etc/cyrus.conf. I suspect the underlying cause is this chunk from lib/cyrusdb_quotalegacy.c(foreach): /* strip off the qr specific path and replace with pattern */ sprintf(strstr(quota_path, FNAME_QUOTADIR) + strlen(FNAME_QUOTADIR), ?/%s*, prefix); /* search for the quotaroots */ glob(quota_path, GLOB_NOSORT, NULL, globbuf); That pattern on our boxes can match thousands of users. I think it needs to be more careful and go looking for `prefix' and `prefix.*', but I haven't dug into it far enough yet. Cheers, Rob -- E-Mail: [EMAIL PROTECTED] PHONE: +44 24 7652 3037 Rob McMahon, IT Services, Warwick University, Coventry, CV4 7AL, England Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Too many open files
Rob McMahon wrote:
Cyrus-2.2.13 (yes, I know it's getting on a bit). Solaris 10 SPARC,
around 115,000 mailboxes.
I was mopping up a few users today with a script which basically does:
while () {
chomp;
my $mb = 'user.' . $_;
$client-setacl($mb, $user = 'c');
$client-delete($mb);
}
After deleting just a few, I start getting errors like Must login
first, System I/O Error (sorry, I haven't got the exact messages to
cut paste). Looking in the logs I see:
Aug 24 16:20:44 narcissus imap[24338]: [ID 240394 local6.error] IOERROR:
opening quota file /var/imap/quota/V/user.??.Trash: Too many open files
Aug 24 16:21:38 narcissus imap[24338]: [ID 816447 local6.error] warning:
cannot open /etc/hosts.deny: Too many open files
and indeed this process has hit the default 256 limit, most of which
seem to be quota files. Other issues were manifesting, IMAP commands
never completing for instance, so I had to kill this process, which in
turn caused the master process to restart. Not good.
I think I've just found an answer to this. Looking at the files open by
one of these processes (I'd upped descriptors to 2048), it appears that
if you try to delete, say, user.a, the IMAP process ends up quota files
for every user beginning with a a ... sounds mad but that is what I've
observed, and if I'm careful about not deleting short names it goes
through okay. I'll try to investigate further. I know this is old
code, and the algorithms have probably changed by now.
Cheers,
Rob
--
E-Mail: [EMAIL PROTECTED] PHONE: +44 24 7652 3037
Rob McMahon, IT Services, Warwick University, Coventry, CV4 7AL, England
Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Pause while selecting INBOX
I'd probably use imtest to connect, get the PID of the server process that I'm connected to, and then attach to that process with ktrace (or whatever) with timestamps enabled. Then I'd select the mailbox -- this is assuming that mutt is only issuing a select when it says Selecting INBOX. Obviously it could be doing any number of things. You can get positive confirmation of which command is taking a long time by enabling telemetry, of course. Using ktrace/strace may not actually be that helpful. cyrus read accesses most files via mmaped memory. The problem is that opening a file + mmaping it might be quick, but then when the process does a memory access to the mmapped region, the whole process is put to sleep waiting on the underlying page fault to finish doing the IO. That doesn't show up in strace/ktrace because it's just a memory access, not a system call. In general, tracing where IO bottlenecks are occuring can be quite tricky. Rob Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: IMAP Date-Time format for Append
We're pretty sure this is related to the timezone being represented as a short name, as opposed to in numeric format (+0400, etc.). The IMAP spec is vague on whether or not this format should be accepted. I believe that this has to do with the way the function from the C Library converts the string, but am not sure. The spec isn't vague at all: http://www.faqs.org/rfcs/rfc3501.html append = APPEND SP mailbox [SP flag-list] [SP date-time] SP literal date-time = DQUOTE date-day-fixed - date-month - date-year SP time SP zone DQUOTE zone= (+ / -) 4DIGIT ; Signed four-digit value of hhmm representing ; hours and minutes east of Greenwich (that is, ; the amount that the given time differs from ; Universal Time). Subtracting the timezone ; from the given time will give the UT form. ; The Universal Time zone is +. Seems pretty clear that what the client is doing is broken. Having said that, the parsing code is in cyrus, so changing the C library won't help. You'll need to dig into the cyrus source.Rob Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Pause while selecting INBOX
We have a moderately sized Cyrus installation with 2 TB of storage and a few thousand simultaneous IMAP sessions. When one of the backup processes is running during the day, there's a noticable slowdown in IMAP client performance. When I start my `mutt' mail reader, it pauses for several seconds at `Selecting INBOX'. That behavior disappears when the backup finishes. Where's the first place to look for this problem? I/O statistics show a higher read bandwidth while the backup is running, but writes still dominate. The backup would typically read all of the files in a single Cyrus partition. Some more information about your setup would be helpful. 1. What OS? 32 or 64bit? 2. How much RAM do you have? How much is used by cache? 3. What sort of drive system is the data stored on? 4. Do you have split data/meta data? 5. What filesystem are you using? 6. What mount options to you have on your volumnes? 7. What IO scheduler are you using? 8. What version of cyrus are you using? 9. Do you have the cyrus status cache enabled? Some things to try if you haven't: 1. Make sure you have noatime (and nodiratime if applicable) set on your FS 2. If on linux, use the deadline IO scheduler 3. If you can, split your data and meta data onto separate volumes (probably a lot of work) 4. Install more RAM and use a 64-bit OS. x64_64 linux can cache many, many more inodes and dentries in memory than x86 linux. 5. Upgrade to the latest cyrus and enable the status cache Rob Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Pause while selecting INBOX
On Sep 26, 2008, at 06:44, Gary Mills wrote: We have a moderately sized Cyrus installation with 2 TB of storage and a few thousand simultaneous IMAP sessions. When one of the backup processes is running during the day, there's a noticable slowdown in IMAP client performance. When I start my `mutt' mail reader, it pauses for several seconds at `Selecting INBOX'. That behavior disappears when the backup finishes. Where's the first place to look for this problem? I/O statistics show a higher read bandwidth while the backup is running, but writes still dominate. The backup would typically read all of the files in a single Cyrus partition. As always, the answer is probably complicated. What's your storage backend look like? Some sort of RAID with cache? The backup is very probably blowing out your cache while running, which can manifest itself in ways that will make it look like the performance profile of your storage has taken on a different personality. Especially with respect to writes, since you might usually be blessed with an abundance of write-back cache that's now fighting with all the reads for the backup. Do you have your cyrus partitions mounted with noatime? Your backups could be causing fs write operations that you really don't want. -rob Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Sieve: UTF-7 changes in 2.3.12p2?
Not out of my head, but we probably need to add a configuration setting in Ingo to switch between the old and new behavior. Please add a request on http://bugs.horde.org. Might be worth trying to make it an imapd.conf option for cyrus as well to choose between the old or new behaviour to make upgrades for users easier. I think a patch should be fairly easy if you look into it. Unfortunately i don't have the time right now... Rob Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Sieve: UTF-7 changes in 2.3.12p2?
Is this a bad joke or am I missing something? Sieve scripts of most non-English-speakers are intentionally broken due to a BC breaking change in a bugfix release version? I'm afraid you're not missing anything. It bit us as well :( Fortunately for us, there weren't that many people with non-english folder names with filter rules, but for other people, this could be a pretty nasty change. The change is actually the correct fix, but dropping it in without any large warning in the changelog or release announcement is pretty rough. Fixing all the existing sieve scripts shouldn't be too hard though. A perl 1-liner could almost do it... perl -pi -e 'use Encode; use Encode::IMAPUTF7; s/\bfileinto\s+([^]*)/$f=$1;from_to($f,IMAP-UTF-7,utf-8);$f;/ge;' /path/to/sievescripts/* You'll need to recompile the script as well. Completely untested, but should get you on the way. Rob Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Sieve: UTF-7 changes in 2.3.12p2?
The mail is delivered to the INBOX (user.fri) The delivery is tried to user.fri.t-APY-ster, NOT user.fri.tAPY-ster Any hints where the error occurs? Yes. Sieve scripts are in utf-8. You now have to use the true utf-8 name of the folder in your sieve script, and it's automatically converted to the modified utf-7 folder name internally. This change is noted in the changelog for 2.3.11 --- The Sieve compiler now translates UTF-8 mailbox names used in fileinto to modified UTF-7 so that messages will be successfully inserted into the mailbox. --- The old way of using the modified utf-7 form in the sieve script was really wrong and hacky. Rob PS. It would be nice if http://cyrusimap.web.cmu.edu/imapd/changes.html was kept up to date or removed... it's very old. Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Several cyrus instances on same host.
Is it possible to run several cyrus imap instances (with different cyrus.conf and imapd.conf files) on the same server? I will like to have all related files for imap server A in one directory (/imapA) and all the related files for imap server B in another directory (/imapB). Is it possible? No problem at all. We run up to 20 different instances of cyrus on some of our machines. You just have to setup separate cyrus.conf and imapd.conf files, separate start/stop scripts, and use the -C option on every command you run. The main thing I'd recommend is that you create separate instance files like: /etc/cyrus-a.conf and /etc/imapd-a.conf /etc/cyrus-b.conf and /etc/imapd-b.conf And completely delete /etc/cyrus.conf and /etc/imapd.conf. That way if you run reconstruct or any other tool on it's own, it'll complain that it couldn't find a imapd.conf file. You have to specify the -C option, so that you explicitly tell the command which instance you want to work with. By not having a default install, this will avoids mistakes with running the wrong command. One other thing you'll have to do is bind each instance to different port numbers, or different IP addresses. Rob Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: cyrus-imap - High consumption of memory
I have a problem with a high memory consumption using cyrus-imap 2.2 on FreeBSD 7.0. Each processes of imapd consume 30mb of ram ! I believe that this is not normal. Is there a way to adjust this? This is probably fine. cyrus mmap's lots of files, which often make the process look bigger that it really is, because the OS will just page in data as required. Also a number of the mmaped files are shared between process (eg skiplist mailboxes.db), so if you have a 30M mailboxes.db, it'll make each process look like it's at least 30M in size, but in reality, that 30M is shared between all processes. Rob Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Seen database issue
Moving the /var/spool/imap directories, and /var/lib/user/{}.seen
files to the new server and reconstructing works fine except that all the
mail shows up as not read on the new sever.
The seen state is keyed on the mailbox uniqueid, so if that changes, the
seen state becomes invalid.
The mailbox uniqueid is based on the mailbox name + the uidvalidity value,
which is itself based on the time the mailbox is created, or recreated due
to a reconstruct.
Now I thought reconstruct should preserve the mailbox uniqueid in most
cases, but it's possible it's not if you're not copying all the data
correctly.
1. Why are you reconstructing the mailboxes. If you copy all the right data,
it should be fine without a reconstruct
2. How are you copying the data? Recommend using rsync -az which will
preserve all the attributes + timestamps
3. Are you using split data/meta data? If so, are you copying the metadata
as well?
4. What flags are you passing to reconstruct?
Rob
Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Any command to get rapidly ALL annotations?
To get a tiny statistic we are going through all mailboxes and use GETANNOTATION to retrieve possible annotations, which is a time consuming progress. GETANNOTATION does not like wildcards like LIST. Yes it does. Bah, seems the draft is up to -13 now, and they've actually changed the IMAP command to GETMETADATA. Anyway, the cyrus implementation still uses GETANNOTATION and seems to implement somewhere around -05 draft. http://ietfreport.isoc.org/all-ids/draft-daboo-imap-annotatemore-05.txt Both * and % list wildcard characters MAY be used in the mailbox name argument to commands to match all possible occurrences of a mailbox name pattern. However, * or % by themselves MUST NOT match the empty string (server) entries. Server entries can only be accessed by explicitly using the empty string as the mailbox name. I'm pretty sure this works from my memory and testing. Rob Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Cyrus infrastructure performance less than expected
On current production platform the frontends use swap massively but the impact is far less than on the new platform. It's not so much how much swap is actually used, but how much is being paged in or paged out at any time. If there are memory pages not being used at all, the OS will swap them out, but that's fine because they're not being used. The problem is when more memory pages are being used than memory available. In that case the OS starts continuously moving things back and forth between RAM and swap, which causes performance to become horrible. Can you run free and vmstat 10 10 (will take 100 seconds to run) on one of your old frontends and one of your new frontends during peak times so we can get an idea of what is going on with memory + swap. Also lets just confirm a process count on your system. Run these commands: ps ax | wc -l ps ax | grep imapd | wc -l On both old and new. We are indeed using 64bits OS and 64bits software. This is the main difference between the old platform which runs a 32 bits software on a 32 bits OS. That will increase memory usage as well. Every pointer in the system (either in the kernel or in an application) is now using twice as much space. Of course not all data is pointers, but depending on the application it can mean considerably more memory is used. Lets start with the free, vmstat and ps data, and we'll go from there. Rob Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: cyrus perdtion , connections not dying
Usually we have around 700k simultaneous pop connections on the real servers now with perdition we have 3000+ connections Er, I didn't write that. That was Ram, the OP. Different person, different institution, similar issues. We don't offer POP here. Ah oops, got a bit mixed up there. We ran into the same problem, the performance limitation on our front end cluster was (is) the number of processes. Unfortunately, the same problem applies. Actually, it's worse. Cyrus processes use more RAM: typically 2.0-3.5 MB per process as opposed to 1.5 - 1.9 MB per process. I've even seen cyrus processes with up to 30MB. There seems to be some confusion here over if the problem is the size of the processes, or the actual number of processes. In many cases, the size of the process won't be the problem. In all UNIX OS's, the size the process is doesn't necessarily correlate very strongly to how much memory it's really using because lots of that can be shared between many processes (COW forking), or just mmaped files into the process space, which might not actually be all in memory at that moment. What's the actual memory status of your production machine (used/free/etc) and the cpu status of your production machine (idle/user/system/iowait)? Then you can work out what your real bottleneck is first. Rob Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: cyrus perdtion , connections not dying
opposed to 1.5 - 1.9 MB per process. I've even seen cyrus processes with up to 30MB. cyrus uses mmap a lot. The processes probably aren't actually really growing, they just look bigger because a user has a folder selected and the cyrus.index file has been mmaped into the process space. If they unselect the folder, the process size will drop back down again. How much RAM does your machine have and how many users are you trying to support? As a point of reference. We have quite old machines with 8G RAM, 8T attached storage that easily handle 2000+ processes under linux just fine. Rob Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Slow local network performance for image attachments
Squirrelmail on localhost:. very fast, in the blink of an eye, likewise with Outlook Express (perish the thought). So I guess it's Thunderbird. That sucks. I guess I'll take this to a Thunderbird forum. Thanks for pointing me in the right direction. Some days after troubleshooting everyone else's computer problems, there's nothing left in the tank for my own... These are a little old, but it's possible they still apply. The default settings for Thunderbird IMAP handling with large emails with attachments sucks... badly. http://bleedingedge.com.au/blog/archives/2005/11/waiting_for_thunderbird.html http://bleedingedge.com.au/blog/archives/2005/11/speeding_thunderbird.html http://bleedingedge.com.au/blog/archives/2005/11/making_thunderbird_fly.html Rob Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Miserable performance of cyrus-imapd 2.3.9 -- seems to be lockingissues
but all attempts to simulate a client load-pattern are devilishly difficult to get right. I can atest to this as well. I created an imapstresstest tool a few years back to attempt to stress our cyrus installs. It attempts to emulate all the main actions of a running IMAP server like lots of IMAP connections (1000) with status/select/fetch/copy/create/rename/delete commands, and lots of LMTP email deliveries, etc. I used to be able to generate reasonable loads, but it never was quite the same as what actual users would do to the system. As an FYI on RAID controllers, we've used ARECA controllers (http://www.areca.com.tw) and they've been excellent in RAID-1, RAID-5 and RAID-6 modes. They include up to 1G of battery backed up cache. Rob Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Dual Quad Core processors
Does Cyrus-imapd take advantage of Dual and\or Quad core processors? We are looking at upgrading our server to either 2x Dual core Xeon's or 1 x Quad core Xeon processor. Does Cyrus have the ability to take advantage of this? Since it uses a multi-process model, yes it does. However that's not what you should upgrade your servers for. cyrus uses very little CPU time on any modern processor. Even with 10,000's of users, our servers with 3 year old 2.4Ghz Netburst Xeons never get more than about 30% CPU usage (out of 400% because they're dual processor, hyper-threaded machines). Instead cyrus is incredibly IO hungry. You should have as much RAM as possible for caching, and a battery backed NVRAM RAID controller to try and improve the random write IO that's generated. These will do much more for you than upgrading your CPU will. Rob Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Non-ASCII chars in MAIL FROM envelope
Yes, but that will block possibly valid mail. Of course I don't accept mail with non-ASCII RCPT TO addresses because Cyrus doesn't allow it, but I should accept non-ASCII MAIL FROM addresses if they are valid. But Cyrus also refuses them. That's the real problem. Can you turn off the 8BITMIME extension that postfix advertises? It seems from the RFC (http://tools.ietf.org/html/rfc4952), if 8BITMIME isn't advertised, then it's up to the sending side to downgrade the SMTP transaction to ASCII, and then you could block non-ASCII MAIL FROM addresses... Rob Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Non-ASCII chars in MAIL FROM envelope
Can you turn off the 8BITMIME extension that postfix advertises? It seems from the RFC (http://tools.ietf.org/html/rfc4952), if 8BITMIME isn't That seems to be the right logic but I can't find the proper way to do it with postfix. Anyway, I'm not sure about the whole 8BITMIME thing and whether it counts for message content or envelope or both? It it counts for both, why does Cyrus lmtp advertise 8BITMIME and then refuses to accept mail with 8bit envelope. I'm really confused and from looking at different list archives I see that I'm not alone. Agreed. From what I can see, the original 8BITMIME extension was 8-bit message content, but has been updated to include 8-bit SMTP transaction envelopes as well. It might be a mismatch between that over time (eg postfix changed at 2.4.5 as well) Rob Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Non-ASCII chars in MAIL FROM envelope
Okay, so you confirm my impression that message content and envelope seem to be treated the same way now (for example with postfix). Then, wouldn't it make sense to do the same with Cyrus? Could parseaddr() be relaxed to do this? It sounds like the right approach, but there's probably quite a bit of code to check for assumptions... Rob Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Plugging into the imap system
There is some system (on freshmeat?) that has a special folder in IMAP for storing calendar events. The program uses the IMAP defined protocol though. http://www.kolab.org/documentation.html It's a reasonably interesting idea. Basically use a sort of folder=table and email=row database mapping to store the data. Ok more email = structured XML blob of data, but the overall concept is that the IMAP server acts as a centralised store of information that multiple clients can sync from. Actually a few nice things about this approach: 1. You reuse an existing protocol and servers rather than creating a bunch of new ones 2. Because IMAP is designed so the server is authoriative and you have long term connections, you can avoid many of the sync issues that irregular sync systems end up dealing with 3. Emails are immutable, so changing anything is a delete + upload new action, so again, other clients can easily detect when changes have occured on the server I really wish the Bynari and Toltec Outlook plugins were just a bit more stable and sane though... Rob Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: 4xusers imap processes?
Its OK that they hang around -- in fact, its encouraged. You could
set the -U flag on imapd, etc to 1, then the process should die
after its first use.
-rob
On Dec 19, 2007, at 14:56, Mike Eggleston wrote:
I run a small shop and have around ~18 users. Checking just now I have
70 imap processes. Is there an idle value I need to set so these
process
don't hang around? All users use sieve, all users but two use imap.
The
two remaining use pop.
/etc/cyrus.conf (part)
SERVICES {
# add or remove based on preferences
imap cmd=imapd listen=imap prefork=10
imaps cmd=imapd -s listen=imaps prefork=5
pop3 cmd=pop3d listen=pop3 prefork=2
pop3s cmd=pop3d -s listen=pop3s prefork=2
sieve cmd=timsieved listen=sieve prefork=10
# these are only necessary if receiving/exporting usenet via NNTP
# nntp cmd=nntpd listen=nntp prefork=3
# nntpscmd=nntpd -s listen=nntps prefork=1
# at least one LMTP is required for delivery
# lmtp cmd=lmtpd listen=localhost:lmtp prefork=1
lmtpunix cmd=lmtpd listen=/var/lib/imap/socket/lmtp
prefork=10
# this is only necessary if using notifications
# notify cmd=notifyd listen=/var/lib/imap/socket/notify
proto=udp prefork=1
}
/etc/cyrus.conf (part)
Mike
Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Cyrus on Solaris at universities?
Ian Eiloart wrote: Just wondering what other universities are runing Cyrus on Solaris? We know of: CMU UCSB Warwick Uni, Solaris 10, 60k users, 1.5TB mail spool (half in SAN, half on a 3510), Sun V890. It was set up with UFS because I felt ZFS was too new at the time and I had worries about its performance, especially with all the fsync'ing going on. I'm still not sure what I'd do on a new box. Rob -- E-Mail: [EMAIL PROTECTED] PHONE: +44 24 7652 3037 Rob McMahon, IT Services, Warwick University, Coventry, CV4 7AL, England Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Cyrus on Solaris at universities?
I do have one ZFS machine, and I don't use it to anywhere near its capabilities - it's just backups. ZFS really did raise the bar on file systems by a big jump, and it's created a new level of expectation. For ages we lived with non-journaled file systems and then when we went to journaled it was how did we ever live without this?. Still, corruption does sometimes happens, and fscking is still a pain. On top of that, there's undetected corruption that can and does occur as well. Now with ZFS having COW + checksums, the integrity level has been raised again and it's the same again, how did we ever live without this? I'm really hoping that BTRFS fills this filesystem hole that needs to be filled in linux (http://oss.oracle.com/projects/btrfs/). The good points are that Chris Mason has a lot of experience with file systems, so it seems he's been able to gather together all the main concepts and issues in his head to develop this new one, including the performance implications of decisions. He's also got a reasonably aggressive sort of timeframe for a filesystem as well (http://oss.oracle.com/projects/btrfs/dist/documentation/todo.html). The one thing that seems to be missing to me is a big test suite. I notice that ZFS advertise the fact that they have a comprehensive set of tests that includes trying to break the filesystem in many nasty ways. I think that's a great idea, and something that should be included as well. I guess time will tell. Rob Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Cyrus on Solaris at universities?
UMBC is, thought I don't speak for them anymore. On Dec 12, 2007, at 18:58, Vincent Fox wrote: Just wondering what other universities are runing Cyrus on Solaris? We know of: CMU UCSB Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: skiplist_unsafe?
I enabled it a few weeks ago before I left UMBC, and it certainly seemed to help write performance on the ZFS pool -- we started seeing the larger batched writes on those devices, instead of a constant stream of itty bity IOs caused by the constant fsync'ing. If you feel that your filesystem/buffercache will do a good job at writing things out to disk, and you've got battery-backed cache on your storage, you should be relatively well off. -rob On Nov 30, 2007, at 19:59, Vincent Fox wrote: How unsafe is setting in imapd.conf skiplist_unsafe: 1 Our /var/cyrus/imap filesystem is on a ZFS mirror set on arrays with dual controllers so OS and/or hardware corruption is remote. The application can scramble it but that can happen whether we have sync or not eh? Anything I am missing? Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: TMPFS for socket and log directories?
On Nov 29, 2007, at 21:34, Vincent Fox wrote: We had sym-linked imap/proc directory to a size-limited TMPFS a while back. Now I'm thinking to do the same for imap/socket and imap/log Any reasons against? Other candidates? You're probably not going to see wins on the socket data, since that directory is probably very well cached and doesn't get any/many writes. imap/log will only give you a win if you're doing any debugging logging for someone, which I hope you're not doing very often as it is. I think that proc is probably the biggest win, since it's quite write- happy. -rob Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: LARGE single-system Cyrus installs?
We went through a similar discussion last year in OpenAFS land, and came the same conclusion -- basically, if your filesystem is reasonably reliable (such as ZFS is), and you can trust your underlying storage not to lose transactions that are in-cache during a 'bad event', the added benefit of fsync() may be less than its performance cost. -rob On Nov 20, 2007, at 08:32, Michael R. Gettes wrote: I am wondering about the use of fsync() on journal'd file systems as described below. Shouldn't there be much less use of (or very little use) of fsync() on these types of systems? Let the journal layer due its job and not force it within cyrus? This would likely save a lot of system overhead. It makes sense to use on non-journal'd fs. I also wonder whether modern arrays even respect FULLFSYNC given their more complex nature and I/O scheduling algorithms. It may be time that fsync() (and fcntl(F_FULLFSYNC)) have become moot since there is likely little way to influence, in an effective targeted way, I/O behavior in complex environments these days. Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: LARGE single-system Cyrus installs?
On Nov 20, 2007, at 15:38, Ken Murchison wrote: Pascal Gienger wrote: Rob Banz [EMAIL PROTECTED] wrote: We went through a similar discussion last year in OpenAFS land, and came the same conclusion -- basically, if your filesystem is reasonably reliable (such as ZFS is), and you can trust your underlying storage not to lose transactions that are in-cache during a 'bad event', the added benefit of fsync() may be less than its performance cost. Would'nt it be nice to have a configuration option to completely turn off fsync() in Cyrus? If you want, with a BIG WARNING in the doc stating NOT TO USE IT unless you know what you doing. :) Its already in imapd.conf(8): skiplist_unsafe Well shiver me timbers! Ya'll rock. -rob Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: LARGE single-system Cyrus installs?
On Nov 20, 2007, at 14:57, Pascal Gienger wrote: Rob Banz [EMAIL PROTECTED] wrote: We went through a similar discussion last year in OpenAFS land, and came the same conclusion -- basically, if your filesystem is reasonably reliable (such as ZFS is), and you can trust your underlying storage not to lose transactions that are in-cache during a 'bad event', the added benefit of fsync() may be less than its performance cost. Would'nt it be nice to have a configuration option to completely turn off fsync() in Cyrus? If you want, with a BIG WARNING in the doc stating NOT TO USE IT unless you know what you doing. :) That's basically what we did with the AFS volserver fileserver. Oddly, when the patch got integrated into the OpenAFS code, they didn't like the idea of it being an option and made it the default ;) Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: LARGE single-system Cyrus installs?
This is where I think the actual user count may really influence this behavior. On our system, during heavy times, we can see writes to the mailboxes file separated by no more than 5-10 seconds. If you're constantly freezing all cyrus processes for the duration of those writes, and those writes are taking any appreciable time at all, you're going to have a stuttering server with big load averages. This shouldn't really be a problem. Yes the whole file is locked for the duration of the write, however there should be only 1 fsync per transaction, which is what would introduce any latency. The actual writes to the db file itself should be basically instant as the OS should just cache them. Still, you have a point that mailboxes.db is a global point of contention, and it is access a lot, so blocking all processes on it for a write could be an issue. Which makes me even more glad that we've split up our servers into lots of small cyrus instances, even less points of contention... Rob Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Collaboration replacement via Toltec/Bynari (was How many people to admin a Cyrus system?)
So, the problem has nothing to do with IMAP, and everything to do with message handling before delivery to the mailbox. If I've assimilated everything right, I think the summary of the problem is: Outlook handles some email messages specially (the example Joon has used is iTIP emails). To actually process these special emails, they have to pass through the Outlook transport mechanism. An email appearing in an IMAP folder is not processed by the Outlook transport mechansim. However an email POPed from a POP server to a local folder is. There's no api way to just invoke the Outlook transport system on an email, it's a black box thing that just happens on certain transports. So just mapping the INBOX to display will result in these special emails never being processed. However POPing them will. I do have a hacky workaround suggestion which might work. 1. Display all folders including INBOX 2. Setup a dummy POP account that points to the same cyrus server 3. Make sure the leave on server option is set 4. Setup a filter rule in Outlook to just delete all the POPed email Then emails arriving in the IMAP inbox will appear as normal, but they'll also be POPed regularly and processed by the Outlook transport mechansim. Unfortunately this solution wont currently work because (1) is impossible in Toltec at the moment. Rob Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Setting the location of proc independent of configdir
I didn't test but doesn't a symlink work? Yes, it does (just tried it on a development system). Definitely, we use it on all our machines. lrwxrwxrwx 1 root root 23 Oct 26 10:50 proc - /tmpfs/imapproc-slot101 Rob Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: LARGE single-system Cyrus installs?
About 30% of all I/O is to mailboxes.db, most of which is read. I haven't personally deployed a split-meta configuration, but I understand the meta files are similarly heavy I/O concentrators. That sounds odd. Given the size and hotness of mailboxes.db, and in most cases the size of mailboxes.db compared to the memory your machine has, basically the OS should end up caching the entire thing in memory. If it has to keep going back to disk to get parts of it, it suggests something is wrong with the OS caching eviction policy. I wonder if it's worth setting up a separate process that just mmaps the whole file and then uses mlock() to keep the pages in memory, that should mean it removes all read IO for mailboxes.db. Rob Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Collaboration replacement via Toltec/Bynari (was How many people to admin a Cyrus system?)
Not a bug, but a feature :) Outlook makes a clear distinction between storage and transport. In IMAP this gets a bit blurred as the INBOX is also the mechanism for receiving new mail. Using the POP3 moves the mail from the IMAP4 INBOX to the Outlook Inbox. This is handled by both Kontact and Horde. I admit I still don't get this at all, but that's because I admit I don't understand Outlook well. From a users perspective though, here is what I see. In Outlook by itself you can setup an IMAP server as a pure email store, and everything works just fine. You see the IMAP INBOX and all the IMAP folders and they work in the way you expect IMAP to, the server is authoriative and the client shows a cached view of the server. In Toltec, it almost works the same way, every other folder does exactly this, except you can't see the INBOX. I just don't understand why the INBOX has to be special if the standard Outlook IMAP integration works without it being? It's really very, very annoying feature, because it destroys much of the point of IMAP, which is that EVERYTHING is on the server, the client only presents a view of what's there. Downloading emails to your local client destroys the safety that IMAP buys you. But once the initial synchronization is done, it just updates that are done. There is no real model for skipping the initial synchronization. I thought that's probably the case. Does it poll/monitor folders regularly to see if there's been any updates from another client on the server, or does it only do it when you start Outlook? Rob Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: OT: Re: How many people to admin a Cyrus system?
On Nov 14, 2007, at 10:01, Zachariah Mully wrote: On Wed, 2007-11-14 at 13:25 +, Ian G Batten wrote: On 13 Nov 07, at 1505, David Chait wrote: One key piece of functionality that seems to be missing from every OSS solution mentioned thus far is mobile device push support (Activesync), this is not to be underestimated as it is for us, a key reason why we are ultimately being forced to adopt Exchange en-mass and abandon our current Cyrus infrastructure. There's purported to be a solution from Concilliant. http:// www.consilient.com/media/2005/c2-for-cyrus.html ian Be forewarned, last I had checked on this the licensing costs were a bit onerous IMO. The software was basically a repackaged BES with their custom client software. After trying consilient, and becoming very frustrated with it, we went with a vendor called NotifyLink for our wireless devices. Their product worked well with our crackberrys palms, integrating with Cyrus and Oracle Calendar. -rob Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: LARGE single-system Cyrus installs?
On Nov 14, 2007, at 15:20, Michael Bacon wrote: Sun doesn't make any SSDs, I don't think, but while I'm not certain, I think the RamSan line (http://www.superssd.com/products/ramsan-400/) has some sort of partnership with Sun. To be honest, I'm not sure which brand we're using, but like RamSan, it's a FC disk that slots into our SAN like any other target. I'd love to find out what your dtrace output says, though. So, is it just your mailboxes db that lies on the SSD? Or your entire 'meta' partition? Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: OT: Re: How many people to admin a Cyrus system?
On Nov 14, 2007, at 16:20, Scott M. Likens wrote: I looked into using NotifyLink with Zimbra. The cost was a bit heavy, and the only option for us would have been them hosting it. (We did not/would not have any Windows Servers/Desktops to run the software). We're using their hosted solution, which has the added benefit of them doing the end-user support for *insert stupid device here.* Your support folks tell you its worth every cent, if they've ever had to support every idiot and their idiot device. However luckily my Manager refused to run any software that did not offer a 'linux solution'. To quote his exact words No Linux, No sale That's a rather bad attitude to have. ;) Then again, I'm of the attitude where I go what, you don't support Solaris? ONLY Linux!? Ok, what about your windows solution again... -rob Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Just in case it is of general interest: ZFS mirroring was the culprit in our case
...though, we have seen super-greedyness of ZFS when resilvering. ;) On Nov 13, 2007, at 09:17, Dale Ghent wrote: Interesting. What's your kernel patch level? We're running on 125101-10 with the exact same configuration as you (mirrored to two arrays, in separate buildings even) and haven't seen this problem. /dale On Nov 13, 2007, at 1:23 AM, Pascal Gienger wrote: Our latency problems went away like a miracle when we detached one half of the mirror (so it is no more a mirror). Read-Rates are doubled (not per device, the total read rate!), latency is cut off. No more latency problems. When attaching the volume again, resilvering puts the system to a halt - reads and writes do block for seconds (!). We will go on directly with Sun to solve the problem. Their lowest I/O-priority to resilver disks does not seem to be effective. It really blocks the kernel and you end up with thousand locks in zfs_zget. We have two SAN volumes in different buildings which are NOT the bottleneck, tests show it. Pascal Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html -- Dale Ghent Specialist, Storage and UNIX Systems UMBC - Office of Information Technology ECS 201 - x51705 Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Timed Actions in Sieve
I had looked into this before, but really haven't had a chance to follow up on it. There is an draft for a sieve date/time extension, which allows you to use time comparisons in conditionals -- for example, the vacation message defined by a date range. This looks like its the current version of the draft: http://tools.ietf.org/html/draft-freed-sieve-date-index-07 On Nov 13, 2007, at 06:24, Ian G Batten wrote: We've been having a chat about how useful it would be to have timed actions in sieve: so that a vacation message could be set up for a duration which would automatically revert, so that a forwarding could be set up for the duration of a short-term project, etc, etc. The naive way is to add support to the sieve interface of choice (the squirrelmail plugin in our case) to handle deferred actions, but I can think of all sorts of security problems with that. Another would be a means to auto-generate regexps to match on Date: headers, but that's really tacky. The full solution would be to have the current time available in sieve scripts, to then match on. Has anyone else thought about this area? ian Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Squatter causing load spike
On Nov 13, 2007, at 16:47, Gerard wrote: I am running squatter at 3am as a cron job on all of my servers. Over the passed week I have one server where squatter spikes the load and ends up locking up the server at around 8am every morning. Yeah, it seems to take that long to run which may be an issue in itself. Has anyone come accross this or have a suggestion on how to get squatter to perform better? What OS? Squatter shouldn't be causing load spikes unless there's some resource contention for lock files, etc. You should also make sure that if you're running cyr_expire, you're NOT running it concurrently with squatter. Bad things happen.* To get around this, I created a nightly shell script which runs cyr_expire, then squatter, and kick that off as a scheduled task in my cyrus.conf. * The bad things come into play if expire squatter are processing the same mailbox at the same time. Squatter will probably just stop, and not tell you why. -rob Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Squatter causing load spike
It seems like the default for cyr_expire runs at 4:00AM (delprune cmd=cyr_expire -E 3 at=0400) and I start squatter at 3:00AM. Do you think that this would cause the spike and server to lock up? We are running RHEL4U4 They both compete for a lot of resources... You probably shouldn't run them both at the same time. My cyrii run on Solaris, so I'm not familiar with what platform specific issues you may be having. You might try running just squatter one night without expire and see how it does. Also, make sure you're running squatter with the option (think it's - s) that makes it only reindex mailboxes that have changed since the last squatter run. -rob Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
OT: Collaboration replacement via Toltec/Bynari (was How many people to admin a Cyrus system?)
So you will have a choice of 3 commercial Outlook plug-ins, What are the 3 commercial Outlook plugins? Obviously the Toltec one, but which others? I've actually always liked the idea of what toltec + bynari were doing. It's basically using the IMAP server as a database where folders = tables, emails = records. The nice things about using an IMAP server is that: 1) you're reusing a protocol, rather than creating another one 2) emails can't be altered, so an update is a delete + add which makes detecting changes easy 3) the server is always authoriative, so syncing tends to be easier Of course reusing an email protocol to do these other things it was never meant to do seems hacky as well. Still, is it really any more hacky than using a blatant file protocol (despite calling files resources and directories collections) than CalDAV? As an FYI, last time I tried Bynari and Toltec, my general conclusions were: Bynari - Annoying they use a binary format for storing the information, but good that they use the 0-9 ACLs to designate what folders are calendar/contacts/etc folders, means you don't need the damn ANNOTATEMORE extension and the patches to make it work with arbitrary values. Seemed a bit flaky. It creates some process that runs next to Outlook, and sometimes that process wouldn't quit when you quit Outlook leaving things in an odd state. Install/removing the software seemed to be prone to failures/screwing up your config. Toltec - Seemed less prone to bugginess than bynari, and using an XML format seemed more open so we could read it ourselves for other features in the future. Using ANNOTATEMORE was a pain. There was a showstopper problem however. When I tested it seemed that Toltec would display all sub-folders of INBOX, and using annotations it would set the Calendar, Contacts, etc folders as appropriate to display those items, and all other folders to display emails, but it would not show the INBOX! At the time I asked support about this it was suggested that you should setup a POP account to download the emails from the inbox to the Local Folders PST file. This seemed to completely defeat the purpose of IMAP to me. Has that been fixed? Also out of interest, how much caching does toltec do? Basically it seems to me to get any decent performance you effectively have to download all the calendar/task/etc items from the appropriate folders and cache them all inside outlook. Any attempt to get from the server on a when needed basis seems like it would never perform well? Rob Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: LARGE single-system Cyrus installs?
However observationally we find that under high email usage that above 10K users on a Cyrus instance things get really bad. Like last week we had a T2000 at about 10,500 users and loads of 5+ and it was bogging down. We moved 1K users off bringing it down to 9,500 and loads dropped to around 1.0 and everything is fine. With 10k users, how much disk space and how many peak simultaneous imap connections does 10k users represent on your system? It's just that I imagine the number of actual users on a system isn't the pure bottleneck per-se, it's really some relationship between number-of-users (more users means more email deliveries) and number-of-users*percent-that-are-connected (more users that connect more often means more imapd processes). Rob Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: squatter running longer than 24 hours
squatter would really benefit from incremental updates. At the moment a single new message in a mailbox containing 20k messages causes it to read in all the existing messages in order to regenerate the index. We spoke to Ken about this ages back, and even offered to pay for the work to make it happen, but it was just around the time CMU hired him, so it never actually happened pity. It would be nice to be able to dedicate a couple of weeks to rummage around in there and actually make it happen... Rob Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: UC Davis Cyrus Incident September 2007
Thanks for your post, it's always interesting to hear other peoples stories. 1st STEP: Perdition mail-proxies in a load-balanced pool and 2 can handle the load most days. Initially we If you have a chance, definitely think about changing from perdition to nginx. There's slightly more work in that I think you'll have to write a custom daemon to do LDAP - nginx http auth protocol mappings, but if you've got any perl experience that should be like a few dozen lines with Net::Server and Net::LDAP. With large numbers of IMAP connections, the improvements were considerable. http://blog.fastmail.fm/2007/01/04/webimappop-frontend-proxies-changed-to-nginx/ for this in both the OS and the application. There is some bottleneck for a resource that once it reaches a certain busy-ness level, everything starts on our hardware such as Zones. The theory being 2 or 3 Zones on a T2000 with say 10K users each, would still let us accomodate the same number of users on the new hardware that we had originally targetted. The root problem seems to be an interaction between Solaris' concept of global memory consistency and the fact that Cyrus spawns many processes that all memory map (mmap) the same file. Whenever any process updates any part of a memory mapped file, Solaris freezes all of the processes that have that file mmaped, updates their memory tables, and then Ick, that will be nasty. This is going to particularly be a problem with a skiplist mailboxes.db. Basically every imapd will have an mmaped copy of the entire mailboxes.db in it's process map. Linux does not seem to have a problem with this at all (at a time in the past we had a 150M mailboxes.db file with 6000+ imapd processes and it worked fine). Clearly there's some low-level OS design issue that means that it's not as easy to share mmaped pages between processes in Solaris compared to Linux. One option you have is that rather than creating separate Zones in the OS, you just create separate cyrus instances yourself. We do this at FastMail. Basically we've partitions all our storage into 300G units, and each partition has a completely separate config dir, cyrus.conf, imapd.conf, mailboxes.db, partition, etc and runs a separate cyrus master process. It required some automation (eg we have scripts that auto-generate the appropriate cyrus.conf, imapd.conf and init scripts), and it means every cyrus command run needs the -C parameter to tell which instance you're dealing with, but it means that each instance of cyrus is much smaller (eg much smaller mailboxes.db file, deliver.db file, etc) and there's less global contention points. Actually the reason for breaking up into smaller units was for replication reasons, but despite that, I think it's been a good move in many respects. Of course if you have global shared folders or want a a murder environment, then that probably just makes things harder and each instance needs a complete mailboxes.db file I believe anyway, but we've decided that we don't need the one main thing murder offers which is global shared folders. Rob Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: LARGE single-system Cyrus installs?
Yesterday I checked my own Cyrus servers to see if I was running out of lowmem, and it sure looked like it. Lowmem had only a couple MB free, and I had 2GB of free memory that was not being used for cache. I checked again today and everything seems to be fine - 150MB of lowmem free and almost no free memory (3GB cached)! Grrr. I'm guessing it can depend on recent operations. From my understanding, the issue is that inode structures can only be in lowmem, but actual cache pages can be anywhere, but you can only have cached file pages if you also have the corresponding inode structure. So if you have lots of small files (eg like cyrus), then you can run out of lowmem for inode structures before you run out of total memory for cached pages. However if your access pattern changes, and you access fewer but larger files, then lowmem might be big enough to hold enough inodes to use up the reset of memory for cache pages. For instance if you've just run cyr_expire, it would have read a lot cyrus.cache files, which are a lot bigger than your average email, so in that case you might end up with all memory used by cache but with lowmem still free. One thing we've found very useful is munin, which has allowed us to track patterns and long term changes over time. The graphs in the blog post are directly from munin. If you can get it setup on your machine, I recommend it as a nice visual way of getting to know your machines over time. http://blog.fastmail.fm/2007/09/21/reiserfs-bugs-32-bit-vs-64-bit-kernels-cache-vs-inode-memory/ In the above case, if my hypothesis were true, you'd be able to see it easily on a munin graph by comparing the memory graph with the open_inodes graph and your known cyr_expire run times to see if there is a relationship. Anyways, I was looking into building a 64-bit kernel. I'm running Debian Sarge (I know, old) on a Dell 2850 with Intel Xeon (Nocona) CPUs and 4GB RAM. My kernel version is 2.6.14.5, built from kernel.org sources. It has High Memory Support (64GB) selected. When I run menuconfig, I'm not seeing any obvious place to switch from 32-bit to 64-bit. Could you elaborate a bit about how you switched to a 64-bit kernel? Also, are you running a full 64-bit distro, or just a 64-bit kernel? Bron did all the OS changes. We're running a 64bit kernel on a 32bit system, it was a lot easier than upgrading the entire system. I believe debian makes this pretty easy to do (we use etch though, not sarge). I've added him to the cc list so he can reply with the details. Rob Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Date problem after migration
There's no really easy way, but this patch might help. Basically it parses the last Received: header to set the internal date. http://cyrus.brong.fastmail.fm/#cyrus-receivedtime-2.3.8.diff The problem is if you apply it, you then have to reconstruct all your mailboxes to rebuild the cyrus.index with the new values. Rob - Original Message - From: Guillaume Postaire [EMAIL PROTECTED] To: info-cyrus@lists.andrew.cmu.edu Sent: Tuesday, October 09, 2007 10:23 PM Subject: Date problem after migration Hi all, We just have done a migration from a very old Cyrus stand alone installation to a new one with murder. During the migration everything went ok and no user complaint, so we destroy our old mailbox. Shortly after that one of our user notice a huge problem with outlook that we don't reproduce with thunderbird (put here whatever imap compliant client). All the mail older than the migration have the date of the migration in outlook. It seems that outlook don't use the header date (wich are ok in the plain text storage) but another information that come from cyrus. We try to analyse what happen and discover we forget to use --syncinternaldates with imapsync. We simulate some migration and this problem don't exist if we had this. How could we correct the date stored in cyrus ? Here is the command we use for migration imapsync --ssl1 --ssl2 --host1 #HOST1# --host2 #HOST2# \ --authuser1 cyrus --password1 #PASSWORD1# --authuser2 cyrus \ --password2 #PASSWORD2# --authmech1 PLAIN --authmech2 PLAIN \ --subscribe --user1 #USER# --user2 #USER# \ --delete2 --expunge --expunge2 Cheers. -- Guillaume Postaire - KARAVEL - Responsable infrastructure [EMAIL PROTECTED] - http://www.promovacances.com - http://www.karavel.com https://www.linkedin.com/in/gpostaire Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: LARGE single-system Cyrus installs?
I suppose that 8 SATA disks for the data and four 15k SAS disks for the metadata would be a good mix. Yes. As I mentioned, our iostat data shows that meta-data is MUCH hotter than email spool data. --- Checking iostat, a rough estimate shows meta data get 2 x the rkB/s and 3 x the wkB/s vs the email spool, even though it's 1/20th the data size and we have the status cache patch! Basically the meta data is very hot, so optimising access to it is important. --- Splitting was definitely a big win for us. Rob Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: LARGE single-system Cyrus installs?
A data point regarding reiserfs/ext3: We are in the process of moving from reiserfs to ext3 (with dir_index). ext3 seems to do substantially better than reiserfs for us, especially for read heavy loads (squatter runs at least twice as fast as it used do). Are you comparing an old reiserfs partition with a new ext3 one where you've just copied the email over to? If so, that's not a fair comparison. What we found was that when we first copied the data from reiserfs - ext3, everything seemed nice, but that's only because when you copy mailboxes over, you're effectively defragmenting all the files and directories in your filesystem. We've actually been thinking of setting up a regular process to just randomly move users around, because the act of moving them effectively defragments them regardless of the filesystem you're using! Give it a month or two of active use though (delivering new emails, deleting old ones, etc), and everything starts getting fragmented again. Then ext3 really started going to crap on us. Machines that had been absolutely fine under reiserfs, the load just blew out to unuseable under ext3. data=ordered in both cases. data=journal didn't seem to make any difference with ext3. data=journal with reiserfs caused amusing kernel memory leaks, which it looks like Fastmail also hit recently. An dedicated journal device would probably make a big difference with data=journal. Talking with Chris Mason about this, data=journal is faster in certain scenarios with lots of small files + fsyncs from different processes, exactly the type of workload cyrus generates! As it turns out, the memory leaks weren't critical, because the the pages do seem to be reclaimed when needed, though it was annoying not knowing exactly how much memory was really free/used. The biggest problem was that with cyrus you have millions of small files, and with a 32bit linux kernel the inode cache has to be in low memory, severely limiting how many files the OS will cache. See this blog post for the gory details, and why a 64-bit kernel was a nice win for us. http://blog.fastmail.fm/2007/09/21/reiserfs-bugs-32-bit-vs-64-bit-kernels-cache-vs-inode-memory/ Rob Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: LARGE single-system Cyrus installs?
The iostat and sar data disagrees with it being an I/O issue. 16 gigs of RAM with about 4-6 of it being used for Cyrus leaves plenty for ZFS caching. Our hardware seemed more than adequate to anyone we described it to. Yes beyond that it's anyone guess. If it wasn't IO limit related, and it wasn't CPU limit related, then there must be some other single resource that things were contending for. My only guess then is it's some global kernel lock or the like. When the load skyrocketed, it must have shown that lots of processes were not in S (sleep) state. Were they in R (run) state or D (io wait) state? Since you're on Solaris, you could use DTrace to find out what they were actually doing/waiting for... Rob Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: LARGE single-system Cyrus installs?
I think what truly scares me about reiser is those rather regular posts to various mailing lists I'm on saying my reiser fs went poof and lost all my data, what should I do? I've commented on this before. I believe it's absolutely hardware related rather than reiserfs related. http://www.mail-archive.com/info-cyrus@lists.andrew.cmu.edu/msg30656.html If you use hardware that's broken, you should expect to get burnt. Rob Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: LARGE single-system Cyrus installs?
Are you comparing an old reiserfs partition with a new ext3 one where you've just copied the email over to? If so, that's not a fair comparison. No, a newly created partitions in both cases. Fragmented partitions are slower still of course. That's strange. What mount options are/were you using? We use/used: reiserfs - rw,noatime,nodiratime,notail,data=journal ext3 - noatime,nodiratime,data=journal If you weren't using notail on reiserfs, that would definitely have a performance impact. Okay, I think that we had a different kernel memory bug. We were running out of memory after 24 hours, and a 20 line test program could exhaust memory in seconds. This bug was in SLES four years back, and it was still there the last time that I looked (some months back now). Wow weird, must be something different. What kernel was it? Do you know where the memory leak was occuring? We never encountered anything that would exhaust memory like that. We did encounter a bug which caused a deadlock inversion problem and would causes processes to get stuck in D state. It appears the code path that causes this has been totally removed sometime between 2.6.20 and 2.6.22, so I think we can finally drop that 1 line patch we've had to apply for the last couple of years... Rob Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: LARGE single-system Cyrus installs?
Anyhow, just wondering if we the lone rangers on this particular edge of the envelope. We alleviated the problem short-term by recycling some V240 class systems with arrays into Cyrus boxes with about 3,500 users each, and brought our 2 big Cyrus units down to 13K-14K users each which seems to work okay. FastMail has many 100,000's of users in a full replicated setup spread across 10 backend servers (+ separate MX/Spam/Web/Frontend servers). We use IBM servers with some off the shelf SATA-to-SCSI RAID DAS (eg like http://www.areasys.com/area.aspx?m=PSS-6120). Hardware will die at some stage, that's what replication is for. Over the years we've tuned a number of things to get the best possible performance. The biggest things we found: 1. Using the status cache was a big win for us I did some analysis at one stage, and found that most IMAP clients issue STATUS calls to every mailbox a user has on a regular basis (every 5 minutes or so on average, but users can usually change it) so they can update the unread count on every mailbox. The default status code has to iterate over the entire cyrus.index file to get the unread count. Although the cyrus.index file is the smallest file, with 10,000's of users connected with clients doing this regularly for every folder, it basically means you either have to have enough memory to keep every cyrus.index hot in memory, or every 5-15 minutes you'll be forcing a re-read of gigabytes of data from disk, or you need a better way. The better way was to have a status cache. http://cyrus.brong.fastmail.fm/#cyrus-statuscache-2.3.8.diff This helped reduce meta data IO a lot for us. 2. Split your email data + metadata IO With the 12 drive SATA-to-SCSI arrays, we get 4 x 150G 10k RPM WD Raptor drives + 8 x (largest you can get) drives. We then build 2 x 2 drive RAID1 + 2 x 4 drive RAID5 arrays. We use the RAID1 arrays for the meta data (cyrus.* except squatter) and the RAID5 arrays for the email data. We find the email to meta ratio about 20-to-1, higher if you have squatter files, so 150G of meta will support up to 3T of email data fine. From our iostat data, this seems to be a nice balance. Checking iostat, a rough estimate shows meta data get 2 x the rkB/s and 3 x the wkB/s vs the email spool, even though it's 1/20th the data size and we have the status cache patch! Basically the meta data is very hot, so optimising access to it is important. 3. Not really related to cyrus, but we switched from perdition to nginx as a frontend POP/IMAP proxy a while back. If you've got lots of IMAP connections, it's really a worthwhile improvement. http://blog.fastmail.fm/2007/01/04/webimappop-frontend-proxies-changed-to-nginx/ 4. Lots of other little things a) putting the proc dir on tmpfs is a good idea b) make sure you have the right filesystem (on linux, reiserfs is much better than ext3 even with ext3s dir hashing) and journaling modes That is our hypothesis right now, that the application has certain limits and if you go beyond a certain number of very active users on a single backend bad things happen. Every application has that problem at some point. Consider something that uses CPU only, and every new unit of work takes the CPU 0.1 seconds, then you can handle 1-10 units of work arriving per second no problem. If 11 units per second arrive, then after 1 second you'll have done 10, but have 1 unit still to do, but another 11 units arrive in that next second again. Basically your outstanding work queue will grow forever in theory. cyrus isn't CPU limited by a long shot, but it can easily become IO limited. This same effect happens with IO, it's just more noticeable because disks are slow. Basically if you start issuing IO requests to a disk system and it can't keep up, the IO queue grows quickly and the system starts crawling. The only way to improve it is reduce your IOPs (eg less users or optimise the application to issue less IOPs in some way) or increase the IOPs your disk system can handle (eg more spindles, faster spindles, NVRAM, etc). That's what 1 (reduce IOPs application generates) 2 (put hot data on faster spindles) above are both about, rather than the other option (reduce users per server). Rob Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Apply Sieve script to messages already in INBOX?
Your other option, of course, is to implement this in the client. I had thought of doing this using the sieve-test utility provided with cyrus, which lets you specify a message + sieve script and prints to stdout the actions that would have been performed. The only issue is on large folders having to retrieve lots of messages and run the utility lots of times. I just thought it would be easier to get the server to do it and just pass back the actions that would be performed. Rob Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Apply Sieve script to messages already in INBOX?
A kludgy solution... what about fetchmail to pull the INBOX contents and resubmit the messages to deliver? Be careful of duplicate delivery suppression! I had a very brief talk with Ken about this ages back, about creating a non-standard extension to IMAP that would allow you to run a sieve script against a bunch of messages in a mailbox, and either perform the action, or return an action response string back for each message. My thinking is I'd love to add to our web interface a way to say either run sieve on selected messages or show actions on selected messages. For the second, it would basically group by the list of returned actions, and let you use a checkbox to decide which ones to actually do. More complex, but more powerful to. In theory it shouldn't be *too* hard, since the sieve bytecode interpreter allows you to specify exactly what callback you want to run for each sieve action, so you could embed it in imapd and setup the appropriate callbacks to do the actions or return the appropriate string responses. Still, it's at least a couple of weeks of programming work, and since it's completely non-standard, I doubt it would make it back into the CVS code base. Rob Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Cyrus IMAP 2.3.9 on Solaris 10 with ZFS and SAN
http://cyrus.brong.fastmail.fm/patches/cyrus-statuscache-2.3.8.diff This cuts back on meta IO traffic considerably for repeated SELECT calls on mailboxes which are unchanged. We are in a similar boat, and it makes a huge difference (also for some other clients) That should be cuts back on meta IO traffic considerably for repeated STATUS calls on mailboxes which are unchanged, not SELECT calls. That's ok though, because the reality is that most clients do many, many more STATUS calls than SELECT calls. Basically every client that has a updated unread count on mailboxes every 5 minutes is calling STATUS on every mailbox every 5 minutes. If you're getting the unseen count, that means stepping through the cyrus.index file of every mailbox every 5 minutes, which if you've got lots of users, with large mailboxes, each with an IMAP client doing a status every 5 minutes on every mailbox, means you could be forcing re-reads of many gigabytes of data every few minutes. If your cache memory isn't big enough, your IO will jump enormously. Basically this patch significantly reduced the amount of read IO from our meta partitions. of passing FDs around between processes. I don't know how it works precisely, but it uses Socket::Pool which had a sourceforge project and doesn't appear to any more. My guess is that I'd rather not look at the code for fear of going totally insane. You can pass file descriptors over UNIX domain sockets, see SCM_RIGHTS. Here is some code Google found - http://search.cpan.org/src/SAMPO/Socket-PassAccessRights-0.03/passfd.c It does use this underneath. It's not that magic (eg fork() automatically copies file descriptors from one process to another, why shouldn't there be an explicit way to do that between non-parent/child processes), though it's not that commonly used either, and the usage api is a bit weird (strange messages via unix sockets). I haven't seen many programs that use it since you have to be quite careful with it (eg you can't read any data from a socket before you read the file descriptor or it breaks), and there have been bugs with many different OS's in the past (even linux 2.4 had a very intermittent bug for a long time in our testing, which is why we had a try this 3 times loop on a bunch of code). In our case it is very useful, since our imappool daemon keeps track of the server + user + currently selected folder, we basically call the imappool daemon to get a socket for a server/user combination, and it returns us one (connects to the server if needed), and tells us the state (eg connected but not logged in, logged in, logged in and which folder is selected). It then locks that socket in it's pool so nothing else can use it (eg another web process). We then do whatever we need with the socket, and when done, we tell the imappool to unlock the socket, and we also tell it the new state it's in (eg which users/folder). That way, the imappool always has the socket (we don't need to pass it back), but ensures that only one process can use it at a time. The main issue is that the code has to be changed to explicitly contact the imappool to get a connection and free it when done, it's not transparent. In our installation it is not possible to log on as cyrus or other admin users from outside, I patched the imapd to check the caller's ip address - and the firewall does the rest so that no forged packet (with an inside sender ip coming from outside...) will trigger it. We use nginx as a frontend proxy. For every IMAP/POP login, it calls out to a process that allows you to authenticate and/or authorise the login (you can return any error message you want, and it gives you information like login name, password and ip address - http://wiki.codemongers.com/NginxMailCoreModule), and what backend server + port to connect to. It uses a small process pool + epoll to easily handle 10,000's of connections. http://blog.fastmail.fm/2007/01/04/webimappop-frontend-proxies-changed-to-nginx/ Rob Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Too many open files
Rob McMahon wrote: Aug 24 16:20:44 narcissus imap[24338]: [ID 240394 local6.error] IOERROR: opening quota file /var/imap/quota/V/user.??.Trash: Too many open files Aug 24 16:21:38 narcissus imap[24338]: [ID 816447 local6.error] warning: cannot open /etc/hosts.deny: Too many open files and indeed this process has hit the default 256 limit, most of which seem to be quota files. Other issues were manifesting, IMAP commands never completing for instance, so I had to kill this process, which in turn caused the master process to restart. Not good. Has anyone got a handle on this bug ? It doesn't seem like a slow descriptor leak: I've just deleted 250 users in chunks of 20, checking carefully on fd usage between each batch, and then on the final 20, blatt, it did it again. It feels like it's some sort of house-keeping function kicking in. I can go around hunting for, and deleting, quota roots. I can up maxfds to try to alleviate the problem. But, I've got no confidence, I've got 4,000 users to delete, and I want to completely automate it, and rely on the service staying up once the deed is done. Rob -- E-Mail: [EMAIL PROTECTED] PHONE: +44 24 7652 3037 Rob McMahon, IT Services, Warwick University, Coventry, CV4 7AL, England Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Too many open files
Cyrus-2.2.13 (yes, I know it's getting on a bit). Solaris 10 SPARC,
around 115,000 mailboxes.
I was mopping up a few users today with a script which basically does:
while () {
chomp;
my $mb = 'user.' . $_;
$client-setacl($mb, $user = 'c');
$client-delete($mb);
}
After deleting just a few, I start getting errors like Must login
first, System I/O Error (sorry, I haven't got the exact messages to
cut paste). Looking in the logs I see:
Aug 24 16:20:44 narcissus imap[24338]: [ID 240394 local6.error] IOERROR:
opening quota file /var/imap/quota/V/user.??.Trash: Too many open files
Aug 24 16:21:38 narcissus imap[24338]: [ID 816447 local6.error] warning:
cannot open /etc/hosts.deny: Too many open files
and indeed this process has hit the default 256 limit, most of which
seem to be quota files. Other issues were manifesting, IMAP commands
never completing for instance, so I had to kill this process, which in
turn caused the master process to restart. Not good.
Any suggestions on this ? I could up maxfds to, say, 1024 for imap, but
then would it just hit that limit instead. No other IMAP process seems
to use more than around 16 descriptors.
Cheers,
Rob
--
E-Mail: [EMAIL PROTECTED] PHONE: +44 24 7652 3037
Rob McMahon, IT Services, Warwick University, Coventry, CV4 7AL, England
Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Replication to more than one replica?
Could you do it then with sync_client -S -u instead of -r to one of the relicas? -u User mode. Remaining arguments are list of users who should be replicated. Which user though? You could do all users, but that takes time. The point is that cyrus programs (lmtpd, imapd, etc) log actions that change mailboxes, so you know exactly which mailboxes need syncing. It's those logs you have to play twice to apply to two replicas to keep replication as close as possible to real time. Rob Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Replication to more than one replica?
It would be a way to keep a second offline replica for backing up to a tape archive, which is what I plan to do. I realize it's slower than the standard rolling replication but the archive is mainly for ooops I deleted that mail kind of scenario. In fact, I'm testing the setup right know with a list of all users. First run will take a while but further ones should only take a couple of minutes. Of course this setup is a workaround and it gets increasingly more difficult with the number of users, here we only have around 500 although many mailboxes are quite large. That's all fine. Just make sure you only have one sync_client in rolling replication mode pulling log files from a particular dir at a time. Rob Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Replication to more than one replica?
Disagree. I am replicating one server to two and have been doing it for quite a while with cyrus 2.3. Can you explain how you're doing this? If you're just running multiple sync_clients with different config files that point to different replica servers, then what you've got is broken because each sync_client will be randomly grabbing different log files, so each replica will be only getting half the required updates and will at any moment be randomly out of sync with the master. The only way you can reliably make this work is. 1. Write a process that that like sync_client, regularly grabs and atomically rotates the log files in the conf/log directory 2. That process then either: a) puts two copies of the log file in separate directories, and you have two separate sync_client processes running in rolling mode picking up the files in those directories, each sync_client with a different config file pointing to the two different replica servers b) it runs a sync_client -f on the file twice, but with separate config filess to each of the two different replica servers Rob Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: cyrus replication validation
Hi If I understand this patch correctly, it doesn't solve the larger problem that I'm interested in: is the data on my replica the same as the data on my primary, or more to the point, are the two data sets converging? ... But I'm really interested in something that can run out of band from csync, imap, etc, that examines files on the primary and replica to know what the variance As mentioned, there's two parts to the patch. The UUID part which helps with the replication, but there's also this bit. 2. You can fetch a computed MD5 of any message on disk via IMAP Using the second, you can do complete validation via IMAP, just iterate through all folders and all messages, get the computed MD5 and compare on both sides. We wanted the same thing you did, some way to guarantee that the message data on both sides was exactly the same. One way of doing that was to use something that runs under the covers to check the messages on disk, which is fine. The other was to basically add something to the IMAP protocol which lets us do the same thing via IMAP. We went the second, because we already had code that given a username, would check their master server and replica server to see that 1. The folder list matched 2. For each folder, message count + unread count + uidvalidity + uidnext matched (eg status results) 3. For each folder, the UID listing matched 4. For each folder, the flags on each UID message matched These were all easy to get via IMAP on both sides and compare. However they were all meta-data related, and didn't help check that the actual email spool data on disk was correct. Which is why we added two FETCH items to the imap protocol with the above patch. FILE.MD5 and FILE.SIZE With these, we can now compare each file on each side of the master/replica set to see that they match. This means we can now check pretty much all meta data + spool data on both sides for consistency, all via IMAP connections, without having having to do any more peeking under the hood. Of course actually having the patch in there is pretty heavily peeking under the hood, but it was easier for us to do that because we already had a script which did steps 1-4, so adding a hack to the IMAP protocol was easier for us than creating a whole new system. Whether this is easier/harder at your site is up to you. Rob Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: RAID type suggestion
Just some thoughts 1. Simon is right about RAID-5 being unreliable. Either use RAID-6 or replication to a completely separate system to ensure you have enough redundancy 2. Look at having separate RAID-1 for OS AND cyrus meta data, and RAID-5+replication or RAID-6 for the data spool From our testing, if you separate out the cyrus meta data (eg mailboxes.db, quota dbs, cyrus.* files, etc), it gets more IO than the email spool data, but takes only 1/20th the storage, so having high speed (eg SAS, 10k RPM drives, etc) for it is definitely worthwhile. Rob Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Recommend how to move 31GB of mail to a new server
Would it be possible to use replication for this? Set up a replica of the first server, copy while both servers online. Then take first server offline and change IPs/servernames/whatever. Yes, it's definitely possible, but potentially quite a bit of work. We actually have some modules that use replication to allow you to move one user at a time between arbitrary servers. Basically it: 1. Creates a new temporary cyrus config dir with db,quota,sieve,user sub-dirs 2. Generates on the fly a new imapd.conf file with the appropriate sync_* params 3. Creates a new mailboxes.db with just the user directories (it hacks the partition to be the same as the destination one wanted, but makes the dirs in the temp imapd.conf for that partition point to the old place) 4. symlinks the users seen,sub,quota,sieve files to the new tmp config dir 5. Runs a sync_client -u to copy the user This basically allow arbitary moving of any single user from any server to any other server on any partition. The nice thing is this can happily run while the user is logged in and doing stuff. Once it's run once, we then disable logins and email deliveries for the user and disconnect them from any imap/pop session, run the sync again (generally very quick), update the db with which server they're on, then re-enable them again. It's quite neat. Unfortunately the code isn't really a releaseable state, there's too many FM only assumptions and details in it. Rob Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: cyrus 2.3.8, internal dates not stored in message file time ?
So, as I understand it, when reconstructing mailboxes, internal dates will be lost. It this intentional, or did I miss something in the docs/manual ? This is why we created this patch. http://cyrus.brong.fastmail.fm/#cyrus-receivedtime-2.3.8 Rob Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Cyrus with a NFS storage. random DBERROR
I suspect that the problem is with mailbox renames, which are not atomic and can take some time to complete with very large mailboxes. I think there's some other issues as well. For instance we still see skiplist seen state databases get corrupted every now and then. It seems certain corruption can result in the skiplist code calling abort() which terminates the sync_server, and causes the sync_client to bail out. I had a back trace on one of them the other day, but the stack frames were all wrong so it didn't seem that useful. HERMES_FAST_RENAME: Translates mailbox rename into filesystem rename() where possible. Useful because sync_client chdir()s into the working directory. Would be less useful in 2.3 with split metadata. It would still be nice to do this to make renames faster anyway. If you did. 1. Add new mailboxes to mailboxes.db 2. Filesystem rename 3. Remove old mailboxes You end up with a race condition, but it's far shorter than the mess you can end up with at the moment if a restart occurs during a rename. Together with my version of delayed expunge this pretty much guarantees that things aren't moving around under sync_client's feet. Its been an awful long time (about a year?) since I last had a sync_client bail out. We are moving to 2.3 over the summer (initially using my own original replication code), so this is something that I would like to sort out. Any suggestions? I can try and keep an eye on bailouts some more, and see if I can get some more details. It would be nice if there was some more logging about why the bail out code path was actually called! Rob Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Cyrus with a NFS storage. random DBERROR
does the IMAP spec specify how large a UUID can be? UUIDs aren't part of the IMAP spec. It's an addition to cyrus to help replication. By default, there's no way to access UUIDs via IMAP at all since they're not part of the IMAP spec. The UUID size chosen was done by David Carter when he implemented replication. It would be possible to change them again, but it would mean changing the cyrus.index file format and upgrading all cyrus.index files. Seemed easier not to do that. Rob Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Cyrus with a NFS storage. random DBERROR
I don't have something to consume make_md5 data, yet, either. My plan is to note the difference between the replica and the primary. On a subsequent run, if those differences aren't gone, then they would be included in a report. Rather than make_md5, check the MD5 UUIDs patch below. Using this, we have a script that regularly checks both sides of a master/replica pair to check everything is consistent between the UUID and the computed MD5. It was this that let us discover the rare didn't unlink old files bug reported about 3 months back. --- http://cyrus.brong.fastmail.fm/ One problem we've had is the inability to easily check that the files on disk correspond to what was originally delivered to check for cyrus data corruption after either a disk problem or some other bug has caused us to be unsure of our data integrity. I wanted to calculate a digest and store it somewhere in the index file, but messing with the file format and fixing sync to still work, etc... it all sounded too painful. So - added is a new option uuidmode in imapd.conf. Set it to md5 and you will get UUIDs of the form: 02(first 11 bytes of the MD5 value for the message) which takes up the same space, but allows pretty good integrity checking. Is it safe? - we calulated that with one billion messages you have a one in 1 billion chance of a birthday collision (two random messages with the same UUID). They then have to get in the same MAILBOXES collection to sync_client to affect each other anyway. The namespace available for generated UUIDs is much smaller than this, since they have no collision risk - but if you had that many delivering you would hit the limits and start getting blank UUIDs anyway. Mitigating even the above risk: you could alter sync_client to not use UUID for copying. It's not like it's been working anyway (see our other UUID related patch). As an integrity check it's much more useful. The attached patch adds the md5 method, a random method which I've never tested and is almost certainly bogus, but is there for educational value[tm], the following FETCH responses in imapd: FETCH UUID = 24 character hex string (02 + first 11 bytes of MD5) FETCH RFC822.MD5 = 32 character hex string (16 bytes of MD5) FETCH RFC822.FILESIZE = size of actual file on disk (via stat or mmap) Totally non-standard of course, but way useful for our replication checking scripts. Embrace and extend 'r' us. Anyone feel like writing an RFC for fetching the digest of a message via IMAP? If the server calculated it on delivery and cached it then you'd have a great way to clean up after a UIDVALIDITY change or other destabilising event without having to fetch every message again. --- Rob Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Cyrus with a NFS storage. random DBERROR
I run it directly, outside of master. That way when it crashes, it can be easily restarted. I have a script that checks that it's running, that the log file isn't too big, and that there are no log- PID files that are too old. If anything like that happens, it pages someone. Ditto, we do almost exactly the same thing. Also if we switch master/replica roles, our code looks for any incomplete log files after stopping the master, and runs those first to ensure that replication was completely up to date. It seems anyone seriously using replication has to unfortunately do these things manually at the moment. Replication just isn't reliable enough, we see sync_client bail out quite regularly, and there's not enough logging to exactly pinpoint why each time. I think there's certain race conditions that still need ironing out, because rerunning sync_client on the same log file that caused a bail out usually succeeds the second time. It would be nice if some code was actually made part of the core cyrus distribution to make this all work properly, including switching master/replica roles. Rob Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
