Re: single instance store and replication
I'm using Cyrus replication. After several tests, it seems that the single instance store facility is not replicated. I mean, the same message sent to several recipient is stored once on the master, but stored several times on the slave. Is there a special thing to do to activate single instance store replication or it just doesn't exist yet? Message UUIDs are used to replicate the single instance store (see docs/text/install-replication). This won't have much effect when you first replicate a mailstore as sync_server in 2.3 only tracks the last few thousand messages that have been uploaded. It becomes much more effective when a replica has been seeded and you switch to rolling replication. Do i have to put the provide_uuid in imap lmtp services both in master and replica cyrus.conf or only in the master cyrus.conf ? Does the sync_machineid has to be set in the imapd.conf of the replica ? When you set up a replica for the first time with a master already in service, do u have to first manually synchronize the mailboxes and only after set the rolling replication ? Is there a way to reinitialize rolling replication on the master ? Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Slow lmtpd
On Thu, 2007-03-01 at 16:41 -0600, Blake Hudson wrote: Actually it's still happening :( I read in the archives that the only way to actually avoid the locking in deliver.db is to also disable sieve, is that true? So, I'm willing to recompile cyrus and comment the calls to dupicate_check() and duplicate_mark() that are done in deliver_mailbox() in imap/lmtpd.c... would that break anything (other than duplicate checking :) ? Is there a cyrus version where specifying duplicatesuppression: 0 actually prevents the duplicate check and thus the database locking issues I could upgrade to a newer version too, no problem. Thanks, Andre Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Info-cyrus Digest, Vol 20, Issue 1
I'm using Cyrus replication. After several tests, it seems that the single instance store facility is not replicated. I mean, the same message sent to several recipient is stored once on the master, but stored several times on the slave. Is there a special thing to do to activate single instance store replication or it just doesn't exist yet? Message UUIDs are used to replicate the single instance store (see docs/text/install-replication). This won't have much effect when you first replicate a mailstore as sync_server in 2.3 only tracks the last few thousand messages that have been uploaded. It becomes much more effective when a replica has been seeded and you switch to rolling replication. After turning on rolling replication, 2 different hardlinks for the same message has been created in /bulk/imap/spool/sync./ on the replica ( one per recipient ) but not the same hardlink for the same message in the spool. Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Cyrus replication problems
Hy all again! I get more information. In copying from my slave 2.3.8 the sync_client and sync_server files to the master in 2.3.7 I am now able to run sync_client -v -m user.toto2 on my master it works! I think there is an incompatibility between the two modules but i'm absolutely unable to read and understand any codes. Now seems to work fine, also when i do a rolling sync! -- Debut du message initial --- De : [EMAIL PROTECTED] A : dkirhlarov [EMAIL PROTECTED] Copies : info-cyrus info-cyrus@lists.andrew.cmu.edu Date : Thu, 1 Mar 2007 17:39:36 +0100 Objet : Re: Cyrus replication problems Does someone can help me with it? It seems replication is rely hard to get working? Thanks! I remove the line syncserver cmd=/usr/cyrus/bin/sync_server listen=csync from my cyrus.conf then i try to sync manually, it doesn't sync anything. If i make as cyrus a sync_client -u -v i will see on the replica : Imapreplica syncserver[1185]: login: [10.1.45.3] cyrus DIGEST-MD5 User logged in So it seems to talk between my two servers. But nothing is synchronize ... But on my master, i get nothing on the var/log/messages ... Sometimes and i do not know how, i get on the master : Feb 27 17:58:02 imaptest sync_client[1109]: MAILBOXES received BAD response: Missing required argument to Mailboxes Feb 27 17:58:02 imaptest sync_client[1109]: Error in do_mailboxes(): bailing out! Feb 27 17:58:16 imaptest sync_client[]: MAILBOXES received BAD response: Missing required argument to Mailboxes Feb 27 17:58:16 imaptest sync_client[]: Error in do_mailboxes(): bailing out! Feb 27 17:58:22 imaptest sync_client[1112]: CREATE received BAD response: Missing required argument to Create Feb 27 17:58:22 imaptest sync_client[1112]: Error in do_mailboxes(): bailing out! I can't reproduce it everytimes... What i would like to do is to synchronize the master on the replica to use the replica as a backup server not High availability. So do i need to make rolling (as i understood, i think no! ?) Do i must put parameters at sync_client -u -v or can it sync every users and mailboxes (with -m) automatically? Can someone helps me please? Thanks a lot! -- Debut du message initial --- De : [EMAIL PROTECTED] A : info-cyrus@lists.andrew.cmu.edu Copies : Date : Mon, 26 Feb 2007 19:25:07 +0300 Objet : Re: Cyrus replication problems On Mon, Feb 26, 2007 at 02:35:05PM +0100, [EMAIL PROTECTED] wrote: I'm not able anymore to make as cyrus user a cyradm localhost I get : cyradm: cannot connect to server http://cyrusimap.web.cmu.edu/archive/message.php?mailbox=archive.info-cyrusmsg=41779 You must start sync_client over different script AFTER starting cyrus imapd server. WBR -- Dmitriy Kirhlarov OILspace, 26 Leninskaya sloboda, bld. 2, 2nd floor, 115280 Moscow, Russia P:+7 495 105 7245 ext.208 F:+7 495 105 7246 E:[EMAIL PROTECTED] Building Successful Supply Chains - One Solution At A Time. www.oilspace.com Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html Envoyez vos cartes de voeux depuis www.laposte.net Elles seront ensuite distribuées par le facteur : pratique et malin ! Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html Envoyez vos cartes de voeux depuis www.laposte.net Elles seront ensuite distribuées par le facteur : pratique et malin ! Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Cyrus replication problems
Hy all again! I get more information. In copying from my slave 2.3.8 the sync_client and sync_server files to the master in 2.3.7 I am now able to run sync_client -v -m user.toto2 on my master it works! I think there is an incompatibility between the two modules but i'm absolutely unable to read and understand any codes. Now seems to work fine, also when i do a rolling sync! -- Debut du message initial --- De : [EMAIL PROTECTED] A : dkirhlarov [EMAIL PROTECTED] Copies : info-cyrus info-cyrus@lists.andrew.cmu.edu Date : Thu, 1 Mar 2007 17:39:36 +0100 Objet : Re: Cyrus replication problems Does someone can help me with it? It seems replication is rely hard to get working? Thanks! I remove the line syncserver cmd=/usr/cyrus/bin/sync_server listen=csync from my cyrus.conf then i try to sync manually, it doesn't sync anything. If i make as cyrus a sync_client -u -v i will see on the replica : Imapreplica syncserver[1185]: login: [10.1.45.3] cyrus DIGEST-MD5 User logged in So it seems to talk between my two servers. But nothing is synchronize ... But on my master, i get nothing on the var/log/messages ... Sometimes and i do not know how, i get on the master : Feb 27 17:58:02 imaptest sync_client[1109]: MAILBOXES received BAD response: Missing required argument to Mailboxes Feb 27 17:58:02 imaptest sync_client[1109]: Error in do_mailboxes(): bailing out! Feb 27 17:58:16 imaptest sync_client[]: MAILBOXES received BAD response: Missing required argument to Mailboxes Feb 27 17:58:16 imaptest sync_client[]: Error in do_mailboxes(): bailing out! Feb 27 17:58:22 imaptest sync_client[1112]: CREATE received BAD response: Missing required argument to Create Feb 27 17:58:22 imaptest sync_client[1112]: Error in do_mailboxes(): bailing out! I can't reproduce it everytimes... What i would like to do is to synchronize the master on the replica to use the replica as a backup server not High availability. So do i need to make rolling (as i understood, i think no! ?) Do i must put parameters at sync_client -u -v or can it sync every users and mailboxes (with -m) automatically? Can someone helps me please? Thanks a lot! -- Debut du message initial --- De : [EMAIL PROTECTED] A : info-cyrus@lists.andrew.cmu.edu Copies : Date : Mon, 26 Feb 2007 19:25:07 +0300 Objet : Re: Cyrus replication problems On Mon, Feb 26, 2007 at 02:35:05PM +0100, [EMAIL PROTECTED] wrote: I'm not able anymore to make as cyrus user a cyradm localhost I get : cyradm: cannot connect to server http://cyrusimap.web.cmu.edu/archive/message.php?mailbox=archive.info-cyrusmsg=41779 You must start sync_client over different script AFTER starting cyrus imapd server. WBR -- Dmitriy Kirhlarov OILspace, 26 Leninskaya sloboda, bld. 2, 2nd floor, 115280 Moscow, Russia P:+7 495 105 7245 ext.208 F:+7 495 105 7246 E:[EMAIL PROTECTED] Building Successful Supply Chains - One Solution At A Time. www.oilspace.com Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html Envoyez vos cartes de voeux depuis www.laposte.net Elles seront ensuite distribuées par le facteur : pratique et malin ! Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html Envoyez vos cartes de voeux depuis www.laposte.net Elles seront ensuite distribuées par le facteur : pratique et malin ! Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Slow lmtpd
On Fri, 2 Mar 2007, Andre Nathan wrote: On Thu, 2007-03-01 at 16:41 -0600, Blake Hudson wrote: Actually it's still happening :( I read in the archives that the only way to actually avoid the locking in deliver.db is to also disable sieve, is that true? So, I'm willing to recompile cyrus and comment the calls to dupicate_check() and duplicate_mark() that are done in deliver_mailbox() in imap/lmtpd.c... would that break anything (other than duplicate checking :) ? You'll lose the ability to effectively use vacation/out-of-office messages. The vacation system uses deliver.db to determine which person has already received the vacation message, to prevent multiple messages from being sent. Is there a cyrus version where specifying duplicatesuppression: 0 actually prevents the duplicate check and thus the database locking issues I could upgrade to a newer version too, no problem. duplicatesuppression: 0 does prevent Cyrus from blocking duplicate messages. However, deliver.db is still used to track message-ids for the purpose of vacation messages, which is why Cyrus is still updating deliver.db. I'm not aware of any configure option that will disable deliver.db entirely. From the earlier discussion on this topic, it sounds to me like you are simply pushing your available hardware too hard without further tuning. You mentioned using ATA-over-Ethernet storage for your mail spool. Have you considered putting your configdirectory files on a local hard drive instead of on the ATA-over-Ethernet storage? There is a *lot* of contention for the files in the config directory, so maybe it would be better to move them onto a drive separate from the mail spool. After running iostat on my cyrus partition (both config and mail spool are kept on a SAN), I'm wondering if I should separate them out as well. This sounds related to the new metapartition and metapartition_files options that were added in v2.3.x. Does anyone have any recommendations or guidance on this topic? Andy Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Slow lmtpd
On Fri, 2007-03-02 at 12:59 -0800, Andrew Morgan wrote:
You'll lose the ability to effectively use vacation/out-of-office
messages. The vacation system uses deliver.db to determine which person
has already received the vacation message, to prevent multiple messages
from being sent.
But does vacation depend on the duplicate_{check,mark} calls from lmtpd?
I didn't read the sieve code, but imap/lmtp_sieve.c also calls those
functions.
From the earlier discussion on this topic, it sounds to me like you are
simply pushing your available hardware too hard without further tuning.
You mentioned using ATA-over-Ethernet storage for your mail spool. Have
you considered putting your configdirectory files on a local hard drive
instead of on the ATA-over-Ethernet storage? There is a *lot* of
contention for the files in the config directory, so maybe it would be
better to move them onto a drive separate from the mail spool.
The machine actually doesn't have any local disks (it's booted via pxe
and the root partition is also on AoE). The directories /var/spool/imap
and /var/lib/imap are each on its own LVM logical volume.
I mounted /var/lib/imap/proc as a memory-based filesystem (using tmpfs),
because of the contant writes to this directory, and yesterday I tried
moving deliver.db to that directory, and creating a symlink, but it
didn't improve the situation a lot.
After running iostat on my cyrus partition (both config and mail spool are
kept on a SAN), I'm wondering if I should separate them out as well. This
sounds related to the new metapartition and metapartition_files options
that were added in v2.3.x.
Does anyone have any recommendations or guidance on this topic?
Yes, people, please share :)
Thanks for the suggestions,
Andre
Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Slow lmtpd
On Fri, 2 Mar 2007, Andre Nathan wrote: On Fri, 2007-03-02 at 09:21 -0300, Andre Nathan wrote: Is there a cyrus version where specifying duplicatesuppression: 0 actually prevents the duplicate check and thus the database locking issues I could upgrade to a newer version too, no problem. I did that, and the lock contention problem with deliver.db is gone. I'm still seeing the problem in some users' cyrus.header files though. Searching the archives, I found some threads regarding this issue [1,2], or more recently in [3]. From the discussion on these threads it seems that the lock problem actually is happening with the quota files. The first two threads are old (from 2003), and they mention a bug filed in bugzilla which is marked as fixed, but the situation I'm seeing, and the one on the third thread (from 2006) seems to be the same problem. Has anyone ever been through that? I have no idea what could be causing the lock contention only for some users, so I'm not sure of any workaround for the problem :/ [1]http://marc.theaimsgroup.com/?l=info-cyrusm=106434064805178w=2 [2]http://marc.theaimsgroup.com/?l=info-cyrusm=106434491911478w=2 [3]http://www.irbs.net/internet/info-cyrus/0610/0390.html Items [1,2] are me... :) The problems I was having went away when I upgraded Cyrus. I can't remember specifically which version it was fixed in, but you shouldn't be running anything older than v2.2.13 these days anyways. In my case, I had an imaps process that was stuck waiting for input from the network for a client that had long since disconnected. The imaps process had a write lock open on the user's quota file, therefore any incoming mail for that user was stuck waiting for the write lock to become available. I doubt this is the problem you are seeing. It sounds to me like you may be hitting the limits of your storage system. I would be fascinated to see the output of the command iostat -x sdb 5 (where 'sdb' is the device you have cyrus on) on your system. Even if you aren't saturating your Gigabit Ethernet link to the ATA-over-Ethernet storage, you may be exceeding the number I/O operations per second. Here is an example of the iostat output on one of my cyrus systems: - avg-cpu: %user %nice%sys %iowait %idle 0.750.000.351.11 97.79 Device:rrqm/s wrqm/s r/s w/s rsec/s wsec/srkB/swkB/s avgrq-sz avgqu-sz await svctm %util sdb 0.20 91.11 13.13 107.68 195.56 1587.0797.78 793.54 14.76 0.302.50 1.34 16.16 - This is one of two murder backends running v2.2.13. The server is a Dell 2850 with two 3.0GHz Xeons and 2GB of RAM. It is attached to an EMC Cx500 SAN, specifically a 7 disk RAID5 array of 10k 146GB Fibre-channel drives. You can view the load average and process counts for these servers at: https://secure.onid.oregonstate.edu/cacti/graph_view.php?action=treetree_id=4 Andy Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Slow lmtpd
On Fri, 2007-03-02 at 17:07 -0800, Andrew Morgan wrote: I doubt this is the problem you are seeing. It sounds to me like you may be hitting the limits of your storage system. I would be fascinated to see the output of the command iostat -x sdb 5 (where 'sdb' is the device you have cyrus on) on your system. Even if you aren't saturating your Gigabit Ethernet link to the ATA-over-Ethernet storage, you may be exceeding the number I/O operations per second. Here are the numbers, but I don't think they'll mean much now, since the current load is pretty low. I'll have more significative numbers on monday, probably. Device:rrqm/s wrqm/s r/s w/s rsec/s wsec/srkB/swkB/s avgrq-sz avgqu-sz await svctm %util etherd/e0.0 0.00 0.00 1.82 37.17 17.78 354.34 8.89 177.17 9.54 0.00 48.15 48.15 187.74 Andre Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Slow lmtpd
You can view the load average and process counts for these servers at: https://secure.onid.oregonstate.edu/cacti/graph_view.php?action=treetree_id=4 OT, but I found this interesting... I wonder when school starts and ends and when holidays are :) https://secure.onid.oregonstate.edu/cacti/graph_image.php?graphid=59rraid=4 Rob Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Slow lmtpd
see the output of the command iostat -x sdb 5 (where 'sdb' is the device you have cyrus on) on your system. Even if you aren't saturating your Gigabit Ethernet link to the ATA-over-Ethernet storage, you may be exceeding the number I/O operations per second. I thought ATA-over-Ethernet had high latency issues that mean lots of random writes like cyrus does is problematic? Hmmm, I can't find a reference for this at the moment, but I'm sure I read about it in a review somewhere. The iostat command above will help determine if the IO is saturated or not, the final column a % utilisation figure, from the man page. %util - Percentage of CPU time during which I/O requests were issued to the device (bandwidth utilization for the device). Device saturation occurs when this value is close to 100%. The other thing of interest would be the load on the machine, and processes in D state. So running: uptime And: ps auxw | grep -v ' S' A few times to see how many processes are in D state vs R state. I've found IO debugging on linux quite hard. Finding CPU bottlenecks is easy with things like oprofile, but I haven't yet been able to find anything that tells me how long a process sits in a blocked state waiting on IO so I can get an idea of which processes are generating the IO, and which partiular parts of the code are generating it. Apparently 2.6.20 has better IO accounting, but I haven't tested it yet: http://kernelnewbies.org/Linux_2_6_20 IO Accounting The present per-task IO accounting isn't very useful. It simply counts the number of bytes passed into read() and write(). So if a process reads 1MB from an already-cached file, it is accused of having performed 1MB of I/O, which is 'wrong'. So this IO accounting implements per-process statistics of storage I/O (i.e.: I/O that _really_ does I/O on the storage device - Linux already had I/O storage statistics but it's not per-task). The data is reported through taskstats and procfs (/proc/$PID/io) So it sounds like you can find out which processes are the issue (imap or lmtp and/or particular users maybe even with a bit of work), but still not which parts of the code are causing the IOs. Rob Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Slow lmtpd
After running iostat on my cyrus partition (both config and mail spool are kept on a SAN), I'm wondering if I should separate them out as well. This sounds related to the new metapartition and metapartition_files options that were added in v2.3.x. Does anyone have any recommendations or guidance on this topic? Yes, people, please share :) We sponsored the patches that split out meta data and spool data and have found splitting them very useful. Even though the meta partition is only 1/20th the size of the spool partition, the meta partitions still get more IO. That shows how lopsided the IO in cyrus is towards meta data, so anything you can do to put the meta data on high-speed, low latency storage is a help. Also the more memory you have the better. Allowing the OS lots of spare memory to cache the cyrus.* files in memory is a great help as well. Rob Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
