Re: very slow syncing, any ideas?
Quoting Marten Lehmann <[EMAIL PROTECTED]>: Hello, i had much better performance with mailutil from UW-Imapd. It uses the IMAP-protocol like imapscyn but is not a scipt but a binary program and uses the imap APPEND command and does noe checks to see wich E-Mails are on the new server. and I hope it only deletes messages from the old server if they are transfered succesfully to the new server? It does not Delete the messages at all. Does it keep the flags of the messages, too? Yes, but i had to use "flushseenstate: 1" Does it also create required folders Yes and subscribe them (if they have been subscribed on the old server)? Im not sure about this, but you can rund imapsync or other programs to du this afterwards If you can answer everything with "yes", then I definetely should have a look at it, because while imapsync works well in general, it is really slow. Regards Marten Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html M.Menge Tel.: (49) 7071/29-70316 Universitaet Tuebingen Fax.: (49) 7071/29-5912 Zentrum fuer Datenverarbeitung mail: [EMAIL PROTECTED] Waechterstrasse 76 72074 Tuebingen smime.p7s Description: S/MIME krytographische Unterschrift Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: very slow syncing, any ideas?
Hello, i had much better performance with mailutil from UW-Imapd. It uses the IMAP-protocol like imapscyn but is not a scipt but a binary program and uses the imap APPEND command and does noe checks to see wich E-Mails are on the new server. and I hope it only deletes messages from the old server if they are transfered succesfully to the new server? Does it keep the flags of the messages, too? Does it also create required folders and subscribe them (if they have been subscribed on the old server)? If you can answer everything with "yes", then I definetely should have a look at it, because while imapsync works well in general, it is really slow. Regards Marten Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: very slow syncing, any ideas?
On Oct 20, 2006, at 04:09, Michael Menge wrote: Hi, i had much better performance with mailutil from UW-Imapd. It uses the IMAP-protocol like imapscyn but is not a scipt but a binary program and uses the imap APPEND command and does noe checks to see wich E-Mails are on the new server. Michael Menge I second the usefulness of mailutil. What also makes it neat is that it can read (natively) any c-client supported mail spool -- be it imap, local filesystem mail folders, etc. So, if you've got backend mail store that has a c-client driver, you can pull the data directly from the filesystem and bypass IMAP on the reading end altogether. -rob Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: very slow syncing, any ideas?
Hi, i had much better performance with mailutil from UW-Imapd. It uses the IMAP-protocol like imapscyn but is not a scipt but a binary program and uses the imap APPEND command and does noe checks to see wich E-Mails are on the new server. Michael Menge Quoting Marten Lehmann <[EMAIL PROTECTED]>: Hello, I'm about to migrate several thousand mailboxes from Maildir to Cyrus using the tool imapsync. It does its job very well and when I tested the migration on a small development server it was very fast. But now on the production system the synchronisation is very slow with a maximum of one message per second (and we have gigabytes of messages in the storage, partically > 10,000 messages per mailbox!). The general load of the system isn't very high, maybe a load average of 30. I disabled the duplicate message suppression. The mailboxes.db is about 8 megabytes big with approx. 13,000 mailboxes and 4 default folders each (Drafts, Junk, Sent, Trash). I have the following entries in my configuration which should provide a better hierarchie and balance of directories than if they were all in one main directory: altnamespace: true hashimapspool: true unixhierarchysep: true virtdomains: userid I also tried to move the old Maildirs to a different server, so that getting messages from the old mailbox and putting it to the new mailbox through IMAP doesn't come up with reads and writes on the same server. But the performance benefit was minimal. But in the end, syncing is still really slow. It would take weeks to sync all mailboxes that way. How else could we move them to the new storage if doing it through IMAP is too slow? On the other hand we would like to keep all flags so I guess syncing it with IMAP is the only choice? What could be the reason to be that slow? Is it the big mailboxes.db? Regards Marten Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html M.Menge Tel.: (49) 7071/29-70316 Universitaet Tuebingen Fax.: (49) 7071/29-5912 Zentrum fuer Datenverarbeitung mail: [EMAIL PROTECTED] Waechterstrasse 76 72074 Tuebingen smime.p7s Description: S/MIME krytographische Unterschrift Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: very slow syncing, any ideas?
On Thu, 19 Oct 2006, Michael Loftis wrote: --On October 19, 2006 11:25:07 PM +0200 Marten Lehmann <[EMAIL PROTECTED]> wrote: Hello, Uhm... LA of 30 is very high. What OS? I assume Linux, vmstat 5 will tell you where you're hitting the wall, but unless you've got an 8 CPU machine LA 30 is rather quite high. Linux LA is a measurement of processes blocked on I/O, processes running and processes waiting to run on a CPU. yes, Linux (2.6.9, RHEL4): Looking at taht i'd say you're VERY badly CPU bound. a simple dd/cp doesn't do anything to the mail but IMAP ops will require some CPU workCyrus also will probably be forcing syncs but your I/O load doesn't look that high (my mfe's run more I/O and they're not storing any mail, just logs and temporary files for virus/spam scanning heh, and they only have a little IDE HDD each) the numebrs in the procs->r column indicate you've got a lot of processes vying for CPU time. the b column indicates you've got a little bit of blocking going on. Processes in the 'D' state may show up in ps or top output (doubtfully top unless you change the default sort). I'd guess you're being CPU bound and cyrus is probably running (usually does by default) at a bit lower priority than your interactive cp operations. 2.6 kernel scheduler does a lot of weird stuff if it determines you're interactive, and then tends to give you priority over daemons. it's very non-deterministic, but I don't think you're running into this. quite likely some inefficiencies in the cyrus code (or intentional backoffs?) but mostly just being CPU bound. I know our MBEs can generally scp or rsync quite a bit faster than they can perform a XFER/RENAME command that moves between mail store servers (IE MURDER host to MURDER host). No clue if it matters in this imapsync case, but if you are using the 2.6 kernel, you should set the I/O scheduler to "deadline" rather than the default "anticipatory". You can set this in the boot loader, such as the following for lilo.conf: append="elevator=deadline" or you can change the scheduler on the fly by: echo "deadline" > /sys/block/sdb/queue/scheduler Replace "sdb" with the block device you store your cyrus mail spool on. Andy Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
RE: very slow syncing, any ideas?
Marten, > What would you do if you would need to migrate from Maildir > to cyrus? It > is not important for me to keep the flags. I would be happy > if I could > move all message files to the appropriate cyrus folder. But > will cyrus > detect them automatically? Or is there any rebuild-command required? I had the same problem when I did a migration. I have a fast mbox to cyrus migration script which you could convert to do maildir easily enough if you have any shell programming knowledge (and if not, time to learn!). You will lose all information though - the messages will be flagged as new. Instead of using an IMAP sync, you feed the email message in as a brand new message to the deliver program. It's fast and effective. Regards, Sarah Walters Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: very slow syncing, any ideas?
Hello, Looking at taht i'd say you're VERY badly CPU bound. a simple dd/cp doesn't do anything to the mail but IMAP ops will require some CPU workCyrus also will probably be forcing syncs but your I/O load doesn't look that high (my mfe's run more I/O and they're not storing any mail, just logs and temporary files for virus/spam scanning heh, and they only have a little IDE HDD each) I found the option berkeley_cachesize which is way too low by default. Transactions just reading or writing db-entries (i.e. changes of flags) are much faster now. Indead, the imapsync processes are causing much load: PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND 20461 root 25 0 79988 36m 2488 R 63 3.6 13:24.10 imapsync 20925 root 25 0 14908 8196 2664 R 57 0.8 0:31.49 imapsync But what does them make so cpu-expensive? I'd guess you're being CPU bound What would you do if you would need to migrate from Maildir to cyrus? It is not important for me to keep the flags. I would be happy if I could move all message files to the appropriate cyrus folder. But will cyrus detect them automatically? Or is there any rebuild-command required? Regards Marten Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: very slow syncing, any ideas?
--On October 19, 2006 11:25:07 PM +0200 Marten Lehmann <[EMAIL PROTECTED]> wrote: Hello, Uhm... LA of 30 is very high. What OS? I assume Linux, vmstat 5 will tell you where you're hitting the wall, but unless you've got an 8 CPU machine LA 30 is rather quite high. Linux LA is a measurement of processes blocked on I/O, processes running and processes waiting to run on a CPU. yes, Linux (2.6.9, RHEL4): Looking at taht i'd say you're VERY badly CPU bound. a simple dd/cp doesn't do anything to the mail but IMAP ops will require some CPU workCyrus also will probably be forcing syncs but your I/O load doesn't look that high (my mfe's run more I/O and they're not storing any mail, just logs and temporary files for virus/spam scanning heh, and they only have a little IDE HDD each) the numebrs in the procs->r column indicate you've got a lot of processes vying for CPU time. the b column indicates you've got a little bit of blocking going on. Processes in the 'D' state may show up in ps or top output (doubtfully top unless you change the default sort). I'd guess you're being CPU bound and cyrus is probably running (usually does by default) at a bit lower priority than your interactive cp operations. 2.6 kernel scheduler does a lot of weird stuff if it determines you're interactive, and then tends to give you priority over daemons. it's very non-deterministic, but I don't think you're running into this. quite likely some inefficiencies in the cyrus code (or intentional backoffs?) but mostly just being CPU bound. I know our MBEs can generally scp or rsync quite a bit faster than they can perform a XFER/RENAME command that moves between mail store servers (IE MURDER host to MURDER host). Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: very slow syncing, any ideas?
On Thu, 2006-10-19 at 21:59 +0200, Marten Lehmann wrote: > Hi, > > > What is I/O load on the new server? on the old server? Was your test set > > representative of your production system? > > the test system has much older hardware than the pretty new production > server (Xeon 3.4 with RAID5 and SCSI HDs). Old and new server are the > same from the hardware side, we have just changed the software. So your old mailserver (maildir) and your new cyrus server are on two different machines? Was the test server also running RAID5 for the spool? > Even syncing from an IMAP server in the internet to my local server was > faster. To which local server, your Cyrus server or your Maildir server? > I guess its none of them. uptime shows > > 21:04:59 up 1 day, 5:46, 7 users, load average: 29.10, 43.29, 41.94 What is the load a result of? Are you doing av checks on the new server? You've still not answered the fundmental question, what is causing this load? > I can easily copy large amounts of files within the system, so it is > definetely not a problem of the filesystem. Syncing is always slow, of > course large mails take a bit longer, but that isn't the bottleneck. Large being *greater* than the fs cache? What fs are you using? Is the same as the one on your old Maildir system? > The question is: What is Cyrus doing in the background when doing an > APPEND in IMAP (I guess thats the way new messages are added)? When I > can copy 20 files within a second in the filesystem, but only 1 in three > seconds through IMAP, whats wrong then? You've not told us whether or not the load is a result of Cyrus processes. Is it? Which Cyrus process? Authenication? IMAP? what? > We tried both: syncing from the same box or from one to another. It was > slow in both cases. I don't understand what you mean, do you mean you tried running imapsync on the old mailserver, then tried running off the new mailserver? > At this moment it is evening here, so there aren't many deliveries > (about 90 per minute). Argh. How much mail do you need to move, # messages and GB? > I guess that the bottleneck is somewhere in Cyrus. We are using > cyrus-imapd-2.2.12 from RHEL4. Is there a problem with berkeley db with > large mailboxes.db? What is Cyrus doing during an IMAP APPEND? Is it > always looking for the folder in mailboxes.db? Is it sync'ing the > harddisk after each APPEND? You need to pinpoint the bottleneck. What does vmstat look like while trying to sync? What filesystem are you running and with what options is it mounted? Are those options the same/different than your test/old servers? I have 60GB of email in 860,000 messages and my mailboxes.db is 323KB. > Our old way to deliver mails sort of broke down, so we urgently needed a > replacement. We worked on Cyrus for some weeks so we knew enough to > build the right configuration, but we didn't had the time to do a > stresstest. I can understand that, but you're shooting in the dark here, and you've given ZERO information that might help anyone on the list help you. Start by finding the bottleneck, don't assume that it's Cyrus. Especially since you didn't see this behavior on your old slower test system, I can't understand why you'd immediately assume it was Cyrus. Don't arbitrarily rule out possible causes for your problem (as it looks like you have). Sometimes it is something as stupid as a dns issue. Or a bad hardware. Or a bad nic. Or a ... Z Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: very slow syncing, any ideas?
Hello, Uhm... LA of 30 is very high. What OS? I assume Linux, vmstat 5 will tell you where you're hitting the wall, but unless you've got an 8 CPU machine LA 30 is rather quite high. Linux LA is a measurement of processes blocked on I/O, processes running and processes waiting to run on a CPU. yes, Linux (2.6.9, RHEL4): procs ---memory-- ---swap-- -io --system-- cpu r b swpd free buff cache si sobibo incs us sy id wa 15 1 34896 800696 11788 4399685 595 670 2763 34 6 14 46 32 2 34896 792796 11332 4627200 1138 1124 1851 1427 83 8 1 9 11 3 34852 794696 9264 4392470 1194 1087 1896 1546 81 8 1 9 33 1 34852 791752 8660 4478800 902 1059 1933 1672 81 9 1 9 13 2 34852 792652 7004 4150400 1220 1037 1863 1476 81 8 2 9 30 2 34852 795584 7572 4093600 423 1222 1876 1484 82 7 2 9 16 3 34852 796836 8816 4411200 817 1268 1927 1613 83 8 1 8 12 1 34852 794000 9888 4434000 249 1135 1821 1458 82 8 3 8 9 2 34852 793328 9876 4617200 1098 976 1816 1091 86 7 2 6 7 5 34852 790188 10332 4831600 1100 1274 1785 959 85 7 2 6 26 1 34852 789504 7656 5255200 1837 1087 1991 1444 81 8 2 9 20 0 34852 794252 6900 4706800 440 1246 1932 1461 83 8 2 7 Does this tell you anything? I have two imapsync processes which cause some load. Regards Marten Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: very slow syncing, any ideas?
Hi, What is I/O load on the new server? on the old server? Was your test set representative of your production system? the test system has much older hardware than the pretty new production server (Xeon 3.4 with RAID5 and SCSI HDs). Old and new server are the same from the hardware side, we have just changed the software. You could be running into any number of bottlenecks that you've not described: 1) disk I/O 2) network I/O Even syncing from an IMAP server in the internet to my local server was faster. 3) message size 4) file system format I guess its none of them. uptime shows 21:04:59 up 1 day, 5:46, 7 users, load average: 29.10, 43.29, 41.94 I can easily copy large amounts of files within the system, so it is definetely not a problem of the filesystem. Syncing is always slow, of course large mails take a bit longer, but that isn't the bottleneck. The question is: What is Cyrus doing in the background when doing an APPEND in IMAP (I guess thats the way new messages are added)? When I can copy 20 files within a second in the filesystem, but only 1 in three seconds through IMAP, whats wrong then? So both the maildir server and the Cyrus server are on the same box? We tried both: syncing from the same box or from one to another. It was slow in both cases. Do their spools share the disks? Controllers? How many users? How many connections? How much mail? At this moment it is evening here, so there aren't many deliveries (about 90 per minute). I guess that the bottleneck is somewhere in Cyrus. We are using cyrus-imapd-2.2.12 from RHEL4. Is there a problem with berkeley db with large mailboxes.db? What is Cyrus doing during an IMAP APPEND? Is it always looking for the folder in mailboxes.db? Is it sync'ing the harddisk after each APPEND? You need to better understand the limits of your system. Yes your development system was fast, but it sounds like you didn't try to replicate the load of doing everything at once on it, so your expectations have been set artifically. Our old way to deliver mails sort of broke down, so we urgently needed a replacement. We worked on Cyrus for some weeks so we knew enough to build the right configuration, but we didn't had the time to do a stresstest. Regards Marten Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: very slow syncing, any ideas?
--On October 19, 2006 8:10:13 PM +0200 Marten Lehmann <[EMAIL PROTECTED]> wrote: Hello, I'm about to migrate several thousand mailboxes from Maildir to Cyrus using the tool imapsync. It does its job very well and when I tested the migration on a small development server it was very fast. But now on the production system the synchronisation is very slow with a maximum of one message per second (and we have gigabytes of messages in the storage, partically > 10,000 messages per mailbox!). The general load of the system isn't very high, maybe a load average of 30. I disabled the duplicate message suppression. The mailboxes.db is about 8 megabytes big with approx. 13,000 mailboxes and 4 default folders each (Drafts, Junk, Sent, Trash). Uhm... LA of 30 is very high. What OS? I assume Linux, vmstat 5 will tell you where you're hitting the wall, but unless you've got an 8 CPU machine LA 30 is rather quite high. Linux LA is a measurement of processes blocked on I/O, processes running and processes waiting to run on a CPU. -- "Genius might be described as a supreme capacity for getting its possessors into trouble of all kinds." -- Samuel Butler Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: very slow syncing, any ideas?
On Thu, 2006-10-19 at 20:10 +0200, Marten Lehmann wrote: > Hello, > > I'm about to migrate several thousand mailboxes from Maildir to Cyrus > using the tool imapsync. It does its job very well and when I tested the > migration on a small development server it was very fast. > But now on the production system the synchronisation is very slow with a > maximum of one message per second (and we have gigabytes of messages in > the storage, partically > 10,000 messages per mailbox!). The general > load of the system isn't very high, maybe a load average of 30. I > disabled the duplicate message suppression. The mailboxes.db is about 8 > megabytes big with approx. 13,000 mailboxes and 4 default folders each > (Drafts, Junk, Sent, Trash). What is I/O load on the new server? on the old server? Was your test set representative of your production system? You could be running into any number of bottlenecks that you've not described: 1) disk I/O 2) network I/O 3) message size 4) file system format etc. etc. Find out what those are, then find out which you're hitting, then come back. > I also tried to move the old Maildirs to a different server, so that > getting messages from the old mailbox and putting it to the new mailbox > through IMAP doesn't come up with reads and writes on the same server. > But the performance benefit was minimal. So both the maildir server and the Cyrus server are on the same box? Do their spools share the disks? Controllers? How many users? How many connections? How much mail? You need to better understand the limits of your system. Yes your development system was fast, but it sounds like you didn't try to replicate the load of doing everything at once on it, so your expectations have been set artifically. Z Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
very slow syncing, any ideas?
Hello, I'm about to migrate several thousand mailboxes from Maildir to Cyrus using the tool imapsync. It does its job very well and when I tested the migration on a small development server it was very fast. But now on the production system the synchronisation is very slow with a maximum of one message per second (and we have gigabytes of messages in the storage, partically > 10,000 messages per mailbox!). The general load of the system isn't very high, maybe a load average of 30. I disabled the duplicate message suppression. The mailboxes.db is about 8 megabytes big with approx. 13,000 mailboxes and 4 default folders each (Drafts, Junk, Sent, Trash). I have the following entries in my configuration which should provide a better hierarchie and balance of directories than if they were all in one main directory: altnamespace: true hashimapspool: true unixhierarchysep: true virtdomains: userid I also tried to move the old Maildirs to a different server, so that getting messages from the old mailbox and putting it to the new mailbox through IMAP doesn't come up with reads and writes on the same server. But the performance benefit was minimal. But in the end, syncing is still really slow. It would take weeks to sync all mailboxes that way. How else could we move them to the new storage if doing it through IMAP is too slow? On the other hand we would like to keep all flags so I guess syncing it with IMAP is the only choice? What could be the reason to be that slow? Is it the big mailboxes.db? Regards Marten Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
