Re: improving concurrency/performance
Hi, I would just request that the tests and comments in this thread should be added to the Cyrus wiki. Kind regards, Tarjei On Mon, 2005-11-07 at 02:46 -0200, Sergio Devojno Bruder wrote: David Lang wrote: (..) I was recently doing some testing of lots of small files on the various filesystems, and I ran into a huge difference (8x) depending on what allocator was used for ext*. the default allocator changed between ext2 and ext3 (you can override it as a mount option) and when reading 1M files (10 dirs of 10 dirs of 10 dirs of 1000 1K files) the time to read them went from ~5 min with the old allocator useed in ext2 to 40 min for the one that's the default for ext3. David Lang (!!) Interesting. You said mount options? man mount man page only show me data=journal, data=ordered, data=writeback, etcetera. How can I change that? -- Sergio Bruder Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html -- Tarjei Huse [EMAIL PROTECTED] Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: deliver mails to user's Bulk folder
Ivan R. Sy Jr. wrote: hi all I got postfix-amavisd(with ClamAV and SpamAssasin)-postfix-cyrus-imapd22 already running smoothly, question now is, i want that the spam mails (already tagged in the MIME headers) get delivered to the user's Bulk folder? Ive already set autocreateinbox and autosubscribe for each user. also how do automated clean up per user's bulk messages for emails delivered 30 days. something like yahoo mail, yes. can anyone give me some leads to this? To have the mail routed to the Bulk folder, you'd probably want to use sieve scripts, such as those at http://wiki.fastmail.fm/index.php/SieveExamples . I don't know if 'global sieve scripts' are possible. To have the entries cleared periodically, use an entry in /etc/cyrus.conf in the events section, such as # Purge trash from folders purgebulkcmd=/usr/sbin/ipurge -d 30 -f user.%.Bulk at=0200 Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: improving concurrency/performance
Hi, Andrew Morgan wrote: On Sun, 6 Nov 2005, Michael Loftis wrote: I'd also be VERY interested since our experience was quite the opposite. ReiserFS was faster than all three, XFS trailing a dismal third (also had corruption issues) and ext3 second or even more dismal third, depending on if you ignored it's wretched large directory performance or not. ReiserFS performed solidly and predictably in all tests. Not the same could be said for XFS and ext3. This was about 2 yrs ago though. Make sure that you format ext3 partitions with dir_index which improves large directory performance. ... but decreases read performance in general... at least that is what I found under RH / Fedora! Look at: http://www.surfnetters.nl/paul/fs/2/read.png http://www.surfnetters.nl/paul/fs/tarcopy-read-ext3.png ... to see that reading from ext3 with dir_index enabled takes about 2h15 to read 20 Gb of mail data, while... http://www.surfnetters.nl/paul/fs/read-plainext3-reiserfs.png http://www.surfnetters.nl/paul/fs/2/read2.png ... without dir_index it takes only 15 minutes! ReiserFS was a bit slower for me with reads, but faster in writes. ReiserFS was also predictable in writes, where ext3 was slow(er) on large directories, but not that dramaticly. (I have graphs of that too.) http://www.surfnetters.nl/paul/fs/2/write.png BTW: I found that chaning dir_index with tune2fs didn't work as expected. If I disabled the dir_indexes, even after a forced fsck, performance was still slow. Enabling didn't give predictable results either: I had to specify it with mkfs. I posted this to the RedHat list once, no-one replied. I decided that my tests where satisfied my questions for what fs to use under RedHat 4 for our NG mail platform... (ext3: supported, lots of (coroner) tools, fast enough, available in the stock kernel (if you need RH), ... and ReiserFS 3 already has a successor, ...) Paul P.S. I also compared FreeBSD's UFS2 under 5.3. I should maybe try again with 6, since that release should have improved filesystem and disk performance in general. We use FreeBSD now, it's a pity we move to RH for this, but Dell hardware maybe says enough. P.S. You can look at the graph's including some comments on http://www.surfnetters.nl/paul/fs/2 and http://www.surfnetters.nl/paul/fs (more rubbish) Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Problem with Subject: XXXX XXXXXX on Cyrus
Hello All, I havevery annoying problemsince I've moved to Cyrus from UW-IMAP: When someone sends e-mail froma number ofservers to my users, and Subject line is in hebrew, the only Subject they see on their Outlook Express or Horde IMP is Subject: XX On the other hand, same message sent to Exchange account looks OK on the Subject line. Is there any solution to this kind of problem? Here are the headers of the message I got on Cyrus account with Subject: XX: Return-Path: [EMAIL PROTECTED]Received: from mail.edu.haifa.ac.il ([unix socket])by mail.edu.haifa.ac.il (Cyrus v2.2.3) with LMTP; Mon, 07 Nov 2005 13:28:26 +0200X-Sieve: CMU Sieve 2.2Received: from localhost (localhost [127.0.0.1])by mail.edu.haifa.ac.il (Postfix) with ESMTP id 73F921BEC4for [EMAIL PROTECTED]; Mon, 7 Nov 2005 13:28:26 +0200 (IST)Received: from mail.edu.haifa.ac.il ([127.0.0.1])by localhost (mail.edu.haifa.ac.il [127.0.0.1]) (amavisd-new, port 10024)with ESMTP id 26288-07 for [EMAIL PROTECTED];Mon, 7 Nov 2005 13:28:26 +0200 (IST)Received: from mr2.haifa.ac.il (mr2.haifa.ac.il [132.74.1.218])by mail.edu.haifa.ac.il (Postfix) with ESMTP id 3EAA61BEA3for [EMAIL PROTECTED]; Mon, 7 Nov 2005 13:28:26 +0200 (IST)Received: from localhost (mr2 [127.0.0.1])by mr2.haifa.ac.il (Postfix) with ESMTP id 62B543803Bfor [EMAIL PROTECTED]; Mon, 7 Nov 2005 13:21:56 +0200 (IST)Received: from mr2.haifa.ac.il ([127.0.0.1])by localhost (mr2.haifa.ac.il [127.0.0.1]) (amavisd-new, port 10026)with ESMTP id 01525-09 for [EMAIL PROTECTED];Mon, 7 Nov 2005 13:21:53 +0200 (IST)Received: from hotmail.com (bay109-f13.bay109.hotmail.com [64.4.19.23])by mr2.haifa.ac.il (Postfix) with ESMTP id 8D12A380C1for [EMAIL PROTECTED]; Mon, 7 Nov 2005 13:21:49 +0200 (IST)Received: from mail pickup service by hotmail.com with Microsoft SMTPSVC; Mon, 7 Nov 2005 03:27:47 -0800Message-ID: [EMAIL PROTECTED]Received: from 64.4.19.200 by by109fd.bay109.hotmail.msn.com with HTTP;Mon, 07 Nov 2005 11:27:47 GMTX-Originating-IP: [132.74.99.84]X-Originating-Email: [EMAIL PROTECTED]X-Sender: [EMAIL PROTECTED]From: "leon kolchinsky" [EMAIL PROTECTED]To: [EMAIL PROTECTED]Cc: [EMAIL PROTECTED]Subject: XXDate: Mon, 07 Nov 2005 11:27:47 +Mime-Version: 1.0Content-Type: text/plain; format=flowedX-OriginalArrivalTime: 07 Nov 2005 11:27:47.0820 (UTC) FILETIME=[47BA9AC0:01C5E38E]X-Virus-Scanned: by amavisd-new at haifa.ac.ilX-Virus-Scanned: by amavisd-new at edu.haifa.ac.ilX-Amavis-Alert: BAD HEADER Non-encoded 8-bit data (char F7 hex) in message header 'Subject' Subject: \367\345\370\361 \344\351\351\362\345\365\n ^ Here are the same message I got on Exchange account with looking OK Subject line: Microsoft Mail Internet Headers Version 2.0Received: from exchsrv02.haifa.edu ([132.74.1.68]) by exchsrv01.haifa.edu with Microsoft SMTPSVC(6.0.3790.1830); Mon, 7 Nov 2005 13:27:39 +0200Received: from mr2.haifa.ac.il ([132.74.1.218]) by exchsrv02.haifa.edu with Microsoft SMTPSVC(6.0.3790.1830); Mon, 7 Nov 2005 13:27:39 +0200Received: from localhost (mr2 [127.0.0.1])by mr2.haifa.ac.il (Postfix) with ESMTP id 690CC380B6for [EMAIL PROTECTED]; Mon, 7 Nov 2005 13:21:56 +0200 (IST)Received: from mr2.haifa.ac.il ([127.0.0.1])by localhost (mr2.haifa.ac.il [127.0.0.1]) (amavisd-new, port 10026)with ESMTP id 01563-03 for [EMAIL PROTECTED];Mon, 7 Nov 2005 13:21:53 +0200 (IST)Received: from hotmail.com (bay109-f13.bay109.hotmail.com [64.4.19.23])by mr2.haifa.ac.il (Postfix) with ESMTP id 98CCC380CEfor [EMAIL PROTECTED]; Mon, 7 Nov 2005 13:21:49 +0200 (IST)Received: from mail pickup service by hotmail.com with Microsoft SMTPSVC; Mon, 7 Nov 2005 03:27:47 -0800Message-ID: [EMAIL PROTECTED]Received: from 64.4.19.200 by by109fd.bay109.hotmail.msn.com with HTTP;Mon, 07 Nov 2005 11:27:47 GMTX-Originating-IP: [132.74.99.84]X-Originating-Email: [EMAIL PROTECTED]X-Sender: [EMAIL PROTECTED]From: "leon kolchinsky" [EMAIL PROTECTED]To: [EMAIL PROTECTED]Cc: [EMAIL PROTECTED]Subject: ÷åøñ äééòåõDate: Mon, 07 Nov 2005 11:27:47 +Mime-Version: 1.0Content-Type: text/plain; format=flowedX-OriginalArrivalTime: 07 Nov 2005 11:27:47.0820 (UTC) FILETIME=[47BA9AC0:01C5E38E]X-Virus-Scanned: by amavisd-new at haifa.ac.ilX-Amavis-Alert: BAD HEADER Non-encoded 8-bit data (char F7 hex) in message header 'Subject': Subject: \367\345\370\361 \344\351\351\362\345\365\nReturn-Path: [EMAIL PROTECTED] Best Regards, Leon Kolchinsky Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Problem with Subject: XXXX XXXXXX on Cyrus
Hello All, I have very annoying problem since I've moved to Cyrus from UW-IMAP: When someone sends e-mail from a number of servers to my users, and Subject line is in hebrew, the only Subject they see on their Outlook Express or Horde IMP is Subject: XX On the other hand, same message sent to Exchange account looks OK on the Subject line. Hi Leon, The problem is that non-encoded 8bit data is not allowed in message headers and Cyrus-IMAPd prevents from any problem by replacing those chars with X. Your testmail were sent from hotmail which should then be called broken. Is there any solution to this kind of problem? From what I know you have to fix the problem before the mail reaches the cyrus message store. Maybe you can fix those mail with amavis, at least amavis warns you that the mails are broken (BAD HEADER). I don't think there is a way to let cyrus handle those broken messages. Regards, Simon Here are the headers of the message I got on Cyrus account with Subject: XX: Return-Path: [EMAIL PROTECTED] Received: from mail.edu.haifa.ac.il ([unix socket]) by mail.edu.haifa.ac.il (Cyrus v2.2.3) with LMTP; Mon, 07 Nov 2005 13:28:26 +0200 X-Sieve: CMU Sieve 2.2 Received: from localhost (localhost [127.0.0.1]) by mail.edu.haifa.ac.il (Postfix) with ESMTP id 73F921BEC4 for [EMAIL PROTECTED]; Mon, 7 Nov 2005 13:28:26 +0200 (IST) Received: from mail.edu.haifa.ac.il ([127.0.0.1]) by localhost (mail.edu.haifa.ac.il [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 26288-07 for [EMAIL PROTECTED]; Mon, 7 Nov 2005 13:28:26 +0200 (IST) Received: from mr2.haifa.ac.il (mr2.haifa.ac.il [132.74.1.218]) by mail.edu.haifa.ac.il (Postfix) with ESMTP id 3EAA61BEA3 for [EMAIL PROTECTED]; Mon, 7 Nov 2005 13:28:26 +0200 (IST) Received: from localhost (mr2 [127.0.0.1]) by mr2.haifa.ac.il (Postfix) with ESMTP id 62B543803B for [EMAIL PROTECTED]; Mon, 7 Nov 2005 13:21:56 +0200 (IST) Received: from mr2.haifa.ac.il ([127.0.0.1]) by localhost (mr2.haifa.ac.il [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id 01525-09 for [EMAIL PROTECTED]; Mon, 7 Nov 2005 13:21:53 +0200 (IST) Received: from hotmail.com (bay109-f13.bay109.hotmail.com [64.4.19.23]) by mr2.haifa.ac.il (Postfix) with ESMTP id 8D12A380C1 for [EMAIL PROTECTED]; Mon, 7 Nov 2005 13:21:49 +0200 (IST) Received: from mail pickup service by hotmail.com with Microsoft SMTPSVC; Mon, 7 Nov 2005 03:27:47 -0800 Message-ID: [EMAIL PROTECTED] Received: from 64.4.19.200 by by109fd.bay109.hotmail.msn.com with HTTP; Mon, 07 Nov 2005 11:27:47 GMT X-Originating-IP: [132.74.99.84] X-Originating-Email: [EMAIL PROTECTED] X-Sender: [EMAIL PROTECTED] From: leon kolchinsky [EMAIL PROTECTED] To: [EMAIL PROTECTED] Cc: [EMAIL PROTECTED] Subject: XX Date: Mon, 07 Nov 2005 11:27:47 + Mime-Version: 1.0 Content-Type: text/plain; format=flowed X-OriginalArrivalTime: 07 Nov 2005 11:27:47.0820 (UTC) FILETIME=[47BA9AC0:01C5E38E] X-Virus-Scanned: by amavisd-new at haifa.ac.il X-Virus-Scanned: by amavisd-new at edu.haifa.ac.il X-Amavis-Alert: BAD HEADER Non-encoded 8-bit data (char F7 hex) in message header 'Subject' Subject: \367\345\370\361 \344\351\351\362\345\365\n ^ Here are the same message I got on Exchange account with looking OK Subject line: Microsoft Mail Internet Headers Version 2.0 Received: from exchsrv02.haifa.edu ([132.74.1.68]) by exchsrv01.haifa.edu with Microsoft SMTPSVC(6.0.3790.1830); Mon, 7 Nov 2005 13:27:39 +0200 Received: from mr2.haifa.ac.il ([132.74.1.218]) by exchsrv02.haifa.edu with Microsoft SMTPSVC(6.0.3790.1830); Mon, 7 Nov 2005 13:27:39 +0200 Received: from localhost (mr2 [127.0.0.1]) by mr2.haifa.ac.il (Postfix) with ESMTP id 690CC380B6 for [EMAIL PROTECTED]; Mon, 7 Nov 2005 13:21:56 +0200 (IST) Received: from mr2.haifa.ac.il ([127.0.0.1]) by localhost (mr2.haifa.ac.il [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id 01563-03 for [EMAIL PROTECTED]; Mon, 7 Nov 2005 13:21:53 +0200 (IST) Received: from hotmail.com (bay109-f13.bay109.hotmail.com [64.4.19.23]) by mr2.haifa.ac.il (Postfix) with ESMTP id 98CCC380CE for [EMAIL PROTECTED]; Mon, 7 Nov 2005 13:21:49 +0200 (IST) Received: from mail pickup service by hotmail.com with Microsoft SMTPSVC; Mon, 7 Nov 2005 03:27:47 -0800 Message-ID: [EMAIL PROTECTED] Received: from 64.4.19.200 by by109fd.bay109.hotmail.msn.com with HTTP; Mon, 07 Nov 2005 11:27:47 GMT X-Originating-IP: [132.74.99.84] X-Originating-Email: [EMAIL PROTECTED] X-Sender: [EMAIL PROTECTED] From: leon kolchinsky [EMAIL PROTECTED] To: [EMAIL PROTECTED] Cc: [EMAIL PROTECTED] Subject: ÷åøñ äééòåõ Date: Mon, 07 Nov 2005 11:27:47 + Mime-Version: 1.0 Content-Type: text/plain; format=flowed X-OriginalArrivalTime: 07 Nov 2005 11:27:47.0820 (UTC) FILETIME=[47BA9AC0:01C5E38E] X-Virus-Scanned: by amavisd-new at haifa.ac.il X-Amavis-Alert: BAD HEADER Non-encoded 8-bit
Re: Problem with Subject: XXXX XXXXXX on Cyrus
Hi, --On 7. November 2005 13:40:05 +0200 [EMAIL PROTECTED] wrote: I have very annoying problem since I've moved to Cyrus from UW-IMAP: When someone sends e-mail from a number of servers to my users, and Subject line is in hebrew, the only Subject they see on their Outlook Express or Horde IMP is Subject: XX On the other hand, same message sent to Exchange account looks OK on the Subject line. Is there any solution to this kind of problem? sure, use a standards-compliant mail program that does RFC 2047-encoding on all headers. Unencoded 8-bit characters aren't allowed in headers. That's why Cyrus replaces them with X. Cheers, Sebastian Hagedorn -- Sebastian Hagedorn - RZKR-R1 (Gebäude 52), Zimmer 18 Zentrum für angewandte Informatik - Universitätsweiter Service RRZK Universität zu Köln / Cologne University - Tel. +49-221-478-5587 Skype: shagedorn pgplJfdOntyuV.pgp Description: PGP signature Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Problem with Subject: XXXX XXXXXX on Cyrus
[EMAIL PROTECTED] wrote: I have very annoying problem since I've moved to Cyrus from UW-IMAP: When someone sends e-mail from a number of servers to my users, and Subject line is in hebrew, the only Subject they see on their Outlook Express or Horde IMP is Subject: XX On the other hand, same message sent to Exchange account looks OK on the Subject line. Is there any solution to this kind of problem? [...] You can set munge8bit option to zero in imapd.conf. I am not sure if it requires patching cmu sources. Check man imapd.conf on your system to find out if it is supported by your binary. -- [en: Andrew] Andrzej Adam Filip : [EMAIL PROTECTED] : [EMAIL PROTECTED] http://anfi.homeunix.net/ Netcraft Site Rank: 483671 All that is necessary for the triumph of evil is that good men do nothing -- Edmund Burke, 18th century Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: improving concurrency/performance
Have you tried running something like postmark http://packages.debian.org/stable/utils/postmark to benchmark your filesystem? The disks are quite fast. bonnie++, for example, shows writes at over 300MB/s. What I'm finding though is that the processes aren't ever pegging them out -- nothing ever goes into iowait. The bottleneck is elsewhere... John -- John Madden UNIX Systems Engineer Ivy Tech Community College of Indiana [EMAIL PROTECTED] Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Authenticating (with cyradm) using an alternate Kerberos instance?
We're running 'saslauthd -a ldap'... Which I guess is irrelevant. When using gssapi authentication, Cyrus never contacts saslauthd. So, I'm still puzzled, but I've got one less thing to look at. -- Lars Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Problem with Subject: XXXX XXXXXX on Cyrus
[EMAIL PROTECTED] wrote: Hello All, Thanks to all who replyed. I understand now that this are the hotmail and many other clients to blame for these kind of headers. But in reality I would like to deal with this problem (not just blame the clients) via Cyrus (UW-IMAP and Exchange seem to deal with the problem pretty well). Many users on my system now very dissapointed by this header handling :( This munge8bit option, Andrzej wrote about is very interesting but I can't see this option with man imapd.conf on my SLES9 system. Is there more information on enabling this munge8bit option? The patch from Igor is also interesting, if nothing else works, I'd like to try it. Best Regards, Leon Kolchinsky munge8bit patch is included in FedoraCore4 rpm I use (from extras repository). As last repsort you may try to extract it from the source rpm. -- [en: Andrew] Andrzej Adam Filip : [EMAIL PROTECTED] : [EMAIL PROTECTED] http://anfi.homeunix.net/ Netcraft Site Rank: 483671 All that is necessary for the triumph of evil is that good men do nothing -- Edmund Burke, 18th century Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: improving concurrency/performance
On Mon, 7 Nov 2005 09:00:08 -0500 (EST) John Madden [EMAIL PROTECTED] wrote: The disks are quite fast. bonnie++, for example, shows writes at over 300MB/s. What I'm finding though is that the processes aren't ever pegging them out -- nothing ever goes into iowait. The bottleneck is elsewhere... It's situations like this Dtrace was made for. But on linux we still have to use some 'gut feeling' to figure it out ... So you say you have fast disks for bonnie, but still see slow imap copy operations? What kind of SAN exactly do you have attached? Because fsync() calls would still be my primary suspect here ... You say you're copying mail spool from one box to another via imap. Is the source box able to provide mails at a fast enough rate? -- Jure Pečar http://jure.pecar.org/ Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: improving concurrency/performance
On Mon, Nov 07, 2005 at 11:59:39AM +0100, Paul Dekkers wrote: Make sure that you format ext3 partitions with dir_index which improves large directory performance. ... but decreases read performance in general... at least that is what I found under RH / Fedora! Yes, processing directory entries in the order returned by readdir() is slower when dir_index is enabled: https://listman.redhat.com/archives/ext3-users/2004-September/msg00029.html Actually you'd get similar slowdown if you create/delete a lot of files in random order. The speed of readdir() w/ dir_index on a newly populated directory should be similar to the speed of readdir() w/o dir_index on a heavily used mail folder. It would be interesting if you could repeat the measurement with LD_PRELOAD'ing the readdir-sorting library posted in: https://listman.redhat.com/archives/ext3-users/2004-September/msg00025.html BTW: I found that chaning dir_index with tune2fs didn't work as expected. If I disabled the dir_indexes, even after a forced fsck, performance was still slow. Enabling didn't give predictable results either: I had to specify it with mkfs. Disabling dir_index will not reorder existing directories. Again, the result should be similar to what you'd get after creating/deleting a lot of files in random order, so that the readdir() order no longer matches the disk layout. Also, this read benchmark only models the case when users download every mail in a folder sequentially (like POP3). It does not tell anything about the case when users randomly open files (by explicit filename instead of using readdir()) inside a folder. I think Cyrus uses its own database instead of doing a readdir() even for POP3, so this benchmark may not even match POP3 usage. Gabor -- - MTA SZTAKI Computer and Automation Research Institute Hungarian Academy of Sciences - Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: improving concurrency/performance
It's situations like this Dtrace was made for. But on linux we still have to use some 'gut feeling' to figure it out ... True. It's that sort of tool that I'm looking for, specifically to look into concurrency on the skiplist db's, as the system load is so low that it seems there's got to be a simple explanation for what's going on. So you say you have fast disks for bonnie, but still see slow imap copy operations? What kind of SAN exactly do you have attached? Because fsync() calls would still be my primary suspect here ... It's an IBM DS-6800. You say you're copying mail spool from one box to another via imap. Is the source box able to provide mails at a fast enough rate? Yeah, the load average there dropped to near zero, with no iowait and only 1-3MB/s coming off its disk. Perhaps it's worth repeating: With a single imapcopy process, the whole thing goes along pretty quickly, but drops off significantly with a second process and comes to basically a crawl with just 5 processes running concurrently. I gambled that I could shorten my migration by running more than one at a time since one only seems to raise the load on the box to 0.80. With 5, I'm only able to get it to around 2.5 and only briefly as the throughput starts to drop off. With a little multi-threaded perl, I wrote a quick benchmark script that (in parallel) grabs a random user, logs in, selects the INBOX, pulls out the first message and logs out. I'm able to do about 230 of those per second, so at least the read performance is more than acceptable. (And the client box here, a 4-CPU Opteron 850, is definitely the bottleneck anyway.) John -- John Madden UNIX Systems Engineer Ivy Tech Community College of Indiana [EMAIL PROTECTED] Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: improving concurrency/performance
On Mon, 7 Nov 2005 12:41:03 -0500 (EST) John Madden [EMAIL PROTECTED] wrote: Perhaps it's worth repeating: With a single imapcopy process, the whole thing goes along pretty quickly, but drops off significantly with a second process and comes to basically a crawl with just 5 processes running concurrently. I gambled that I could shorten my migration by running more than one at a time since one only seems to raise the load on the box to 0.80. With 5, I'm only able to get it to around 2.5 and only briefly as the throughput starts to drop off. That is a start. Try to strace -tt all of imapd processes running concurrently and examine in which syscalls most time is spent. I hope that would give you at least a lead ... For example, on my production system I see some suspicious long pauses at fcntl64(0x8, 0x7, 0xsomeaddr, 0xsomeotheraddr) calls ... lets dig what this is. -- Jure Pečar http://jure.pecar.org/ Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: improving concurrency/performance
On Mon, 7 Nov 2005 22:31:42 +0100 Jure Pečar [EMAIL PROTECTED] wrote: For example, on my production system I see some suspicious long pauses at fcntl64(0x8, 0x7, 0xsomeaddr, 0xsomeotheraddr) calls ... lets dig what this is. As expected, these are from locking operations. 0x8 is file descriptor, which, if I read lsof output correctly, points to config/socket/imap-0.lock (what would that be?) and 0x7 is F_SETLKW which reads as set lock or wait for it to be released in the manual page. I'm sure some cyrus expert (Ken? :) can explain immediately the role of imap-0.lock and all the locking going on around it ... And if there's anything we can do to speed it up ... -- Jure Pečar http://jure.pecar.org/ Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
only numeric Unknown Error Code in logs; KEMessage from imap_err.strings not being logged
-BEGIN PGP SIGNED MESSAGE- Hash: RIPEMD160 hi all. i've cyrus imap v2.2.12 on osx 10.4.3. in my error logs, i'm seeing errors like: devbox lmtp[11095]: Unknown Error Code: -### where these error_codes are defined in: ./imap/imap_err.strings as, generally : KEManager -### = imap; KEMessage -### = description of error; when correctly mapped, i'd expected to see the more-decriptive text in error-logs, e.g.: devbox lmtp[11095]: Error : description of error or some such ... fwiw, i've posted a bug ~ a month ago @: https://bugzilla.andrew.cmu.edu/show_bug.cgi?id=2721 suggestions? thx! richard - -- /\ \ / ASCII Ribbon Campaign X against HTML email, vCards / \ micro$oft attachments [GPG] OpenMacNews at gmail dot com fingerprint: 780A 5C81 D446 C616 B113 AA3A 9BF4 3736 88A5 678E -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.2 (Darwin) iEYEAREDAAYFAkNwNVkACgkQm/Q3NoilZ46tWQCfV9XiRwMwyHQOWGozStrLgHmk 25EAn2xNVKmyqMoJh9TAz9qcHLRv5gyT =bcys -END PGP SIGNATURE- Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
