Re: Cyrus, Solaris 10, ZFS? (and NIS?)
On Oct 5, 2006, at 10:50 PM, Elizabeth Schwartz wrote: On 10/5/06, Igor Brezac <[EMAIL PROTECTED]> wrote: Already done. man imapd.conf :) unix_group_enable: 0 Cool :) I was looking at an older cyrus distribution that doesn't seem to have it... -rob Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Cyrus, Solaris 10, ZFS? (and NIS?)
On 10/5/06, Igor Brezac <[EMAIL PROTECTED]> wrote: Already done. man imapd.conf :)unix_group_enable: 0Thanks!!! Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
RE: Cyrus, Solaris 10, ZFS? (and NIS?)
> -Original Message- > From: [EMAIL PROTECTED] [mailto:info-cyrus- > [EMAIL PROTECTED] On Behalf Of Michael Loftis > Sent: Thursday, October 05, 2006 5:37 PM > To: Chaskiel M Grundman; [EMAIL PROTECTED] > Cc: info-cyrus@lists.andrew.cmu.edu > Subject: Re: Cyrus, Solaris 10, ZFS? (and NIS?) > > > > --On October 5, 2006 4:46:54 PM -0400 Chaskiel M Grundman > <[EMAIL PROTECTED]> wrote: > > > > > > > > mynewstate is taking 8s to run, and very little of the time is taken up > > in local subroutines. > > auth_unix.c:mynewstate calls getpwnam, and then iterates over all the > > groups using getgrent(), > > checking to see what groups the user is in. The fact that imapd does > this > > twice might be a bug, but even if it didn't do it twice, it would still > > be slow. > > > > Is running "getent group" slow? > > We had to patch this out of our Cyrus frontends using LDAP as well because > it iterates instead of retrieves. We just decided not to support groups > in > the ACL's. > > I'd suspect this is exactly whats going on is this code is still there in > latest Cyrus and it's building the ACL representation. If you don't care > about groups you can find, and remove, that code as we did. Cyrus already deals with this deficiency; unix_group_enable: 0 (not really cyrus fault). There is at least one other and more effective way to implement group ACLs especially if you use LDAP via pts. See man imapd.conf (unfortunately not much more documentation than that) -Igor Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Cyrus, Solaris 10, ZFS? (and NIS?)
On Oct 5, 2006, at 10:05 PM, Kjetil Torgrim Homme wrote: On Thu, 2006-10-05 at 16:46 -0400, Chaskiel M Grundman wrote: mynewstate is taking 8s to run, and very little of the time is taken up in local subroutines. auth_unix.c:mynewstate calls getpwnam, and then iterates over all the groups using getgrent(), checking to see what groups the user is in. The fact that imapd does this twice might be a bug, but even if it didn't do it twice, it would still be slow. to speed up initgroups, make sure you have the netid.byname NIS map. if you're not familiar with it, the keys should be "[EMAIL PROTECTED]", the values are "UID:GID,GID,GID,...". an example for my account, which is uid 1232 in domain "ifi": key: "[EMAIL PROTECTED]" value: "1232:0,6,15,7411,11232" presto, no iteration through the complete group map needed! Unfortunately, in the cyrus implementation (they don't call the "real" initgroups() specifically), they're iterating through the group map entry by entry... The (somewhat mysterious) netid map isn't going to help :( -rob Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
RE: Cyrus, Solaris 10, ZFS? (and NIS?)
> -Original Message- > From: [EMAIL PROTECTED] [mailto:info-cyrus- > [EMAIL PROTECTED] On Behalf Of Robert Banz > Sent: Thursday, October 05, 2006 7:34 PM > To: Chaskiel M Grundman > Cc: [EMAIL PROTECTED]; info-cyrus@lists.andrew.cmu.edu > Subject: Re: Cyrus, Solaris 10, ZFS? (and NIS?) > > > On Oct 5, 2006, at 4:46 PM, Chaskiel M Grundman wrote: > > > > > > > --On Thursday, October 05, 2006 04:13:18 PM -0400 Elizabeth > > Schwartz <[EMAIL PROTECTED]> wrote: > > > >> http://www.gsd.harvard.edu/users/betsys/dapptrace.timed > > > > The interesting bit seems to be here: > > . . -> mynewstate(0x165769, 0x40404040, 0x0) > > . . -> mycanonifyid(0x165769, 0x0, 0x0) > > . . -> libcyrus_config_getswitch(0x2, 0x11AF55, 0x5) > > 43 5 <- libcyrus_config_getswitch = 84 > >135 20 <- mycanonifyid = 292 > > . . -> xmalloc(0x5C, 0x11AF5D, 0x5) > > 38 14 <- xmalloc = 28 > > . . -> libcyrus_config_getswitch(0x1, 0x0, 0x1647CB) > > 40 3 <- libcyrus_config_getswitch = 84 > > . . -> xrealloc(0x0, 0x4, 0xE488) > > 68 14 <- xrealloc = 64 > > . . -> xstrdup(0x16871C, 0x4, 0xE488) > > . . -> xmalloc(0x9, 0x2A0031, 0x168724) > > 30 7 <- xmalloc = 28 > > 94 16 <- xstrdup = 40 > > 8235260 109820 <- mynewstate = 356 > > > > mynewstate is taking 8s to run, and very little of the time is > > taken up in local subroutines. > > auth_unix.c:mynewstate calls getpwnam, and then iterates over all > > the groups using getgrent(), > > checking to see what groups the user is in. The fact that imapd > > does this twice might be a bug, but even if it didn't do it twice, > > it would still be slow. > > > > Is running "getent group" slow? > > Oy yes. > > The only "way" to find out what groups a user is in, of course, is to > interate over the groups file (or map) and look at the whole list of > users assigned to a group. Ugly. > > Now, if you can't think of any reasons you'd actually care about > someone's group membership, it wouldn't be out of the question to > remove said junk out of the auth_state function in auth_unix.c. > Sendmail contains a nice option to turn off initgroups() like > functionality, perhaps Cyrus could use one as well? Already done. man imapd.conf :) unix_group_enable: 0 -Igor Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Cyrus, Solaris 10, ZFS? (and NIS?)
On Thu, 2006-10-05 at 16:46 -0400, Chaskiel M Grundman wrote: > mynewstate is taking 8s to run, and very little of the time is taken up in > local subroutines. > auth_unix.c:mynewstate calls getpwnam, and then iterates over all the > groups using getgrent(), > checking to see what groups the user is in. The fact that imapd does this > twice might be a bug, but even if it didn't do it twice, it would still be > slow. to speed up initgroups, make sure you have the netid.byname NIS map. if you're not familiar with it, the keys should be "[EMAIL PROTECTED]", the values are "UID:GID,GID,GID,...". an example for my account, which is uid 1232 in domain "ifi": key: "[EMAIL PROTECTED]" value: "1232:0,6,15,7411,11232" presto, no iteration through the complete group map needed! -- Kjetil T. Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Cyrus, Solaris 10, ZFS? (and NIS?)
Coupla random details:I got imap running on a nonstandard port by adding this to the cyrus conf file:imaptest cmd="imapd -T 59 " listen= prefork=1The -T 59 is so that it is easy to find in the process list (thanks stpierre!!) Making the server an nis slave server, vs a client of another server, didn't seem to affect performance, except that I saw more paging when it was a slave server. I turned off the slave server stuff after a bit because it seemed like it was just adding to the load. Although I removed most of the nis entries from nsswitch.conf, I still had "groups files nis" . Oops. Have now removed it. This particular server is a cyrus black box so I just need to make sure that various system processes have access to their stuff; cyrus runs everything as user cyrus. So, um, if group stuff is so slow, why is this becoming such an explosive problem now? Is this a new addition in recent versions of cyrus? I will also check to see if there's anything funky with my NIS group file. We generate a bunch of the NIS map files using our GUI user management software and I've found oddball bugs before. (I've got group names with caps, group names with dashes and underscores and group names with more than 8 characters, but it's been like this for a long time) Right now performance is great; I have to wait until mid-day tomorrow to see what's up. continuing thanks for all the help. Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Cyrus, Solaris 10, ZFS? (and NIS?)
On Oct 5, 2006, at 4:46 PM, Chaskiel M Grundman wrote: --On Thursday, October 05, 2006 04:13:18 PM -0400 Elizabeth Schwartz <[EMAIL PROTECTED]> wrote: http://www.gsd.harvard.edu/users/betsys/dapptrace.timed The interesting bit seems to be here: . . -> mynewstate(0x165769, 0x40404040, 0x0) . . -> mycanonifyid(0x165769, 0x0, 0x0) . . -> libcyrus_config_getswitch(0x2, 0x11AF55, 0x5) 43 5 <- libcyrus_config_getswitch = 84 135 20 <- mycanonifyid = 292 . . -> xmalloc(0x5C, 0x11AF5D, 0x5) 38 14 <- xmalloc = 28 . . -> libcyrus_config_getswitch(0x1, 0x0, 0x1647CB) 40 3 <- libcyrus_config_getswitch = 84 . . -> xrealloc(0x0, 0x4, 0xE488) 68 14 <- xrealloc = 64 . . -> xstrdup(0x16871C, 0x4, 0xE488) . . -> xmalloc(0x9, 0x2A0031, 0x168724) 30 7 <- xmalloc = 28 94 16 <- xstrdup = 40 8235260 109820 <- mynewstate = 356 mynewstate is taking 8s to run, and very little of the time is taken up in local subroutines. auth_unix.c:mynewstate calls getpwnam, and then iterates over all the groups using getgrent(), checking to see what groups the user is in. The fact that imapd does this twice might be a bug, but even if it didn't do it twice, it would still be slow. Is running "getent group" slow? Oy yes. The only "way" to find out what groups a user is in, of course, is to interate over the groups file (or map) and look at the whole list of users assigned to a group. Ugly. Now, if you can't think of any reasons you'd actually care about someone's group membership, it wouldn't be out of the question to remove said junk out of the auth_state function in auth_unix.c. Sendmail contains a nice option to turn off initgroups() like functionality, perhaps Cyrus could use one as well? -rob Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Cyrus, Solaris 10, ZFS? (and NIS?)
--On October 5, 2006 4:46:54 PM -0400 Chaskiel M Grundman <[EMAIL PROTECTED]> wrote: mynewstate is taking 8s to run, and very little of the time is taken up in local subroutines. auth_unix.c:mynewstate calls getpwnam, and then iterates over all the groups using getgrent(), checking to see what groups the user is in. The fact that imapd does this twice might be a bug, but even if it didn't do it twice, it would still be slow. Is running "getent group" slow? We had to patch this out of our Cyrus frontends using LDAP as well because it iterates instead of retrieves. We just decided not to support groups in the ACL's. I'd suspect this is exactly whats going on is this code is still there in latest Cyrus and it's building the ACL representation. If you don't care about groups you can find, and remove, that code as we did. Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Cyrus, Solaris 10, ZFS? (and NIS?)
--On Thursday, October 05, 2006 04:13:18 PM -0400 Elizabeth Schwartz <[EMAIL PROTECTED]> wrote: http://www.gsd.harvard.edu/users/betsys/dapptrace.timed The interesting bit seems to be here: . . -> mynewstate(0x165769, 0x40404040, 0x0) . . -> mycanonifyid(0x165769, 0x0, 0x0) . . -> libcyrus_config_getswitch(0x2, 0x11AF55, 0x5) 43 5 <- libcyrus_config_getswitch = 84 135 20 <- mycanonifyid = 292 . . -> xmalloc(0x5C, 0x11AF5D, 0x5) 38 14 <- xmalloc = 28 . . -> libcyrus_config_getswitch(0x1, 0x0, 0x1647CB) 40 3 <- libcyrus_config_getswitch = 84 . . -> xrealloc(0x0, 0x4, 0xE488) 68 14 <- xrealloc = 64 . . -> xstrdup(0x16871C, 0x4, 0xE488) . . -> xmalloc(0x9, 0x2A0031, 0x168724) 30 7 <- xmalloc = 28 94 16 <- xstrdup = 40 8235260 109820 <- mynewstate = 356 mynewstate is taking 8s to run, and very little of the time is taken up in local subroutines. auth_unix.c:mynewstate calls getpwnam, and then iterates over all the groups using getgrent(), checking to see what groups the user is in. The fact that imapd does this twice might be a bug, but even if it didn't do it twice, it would still be slow. Is running "getent group" slow? Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Cyrus, Solaris 10, ZFS? (and NIS?)
On Oct 5, 2006, at 1:59 PM, Elizabeth Schwartz wrote: >There's a bug in ZFS regarding performance problems when fsync'ing >file descriptors -- there's apparently going to be a patch coming >"real soon now" -- your options are: Thanks! Ugh, that would be bad news. Except, I think the delay is happening earlier. The delay is between the A001 login and A001 ok, and the only zfs file system is the mail spool (I should have spelled that out) Have you verified the contents of /etc/nsswitch.conf ? Have you tried pointing its ypservers at localhost (since you say this server is also a NIS slave)? Also check the contents of /etc/nscd.conf on your old server and compare it to the new. I've been playing with the dtrace toolkit this afternoon - still a bit stuck on how to get from my tcp port connection to the process number to run dtrace dtruss (lsof *should* be the right tool but I'm missing a step somewhere) Do this: To simplify, telnet to your imap server's imap port from a machine that otherwise has no connections to that server. On the imap server, and assuming that your imapd processes run under a certain username such as "cyrus", run the following command, replacing the -u argument in pgrep with your imapd user and the IP address in the egrep portion with the IP address you are telnetting from: pfiles `pgrep -u cyrus imapd` | egrep '(imapd|xxx.xxx.xxx.xxx)' You will then get output similar to the following: 25723: imapd 26910: imapd 25084: imapd 27962: imapd peername: AF_INET xxx.xxx.xxx.xxx port: 50630 peername: AF_INET xxx.xxx.xxx.xxx port: 50630 peername: AF_INET xxx.xxx.xxx.xxx port: 50630 8772: imapd With the above, imapd PID 27962 is your process to debug your problem with. Run 'pldd | grep libsasl2' and not the full name of the libsasl2 library your imapd uses, such as "libsasl2.so.2.0.21" sans the preceding path (ie, leave off /usr/local/lib) Download the Dtrace Toolkit (google for it) and run the dappprof program. Now run it against the imapd PID and only key in on the calls made by the SASL library: dappprof -u libsasl2.so.2.0.21 -F -p Over in your imapd telnet window, issue a login command (A001 LOGIN foo bar). After the login returns, hit Ctrl-C in dappprof and look at the times (which are in microseconds, so move the decimal place as you see fit to better understand the times). Look for which function in SASL takes the longest to return. /dale -- Dale Ghent UNIX Systems Specialist UMBC - Office of Information Technology ECS 201 - x51705 Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Cyrus, Solaris 10, ZFS? (and NIS?)
Fascinating stuff. Here is a timed dapptrace of a fairly slow response( I think it was at least 10 seconds from A001 login to the response)http://www.gsd.harvard.edu/users/betsys/dapptrace.timed I am thinking something is off with sasl? I really haven't done much with sasl, here's the config file# You must specify the authentication mechanisms you wish to use.# This defaults to "pam" for PAM support, but may also include # "shadow", "sasldb", "kerberos5", etc.# See saslauthd(8) for more mechanisms.MECHANISM=shadow# Location of config file for mechanism# See /opt/csw/share/docs/saslauthd/LDAP_SASLAUTHD for ldap sample # This parameter is optional.# CONFIG_FILE="/opt/csw/etc/saslauthd.conf"# Any optional additional parameters for saslauthd# e.g. -c -n # See saslauthd(8) for the description of parameters # This parameter is optional# PARAMS="-n 5 -c" Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Failing to authenticate on the frontends
--On October 5, 2006 11:42:36 AM +0100 Jesus Roncero <[EMAIL PROTECTED]> wrote: Umm, I got it to work using DIGEST-MD5, but apparently, all communications are encrypted after the authentication. Is there a way in which all the communications between the frontends and the backends are *not* encrypted, except, probably, the authentication dialog? I guess that's what CRAM-MD5 is for, but the frontend refuses to talk to the backend if it is presented with CRAM-MD5 only. Is there any way to do this or I am doing something really wrong? :) See earlier in this thread. It's not at all possible in stock Cyrus. You have to patch it to allow that. I've got one for older versions of cyrus, 2.1.17 ish, but they'll need cleanup. Thanks to Henrique de Moraes Holscuh who provided me with them. 1813.patch Description: application/text Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Failing to authenticate on the frontends
--On October 5, 2006 10:30:55 AM +0100 Jesus Roncero <[EMAIL PROTECTED]> wrote: Umm, isn't there an option on the configuration to disable referrals? If not, do you have that patch available? Not in 2.0.x for sure. Maybe in newer releases. We're running a 'very old' release of Cyrus here. Also, one question on the communication between the frontends and backends. I made them speak using TLS and plain, but would like to use CRAM-MD5 or DIGEST-MD5 and no TLS at all. Is that possible? Because when I disable TLS and force it to use the MD5 thing, the frontend complains that there are no mechs available. -- Jesus Roncero <[EMAIL PROTECTED]> System Developer Tel: +44 (0) 845 666 7778 http://www.mxtelecom.com -- "Genius might be described as a supreme capacity for getting its possessors into trouble of all kinds." -- Samuel Butler Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Connection to server timed out & Cannot copy to Sent folder
I've recently switched all of my servers to Cyrus IMAP, and have encountered a problem that didn't appear during testing on my home network. When using Thunderbird, it will periodically hang when accessing a folder, then a dialog will be displayed with this message: Connection to server mail.example.com timed out The strange thing is that while it hangs on one folder, I can access other folders in the same account. I am also able to access other accounts on the same server, but the behaviour is inconsistent. It certainly doesn't seem to be related to DNS or any other network service, and it only seems to happen with Thunderbird. I see no strange messages in either the IMAP or debug logs. Other users have reported that they periodically get an error from Thunderbird saying that cannot save a copy to the Sent folder after sending a message. After searching the web for a solution, I found that some people have had success with reducing the cached connections to 1, but this hasn't helped my case at all. My attempts to troubleshoot the problem by tailing the logs or capturing packets have been fruitless, partly because I can not reproduce the problem at will. I'm getting quite frustrated, and will appreciate any suggestions, especially if there is server configuration that would fix the problem. Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Cyrus, Solaris 10, ZFS? (and NIS?)
On Oct 5, 2006, at 13:59, Elizabeth Schwartz wrote: >There's a bug in ZFS regarding performance problems when fsync'ing >file descriptors -- there's apparently going to be a patch coming >"real soon now" -- your options are: Thanks! Ugh, that would be bad news. Except, I think the delay is happening earlier. The delay is between the A001 login and A001 ok, and the only zfs file system is the mail spool (I should have spelled that out) I've been playing with the dtrace toolkit this afternoon - still a bit stuck on how to get from my tcp port connection to the process number to run dtrace dtruss (lsof *should* be the right tool but I'm missing a step somewhere) "topsyscall" is also really a good way to see what fun things are going on with the machine. Damn I'm in love with dtrace ;) Once you find out what your PID is after you connect, connect 'dapptrace' to it. It'll give you a run down of all of the function calls* that are going on (as they're going on), so you'll really get an idea of what's going on. *yes, function calls. Not just syscalls. Don't ask how it works -- it's magic. -rob Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Cyrus, Solaris 10, ZFS? (and NIS?)
I'd also appreciate any hints for debugging IMAP. Is there a way to get cyrus to open *one* connection on an offbeat port for me to play with? I am not clear on how I can truss an entire interaction when there are so many processes. By the time I identify it I've missed the beginning. from a computer A do: $ telnet server_imap 43 Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. * OK [CAPABILITY IMAP4 IMAP4rev1 LITERAL+ ID AUTH=PLAIN AUTH=LOGIN SASL-IR] moldavite Cyrus IMAP4 v2.3.7 server ready At this point, log in the imap server as root and do: $ lsof -n -i [EMAIL PROTECTED] COMMAND PID USER FD TYPEDEVICE SIZE/OFF NODE NAME imapd 23017 cyrus0u IPv4 0x300020360580t119 TCP ip_server_imap:imap->ip_computer_A:56040 (ESTABLISHED) imapd 23017 cyrus1u IPv4 0x300020360580t119 TCP ip_server:imap->ip_computer_A:56040 (ESTABLISHED) imapd 23017 cyrus2u IPv4 0x300020360580t119 TCP ip_server:imap->ip_computer_A:56040 (ESTABLISHED) Now you can do: truss -p 23017 Go back to the client and manually type the login command: 1 LOGIN "username" "password" Now you should have the truss output of the login. Yann Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Cyrus, Solaris 10, ZFS? (and NIS?)
>There's a bug in ZFS regarding performance problems when fsync'ing>file descriptors -- there's apparently going to be a patch coming>"real soon now" -- your options are:Thanks! Ugh, that would be bad news. Except, I think the delay is happening earlier. The delay is between the A001 login and A001 ok, and the only zfs file system is the mail spool (I should have spelled that out) I've been playing with the dtrace toolkit this afternoon - still a bit stuck on how to get from my tcp port connection to the process number to run dtrace dtruss (lsof *should* be the right tool but I'm missing a step somewhere) Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Failing to authenticate on the frontends
On Thu, 5 Oct 2006, Jesus Roncero wrote:
Michael Loftis wrote:
Authentication of the user happens at the frontend. The frontend then
uses the proxy credentials to authorize as the user on the backend.
The backends don't need a full user database, just the proxy
information.
Yes and no. If an IMAP client support referrals, the frontends will
return a referral to the appropriate backend. So, the client may connect
to the backend as well in some cases.
Oops, I forgot about that detail. We locally patched referrals out of our
IMAP proxies.
Umm, isn't there an option on the configuration to disable referrals? If not,
do you have that patch available?
I've attached the patch to this message. We use it with Cyrus v2.2.12
here. I don't know if it works or applies cleanly to v2.3.x.
Andy--- cyrus-imapd-2.2.10.dist/imap/proxyd.c 2004-11-23 09:40:15.0
-0800
+++ cyrus-imapd-2.2.10/imap/proxyd.c2005-01-14 12:50:11.965210408 -0800
@@ -1227,7 +1227,7 @@
/* Cleanup Globals */
proxyd_cmdcnt = 0;
-disable_referrals = 0;
+disable_referrals =
config_getswitch(IMAPOPT_PROXYD_DISABLE_MAILBOX_REFERRALS);
supports_referrals = 0;
proxyd_userisadmin = 0;
proxyd_starttls_done = 0;
@@ -2823,6 +2823,10 @@
prot_printf(proxyd_out, "* CAPABILITY ");
prot_printf(proxyd_out, CAPABILITY_STRING);
+if (config_getswitch(IMAPOPT_PROXYD_DISABLE_MAILBOX_REFERRALS) == 0) {
+ prot_printf(proxyd_out, " MAILBOX-REFERRALS");
+}
+
if (config_getint(IMAPOPT_IMAPIDLEPOLL) > 0) {
prot_printf(proxyd_out, " IDLE");
}
diff -r -u cyrus-imapd-2.2.10.dist/imap/version.h
cyrus-imapd-2.2.10/imap/version.h
--- cyrus-imapd-2.2.10.dist/imap/version.h 2004-11-23 09:52:52.0
-0800
+++ cyrus-imapd-2.2.10/imap/version.h 2005-01-14 12:46:38.391024296 -0800
@@ -55,7 +55,7 @@
/* CAPABILITIES are now defined here, not including sasl ones */
#define CAPABILITY_STRING "IMAP4 IMAP4rev1 ACL QUOTA LITERAL+ " \
- "MAILBOX-REFERRALS NAMESPACE UIDPLUS ID " \
+ "NAMESPACE UIDPLUS ID " \
"NO_ATOMIC_RENAME UNSELECT " \
"CHILDREN MULTIAPPEND BINARY " \
"SORT THREAD=ORDEREDSUBJECT THREAD=REFERENCES " \
diff -r -u cyrus-imapd-2.2.10.dist/lib/imapoptions
cyrus-imapd-2.2.10/lib/imapoptions
--- cyrus-imapd-2.2.10.dist/lib/imapoptions 2004-07-21 12:07:45.0
-0700
+++ cyrus-imapd-2.2.10/lib/imapoptions 2005-01-14 12:47:41.125560426 -0800
@@ -630,6 +630,10 @@
connections that these referrals would cause, thus resulting in a higher
authentication load on the respective backend server. */
+{ "proxyd_disable_mailbox_referrals", 0, SWITCH }
+/* Set to true to disable the use of mailbox-referrals on the
+ proxy servers.*/
+
{ "proxyservers", NULL, STRING }
/* A list of users and groups that are allowed to proxy for other
users, seperated by spaces. Any user listed in this will be
Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: *any* user's folder structure
On Thu, 5 Oct 2006, Kevin Kruzich wrote: With the imapd.conf as show below in use I can see *any* user's folder structure (cannot see the contents) when I do subscribe - query (showing those that I'm not explicity subscribed to). This is with Thunderbird or Outlook. I see a "user" top level folder, all users below that, and have the ability to expand any/everything from there. Yeah, I thought foolstupidclients would work but it did not. This may be just bad ACL assignment (which are assigned by default), here's mine and a few other random ones: kkruzich lrswipcda magosto lrswipcda dcollins lrswipcda rcotto lrswipcda Another behavior, most likely related to this issue, is I can *create* folders outside of my own tree --where they appear as /imap/THISFOLDER but I cannot delete these. I don't want to be able to do this. Any comments would be greatly appreciated. It sounds like you are connecting as a user listed in the "admins:" setting in imapd.conf. Andy Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Cyrus, Solaris 10, ZFS? (and NIS?)
On Oct 5, 2006, at 12:40, Elizabeth Schwartz wrote: Is anyone happily running all of the above? All of the above except NIS? Any tuning hints? I'm running Solaris 10 (06/06), cyrus 2.3.7 (Blastwave build) , sendmail 8.13.8 (ditto), mailspool on a zfs filesystem, authenticating via NIS. I've already solved one problem with VERY slow sendmail response, turned out to be a Solaris NIS bug ( patch 123186-01) Run procsystime (from the dtrace toolkit) on the cyrus imap processes. I'm going to bet that they do a lot of fdsync's. There's a bug in ZFS regarding performance problems when fsync'ing file descriptors -- there's apparently going to be a patch coming "real soon now" -- your options are: 1) Move your mail spool off of ZFS. 2) Remove all of the fsync() calls from cyrus. (this may mean removing them from berkely db ;) ) -rob Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
*any* user's folder structure
With the imapd.conf as show below in use I can see *any* user's folder structure (cannot see the contents) when I do subscribe - query (showing those that I'm not explicity subscribed to). This is with Thunderbird or Outlook. I see a "user" top level folder, all users below that, and have the ability to expand any/everything from there. Yeah, I thought foolstupidclients would work but it did not. This may be just bad ACL assignment (which are assigned by default), here's mine and a few other random ones: kkruzich lrswipcda magosto lrswipcda dcollins lrswipcda rcotto lrswipcda Another behavior, most likely related to this issue, is I can *create* folders outside of my own tree --where they appear as /imap/THISFOLDER but I cannot delete these. I don't want to be able to do this. Any comments would be greatly appreciated. --- imapd.conf --- configdirectory: /var/lib/imap foolstupidclients: true partition-default: /imap3 partition-imap2: /imap2 partition-imap3: /imap3 partition-imap: /imap sasl_mech_list: PLAIN sasl_pwcheck_method: auxprop seenstate_db: flat sendmail: /usr/sbin/sendmail sievedir: /var/lib/imap/sieve --- -- Kevin Kruzich UNIX Systems Administrator Linkshare Corporation Tel 646-654-6000 x344 Fax 646-602-0160 [EMAIL PROTECTED] Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Sieve DBERROR with avelsieve
On Thu, October 5, 2006 8:49 am, Mike Husmann wrote: > Hey all, > I just upgraded the avelsieve plugin in my Squirrelmail site install to > 1.9.7 and > have suddenly started receiving errors. Every time I make a modification > using > the sieve plugin, I get the following error: > > Oct 5 08:24:38 rusty sieve[18879]: DBERROR: error exiting application: > Invalid > argument I found one additional error coming from sieve that corresponds with the above error: Oct 5 11:57:15 rusty sieve[29932]: DBERROR �^H^H: db4 When using sivtest, I get the same error every time I execute LOGOUT, but only when using TLS... sivtest -a bebo -u bebo -t "" imap.morningside.edu this yields the error sivtest -a bebo -u bebo imap.morningside.edu this does not I am using tls on cyrus 2.2.12.. Something wrong in my config? Possible bug? Thanks in advance. Mike Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Cyrus, Solaris 10, ZFS? (and NIS?)
Is anyone happily running all of the above? All of the above except NIS? Any tuning hints?I'm running Solaris 10 (06/06), cyrus 2.3.7 (Blastwave build) , sendmail 8.13.8 (ditto), mailspool on a zfs filesystem, authenticating via NIS. I've already solved one problem with VERY slow sendmail response, turned out to be a Solaris NIS bug ( patch 123186-01)Current problem is that IMAP response is intermittently CRAWLING. http://enki.gsd.harvard.edu/cgi-bin/larrd-grapher.cgi?host=hathor.gsd.harvard.edu&service=imap2&graph=dailyWhen it gets slow I also see paging, high number of processes, no particular errors. The delay comes AFTER the initial response from the server, between the A001 login user passwd and the A001 OK response. So it still could be NIS or it could be some kinda system resource issue. I've tried making the local server an NIS slave (just seems to increase paging and thrashing without changing response time), turning off NIS for everything except netgroups (working on those ) I know NIS is suboptimal and I aim to get rid of NIS completely but I wasn't planning to do so this weekend (all of our web-based user management tools are entangled with NIS) The server is a dual-processor Sun480R; considering that we were running with no load issues with Solaris 8 on a 220R I think this machine should be adequate for our 1000-odd users. I'd also appreciate any hints for debugging IMAP. Is there a way to get cyrus to open *one* connection on an offbeat port for me to play with? I am not clear on how I can truss an entire interaction when there are so many processes. By the time I identify it I've missed the beginning. thanks for any cluesBetsy Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Sieve DBERROR with avelsieve
Hey all,
I just upgraded the avelsieve plugin in my Squirrelmail site install to 1.9.7
and
have suddenly started receiving errors. Every time I make a modification using
the sieve plugin, I get the following error:
Oct 5 08:24:38 rusty sieve[18879]: DBERROR: error exiting application: Invalid
argument
But for that same transaction, there is no mention of any error in the debug
log:
Oct 5 08:24:37 rusty sieve[18879]: executed
Oct 5 08:24:37 rusty sieve[18879]: accepted connection
Oct 5 08:24:37 rusty sieve[18879]: mystore: starting txn 2147484734
Oct 5 08:24:37 rusty sieve[18879]: mystore: committing txn 2147484734
Oct 5 08:24:37 rusty sieve[18879]: entered bc_action_emit with filelen: 16
Oct 5 08:24:37 rusty sieve[18879]: entered bc_action_emit with filelen: 96
Oct 5 08:24:37 rusty sieve[18879]: entered bc_action_emit with filelen: 192
Oct 5 08:24:37 rusty sieve[18879]: entered bc_action_emit with filelen: 300
Oct 5 08:24:37 rusty sieve[18879]: entered bc_action_emit with filelen: 528
Oct 5 08:24:37 rusty sieve[18879]: entered bc_action_emit with filelen: 644
Oct 5 08:24:37 rusty sieve[18879]: entered bc_action_emit with filelen: 748
Oct 5 08:24:37 rusty sieve[18879]: entered bc_action_emit with filelen: 860
Oct 5 08:24:37 rusty sieve[18879]: entered bc_action_emit with filelen: 988
Oct 5 08:24:37 rusty sieve[18879]: entered bc_action_emit with filelen: 1108
Oct 5 08:24:37 rusty sieve[18879]: entered bc_action_emit with filelen: 1232
Oct 5 08:24:37 rusty sieve[18879]: entered bc_action_emit with filelen: 1404
Oct 5 08:24:38 rusty sieve[18879]: entered bc_action_emit with filelen: 1508
Oct 5 08:24:38 rusty master[18370]: process 18879 exited, status 0
I have passed this on as a probable bug in the plugin, but I wanted to see if
there
was anything I can do about it (or if I had anything wrong) as well.
imapd.conf:
# server conf
servername: rusty.morningside.edu
umask: 077
reject8bit: no
quotawarn: 90
timeout: 30
poptimeout: 10
# singleinstancestore: yes
username_to_lower: yes
lmtp_downcase_rcpt: yes
lmtp_over_quota_perm_failure: yes
altnamespace: yes
# user conf
postmaster: postmaster
admins: cyrus cyrusadm
# directory and file locations
configdirectory: /var/spool/cyrus-imap
partition-default: /var/spool/cyrus-imap
sievedir: /var/spool/cyrus-imap/sieve
sievenotifier: mailto
sendmail: /usr/sbin/sendmail
# authentication
allowanonymouslogin: no
allowplaintext: yes
sasl_mech_list: plain
sasl_minimum_layer: 0
sasl_pwcheck_method: saslauthd
# new user automated creates
autocreate_sieve_script: /var/spool/cyrus-imap/sieve/phpscript
autocreate_sieve_compiledscript: /var/spool/cyrus-imap/sieve/phpscript.bc
generate_compiled_sieve_script: yes
sieve_maxscriptsize: 64
sieve_maxscripts: 50
autocreateinboxfolders: Spam
autosubscribeinboxfolders: Spam
autocreatequota: 51200
createonpost: yes
# security certificate information
tls_cert_file: /etc/ssl/certs/imap.morningside.edu.crt
tls_key_file: /etc/ssl/certs/imap.morningside.edu.key
tls_ca_file: /etc/ssl/certs/imap.morningside.edu.ca-bundle
cyrus.conf:
# standard standalone server implementation
START {
# do not delete this entry!
recover cmd="ctl_cyrusdb -r"
# this is only necessary if using idled for IMAP IDLE
# idledcmd="idled"
}
# UNIX sockets start with a slash and are put into /var/spool/cyrus-imap/socket
SERVICES {
# add or remove based on preferences
imap cmd="imapd" listen="imap" prefork=0
imaps cmd="imapd -s" listen="imaps" prefork=0
pop3 cmd="pop3d" listen="pop3" prefork=0
pop3s cmd="pop3d -s" listen="pop3s" prefork=0
sieve cmd="timsieved" listen="sieve" prefork=0
# at least one LMTP is required for delivery
# lmtp cmd="lmtpd" listen="lmtp" prefork=0
lmtpunix cmd="lmtpd" listen="/var/spool/cyrus-imap/socket/lmtp" prefork=0
# this is only necessary if using notifications
notifycmd="notifyd" listen="/var/spool/cyrus-imap/socket/notify"
proto="udp" prefork=1
}
EVENTS {
# this is required
checkpointcmd="ctl_cyrusdb -c" period=15
# this is only necessary if using duplicate delivery suppression
delprune cmd="ctl_deliver -E 3" at=0400
# this is only necessary if caching TLS sessions
tlsprune cmd="tls_prune" at=0400
Thanks in advance,
Mike
Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Failing to authenticate on the frontends
Jesus Roncero wrote: Oops, I forgot about that detail. We locally patched referrals out of our IMAP proxies. Umm, isn't there an option on the configuration to disable referrals? If not, do you have that patch available? Also, one question on the communication between the frontends and backends. I made them speak using TLS and plain, but would like to use CRAM-MD5 or DIGEST-MD5 and no TLS at all. Is that possible? Because when I disable TLS and force it to use the MD5 thing, the frontend complains that there are no mechs available. Umm, I got it to work using DIGEST-MD5, but apparently, all communications are encrypted after the authentication. Is there a way in which all the communications between the frontends and the backends are *not* encrypted, except, probably, the authentication dialog? I guess that's what CRAM-MD5 is for, but the frontend refuses to talk to the backend if it is presented with CRAM-MD5 only. Is there any way to do this or I am doing something really wrong? :) -- Jesus Roncero <[EMAIL PROTECTED]> System Developer Tel: +44 (0) 845 666 7778 http://www.mxtelecom.com Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Failing to authenticate on the frontends
Michael Loftis wrote: Authentication of the user happens at the frontend. The frontend then uses the proxy credentials to authorize as the user on the backend. The backends don't need a full user database, just the proxy information. Yes and no. If an IMAP client support referrals, the frontends will return a referral to the appropriate backend. So, the client may connect to the backend as well in some cases. Oops, I forgot about that detail. We locally patched referrals out of our IMAP proxies. Umm, isn't there an option on the configuration to disable referrals? If not, do you have that patch available? Also, one question on the communication between the frontends and backends. I made them speak using TLS and plain, but would like to use CRAM-MD5 or DIGEST-MD5 and no TLS at all. Is that possible? Because when I disable TLS and force it to use the MD5 thing, the frontend complains that there are no mechs available. -- Jesus Roncero <[EMAIL PROTECTED]> System Developer Tel: +44 (0) 845 666 7778 http://www.mxtelecom.com Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: master / slave replication (sync_server / sync_client)
... sorry. I forgott some additional infos: On the slave side I get in /var/log/debug.log Oct 5 11:18:22 acsvfbsd04 master[18794]: about to exec /usr/local/cyrus/bin/sync_server Oct 5 11:18:22 acsvfbsd04 syncserver[18794]: executed Oct 5 11:18:22 acsvfbsd04 syncserver[18794]: accepted connection Oct 5 11:18:22 acsvfbsd04 syncserver[18794]: cmdloop(): startup Oct 5 11:18:25 acsvfbsd04 syncserver[18794]: accepted connection Oct 5 11:18:25 acsvfbsd04 syncserver[18794]: cmdloop(): startup Oct 5 11:19:09 acsvfbsd04 syncserver[18794]: accepted connection Oct 5 11:19:09 acsvfbsd04 syncserver[18794]: cmdloop(): startup Oct 5 11:19:12 acsvfbsd04 syncserver[18794]: accepted connection Oct 5 11:19:12 acsvfbsd04 syncserver[18794]: cmdloop(): startup while I start sync_client -r -v. Regards, Martin 2006/10/5, Martin Schweizer <[EMAIL PROTECTED]>: Hello Andy Telnet to port 2005 works perfect. I also see traffic on the slave (destination mail server) with tcpdump. But on the master I get (sync_client -r -v): Can not connect to server 'xxx.xxx.xx', retrying in 15 seconds I also see no entry in /var/log/debug.log for sync_client. Any ideas? Regards, Martin 2006/10/4, Andy Fiddaman <[EMAIL PROTECTED]>: > > > On Wed, 4 Oct 2006, Martin Schweizer wrote: > > ; Hello Andy > ; > ; Thank you for the hint. I change it as you desribed. > ; > ; Now after waiting some time I see no replication on both site. Is the > ; firewall the problem (only open 2005/tcp)? Or what do I wrong again? > ; > ; Regards, > ; Martin > > Try telnet from the master to the slave on port 2005 - this checks that > syncserver is running ok (2005 is all you need through the firewall): > > # telnet slave 2005 > Trying xx.xxx.xxx.xxx... > Connected to slave > Escape character is '^]'. > * SASL CRAM-MD5 > * OK mailstore.x.xxx Cyrus sync server v2.3.7 > > If it isn't running (connection refused) then make sure you're logging > debug messages (local6.debug in syslog.conf) and look to see what's wrong > - make sure you put csync in /etc/services > > Try running sync_client manually on the master with verbose flag: > (as your Cyrus user) > > % sync_client -r -v > > Hopefully that will give a clue. > > Andy > > -- Martin Schweizer [EMAIL PROTECTED] Fax: +1 619 3300587 Tel.: +1 619 3300597 (VoIP) -- Martin Schweizer [EMAIL PROTECTED] Fax: +1 619 3300587 Tel.: +1 619 3300597 (VoIP) Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: master / slave replication (sync_server / sync_client)
Hello Andy Telnet to port 2005 works perfect. I also see traffic on the slave (destination mail server) with tcpdump. But on the master I get (sync_client -r -v): Can not connect to server 'xxx.xxx.xx', retrying in 15 seconds I also see no entry in /var/log/debug.log for sync_client. Any ideas? Regards, Martin 2006/10/4, Andy Fiddaman <[EMAIL PROTECTED]>: On Wed, 4 Oct 2006, Martin Schweizer wrote: ; Hello Andy ; ; Thank you for the hint. I change it as you desribed. ; ; Now after waiting some time I see no replication on both site. Is the ; firewall the problem (only open 2005/tcp)? Or what do I wrong again? ; ; Regards, ; Martin Try telnet from the master to the slave on port 2005 - this checks that syncserver is running ok (2005 is all you need through the firewall): # telnet slave 2005 Trying xx.xxx.xxx.xxx... Connected to slave Escape character is '^]'. * SASL CRAM-MD5 * OK mailstore.x.xxx Cyrus sync server v2.3.7 If it isn't running (connection refused) then make sure you're logging debug messages (local6.debug in syslog.conf) and look to see what's wrong - make sure you put csync in /etc/services Try running sync_client manually on the master with verbose flag: (as your Cyrus user) % sync_client -r -v Hopefully that will give a clue. Andy -- Martin Schweizer [EMAIL PROTECTED] Fax: +1 619 3300587 Tel.: +1 619 3300597 (VoIP) Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
