Re: FastMail.FM Patchset Updated
On Wed, 21 Nov 2007 07:40:21 +0100 (CET), "Simon Matter" <[EMAIL PROTECTED]> said: > Hi Bron, > > Did you consider this one > http://bugzilla.andrew.cmu.edu/show_bug.cgi?id=3006 in the patch above? > From a quick look it seems both patches conflict, is #3006 obsolete now? You're right, they probably do conflict and Ken has applied 3006. I should have checked on that. Hmm... Actually, my patch doesn't even fix #3006. It does sort of suck to be working from a stable past point rather than the latest CVS actually when I do these patches - though it's nice from a stability point of view. OK - Ken, would you like me to re-build this patch on top of current CVS, or would you like to do it yourself? My patch has the replication bugfixes which are still worth doing, but Simon's logic is actually correct for the test while mine isn't (though I renamed the function he's using...) Bron. -- Bron Gondwana [EMAIL PROTECTED] Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: LARGE single-system Cyrus installs?
On 20 Nov 07, at 1756, David Lang wrote: however a fsync on a journaled filesystem just means the data needs to be written to the journal, it doesn't mean that the journal needs to be flushed to disk. on ext3 if you have data=journaled then your data is in the journal as well and all that the system needs to do on a fsync is to write things to the journal (a nice sequential write), Assuming the journal is on a distinct device and the distinct device can take the load. It isn't on ZFS, although work is in progress. One of the many benefits of the sadly underrated Solaris Disksuite product was the metatrans devices, which at least permitted metadata updates to go to a distinct device. When the UFS logging code went into core Solaris (the ON integration) that facility was dropped, sadly. My Pillar NFS server does data logging to distinct disk groups, but mostly --- like such boxes tend to do --- relies on 12GB of RAM and a battery. A sequential write is only of benefit if the head is in the right place and the platter is at the right rotational position and the write is well-matched to the transfer rate of the spindle: if the spindle is doing large sequential writes while also servicing reads and writes elsewhere, or can't keep up with writing tracks flat out, the problems increase. for cyrus you should have the same sort of requirements that you would have for a database server, including the fact that without a battery-backed disk cache (or solid state drive) to handle your updates, you end up being throttled by your disk rotation rate (you can only do a single fsync write per rotation, and that good only if you don't have to seek), RAID 5/6 arrays are even worse, as almost all systems will require a read of the entire stripe before writing a single block (and it's parity block) back out, and since the stripe is frequently larger then the OS readahead, the OS throws much of the data away immediatly. if we can identify the files that are the bottlenecks it would be very interesting to see the result of puttng them on a solid-state drive. I've split the meta-data out into separate partitions. The meta data is stored in ZFS filesystems in a pool which is a RAID 0+1 4 disk group with SAS drives, the message data is coming out of the lowest QoS on my Pillar. A ten second fsstat on VM operations shows that by request (this measures filesystem activity, not the implied disk activity) it's the meta partitions taking the pounding (ten second sample): map addmap delmap getpag putpag pagio 0 0 0 45 0 0 /var/imap 11 11 11 17 0 0 /var/imap/meta-partition-1 290290290463 5 0 /var/imap/meta-partition-2 139139139183 3 0 /var/imap/meta-partition-3 66 66 66106 10 0 /var/imap/meta-partition-7 347347342454 16 0 /var/imap/meta-partition-8 57 57 57 65 5 0 /var/imap/meta-partition-9 4 4 8 4 0 0 /var/imap/partition-1 11 11 22 14 0 0 /var/imap/partition-2 1 1 2 1 0 0 /var/imap/partition-3 6 6 12 49 10 0 /var/imap/partition-7 15 15 28457 0 0 /var/imap/partition-8 1 1 2 2 0 0 /var/imap/partition-9 Similarly, by non-VM operation: new name name attr attr lookup rddir read read write write file remov chng get setops ops ops bytes ops bytes 0 0 0 2.26K 0 6.15K 0 0 045 1.22K / var/imap 0 0 0 356 0707 0 0 0 6 3.03K / var/imap/meta-partition-1 3 0 3 596 0902 0 6 135K90 305K / var/imap/meta-partition-2 0 0 0 621 0 1.08K 0 0 0 3 1.51K / var/imap/meta-partition-3 3 0 3 1.04K 0 1.70K 0 6 149K36 650K / var/imap/meta-partition-7 0 0 0 2.28K 0 4.24K 0 0 0 7 1.87K / var/imap/meta-partition-8 0 0 018 0 32 0 0 0 2 176 / var/imap/meta-partition-9 2 2 222 0 30 0 1 2.37K 2 7.13K / var/imap/partition-1 3 41284 0157 0 1 677 3 7.51K / var/imap/partition-2 1 1 1 1.27K 0 2.16K 0 0 0 1 3.75K / var/imap/partition-3 2 2 435 0 56 0 1 3.97K36 279K / var/imap/partition-7 1 2 1 256 0514 0 0 0 1 3.75K / var/imap/partition-8 0 0 0 0 0 0 0 0 0 0 0 / var/imap/partition-9 And looking at the real IO load, ten seconds of zpool (for the meta data and /var/imap_ capacity operationsbandwidth pool used avail read write rea
Re: FastMail.FM Patchset Updated
Bron Gondwana wrote: > On Wed, 21 Nov 2007 07:40:21 +0100 (CET), "Simon Matter" <[EMAIL PROTECTED]> > said: > >> Hi Bron, >> >> Did you consider this one >> http://bugzilla.andrew.cmu.edu/show_bug.cgi?id=3006 in the patch above? >> From a quick look it seems both patches conflict, is #3006 obsolete now? > > You're right, they probably do conflict and Ken has applied 3006. I should > have checked on that. Hmm... > > Actually, my patch doesn't even fix #3006. It does sort of suck to be working > from a stable past point rather than the latest CVS actually when I do these > patches - though it's nice from a stability point of view. > > OK - Ken, would you like me to re-build this patch on top of current CVS, or > would you like to do it yourself? My patch has the replication bugfixes which > are still worth doing, but Simon's logic is actually correct for the test > while mine isn't (though I renamed the function he's using...) If you have the time. I'm off banging on other things at the moment. -- Kenneth Murchison Systems Programmer Project Cyrus Developer/Maintainer Carnegie Mellon University Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: FastMail.FM Patchset Updated
Bron Gondwana wrote: > As usual you can get the patches here: > > http://cyrus.brong.fastmail.fm/ > > > I've been busy with Cyrus _again_ - so much for my theory > that I was taking a break. > > OK - here's what's new. > > * http://cyrus.brong.fastmail.fm/patches/cyrus-skiplist-bugfixes-2.3.10.diff > http://cyrus.brong.fastmail.fm/patches/cyrus-skiplist-robustify-2.3.10.diff > > Skiplist issues - there were two things that could be found > in recovery that actually bit us during the whole "restart > every single store with the new skiplist code" project the > other day. ADD where the record already existed and DELETE > where it didn't. The later also had an obnoxious bug where > it would instead delete _the_alphabetically_NEXT_record_ > silently. Ooops. > > I rolled these two into my bugfix and robustify patches, not > realising Ken had already applied the previous copies upstream. > Ken - do you want me to break this out as a separate patch on > top of the others? Yes, if you wouldn't mind. -- Kenneth Murchison Systems Programmer Project Cyrus Developer/Maintainer Carnegie Mellon University Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
how to limit pop/imap login password attempts
Hi, I've been trying to figure out, how to limit login attempts for cyrus pop/imap daemons. I'm trying to prevent brute-force password guessing. I'm using cyrus sasl with /etc/sasldb2 user database, which also authenticates postfix users. I'd like to solve this problem through sasl so I won't have to figure the same for postfix or keep different passwords for mailboxes and smtp. Is there any mechanism to do this through sasl or do I have to try doing it through a firewall? I'm running debian etch system. If imap and pop do not allow multiple login attempts within a single session, I could try to work around this problem using iptables with the recent module but it's like scratching your left ear with your right hand around the back of your head. thanks for any pointers Martin Kraus Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Bad problems with 2.3.10 on FreeBSD
Rich Wales wrote: > I recently upgraded my system to 2.3.10. My main Cyrus server runs > FreeBSD 6.2, replicating to a second server running Ubuntu 7.10. I > had decided to upgrade now from 2.3.9 to 2.3.10 because of replication > "bailing out" errors in 2.3.9 that are reportedly fixed in 2.3.10. > > At first, I was having lots of seg-fault errors, but I added the > getgrouplist() patch to auth_unix.c on my FreeBSD system, and that > seemed to take care of the core-dumps. > > However, I started seeing other problems on the FreeBSD box -- such > as "ctl_cyrusdb -c" checkpoint jobs hanging, and other mail delivery > attempts hanging (or timing out) and piling up in my Postfix queue. > I had to restart Cyrus frequently on the FreeBSD system in order to > clear the logjams. > > I finally gave up and went back to 2.3.9 on both servers. Given the > instability of 2.3.10 on my main server, I felt I had no choice. > > Any ideas what might have been happening? If I can't figure out how > to get 2.3.10 running stably, I assume I'm going to either have to > stick with 2.3.9 (and figure out some other way around the replication > bailout errors until 2.3.11 comes out), or else move my main Cyrus > operations onto some other platform (maybe another Ubuntu box). > Did you check anything with the FBSD port maintainer? There may be patches available that cannot be commited due to the freeze. --per Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Creating mailboxes with cyradm - useraccount with a 'dot'
Hi mailinglist. ( newbie qestion ahead ;-) ) At the moment I am experimenting with cyrus, running on fedora 7. Problem: Creating mailboxes with cyradm works fine until I want to create a mailbox for a account like 'peter.testaccount': In cyradm: > cm user.peter.testaccount > createmailbox: Permission denied While it seems to be logical to me, that mailboxnames with a dot are not working out-of-the-box because of the way cyrus organizes it's mailboxes (user..), it would be nice, if sombody could give me a hint - how to cope with useraccounts with a 'dot' in them and the corresponding cyrus IMAP mailboxes. Thanks for your help in advance... Stefan Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: how to limit pop/imap login password attempts
Martin Kraus wrote: > Hi, > I've been trying to figure out, how to limit login attempts for cyrus > pop/imap daemons. I'm trying to prevent brute-force password guessing. I'm > using cyrus sasl with /etc/sasldb2 user database, which also authenticates > postfix users. I'd like to solve this problem through sasl so I won't have to > figure the same for postfix or keep different passwords for mailboxes and > smtp. Is there any mechanism to do this through sasl or do I have to try doing > it through a firewall? > > I'm running debian etch system. If imap and pop do not allow multiple login > attempts within a single session, I could try to work around this problem > using iptables with the recent module but it's like scratching your left ear > with your right hand around the back of your head. Hi Martin, A couple of ideas come to mind. You could force the use of the 'NODICT' security flag, or force the use of mechanism which support it. see: http://www.sendmail.org/~ca/email/cyrus2/mechanisms.html Using those mechanisms would probably require a change in the way your users authenticate. Another idea, and this isn't really an approach that will work today, is to use the ldapdb auxprop plugin to store your passwords, and make use of the openldap ppolicy module to enforce password policy. This doesn't really work, because openldap ppolicy does not (yet) enforce password policy when sasl bind (which ldapdb uses) is in use. It only support simple bind. I haven't actually looked at OpenLDAP 2.4.x yet to see if it's supported. A modification to the ldapdb plugin could probably be made to perform a simple bind just after the step where it retrieves the userPassword attribute. - Dan Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Creating mailboxes with cyradm - useraccount with a 'dot'
Hi, If you use unixhierarchysep: 0 in you /etc/imapd.conf (which is default) the . is used as hierarchyseperator. So in your case user.peter.testaccount would be the Folder testaccount of the user peter. There are two way use ^ to replace . cm user.peter^testaccount Or use unixhierarchysep: 1 so / is used as hierarchyseperator. cm user/peter.testaccount Regards Michael Quoting "Kuhlemann, Stefan" <[EMAIL PROTECTED]>: Hi mailinglist. ( newbie qestion ahead ;-) ) At the moment I am experimenting with cyrus, running on fedora 7. Problem: Creating mailboxes with cyradm works fine until I want to create a mailbox for a account like 'peter.testaccount': In cyradm: cm user.peter.testaccount createmailbox: Permission denied While it seems to be logical to me, that mailboxnames with a dot are not working out-of-the-box because of the way cyrus organizes it's mailboxes (user..), it would be nice, if sombody could give me a hint - how to cope with useraccounts with a 'dot' in them and the corresponding cyrus IMAP mailboxes. Thanks for your help in advance... Stefan Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html M.Menge Tel.: (49) 7071/29-70316 Universitaet Tuebingen Fax.: (49) 7071/29-5912 Zentrum fuer Datenverarbeitung mail: [EMAIL PROTECTED] Waechterstrasse 76 72074 Tuebingen smime.p7s Description: S/MIME krytographische Unterschrift Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: how to limit pop/imap login password attempts
Dan White wrote: > A modification to the ldapdb plugin could probably be made to perform a > simple bind just after the step where it retrieves the userPassword > attribute. On second thought, that doesn't actually work. The auxprop plugin doesn't know anything about the password, or sasl exchange, that the users performs, only what the correct userPassword is... - Dan Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
AW: Creating mailboxes with cyradm - useraccount with a 'dot'
> -Ursprüngliche Nachricht- > Von: [EMAIL PROTECTED] > [mailto:[EMAIL PROTECTED] Im Auftrag > von Michael Menge > Gesendet: Mittwoch, 21. November 2007 17:07 > An: info-cyrus@lists.andrew.cmu.edu > Betreff: Re: Creating mailboxes with cyradm - useraccount with a 'dot' [...] > > There are two way use ^ to replace . > > cm user.peter^testaccount > > Or use unixhierarchysep: 1 so / is used as hierarchyseperator. > > cm user/peter.testaccount > > Regards > > Michael > [...] Thanks... I will use 'unixhierarchysep' :-) Btw. - are there any drawbacks coming with 'unixhierarchysep: 1' ? Not that I am running into problems later on? (...so that I will miss some very usefull fuctions) Stefan Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: pop3d 100% cpu
I have... the problem was actually with our sendmail it is supposed to have a message size limit... for the maximum size of message that sendmail will transfer for it turns out that during one of the updates, that option was not set... so by default it is unlimited, so when a user wanted to send a 170MB home video from her outlook, the server was sitting there trying to transfer it... The problem was fixed by removing the mailq and resetting the option in sendmail. I guess that for the pop, your server received a large message and one(or more) of the user(s) is trying to download that to their computer? I think cyrus has a max_size option ... or may be lmtp... which tells it to fail if it finds a message larger than that -- -- -- Rafael Mahecha E-mail Administrator Office of Information Management Jackson State University JSU e-Center 1230 Raymond Road Jackson, MS 39204 [EMAIL PROTECTED] (601)-979-1783 http://www.jsums.edu On Tue, November 20, 2007 3:51 pm, Faust wrote: > > > About 5 to 10 minutes after starting cyrus_imapd the process pop3d will take > 100% cpu usage and the system will freece. > > Anyone else has seen this problem? > > I am using fedora 8 with the following packages; > > cyrus-sasl-plain-2.1.22-6 > cyrus-imapd-utils-2.3.9-7.fc7 > cyrus-sasl-lib-2.1.22-6 > cyrus-sasl-md5-2.1.22-6 > cyrus-imapd-perl-2.3.9-7.fc7 > cyrus-imapd-2.3.9-7.fc7 > cyrus-sasl-2.1.22-6 > > > Faust > > > Cyrus Home Page: http://cyrusimap.web.cmu.edu/ > Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki > List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html > Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: LARGE single-system Cyrus installs?
On Wed, 21 Nov 2007, Ian G Batten wrote: >> however a fsync on a journal ed filesystem just means the data needs to be >> written to the journal, it doesn't mean that the journal needs to be >> flushed to >> disk. >> >> on ext3 if you have data=journal ed then your data is in the journal as well >> and >> all that the system needs to do on a fsync is to write things to the >> journal (a >> nice sequential write), > > Assuming the journal is on a distinct device and the distinct device can take > the load. It isn't on ZFS, although work is in progress. I was responding to the comments about ext3 and other journal ed filesystems as alternatives to zfs and the claim that doing a fsync on one of them required flushing the entire journal. sorry if I wasn't clear enough about this. David Lang Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
prefork mupdate slaves
With the following service entry on a traditional murder frontend (v2.3.10): mupdate cmd="/usr/local/cyrus/bin/mupdate" listen="3905" prefork=1 I notice that 2 mupdate processes are spawned at startup: Nov 21 14:43:43 cyrus-fe3 mupdate[6935]: successful mupdate connection to cyrus-mm.onid.oregonstate.edu Nov 21 14:43:43 cyrus-fe3 mupdate[6935]: unready for connections Nov 21 14:43:43 cyrus-fe3 mupdate[6935]: synchronizing mailbox list with master mupdate server Nov 21 14:43:43 cyrus-fe3 mupdate[6931]: successful mupdate connection to cyrus-mm.onid.oregonstate.edu Nov 21 14:43:43 cyrus-fe3 mupdate[6931]: unready for connections Nov 21 14:43:43 cyrus-fe3 mupdate[6931]: synchronizing mailbox list with master mupdate server Nov 21 14:43:48 cyrus-fe3 mupdate[6935]: mailbox list synchronization complete Nov 21 14:43:52 cyrus-fe3 mupdate[6931]: mailbox list synchronization complete And those 2 mupdate processes remain running. If I set prefork=0, then no mupdate processes are ever spawned, as far as I can tell. On my old v2.2.13 murder frontends, prefork=1 only causes 1 mupdate process to be spawned. Is this a bug? Should I be concerned that 2 mupdate slaves are running? Thanks, Andy Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
mixing versions in murder
Are there any DOs or DON'Ts when mixing different versions of Cyrus in a traditional murder? Specifically, I have a murder at version 2.2.13, and I'd like to upgrade to version 2.3.10. Can I upgrade the frontends first? Do I need to upgrade all of them at once? Thanks, Andy Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
reconstruct -u or -U
The changelog says: Added -u and -U options to reconstruct -- courtesy of David Carter. But I can't find those options listed in the manpage or the built-in help of reconstruct. What do those options do? Andy Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: how to limit pop/imap login password attempts
On Nov 21, 2007 2:27 PM, Martin Kraus <[EMAIL PROTECTED]> wrote:
> Hi,
> I've been trying to figure out, how to limit login attempts for cyrus
> pop/imap daemons. I'm trying to prevent brute-force password guessing.
You can try to use nginx as a proxy imap, pop and smtp protocol. (and
HTTP of course).
The goal is to have the same frontend for multiple pop/imap server and redirect
any connection to the good one depending on the username.
You need to provide a small application that depending the username
will give the
address of the server where the imap/pop account is stored. Some perl
and php sample
are on the nginx wiki
Here is a simple I wrote in python
#!/bin/env python
import sys, BaseHTTPServer
class NginxAuth(BaseHTTPServer.BaseHTTPRequestHandler):
def do_GET(self):
print 'GET', self.client_address, self.path, self.headers
user=self.headers["Auth-User"]
password=self.headers["Auth-Pass"]
protocol=self.headers["Auth-Protocol"]
self.send_response(200, 'OK')
self.send_header('Auth-Status', 'OK')
self.send_header('Auth-Server', '127.0.0.1')
self.send_header('Auth-Port', '143')
self.end_headers()
server=BaseHTTPServer.HTTPServer(('127.0.0.1',8081), NginxAuth)
server.serve_forever()
Here I redirect all connection to my unique server 127.0.0.1 without
doing any check on the user/password (Auth-Status='OK')
If the password was wrong, then the imap server will reject the
connection anyway.
But you can keep a log of all connections with a timestamp and reject
the connection
if the password is changing too often in a small amount of time.
Dont forget to share your experiences if you get some success.
> I'm
> using cyrus sasl with /etc/sasldb2 user database, which also authenticates
> postfix users. I'd like to solve this problem through sasl so I won't have to
> figure the same for postfix or keep different passwords for mailboxes and
> smtp. Is there any mechanism to do this through sasl or do I have to try doing
> it through a firewall?
>
> I'm running debian etch system. If imap and pop do not allow multiple login
> attempts within a single session, I could try to work around this problem
> using iptables with the recent module but it's like scratching your left ear
> with your right hand around the back of your head.
>
> thanks for any pointers
> Martin Kraus
>
> Cyrus Home Page: http://cyrusimap.web.cmu.edu/
> Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
> List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
>
--
Alain Spineux
aspineux gmail com
May the sources be with you
Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: LARGE single-system Cyrus installs?
>>> Would'nt it be nice to have a configuration option to completely >>> turn off fsync() in Cyrus? If you want, with a BIG WARNING in the >>> doc stating NOT TO USE IT unless you know what you doing. :) >> >> Its already in imapd.conf(8): >> >> skiplist_unsafe > >Well shiver me timbers! Ya'll rock. Note, however, that fsync() still serves a purpose with data journalling - without it, your application writes may be sitting in buffer cache, and may be arbitrarily re-ordered before hitting the disk. The fsync() should not return until the blocks have reached the disk (or journal), and thus forms a syncronisation point (which is critical for maintaining sanity of on-disk data structures like skiplists). -- Andrew McNamara, Senior Developer, Object Craft http://www.object-craft.com.au/ Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: prefork mupdate slaves
On Wed, Nov 21, 2007 at 02:58:11PM -0800, Andrew Morgan wrote: > With the following service entry on a traditional murder frontend > (v2.3.10): > >mupdate cmd="/usr/local/cyrus/bin/mupdate" listen="3905" prefork=1 > > I notice that 2 mupdate processes are spawned at startup: > > Nov 21 14:43:43 cyrus-fe3 mupdate[6935]: successful mupdate connection to > cyrus-mm.onid.oregonstate.edu > Nov 21 14:43:43 cyrus-fe3 mupdate[6935]: unready for connections > Nov 21 14:43:43 cyrus-fe3 mupdate[6935]: synchronizing mailbox list with > master mupdate server > Nov 21 14:43:43 cyrus-fe3 mupdate[6931]: successful mupdate connection to > cyrus-mm.onid.oregonstate.edu > Nov 21 14:43:43 cyrus-fe3 mupdate[6931]: unready for connections > Nov 21 14:43:43 cyrus-fe3 mupdate[6931]: synchronizing mailbox list with > master mupdate server > Nov 21 14:43:48 cyrus-fe3 mupdate[6935]: mailbox list synchronization complete > Nov 21 14:43:52 cyrus-fe3 mupdate[6931]: mailbox list synchronization complete > > And those 2 mupdate processes remain running. > > If I set prefork=0, then no mupdate processes are ever spawned, as far as > I can tell. > > On my old v2.2.13 murder frontends, prefork=1 only causes 1 mupdate > process to be spawned. > > Is this a bug? Should I be concerned that 2 mupdate slaves are running? I have two on the murder front end as well: UID PID PPID CSTIME TTY TIME CMD cyrus 765 708 0 Sep 24 ? 332:49 mupdate cyrus 768 708 0 Sep 24 ?1:35 mupdate On the murder back end, I actually have three, although one terminates almost immediately with this error: bind: /imap/mupdate/conf/socket/mupdate.target: Address already in use bind failed service mupdate pid 1595 in READY state: terminated abnormally I also have `prefork=1' in both places. This is with cyrus-imapd-2.3.8. -- -Gary Mills--Unix Support--U of M Academic Computing and Networking- Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: reconstruct -u or -U
On Wed, Nov 21, 2007 at 03:06:39PM -0800, Andrew Morgan wrote: > The changelog says: > >Added -u and -U options to reconstruct -- courtesy of David Carter. > > But I can't find those options listed in the manpage or the built-in help > of reconstruct. What do those options do? The changelog needs to be updated. They got renamed as -g and -G. They either wipe or calculate the message GUID (globally unique identifier, we hope!) which is the sha1 of the message contents now. Bron. Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Murder in replicated mode
Hi!
I'm trying to setup murder in replicated mode. My schema is:
-Two servers + one shared storage
-Redhat Cluster Suite (RHEL 5.1) with GFS2 working.
-Cyrus 2.3.10 in both servers working.
-spool and sieve directories on GFS
-config dir on local filesystems.
I have configured Cyrus in replicated mode but when I create an
account in the master server, it isn't replicated unless Cyrus
is restart in either node. It isnt't an authentication problem,
when I create an account with cyradmin mupdate do nothing.
My config files:
Master imapd.conf
configdirectory: /var/lib/cyrus
partition-default: /var/cyrus/spool
admins: cyrus
sievedir: /var/cyrus/sieve
sendmail: /usr/sbin/sendmail
hashimapspool: true
sasl_pwcheck_method: auxprop
sasl_mech_list: PLAIN
tls_cert_file: /var/cyrus/etc/cyrus-cert.pem
tls_key_file: /var/cyrus/etc/cyrus-key.pem
tlscache_db: skiplist
autocreatequota: 100
allowplaintext: 1
defaultdomain: xcoop.coop
mupdate_config: replicated
-Master cyrus.conf-
START {
# do not delete this entry!
recover cmd="ctl_cyrusdb -r"
# this is only necessary if using idled for IMAP IDLE
#idled cmd="idled"
}
# UNIX sockets start with a slash and are put into /var/imap/socket
SERVICES {
# add or remove based on preferences
imapcmd="imapd" listen="imap" prefork=0
imaps cmd="imapd -s" listen="imaps" prefork=0
pop3cmd="pop3d" listen="pop3" prefork=0
pop3s cmd="pop3d -s" listen="pop3s" prefork=0
sieve cmd="timsieved" listen="sieve" prefork=0
mupdate cmd="mupdate -m" listen=3905 prefork=1
# these are only necessary if receiving/exporting usenet via NNTP
#nntp cmd="nntpd" listen="nntp" prefork=0
#nntps cmd="nntpd -s" listen="nntps" prefork=0
# at least one LMTP is required for delivery
#lmtp cmd="lmtpd" listen="lmtp" prefork=0
lmtpunixcmd="lmtpd" listen="/var/imap/socket/lmtp" prefork=0
# this is required if using notifications
#notify cmd="notifyd" listen="/var/imap/socket/notify" proto="udp"
prefork=1
}
EVENTS {
# this is required
checkpoint cmd="ctl_cyrusdb -c" period=30
# this is only necessary if using duplicate delivery suppression,
# Sieve or NNTP
delprunecmd="cyr_expire -E 3" at=0400
# this is only necessary if caching TLS sessions
tlsprunecmd="tls_prune" at=0400
}
Slave imapd.conf---
configdirectory: /var/lib/cyrus
partition-default: /var/cyrus/spool
admins: cyrus
sievedir: /var/cyrus/sieve
sendmail: /usr/sbin/sendmail
hashimapspool: true
sasl_pwcheck_method: auxprop
sasl_mech_list: PLAIN
tls_cert_file: /var/cyrus/etc/cyrus-cert.pem
tls_key_file: /var/cyrus/etc/cyrus-key.pem
tlscache_db: skiplist
autocreatequota: 100
allowplaintext: 1
defaultdomain: xcoop.coop
mupdate_server: rh-cluster1
mupdate_username: cyrus
mupdate_authname: cyrus
mupdate_password: 123456
mupdate_config: replicated
Slave cyrus.conf-
START {
# do not delete this entry!
recover cmd="ctl_cyrusdb -r"
# this is only necessary if using idled for IMAP IDLE
#idled cmd="idled"
}
# UNIX sockets start with a slash and are put into /var/imap/socket
SERVICES {
# add or remove based on preferences
imapcmd="imapd" listen="imap" prefork=0
imaps cmd="imapd -s" listen="imaps" prefork=0
pop3cmd="pop3d" listen="pop3" prefork=0
pop3s cmd="pop3d -s" listen="pop3s" prefork=0
sieve cmd="timsieved" listen="sieve" prefork=0
mupdate cmd="mupdate" listen=3905 prefork=1
# these are only necessary if receiving/exporting usenet via NNTP
#nntp cmd="nntpd" listen="nntp" prefork=0
#nntps cmd="nntpd -s" listen="nntps" prefork=0
# at least one LMTP is required for delivery
#lmtp cmd="lmtpd" listen="lmtp" prefork=0
lmtpunixcmd="lmtpd" listen="/var/imap/socket/lmtp" prefork=0
# this is required if using notifications
#notify cmd="notifyd" listen="/var/imap/socket/notify" proto="udp"
prefork=1
}
EVENTS {
# this is required
checkpoint cmd="ctl_cyrusdb -c" period=30
# this is only necessary if using duplicate delivery suppression,
# Sieve or NNTP
delprunecmd="cyr_expire -E 3" at=0400
# this is only necessary if caching TLS sessions
tlsprunecmd="tls_prune" at=0400
}
Regards!
--
--
Diego Woitasen
Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
Re: Murder in replicated mode
On Wed, Nov 21, 2007 at 09:41:15PM -0300, Diego Woitasen wrote: > Hi! > I'm trying to setup murder in replicated mode. My schema is: > > -Two servers + one shared storage > -Redhat Cluster Suite (RHEL 5.1) with GFS2 working. > -Cyrus 2.3.10 in both servers working. > -spool and sieve directories on GFS > -config dir on local filesystems. > > I have configured Cyrus in replicated mode but when I create an > account in the master server, it isn't replicated unless Cyrus > is restart in either node. It isnt't an authentication problem, > when I create an account with cyradmin mupdate do nothing. >From memory you may have to actually deliver a message to one of the user's folders before it will replicate. At least that's what the docs suggested. Of course we always LMTP deliver a message right at creation time, so we never bothered to check this. Bron. Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: FastMail.FM Patchset Updated
On Wed, Nov 21, 2007 at 06:37:17AM -0500, Ken Murchison wrote: > Bron Gondwana wrote: >> On Wed, 21 Nov 2007 07:40:21 +0100 (CET), "Simon Matter" >> <[EMAIL PROTECTED]> said: >>> Hi Bron, >>> >>> Did you consider this one >>> http://bugzilla.andrew.cmu.edu/show_bug.cgi?id=3006 in the patch above? >>> From a quick look it seems both patches conflict, is #3006 obsolete now? >> You're right, they probably do conflict and Ken has applied 3006. I >> should >> have checked on that. Hmm... >> Actually, my patch doesn't even fix #3006. It does sort of suck to be >> working >> from a stable past point rather than the latest CVS actually when I do >> these >> patches - though it's nice from a stability point of view. >> OK - Ken, would you like me to re-build this patch on top of current CVS, >> or >> would you like to do it yourself? My patch has the replication bugfixes >> which >> are still worth doing, but Simon's logic is actually correct for the test >> while mine isn't (though I renamed the function he's using...) > > If you have the time. I'm off banging on other things at the moment. Attached are tidied up versions of both my earlier patches on top of what you've already got in CVS. They apply fine on top of current CVS trunk: [EMAIL PROTECTED]:/work/cvs/cyrus-imapd# patch -p1 < /work/cyrus-imapd-2.3.10/patches/cyrus-skiplist-newfixes-2.3.10.diff patching file lib/cyrusdb_skiplist.c Hunk #1 succeeded at 2113 (offset -1 lines). Hunk #2 succeeded at 2140 (offset -1 lines). Hunk #3 succeeded at 2166 (offset -1 lines). Hunk #4 succeeded at 2206 (offset -1 lines). [EMAIL PROTECTED]:/work/cvs/cyrus-imapd# patch -p1 < /work/cyrus-imapd-2.3.10/patches/cyrus-deletemode-userfix-2.3.10.diff patching file imap/mboxname.c patching file imap/mboxlist.c patching file imap/mboxlist.h patching file imap/mboxname.h patching file imap/imapd.c Hunk #1 succeeded at 5006 (offset 2 lines). Hunk #2 succeeded at 5098 (offset 2 lines). Bron. Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: FastMail.FM Patchset Updated (new patches)
On Wed, Nov 21, 2007 at 06:37:17AM -0500, Ken Murchison wrote:
> Bron Gondwana wrote:
>> On Wed, 21 Nov 2007 07:40:21 +0100 (CET), "Simon Matter"
>> <[EMAIL PROTECTED]> said:
>>> Hi Bron,
>>>
>>> Did you consider this one
>>> http://bugzilla.andrew.cmu.edu/show_bug.cgi?id=3006 in the patch above?
>>> From a quick look it seems both patches conflict, is #3006 obsolete now?
>> You're right, they probably do conflict and Ken has applied 3006. I
>> should
>> have checked on that. Hmm...
>> Actually, my patch doesn't even fix #3006. It does sort of suck to be
>> working
>> from a stable past point rather than the latest CVS actually when I do
>> these
>> patches - though it's nice from a stability point of view.
>> OK - Ken, would you like me to re-build this patch on top of current CVS,
>> or
>> would you like to do it yourself? My patch has the replication bugfixes
>> which
>> are still worth doing, but Simon's logic is actually correct for the test
>> while mine isn't (though I renamed the function he's using...)
>
> If you have the time. I'm off banging on other things at the moment.
I guess I should attach them!
Bron.
Fix Delayed Delete replication
Candidate for upstream - now respun on top of
Simon Matter's allowusermoves fixes from CVS
Deleting user.foo was broken on replicas thanks to
me choosing a bad method for making it work.
Also, it was leaving foo.TIMESTAMP.sub files because
it considered that to be a valid "user". Oops.
This patch takes a different approach (on top of the
delayed delete code already in Cyrus 2.3.10):
1) mboxname_isusermailbox() no longer returns true for
DELETED.user.foo.TIMESTAMP
2) The "can I rename this" code checks for the target
name being either another username or something in the
DELETED. namespace.
2a) mboxname_isdeletedmailbox() rather than the somewhat
(IMHO) misplaced mboxlist_in_deletedhierarchy().
3) if "forceuser" is passed (only done by sync_server),
then all ACL checks are completely bypassed, so the
rename will always succeed on the replica.
Index: cyrus-imapd-2.3.10/imap/mboxname.c
===
--- cyrus-imapd-2.3.10.orig/imap/mboxname.c 2007-11-22 01:22:08.0 -0500
+++ cyrus-imapd-2.3.10/imap/mboxname.c 2007-11-22 01:29:17.0 -0500
@@ -600,35 +600,47 @@
{
const char *p;
const char *start = name;
-const char *deletedprefix = config_getstring(IMAPOPT_DELETEDPREFIX);
-size_t len = strlen(deletedprefix);
-int isdel = 0;
/* step past the domain part */
if (config_virtdomains && (p = strchr(start, '!')))
start = p + 1;
-/* step past any deletedprefix */
-if (mboxlist_delayed_delete_isenabled() && strlen(start) > len+1 &&
- !strncmp(start, deletedprefix, len) && start[len] == '.') {
- start += len + 1;
- isdel = 1; /* there's an add'l sep + hextimestamp on isdel folders */
-}
-
/* starts with "user." AND
* we don't care if it's an inbox OR
- * there's no dots after the username OR
- * it's deleted and there's only one more dot
+ * there's no dots after the username
*/
-if (!strncmp(start, "user.", 5) &&
- (!isinbox || !strchr(start+5, '.') ||
- (isdel && (p = strchr(start+5, '.')) && !strchr(p+1, '.'
- return (char*) start+5; /* could have trailing bits if isinbox+isdel */
+if (!strncmp(start, "user.", 5) && (!isinbox || !strchr(start+5, '.')))
+ return (char*) start+5;
else
return NULL;
}
/*
+ * If (internal) mailbox 'name' is a DELETED mailbox
+ * returns boolean
+ */
+int mboxname_isdeletedmailbox(const char *name)
+{
+static const char *deletedprefix = NULL;
+static int deletedprefix_len = 0;
+int domainlen = 0;
+char *p;
+
+if (!mboxlist_delayed_delete_isenabled()) return(0);
+
+if (!deletedprefix) {
+ deletedprefix = config_getstring(IMAPOPT_DELETEDPREFIX);
+ deletedprefix_len = strlen(deletedprefix);
+}
+
+if (config_virtdomains && (p = strchr(name, '!')))
+ domainlen = p - name + 1;
+
+return ((!strncmp(name + domainlen, deletedprefix, deletedprefix_len) &&
+ name[domainlen + deletedprefix_len] == '.') ? 1 : 0);
+}
+
+/*
* Translate (internal) inboxname into corresponding userid.
*/
char *mboxname_inbox_touserid(const char *inboxname)
Index: cyrus-imapd-2.3.10/imap/mboxlist.c
===
--- cyrus-imapd-2.3.10.orig/imap/mboxlist.c 2007-11-22 01:29:03.0 -0500
+++ cyrus-imapd-2.3.10/imap/mboxlist.c 2007-11-22 01:32:08.0 -0500
@@ -1223,8 +1223,7 @@
isusermbox = 1;
} else if ((config_getswitch(IMAPOPT_ALLOWUSERMOVES) &&
mboxname_isusermailbox(newname, 1)) ||
- mboxlist_in_deletedhierarchy(oldname) ||
- mboxlist_in_deletedhierarchy(newname)) {
+ mboxname_isdeletedmailbox(newname)) {
/* Special case of renaming a user */
access = cyrus_acl_myrights(auth_state, oldacl);
if (!(access & ACL_D
Re: Murder in replicated mode
On Thu, Nov 22, 2007 at 12:50:47PM +1100, Bron Gondwana wrote: > On Wed, Nov 21, 2007 at 09:41:15PM -0300, Diego Woitasen wrote: > > Hi! > > I'm trying to setup murder in replicated mode. My schema is: > > > > -Two servers + one shared storage > > -Redhat Cluster Suite (RHEL 5.1) with GFS2 working. > > -Cyrus 2.3.10 in both servers working. > > -spool and sieve directories on GFS > > -config dir on local filesystems. > > > > I have configured Cyrus in replicated mode but when I create an > > account in the master server, it isn't replicated unless Cyrus > > is restart in either node. It isnt't an authentication problem, > > when I create an account with cyradmin mupdate do nothing. > > >From memory you may have to actually deliver a message to one of > the user's folders before it will replicate. At least that's what > the docs suggested. Of course we always LMTP deliver a message > right at creation time, so we never bothered to check this. Yes, that's the way it goes. When I setup my system (migrated from an ages-old cyrus system using imapsync), I had first imapsync'd the primary system from the old system, before setting up the replica. So, when the replica was up and running, I did a sync_client -u for every user and a sync_client -m for every bulletin board in the primary system... took a couple of days :) (I had the replica running during following imapsyncs and the final one, so of course the changes got propagated automatically then.) --Janne Peltonen Imap admin Univ of Helsinki -- Janne Peltonen <[EMAIL PROTECTED]> Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: LARGE single-system Cyrus installs?
Vincent Fox <[EMAIL PROTECTED]> wrote: > This thought has occurred to me: > > ZFS prefers reads over writes in it's scheduling. > > I think you can see where I'm going with this. My WAG is something > related to Pascal's, namely latency. What if my write requests to > mailboxes.db > or deliver.db start getting stacked up, due to the favoritism shown to > reads? I got substantial benefits from setting compression=on and recordsize=32K on the filesystem where deliver.db resides. After talking with our SAN staff it showed up that storage was our problem - it has problems with concurrent write and read calls, the system won't answer read requests if the write channel is "full". I don't know whether it is a firmwire issue or a non-capability of the storage system. Lowering ZFS' recordsize and activating compression on that partition cut down i/o rate and things are going normal here again. Thanks to all who helped! Pascal PS: The mirror resilvering problem was a misconfiguration of a brocade switch... Sometimes you don't see the forest due to so many trees (german proverb, "Man sieht den Wald vor lauter Bäumen nicht")... Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
