Re: FastMail.FM Patchset Updated

[Simon Matter]

> As usual you can get the patches here:
>
> http://cyrus.brong.fastmail.fm/
>
>
> I've been busy with Cyrus _again_ - so much for my theory
> that I was taking a break.
>
> OK - here's what's new.
>
> *
> http://cyrus.brong.fastmail.fm/patches/cyrus-skiplist-bugfixes-2.3.10.diff
>   http://cyrus.brong.fastmail.fm/patches/cyrus-skiplist-robustify-2.3.10.diff
>
>   Skiplist issues - there were two things that could be found
>   in recovery that actually bit us during the whole "restart
>   every single store with the new skiplist code" project the
>   other day.  ADD where the record already existed and DELETE
>   where it didn't.  The later also had an obnoxious bug where
>   it would instead delete _the_alphabetically_NEXT_record_
>   silently.  Ooops.
>
>   I rolled these two into my bugfix and robustify patches, not
>   realising Ken had already applied the previous copies upstream.
>   Ken - do you want me to break this out as a separate patch on
>   top of the others?
>
> *
> http://cyrus.brong.fastmail.fm/patches/cyrus-sync-renamedmailbox-2.3.10.diff
>
>   DelayedDelete of entire users was causing excess copying.
>   This fixes it, but the solution is less than ideal and causes
>   excess messages about folders not existing during an account
>   create.  Annoying.  I'd like a better fix, but this is enough
>   for now.  Found this one after fixing...
>
> *
> http://cyrus.brong.fastmail.fm/patches/cyrus-deletemode-userfix-2.3.10.diff
>
>   This is for upstream.  I made a bogus design decision in the
>   DelayedDelete code that Ken accepted, and it was causing
>   bailouts and all sorts of yuckyness.  Made the conditions for
>   allowing a folder rename into the DELETED. namespace a lot more
>   explicit and correct rather than DELETED.user.foo.TIMESTAMP
>   being considered a user's mailbox!  The user cleanup script
>   no longer causes massive bailouts on sync.

Hi Bron,

Did you consider this one
http://bugzilla.andrew.cmu.edu/show_bug.cgi?id=3006 in the patch above?
>From a quick look it seems both patches conflict, is #3006 obsolete now?

Thanks,
Simon

>
> *
> http://cyrus.brong.fastmail.fm/patches/cyrus-expunged-nocache-2.3.10.diff
>
>   A little thing to shut up the issue that used to cause segfaults
>   and now just causes logging instead.  Cache offsets in the .expunge
>   file can be bogus for deeper architectural reasons.  Rather than
>   fix the underlying reasons I just ignore them completely when
>   running cyr_expire.  At least that way we're not reading bogus
>   cache records.
>
> * http://cyrus.brong.fastmail.fm/patches/cyrus-fastrename-2.3.9.diff
>
>   UPDATED.  It turns out it really doesn't matter what YOU can see
>   when you're checking if you can use FastRename.  It matters if
>   there are subfolders at all.  Change to passing isadmin true
>   and not passing the username to mboxlist_count_inferiors().
>   Also need to check if the target path has inferiors to avoid
>   log messages and partial move failures that have to back out.
>   Much nicer this way.  This means fastrename on replicas isn't
>   totally broken any more (before, it would never see the subfolders
>   because the replication user didn't have ACLs on them and isadmin
>   was being set to false explicitly)
>
>
> Ken - I'd love to see the deletemode-userfix and skiplist stuff
> go upstream.  I know you're not happy with fastrename yet, and
> fair enough - it's an extra risk and if a shutdown happens in
> the middle of the operation things can get very confused!  The
> other two patches are not really long-term good for the Cyrus
> codebase so I'd prefer to fix the underlying issues instead :)
>
> Bron.
> 
> Cyrus Home Page: http://cyrusimap.web.cmu.edu/
> Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
> List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
>



Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

FastMail.FM Patchset Updated

[Bron Gondwana]

As usual you can get the patches here:

http://cyrus.brong.fastmail.fm/


I've been busy with Cyrus _again_ - so much for my theory
that I was taking a break.

OK - here's what's new.

* http://cyrus.brong.fastmail.fm/patches/cyrus-skiplist-bugfixes-2.3.10.diff
  http://cyrus.brong.fastmail.fm/patches/cyrus-skiplist-robustify-2.3.10.diff

  Skiplist issues - there were two things that could be found
  in recovery that actually bit us during the whole "restart
  every single store with the new skiplist code" project the
  other day.  ADD where the record already existed and DELETE
  where it didn't.  The later also had an obnoxious bug where
  it would instead delete _the_alphabetically_NEXT_record_
  silently.  Ooops.

  I rolled these two into my bugfix and robustify patches, not
  realising Ken had already applied the previous copies upstream.
  Ken - do you want me to break this out as a separate patch on
  top of the others?

* http://cyrus.brong.fastmail.fm/patches/cyrus-sync-renamedmailbox-2.3.10.diff

  DelayedDelete of entire users was causing excess copying.
  This fixes it, but the solution is less than ideal and causes
  excess messages about folders not existing during an account
  create.  Annoying.  I'd like a better fix, but this is enough
  for now.  Found this one after fixing...

* http://cyrus.brong.fastmail.fm/patches/cyrus-deletemode-userfix-2.3.10.diff

  This is for upstream.  I made a bogus design decision in the 
  DelayedDelete code that Ken accepted, and it was causing
  bailouts and all sorts of yuckyness.  Made the conditions for
  allowing a folder rename into the DELETED. namespace a lot more
  explicit and correct rather than DELETED.user.foo.TIMESTAMP
  being considered a user's mailbox!  The user cleanup script
  no longer causes massive bailouts on sync.

* http://cyrus.brong.fastmail.fm/patches/cyrus-expunged-nocache-2.3.10.diff

  A little thing to shut up the issue that used to cause segfaults 
  and now just causes logging instead.  Cache offsets in the .expunge
  file can be bogus for deeper architectural reasons.  Rather than
  fix the underlying reasons I just ignore them completely when
  running cyr_expire.  At least that way we're not reading bogus
  cache records.

* http://cyrus.brong.fastmail.fm/patches/cyrus-fastrename-2.3.9.diff

  UPDATED.  It turns out it really doesn't matter what YOU can see
  when you're checking if you can use FastRename.  It matters if
  there are subfolders at all.  Change to passing isadmin true
  and not passing the username to mboxlist_count_inferiors().
  Also need to check if the target path has inferiors to avoid
  log messages and partial move failures that have to back out.
  Much nicer this way.  This means fastrename on replicas isn't
  totally broken any more (before, it would never see the subfolders
  because the replication user didn't have ACLs on them and isadmin
  was being set to false explicitly)


Ken - I'd love to see the deletemode-userfix and skiplist stuff
go upstream.  I know you're not happy with fastrename yet, and
fair enough - it's an extra risk and if a shutdown happens in 
the middle of the operation things can get very confused!  The
other two patches are not really long-term good for the Cyrus
codebase so I'd prefer to fix the underlying issues instead :)

Bron.

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

pop3d 100% cpu

[Faust]

About 5 to 10 minutes after starting cyrus_imapd the process pop3d will take
100% cpu usage and the system will freece.

Anyone else has seen this problem?

I am using fedora 8 with the following packages;

cyrus-sasl-plain-2.1.22-6
cyrus-imapd-utils-2.3.9-7.fc7
cyrus-sasl-lib-2.1.22-6
cyrus-sasl-md5-2.1.22-6
cyrus-imapd-perl-2.3.9-7.fc7
cyrus-imapd-2.3.9-7.fc7
cyrus-sasl-2.1.22-6


Faust


Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

pop3d 100% cpu

[Faust]

About 5 to 10 minutes after starting cyrus_imapd the process pop3d will take
100% cpu usage and the system will freece.

Anyone else has seen this problem?

I am using fedora 8 with the following packages;

cyrus-sasl-plain-2.1.22-6
cyrus-imapd-utils-2.3.9-7.fc7
cyrus-sasl-lib-2.1.22-6
cyrus-sasl-md5-2.1.22-6
cyrus-imapd-perl-2.3.9-7.fc7
cyrus-imapd-2.3.9-7.fc7
cyrus-sasl-2.1.22-6


Faust


Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

pop3d 100% cpu

[Faust]

About 5 to 10 minutes after starting cyrus_imapd the process pop3d will 
take
100% cpu usage and the system will freece.

Anyone else has seen this problem?

I am using fedora 8 with the following packages;

cyrus-sasl-plain-2.1.22-6
cyrus-imapd-utils-2.3.9-7.fc7
cyrus-sasl-lib-2.1.22-6
cyrus-sasl-md5-2.1.22-6
cyrus-imapd-perl-2.3.9-7.fc7
cyrus-imapd-2.3.9-7.fc7
cyrus-sasl-2.1.22-6


Faust

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: building cyrus 2.3.10 on x86_64

[Andrew Morgan]

On Sun, 18 Nov 2007, Gabor Gombas wrote:

> On Fri, Nov 16, 2007 at 05:03:39PM -0800, Andrew Morgan wrote:
>
>> I was able to get it to compile cleanly by adding "-fPIC" to the CFLAGS
>> definition in each Makefile.  I'm not sure if this is the correct solution
>> though!
>
> Yes, that's needed on x86_64 (at least until Cyrus starts providing
> shared libraries for libcyrus & libimap).

Just to follow up, I found the easy way to do this was to run configure 
like so:

   CFLAGS="-g -O2 -fPIC" ./configure

Much easier than editing a bunch of auto-generated Makefiles!

Andy

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: LARGE single-system Cyrus installs?

[John Madden]

> > Would'nt it be nice to have a configuration option to completely
> turn off 
> > fsync() in Cyrus? If you want, with a BIG WARNING in the doc stating
> NOT TO 
> > USE IT unless you know what you doing. :)
> 
> Its already in imapd.conf(8):
> 
> skiplist_unsafe

I see most of our writes going to the spool filesystems, not so much the
meta filesystem, so I'd prefer to see something where we can keep the
main databases fsnyc()ing properly but allow the individual mailboxes to
just rely on filesystem journaling.  Is there a
cacheandindexfile_unsafe? :)

John



-- 
John Madden
Sr. UNIX Systems Engineer
Ivy Tech Community College of Indiana
[EMAIL PROTECTED]


Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: LARGE single-system Cyrus installs?

[Rob Banz]

On Nov 20, 2007, at 14:57, Pascal Gienger wrote:

> Rob Banz <[EMAIL PROTECTED]> wrote:
>
>>
>> We went through a similar discussion last year in OpenAFS land, and
>> came the same conclusion -- basically, if your filesystem is
>> reasonably reliable (such as ZFS is), and you can trust your
>> underlying storage not to lose transactions that are in-cache  
>> during a
>> 'bad event', the added benefit of fsync() may be less than its
>> performance cost.
>
> Would'nt it be nice to have a configuration option to completely  
> turn off fsync() in Cyrus? If you want, with a BIG WARNING in the  
> doc stating NOT TO USE IT unless you know what you doing. :)

That's basically what we did with the AFS volserver & fileserver.   
Oddly, when the patch got integrated into the OpenAFS code, they  
didn't like the idea of it being an option and made it the default ;)



Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: LARGE single-system Cyrus installs?

[Rob Banz]

On Nov 20, 2007, at 15:38, Ken Murchison wrote:

> Pascal Gienger wrote:
>> Rob Banz <[EMAIL PROTECTED]> wrote:
>>> We went through a similar discussion last year in OpenAFS land, and
>>> came the same conclusion -- basically, if your filesystem is
>>> reasonably reliable (such as ZFS is), and you can trust your
>>> underlying storage not to lose transactions that are in-cache  
>>> during a
>>> 'bad event', the added benefit of fsync() may be less than its
>>> performance cost.
>> Would'nt it be nice to have a configuration option to completely  
>> turn off fsync() in Cyrus? If you want, with a BIG WARNING in the  
>> doc stating NOT TO USE IT unless you know what you doing. :)
>
> Its already in imapd.conf(8):
>
> skiplist_unsafe

Well shiver me timbers!  Ya'll rock.

-rob

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: LARGE single-system Cyrus installs?

[Ken Murchison]

Pascal Gienger wrote:
> Rob Banz <[EMAIL PROTECTED]> wrote:
> 
>> We went through a similar discussion last year in OpenAFS land, and
>> came the same conclusion -- basically, if your filesystem is
>> reasonably reliable (such as ZFS is), and you can trust your
>> underlying storage not to lose transactions that are in-cache during a
>> 'bad event', the added benefit of fsync() may be less than its
>> performance cost.
> 
> Would'nt it be nice to have a configuration option to completely turn off 
> fsync() in Cyrus? If you want, with a BIG WARNING in the doc stating NOT TO 
> USE IT unless you know what you doing. :)

Its already in imapd.conf(8):

skiplist_unsafe

-- 
Kenneth Murchison
Systems Programmer
Project Cyrus Developer/Maintainer
Carnegie Mellon University

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Login as user x?

[Simon Matter]

> All-
>   When I migrated off UW-IMAP, I discovered that I could login into a
> users account as the user, using my admin login/pass and passing the
> username as something like "$adminuser#$username". Does Cyrus have any
> similar functionality? The reason being that managing our blackberries
> with BIS2.0 is rapidly becoming untenable as the person has to around to
> login to our Cyrus server for us. If we had the ability to login as an
> admin user, we'd never have to schedule them to come in for service.

You may want to check the proxyservers parameter in imapd.conf. With it
you can authenticate as proxyuser and authorize as user. I have
"proxyservers: proxyadm" on my server and use it with imapsync for
migrations.

Regards,
Simon


Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: LARGE single-system Cyrus installs?

[Pascal Gienger]

Rob Banz <[EMAIL PROTECTED]> wrote:

>
> We went through a similar discussion last year in OpenAFS land, and
> came the same conclusion -- basically, if your filesystem is
> reasonably reliable (such as ZFS is), and you can trust your
> underlying storage not to lose transactions that are in-cache during a
> 'bad event', the added benefit of fsync() may be less than its
> performance cost.

Would'nt it be nice to have a configuration option to completely turn off 
fsync() in Cyrus? If you want, with a BIG WARNING in the doc stating NOT TO 
USE IT unless you know what you doing. :)



Pascal (in train of reconfiguring our SAN to make more cyrus checks)

PS: Putting deliver.db on tempfs seems to be a nice idea, but in current 
cyrus code you may not give extra paths to single cyrus databases. Our 
actual deliver.db on one machine is ca 600 MB tall, so it won't be of any 
problem to store it completely on tmpfs.


Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: LARGE single-system Cyrus installs?

[David Lang]

On Tue, 20 Nov 2007, Ian G Batten wrote:

> On 20 Nov 07, at 1332, Michael R. Gettes wrote:
>
>> I am wondering about the use of fsync() on journal'd file systems
>> as described below.  Shouldn't there be much less use of (or very
>> little use) of fsync() on these types of systems?  Let the journal
>> layer due its job and not force it within cyrus?  This would likely
>> save a lot of system overhead.
>
> fsync() forces the data to be queued to the disk.  A journaling
> filesystem won't usually make any difference, because no one wants to
> keep an intent log of every 1 byte write, or the 100 overwrites of
> the same block.  If you want every write() to go to disk,
> immediately, the filesystem layout doesn't really matter: it's just a
> matter of disk bandwidth.  Journalling filesystems are more usually
> concerned with metadata consistency, so that the filesystem isn't
> actively corrupt if the music stops at the wrong point in a directory
> create or something.

however a fsync on a journaled filesystem just means the data needs to be 
written to the journal, it doesn't mean that the journal needs to be flushed to 
disk.

on ext3 if you have data=journaled then your data is in the journal as well and 
all that the system needs to do on a fsync is to write things to the journal (a 
nice sequential write), and everything is perfectly safe. if you have 
data=ordered (the default for most journaled filesystems) then your data isn't 
safe when the journal is written and two writes must happen on a fsync (one for 
the data, one for the metadata)

for cyrus you should have the same sort of requirements that you would have for 
a database server, including the fact that without a battery-backed disk cache 
(or solid state drive) to handle your updates, you end up being throttled by 
your disk rotation rate (you can only do a single fsync write per rotation, and 
that good only if you don't have to seek), RAID 5/6 arrays are even worse, as 
almost all systems will require a read of the entire stripe before writing a 
single block (and it's parity block) back out, and since the stripe is 
frequently larger then the OS readahead, the OS throws much of the data away 
immediatly.

if we can identify the files that are the bottlenecks it would be very 
interesting to see the result of puttng them on a solid-state drive.

David Lang

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Login as user x?

[Alain Spineux]

On Nov 20, 2007 7:34 PM, Zachariah Mully <[EMAIL PROTECTED]> wrote:
> All-
> When I migrated off UW-IMAP, I discovered that I could login into a
> users account as the user, using my admin login/pass and passing the
> username as something like "$adminuser#$username". Does Cyrus have any
> similar functionality?

yes, you can try something like

# imtest -a [EMAIL PROTECTED] -w adminpassword -u [EMAIL PROTECTED]  -v 
localhost


> The reason being that managing our blackberries
> with BIS2.0 is rapidly becoming untenable as the person has to around to
> login to our Cyrus server for us. If we had the ability to login as an
> admin user, we'd never have to schedule them to come in for service.
>
> Thanks,
> Z
>
> 
> Cyrus Home Page: http://cyrusimap.web.cmu.edu/
> Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
> List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
>



-- 
Alain Spineux
aspineux gmail com
May the sources be with you

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: LARGE single-system Cyrus installs?

[Rob Banz]

We went through a similar discussion last year in OpenAFS land, and  
came the same conclusion -- basically, if your filesystem is  
reasonably reliable (such as ZFS is), and you can trust your  
underlying storage not to lose transactions that are in-cache during a  
'bad event', the added benefit of fsync() may be less than its  
performance cost.

-rob

On Nov 20, 2007, at 08:32, Michael R. Gettes wrote:

> I am wondering about the use of fsync() on journal'd file systems
> as described below.  Shouldn't there be much less use of (or very
> little use) of fsync() on these types of systems?  Let the journal
> layer due its job and not force it within cyrus?  This would likely
> save a lot of system overhead.  It makes sense to use on non-journal'd
> fs.  I also wonder whether modern arrays even respect FULLFSYNC
> given their more complex nature and I/O scheduling algorithms.  It
> may be time that fsync() (and fcntl(F_FULLFSYNC)) have become moot
> since there is likely little way to influence, in an effective
> targeted way, I/O behavior in complex environments these days.


Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Login as user x?

[Zachariah Mully]

All-
When I migrated off UW-IMAP, I discovered that I could login into a
users account as the user, using my admin login/pass and passing the
username as something like "$adminuser#$username". Does Cyrus have any
similar functionality? The reason being that managing our blackberries
with BIS2.0 is rapidly becoming untenable as the person has to around to
login to our Cyrus server for us. If we had the ability to login as an
admin user, we'd never have to schedule them to come in for service.

Thanks,
Z


Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Bad problems with 2.3.10 on FreeBSD

[Rich Wales]

I recently upgraded my system to 2.3.10.  My main Cyrus server runs
FreeBSD 6.2, replicating to a second server running Ubuntu 7.10.  I
had decided to upgrade now from 2.3.9 to 2.3.10 because of replication
"bailing out" errors in 2.3.9 that are reportedly fixed in 2.3.10.

At first, I was having lots of seg-fault errors, but I added the
getgrouplist() patch to auth_unix.c on my FreeBSD system, and that
seemed to take care of the core-dumps.

However, I started seeing other problems on the FreeBSD box -- such
as "ctl_cyrusdb -c" checkpoint jobs hanging, and other mail delivery
attempts hanging (or timing out) and piling up in my Postfix queue.
I had to restart Cyrus frequently on the FreeBSD system in order to
clear the logjams.

I finally gave up and went back to 2.3.9 on both servers.  Given the
instability of 2.3.10 on my main server, I felt I had no choice.

Any ideas what might have been happening?  If I can't figure out how
to get 2.3.10 running stably, I assume I'm going to either have to
stick with 2.3.9 (and figure out some other way around the replication
bailout errors until 2.3.11 comes out), or else move my main Cyrus
operations onto some other platform (maybe another Ubuntu box).

-- 
Rich Wales  ===  Palo Alto, CA, USA  === [EMAIL PROTECTED]
http://www.richw.org   ===   http://en.wikipedia.org/wiki/User:Richwales
"The difference between theory and practice is that, in theory,
theory and practice are identical -- whereas in practice, they aren't."

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Maybe too much of a good thing?

[Sebastian Hagedorn]

--On 20. November 2007 11:08:30 -0500 Ken Murchison <[EMAIL PROTECTED]> 
wrote:



OK, let me know what you find out.


Working on it.


I didn't change the logic if/when SSL_accept() fails, because if its an
SSL_wrapped process, there is nothing to fall back on (the application
protocol hasn't started yet).  Perhaps your dial-in clients take longer
than 3 minutes to complete the handshake.


No, because it fails *immediately*.


Hmm, from this particular log, I don't see the debug message tellinng us
that we're waiting for more input.  When I test locally (over localhost),
I always get at least in "-> waiting" log message before the "-> done"
messsage.


Sure, for the most part I get those as well. But there are exceptions. And 
of course in principle SSL_get_error() can return something unexpected the 
first time around. So that's what I'll be looking for.

--
.:.Sebastian Hagedorn - RZKR-R1 (Gebäude 52), Zimmer 18.:.
Zentrum für angewandte Informatik - Universitätsweiter Service RRZK
.:.Universität zu Köln / Cologne University - ✆ +49-221-478-5587.:.
  .:.:.:.Skype: shagedorn.:.:.:.

pgpWjBeSg04KI.pgp
Description: PGP signature

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Maybe too much of a good thing?

[Ken Murchison]

Sebastian Hagedorn wrote:
> Well,
> 
> the new patch works as intended (processes time out yet remain 
> straceable), but looks like it might be overzealous:
> 
> Nov 20 16:46:30 lvr13 pop3s[25622]: accepted connection
> Nov 20 16:46:30 lvr13 pop3s[25622]: error or timeout in SSL_accept() -> 
> done
> Nov 20 16:46:30 lvr13 pop3s[25622]: pop3s failed: ug-out-1314.google.com 
> [66.249.92.171]
> Nov 20 16:46:30 lvr13 pop3s[25622]: Fatal error: tls_start_servertls() 
> failed
> Nov 20 16:46:30 lvr13 master[32385]: process 25622 exited, status 75
> Nov 20 16:46:30 lvr13 master[32385]: service pop3s pid 25622 in BUSY 
> state: terminated abnormally
> 
> I understand that the timeout is not the only possible error condition, 
> but failing immediately seems strange. There are definitely more "error 
> or timeout in SSL_accept() -> done" messages than there used to be stuck 
> processes! Perhaps we should log the error code returned by 
> SSL_get_error()? I think I'll add some more logging locally.

OK, let me know what you find out.

I didn't change the logic if/when SSL_accept() fails, because if its an 
SSL_wrapped process, there is nothing to fall back on (the application 
protocol hasn't started yet).  Perhaps your dial-in clients take longer 
than 3 minutes to complete the handshake.

Hmm, from this particular log, I don't see the debug message tellinng us 
that we're waiting for more input.  When I test locally (over 
localhost), I always get at least in "-> waiting" log message before the 
"-> done" messsage.

-- 
Kenneth Murchison
Systems Programmer
Project Cyrus Developer/Maintainer
Carnegie Mellon University

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Maybe too much of a good thing?

[Sebastian Hagedorn]

Well,

the new patch works as intended (processes time out yet remain straceable), 
but looks like it might be overzealous:


Nov 20 16:46:30 lvr13 pop3s[25622]: accepted connection
Nov 20 16:46:30 lvr13 pop3s[25622]: error or timeout in SSL_accept() -> done
Nov 20 16:46:30 lvr13 pop3s[25622]: pop3s failed: ug-out-1314.google.com 
[66.249.92.171]
Nov 20 16:46:30 lvr13 pop3s[25622]: Fatal error: tls_start_servertls() 
failed

Nov 20 16:46:30 lvr13 master[32385]: process 25622 exited, status 75
Nov 20 16:46:30 lvr13 master[32385]: service pop3s pid 25622 in BUSY state: 
terminated abnormally


I understand that the timeout is not the only possible error condition, but 
failing immediately seems strange. There are definitely more "error or 
timeout in SSL_accept() -> done" messages than there used to be stuck 
processes! Perhaps we should log the error code returned by 
SSL_get_error()? I think I'll add some more logging locally.

--
Sebastian Hagedorn - Postmaster - RZKR-R1 (Gebäude 52), Zimmer 18
Zentrum für angewandte Informatik - Universitätsweiter Service RRZK
Universität zu Köln / Cologne University - Tel. +49-221-478-5587
Skype: shagedorn

pgpXc94aPjRDt.pgp
Description: PGP signature

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: LARGE single-system Cyrus installs?

[Ian G Batten]

On 20 Nov 07, at 1332, Michael R. Gettes wrote:

> I am wondering about the use of fsync() on journal'd file systems
> as described below.  Shouldn't there be much less use of (or very
> little use) of fsync() on these types of systems?  Let the journal
> layer due its job and not force it within cyrus?  This would likely
> save a lot of system overhead.

fsync() forces the data to be queued to the disk.  A journaling  
filesystem won't usually make any difference, because no one wants to  
keep an intent log of every 1 byte write, or the 100 overwrites of  
the same block.  If you want every write() to go to disk,  
immediately, the filesystem layout doesn't really matter: it's just a  
matter of disk bandwidth.  Journalling filesystems are more usually  
concerned with metadata consistency, so that the filesystem isn't  
actively corrupt if the music stops at the wrong point in a directory  
create or something.

ian


Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

RE: problem with timsieved 2.3.9

[Peter Nerád]

Thanks... a forgot to check the port 2000 on my machine.

tcp0  0 0.0.0.0:20000.0.0.0:*
LISTEN  10139/asterisk
tcp0  0 :::2000 :::*
LISTEN  11723/cyrus-master

so i stoped asterisk...
i need to reconfigure asterisk instalation to use another port. 

Thank you very much..


-Original Message-
From: Alain Spineux [mailto:[EMAIL PROTECTED] 
Sent: Tuesday, November 20, 2007 4:10 PM
To: [EMAIL PROTECTED]
Cc: info-cyrus@lists.andrew.cmu.edu
Subject: Re: problem with timsieved 2.3.9

Can you test this ?

# netstat -a -n -p  | grep 2000
tcp0  0 127.0.0.1:2000  0.0.0.0:*
 LISTEN  9436/cyrmaster

Did you check you cyrus.conf file ?

On Nov 20, 2007 3:39 PM, Peter Nerád <[EMAIL PROTECTED]> wrote:
>
>
>
>
> Hi
>
> I have a strange problem. After upgrade to cyrus 2.3.9 I can't login to
> timsieved server. Old sieve scripts works fine but I can't add new ones.
> telnet localhost sieve, or sievtest -a root -u root localhost not working.
I
> have no response on these commands from timsieved. There are no error
> messages in logs. I can login to imapd or pop3d, so auth woks fine.
>
> Thanks for any help.
>
>
>
> This is my imapd.conf
>
>
>
> configdirectory: /var/lib/imap
>
> partition-default: /var/spool/imap
>
> admins: cyrus root
>
> allowplaintext: true
>
> sievedir: /var/lib/imap/sieve
>
> sendmail: /usr/sbin/sendmail
>
> hashimapspool: true
>
> sasl_pwcheck_method: saslauthd
>
> sasl_mech_list: PLAIN
>
> #tls_cert_file: /etc/pki/cyrus-imapd/cyrus-imapd.pem
>
> #tls_key_file: /etc/pki/cyrus-imapd/cyrus-imapd.pem
>
> #tls_ca_file: /etc/pki/tls/certs/ca-bundle.crt
>
>
>
>
>
> Sorry for my English.
>
>
>
>
> 
> Cyrus Home Page: http://cyrusimap.web.cmu.edu/
> Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
> List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
>



-- 
Alain Spineux
aspineux gmail com
May the sources be with you


Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: problem with timsieved 2.3.9

[Patrick T. Tsang]

Hello,

Can you try this?

...
sasl_mech_list: PLAIN LOGIN
allowplaintext: yes
pwcheck_method: PLAIN
...



  - Original Message - 
  From: Peter Nerád 
  To: info-cyrus@lists.andrew.cmu.edu 
  Sent: Tuesday, November 20, 2007 10:39 PM
  Subject: problem with timsieved 2.3.9


  Hi

  I have a strange problem. After upgrade to cyrus 2.3.9 I can't login to 
timsieved server. Old sieve scripts works fine but I can't add new ones. telnet 
localhost sieve, or sievtest -a root -u root localhost not working. I have no 
response on these commands from timsieved. There are no error messages in logs. 
I can login to imapd or pop3d, so auth woks fine.

  Thanks for any help.

   

  This is my imapd.conf

   

  configdirectory: /var/lib/imap

  partition-default: /var/spool/imap

  admins: cyrus root

  allowplaintext: true

  sievedir: /var/lib/imap/sieve

  sendmail: /usr/sbin/sendmail

  hashimapspool: true

  sasl_pwcheck_method: saslauthd

  sasl_mech_list: PLAIN

  #tls_cert_file: /etc/pki/cyrus-imapd/cyrus-imapd.pem

  #tls_key_file: /etc/pki/cyrus-imapd/cyrus-imapd.pem

  #tls_ca_file: /etc/pki/tls/certs/ca-bundle.crt

   

   

  Sorry for my English.

   

   



--


  
  Cyrus Home Page: http://cyrusimap.web.cmu.edu/
  Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
  List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

RE: problem with timsieved 2.3.9

[Peter Nerád]

Yes i can...

My problem was not solved...

I still have no answer from timsieved server.

 

  _  

From: Patrick T. Tsang [mailto:[EMAIL PROTECTED] 
Sent: Tuesday, November 20, 2007 4:02 PM
To: [EMAIL PROTECTED]; info-cyrus@lists.andrew.cmu.edu
Subject: Re: problem with timsieved 2.3.9

 

Hello,

 

Can you try this?

 

...

sasl_mech_list: PLAIN LOGIN
allowplaintext: yes
pwcheck_method: PLAIN
...

 

 

 

- Original Message - 

From: Peter Ner  ád 

To: info-cyrus@lists.andrew.cmu.edu 

Sent: Tuesday, November 20, 2007 10:39 PM

Subject: problem with timsieved 2.3.9

 

Hi

I have a strange problem. After upgrade to cyrus 2.3.9 I can't login to
timsieved server. Old sieve scripts works fine but I can't add new ones.
telnet localhost sieve, or sievtest -a root -u root localhost not working. I
have no response on these commands from timsieved. There are no error
messages in logs. I can login to imapd or pop3d, so auth woks fine.

Thanks for any help.

 

This is my imapd.conf

 

configdirectory: /var/lib/imap

partition-default: /var/spool/imap

admins: cyrus root

allowplaintext: true

sievedir: /var/lib/imap/sieve

sendmail: /usr/sbin/sendmail

hashimapspool: true

sasl_pwcheck_method: saslauthd

sasl_mech_list: PLAIN

#tls_cert_file: /etc/pki/cyrus-imapd/cyrus-imapd.pem

#tls_key_file: /etc/pki/cyrus-imapd/cyrus-imapd.pem

#tls_ca_file: /etc/pki/tls/certs/ca-bundle.crt

 

 

Sorry for my English.

 

 


  _  



Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html


Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: problem with timsieved 2.3.9

[Alain Spineux]

Can you test this ?

# netstat -a -n -p  | grep 2000
tcp0  0 127.0.0.1:2000  0.0.0.0:*
 LISTEN  9436/cyrmaster

Did you check you cyrus.conf file ?

On Nov 20, 2007 3:39 PM, Peter Nerád <[EMAIL PROTECTED]> wrote:
>
>
>
>
> Hi
>
> I have a strange problem. After upgrade to cyrus 2.3.9 I can't login to
> timsieved server. Old sieve scripts works fine but I can't add new ones.
> telnet localhost sieve, or sievtest –a root –u root localhost not working. I
> have no response on these commands from timsieved. There are no error
> messages in logs. I can login to imapd or pop3d, so auth woks fine.
>
> Thanks for any help.
>
>
>
> This is my imapd.conf
>
>
>
> configdirectory: /var/lib/imap
>
> partition-default: /var/spool/imap
>
> admins: cyrus root
>
> allowplaintext: true
>
> sievedir: /var/lib/imap/sieve
>
> sendmail: /usr/sbin/sendmail
>
> hashimapspool: true
>
> sasl_pwcheck_method: saslauthd
>
> sasl_mech_list: PLAIN
>
> #tls_cert_file: /etc/pki/cyrus-imapd/cyrus-imapd.pem
>
> #tls_key_file: /etc/pki/cyrus-imapd/cyrus-imapd.pem
>
> #tls_ca_file: /etc/pki/tls/certs/ca-bundle.crt
>
>
>
>
>
> Sorry for my English.
>
>
>
>
> 
> Cyrus Home Page: http://cyrusimap.web.cmu.edu/
> Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
> List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
>



-- 
Alain Spineux
aspineux gmail com
May the sources be with you

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: One more attempt: stuck processes

[Sebastian Hagedorn]

--On 20. November 2007 15:59:18 +0100 Sebastian Hagedorn 
<[EMAIL PROTECTED]> wrote:



I can fix this myself, but it's probably easier if you do it.


Just FYI: I fixed it locally with a 3 minute timeout and it compiled fine. 
I'll start testing it now.

--
.:.Sebastian Hagedorn - RZKR-R1 (Gebäude 52), Zimmer 18.:.
Zentrum für angewandte Informatik - Universitätsweiter Service RRZK
.:.Universität zu Köln / Cologne University - ✆ +49-221-478-5587.:.
  .:.:.:.Skype: shagedorn.:.:.:.

pgpoLn6l11ZH7.pgp
Description: PGP signature

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: One more attempt: stuck processes

[Sebastian Hagedorn]

--On 20. November 2007 09:20:42 -0500 Ken Murchison <[EMAIL PROTECTED]> 
wrote:



OK.  Can you both try this alternate patch?  It should be portable, and
GDB shouldn't cause it to kick out.  I've set it up so that for
SSL-wrapped services it will timeout after 3 minutes, otherwise it uses
the service-specific timeout.


Thanks for working on this. I think you missed a spot:

gcc -c -I.. -I./../sieve -I./../imap -I./../lib  -I/usr/include/db4 
-I/usr/kerberos/include   -I/usr/include/et  -DHAVE_CONFIG_H 
-I/usr/kerberos/include   -O2 -g -pipe -march=i386 -mcpu=i686 -fPIC \

parser.c
parser.c: In function `cmd_starttls':
parser.c:834: warning: passing arg 3 of `tls_start_servertls' makes integer 
from pointer without a cast
parser.c:834: warning: passing arg 4 of `tls_start_servertls' from 
incompatible pointer type
parser.c:834: warning: passing arg 5 of `tls_start_servertls' from 
incompatible pointer type

parser.c:834: too few arguments to function `tls_start_servertls'

I can fix this myself, but it's probably easier if you do it.
--
.:.Sebastian Hagedorn - RZKR-R1 (Gebäude 52), Zimmer 18.:.
Zentrum für angewandte Informatik - Universitätsweiter Service RRZK
.:.Universität zu Köln / Cologne University - ✆ +49-221-478-5587.:.
  .:.:.:.Skype: shagedorn.:.:.:.

pgpImfrnNAhAO.pgp
Description: PGP signature

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: One more attempt: stuck processes

[Ken Murchison]

Gary Mills wrote:

On Mon, Nov 19, 2007 at 12:35:46PM -0500, Ken Murchison wrote:

Sebastian Hagedorn wrote:
-- Ken Murchison <[EMAIL PROTECTED]> is rumored to have mumbled on 
17. November 2007 11:21:38 -0500 regarding Re: One more attempt: stuck 
processes:



Here's a patch that seems to fix the problem.  I did some basic testing
(Linux only) to make sure that it doesn't break anything else, but its
always possible that it has some unforseen side effects.  Keep an eye on
it and let me know if you see anything unusual.

Thanks, it seems to be working fine so far! Enjoy your weekend now ...



Gary, have you tried the patch?


On Solaris 9, SO_RCVTIMEO is not mentioned in the setsockopt man page.
My reading tells me that it is defined in the header file, but if it's
actually used, setsockopt() will return an error.  I understand that
this is the case for other operating systems too.  This seems to be a
known problem with setsockopt().  I just checked the Opensolaris source;
it does function there.  This is the code:

 case SO_RCVTIMEO:
if (optlen == sizeof (uint32_t))
  sockets[i].in_timeout = *(uint32_t *)optval;
else {
errno = EINVAL;
}
break;

Note that the option has to be an int, and the length has to be
that of an int as well.  Linux wants a `struct timeval'.  I wonder
if there's a standard in this area?


OK.  Can you both try this alternate patch?  It should be portable, and 
GDB shouldn't cause it to kick out.  I've set it up so that for 
SSL-wrapped services it will timeout after 3 minutes, otherwise it uses 
the service-specific timeout.


--
Kenneth Murchison
Systems Programmer
Project Cyrus Developer/Maintainer
Carnegie Mellon University
Index: imapd.c
===
RCS file: /afs/andrew/system/cvs/src/cyrus/imap/imapd.c,v
retrieving revision 1.535
diff -u -r1.535 imapd.c
--- imapd.c	9 Nov 2007 12:54:22 -	1.535
+++ imapd.c	20 Nov 2007 14:15:24 -
@@ -148,6 +148,7 @@
 /* end PROXY STUFF */
 
 /* per-user/session state */
+int imapd_timeout;
 struct protstream *imapd_out = NULL;
 struct protstream *imapd_in = NULL;
 struct protgroup *protin = NULL;
@@ -701,7 +702,6 @@
 #endif
 {
 socklen_t salen;
-int timeout;
 sasl_security_properties_t *secprops = NULL;
 struct sockaddr_storage imapd_localaddr, imapd_remoteaddr;
 char localip[60], remoteip[60];
@@ -778,9 +778,10 @@
 proc_register("imapd", imapd_clienthost, NULL, NULL);
 
 /* Set inactivity timer */
-timeout = config_getint(IMAPOPT_TIMEOUT);
-if (timeout < 30) timeout = 30;
-prot_settimeout(imapd_in, timeout*60);
+imapd_timeout = config_getint(IMAPOPT_TIMEOUT);
+if (imapd_timeout < 30) imapd_timeout = 30;
+imapd_timeout *= 60;
+prot_settimeout(imapd_in, imapd_timeout);
 prot_setflushonread(imapd_in, imapd_out);
 
 /* we were connected on imaps port so we should do 
@@ -6626,6 +6627,7 @@
   
 result=tls_start_servertls(0, /* read */
 			   1, /* write */
+			   imaps ? 180 : imapd_timeout,
 			   layerp,
 			   &auth_id,
 			   &tls_conn);
Index: lmtpengine.c
===
RCS file: /afs/andrew/system/cvs/src/cyrus/imap/lmtpengine.c,v
retrieving revision 1.122
diff -u -r1.122 lmtpengine.c
--- lmtpengine.c	10 Oct 2007 15:14:39 -	1.122
+++ lmtpengine.c	20 Nov 2007 14:15:25 -
@@ -1579,6 +1579,7 @@
   
 		r=tls_start_servertls(0, /* read */
   1, /* write */
+  360, /* 6 minutes */
   layerp,
   &auth_id,
   &(cd.tls_conn));
Index: mupdate.c
===
RCS file: /afs/andrew/system/cvs/src/cyrus/imap/mupdate.c,v
retrieving revision 1.99
diff -u -r1.99 mupdate.c
--- mupdate.c	10 Oct 2007 15:14:39 -	1.99
+++ mupdate.c	20 Nov 2007 14:15:25 -
@@ -1941,6 +1941,7 @@
   
 result=tls_start_servertls(C->pin->fd, /* read */
 			   C->pout->fd, /* write */
+			   180, /* 3 minutes */
 			   layerp,
 			   &auth_id,
 			   &C->tlsconn);
Index: nntpd.c
===
RCS file: /afs/andrew/system/cvs/src/cyrus/imap/nntpd.c,v
retrieving revision 1.60
diff -u -r1.60 nntpd.c
--- nntpd.c	10 Oct 2007 15:14:39 -	1.60
+++ nntpd.c	20 Nov 2007 14:15:25 -
@@ -142,6 +142,7 @@
 
 sasl_conn_t *nntp_saslconn; /* the sasl connection context */
 
+int nntp_timeout;
 char newsprefix[100] = "";
 char *nntp_userid = 0, *newsmaster;
 struct auth_state *nntp_authstate = 0, *newsmaster_authstate;
@@ -523,7 +524,6 @@
 char localip[60], remoteip[60];
 char hbuf[NI_MAXHOST];
 int niflags;
-int timeout;
 sasl_security_properties_t *secprops=NULL;
 char unavail[1024];
 
@@ -590,9 +590,10 @@
 proc_register("nntpd", nntp_clienthost, NULL, NULL);
 
 /* 

problem with timsieved 2.3.9

[Peter Nerád]

Hi

I have a strange problem. After upgrade to cyrus 2.3.9 I can't login to
timsieved server. Old sieve scripts works fine but I can't add new ones.
telnet localhost sieve, or sievtest -a root -u root localhost not working. I
have no response on these commands from timsieved. There are no error
messages in logs. I can login to imapd or pop3d, so auth woks fine.

Thanks for any help.

 

This is my imapd.conf

 

configdirectory: /var/lib/imap

partition-default: /var/spool/imap

admins: cyrus root

allowplaintext: true

sievedir: /var/lib/imap/sieve

sendmail: /usr/sbin/sendmail

hashimapspool: true

sasl_pwcheck_method: saslauthd

sasl_mech_list: PLAIN

#tls_cert_file: /etc/pki/cyrus-imapd/cyrus-imapd.pem

#tls_key_file: /etc/pki/cyrus-imapd/cyrus-imapd.pem

#tls_ca_file: /etc/pki/tls/certs/ca-bundle.crt

 

 

Sorry for my English.

 

 


Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: LARGE single-system Cyrus installs?

[Michael R. Gettes]

I am wondering about the use of fsync() on journal'd file systems
as described below.  Shouldn't there be much less use of (or very
little use) of fsync() on these types of systems?  Let the journal
layer due its job and not force it within cyrus?  This would likely
save a lot of system overhead.  It makes sense to use on non-journal'd
fs.  I also wonder whether modern arrays even respect FULLFSYNC
given their more complex nature and I/O scheduling algorithms.  It
may be time that fsync() (and fcntl(F_FULLFSYNC)) have become moot
since there is likely little way to influence, in an effective
targeted way, I/O behavior in complex environments these days.

/mrg

On Nov 19, 2007, at 23:40, Andrew McNamara wrote:

>>> In production releases of ZFS fsync() essentially triggers sync()  
>>> (fixed in
>>> Solaris Next).
> [...]
>> Skiplist requires two fsync calls per transaction (single
>> untransactioned actions are also one transaction), and it
>> also locks the entire file for the duration of said
>> transaction, so you can't have two writes happening at
>> once.  I haven't built Cyrus on our Solaris box, so I don't
>> know if it uses fcntl there, it certainly does on the Linux
>> systems, but it can fall back to flock if fcntl isn't
>> available.
>
> Note that ext3 effectively does the same thing as ZFS on fsync() -  
> because
> the journal layer is block based and does no know which block belongs
> to which file, the entire journal must be applied to the filesystem to
> achieve the expected fsync() symantics (at least, with data=ordered,
> it does).
>
> -- 
> Andrew McNamara, Senior Developer, Object Craft
> http://www.object-craft.com.au/
> 
> Cyrus Home Page: http://cyrusimap.web.cmu.edu/
> Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
> List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html


Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: 2.3.10 Upgrade Question

[Bron Gondwana]

On Tue, 20 Nov 2007 11:54:24 +, "Ian G Batten" <[EMAIL PROTECTED]> said:
> 
> On 20 Nov 07, at 1146, Bron Gondwana wrote:
> 
> >
> > The index files are pretty small, and they rebuild fast :)  They  
> > get streamed
> > into new copies every single expunge anyway.
> 
> What's involved in the rebuild?  I have users with tens of thousands  
> of messages in a single mailbox, so delaying that open while it reads  
> and indexes a few gigs is bad.  Does the rebuild need to open every  
> message?  Read headers?  Bodies?  What?

No - thankfully everything you need is in the index record itself.
If you want the new sha1 GUIDs you need to reconstruct with the -G
flag, but that's not required for the upgrade.  The upgrade just pads
the GUID field with zeros.

It's really quick, we had folders with 200k messages in them, and the
upgrade was only a couple of seconds on a pretty busy server.  It's still
only 96 bytes per record or something, so that's pretty much streaming what,
18Mb or something - hardly giant given that it's all sequential reads and
writes, and the CPU isn't doing much work.

Bron.
-- 
  Bron Gondwana
  [EMAIL PROTECTED]


Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: 2.3.10 Upgrade Question

[Ian G Batten]

On 20 Nov 07, at 1146, Bron Gondwana wrote:

>
> The index files are pretty small, and they rebuild fast :)  They  
> get streamed
> into new copies every single expunge anyway.

What's involved in the rebuild?  I have users with tens of thousands  
of messages in a single mailbox, so delaying that open while it reads  
and indexes a few gigs is bad.  Does the rebuild need to open every  
message?  Read headers?  Bodies?  What?

ian

>
> Also, it will only upgrade each index as it is opened for the first  
> time, so
> the load isn't one giant hit.
>
> Bron.
> -- 
>   Bron Gondwana
>   [EMAIL PROTECTED]
>


Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: 2.3.10 Upgrade Question

[Bron Gondwana]

On Tue, 20 Nov 2007 12:05:43 +0100 (CET), "Simon Matter" <[EMAIL PROTECTED]> 
said:
> > In a NON-replicated setup, do the changes to the GUID have an
> > impact?  Can I just put 2.3.10 on with a quick restart of the
> > mailsystem, or is there More To It?
> >
> > I have 1.7TB of mail, about 40K mailboxes, about 10 million pieces of
> > mail.  So I don't want to do an upgrade which will kick off some huge
> > rebuild-fest without planning.
> 
> I have a server with ~ half the size and upgrade has worked without doing
> anything special.

The index files are pretty small, and they rebuild fast :)  They get streamed
into new copies every single expunge anyway.

Also, it will only upgrade each index as it is opened for the first time, so
the load isn't one giant hit.

Bron.
-- 
  Bron Gondwana
  [EMAIL PROTECTED]


Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: 2.3.10 Upgrade Question

[Simon Matter]

> In a NON-replicated setup, do the changes to the GUID have an
> impact?  Can I just put 2.3.10 on with a quick restart of the
> mailsystem, or is there More To It?
>
> I have 1.7TB of mail, about 40K mailboxes, about 10 million pieces of
> mail.  So I don't want to do an upgrade which will kick off some huge
> rebuild-fest without planning.

I have a server with ~ half the size and upgrade has worked without doing
anything special.

Simon

>
> ian
>
> 
> Cyrus Home Page: http://cyrusimap.web.cmu.edu/
> Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
> List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
>



Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

2.3.10 Upgrade Question

[Ian G Batten]

In a NON-replicated setup, do the changes to the GUID have an  
impact?  Can I just put 2.3.10 on with a quick restart of the  
mailsystem, or is there More To It?

I have 1.7TB of mail, about 40K mailboxes, about 10 million pieces of  
mail.  So I don't want to do an upgrade which will kick off some huge  
rebuild-fest without planning.

ian


Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Vacation

[Ian G Batten]

On 19 Nov 07, at 2139, Scott Adkins wrote:

> This does bring up an important point... we had to disable duplicate
> suppression on our server because it was just causing too much  
> contention.
> When every single mail message being delivered has to check against  
> the
> database, it is a problem.

The checking's not the problem, as the 99.9% case is an update.  In  
other words, if the rate of duplicates were high, you could lock it  
for reading, check the message was unique, and then if that gave a  
positive response lock it for update and perform the update.  Yes,  
there's a race condition, but it's small and benign.

However, in the real world this gains you nothing, because the rate  
of duplicates is vanishingly small.  You buy complexity and a race  
condition for degraded performance (the lock upgrade is expensive).

There are other possible solutions.  One would be a per-user, or per- 
user-initial-letter, duplicate database.  Another would be to not  
bother syncing the database (I think this is what berkeley-nosync  
does, but that doesn't help skiplist users).

ian


Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Vacation

[Lars Schimmer]

Scott Adkins <[EMAIL PROTECTED]> wrote: 

> Why can't vacation messages, or better yet, sieve in general (since there
> might be a use for something mroe than vacation messages down the road?),
> get its own database and not tie it in so much to duplicate suppression?

Oh yes, there is some need for sieve.
We sort some mails out with sieve.
Me personal sort mails with sieve into IMAP folders in cyrus based on headers.
I use different IMAP clients on different PCs, and I don´t want to let the
client sort the mails.
For office use we sort on base of email adresses into public folders.
In the end we don´t use the vacation function at all ;-)

Cya
Lars Schimmer


Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html