Re: Cyrus (Sieve) and external programs

[David Lang]

FYI, the software I mentioned below is popfile, which seems to have stopped 
major development in 2011, but someone is still fixing it to work with later 
releases


http://getpopfile.org/

Even if you don't use this, you can look at how it works to build a replacement.

David Lang


On Tue, 29 May 2018, David Lang wrote:

several years ago I used a system that subscribed to the various folders and 
watched for messages to appear. Unfortunantly, that software was abandoned 
and I haven't gone looking for a replacement in the last several years.


but it did work very well, at very little load on the system (fastmail was a 
little concerned at seeing many connections from me, but once they learned 
that they were mostly idle watching for new message notifications, they 
relaxed a bit)


David Lang

On Tue, 29 May 2018, Pedro silva wrote:


Date: Tue, 29 May 2018 10:16:37 +0100
From: Pedro silva 
To: Sven Schwedas 
Cc: info-cyrus@lists.andrew.cmu.edu,
Info-cyrus 


Subject: Re: Cyrus (Sieve) and external programs

Hello Sven,

Thanks for your answer.

Actually I would like to set up a system were a user would move his
messages to a spam folder, and by doing so, the message would be sent to
the spam learning system.

Ive seen something like this in other distributions (eg.:
https://wiki.dovecot.org/HowTo/AntispamWithSieve) I was just wondering
if the same is possible with cyrus or cyrus sieve.

Best regards,

Pedro Silva

On 28-05-2018 15:59, Sven Schwedas wrote:


On 2018-05-28 16:41, Pedro silva wrote:


Hello,

I'm trying to set up a spam learning system, and I would like to pipe
email places in (for example) the SPAM folder to an external program.

I have cyrus 2.4.17

Does any one know how or if this is possible in cyrus (or sieve)?


If you want to (read only) process data already received to train your
spam filter, just use the individual mail files in cyrus' spool.

If you want to act on mails being received and possibly delete/redirect
them, you normally want hook into your MTA, not cyrus/sieve.

--
Mit freundlichen Grüßen, / Best Regards,
Sven Schwedas, Systemadministrator
✉ sven.schwe...@tao.at | ☎ +43 680 301 7167
TAO Digital   | Teil der TAO Beratungs- & Management GmbH
Lendplatz 45  | FN 213999f/Klagenfurt, FB-Gericht Villach
A8020 Graz| https://www.tao-digital.at


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Cyrus (Sieve) and external programs

[David Lang]

several years ago I used a system that subscribed to the various folders and 
watched for messages to appear. Unfortunantly, that software was abandoned and I 
haven't gone looking for a replacement in the last several years.


but it did work very well, at very little load on the system (fastmail was a 
little concerned at seeing many connections from me, but once they learned that 
they were mostly idle watching for new message notifications, they relaxed a 
bit)


David Lang

 On Tue, 29 May 
2018, Pedro silva wrote:



Date: Tue, 29 May 2018 10:16:37 +0100
From: Pedro silva 
To: Sven Schwedas 
Cc: info-cyrus@lists.andrew.cmu.edu,
Info-cyrus 
Subject: Re: Cyrus (Sieve) and external programs

Hello Sven,

Thanks for your answer.

Actually I would like to set up a system were a user would move his
messages to a spam folder, and by doing so, the message would be sent to
the spam learning system.

Ive seen something like this in other distributions (eg.:
https://wiki.dovecot.org/HowTo/AntispamWithSieve) I was just wondering
if the same is possible with cyrus or cyrus sieve.

Best regards,

Pedro Silva

On 28-05-2018 15:59, Sven Schwedas wrote:


On 2018-05-28 16:41, Pedro silva wrote:


Hello,

I'm trying to set up a spam learning system, and I would like to pipe
email places in (for example) the SPAM folder to an external program.

I have cyrus 2.4.17

Does any one know how or if this is possible in cyrus (or sieve)?


If you want to (read only) process data already received to train your
spam filter, just use the individual mail files in cyrus' spool.

If you want to act on mails being received and possibly delete/redirect
them, you normally want hook into your MTA, not cyrus/sieve.

--
Mit freundlichen Grüßen, / Best Regards,
Sven Schwedas, Systemadministrator
✉ sven.schwe...@tao.at | ☎ +43 680 301 7167
TAO Digital   | Teil der TAO Beratungs- & Management GmbH
Lendplatz 45  | FN 213999f/Klagenfurt, FB-Gericht Villach
A8020 Graz| https://www.tao-digital.at


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Cyrus 3.0.3 XAPPEPUSH

[David Lang]

on the contrary, thanks for sending it to the list, besides others who may want 
to do the exact same thing, it also helps others who may not have realized that 
such things could be done easily.


at 28k, it's not like it was a massive e-mail

David Lang


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: problem with one user after a crash

[David Lang]

attached is the compressed .seen file if anyone can salvage it.

David Lang

On Thu, 10 Jan 2013, David Lang wrote:


On Thu, 10 Jan 2013, Andrew Morgan wrote:


On Thu, 10 Jan 2013, David Lang wrote:


On Thu, 10 Jan 2013, Andrew Morgan wrote:

A corrupted seen file is the only thing that makes sense to me.  If other 
users can open the same folder, then the cyrus.header and cyrus.index 
files must be sane.


As an experiment, you could move your seen file from lang.seen (or 
whatever it's called) to lang.seen.bak.  Then connect to IMAP as yourself 
and try to open the folder.  If it works, then it must have been a 
corrupted seen file, and you can use skiplist.py to recover as much of it 
as possible.


Ok, the good news is that this seems to be the problem.

unfortunantly the skiplist recovery tool is not working.

# ./skiplist.py dlang.seen.bak >dlang.seen.txt
Traceback (most recent call last):
 File "./skiplist.py", line 172, in 
   values, keys = getkeys(fp)
 File "./skiplist.py", line 152, in getkeys
   spointer = unpack('>I', str_p)[0]
struct.error: unpack requires a string argument of length 4

# file dlang.seen.bak
dlang.seen.bak: Cyrus skiplist DB

I tried enabling debug mode in skiplist.py and I'm not seeing anything 
different. This confuses me. I'm not that familiar with python, but as I 
read the code, get_header() should be writing a bunch of stuff before it 
gets to the getkeys() section that failing.


Hmmm, I haven't looked at the code in skiplist.py much.  I have an older 
version of skiplist.py, which I have attached to this email.  Honestly, I 
haven't used this since I upgraded to Cyrus v2.3.something.  I think there 
were some bugs in skiplist on the older versions.  :)


Give the attached skiplist.py a shot!  Worst case, you'll have to start 
over with no Seen history.  :(


It dies as well. It turns out that it's sending the debug messages to stdout 
not stderr. It looks like it processes 96 sections before it dies.


with debug off I just get the error, with debug on I get a bunch of stuff 
that looks fairly similar, ending with:


[*] 
--

[INORDER] Record type INORDER
[INORDER] Key size 16 (16)
[INORDER] Key String 7536cca646f98c5d
[INORDER] Data size 259 (260)
[INORDER] Data String 1 1351316330 4755 1351316487 
1,285,528,596,807,811,984:1003,1005:1010,1012,1014,1016,1018:1019,1075,1239,1619,1821,1920,2409,2511:2513,2646,2713,2787,3076,3080:3081,3084:3085,3090,3124,3126,3148,3160,3385,4056,4108,4186,4221,4256,4320,4390,4401,4585,4618,4627

[INORDER] Skip pointer 16264
[INORDER] Total Skip pointers: 1
[*] 
--

[INORDER] Record type INORDER
[INORDER] Key size 16 (16)
[INORDER] Key String 782cc11f4fb291b5
[INORDER] Data size 127 (128)
[INORDER] Data String 1 1351316303 46217 1351316313 
1:13755,22847:22986,26559:26563,26565:26749,29906:29910,30316,

Traceback (most recent call last):
 File "/home/dlang/skiplist.py", line 167, in 
   values, keys = getkeys(fp)
 File "/home/dlang/skiplist.py", line 147, in getkeys
   spointer = unpack('>I', str_p)[0]
struct.error: unpack requires a string argument of length 4

with >24k messages in my inbox, I really hope I don't loose too much of my 
seen data. anyone else have suggestions?


David Lang


dlang.seen.bak.gz
Description: Binary data

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: problem with one user after a crash

[David Lang]

On Thu, 10 Jan 2013, Andrew Morgan wrote:


On Thu, 10 Jan 2013, David Lang wrote:


On Thu, 10 Jan 2013, Andrew Morgan wrote:

A corrupted seen file is the only thing that makes sense to me.  If other 
users can open the same folder, then the cyrus.header and cyrus.index 
files must be sane.


As an experiment, you could move your seen file from lang.seen (or 
whatever it's called) to lang.seen.bak.  Then connect to IMAP as yourself 
and try to open the folder.  If it works, then it must have been a 
corrupted seen file, and you can use skiplist.py to recover as much of it 
as possible.


Ok, the good news is that this seems to be the problem.

unfortunantly the skiplist recovery tool is not working.

# ./skiplist.py dlang.seen.bak >dlang.seen.txt
Traceback (most recent call last):
 File "./skiplist.py", line 172, in 
   values, keys = getkeys(fp)
 File "./skiplist.py", line 152, in getkeys
   spointer = unpack('>I', str_p)[0]
struct.error: unpack requires a string argument of length 4

# file dlang.seen.bak
dlang.seen.bak: Cyrus skiplist DB

I tried enabling debug mode in skiplist.py and I'm not seeing anything 
different. This confuses me. I'm not that familiar with python, but as I 
read the code, get_header() should be writing a bunch of stuff before it 
gets to the getkeys() section that failing.


Hmmm, I haven't looked at the code in skiplist.py much.  I have an older 
version of skiplist.py, which I have attached to this email.  Honestly, I 
haven't used this since I upgraded to Cyrus v2.3.something.  I think there 
were some bugs in skiplist on the older versions.  :)


Give the attached skiplist.py a shot!  Worst case, you'll have to start over 
with no Seen history.  :(


It dies as well. It turns out that it's sending the debug messages to stdout not 
stderr. It looks like it processes 96 sections before it dies.


with debug off I just get the error, with debug on I get a bunch of stuff that 
looks fairly similar, ending with:


[*] 
--

[INORDER] Record type INORDER
[INORDER] Key size 16 (16)
[INORDER] Key String 7536cca646f98c5d
[INORDER] Data size 259 (260)
[INORDER] Data String 1 1351316330 4755 1351316487 
1,285,528,596,807,811,984:1003,1005:1010,1012,1014,1016,1018:1019,1075,1239,1619,1821,1920,2409,2511:2513,2646,2713,2787,3076,3080:3081,3084:3085,3090,3124,3126,3148,3160,3385,4056,4108,4186,4221,4256,4320,4390,4401,4585,4618,4627

[INORDER] Skip pointer 16264
[INORDER] Total Skip pointers: 1
[*] 
--

[INORDER] Record type INORDER
[INORDER] Key size 16 (16)
[INORDER] Key String 782cc11f4fb291b5
[INORDER] Data size 127 (128)
[INORDER] Data String 1 1351316303 46217 1351316313 
1:13755,22847:22986,26559:26563,26565:26749,29906:29910,30316,

Traceback (most recent call last):
  File "/home/dlang/skiplist.py", line 167, in 
values, keys = getkeys(fp)
  File "/home/dlang/skiplist.py", line 147, in getkeys
spointer = unpack('>I', str_p)[0]
struct.error: unpack requires a string argument of length 4

with >24k messages in my inbox, I really hope I don't loose too much of my seen 
data. anyone else have suggestions?


David Lang#!/usr/bin/env python
# -*- Mode: Python; tab-width: 4 -*-
#
# Cyrus Imapd Skiplist db recovery tool
#
# Copyright (C) 2004 Gianluigi Tiesi 
# Copyright (C) 2004 NetFarm S.r.l.  [http://www.netfarm.it]
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by the
# Free Software Foundation; either version 2, or (at your option) any later
# version.
#
# This program is distributed in the hope that it will be useful, but
# WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTIBILITY
# or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
# for more details.
# ==

__version__= '0.1'
__doc__="""Cyrus skiplist db recover"""

from sys import argv,exit,stdout,stderr
from struct import unpack
from time import localtime, strftime

### User Conf
debug = 0
###

TIMEFMT ='%a, %d %b %Y %H:%M:%S %z'
PADDING = '\xff' * 4
INORDER = 1
ADD = 2
DELETE  = 4
COMMIT  = 255
DUMMY   = 257
HEADER  = -1
MAIN= -2

types = {
1:   'INORDER',
2:   'ADD',
4:   'DELETE',
255: 'COMMIT',
257: 'DUMMY',
-1:  'HEADER',
-2:  '*'
}

def log(rtype, text):
global debug
if debug:
out = '[%s] %s\n' % (types[rtype], text)
stdout.write(out)
stdout.flush()

def roundto4(value):
if value % 4:
return ((value / 4) + 1) * 4
return value

def g

Re: problem with one user after a crash

[David Lang]

On Thu, 10 Jan 2013, Andrew Morgan wrote:

> A corrupted seen file is the only thing that makes sense to me.  If other 
> users can open the same folder, then the cyrus.header and cyrus.index files 
> must be sane.
>
> As an experiment, you could move your seen file from lang.seen (or whatever 
> it's called) to lang.seen.bak.  Then connect to IMAP as yourself and try to 
> open the folder.  If it works, then it must have been a corrupted seen file, 
> and you can use skiplist.py to recover as much of it as possible.

Ok, the good news is that this seems to be the problem.

unfortunantly the skiplist recovery tool is not working.

# ./skiplist.py dlang.seen.bak >dlang.seen.txt
Traceback (most recent call last):
   File "./skiplist.py", line 172, in 
 values, keys = getkeys(fp)
   File "./skiplist.py", line 152, in getkeys
 spointer = unpack('>I', str_p)[0]
struct.error: unpack requires a string argument of length 4

# file dlang.seen.bak
dlang.seen.bak: Cyrus skiplist DB

I tried enabling debug mode in skiplist.py and I'm not seeing anything 
different. This confuses me. I'm not that familiar with python, but as I read 
the code, get_header() should be writing a bunch of stuff before it gets to the 
getkeys() section that failing.

David Lang

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: problem with one user after a crash

[David Lang]

On Thu, 10 Jan 2013, Andrew Morgan wrote:

> On Thu, 10 Jan 2013, David Lang wrote:
>
>> I has my home mail server crash, and after the crash, one user (me) is 
>> unable to
>> acess any folders.
>> 
>> When I manually telnet to the IMAP port, I can login, I can list and run 
>> other
>> commands, but as soon as I do a select of any folder (mine or any other 
>> shared
>> folder) I get disconnected.
>> 
>> Other users have no problems accessing the same folder.
>> 
>> This is with Cyrus 2.2 on Ubuntu (I need to upgrade, but have not had the 
>> time
>> to do so yet)
>> 
>> Any suggestions on what may be wrong and how to diagnose this?
>
> Check your syslog files, whichever one Cyrus is logging to.  I suspect you'll 
> see something related to your "seen" file.
>
> A corrupt seen file could be causing your problem.  Seen files are stored in 
> {$configdir}/user/prefix/username.seen.  My seen file is:
>
>  /var/spool/cyrus/config/user/m/morgan.seen
>
> If it is corrupt, you may be able to repair it.  Seen files have been stored 
> by default in Skiplist format for quite a while.  You can google get 
> "skiplist.py", a script to fix corrupted Skiplist files, from:
>
>  http://oss.netfarm.it/python-cyrus.php
>
> Hope this helps!

nothing useful shows up in the logs

Jan 10 13:19:12 asgard cyrus/imap[22884]: login: localhost [127.0.0.1] 
dl...@lang.hm plaintext Userlogged in
Jan 10 13:19:47 asgard master[1220]: process 22884 exited, signaled to death by 
7
Jan 10 13:19:47 asgard master[1220]: service imap pid 22884 in BUSY state: 
terminated abnormally


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

problem with one user after a crash

[David Lang]

I has my home mail server crash, and after the crash, one user (me) is unable 
to 
acess any folders.

When I manually telnet to the IMAP port, I can login, I can list and run other 
commands, but as soon as I do a select of any folder (mine or any other shared 
folder) I get disconnected.

Other users have no problems accessing the same folder.

This is with Cyrus 2.2 on Ubuntu (I need to upgrade, but have not had the time 
to do so yet)

Any suggestions on what may be wrong and how to diagnose this?

David Lang


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Cyrus imap server and filesystem type.

[David Lang]

On Mon, 3 Oct 2011, Bron Gondwana wrote:

> On Mon, Oct 03, 2011 at 01:13:44PM -0700, David Lang wrote:
>> reiserfs3 should work well as well (I don't use it as I dislike the
>> failure modeof it's filesystem check with filesystem images)
>
> If someone is emailing you reiserfs images which wind up aligning at
> block sizes after Cyrus adds its headers at the top, you have bigger
> problems!
>
> It's an annoying failure mode, but pretty understandable - just don't
> store reiserfs filesystems as files on it, and you're fine.

I don't have any filesystems that are _only_ used for Cyrus, as such there is a 
good chance of filesystem images existing somewhere on the filesystem.

I realize it's not a deal killer for everyone, but for me it's enough to keep 
me 
elsewhere (Hans Reisers benchmark cheating early on made me very skeptical of 
it 
and I moved to XFS instead)

David Lang

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/

Re: Cyrus imap server and filesystem type.

[David Lang]

On Mon, 3 Oct 2011, Josef Karliak wrote:


Hi there,
what filesystem type do you use for Cyrus imapd ? I use SLES11x64 (or 
opensuse 11.4).

I use Reiserfs3.6, so far so good. But couldn't be better ? :)
Thanks for share your expericiencies.
J.K.


ext2/3 were really bad in my experience. XFS works very well for me (except for 
slow expunge times without delayed expunge enabled) ext4 should also work well.


reiserfs3 should work well as well (I don't use it as I dislike the failure mode 
of it's filesystem check with filesystem images)


I don't think you'll get significantly better performance switching to a 
different filesystem, but it's worth checking.


David Lang

binCp0FA0bnpl.bin
Description: Veřejný PGPklíč


Cyrus Home Page: http://www.cyrusimap.org/

List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/

Re: DECISION: what reconstruct -f should do

[David Lang]

On Tue, 26 Apr 2011, Bron Gondwana wrote:

> On Tue, 26 Apr 2011 10:19 -0700, "David Lang"  
> wrote:
>> I can easily see someone wanting to have INBOX.sub containing 
>> INBOX.sub.folder1
>> and INBOX.sub.folder2 as an organizational mechanism
>>
>> if you were to create INBOX.sub and the user didn't want it, could they 
>> remove
>> it without affecting INBOX.sub.folder?
>
> No, not easily.  It would delete the whole tree by default.
>
>> if you don't create INBOX.sub and the user wants it, will they run into any
>> grief when they try to create INBOX.sub due to the fact that the directory
>> already exists?
>
> No, that's not a problem.  It will create just fine.

Ok, in this case your logic makes sense.

I would suggest that reconstruct report these cases, so that if the user did 
want INBOX.sub they have a reminder from the fact that it could have been 
created, but was not.

David Lang

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/

Re: DECISION: what reconstruct -f should do

[David Lang]

On Tue, 26 Apr 2011, Bron Gondwana wrote:

> On Sat, Apr 23, 2011 at 01:07:00AM +0200, Bron Gondwana wrote:
>> 3) add the mailbox if there's a directory, don't require
>>cyrus.header.
>>
>> 4) like (3) - but check that there's at least one cyrus.* file
>>OR at least one message file in the directory before
>>creating the mailbox.  (so an empty directory doesn't generate
>>a bogus mailbox, and neither does one containing nothing that
>>looks like it belongs in a mailbox)
>
> The clear winner appears to be (3) - the suggestion of adding
> yet-another-switch[tm] seems a bit silly to me, you're already
> specifying -f which pretty much means "I'm recovering from something
> that got the filesystem out of sync with mailboxes.db".
>
> But - there was one valid concern.  Assuming this structure:
>
> $conf/user/foo/
> $conf/user/foo/[contents]
> $conf/user/foo/sub/
> $conf/user/foo/sub/folder/
> $conf/user/foo/sub/folder/[contents]
>
> It's legal for the following to exist:
>
> INBOX
> INBOX.sub.folder
>
> Without INBOX.sub
>
> So I'm going to implement a modified (3) as follows:
>
> a) if cyrus.header, it's a folder
> b) if spool files, it's a folder
> c) if a directory with no subdirectories it's a folder
>
> Otherwise it's an intermediate folder, so we recurse without
> creating a mailboxes.db entry.  Basically, (4) but still
> creating leaf folders, just not intermediate ones that don't
> have any other content.

I can easily see someone wanting to have INBOX.sub containing INBOX.sub.folder1 
and INBOX.sub.folder2 as an organizational mechanism

if you were to create INBOX.sub and the user didn't want it, could they remove 
it without affecting INBOX.sub.folder?

if you don't create INBOX.sub and the user wants it, will they run into any 
grief when they try to create INBOX.sub due to the fact that the directory 
already exists?

Daivd Lang

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/

Re: POLL: what should reconstruct -f do?

[David Lang]

On Sat, 23 Apr 2011, Bron Gondwana wrote:

> The question came up from the following bug report:
>
> http://bugzilla.cyrusimap.org/bugzilla3/show_bug.cgi?id=3449
>
> Where there were spool files on disk, but no meta data left.
> Reconstruct gave no information about the files on disk at
> all.
>
> I see 4 options, can I'd like some opinions on what people
> think reconstruct should do.  Speak now(ish) or hold your
> peace!
>
> 1) what we do now - require a cyrus.header in the directory
>   or ignore it.
>
> 2) like (1) but warn about the directory with no cyrus.header
>
> 3) add the mailbox if there's a directory, don't require
>   cyrus.header.
>
> 4) like (3) - but check that there's at least one cyrus.* file
>   OR at least one message file in the directory before
>   creating the mailbox.  (so an empty directory doesn't generate
>   a bogus mailbox, and neither does one containing nothing that
>   looks like it belongs in a mailbox)

I think either 3 is the best answer with 4 being a reasonably close second.

I tend to be a person who would rather have extra stuff show up and deal with 
it 
rather than run the risk of not getting something that I need.

I don't think that there's a real problem with creating 'extra' mailboxes if 
there are extra directories, it's easy enough for the user to delete them. 
saying that there needs to be a message or a cyrus.* file is a huristic that 
sounds like it will work most of the time, but not always.

David Lang

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/

Re: Cyrus IMAP running in linux: recommended file system

[David Lang]

On Fri, 17 Dec 2010, Bron Gondwana wrote:

> On Thu, Dec 16, 2010 at 06:09:58PM -0800, David Lang wrote:
>> On Thu, 16 Dec 2010, Lucas Zinato Carraro wrote:
>>
>>> I know that this question is controversy and there is no exact answer.
>>>
>>> But  currently what is the best file system in  Linux to handle
>>> thousands of small files?
>>> I have mailboxes with 4Gb. And messages with ~4Kb.
>>> Ext4 ?
>>
>> depending on who you ask you will get
>>
>> ext4
>> btrfs
>> XFS
>>
>> my personal opinion is that ext4 is still new enought that I don't trust it
>> fully (they are still finding too many problems and having to scramble to fix
>> them)
>
> Disagree - it's been quite stable for about a year now, and usable for at
> least two.  The stuff they're finding is skanky little edge cases that
> we haven't run in to at all, so I'm happy.
>
> We're running about 50% on ext4.  About 1% on btrfs (a testing store, no real
> users)

that's very fair. I'm probably biased on the fact that I read the linux-kernel 
list and so see all the 'how in the world is it doing _that_' type of problems 
that come up :-)

>> this makes btrfs much too new.
>
> Agree.
>
>> so I opt for XFS
>
> Performance with zillions of small files isn't XFS's strong suit.

lots of files works fine on XFS, what it is slow with is metadata operations 
(creating or deleting lots of files), delayed expunge hides the latter problem.

David Lang

>> you want to avoid ext2/ext3 as they do very poorly with large numbers of 
>> files
>> in one directory.
>
> Definitely agree.  ext3 was unusable.
>
>> jfs and reiserfs (both 3 and 4) are options, but they are both used so seldom
>> that I would not be comfortable trusting them.
>
> I wouldn't trust reiser4, but reiserfs (version 3) is super stable - nobody
> is messing with it much.  The only interesting question is the last bit of
> BKL removal.  Otherwise it's rock solid.  We've been using it for the last 8
> years, and it's been solid for about the last 5.
>
>> some people consider ext4 well tested enough, and then the debate between it 
>> and
>> XFS gets much more interesting.
>
> yes, it does.
>
>> Ted Tso will tell you that ext4 is only extensivly tested in the simple cases
>> (single drive, home system type of thing), while XFS has had a lot of 
>> attention
>> and testing on very large and sophisticated drive systems, so the larger and
>> more powerful the drive subsystem you have the more likely you are to run 
>> into
>> some corner case that the ext4 developers just haven't run into yet, but that
>> the XFS developers have handled.
>
> Our large sophisticated drive systems emulate a single drive well enough that
> it works just fine.  We're not running layers of LVM magic though.  All the
> funky RAID stuff is handled by hardware.
>
> Bron.
>

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/

Re: Cyrus IMAP running in linux: recommended file system

[David Lang]

On Thu, 16 Dec 2010, Lucas Zinato Carraro wrote:

> I know that this question is controversy and there is no exact answer.
>
> But  currently what is the best file system in  Linux to handle
> thousands of small files?
> I have mailboxes with 4Gb. And messages with ~4Kb.
> Ext4 ?

depending on who you ask you will get

ext4
btrfs
XFS

my personal opinion is that ext4 is still new enought that I don't trust it 
fully (they are still finding too many problems and having to scramble to fix 
them)

this makes btrfs much too new.

so I opt for XFS

you want to avoid ext2/ext3 as they do very poorly with large numbers of files 
in one directory.

jfs and reiserfs (both 3 and 4) are options, but they are both used so seldom 
that I would not be comfortable trusting them.


some people consider ext4 well tested enough, and then the debate between it 
and 
XFS gets much more interesting.

Ted Tso will tell you that ext4 is only extensivly tested in the simple cases 
(single drive, home system type of thing), while XFS has had a lot of attention 
and testing on very large and sophisticated drive systems, so the larger and 
more powerful the drive subsystem you have the more likely you are to run into 
some corner case that the ext4 developers just haven't run into yet, but that 
the XFS developers have handled.

David Lang

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/

RE: disable IMAP IDLE

[David Lang]

On Wed, 24 Nov 2010, Robert Mueller wrote:

>> I was asked by IT to not permit IDLE since the current server went down
>> after 4-500 blackberries ate up all the (limited) capabilities of that
>> machine.
>
> I'd really be surprised if that was a problem these days. We have
> machines that have 1 connections quite fine. Yes they're fairly
> loaded servers with fast IO and lots of RAM, but even a pretty basic
> server should handle 500 connections no problem.
>
> Have you actually tested that long lived IDLE connections are a problem?

if they are idle they eat up very few resources, if they are checking things 
frequently, forcing them to reconnect each time will just eat up more resources.

David Lang

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/

Re: ext3 / XFS [Was: Re: Does anyone allow unlimited or extremely large quotas?]

[David Lang]

On Tue, 16 Nov 2010, Adam Tauno Williams wrote:

> On Tue, 2010-11-16 at 11:25 -0800, David Lang wrote:
>>> I don't actually know what sort of problems I'm referring to, hence the
>>> question.  The big problem I can imagine would be opendir() and
>>> readdir() with a huge number of files in a directory, but the cyrus code
>>> doesn't appear to do that in a lot of places that would matter to a user
>>> (deleting an entire folder, delete sieve scripts, etc) in the course of
>>> normal operations.
>> This is depends on what filesystem you are useing, I have mailboxes with 
>> hundreds
>> of thousands of messages in them on XFS and have no problems, but on ext3 I
>> start seeing slowdowns with a bit over ten thousand messages.
>
> Was dir_index enabled on that ext3 filesystem?  Prior to dir-index ext3
> was very slow for large folders. dir_index is not enabled by default in
> ext3.

yes, even with dir-index I see slowdown on large, busy folders. not as bad as 
without them, but still there.

without dir-index a folder that at one time had 10K files in it becomes 
unusuably slow forever, with dir-index the slowdown isn't as bad, but it's 
still 
there.

remember that dir-index only helps for the case where you are looking for a 
single file, if you are walking the entire directory it has little, if any 
effect.

XFS is slow deleting large numbers of files as noted by others, but delayed 
expunge sidesteps that issue.

David Lang

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/

Re: Does anyone allow unlimited or extremely large quotas?

[David Lang]

On Tue, 16 Nov 2010, Ciprian wrote:

> Adam Tauno Williams wrote:
>> I think the issue you will encounter first is clients will start to fall
>> down when folders exceed a 'reasonable' number of messages.  Common IMAP
>> clients I've seen start to exhibit severe performance issues beyond a
>> few hundred thousand messages.
>>
> Older versions of Outlook (e.g. Office 2003) will choke well before that
> (I think the limit was around 35.000 on XP SP2 ) also couple that with
> the local Outlook file store for that IMAP account which used to be
> limited to 2G. We generally advice our users to avoid going past 20.000
> messages in one folder.

the nice thing is that this is per-folder, and generally shows up as a slowdown 
as you get large, so the user can just create a subfolder and move messages in 
to it to work around the client issues.

This should mostly be self-regulating as a result.

David Lang

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/

Re: Fwd: Re: Does anyone allow unlimited or extremely large quotas?

[David Lang]

On Tue, 16 Nov 2010, Dave McMurtrie wrote:

> I didn't realize that I only responded to Rob here.  Perhaps my
> additional information will shed some light on the kind of information
> I'm looking for.
>
>  Original Message 
> Subject: Re: Does anyone allow unlimited or extremely large quotas?
> Date: Tue, 16 Nov 2010 07:06:53 -0500
> From: Dave McMurtrie 
> To: Rob Mueller 
>
> On 11/16/2010 06:45 AM, Rob Mueller wrote:
>>
>>> This may be slightly off-topic, so apologies in advance. Is there
>>> anyone out there who allows unlimited quota for their users or provides
>>> extremely large quotas when asked for?
>>
>> What do you consider extremely large? And what sort of problems are you
>> referring to?
>
> I don't actually know what sort of problems I'm referring to, hence the
> question.  The big problem I can imagine would be opendir() and
> readdir() with a huge number of files in a directory, but the cyrus code
> doesn't appear to do that in a lot of places that would matter to a user
> (deleting an entire folder, delete sieve scripts, etc) in the course of
> normal operations.

this depends on what filesystem you are useing, I have mailboxes with hundreds 
of thousands of messages in them on XFS and have no problems, but on ext3 I 
start seeing slowdowns with a bit over ten thousand messages.

>> The usual issue is just the huge number of emails and thus files that
>> accumulate. Creating a fresh replica, body searching, reconstructing,
>> etc all take quite a bit of time because of the large amount of random
>> IOs. Apart from that, everything does actually work ok...
>
> The only issue we ever had was with a bboard that our network group
> sends automated system messages to.  Something in their environment went
> haywire and we ended up with ~1.5 million messages in that bboard.  They
> were unable to find a client that was willing to deal with the folder to
> be able to clean it up.  I was able to connect using imtest and SELECT
> and FETCH messages without any problems, though.  I also recall that
> replication was broken by this folder, but I don't remember exactly why.

alpine and mulberry have no problem with huge numbers of messages.

David Lang

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/

Re: too many files in a directory

[David Lang]

On Fri, 5 Nov 2010, Ross Boylan wrote:

> Thanks for your reply.
> On 11/5/2010 6:18 PM, David Lang wrote:
>>> I have a narrow question and a broader one.  Narrowly, if I create some
>>> other folders and move some of the messages into them, will it help?  My
>>> understanding is that cyrus tries to avoid copying or moving message
>>> files around on disk, and so I suspect the files will continue to take
>>> up space in the original directory even if I move them.
>> 
>> if you copy them to the new folder, 
> are you talking about a filesystem copy, or a copy via imap?

copy via imap. you really don't want to go mucking around on the filesystem 
under cyrus. Reserv that for emergancies ;-)

>> delete them from the old folder, and expunge them, then the directory slots 
>> will be freed up. I don't think that ext3 will actually shrink the 
>> directory, but it will re-use these available slots for new files. 
>
> [deleted stuff on XFS]
> Yes, I noticed XFS seems to have quite a few people with good experiences. 
> That's another one I might consider.  I was a little reluctant because I've 
> also seen (not in this list) people saying it ate their data.  Of course, a 
> lot of people said that about reiserfs too, and probably every other file 
> system...

the other thing to look at is _when_it ate their data. like linux, most 
filesystems are under continuing development. If XFS ate their data 5+ years 
ago, so many improvements have gone in thatI wouldn't care. If they say it ate 
their data last year, I would be worried.

David Lang

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/

Re: too many files in a directory

[David Lang]

On Fri, 5 Nov 2010, Ross Boylan wrote:

> My Cyrus 2.2.13 server (Debian stable) is running on an ext3
> filesystem.  One folder contains so many files that it keeps exceeding
> the limit ext3 can cope with (or at least, more than the directory index
> can handle).  I've been able to recover with e2fsck (I don't know if it
> rebalances the tree or makes more room), but clearly need to do
> something else.
>
> I have a narrow question and a broader one.  Narrowly, if I create some
> other folders and move some of the messages into them, will it help?  My
> understanding is that cyrus tries to avoid copying or moving message
> files around on disk, and so I suspect the files will continue to take
> up space in the original directory even if I move them.

if you copy them to the new folder, delete them from the old folder, and 
expunge 
them, then the directory slots will be freed up. I don't think that ext3 will 
actually shrink the directory, but it will re-use these available slots for new 
files.

> More broadly, any other suggestions for dealing with this situation?
> I'm also having very slow backup times, I think the result of the long
> time it takes to traverse the directory.  I might go to reiser3, which I
> used successfully on a different system.  I thought the reiser file
> system seemed like a bad long-term bet after the architect was jailed
> for murder.  But I see from the archives that people are still using it
> and having good experiences.

this is why I use XFS instead of ext3 :-)

ext4 is also better, but it's new enough that I don't use it for anything 
critical yet.

> Thanks for any advice.
> Ross
>
> P.S. For the record, the failures do not lose any messages; they result
> in messages going to my main inbox when there's an error on attempted
> delivery to the subfolder.

probably what's happening is that something is taking long enough that the 
delivery to the subfolder 'fails' and it falls back to delivering to the main 
inbox instead.

how many files do you have in the problem folders?

David Lang

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/

Re: Cyrus IMAPd 2.4.0 Released

[David Lang]

I'm happy to see this.

Is there anyone packaging this up for the common linux distros?

David Lang

On Mon, 11 Oct 2010, Ken Murchison wrote:

> Date: Mon, 11 Oct 2010 17:48:54 -0400
> From: Ken Murchison 
> To: Cyrus Mailing List ,
> cyrus-annou...@lists.andrew.cmu.edu, cyrus-de...@lists.andrew.cmu.edu,
> cyrus-proj...@lists.andrew.cmu.edu
> Subject: Cyrus IMAPd 2.4.0 Released
> 
> I am pleased to announce the release of Cyrus IMAPd 2.4.0.  This
> release should be considered production quality but hasn't yet been
> widely tested.  Major changes in the release are the following:
>
> - All databases are now skiplist by default
> - Charset subsystem has been rewritten - Unicode 5.2 and UTF-8 in Sieve
> - Core mailbox handling code has largely been rewritten
> - Replication code largely rewritten
> - Several LEMONADE extensions have been implemented
> - Added support for marking QoS on traffic
>
> For full details, please see:
> http://www.cyrusimap.org/docs/cyrus-imapd/2.4.0/changes.php
> http://www.cyrusimap.org/docs/cyrus-imapd/2.4.0/install-upgrade.php
>
>
> URL for this release:
> ftp://ftp.cyrusimap.org/cyrus-imapd/cyrus-imapd-2.4.0.tar.gz
>
>
> Special thanks go to Bron Gondwana whose stellar work on redesigning and
> refactoring the code has been instrumental in making this release possible.
>
>
> Questions and comments can be directed to
> info-cyrus@lists.andrew.cmu.edu (public list), or cyrus-b...@andrew.cmu.edu.
>
>
>

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/

Re: competition

[David Lang]

On Mon, 20 Sep 2010, Andrew Morgan wrote:

> On Mon, 20 Sep 2010, Vincent Fox wrote:
>
>> Umm, what?  We run Cyrus IMAP server with no Murder
>> for 20K+ people.  Murder may be a feature but it's not a
>> deployment requirement.
>>
>> We used Perdition, originally just thrown up to provide a
>> transparent bridge as we migrated from Uwash to Cyrus.
>> But as things moved along, we ended up with Perdition
>> and multiple Cyrus backends and no good reason to switch
>> over to a Murder.
>
> I end up granting myself rights to various users' mailboxes to investigate
> when we see one of our users sending out spam.  It usually turns out that
> they have been phished recently.  Once I grant myself rights to their
> mailbox, I see the mailbox in my regular email client (Alpine) in the
> "Other Users" hierarchy.
>
> Does this work without Murder?

it definantly works with a single back-end. In the case of a multi-server 
setup, 
it depends on if your murder substatute will show you the folders across 
back-ends or not in a listing.

you can always connect directly to the back-end (bypassing murder or it's 
replacement) to see all "other users" for that back-end.

David Lang

>  How do you investigate users' mailboxes in
> cases like this?
>
>   Andy
> 
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
>

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/

Re: [PATCH] Disable reverse DNS lookups

[David Lang]

On Thu, 9 Sep 2010, Guilherme Manika wrote:


 Oh, sure.

 Cyrus does a reverse DNS lookup to log the user's domain name to syslog. This 
adds an overhead to each incoming connection that increases the number of 
simultaneous connections to the system and may become devastating if there is a 
problem in external DNS connectivity. We added this when the DNS servers for a 
large part of our customer base experienced such problems, as that essentially 
destroyed our POP3 servers.

 Needless to say, this is only a problem for people operating large 
installations.


or very small ones that do not have reverse DNS properly setup (not an uncommon 
situation)


David Lang


Guilherme


Em 09/09/2010, às 08:38, Jeroen van Meeuwen (Kolab Systems) escreveu:


Guilherme Manika wrote:

 This patch adds a "disablereverselookups" option to imapd.conf that
disables reverse DNS lookups in imapd and pop3d.

 It doesn't affect other services (lmtp, mupdate, etc.) because they are
not Internet-facing services and so do not rely on external DNS to work.
That's probably acceptable.



Mind sharing with us what the purpose of this patch is?

Kind regards,

--
Jeroen van Meeuwen
Senior Engineer, Kolab Systems AG

e: vanmeeu...@kolabsys.com
t: +316 42 801 403
w: http://www.kolabsys.com

pgp: 9342 BF08

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/



Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/

Re: visibility of Mailbox-folders

[David Lang]

On Tue, 19 Jan 2010, Dr. Harry Knitter wrote:

>>
>> That's interesting. Since cyradm is an imap client, why does it see the
>> mailboxes and your other clients don't? Can it be that your clients are
>> configured to use another folder separator or something? I really don't
>> understand this.
>>
>> Simon
>>
>
> I don' t understand it either. We have tried it with different Client programs
> (Thunderbird, Outlook, KMail unfortunately has problems with large amounts of
> imap folders). The result was allways  the same.
> Other folders which show the same results in cyradm show their subfolders and
> files without any problems.
>
> So what can I do to solve this problem?


Silly question (since I haven't been following this thread), have you checked 
the subscribed status of the folders?

David Lang

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Future Ideas wiki page

[David Lang]

On Wed, 13 Jan 2010, Bron Gondwana wrote:

> On Wed, 13 Jan 2010 17:40 -0800, "David Lang"  
> wrote:
>>> Absolutely - two issues.
>>>
>>> 1: how to you give folders UIDs?
>>
>> I thought that there was mention in your list of addressing folders by
>> UID for
>> replication purposes.
>
> UniqueID - it's an internal 16 hex digit field.  Tacking one of those on every
> result item would be expensive for bandwidth.


If you have a lot of of messages being identified it will be noticable, but 
I'll 
point out that the user dowloading one word doc attachement is several thousand 
messages worth of IDs. Another option would be to use the folder name (but you 
would have to worry about escaping some characters, etc)

>>> The first one is the killer, because you'd have to be able to map back to 
>>> select
>>> by UID as well.  SELECT 195455, or something.
>>>
>>> Actually, the whole "SELECTED" idea would probably be discarded, instead 
>>> just
>>> addressing everything by UID and never actually selecting folders.
>>
>> yes, for some things this is extremely useful for others it's just added
>> complication.
>
> There's not much that it's useful for with cross folder work!
>
> Internally our web interface actually already does exactly what you describe 
> so it
> can support cross folder searching.  Every UnqMsg is :, where
> FolderId is a reference to a database field which contains a mapping to the
> underlying folder.  We also store a bunch of metadata about how to present 
> that
> particular folder to the user (number of messages to display, type of preview,
> default sort, etc)

if the only thing that can really use this is cross folder searching, then only 
implement this as output from the server in search results (and possibly 
sorting 
commands), allow it for input (message specification) for other commands. Worst 
case you could require the client to select the folder.

One other thing that something like this would be useful for is for things that 
want to watch for messages from any of several folders. Today they need to 
either open a seperate connection for each folder they want to watch, or cycle 
through the folders polling them.

I suspect that if this addressing mechanism was implemented, it would be about 
as easy to add it everywhere that UID is an option as it would be to only add 
it 
to some commands.

>>> By which time, why not just define a brand new protocol not called IMAP 
>>> which
>>> includes the good bits of what IMAP currently does, and discards anything 
>>> that
>>> doesn't fit the multi-folder worldview.  So long as you made the storage and
>>> meta-data requirements compatible with already existing Cyrus and other IMAP
>>> servers, you could just write a whole new daemon that talked your new 
>>> protocol
>>> and be happy with that.
>>
>> you could, but just like not every client uses UIDs and still use message
>> numbers, this is being done in a backwards compatible manner so that the
>> client
>> can use either one, and a server can support both.
>
> The problem is you keep paying a higher and higher complexity cost at the 
> server end.
> Once you start talking not having a folder selected, that cost (and associated
> locking issues) is going to blow your implementation complexity way up.  
> Cross folder is
> far enough away from the initial design of IMAP that the impedance mismatch 
> is grating
> pretty badly!

I'm missing the locking issues here. even with single folder searching you can 
have a message disappear immediatly after the search results have been 
provided. 
Folders don't get locked when they are selected. What is it that needs to be 
locked for this sort of addressing?

I agree with the climbing complexity, but is there really so much stuff that 
_everyone_ agrees should be dumped with IMAPv4 that it's worth the break? If so 
it may be time for IMAPv5, but have we really moved that far?

Most of the time you really do want to act on one folder at a time, and as such 
selecting that folder makes sense.

David Lang

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Future Ideas wiki page

[David Lang]

On Wed, 13 Jan 2010, Bron Gondwana wrote:

> On Wed, 13 Jan 2010 16:26 -0800, "David Lang"  
> wrote:
>> On Sat, 9 Jan 2010, Bron Gondwana wrote:
>>
>>> On Sat, 09 Jan 2010 15:10 +0100, "Alexey Melnikov" 
>>>  wrote:
>>>> Bron Gondwana wrote:
>>>>> While we're at it, I'm much more interested in cross-folder searching 
>>>>> with sort
>>>>> order that doesn't require folder as the first item, but that's 
>>>>> significantly more
>>>>> complex!
>>>>>
>>>>>
>>>> <http://tools.ietf.org/html/draft-ietf-morg-multimailbox-search-01>?
>>>> If you have some additional use cases in mind, please let me know.
>>>
>>> "with sort order that doesn't require folder as the first item".  I guess 
>>> you could return
>>> multiple ESEARCH responses with the same folder mentioned to get the 
>>> ordering...
>>>
>>> It's still more folder-centric than otherwise, and it's going to make 
>>> following threads
>>> across folders (say INBOX and a couple of time based archive folders) 
>>> tricky!
>>
>> thinking about this a bit more, it sounds almost like what you are
>> wanting is a
>> third mode of addressing messages.
>>
>> currently we can do
>>
>> message # in this folder
>>
>> messageUID in this folder
>>
>> and something like folderUID:messageUID would open up what you are
>> looking for
>> (probably plus more)
>>
>> Would it be possible to take a character that can't appear in a message#
>> or UID
>> position in the existing protocol and define it as a delimiter for this?
>> (I used
>> ':' in my example above as I believe that '-' is used to indicate a range
>> of
>> messages)
>>
>> If it is, then this would 'just' be a new addressing option like UID
>> currently
>> is, and like UID, clients would opt-in to this new mode. (it would still
>> need a
>> RFC for the new mode, but does this sound like a solution to what you are
>> looking for?
>
> Absolutely - two issues.
>
> 1: how to you give folders UIDs?

I thought that there was mention in your list of addressing folders by UID for 
replication purposes.

> 2: how do "ranges" work?

a range within a folder works as-is, a range with different folders is an error

> The first one is the killer, because you'd have to be able to map back to 
> select
> by UID as well.  SELECT 195455, or something.
>
> Actually, the whole "SELECTED" idea would probably be discarded, instead just
> addressing everything by UID and never actually selecting folders.

yes, for some things this is extremely useful for others it's just added 
complication.

> By which time, why not just define a brand new protocol not called IMAP which
> includes the good bits of what IMAP currently does, and discards anything that
> doesn't fit the multi-folder worldview.  So long as you made the storage and
> meta-data requirements compatible with already existing Cyrus and other IMAP
> servers, you could just write a whole new daemon that talked your new protocol
> and be happy with that.

you could, but just like not every client uses UIDs and still use message 
numbers, this is being done in a backwards compatible manner so that the client 
can use either one, and a server can support both.

David Lang

> Bron ( yes, I have been tempted to write something that talks sync_client 
> protocol,
>why do you ask ;)
>

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Future Ideas wiki page

[David Lang]

On Wed, 13 Jan 2010, Wil Cooley wrote:


And anyway, would it
be faster to open and list 1,000 files in 23 directories than to open one
directory and list 23,000 files? Would that be overshadowed by the cost of
opening all 23,000 files (which I presume it would need to if it were resorting
to listing them).


there are two scenerios that are important here

1. open a directory and list 23,000 files

for this case, just about all filesystems will be faster with one directory as 
they can just do a sequential walk through the directory.


2. open file 22,987

for this case you will see a drastic difference between different filesystems 
(and potentially even the same filesystem with different options). For example, 
this will be very slow on Ext2, or on Ext3 without hashdir enabled. It's 
supposed to be much faster on Ext3 with hashdir, but I've had mixed results when 
I've tried it. On the other hand this is a very fast operation on XFS.


David Lang


signature.asc
Description: OpenPGP digital signature

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Future Ideas wiki page

[David Lang]

On Sat, 9 Jan 2010, Bron Gondwana wrote:

> On Sat, 09 Jan 2010 15:10 +0100, "Alexey Melnikov" 
>  wrote:
>> Bron Gondwana wrote:
>>> While we're at it, I'm much more interested in cross-folder searching with 
>>> sort
>>> order that doesn't require folder as the first item, but that's 
>>> significantly more
>>> complex!
>>>
>>>
>> <http://tools.ietf.org/html/draft-ietf-morg-multimailbox-search-01>?
>> If you have some additional use cases in mind, please let me know.
>
> "with sort order that doesn't require folder as the first item".  I guess you 
> could return
> multiple ESEARCH responses with the same folder mentioned to get the 
> ordering...
>
> It's still more folder-centric than otherwise, and it's going to make 
> following threads
> across folders (say INBOX and a couple of time based archive folders) tricky!

thinking about this a bit more, it sounds almost like what you are wanting is a 
third mode of addressing messages.

currently we can do

message # in this folder

messageUID in this folder

and something like folderUID:messageUID would open up what you are looking for 
(probably plus more)

Would it be possible to take a character that can't appear in a message# or UID 
position in the existing protocol and define it as a delimiter for this? (I 
used 
':' in my example above as I believe that '-' is used to indicate a range of 
messages)

If it is, then this would 'just' be a new addressing option like UID currently 
is, and like UID, clients would opt-in to this new mode. (it would still need a 
RFC for the new mode, but does this sound like a solution to what you are 
looking for?

David Lang

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Future Ideas wiki page

[David Lang]

On Fri, 8 Jan 2010, Bron Gondwana wrote:

> On Fri, 08 Jan 2010 09:56 -0800, "David Lang"  
> wrote:
>> one thing that I saw mentioned elsewhere as a limitation of IMAP (and
>> therefor I
>> don't know if there is a way to address it reasonably) is the lack of a
>> fuzzy
>> search capability.
>
> Without a specification document, it's hard to add anything that you expect
> clients to actually use.

true, but without a sample implementation it's unlikely to become a standard, 
so 
the discussion needs to stop somewhere.

>> the IMAP search is a exact match search, it would be useful to have the
>> hooks to
>> be able to use a search-engine like search capibility as well (not just
>> exact
>> matches, but matches with only some of the search terms, matches with
>> plural
>> versions of the search terms, etc)
>
> Yes, that would be lovely to have.  You'd probably run a separate 
> search-engine
> process and have the IMAP server just send out a request and map the document
> IDs back to folder/uid on response.

sounds right.

>> As I understand it this would require a slight variation of the search
>> request
>> to indicate that you want the fuzzy match, and a variation of the search
>> response to be able to indicate the quality of each match returned.
>
> It would require a brand new spec for the search result - an ordered list of
> UIDs wouldn't cut it any more!
>
> While we're at it, I'm much more interested in cross-folder searching with 
> sort
> order that doesn't require folder as the first item, but that's significantly 
> more
> complex!

If we have to define a new search response, it may be reasonable to handle 
both. 
Does anyone who has been part of the IMAP standards work want to think/talk 
about what would work well and fit the flavor of IMAP?

On the same subject, what other wishes do people have related to search (if 
there are enough things people are interested in, possibly someone will get 
motivated enough to write the code to make it work)

> Thankfully, this is all pretty orthagonal to everything that I'm doing, so 
> it's not
> a consideration I need to give much thought to at the moment.  Someone else
> who considers it worth putting effort in to could do it pretty independently.

agreed.

> The charset changes would allow an initial pre-processing pass to spit out the
> "document" as UTF-8 rather than its original MIME encoding for processing by
> the search engine, but that's the only interaction it would have.  If the 
> search
> engine supports a chunked input, it would probably be worth embedding that
> target into the lib/charset.c as a character filter sink, and chaining the 
> documents
> into it rather than building an entire buffer at once.  There's already code 
> that
> does that just using a standard buffer and sending it to the squatter callback
> whenever it reaches a fixed size, then resetting it.  Easy enough to do.

If that interface is enough more efficiant, it may be worth making that a 
requirement and let the external tool deal with splitting it back up if needed.

So, if I am understanding you correctly, the hooks into Cyrus to support 
something like this are fairly easy to do, the hard thing would be the IMAP 
command and response.

David Lang

> Regards,
>
> Bron.
>

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Future Ideas wiki page

[David Lang]

one thing that I saw mentioned elsewhere as a limitation of IMAP (and therefor 
I 
don't know if there is a way to address it reasonably) is the lack of a fuzzy 
search capability.

the IMAP search is a exact match search, it would be useful to have the hooks 
to 
be able to use a search-engine like search capibility as well (not just exact 
matches, but matches with only some of the search terms, matches with plural 
versions of the search terms, etc)

As I understand it this would require a slight variation of the search request 
to indicate that you want the fuzzy match, and a variation of the search 
response to be able to indicate the quality of each match returned.

David Lang



  On Thu, 7 Jan 2010, Bron Gondwana wrote:

> Date: Thu, 7 Jan 2010 20:01:52 -0800
> From: Bron Gondwana 
> To: cyrus-de...@lists.andrew.cmu.edu, cyrus-proj...@lists.andrew.cmu.edu
> Cc: info-cyrus@lists.andrew.cmu.edu
> Subject: Future Ideas wiki page
> 
> Hi All,
>
> I've set up a new wiki page here:
>
> http://cyrusimap.web.cmu.edu/twiki/bin/view/Cyrus/FutureIdeas
>
> Linked from the roadmap:
>
> http://cyrusimap.web.cmu.edu/twiki/bin/view/Cyrus/RoadMap
>
> I've updated the Roadmap with the items I have ready right
> now for 2.4, and put everything else into the bright future
> under 2.5 for now, subject to "actually gets finished and
> stable", obviously.
>
> Most of these ideas have been fleshed out in some detail
> already in my postings to the devel mailing list.  Some are
> still a little bit raw (like having the mailbox path on disk
> depend on the uniqueid rather than the actual mailbox name.
> I'll expand on this another time... and it's not yet mentioned
> on the wiki, but it makes a lot of things nicer!)
>
> Anyway, I'd love feedback on any or all of it, and if there
> are other things that you feel are really important for the
> future viability of Cyrus I'd love to hear about them as well.
> I haven't yet had a chance to look at the QRESYNC stuff that
> Ken's already done for 2.4, and we might wind up releasing
> a 2.4 without a lot of these changes just because there's a
> lot of work in there!
>
> That said, I'm pushing myself pretty aggressively to have that
> list finished by April of THIS YEAR.  Particularly the low
> bandwidth replication, which depends on all pretty much all
> of the others!
>
> Regards,
>
> Bron.
> 
> Cyrus Home Page: http://cyrusimap.web.cmu.edu/
> Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
> List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
>

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Exec'ing a script from Cyrus when imapd has a client

[David Lang]

On Thu, 12 Nov 2009, Greg A. Woods wrote:

> At Thu, 12 Nov 2009 11:55:25 -0800 (PST), David Lang 
>  wrote:
> Subject: Re: Exec'ing a script from Cyrus when imapd has a client
>>
>> no, SMTP only works if you have network connectivity that is up most of the
>> time. it will handle short outages, but it will not handle the case where 
>> your
>> network connectivity is off most of the time
>
> If your link is off most of the time then use UUCP -- that's what it was
> designed for.  Tunnel it through SSL if you're using public IP
> addressing and routing for the same link.

you can only do this if the servers you are connecting to support UUCP, most do 
not.

but even if you do this, it's valid to want UUCP to check frequently when imapd 
has a client and infrequently (if at all) when it doesn't.

> Or just use IMAP from a client MUA -- your link will be up when you want
> to read mail and all your mail will be available at the IMAP server at
> that time.

over a slow, high latency, error-prone link IMAP is very painful to use.

> All these excuses for doing strange things with things like fetchmail
> are really really lame and stretching the imagination beyond belief!



>>> Use proper client/server protocols for dynamic IP clients!
>>
>> SMTP is not a proper protocol for a dynamic IP environement.
>
> Indeed, SMTP is not a client/server style protocol in the sense I meant.
>
> IMAP would be a proper client/server style protocol in the sense I meant.

so you are claiming that his business need is invalid. you are not in a 
position 
to declare this.

>> Thunderbird? my understanding (from watching people use it) is that it wants 
>> to
>> pull a copy of all your mail to the local box before processing it. how is 
>> this
>> a proper IMAP client?
>
> How is it not a proper IMAP client?  Like I said, pick your poison.
> Some MUAs will want to copy everything over at once for one kind of
> performance profile, some will request only headers (enough to form the
> summary index) for another kind of performance profile.  Thunderbird
> fully understands the concept of multiple folders on arbitrary servers,
> and it more or less speaks true IMAPv4, giving the user an interface to
> do most everything that IMAPv4 will easily allow.  It has additional
> features that make it possible to work offline.  That sounds like a
> proper IMAP client to me.

that 'more or less speaks true IMAPv4' makes me say it's not a proper IMAP 
client.

I also consider any client that uses IMAP to pull the data to the local system 
and does everything else there to be a POP client that happens to use IMAP 
to fetch it's messages.

A proper IMAP client would fetch only the portions of a message that the user 
needs, and would use the capabilities of the server (or at least the basic 
IMAPv4 capabilities, they may not use all of the enhancements) rather than 
duplicating the functionality on local copies.

this doesn't prevent the client from offering a disconnected mode of operation, 
but if disconnected mode is not in use, the server should be used.

David Lang


Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Exec'ing a script from Cyrus when imapd has a client

[David Lang]

On Thu, 12 Nov 2009, Greg A. Woods wrote:

> At Thu, 12 Nov 2009 21:21:24 +0100, Xavier Bestel  
> wrote:
> Subject: Re: Exec'ing a script from Cyrus when imapd has a client
>>
>> Do you gain anything if Cyrus doesn't
>> fulfill the needs of some users ?
>
> Did I ever say Cyrus should or should not meet the needs of some users?
>
> I think you've got something backwards here.
>
> Why should any one "product" meet the needs of all users?  That's the
> "if all you've got is a hammer..." analogy.  Cyrus IMAP will _NEVER_
> meet the needs of all users, and that's fundamentally what I've been
> trying to say from the start.  If you don't need it then don't try to
> wedge it into your implementation!
>
> We all gain if we can avoid the "all you've got is a hammer" trap, and
> indeed we all gain if we can help each other avoid that trap too.
>
> If I'm not too mistaken it seems most everyone who wanted to use an IMAP
> server as a client-level tool (by employing the likes of fetchmail) were
> clearly falling for the "I have this Cyrus IMAP Hammer and I want to use
> it to manage all of my e-mail even though I'm not running a server" trap.
> Certainly that seemed to me to be the OP's situation.
>
> The real fear I have is that as a result others will believe that this
> kind of use is condoned and approved, or worse that these folks were
> actually setting up such things for other groups of users.  In both
> cases we end up with naive users who will not understand the issues
> involved.   Issues such as mangled headers, unreliable delivery, loss of
> e-mail, and so on.

who are you to officially condone or approve any particular use?

who on this list (or any list) has the authority to do so?

it's fine to suggest other solutions, but if people then say that those 
solutions cannot be used in these cases (and especially when they give the 
reasons) continuing to argue that it's wrong and evil to use the tool that way 
is not productive. you may not choose that solution, but that may be because 
you 
have resources available to you that make a different solution better.

> Sure, it's all fine and dandy for someone to want to learn about a
> product like Cyrus IMAP (or even fetchmail) by installing and using it
> in some form where they can personally make use of it.
>
> However if the goal is just to make something work in the real world for
> end users then we really need to go back to the fundamental end-user
> requirements and figure out how best to meet them without creating
> hidden problems along the way simply because we've got this hammer in
> our hands and we're dying to bang away on something.  If the user wants
> some screws installed then we'd be doing them a huge favour if we would
> go and find the proper screwdriver to do the job for them!

even if the OP was using UUCP for the mail transport, the question he asked 
(how 
to have a job run frequently when a user is logged in, and not run if there are 
no users logged in) is still a very useful question to get an answer to.

you have focused on the fact that he wants to use fetchmail as the transport 
between the full-time internet and his intermittently connected network and are 
telling everyone that he absolutly, under no conditions should try to do what 
he's attempting.

that's where we have severe disagreements.

most of the rest of us see situations where the basic approach he's doing could 
be the best possible approach. we may quibble over exact details of some of the 
things (use fetchmail vs UUCP vs other), but that doesn't make the basic 
approach invalid.

David Lang

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Exec'ing a script from Cyrus when imapd has a client

[David Lang]

On Thu, 12 Nov 2009, Gabor Gombas wrote:

> On Thu, Nov 12, 2009 at 11:55:25AM -0800, David Lang wrote:
>
>>> Use SMTP to breech the unreliable link!  It's safe, proven, and designed
>>> for that very task!
>>
>> no, SMTP only works if you have network connectivity that is up most of the
>> time. it will handle short outages, but it will not handle the case where 
>> your
>> network connectivity is off most of the time
>
> ETRN can solve this, you just need a relay that understands ETRN and is
> willing to hold your e-mails while you're off-line. ODMR is better for
> dynamic addresses (you don't need dynamic DNS) but it seems to be less
> supported. I have not used either yet, so YMMV.

the big advantage of the fetchmail approach that the initial poster was asking 
about is that it does not require special configuration of the upstream mail 
server.

I definantly agree that it is not the best possible approach. but as far as I 
can tell, every approach that's better requires cooperation from a mail server 
that does have full-time Internet connectivity. unfortunantly that's not always 
available (at least, not available at an affordable price in time and money)

David Lang

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Exec'ing a script from Cyrus when imapd has a client

[David Lang]

On Thu, 12 Nov 2009, Greg A. Woods wrote:

> At Sun, 8 Nov 2009 06:54:30 +1100, Bron Gondwana  wrote:
> Subject: Re: Exec'ing a script from Cyrus when imapd has a client
>>
>> On Sat, Nov 07, 2009 at 11:08:31AM -0500, Greg A. Woods wrote:
>>>
>>> Just get a forwarding alias installed on the remote mail server and then
>>> you'll be using the MTA to move your mail in a robust, secure, and
>>> fail-safe manner to the IMAP server where you desire it to be finally
>>> delivered.
>>
>> Maybe unreliable network connectivity?
>
> That's _exactly_ where you want to use SMTP or some other store and
> forward mechanism to create a robust and reliable mail transport link!
>
> Use SMTP to breech the unreliable link!  It's safe, proven, and designed
> for that very task!

no, SMTP only works if you have network connectivity that is up most of the 
time. it will handle short outages, but it will not handle the case where your 
network connectivity is off most of the time

>>  A dynamic IP where they don't
>> want a stale DynDNS pointer to cause someone else to get the mail?
>
> Well, amateurs can and will run whatever hacks they want, and they're
> not usually interested in doing the kinds of things necessary for
> production systems in the first place either.
>
> Further, if anyone is stupid enough to try to use dynamic IP addresses
> where static IP addresses are REQUIRED for proper functionality and
> robust operations then they get every problem they deserve and I have no
> interest whatsoever in catering to any of their hacks and abominations.
>
> Use proper client/server protocols for dynamic IP clients!

SMTP is not a proper protocol for a dynamic IP environement.

>> Pull vs Push in the abstract is an age old question that never has only
>> one answer, much as you are trying to paint it that way.
>
> Well, in Internet e-mail delivery there has always been one and only one
> answer to the push vs. pull philosophy.  I'm only talking about e-mail
> here.

not true, in the beginning UUCP was the primary mechanism for transporting 
e-mail. it was designed for the (then current) environement where connectivity 
was very intermittent and expensive to leave idle (which happens to match the 
use case here as well, but using UUCP takes far more cooperation on the part of 
the Internet server, thus the fetchmail approach)

> Fight the way e-mail has always worked and you have to fight the whole
> infrastructure and use fringe tools with known risks and problems.
>
> If you want your e-mail to work reliably then you have to work with the
> existing infrastructure and with the tried and tested tools that
> designed and implemented to work that way.
>
> Note how even in SMTP the proposed mechanisms for pull-like
> functionality have been lost, broken, and forgotten forever, and even
> there, like in UUCP, it's still fundamentally store-and-FORWARD even if
> the client makes the call.  Nobody has _ever_ made "pull" work for
> e-mail in any significant widely accepted and implemented way.

UUCP that's acttivated when the client connects and tickles the server to let 
it 
know that it's connected is effectivly a pull mechanism.

>>> The rest of this is kinda just BS about how to use a proper IMAP client.
>>
>> Er, you know a perfect IMAP client?  I've never been able to find a good
>> one, which is why I use offlineimap to local Maildirs and mutt to talk
>> to them.
>
> I didn't say "perfect" -- I said "proper".   :-)
>
> Mutt is not a proper IMAP client so far as I can tell, for example.
>
> Pine, Emacs Wanderlust, Thunderbird, Apple Mail, etc. are all "proper"
> IMAP clients in most respects, I think.  Pick your poison.  :-)

Thunderbird? my understanding (from watching people use it) is that it wants to 
pull a copy of all your mail to the local box before processing it. how is this 
a proper IMAP client?

David Lang

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: VMware for Cyrus?

[David Lang]

On Mon, 9 Nov 2009, Sebastian Hagedorn wrote:


--On 9. November 2009 14:10:54 +0100 Simon Matter 
wrote:


While virtualization has advantages it has also disadvantages. One thing
is that it introduces an additional layer of complexity into the game.
It's my impression that in many areas virtualization gets introduced not
because of technical reasons but because of political pressure.


In our case I wouldn't necessarily call it political pressure ... it's more
like organizational pressure. We have fewer personnel resources than we
used to, and have to run more systems with them!


and every time you introduce virtualization you introduce an additional system 
that you need to run.


remember that you need to admin all the virtual machines, just like you would if 
they were on their own physical boxes, plus you now need to admin the host OS.


David Lang
Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: VMware for Cyrus?

[David Lang]

On Mon, 9 Nov 2009, Sebastian Hagedorn wrote:


--On 9. November 2009 08:37:46 -0300 Reinaldo de Carvalho
 wrote:


You need analyse the I/O consumition.

# iostat -d 1


I trust real world experiences more than benchmarks ... either people on
this list have successfully run Cyrus under ESX or they haven't. I don't
want to be the first to try it.


iostat is not a benchmark, it's a tool to measure your I/O patterns (number of 
I/O's, size of transfers, etc)


while I understand your desire to have real-world experiance, you do need to 
know your usage patterns to size the system accordingly.


David Lang
Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Exec'ing a script from Cyrus when imapd has a client

[David Lang]

On Mon, 26 Oct 2009, Xavier Bestel wrote:

> On Mon, 2009-10-26 at 10:07 -0700, David Lang wrote:
>> On Mon, 26 Oct 2009, Greg A. Woods wrote:
>>
>>> At Fri, 23 Oct 2009 13:37:30 -0700 (PDT), David Lang 
>>>  wrote:
>>> Subject: Re: Exec'ing a script from Cyrus when imapd has a client
>>>>
>>>> I possibly missed it, but I didn't see anything that said that fetchmail 
>>>> was
>>>> grabbing things via IMAP.
>>>
>>> Yup, I think you missed it.
>>>
>>>> if you have intermittent/expensive-per-min internet connectivity doing 
>>>> something
>>>> like this has value.
>>>
>>> Nope, not really.   All modern useful IMAP clients can work offline too.
>>>
>>> All another IMAP server is doing is adding to the complexity _and_
>>> decreasing, i.e. lowering, the robustness of the overall solution.
>>>
>>>> another reason to run your own server is just to be free from quotas. many 
>>>> ISPs
>>>> have small mail quotas.
>>>
>>> All modern useful IMAP clients can also store message locally -- moving
>>> them from server to server, or server to local (or back), is as simple
>>> as selecting and saving/dragging messages between folders.
>>
>> in my mind, having the IMAP client copy all messages to the local drive goes 
>> a
>> long way to defeating the benifits of using IMAP in the first place.
>
> The drive is not exactly local, it's on a separate server (which does
> mainly mail and file server), which is accessed remotely or not,
> depending on who uses it and when.

I was responding to Greg, who was saying that all modern IMAP clients will copy 
the mail to the local drive so that they can work offline.

David Lang

>> what do you consider a 'modern IMAP client' that is actually reasonably
>> efficiant to use? there are a lot of 'IMAP clients' out there that treat 
>> IMAP as
>> if it was POP (downloading everything and then working on it locally, taking
>> _no_ advantage of the server capabilities) I am interested in finding such a
>> client because at the moment I am using pine and mulberry, both of which are
>> very good at using the server, but not exactly 'modern'.
>
> I admit I have yet to find the ideal IMAP client, efficiency-wise. But
> that's another problem.
>
>   Xav
>
>
>
>

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Exec'ing a script from Cyrus when imapd has a client

[David Lang]

On Mon, 26 Oct 2009, Greg A. Woods wrote:

> At Fri, 23 Oct 2009 13:37:30 -0700 (PDT), David Lang 
>  wrote:
> Subject: Re: Exec'ing a script from Cyrus when imapd has a client
>>
>> I possibly missed it, but I didn't see anything that said that fetchmail was
>> grabbing things via IMAP.
>
> Yup, I think you missed it.
>
>> if you have intermittent/expensive-per-min internet connectivity doing 
>> something
>> like this has value.
>
> Nope, not really.   All modern useful IMAP clients can work offline too.
>
> All another IMAP server is doing is adding to the complexity _and_
> decreasing, i.e. lowering, the robustness of the overall solution.
>
>> another reason to run your own server is just to be free from quotas. many 
>> ISPs
>> have small mail quotas.
>
> All modern useful IMAP clients can also store message locally -- moving
> them from server to server, or server to local (or back), is as simple
> as selecting and saving/dragging messages between folders.

in my mind, having the IMAP client copy all messages to the local drive goes a 
long way to defeating the benifits of using IMAP in the first place.

what do you consider a 'modern IMAP client' that is actually reasonably 
efficiant to use? there are a lot of 'IMAP clients' out there that treat IMAP 
as 
if it was POP (downloading everything and then working on it locally, taking 
_no_ advantage of the server capabilities) I am interested in finding such a 
client because at the moment I am using pine and mulberry, both of which are 
very good at using the server, but not exactly 'modern'.

David Lang

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Exec'ing a script from Cyrus when imapd has a client

[David Lang]

On Fri, 23 Oct 2009, Greg A. Woods wrote:

> At Fri, 23 Oct 2009 13:00:34 -0700 (PDT), David Lang 
>  wrote:
> Subject: Re: Exec'ing a script from Cyrus when imapd has a client
>>
>> as long as you are willing to limit yourself to a single MUA on a single
>> desktop/laptop.
>>
>> if you want to be able to access your mail from different devices you need a
>> mail server, not just a MUA
>
> Huh?  What in the heck are you talking about?
>
> I run multiple IMAP clients (some on the same computer, some on
> different computers) all _simultaneously_, all accessing a half-dozen
> different mail accounts on different servers around the Internet.
>
> All my MUAs access the same folders and same messages directly, and
> simultaneously.
>
> I certainly don't need yet another IMAP server and a whole bunch of
> unnecessary complexity with things like fetchmail just to do this.

I possibly missed it, but I didn't see anything that said that fetchmail was 
grabbing things via IMAP.

if the remote account is POP then doing something like this has value

if you have intermittent/expensive-per-min internet connectivity doing 
something 
like this has value.

I did something similar to this several years ago where a non-profit only had 
dial-up internet. I ran a local cyrus server and then had a process to bring up 
the internet connection as-needed. In that case I just polled for mail every 
half hour and people were willing to live with that at that time. In this case 
I 
actually used UUCP through a fixed mail server to do the routing instead of 
fetchmail, but the basic concept is the same.

another reason to run your own server is just to be free from quotas. many ISPs 
have small mail quotas.

David Lang

> What I would really like to learn is why anyone would falsely believe
> that they do somehow need their own IMAP server for this reason.  There
> must be some false conception or expectation permeating some parts of
> the ether out there.
>
>

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Exec'ing a script from Cyrus when imapd has a client

[David Lang]

On Fri, 23 Oct 2009, Greg A. Woods wrote:

> At Thu, 22 Oct 2009 18:43:41 -0700 (PDT), David Lang 
>  wrote:
> Subject: Re: Exec'ing a script from Cyrus when imapd has a client
>>
>> there can be cases where you are providing mail services for several people, 
>> or
>> have multiple machines you use yourself where having an IMAP server is
>> worthwhile.
>
> Neither of those things make any real sense whatsoever.  They certainly
> don't define any clear requirements that make sense in this context.
>
> Every modern and useful IMAP-capable MUA can collect e-mail from any
> combination of many IMAP servers anywhere and everywhere all at once.
>
> If "fetchmail" can fetch the mail from an IMAP server, then so can any
> MUA.
>
> Just get rid of all the unnecessary complexity in the middle and just
> use the MUA for what it's designed to be used for!

as long as you are willing to limit yourself to a single MUA on a single 
desktop/laptop.

if you want to be able to access your mail from different devices you need a 
mail server, not just a MUA

David Lang

>
>
>> now, it's unusual to use something like this without having a full MTA, but 
>> it's
>> not unheard of.
>
> It's not unusual for people to create all kinds of crazy complicated
> setups that have no real purpose, in every domain in life.
>
> I'm sure I make my own life more complicated than it needs to be in some ways.
>
> However things do not _need_ to be made more complicated than necessary,
>
> Here the OP's question provides a perfect clue showing that something is
> far more complicated than it needs to be because we see that it will
> even have to get more complex (and even less robust) before it begins to
> work the way it would actually work without any of this unnecessary
> complexity in the middle in the first place.
>
>

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Exec'ing a script from Cyrus when imapd has a client

[David Lang]

On Thu, 22 Oct 2009, Greg A. Woods wrote:


At Tue, 20 Oct 2009 22:54:12 +0200, Xavier Bestel  wrote:
Subject: Re: Exec'ing a script from Cyrus when imapd has a client


Le mardi 20 octobre 2009 à 13:00 -0700, David Lang a écrit :

On Tue, 20 Oct 2009, Greg A. Woods wrote:


At Tue, 20 Oct 2009 19:36:24 +0200, Xavier Bestel  wrote:
Subject: Exec'ing a script from Cyrus when imapd has a client


I have a small install with cyrus-imapd 2.3.14, which reads some of its
mails with fetchmail. To limit the delay in mail delivery, fetchmail
awakes each minute to get mails.
What I would like is let fetchmail do that only when there's a client
actually reading its mails, i.e. an MUA actually connected to imapd.


I don't get it.  Are you saying you are using fetchmail to inject
messages into a locally running Cyrus install which you then connect to
with a locally running IMAP MUA?


I think what he is saying is that he does not have a MTA. he uses fetchmail to
download mail from elsewhere and put it in cyrus.

currently he crons fetchmail to run once a min so that when people are logged in
they see new mail with low latencies.

however, if nobody is logged in to Cyrus, this is a waste of time, and he would
be better off running fetchmail less frequently (or not at all).

so he is asking if there is a way to tell if anyone is connected to Cyrus or
not, so that if not he can skip the fetchmail run.


Yes, that's it, precisely.



OK, that's just the kind of crazy overkill and complexity I was talking about.

If you don't have an MTA (but you do have an IMAP-capable MUA), then you
really do not need or want Cyrus, and you certainly do not want
fetchmail either.

Just use your IMAP MUA directly for goodness sake!


there can be cases where you are providing mail services for several people, or 
have multiple machines you use yourself where having an IMAP server is 
worthwhile.


and if you are going to setup an IMAP server, why use anything less than the 
best? ;-)


now, it's unusual to use something like this without having a full MTA, but it's 
not unheard of.


David Lang
Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: cyrus replication : master to replica and replica to master

[David Lang]

On Thu, 22 Oct 2009, David Touzeau wrote:

> On Thu, Oct 22, 2009 at 12:56:03AM -0700, Jon . wrote:
>> On Wed, Oct 21, 2009 at 9:20 PM, Rob Mueller  wrote:
>> ...
>>
>>> The difference between "in theory this would work" and the practice
> of
>>> actually doing it are huge. Basically it works only if you are 100%
> sure
>>> that only one side is ever being accessed at a time. eg.
> IMAP/POP/LMTP/etc.
>
> Pretty much.  With appropriate fencing, non-local bind and a service IP
> address that's feasible.  But Rob won't let me do it.  Fair enough too,
> it's pretty messy.

implementing this should not be that hard

allow non-local bind in /etc/sysctl

heartbeat (linux-ha.org) can handle moving the service IP and fencing (up to 
and 
including turning a box off if the cluster decides that it has failed)

I've been doing this (without going to the extent of turning the failed box 
off) 
on my firewalls for years. it sounds more complicated than it is.

David Lang

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Exec'ing a script from Cyrus when imapd has a client

[David Lang]

On Tue, 20 Oct 2009, Greg A. Woods wrote:

> At Tue, 20 Oct 2009 19:36:24 +0200, Xavier Bestel  
> wrote:
> Subject: Exec'ing a script from Cyrus when imapd has a client
>>
>> I have a small install with cyrus-imapd 2.3.14, which reads some of its
>> mails with fetchmail. To limit the delay in mail delivery, fetchmail
>> awakes each minute to get mails.
>> What I would like is let fetchmail do that only when there's a client
>> actually reading its mails, i.e. an MUA actually connected to imapd.
>
> I don't get it.  Are you saying you are using fetchmail to inject
> messages into a locally running Cyrus install which you then connect to
> with a locally running IMAP MUA?

I think what he is saying is that he does not have a MTA. he uses fetchmail to 
download mail from elsewhere and put it in cyrus.

currently he crons fetchmail to run once a min so that when people are logged 
in 
they see new mail with low latencies.

however, if nobody is logged in to Cyrus, this is a waste of time, and he would 
be better off running fetchmail less frequently (or not at all).

so he is asking if there is a way to tell if anyone is connected to Cyrus or 
not, so that if not he can skip the fetchmail run.

David Lang


> Maybe you should just eliminate fetchmail from the picture and then see
> if things make more sense.  Just point your MUA at the originating IMAP
> server and eliminate everything in between.
>
> If you're using an IMAP capable client, and you're using IMAP to read
> your e-mail, then you don't need or want fetchmail -- it's just a rather
> nasty old hack for the days before IMAP.  Personally I think the last
> usable copy of fetchmail should be archived away on an ancient 9-track
> magnetic tape reel and put behind a little glass door which has "Break
> glass in case of emergency" written on it.  Everything is ever so much
> easier and simpler if you can just eliminate all the unnecessary
> complexities, such as fetchmail.
>
> (if you want to keep a copy of your messages on your local machine, then
> you can and should probably just use your MUA to do that)
>
>

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

server-side signature verification through IMAP?

[David Lang]

the attached messages were posted to the mulberry mailing list

short version, in order to do s/mime verification the client must retreive the 
entire message to do the verification client-side.


is there any way to do this server-side?

David Lang--- Begin Message ---
Hi.

I've finally identified the source of a problem that has
gradually been driving me nuts.  Part of the finger points at
IMAP, but Mulberry's behavior makes it much worse.  I thought
I'd at least warn others about the problem.

Suppose a message is received with, say, an introductory body
part containing a few kilobytes of comments or information,
followed by many megabytes of "attachment" body parts.  The
obvious thing to do is to open (or synchronize, if working
largely offline) the first body part, read it, and then download
the additional body parts as needed.   Asynchronous prefetching
(which Mulberry doesn't do) aside, that is a model that ought to
be supported by any competent IMAP client-server pair and
Mulberry does it very nicely.

However, if the message is signed with S/MIME, the signature is
over the entire message, including all body parts.  Mulberry
wants to verify signatures when messages are opened.  One can
uncheck that preference option, but, if one does, it is
relatively hard to verify selectively -- no provision for adding
a "verify signature" button, no "verify sig" entry in the
per-message pull-down from the TOC page, etc.  There is a
per-message Verify/Decrypt entry on the main "Message"
pull-down, but it seems to often be grayed out even when signed
messages are selected.

So, suppose one keeps the "Verify signed messages on opening"
preference checked.  Now, if one has a signed message and tries
to open that first body part, Mulberry feels obligated to
download the entire, multi-megabyte message in order to verify
the signature.   No provisions for a "this is going to take a
while, do you mean it" warning or anything else (such as the
traditional "the message you are about to open is large..."
warning/question) -- just innocently click on a message which is
obviously "small first body part, huge attachment(s)" (and less
obviously signed, little pencil icon notwithstanding) and then
go sit on one's hands for a while.  

Mulberry then manages to add insult to injury: when one actually
selects the attachment to open or download it, it generates the
"message you are about to open is large" warning and then, upon
getting a "yes", proceeds to download it again.  I understand
enough of Mulberry's internal model to know why that happens
but, as the number of people who are automatically signing
messages by default continues to rise, the overall picture is
going to become a fairly nasty combination of misfeatures.

At least a "this message is rather large, do you really want to
take the time to download all of the body parts so the signature
can be verified?" dialog box would seem to be in order.

Obviously this will never be noticed by anyone with fast LAN
connections to their IMAP servers.  But for those of us who are
either not as well off technologically or who travel extensively
to environments with poor connectivity...

   john

--- End Message ---
--- Begin Message ---
--On Tuesday, October 13, 2009 6:03 PM -0400 John C Klensin 
 wrote:

> However, if the message is signed with S/MIME, the signature is
> over the entire message, including all body parts.

Ouch. That's a nasty drawback. I wonder if there are IMAP extensions to do 
the verification server-side?


--- End Message ---

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Does Cyrus benefit greatly from increased FS buffer cache?

[David Lang]

On Thu, 16 Apr 2009, Sebastian Hagedorn wrote:


--On 16. April 2009 10:58:15 +1000 Rob Mueller  wrote:


http://blog.fastmail.fm/2007/09/21/reiserfs-bugs-32-bit-vs-64-bit-kernel
s-cache-vs-inode-memory/

Anyone have any specific thoughts?  Is there any other benefit we might
see from large memory allocation in 64-bit architecture?


Given that I wrote that blog post, I can only tell you that in our
environment, 64-bit kernels made a big difference.


I wonder if ext3 behaves differently, Red Hat's 32-bit behaves differently,
or if something altogether different is going on. We are currently running
RHEL 3 in 32-bit mode, our servers have 16 GB, and most of it is used for
caching:

# free
total   used   free sharedbuffers cached
Mem:  16214344   16197612  16732  0  86944   13415172
-/+ buffers/cache:2695496   13518848
Swap:  4192944   84364184508

So it would seem that a 64-bit kernel wouldn't improve on that, right? Or
is that a difference between 2.4 and 2.6?


64 bit kernels will be significantly more efficiant, and more reliable with that 
much memory.


in addition, in 64 bit mode the system is able to use twice as many registers on 
the CPU, which can frequently be a significant win in and of itself (even on 
machines with 1G of ram)


I've run both kernels on the same system and always have found that the 64 bit 
kernel is an advantage.


I tend to do the same thing for userspace (unless I am running something that 
doesn't work with 64 bit userspace), but there the benifit is more hit-and-miss


David Lang

ATT1922831.dat
Description: ATT1922831.dat

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: List to Spam Harvest

[David Lang]

On Fri, 27 Feb 2009, John Thomas wrote:

> I know little, so please forgive if this is wrong.
> The following link seems to be crawled by Google and exposes our email
> addresses to spam harvesters.  I wonder if it makes sense and is
> possible to not do this or obfuscate the addresses?
> http://cyrusimap.web.cmu.edu/archive/mailbox.php?mailbox=archive.info-cyrus

if you send mail to a public mailing list it can be harvested by spammers.

David Lang

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Security risk of POP3 & IMAP protocols

[David Lang]

On Fri, 13 Feb 2009, Ian Batten wrote:

> On 13 Feb 09, at 0149, Joseph Brennan wrote:
>>
>> The protocol itself is no less secure than POP.
>
> Security isn't about protocols, it's about systems, and I suspect POP3
> vs IMAP is metonymic for local vs remote mail storage.
>
> I can see an argument that says that one problem with IMAP is that
> your entire mail store, which is much more interesting to an attacker
> than a message in flight or your current mail pending collection a la
> POP3, is under someone else's control.  So if, say, you use a whole
> disk encryption product, mail delivered via traditional POP3 will be
> wrapped in the arms of the encryption immediately after collection,
> while mail stored on a remote server and accessed via IMAP will have
> whatever security features the server has.
>
> If you control the IMAP server (for some suitable value of `you') then
> a risk assessment is the same task in both scenarios.  However, if, as
> is common in many situations, the IMAP server isn't within the scope
> of a risk assessment, then I can imagine that your 27001 life is a
> little easier if you don't have a large pool of potentially sensitive
> data under someone else's (for some value of `someone else')
> control.   Data at rest is a different class of problem to data in
> motion, and IMAP implies a _lot_ of data at rest.
>
> To make this more concrete, imagine you're an HR department within a
> large enterprise, handling job applications, CVs, disciplinary
> processes, dismissals, etc.  You need to demonstrate your compliance
> with your local data protection regulations.  The theft of a day's
> email would be severely embarrassing, but is analogous to the theft of
> a day's postal mail: a risk which most businesses would accept.  It
> would expose limited amounts of information about a small subset of
> your employees.
>
> However, the theft of a year's or a decade's email would expose
> substantial information about a large percentage of your employees,
> and would be analogous to allowing a few filing cabinets to be stolen.
>
> Your email system is run by your corporation's IT function in another
> jurisdiction which has laxer data protection laws --- say, an EU
> company whose head office is in the USA.
>
> Do you (a) store all your long term records in the other jurisdiction
> or (b) store them locally?
>
> Now I'm not defending the argument, and indeed here we have ~4TB of
> email on our Cyrus servers.  But I don't think the position is
> entirely without merit, and having gone through the simplifying and
> distorting mirror of sales droids I can see where it's come from...

the flip side of the complience issue is that it's a LOT easier to control 
retention policies (including backups) on a central server than on everybody's 
individual desktops/laptops.

as for the concerns about laxer data security in other juristictions, that's 
something that needs to be addressed when you outsource your mail (via contract 
with whoever you are having host your mail for you)

David Lang

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: choosing a file system

[David Lang]

On Sat, 3 Jan 2009, Rob Mueller wrote:

>> But the new Solid-State-Disks seem very promising. They are claimed to
>> give 30x the throughput of a 15k rpm disk. If IO improves by 30 times
>> that should make all these optimizations unnecessary.
>> As my boss used to tell me ... Good hardware always compensates for
>> not-so-good software.
>
> What we've found is that the meta-data (eg mailbox.db, seen db's, quota
> files, cyrus.* files) use WAY more IO than the email data, but only use
> 1/20th the space.
>
> By separating the meta data onto RAID1 10k/15k RPM drives, and the email
> data onto RAID5/6 7.2k RPM drives, you can get a good balance of
> space/speed.

how do you move the cyrus* files onto other drives?

David Lang

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: choosing a file system

[David Lang]

On Wed, 31 Dec 2008, Adam Tauno Williams wrote:

> On Wed, 2008-12-31 at 11:47 +0100, LALOT Dominique wrote:
>> Thanks for everybody. That was an interesting thread. Nobody seems to
>> use a NetApp appliance, may be due to NFS architecture problems.
>
> Personally, I'd never use NFS for anything.  Over the years I've had way
> to many NFS related problems on other things to ever want to try it
> again.

NFS has some very interesting capabilities and limitations. it's really bad for 
multiple processes writing to the same file (the cyrus* files for example) and 
for atomic actions (writing the message files for example)

there are ways that you can configure it that will work, but unless you already 
have a big NFS server you are probably much better off using a mechanism that 
makes the drives look more like local drives (SAN, iSCSI, etc) or try one of 
the 
cluster filesystems that has different tradeoffs than NFS does

>> I believe I'll look to ext4 that seemed to be available in last
>> kernel, and also to Solaris, but we are not enough to support another
>> OS.
>
> We've used Cyrus on XFS for almost a years, no problems.
>
> In regards to ext3 I'd pay attention to the vintage of problem reports
> and performance issues;  ext3 of several years ago is not the ext3 of
> today, many improvements have been made.  "data=writeback" mode can help
> performance quite a bit, as well as enabling "dir_index" if it isn't
> already (did it ever become the default?).  The periodic fsck can also
> be disabled via tune2fs.   I only point this out since, if you already
> have any ext3 setup,  trying the above are all painless and might buy
> you something.

it's definantly worth testing different filesystems. I last did a test about 
two 
years ago and confirmed XFS as my choice. I have one instance of cyrus still 
running on ext3 and I definantly see it as a user in the performance.

David Lang

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: suggestion need to design an email system.

[David Lang]

On Thu, 18 Sep 2008, Adam Tauno Williams wrote:

> On Wed, 2008-09-17 at 21:12 -0700, David Lang wrote:
>> On Wed, 17 Sep 2008, Wesley Craig wrote:
>>> On 17 Sep 2008, at 11:40, Jens Hoffrichter wrote:
>>>> Why does cyrus need it's own
>>>> structure for the mailboxes, which is similar, but not wholly
>>>> compatible, to maildir. Maildir and cyrus both suffer from the same
>>>> disadvantages (huge needs in terms of inodes etc.), yet I see no
>>>> distinctive advantage for the cyrus mailbox format to maildir.
>>> Performance.  I could go on & on, but that's the answer, basically.
>> actually, I suspect that Cyrus existed before Maildir and since Cyrus is a
>> 'black box' server there's no advantage to moving to maildir.
>> Also Cyrus has cache, index, and flags stored seperatly from the message 
>> itself,
>> which means that when these things are changed the message file itself 
>> doesn't
>> need to change.
>
> Yep,  which is why the sealed mailstore is a good idea: meta-data.  You
> can keep reliable meta-data if third parties can go in and muck about.

one of those "can" should be "can't" :-)

David Lang

>> doign a quick google check on maildir it also appears that maildir is not as
>> standard as people think it is, it's defined almost entirely by the
>> implementation (DJB started it, but never worked to turn it into a standard 
>> for
>> others to use)
>
>
> 
> Cyrus Home Page: http://cyrusimap.web.cmu.edu/
> Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
> List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
>

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: suggestion need to design an email system.

[David Lang]

On Wed, 17 Sep 2008, Scott M. Likens wrote:

> Hi David,
>
> If you want ZFS you have several choices, OS X (Leopard), as there is a
> development version that supports RAID-Z and works quite well.  You
> can't boot off of it, but that will change in Snow Leopard.
>
> Additionally you can use Opensolaris such as Nexenta, or you can use
> fuse to get ZFS into Linux.

fuse on linux is not a real option for most uses due to the performance hit.

so you have OS X and Solaris (with opensolaris varients), that's not a very 
wide 
supproted base for running servers.

> Truthfully I would rather use Nexenta to get ZFS which at least gives
> you a decent working system... Unlike a standard Solaris where you are
> forced to get either Sun One (Forte?) or an ancient version of gcc off
> of sunfreeware and start building.   :(

no comment

> However I do look forward to the day when Sun dual licenses ZFS as GPL
> and sticks it in the Linux kernel and throws us all a wrench, for good
> or bad ZFS introduces some excellent overlooked ideas that it's about
> damned time someone introduced.

I don't disagree that ZFS is a prod to get development on filesystems moving 
again or to say that they didn't introduce any good ideas, but I do disagree 
with people who seem to think that ZFS is perfect, and is so much better then 
any other filesystem that the availablity of ZFS should be the only 
consideration on what OS to run.

David Lang

> Scott
>
> David Lang wrote:
>> On Wed, 17 Sep 2008, Scott M. Likens wrote:
>>
>>> With ZFS your leaving a "ton" of stone-age worries behind.  You can go
>>> much beyond inodes in the perks of ZFS.
>>
>> and gaining some new worries along the way. while some are convinced
>> that ZFS is the best thing ever others see it as trading a set of
>> known problems for a set of unknown problems (plus it severly limits
>> what OS you can run, which can bring it's own set of problems along)
>>
>> David Lang
>>
>>> Vincent Fox wrote:
>>>> Wesley Craig wrote:
>>>>
>>>>>>  Maildir and cyrus both suffer from the same
>>>>>> disadvantages (huge needs in terms of inodes etc.),
>>>>>>
>>>> With ZFS, inodes are among the many stone-age worries you leave behind.
>>>>
>>>> ;-)
>>>>
>>>> 
>>>> Cyrus Home Page: http://cyrusimap.web.cmu.edu/
>>>> Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
>>>> List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>> 
>>> Cyrus Home Page: http://cyrusimap.web.cmu.edu/
>>> Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
>>> List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
>>>
>
>

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: suggestion need to design an email system.

[David Lang]

On Wed, 17 Sep 2008, Scott M. Likens wrote:

> With ZFS your leaving a "ton" of stone-age worries behind.  You can go
> much beyond inodes in the perks of ZFS.

and gaining some new worries along the way. while some are convinced that ZFS 
is 
the best thing ever others see it as trading a set of known problems for a set 
of unknown problems (plus it severly limits what OS you can run, which can 
bring 
it's own set of problems along)

David Lang

> Vincent Fox wrote:
>> Wesley Craig wrote:
>>
>>>>  Maildir and cyrus both suffer from the same
>>>> disadvantages (huge needs in terms of inodes etc.),
>>>>
>> With ZFS, inodes are among the many stone-age worries you leave behind.
>>
>> ;-)
>>
>> 
>> Cyrus Home Page: http://cyrusimap.web.cmu.edu/
>> Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
>> List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
>>
>>
>> !DSPAM:48d1d451236501804284693!
>>
>>
>>
>
> 
> Cyrus Home Page: http://cyrusimap.web.cmu.edu/
> Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
> List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
>

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: suggestion need to design an email system.

[David Lang]

On Wed, 17 Sep 2008, Wesley Craig wrote:

> On 17 Sep 2008, at 11:40, Jens Hoffrichter wrote:
>> Why does cyrus need it's own
>> structure for the mailboxes, which is similar, but not wholly
>> compatible, to maildir. Maildir and cyrus both suffer from the same
>> disadvantages (huge needs in terms of inodes etc.), yet I see no
>> distinctive advantage for the cyrus mailbox format to maildir.
>
> Performance.  I could go on & on, but that's the answer, basically.

actually, I suspect that Cyrus existed before Maildir and since Cyrus is a 
'black box' server there's no advantage to moving to maildir.

Also Cyrus has cache, index, and flags stored seperatly from the message 
itself, 
which means that when these things are changed the message file itself doesn't 
need to change.

doign a quick google check on maildir it also appears that maildir is not as 
standard as people think it is, it's defined almost entirely by the 
implementation (DJB started it, but never worked to turn it into a standard for 
others to use)

David Lang

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Cyrus vs Dovecot

[David Lang]

On Wed, 13 Aug 2008, Wesley Craig wrote:

> On 13 Aug 2008, at 10:31, kbajwa wrote:
>> I think you are missing a point which is most important, i.e., what
>> type of
>> support Cyrus vs Dovecot offers. In my experience:
>>
>> Cyrus  =  0
>> Dovecot=  100
>
> As someone who answers many help requests for cyrus (and I'm very far
> from the only one), I can honestly say I've never seen a requests
> from you.  Perhaps you've had a lot of occasion to ask for help with
> Dovecot.  I'm happy to hear you've gotten that help.  Community is a
> lot of what open source software is about.  As for your experience
> with the cyrus imapd community, perhaps your sample size is too small.
>
> Or perhaps you're thinking of paid support?  Because I know very well
> that you can get that for cyrus imap.

can you provide links to where from?

David Lang

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Cyrus infrastructure performance less than expected

[David Lang]

On Tue, 29 Apr 2008, Eric Déchaux wrote:


Pascal Gienger a écrit :

Eric Déchaux <[EMAIL PROTECTED]> wrote:


The older infrastructure can stand the 42 000 concurrent sessions, the
new one can't : I was expecting each frontend to be able to handle 5 500
concurrent sessions but they are not. Around 3 000 / 3 500 concurrent
sessions the frontends begin to SWAP and are not more able keep up the
load.


Did I undertand correctly: You are virtualizing each component and 
your frontends begin to swap in their virtual environment?
Is there any reason why you don't assign more RAM to them? Does your 
frontend virtual machines run a 64 bit kernel?
That's it. The frontend begin to swap, not the ESX. We have scaled the 
ESX RAM to avoid SWAPING on the ESX level.
In fact we can't assign anymore memory as the X4600 can't handle more 
than 64 Gb.


Another point is we have doubled total RAM compared to the actual 
platfrom and I don't understand why it behaves so badly.


were you virtualized before? adding virtualization causes a fairly significant 
overhead.


David Lang



We abandoned all Linux for our Cyrus Servers and switched to Solaris 
10 with Zoning and ZFS. We have less concurrent users than you but 
more storage (10 T at the moment).


I would have loved to put Solaris, Zones and Massaging Server here but 
it was not a possibility. Custorme chose was VMware + Linux + Cyrus.



Many thanks.
--
Eric


Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Miserable performance of cyrus-imapd 2.3.9 -- seems to be lockingissues

[David Lang]

 disks to 3.

ZFS is not available everywhere, and it is not suitable for all workloads 
(specificly database type workloads, which is a fair approximation for cyrus)

you say it's not worth reducing 4 disks to 3, but what about 6 disks to 4?
(useing your example of a machine with 4 SATA drives it's the difference 
between 
useing the machine you have or buying a new one)

if that's not enough, what about 8 disks to 5? (6 if you do raid 6 or want a 
hot-spare)

what is the point that you would consider the difference valid?

David Lang

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Miserable performance of cyrus-imapd 2.3.9 -- seems to be lockingissues

[David Lang]

On Tue, 4 Mar 2008, Ian G Batten wrote:

>  software RAID5 is a performance
> disaster area at the best of times unless it can take advantage of
> intimate knowledge of the intent log in the filesystem (RAID-Z does
> this),

actually, unless you have top-notch hardware raid controllers, software raid 5 
may be better then hardware raid 5. many controllers only do a decent job doing 
raid 0 or raid 1. this is something to measure with your particular hardware. 
I've seen many cases where the cards do a horrible job with raid 5 compared to 
software.

> and three-disk RAID5 assemblages are a performance disaster
> area irrespective of hardware in a failure scenario.  The rebuild
> will involve taking 50% of the IO bandwidth of the two remaining
> disks in order to saturate the new target; rebuild performance ---
> contrary to intuition --- improves with larger assemblages as you can
> saturate the replacement disk with less and less of the bandwidth of
> the surviving spindles.

this is true, up to the point where the bus gets saturated with the re-sync 
info, after that more disks will not improve the rebuild time.

> For a terabyte, 3x500GB SATA drives in a RAID5 group will be blown
> out of the water by 4x500GB SATA drives in a RAID 0+1 configuration
> in terms of performance and (especially) latency, especially if it
> can do the Solaris trick of not faulting an entire RAID 0 sub-group
> if one spindle fails.  Rebuild still isn't pretty, mind you.

either of these cases will survive a single drive failure, what I would look at 
is either 3x1TB drives in raid 1, or 4x500G drives in raid 6 to get the ability 
to survive 2 drives failing.

it takes long enough to rebuild an array with large drives that the chances of 
a 
second drive failing during the rebuild become noticable.

David Lang

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: LARGE single-system Cyrus installs?

[David Lang]

On Thu, 22 Nov 2007, Gabor Gombas wrote:

> On Tue, Nov 20, 2007 at 09:56:37AM -0800, David Lang wrote:
>
>> for cyrus you should have the same sort of requirements that you would have 
>> for
>> a database server, including the fact that without a battery-backed disk 
>> cache
>> (or solid state drive) to handle your updates, you end up being throttled by
>> your disk rotation rate (you can only do a single fsync write per rotation, 
>> and
>> that good only if you don't have to seek), RAID 5/6 arrays are even worse, as
>> almost all systems will require a read of the entire stripe before writing a
>> single block (and it's parity block) back out, and since the stripe is
>> frequently larger then the OS readahead, the OS throws much of the data away
>> immediatly.
>
> You're mixing things up. Readahead has absolutely zero influence on when
> data is evicted from the cache.

if the system is set to do a 1M readahead and to do that readahead it needs to 
read in 5M of data to verify the integrity, the system doesn't keep all 5M of 
data in it's cache, only the 1M that is it's readahead (or at least in some 
cases this is true)

David Lang

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: LARGE single-system Cyrus installs?

[David Lang]

On Wed, 21 Nov 2007, Ian G Batten wrote:

>> however a fsync on a journal ed filesystem just means the data needs to be
>> written to the journal, it doesn't mean that the journal needs to be 
>> flushed to
>> disk.
>> 
>> on ext3 if you have data=journal ed then your data is in the journal as well 
>> and
>> all that the system needs to do on a fsync is to write things to the 
>> journal (a
>> nice sequential write),
>
> Assuming the journal is on a distinct device and the distinct device can take 
> the load.  It isn't on ZFS, although work is in progress.

I was responding to the comments about ext3 and other journal ed filesystems as 
alternatives to zfs and the claim that doing a fsync on one of them required 
flushing the entire journal. sorry if I wasn't clear enough about this.

David Lang


Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: LARGE single-system Cyrus installs?

[David Lang]

On Tue, 20 Nov 2007, Ian G Batten wrote:

> On 20 Nov 07, at 1332, Michael R. Gettes wrote:
>
>> I am wondering about the use of fsync() on journal'd file systems
>> as described below.  Shouldn't there be much less use of (or very
>> little use) of fsync() on these types of systems?  Let the journal
>> layer due its job and not force it within cyrus?  This would likely
>> save a lot of system overhead.
>
> fsync() forces the data to be queued to the disk.  A journaling
> filesystem won't usually make any difference, because no one wants to
> keep an intent log of every 1 byte write, or the 100 overwrites of
> the same block.  If you want every write() to go to disk,
> immediately, the filesystem layout doesn't really matter: it's just a
> matter of disk bandwidth.  Journalling filesystems are more usually
> concerned with metadata consistency, so that the filesystem isn't
> actively corrupt if the music stops at the wrong point in a directory
> create or something.

however a fsync on a journaled filesystem just means the data needs to be 
written to the journal, it doesn't mean that the journal needs to be flushed to 
disk.

on ext3 if you have data=journaled then your data is in the journal as well and 
all that the system needs to do on a fsync is to write things to the journal (a 
nice sequential write), and everything is perfectly safe. if you have 
data=ordered (the default for most journaled filesystems) then your data isn't 
safe when the journal is written and two writes must happen on a fsync (one for 
the data, one for the metadata)

for cyrus you should have the same sort of requirements that you would have for 
a database server, including the fact that without a battery-backed disk cache 
(or solid state drive) to handle your updates, you end up being throttled by 
your disk rotation rate (you can only do a single fsync write per rotation, and 
that good only if you don't have to seek), RAID 5/6 arrays are even worse, as 
almost all systems will require a read of the entire stripe before writing a 
single block (and it's parity block) back out, and since the stripe is 
frequently larger then the OS readahead, the OS throws much of the data away 
immediatly.

if we can identify the files that are the bottlenecks it would be very 
interesting to see the result of puttng them on a solid-state drive.

David Lang

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: squatter running longer than 24 hours

[David Lang]

On Mon, 22 Oct 2007, Rob Mueller wrote:

>> squatter would really benefit from incremental updates. At the moment a
>> single new message in a mailbox containing 20k messages causes it to read
>> in all the existing messages in order to regenerate the index.
>
> We spoke to Ken about this ages back, and even offered to pay for the work
> to make it happen, but it was just around the time CMU hired him, so it
> never actually happened pity. It would be nice to be able to dedicate a
> couple of weeks to rummage around in there and actually make it happen...

postgres has full-text search capabilities at acceptable performance on very 
large databases, their code is BSD so anything relavent coudl be merged into 
cyrus. it may be worth someone looking into their logic.

David Lang

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: UC Davis Cyrus Incident September 2007

[David Lang]

 Omen Wild (University of California Davis)
The root problem seems to be an interaction between Solaris' concept of
global memory consistency and the fact that Cyrus spawns many processes
that all memory map (mmap) the same file.  Whenever any process updates
any part of a memory mapped file, Solaris freezes all of the processes
that have that file mmaped, updates their memory tables, and then
re-schedules the processes to run.  When we have problems we see the load
average go extremely high and no useful work gets done by Cyrus.  Logins
get processed by saslauthd, but listing an inbox either takes a long
time or completely times out.

Apparently AIX also runs into this issue.  I talked to one email
administrator that had this exact issue under AIX.  That admin talked to
the kernel engineers at IBM who explained that this is a feature, not a
bug.  They eventually switched to Linux which solved their issues,
although they did move to more Linux boxes with fewer users per box.


Oh man... Horrible memories just flood right back... Wow.  I was reading
your e-mail and thinking to myself that this sounded like the same problem
we had.  Then I got to the above section and *bam*, there it was...  We
had significant problems with our e-mail last year (this year was a perfect
start!) a week before students came back.  We didn't resolve the problems
until the end of September and we were dismayed at our final solution.

We run Tru64 5.1b on a 4 member cluster.  Tru64's kernel suffers from the
same exact issue as described above.  We have regularly 12,000 cyrus procs
running at any one time during the day, and that cluster also receives on
average 300k-500k e-mails each day (that is after spam/virus work).

What was finally identified was that the number of "processes" that were
mapped to that single physical "executable" (/usr/cyrus/imapd) was causing
a lot of lock contention in the kernel.  The executable would have a link
list of all the processes running off of it in kernel memory.  When one
of the processes would go away, the kernel would start at the beginning
of the list and search for the process in order to clean up its resources.
During that time, the kernel would lock everything and execution would
essentially stop for everything (basically, the whole system appeared to
simply freeze on us).  The kernel would reach a time threshold and stop
in order to let other things happen (unfreeze).  This time was very short,
but if we had a lot of processes going away in a very short period of time,
we would noticeably see the freeze, since the kernel was going into this
lock-down mode a lot in a very short period of time.  That is a simplified
view of what really happened.


could someone whip up a small test that could be used to check different 
operating systems (and filesystems) for this concurrancy problem?


it doesn't even need to use any cyrus code, (in fact it would probably be better 
if it didn't)


it sounds like there are a couple different aspects to check

1. large number of copies of a single program running, find the impact of 
starting and stopping a process

1a. single process that forks lots of copies
1b. master process that execs lots of copies

2. large number of processes mmapping a single file.
2a. impact to add or remove a process from this group
2b. impact on modifying this file


personally I expect 1b and 1a to be significantly different on different OSs. 
some OSs will gain huge memory savings in 1a due to copy-on-write savings (and 
to partially account for this it may be worth making the program allocate a 
chink of ram and write to it after the fork), while on other OSs the overhead of 
multiple mappings of a page will dominate.


David Lang

--On Tuesday, October 16, 2007 3:39 PM -0700 Vincent Fox <[EMAIL PROTECTED]> 
wrote:


 Omen Wild (University of California Davis)
The root problem seems to be an interaction between Solaris' concept of
global memory consistency and the fact that Cyrus spawns many processes
that all memory map (mmap) the same file.  Whenever any process updates
any part of a memory mapped file, Solaris freezes all of the processes
that have that file mmaped, updates their memory tables, and then
re-schedules the processes to run.  When we have problems we see the load
average go extremely high and no useful work gets done by Cyrus.  Logins
get processed by saslauthd, but listing an inbox either takes a long
time or completely times out.

Apparently AIX also runs into this issue.  I talked to one email
administrator that had this exact issue under AIX.  That admin talked to
the kernel engineers at IBM who explained that this is a feature, not a
bug.  They eventually switched to Linux which solved their issues,
although they did move to more Linux boxes with fewer users per box.


Oh man... Horrible memories just flood right back... Wow.  I was reading
your e-mail and thinking 

Re: LARGE single-system Cyrus installs?

[David Lang]

On Tue, 9 Oct 2007, Andrew Morgan wrote:

> On Sat, 6 Oct 2007, Rob Mueller wrote:
>
>> As it turns out, the memory leaks weren't critical, because the the pages do
>> seem to be reclaimed when needed, though it was annoying not knowing exactly
>> how much memory was really free/used. The biggest problem was that with
>> cyrus you have millions of small files, and with a 32bit linux kernel the
>> inode cache has to be in low memory, severely limiting how many files the OS
>> will cache.
>>
>> See this blog post for the gory details, and why a 64-bit kernel was a nice
>> win for us.
>>
>> http://blog.fastmail.fm/2007/09/21/reiserfs-bugs-32-bit-vs-64-bit-kernels-cache-vs-inode-memory/
>
> Yesterday I checked my own Cyrus servers to see if I was running out of
> lowmem, and it sure looked like it.  Lowmem had only a couple MB free, and
> I had 2GB of free memory that was not being used for cache.
>
> I checked again today and everything seems to be fine - 150MB of lowmem
> free and almost no free memory (3GB cached)!  Grrr.
>
> Anyways, I was looking into building a 64-bit kernel.  I'm running Debian
> Sarge (I know, old) on a Dell 2850 with Intel Xeon (Nocona) CPUs and 4GB
> RAM.  My kernel version is 2.6.14.5, built from kernel.org sources.  It
> has "High Memory Support (64GB)" selected.
>
> When I run menuconfig, I'm not seeing any obvious place to switch from
> 32-bit to 64-bit.  Could you elaborate a bit about how you switched to a
> 64-bit kernel?  Also, are you running a full 64-bit distro, or just a
> 64-bit kernel?

you need a full 64 bit toolchain to compile a 64 bit kernel, the easy way to do 
this is to compile the kernel on a 64 bit distro.

if you have the toolchain you can add arch=x86_64 to your make command

if you are not converting everything over to 64 bit remember to enable 32 bit 
userspace (cyrus won't take advantage of all the ram, but the kernel will, so 
it's definantly still a win)

with some older versions of the iptables binaries you can run into trouble with 
a 64 bit kernel and 32 bit userspace. unless you take the time to make sure 
that 
you aren't running versions that have this problem don't execute any iptables 
commands when running in mixed mode.

David Lang

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: need help recovering from disk crash

[David Lang]

On Wed, 26 Sep 2007, Rudy Gevaert wrote:

> David Lang wrote:
>> On Tue, 25 Sep 2007, David Lang wrote:
>> 
>>> On Tue, 25 Sep 2007, Scott M. Likens wrote:
>>> 
>>>> Hi,
>>>> 
>>>> If you have a dump of the mailbox's (ctl_mboxlist) then you can restore
>>>> those, personally I back those up weekly as well as /var/spool/imap
>>> I don't think I have that.
>>> 
>>>> If you don't, re-add the users, then do reconstruct -r -f user.username
>>>> (obviously replace username with the username in question) and it will
>>>> reconstruct the mailbox and find all the folders for you and add them to
>>>> the mailboxes db.
>>>> 
>>>> Then do that on all your users, and you should be good.
>>> Ok, that's not bad.
>> 
>> I'm running into a problem with this, I can do this for one domain, but not 
>> for a second.
>
> I don't know if 2.2 supports virtual domains.  You'd better check that.  I 
> think there is a bug in reconstruct with virtual domains.  I have to run:
>
> reconstruct -r -f user/[EMAIL PROTECTED]
> and
> reconstruct -r -f user/login/[EMAIL PROTECTED]
>
> (I'm also using unix hierarchy seperator, but you aren't I think.)

I tracked this down last night. Since I was also fighting SASL (my old SASL had 
been connected to a postgres database, the new one wasn't compiled to support 
that, so I was having to learn a different way to set passwords) I had gotten 
[EMAIL PROTECTED] to work and be able to login, and apparently if you login 
with one 
virtual domain you can't do anything to any other virtual domain, even if you 
are listed as the admin user. I setup a password for the user cyrus and set it 
to be the admin user and was able to work with all the different domains.

I did run into one other bug that I couldn't figure out. no user was able to 
modify their INBOX (delete items, etc). I could copy messages to folders, but 
not delete anything.

to get back online (and since this is just my home machine) I ended up setting 
anyone all acl's on every inbox.

it sounds like I really need to upgrade to 2.3.x, I downloaded it, but haven't 
compiled it yet.

David Lang

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: need help recovering from disk crash

[David Lang]

On Tue, 25 Sep 2007, David Lang wrote:

> On Tue, 25 Sep 2007, Scott M. Likens wrote:
>
>> Hi,
>>
>> If you have a dump of the mailbox's (ctl_mboxlist) then you can restore
>> those, personally I back those up weekly as well as /var/spool/imap
>
> I don't think I have that.
>
>> If you don't, re-add the users, then do reconstruct -r -f user.username
>> (obviously replace username with the username in question) and it will
>> reconstruct the mailbox and find all the folders for you and add them to
>> the mailboxes db.
>>
>> Then do that on all your users, and you should be good.
>
> Ok, that's not bad.

I'm running into a problem with this, I can do this for one domain, but not for 
a second.

David Lang

>> As far as 2.3.9 vs. 2.2... if you're only dealing with the message
>> spool... it should be easy to massage it together.  if you're trying to
>> bring back duplicate, sieve, seen, etc... it might be a hassle.
>
> it would be really nice to retain the seen state, I don't care about 
> duplicates
> and don't have sieve in use. what else would I loose?
>
>> Rather vague email I just wrote... but you seemed to have the basics...
>> if not reply all.
>
> thanks.
>
> David Lang
>
>> Scott
>>
>> [EMAIL PROTECTED] wrote:
>>> I lost my OS drive on my home server, the mail partition was on a raid array
>>> and survived, I have some of the rest of the config info, but it looks like 
>>> I
>>> lost the configdir contents (the directories are still there, but the files 
>>> are
>>> missing) I may be able to recover some stuff from lost+found if I can get 
>>> hints
>>> on what to search for.
>>>
>>> now, to make things more interesting, the old system was running gentoo and 
>>> has
>>> cyrus 2.3.recent on it
>>>
>>> the new system is ubuntu with 2.2.something on it (I couldn't get a recent
>>> gentoo installer to run reliably on this hardware)
>>>
>>> can I make this work or should I compile 2.3.9?
>>>
>>> with reconstruct -m now working how can I recover the mailbox?
>>>
>>> the good news is that I only have 3 users on the system (with about 3G of 
>>> mail
>>> in several hundred folders betwen us) so manual fixes may be practical
>>>
>>> the config files were saved, and are:
>>>
>>> #cat imapd.conf |grep -v "^#" |grep "^[a-z]"
>>> configdirectory:/var/imap
>>> partition-default:  /movies/imap
>>> sievedir:   /var/imap/sieve
>>> virtdomains:yes
>>> admins: cyrus
>>> hashimapspool:  yes
>>> allowanonymouslogin:no
>>> allowplaintext: yes
>>> sasl_pwcheck_method: auxprop
>>> sasl_auxprop_plugin: sasldb
>>> sasl_mech_list: PLAIN
>>>
>>>
>>> # cat cyrus.conf |grep -v "^ *#" |grep "[a-z]"
>>>recover   cmd="ctl_cyrusdb -r"
>>>idled cmd="idled"
>>>imap  cmd="imapd" listen="imap2" prefork=0
>>>pop3  cmd="pop3d" listen="pop-3" prefork=0
>>>imaps cmd="imapd -s" listen="imaps" prefork=0
>>>pop3s cmd="pop3d -s" listen="pop3s" prefork=0
>>>sieve cmd="timsieved" listen="sieve" prefork=0
>>>lmtpunix  cmd="lmtpd" listen="/var/imap/socket/lmtp" prefork=0
>>>checkpointcmd="ctl_cyrusdb -c" period=30
>>>    delprune  cmd="ctl_deliver -E 3" period=1440
>>>tlsprune  cmd="tls_prune" period=1440
>>> [EMAIL PROTECTED]:/etc# cat cyrus.conf |grep -v "^ *#" |grep "[a-z{}]"
>>> START {
>>>recover   cmd="ctl_cyrusdb -r"
>>>idled cmd="idled"
>>> }
>>> SERVICES {
>>>imap  cmd="imapd" listen="imap2" prefork=0
>>>pop3  cmd="pop3d" listen="pop-3" prefork=0
>>>imaps cmd="imapd -s" listen="imaps" prefork=0
>>>pop3s cmd="pop3d -s" listen="pop3s" prefork=0
>>>sieve cmd="timsieved" listen="sieve" prefork=0
>>>lmtpunix  cmd="lmtpd" listen="/var/imap/socket/lmtp" prefork=0
>>> }
>>> EVENTS {
>>>checkpointcmd="ctl_cyrusdb -c" period=30
>>>delprune  cmd="ctl_deliver -E 3" period=1440
>>>tlsprune  cmd="tls_prune" period=1440
>>> }
>>>
>>> David Lang
>>> 
>>> Cyrus Home Page: http://cyrusimap.web.cmu.edu/
>>> Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
>>> List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
>>>
>>>
>>> !DSPAM:46f8b94b179948275421122!
>>>
>>>
>>>
>>
>>
> 
> Cyrus Home Page: http://cyrusimap.web.cmu.edu/
> Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
> List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
>

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: need help recovering from disk crash

[David Lang]

On Tue, 25 Sep 2007, Scott M. Likens wrote:

> Hi,
>
> If you have a dump of the mailbox's (ctl_mboxlist) then you can restore
> those, personally I back those up weekly as well as /var/spool/imap

I don't think I have that.

> If you don't, re-add the users, then do reconstruct -r -f user.username
> (obviously replace username with the username in question) and it will
> reconstruct the mailbox and find all the folders for you and add them to
> the mailboxes db.
>
> Then do that on all your users, and you should be good.

Ok, that's not bad.

> As far as 2.3.9 vs. 2.2... if you're only dealing with the message
> spool... it should be easy to massage it together.  if you're trying to
> bring back duplicate, sieve, seen, etc... it might be a hassle.

it would be really nice to retain the seen state, I don't care about duplicates 
and don't have sieve in use. what else would I loose?

> Rather vague email I just wrote... but you seemed to have the basics...
> if not reply all.

thanks.

David Lang

> Scott
>
> [EMAIL PROTECTED] wrote:
>> I lost my OS drive on my home server, the mail partition was on a raid array
>> and survived, I have some of the rest of the config info, but it looks like I
>> lost the configdir contents (the directories are still there, but the files 
>> are
>> missing) I may be able to recover some stuff from lost+found if I can get 
>> hints
>> on what to search for.
>>
>> now, to make things more interesting, the old system was running gentoo and 
>> has
>> cyrus 2.3.recent on it
>>
>> the new system is ubuntu with 2.2.something on it (I couldn't get a recent
>> gentoo installer to run reliably on this hardware)
>>
>> can I make this work or should I compile 2.3.9?
>>
>> with reconstruct -m now working how can I recover the mailbox?
>>
>> the good news is that I only have 3 users on the system (with about 3G of 
>> mail
>> in several hundred folders betwen us) so manual fixes may be practical
>>
>> the config files were saved, and are:
>>
>> #cat imapd.conf |grep -v "^#" |grep "^[a-z]"
>> configdirectory:/var/imap
>> partition-default:  /movies/imap
>> sievedir:   /var/imap/sieve
>> virtdomains:yes
>> admins: cyrus
>> hashimapspool:  yes
>> allowanonymouslogin:no
>> allowplaintext: yes
>> sasl_pwcheck_method: auxprop
>> sasl_auxprop_plugin: sasldb
>> sasl_mech_list: PLAIN
>>
>>
>> # cat cyrus.conf |grep -v "^ *#" |grep "[a-z]"
>>recover   cmd="ctl_cyrusdb -r"
>>idled cmd="idled"
>>imap  cmd="imapd" listen="imap2" prefork=0
>>pop3  cmd="pop3d" listen="pop-3" prefork=0
>>imaps cmd="imapd -s" listen="imaps" prefork=0
>>pop3s cmd="pop3d -s" listen="pop3s" prefork=0
>>sieve cmd="timsieved" listen="sieve" prefork=0
>>lmtpunix  cmd="lmtpd" listen="/var/imap/socket/lmtp" prefork=0
>>checkpointcmd="ctl_cyrusdb -c" period=30
>>delprune  cmd="ctl_deliver -E 3" period=1440
>>tlsprune  cmd="tls_prune" period=1440
>> [EMAIL PROTECTED]:/etc# cat cyrus.conf |grep -v "^ *#" |grep "[a-z{}]"
>> START {
>>recover   cmd="ctl_cyrusdb -r"
>>idled cmd="idled"
>> }
>> SERVICES {
>>imap  cmd="imapd" listen="imap2" prefork=0
>>pop3  cmd="pop3d" listen="pop-3" prefork=0
>>imaps cmd="imapd -s" listen="imaps" prefork=0
>>pop3s cmd="pop3d -s" listen="pop3s" prefork=0
>>sieve cmd="timsieved" listen="sieve" prefork=0
>>lmtpunix  cmd="lmtpd" listen="/var/imap/socket/lmtp" prefork=0
>> }
>> EVENTS {
>>checkpointcmd="ctl_cyrusdb -c" period=30
>>delprune  cmd="ctl_deliver -E 3" period=1440
>>tlsprune  cmd="tls_prune" period=1440
>> }
>>
>> David Lang
>> 
>> Cyrus Home Page: http://cyrusimap.web.cmu.edu/
>> Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
>> List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
>>
>>
>> !DSPAM:46f8b94b179948275421122!
>>
>>
>>
>
>

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Virtual Domains

[David Lang]

I had all sorts of problems getting this to work.

I have my firewall running sendmail sending the messages to an internal server 
via lmtp, and then authenticating against postgres. the biggest problems were 
getting the lmtp connection to include the domain of the destination and 
makeing 
the authentication pass through the domain the user typed in.

David Lang

my cyrus.conf is

asgard dlang # cat /etc/cyrus.conf
# $Header: /var/cvsroot/gentoo-x86/net-mail/cyrus-imapd/files/cyrus.conf,v 1.4 
2004/07/18 04:02:23 dragonheart Exp $

# Standard standalone server configuration.

START {
   # Do not delete this entry!
   recover   cmd="ctl_cyrusdb -r"

   # This is only necessary if using idled for IMAP IDLE.
   idled cmd="idled"
}

# UNIX sockets start with a slash and are put into /var/imap/socket.
SERVICES {
   # Add or remove based on preferences.
   imap  cmd="imapd" listen="imap2" prefork=0
   pop3  cmd="pop3d" listen="pop-3" prefork=0

   # Don't forget to generate the needed keys for SSL or TLS
   # (see doc/html/install-configure.html).
   imaps cmd="imapd -s" listen="imaps" prefork=0
   pop3s cmd="pop3d -s" listen="pop3s" prefork=0

   sieve cmd="timsieved" listen="sieve" prefork=0

   # at least one LMTP is required for delivery
   lmtp  cmd="lmtpd -a" listen="lmtp" prefork=0
   lmtpunix  cmd="lmtpd" listen="/var/imap/socket/lmtp" prefork=0

   # this is only necessary if using notifications
   #notify   cmd="notifyd" listen="/var/imap/socket/notify" proto="udp" 
prefork=1
}

EVENTS {
   # This is required.
   checkpointcmd="ctl_cyrusdb -c" period=30

   # This is only necessary if using duplicate delivery suppression.
   delprune  cmd="ctl_deliver -E 3" period=1440

   # This is only necessary if caching TLS sessions.
   tlsprune  cmd="tls_prune" period=1440
}

my imapd.conf
asgard dlang # cat /etc/imapd.conf
# $Header: /var/cvsroot/gentoo-x86/net-mail/cyrus-imapd/files/imapd.conf,v 1.5 
2004/08/27 06:02:45 langthang Exp $

# Don't forget to use chattr +S (if you are using ext[23])
# when you change these directories (read the docs).
configdirectory:/var/imap
partition-default:  /movies/imap
sievedir:   /var/imap/sieve
virtdomains:yes
#defaultdomain  lang.hm

#tls_ca_path:/etc/ssl/certs
#tls_cert_file: /etc/ssl/cyrus/server.crt
#tls_key_file:  /etc/ssl/cyrus/server.key

# Don't use an everyday user as admin.
admins: cyrus

hashimapspool:  yes
allowanonymouslogin:no
allowplaintext: yes

# Allow renaming of top-level mailboxes.
#allowusermoves: yes

# Use this if sieve-scripts could be in ~user/.sieve.
#sieveusehomedir:   yes

# Use saslauthd if you want to use pam for imap.
# But be warned: login with DIGEST-MD5 or CRAM-MD5
# is not possible using pam.
#sasl_pwcheck_method:   saslauthd


## This is a recommended authentication method if you
## emerge cyrus-sasl with 'postgres' or 'mysql'
## To use with mysql database uncomment those lines below.

sasl_pwcheck_method: auxprop
sasl_auxprop_plugin: sql

## possible values for sasl_auxprop_plugin 'mysql', 'pgsql', 'sqlite'.
sasl_sql_engine: pgsql

## all possible values.
sasl_mech_list: LOGIN PLAIN CRAM-MD5 DIGEST-MD5
## or limit to CRAM-MD5 only
#sasl_mech_list: CRAM-MD5

## change below to suit your setup.
sasl_sql_user: mailuser
sasl_sql_passwd: password
sasl_sql_database: maildb
sasl_sql_hostnames: localhost
sasl_sql_select: SELECT clear FROM users WHERE email = '[EMAIL PROTECTED]'


my sendmail.mc

bifrost:/etc/mail# cat sendmail.mc
define(`_USE_ETC_MAIL_')dnl
include(`/usr/share/sendmail/cf/m4/cf.m4')dnl
VERSIONID(`DI Basebuild 3.1 07-20-05')
OSTYPE(`debian')dnl
DOMAIN(`debian-mta')dnl
dnl # Items controlled by /etc/mail/sendmail.conf - DO NOT TOUCH HERE
undefine(`confHOST_STATUS_DIRECTORY')dnl#DAEMON_HOSTSTATS
dnl # Items controlled by /etc/mail/sendmail.conf - DO NOT TOUCH HERE
FEATURE(`virtusertable',`hash /etc/mail/virtusertable')
VIRTUSER_DOMAIN_FILE(`/etc/mail/virtdomaintable')
FEATURE(`mailertable',`hash /etc/mail/mailertable')
FEATURE(`use_cw_file')
FEATURE(`preserve_local_plus_detail')
FEATURE(always_add_domain)
FEATURE(nouucp,`reject')
define(`confLOCAL_MAILER',`cyrusv2')
define(`CYRUSV2_MAILER_ARGS',`TCP asgard lmtp')
dnl MAILER(`smtp')
MAILER(`cyrusv2')
MAILER(`smtp')
MAILER_DEFINITIONS
Mlmtp,  P=[IPC], F=lsDFMnqA@/:|SmXz, E=\r\n,
 S=EnvFromL, R=EnvToL/H

Re: Global mailbox

[David Lang]

On Mon, 30 Jul 2007, Boris Andratzek wrote:

> Lorenzo Milesi wrote:
>> Michael Menge ha scritto:
>>> Hi,
>>>
>>> as far as i know the seen status is managed for each user, but the
>>> other flags  such as deleted, spam, or other custom flags are managed
>>> per email and are GLOBAL.
>>
>> Hi
>> Thanks for your reply.
>> Isn't it possible to make also the seen flag global?
>>
>
> Hej Lorenzo,
>
>
> I started a thread with quite the same in here ('mark shared folder's
> mails as read') and in debian.user.de and had to consider that this is
> almost impossible. I also thought about server-side filtering with sieve
> and this might be an option for you but it'll be a lot of work

what you could do as a work-around is make a subfolder 'read' and configure 
your 
clients to move messages that have been read into that folder.

David Lang

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Cyrus with a NFS storage. random DBERROR

[David Lang]

On Sat, 9 Jun 2007, Rob Mueller wrote:

> So - added is a new option "uuidmode" in imapd.conf. Set it to md5 and you
> will get UUIDs of the form: 02(first 11 bytes of the MD5 value for the
> message) which takes up the same space, but allows pretty good integrity
> checking.
>
> Is it safe? - we calulated that with one billion messages you have a one in
> 1 billion chance of a birthday collision (two random messages with the same
> UUID). They then have to get in the same MAILBOXES collection to sync_client
> to affect each other anyway. The namespace available for generated UUIDs is
> much smaller than this, since they have no collision risk - but if you had
> that many delivering you would hit the limits and start getting blank UUIDs
> anyway.

does the IMAP spec specify how large a UUID can be?

David Lang

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: load balancing at fastmail.fm

[David Lang]

On Mon, 12 Feb 2007, urgrue wrote:

If it's using block level replication, how does it offer instant recovery 
on filesystem corruption? Does it track every block written to disk, and 
can thus roll back to effectively what was on disk at a particular instant 
in time, so you then just remount the filesystem and the replay of the 
journal should restore to a good state?
Yes. I may be wrong but to my understanding at least NetApp has this 
capability.


No, NetApp takes snapshots of the filesystems on a schedule (hourly, daily, 
weekly, etc), and you can read files off of those snapshots. you cannot getany 
more granular then that.


David Lang

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Clustering and replication

[David Lang]

On Mon, 29 Jan 2007, Tom Samplonius wrote:


- "Bron Gondwana" <[EMAIL PROTECTED]> wrote:

On Fri, Jan 26, 2007 at 12:20:15PM -0800, Tom Samplonius wrote:



* the system monitoring scripts do a 'du -s' on the sync directory every
  2 minutes and store the value in a database so our status commands can
  see if any store is behind (the trigger for noticing is 10kb, that's a
  couple of minutes worth of log during the U.S. day).  This also emails
  us if it gets above 100kb (approx 20 mins behind)


 And what do you do if it gets behind?  I have three Cyrus groups right now, 
that are never going to catch up.  They log about 20KB in 20 minutes, so the 
update rate is not that high.  The machines are dedicated, and the replicas 
aren't doing anything.  tcpdump confirms that there is traffic to the replica, 
but the entire sync_client is so opaque it is hard to see what it is doing. 
So sync_client can't keep up at all, and since it also quits from time to 
time, it gets even worse.


 I'm planning to hack the log, and add some logging to sync_client, 
particularly to find the number of records per second it is able to process. 
And then maybe someway to find why it quits all the time.


 Either that, or my only alternative is to switch to using DRBD to sync the 
filesystem to a standby server.



* a "monitorsync" process that runs from cron every 10 minutes and reads
  the contents of the sync directory, comparing any log-(\d+) file's PID
  with running processes to ensure it's actually being run and tries to
  sync_client -r -f the file if there isn't one.  It also checks that
  there is a running sync_client -r process (no -f) for the store.


 Wow, a lot of protection to protect against sync_client just exiting. 
sync_client isn't very big, so it shouldn't be that hard to find the different 
places that it exits, and fix them?



* a weekly "checkreplication" script which logs in as each user to both
  the master and replica via IMAP and does a bunch of lists, examines,
  even fetches and compares the output to ensure they are identical.

Between all that, I'm pretty comfortable that replication is correct and
we'll be told if it isn't.  It's certainly helped us find our share
of issues with the replication system!


 Well, I know our replicas are out of sync, so we just don't use them.  I just 
hope the master's don't fail.  Each pair has about 30,000 accounts, and about 
300GB of online mail.


Tom,
  in your situation you may want to seriously look at disabling fsync. doing so 
could let your replicas keep up.


it's definantly not ideal, but if I was forced to choose between

1. single box with fsync and no replica

or

2. master without fsync and replicas without fsync, but up to date

I would choose 2, as it won't loose any data due to a master failing, no matter 
what happens on the master, and I'm only vunerable to something that would take 
down both the primary and it's replica at the same time (don't have them both on 
the same UPS!)


David Lang

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Patches used at FastMail.FM

[David Lang]

On Wed, 10 Jan 2007, Rob Mueller wrote:

but this is in conflict with the the idea that in a large installation of 
people who don't know each other the 'anyone' permission doesn't make 
sense.


what is really desired for + addressing is to say that messages that arrive 
via the lmtp interface are allowed to write to all folders (not just the 
inbox folders) without allowing other users on the system to write 
arbatrary data to other people's folders via the IMAP interface.


at least if it's arriving via the lmtp interface you have reason to believe 
that it's been (somewhat) validated by your MTA.


That's really what the "p" permission is all about:

 p - post (send mail to submission address for mailbox,
 not enforced by IMAP4 itself)

So setting "anyone p" means that email via LMTP can be put into any persons 
folder by the delivery agent, but that folder isn't visible or accessible via 
any IMAP commands.


At least that how I believe it works, and what we've observed. Maybe Ken can 
clarify?


Ok, I thought that 'post' pre-dated lmtp and was the IMAP function to write a 
message into the folder.


i.e. a program like imapsync would need the 'p' permission to write the 
messages, (but would need other permissions to check for messages, set flags, 
etc)


I'll play around with things a bit while waiting for clarification.

David Lang

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Patches used at FastMail.FM

[David Lang]

On Wed, 10 Jan 2007, Rob Mueller wrote:



the usual reason for allowing the "anyone" ACL is to allow for + addressing 
to

work.

is there another way to do this?


The admin user can still set the anyone acl, it's just non-admin users can't 
change/set it. The way we do this to allow + addressing is when we create the 
users top level folder, we set the "anyone p" acl on it, and any new folders 
created after that by the user automatically inherit it.


but this is in conflict with the the idea that in a large installation of people 
who don't know each other the 'anyone' permission doesn't make sense.


what is really desired for + addressing is to say that messages that arrive via 
the lmtp interface are allowed to write to all folders (not just the inbox 
folders) without allowing other users on the system to write arbatrary data to 
other people's folders via the IMAP interface.


at least if it's arriving via the lmtp interface you have reason to believe that 
it's been (somewhat) validated by your MTA.


David Lang

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Patches used at FastMail.FM

[David Lang]

On Tue, 9 Jan 2007, Ken Murchison wrote:



Disable "anyone" ACL



the usual reason for allowing the "anyone" ACL is to allow for + addressing to 
work.


is there another way to do this?

in most cases I think that a global 'allow + addressing' config option is really 
more appropriate then having to configure things on a per-folder basis, possibly 
with a 'no, don't allow + addressing to this folder' override.


David Lang

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: IMAP sync tool (rsync for IMAP)

[David Lang]

On Thu, 21 Dec 2006, Florin Andrei wrote:

I'm currently using two IMAP accounts, one on Cyrus-2.2 the other on 
Cyrus-2.3.


The one on Cyrus-2.3 will get decomissioned so I need to transfer all my 
email, preserving the folders/subfolders tree, under a specific folder 
(oldmail/foo/bar) on the 2.2 server.
I need to do the bulk of the transfer sometime soon, then sync up again a few 
times after that, until the day the account on the 2.3 server gets nuked.


Essentially, I need a tool that I can point at servers A and B and tell it 
"get all the email from my account on server A to a specific folder on my 
account on server B, preserving the subfolders hierarchy".
The tool needs to be smart enough to repeat the operation later on but then 
it must only transfer the new messages.
The tool may run on one or the other IMAP servers, or even on a 3rd machine, 
since it should be network-based. Pretty much all systems are Linux 'round 
here, some Windows stragglers too.


Sort of like rsync for IMAP, if that makes sense.

So far, the only tool I've found is imapsync:

http://freshmeat.net/projects/imapsync/

Anyone tried it with Cyrus? Good/bad experiences?


I just used it to move from 2.1 to 2.3, there were a handful of messages it 
didn't like (~30 out of a few hundred thousand messages) but it appears to have 
worked well enough to fix the last few messages manually.


most of the errors were cases of invalid headers that 2.3 wouldn't accept, but 
2.1 obviously did.


David Lang


Are there any other tools that work better with Cyrus?




Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Anyone with experience using imapsync

[David Lang]

--On Monday, August 21, 2006 10:29:26 PM -0700 Rob Tanner 
<[EMAIL PROTECTED]> wrote:



David,

I tried various combinations based on your suggestions.  The --prefix2
"INBOX."  was the most interesting.  Instead of userxyz, I ended up with
INBOXxyz.  Just a variation on the same problem.
One thing I did forget to mention.  The saslauthd program is pointing to
the shadow file at the moment, and none of the folks (ultimately I need
to move about 4,500 accounts) are in the shadow file.  I have to
eventually switch the pointer to LDAP but that requires some other prep
that I'm not yet ready for.  But could that possibly explain my problem?


I think that the fact that it's dropping the . when it is trying to access 
the destination folder is the key.


when you add the --prefix INBOX. you may have to quote it to have it keep 
the . in the name.


when I get home tonight I'll check the command line I've been useing at 
home.


David Lang


-- Rob

On 08/21/2006 05:21 PM, David Lang wrote:

I'm working with it to copy some things currently, I'm doing it a user
at a time, and found that I needed to set --prefix2 "INBOX." (note the
. ) to get things to copy properly. it looks like you may need to set
--sep or --sep2 to "." as well.

also, take a look at imapcopy, if you are just doing a one-way move it
may be easier to setup (imapsync can keep the two in sync while you
are testing)

David Lang

On Mon, 21 Aug 2006, Rob Tanner wrote:


Hi,

I'm trying to migrate mail from one IMAP server to another using the
perl program imapsync.  Both the source and destination servers are
Cyrus IMAP4 v2.2.3 servers.  I have added a second partition to the
destination server and made it the default by configuring imapd.conf
as follows:

partition-default: /var/spool/imap
partition-belgarath: /var/spool/belgarath
defaultpartition: belgarath

With this setup, I can use cyradm and by hand correctly adds users to
the new partition.  When I use imapsync to copy users over.  Instead
of folders such as user.xyz and user.xyx.sent and user.xyz.drafts, I
get userxyz and userxyzsent and userxyzdrafts -- all as separate and
entirely independent folders and not even a hierarchy.

Here's a script I used just to test and move over one user:


# ! /bin/bash

./imapsync \
  --host1 belgarath.linfield.edu --user1 cyrus --passfile1
/home/rtanner/imapsync/cyrus.pwd \
  --host2 polgara.linfield.edu --user2 cyrus --passfile2
/home/rtanner/imapsync/cyrus.pwd \
  --syncinternaldates \
  --subscribe \
  --include "^user\.aabryan.*$"


Any idea what I'm doing wrong?

Thanks,
Rob











Cyrus Home Page: http://asg.web.cmu.edu/cyrus
Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Cyrus WebMail

[David Lang]

On Wed, 30 Nov 2005, Andrzej Kwiatkowski wrote:


2005/11/30, Bill Kearney < [EMAIL PROTECTED]
<mailto:[EMAIL PROTECTED]> >:


SquirrelMail!!


Or it you want less bloat, try Ilohamail.


I was wondering if someone tried to write some
webmail application for Cyrus IMAPD.

Using IMAP connection for Webmail are not this,
what i want for today.


Why not?


Is there some documentation about Cyrus internal databases,
or maybe API about this stuff ?


If you use an IMAP webmail client it'll work with nearly all IMAP
servers.


Hello.
Thanks for your replies.
For now i'm only testing varios possibilities.

Now i'm using squirellmail in my architecture, but as for me it have
some
limitations.

For now i've got any IMAP server and it's ok, SM works great .
In future i will have four or more backend Cyrus Servers.

One possible is to have one webmail instance dedicated for
one cyrus server, second to use vlogin plugin.

I need third possibility: before login, select in sql database to get
IMAP
server corresponding to login.

Vlogin has this feature, but it need special table for preferences.
I've got table with colum mailhost (used by postfix too) and i want to
use it.

Have you got any idea how to do this ?
Thanks
AK


Cyrus makes this really easy, when you migrate to multiple IMAP servers 
implment cyrus murder, this makes your cluster of IMAP servers look like a 
single server to the outside world, so everything (including SM) works 
without modification with no need to do extra lookups.


this also means that cyrus continues to be a black box to the outside 
world and you can move users from server to server without having to 
reconfigure anything (it's just a simple command on the murder server(s))


David Lang

--
There are two ways of constructing a software design. One way is to make it so 
simple that there are obviously no deficiencies. And the other way is to make 
it so complicated that there are no obvious deficiencies.
 -- C.A.R. Hoare


Cyrus Home Page: http://asg.web.cmu.edu/cyrus
Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Cyrus Home Page: http://asg.web.cmu.edu/cyrus
Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: improving concurrency/performance

[David Lang]

On Mon, 7 Nov 2005, Sergio Devojno Bruder wrote:


David Lang wrote:

(..)
I was recently doing some testing of lots of small files on the various 
filesystems, and I ran into a huge difference (8x) depending on what 
allocator was used for ext*. the default allocator changed between ext2 and 
ext3 (you can override it as a mount option) and when reading 1M files (10 
dirs of 10 dirs of 10 dirs of 1000 1K files) the time to read them went 
from ~5 min with the old allocator useed in ext2 to 40 min for the one 
that's the default for ext3.


David Lang

(!!) Interesting. You said mount options? man mount man page only show me 
data=journal, data=ordered, data=writeback, etcetera.


How can I change that?


I found more things listed under /usr/src/linux/Documentation/filesystems

there are ext2.txt and ext3.txt files that list all the options available.

note that with my test all the files were created in order, it may be that 
if the files are created in a random order things would be different, so 
further testing is warrented.


I was doing tests to find how long it took to tar/untar these files  (with 
the tarball on a different drive).


here are the notes I made at the time. this was either 2.6.8.1 or 2.6.13.4 
(I upgraded about that time, but I'm not sure what the exact timeing was


note that on my cyrus server I actually use XFS with very large folders 
(20,000 mails in one folder) and it seems lightning fast. I haven't 
reconciled that observed bahavior with the tests listed below


the fact that on ext* filesystems I had tests range from 5 min to 80 min 
is somewhat scary. I did make sure to clear memory (by reading a file 
larger then available ram and doing a sync) between tests


David Lang



on ext2 reading the tarball takes 53 seconds, createing the tar takes 10m, 
untarring it takes 4 min, copying it between drives on different 
controllers takes 62 seconds.


XFS looks bad for small files (13 min to untar, 9:41 to tar), but good for 
large files (47 sec to read)


reiserfs: reading the tar 43 sec 4:50 to tar 2:06 to untar (it was 
designed for tiny files and it appears to do that well)


  a couple tests I ran on reiserfs that I hadn't thought to run on the 
others, untaring on top of an existing directory took 7m, ls -lR took 
2:40, ls -flR (unsorted) took 2:40, find . -print took 21sec, rm -r took 
3m


jfs: 57 sec to read, untar 15:30, no other tests run

ext3: untar 3:30, read 64sec, tar 5:46, untarring on top of an existing 
directory 5:20, ls -lR 53 sec, ls -flR 47 sec, find . -print 7 sec


enabling dir_hash changed the read (36 sec) ls -flr (57 sec), ls -lR 61 
sec, find (25 sec), tar 81m!!!


turning off dir_hash and removing the journal (effectivly turning it into 
ext2 again) and mounting noatime

the tar goes to  34 min

mounting with oldalloc,noatime untar is 4:45, tar is 5:51.


--
There are two ways of constructing a software design. One way is to make it so 
simple that there are obviously no deficiencies. And the other way is to make 
it so complicated that there are no obvious deficiencies.
 -- C.A.R. Hoare

Cyrus Home Page: http://asg.web.cmu.edu/cyrus
Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: improving concurrency/performance

[David Lang]

On Mon, 7 Nov 2005, Jure Pe?ar wrote:


On Sun, 6 Nov 2005 14:20:03 -0800 (PST)
Andrew Morgan <[EMAIL PROTECTED]> wrote:


mkfs -t ext3 -j -m 1 -O dir_index /dev/sdb1
tune2fs -c 0 -i 0 /dev/sdb1


What about 1k blocks? I think they'd be more useful than 4k on mail
spools ...


I was recently doing some testing of lots of small files on the various 
filesystems, and I ran into a huge difference (8x) depending on what 
allocator was used for ext*. the default allocator changed between ext2 
and ext3 (you can override it as a mount option) and when reading 1M files 
(10 dirs of 10 dirs of 10 dirs of 1000 1K files) the time to read them 
went from ~5 min with the old allocator useed in ext2 to 40 min for the 
one that's the default for ext3.


David Lang

--
There are two ways of constructing a software design. One way is to make it so 
simple that there are obviously no deficiencies. And the other way is to make 
it so complicated that there are no obvious deficiencies.
 -- C.A.R. Hoare

Cyrus Home Page: http://asg.web.cmu.edu/cyrus
Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: [POLL] pop3d/nntpd and IMAP flags

[David Lang]

On Tue, 25 Oct 2005, Henrique de Moraes Holschuh wrote:


On Tue, 25 Oct 2005, Ken Murchison wrote:

It would be fairly straightforward to have an option that updated \Seen
state whenever a POP3 client issues a RETR command or an NNTP client
issues a BODY or ARTICLE command.


I vote for this change, and if it is optional, I'd vote for it to be the
default behaviour.


I obviously haven't used pop and imap on the same folder much, this is 
actually the behavior I would have expected (it's the same message, 
however you read it, the fact that you have read it means it's seen.)



My question is what do people think of the interaction between
pop3d/nntpd and the \Deleted flag?  Should these daemons ignore articles
that have this flag set?  Should a POP3 DELE command or a NNTP cancel
message just set the \Deleted flag instead of expunging the message?


I vote for two possible behaviours, selectable via imapd.conf:
 1. pop3/nntp DELE/cancel sets \Delete flag. pop3 QUIT causes expunge
 2. what we have now (this would be the default).


#1 sounds like the proper flag, there's not much that's more annoying then 
having fetchmail crach in the middle of a run and result in loosing 
messages as a result. and expunging after each message can kill your 
server (in fact, if there was a lazy expunge this would be the perfect 
situation to use it)



Should any setting which enables pop3d/nntpd to use IMAP flags be global
or per-mailbox?


I'd be happy enough with it being global.  If it is made per-mailbox, IMHO
it would be good to have it work as follows: a "global" option that applies
to every mailbox, and a per-mailbox annotation that overrides the global
option for this mailbox subtree.


I agree with those who say it should be both, with the specific overriding 
the general.


David Lang

--
There are two ways of constructing a software design. One way is to make it so 
simple that there are obviously no deficiencies. And the other way is to make 
it so complicated that there are no obvious deficiencies.
 -- C.A.R. Hoare

Cyrus Home Page: http://asg.web.cmu.edu/cyrus
Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Imap timeout with 27k messages...

[David Lang]

the problem is reated to the speed of accessing evrything on the 
filesystem. there's no easy fix for this (other then possibly extending 
the timeouts)


to 'fix' this you could create a new folder and manually copy a bunch of 
the messages to that new folder then run reconstruct on both the old and 
new folders.


beyond that do some testing with huge message folders on different 
filesystems. you may find that other filesystems handle huge folders 
better then what you're useing.


David Lang

 On Mon, 8 Aug 2005, Jared Watkins wrote:


Date: Mon, 08 Aug 2005 13:26:31 -0400
From: Jared Watkins <[EMAIL PROTECTED]>
Reply-To: Jared Watkins <[EMAIL PROTECTED]>
To: info-cyrus@lists.andrew.cmu.edu
Subject: Imap timeout with 27k messages...

Hello all..

I have a situation here where an 'exempt' user has accumulated nearly 27k 
messages and 1.5G of mail in their sent items folder and now any attempt to 
access this folder has imap timeout problems and stuck processes.  The cyradm 
utility is also not able to work with the folder... attempts to rename.. or 
reconstruct it results in a stuck process.  This under RHEL3 with reiserfs 
and cyrus 2.2.3.. I've reviewed the changelog and I don't see anything 
obviously related to this up to 2.2.12.


Any ideas on what might be causing this... or what I can do to fix it... 
short of deleting the folder?


Thanks,
Jared
---
Cyrus Home Page: http://asg.web.cmu.edu/cyrus
Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html



--
There are two ways of constructing a software design. One way is to make it so 
simple that there are obviously no deficiencies. And the other way is to make 
it so complicated that there are no obvious deficiencies.
 -- C.A.R. Hoare
---
Cyrus Home Page: http://asg.web.cmu.edu/cyrus
Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

RE: Database backend?

[David Lang]

I would be interested in this, thanks.
David Lang
On Wed, 11 May 2005, ¿øÅÂȯ wrote:
Date: Wed, 11 May 2005 23:29:55 +0900
From: ¿øÅÂȯ <[EMAIL PROTECTED]>
To: Markus Heller <[EMAIL PROTECTED]>
Cc: info-cyrus@lists.andrew.cmu.edu
Subject: RE: Database backend?
Hi,
I had an experience to implement mysql backend to store mailbox paths.
The mysql backend was intended for special purpose.
But it would help you to start to make your DBMS backend implementation.
Let me know if you want it. ;-)
Tawan Won
 _
From: [EMAIL PROTECTED] ÀÌ(°¡) ´ÙÀ½ »ç¶÷ ´ë½Å º¸³¿
Markus Heller
Sent: 2005-05-11 (¼ö) ¿ÀÈÄ 8:01
To: info-cyrus@lists.andrew.cmu.edu
Subject: Database backend?

Dear List,
has anybody tried to store the contents of the imap repository in a
database
like postgres? Is there an interface for postgres or mysql?
Thanks in advance,
Markus
---
Cyrus Home Page: http://asg.web.cmu.edu/cyrus
<http://asg.web.cmu.edu/cyrus>
Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu
<http://cyruswiki.andrew.cmu.edu>
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
<http://asg.web.cmu.edu/cyrus/mailing-list.html>

--
There are two ways of constructing a software design. One way is to make it so 
simple that there are obviously no deficiencies. And the other way is to make 
it so complicated that there are no obvious deficiencies.
 -- C.A.R. Hoare
---
Cyrus Home Page: http://asg.web.cmu.edu/cyrus
Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Large email account

[David Lang]

On Mon, 20 Dec 2004, Prasanna Buddhika wrote:
Date: Mon, 20 Dec 2004 16:14:17 -0800 (PST)
From: Prasanna Buddhika <[EMAIL PROTECTED]>
To: info-cyrus@lists.andrew.cmu.edu
Subject: Large email account
Hi,
I have win2000 server running exchange 2000 server.
There are 2 large email accounts (70 emails in a
single account) which we don't want to delete. But I
want to move these 2 accounts to Linux machine with
cyrus-imap server. Migration went very well. Only
problem is the speed. When the 2 mail accounts on the
exchange server and when I use exchange method to
access the account via Microsoft outlook client it
won't take even 1 minute to show all the massages. But
if I use cyrus-imap to retriew (sync) the email it
takes 30 minutes. But after sync it's fast. Only
problem is when ever I logged off the system and log
back in to the system again I have to wait 30 minutes
to access the mail. Can anyone have a good suggestion
to speedup this apart from deleting emails.:)
Thank you,
Prasan
What filesystem did you use on the linux box?
the ext2/ext3 filesystems slow down significantly when you have large 
files or directories with large numbers of files in them, and you are 
definantly far beyone the point where they will slow down (around 10,000 
messages they start to slow down and ~15,000-20,000 they become very bad)

if you want to use ext2/ext3 you will need to reasearch and implement the 
htree filesystem extention and see how well it helps you. Otherwise try 
converting to XFS or Reiserfs (my personal prefrence is XFS, but test for 
yourself)

one way to tell that it's a server-side problem is to login to the server, 
cd to the directory for the mailbox and do a ls, you will be amazed at how 
long it will take to respond.

once you get the server working well then the next step is to get a better 
client then outlook as the other posts have pointed out, but if you were 
to switch to another client without fixing the server you would find that 
every action you take is very slow (now moving from outlook you may not 
notice quite how slow, as outlook is one of the slower clients to start 
with)

David Lang
 -- 
There are two ways of constructing a software design. One way is to make it so simple that there are obviously no deficiencies. And the other way is to make it so complicated that there are no obvious deficiencies.
 -- C.A.R. Hoare
---
Cyrus Home Page: http://asg.web.cmu.edu/cyrus
Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Once again: action hooks

[David Lang]

Take a look at Popfile (on sourceforge) it will train when you move a 
message into a folder and retrain when you move a message to another 
folder.

the current version is single user (you would need to run one copy per 
user), but the development version is adding multi-user capability)

David Lang
 On Tue, 25 Jan 2005, Denis Jedig wrote:
Date: Tue, 25 Jan 2005 12:19:27 +0100
From: Denis Jedig <[EMAIL PROTECTED]>
To: info-cyrus@lists.andrew.cmu.edu
Subject: Once again: action hooks
Greetings, list.
I already crawled the web, checked the docs and posted to the local 
newsgroups, but I am still lacking a solution for my problem:

I want to train bayesian filters via putting messages into and removing them 
out of IMAP mailboxes [*]
I considered several ways to achieve this:
a) start the learning process for everything in an IMAP mailbox periodically 
(cron)
b) use notifications for starting the learning process

However, the problem for both approaches is:
there is no clean way to "unlearn" a message if it is moved out (not deleted) 
of a mailbox.

So I would like to know, if there is some way to implement notifications on 
message move actions.

Thank you in advance.
[1] this seems the most convinient way to do so for the users. Forwarding 
messages to a specific e-mail address does not work really well - we would 
need two e-mail addresses per user, per bayes bucket, one for "learn", one 
for "unlearn" and the users do not understand this concept.

Denis Jedig
---
Cyrus Home Page: http://asg.web.cmu.edu/cyrus
Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
--
There are two ways of constructing a software design. One way is to make it so 
simple that there are obviously no deficiencies. And the other way is to make 
it so complicated that there are no obvious deficiencies.
 -- C.A.R. Hoare
---
Cyrus Home Page: http://asg.web.cmu.edu/cyrus
Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Can I use Cyrus IMAP w/Outlook?

[David Lang]

Paul, I recently took a look at useing thunderbird 1.0 with IMAP and found 
that it was storing a lot of info locally, is it really that good an IMAP 
client?

David Lang
 On Wed, 12 Jan 2005, Paul Dekkers wrote:
Date: Wed, 12 Jan 2005 10:17:10 +0100
From: Paul Dekkers <[EMAIL PROTECTED]>
To: Thomas Kessler <[EMAIL PROTECTED]>
Cc: info-cyrus@lists.andrew.cmu.edu
Subject: Re: Can I use Cyrus IMAP w/Outlook?
Thomas Kessler wrote:
Can I use Cyrus IMAP w/Outlook? Is this even possible? I setup Cyrus 
assuming that I could, but I haven’t been able to find any saying that is 
works, just that Insight Server application that works with Cyrus IMAP. If 
I can’t use Cyrus with Outlook, can someone recommend an IMAP server that 
will?

Of course you can use it. It's not the best IMAP client in the world (I would 
rather use (and support) Mozilla/Thunderbird ;-) or, ok, maybe even rather 
Outlook Express then Outlook because of its IMAP support), but it surely 
works. Just set up an IMAP account with the correct account info & 
credentials.

Paul
P.S. If authentication fails against the Cyrus server you should probably 
tune your sasl_mech_list to something like "PLAIN LOGIN" depending on your 
backend ofcourse.


---
Cyrus Home Page: http://asg.web.cmu.edu/cyrus
Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
--
There are two ways of constructing a software design. One way is to make it so 
simple that there are obviously no deficiencies. And the other way is to make 
it so complicated that there are no obvious deficiencies.
 -- C.A.R. Hoare
---
Cyrus Home Page: http://asg.web.cmu.edu/cyrus
Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: best filesystem for imap server

[David Lang]

On Thu, 2 Dec 2004, John Madden wrote:
Date: Thu, 2 Dec 2004 14:53:07 -0500 (EST)
From: John Madden <[EMAIL PROTECTED]>
To: [EMAIL PROTECTED]
Cc: [EMAIL PROTECTED]
Subject: Re: best filesystem for imap server
I think they use capacitors that will hold enough charge to allow
flushing the buffers to disk when there's a power loss.
And another set of caps to keep the spindles spinning so that data can be
written?  I'm not yet willing to buy the bridge you're selling. :)
10 or so years ago when the drives had significantly more rotating mass 
and significantly lower data density there were (high-end SCSI) drives 
that could use their rotational energy to power their electronics to write 
the data and adjust the dataclock as the spindle slowed, but I don't think 
any drive does this anymore.

David Lang
--
There are two ways of constructing a software design. One way is to make it so 
simple that there are obviously no deficiencies. And the other way is to make 
it so complicated that there are no obvious deficiencies.
 -- C.A.R. Hoare
---
Cyrus Home Page: http://asg.web.cmu.edu/cyrus
Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: best filesystem for imap server

[David Lang]

On Thu, 2 Dec 2004, Jules Agee wrote:
Date: Thu, 02 Dec 2004 10:11:21 -0800
From: Jules Agee <[EMAIL PROTECTED]>
To: [EMAIL PROTECTED]
Subject: Re: best filesystem for imap server
David Lang wrote:
also note that if you are useing IDE drives you have no way of really 
knowing when the data has hit the platter (as opposed to just being in the 
buffer of the drive) as many of the drives will lie to you and tell you 
the write is complete once it hits the buffers.
I think they use capacitors that will hold enough charge to allow flushing 
the buffers to disk when there's a power loss.
they used to, but nowdays when the bugger is 8M (or larger), potentially 
with many seeks they don't have any capacitors large enough to hold that 
much power (disassemble a failed drive sometime and try to find any 
significant capacitors in it)

David Lang
--
Jules Agee
System Administrator
Pacific Coast Feather Co.
[EMAIL PROTECTED]  x284
---
Cyrus Home Page: http://asg.web.cmu.edu/cyrus
Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
--
There are two ways of constructing a software design. One way is to make it so 
simple that there are obviously no deficiencies. And the other way is to make 
it so complicated that there are no obvious deficiencies.
 -- C.A.R. Hoare
---
Cyrus Home Page: http://asg.web.cmu.edu/cyrus
Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

RE: best filesystem for imap server

[David Lang]

On Wed, 1 Dec 2004, Jim Miller wrote:
I feel that XFS is a bad choice since it is not a 'truly' journaled file
system.  If you have a power failure/system crash/lockup, etc., etc. You
could very easily end up with a corrupt file system -- XFS doesn't write out
to the disks immediately (caching unwritten data to memory).  EXT3 is
journaled but very slow.  ReiserFS is a better choice for a journaled file
system and if you can hold off until all the bugs are worked out, Reiser4FS
would be the best choice (IMHO).
note that most journaling filesystems journal the metadata, not the file 
data (and ext3 does this as well by default, but it has a mode to enable 
journaling everything)

and actually ext3 had the option to journal everything becouse in the 
initial implementation the peopel writing the code couldn't seperate the 
two types of data so to simplify things they journaled everything.

the reason that not everything is journaled is a simple performance issue. 
having to write the data to the journal, read it from the journal and 
write it to the final location, then update the journal requires a LOT 
more IO bandwidth then if you just do this for the metadata.

personally I have trouble trusting reiserfs ever since it was revealed 
that one reason that it was doing so well on benchmarks is that it delaye 
up to 30 seconds before writing anything to disk so in many cases the 
benchmark was completed before any disk activity took place. This has been 
changed, but it leaves a bad taste behind.

also note that if you are useing IDE drives you have no way of really 
knowing when the data has hit the platter (as opposed to just being in the 
buffer of the drive) as many of the drives will lie to you and tell you 
the write is complete once it hits the buffers.

David Lang

Jim
---
Cyrus Home Page: http://asg.web.cmu.edu/cyrus
Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
--
There are two ways of constructing a software design. One way is to make it so 
simple that there are obviously no deficiencies. And the other way is to make 
it so complicated that there are no obvious deficiencies.
 -- C.A.R. Hoare
---
Cyrus Home Page: http://asg.web.cmu.edu/cyrus
Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: best filesystem for imap server

[David Lang]

I've done some testing and seen a HUGE speedup when switching from EXT2/3 
to XFS. unfortunantly I haven't had a chance to do the same comparison 
with Reiserfs (I need to, but haven't had time)

I was even able to see a dramatic difference with a single user accessing 
a fairly large mailbox (thousands of messages in the inbox)

David Lang
 On Wed, 1 Dec 2004, John 
Madden wrote:

Date: Wed, 1 Dec 2004 13:12:57 -0500 (EST)
From: John Madden <[EMAIL PROTECTED]>
To: [EMAIL PROTECTED]
Cc: [EMAIL PROTECTED]
Subject: Re: best filesystem for imap server
I dont want to start a religious battle, but could I have some opinions
on filesystems for a 100ish user imap server? I have 2x 250G western
digital disks to use.
I think the performance of those disks (and the RAID you put on them) will
be much more significant that the filesystem you use, considering the size
of your user population.  And given that factor, I'd say that even ext3
won't give you any problems performance-wise.  Still, reiserfs, IMO, would
be preferable for mail files.
John

--
John Madden
UNIX Systems Engineer
Ivy Tech State College
[EMAIL PROTECTED]
---
Cyrus Home Page: http://asg.web.cmu.edu/cyrus
Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
--
There are two ways of constructing a software design. One way is to make it so 
simple that there are obviously no deficiencies. And the other way is to make 
it so complicated that there are no obvious deficiencies.
 -- C.A.R. Hoare
---
Cyrus Home Page: http://asg.web.cmu.edu/cyrus
Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: setrlimit: Unable to set file descriptors limit to -1: Operationnot permitted

[David Lang]

On Mon, 18 Oct 2004, Luca Olivetti wrote:
That's not actually a problem (under linux it's not possible to use -1 and so 
it retries with the maximum value) and it's not the cause of the "signaled to 
death by 11". Among other causes it's possible that cyrus is using one 
version of berkeley db while openldap is using another.
what is the maximum value for file decripters?
David Lang
--
There are two ways of constructing a software design. One way is to make it so simple 
that there are obviously no deficiencies. And the other way is to make it so 
complicated that there are no obvious deficiencies.
 -- C.A.R. Hoare
---
Cyrus Home Page: http://asg.web.cmu.edu/cyrus
Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Funding Cyrus High Availability

[David Lang]

On Mon, 20 Sep 2004, Paul Dekkers wrote:
David Carter wrote:
5. Active/Active
designate one of the boxes as primary and identify all items in the 
datastore that absolutly must not be subject to race conditions between 
the two boxes (message UUID for example). In addition to implementing 
the replication needed for #1 modify all functions that need to update 
these critical pieces of data to update them on the master and let the 
master update the other box.
We may be talking at cross purposes (and its entirely likely that I've
got the wrong end of the stick!), but I consider active-active to be
the case where there is no primary: users can make changes to either
system, and if the two systems lose touch with each other they have
to resolve their differences when contact is reestablished.
I'd go for #5 as well:
Since this is a setup where there is no primary at all, I suppose this is 
quite some different design then the #1-4 solutions. And because of that, I 
would think that it's rather useless to have these steps done in order to get 
#5 right, but I might as well be wrong.
actually I think most of the work nessasary for #1 is also needed for 
#5-6.

for #1 you need to have the ability for a system report all it's changes 
to a daemon and the ability for a system to read in changes and implement 
them. #5 needs the same abilities plus the ability to resolve conflicts.

the HA steps of #2 and #4 don't gain that much, but they can also be done 
external to cyrus so it's not a problem to skip them.

#3 involves changes to the update code to have cyrus take special actions 
with soem types of updates. there would need to be changes in the same 
area for #5, but they would be different.

David Lang
--
There are two ways of constructing a software design. One way is to make it so simple 
that there are obviously no deficiencies. And the other way is to make it so 
complicated that there are no obvious deficiencies.
 -- C.A.R. Hoare
---
Cyrus Home Page: http://asg.web.cmu.edu/cyrus
Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Funding Cyrus High Availability

[David Lang]

On Mon, 20 Sep 2004, David Carter wrote:
On Sun, 19 Sep 2004, David Lang wrote:
assiming that the simplest method would cost ~$3000 to code I would make a 
wild guess that the ballpark figures would be

1. active/passive without automatic failover $3k
2. active/passive with automatic failover (limited to two nodes or withing 
a murder cluster) $4k

3. active/passive with updates pushed to the master $5k
4. #3 with auto failover (failover not limited to two nodes or a single 
murder cluster) $7k

5. active/active (limited to a single geographic location) $10k
6. active/active/active (no limits) $30k
in addition to automaticly re-merge things after a split-brin has happened 
would probably be another $5k
I think that you are missing a zero (or at least a fairly substantial 
multipler!) from 5. 1 -> 4 can be done without substantial changes to the 
Cyrus core code, and Ken would be able to use my code as a reference 
implementation, even if he wanted to recode everything from scratch. 5 and 6 
would require a much more substantial redesign and I suspect quite a lot of 
trial and error as this is unexplored territory for IMAP servers.
Thanks, this is exactly the type of feedback that I was hopeing to get. so 
you are saying that #5 is more like $50k-100k and #6 goes up from there

Ok folks, how much are you really willing to pay for this and since the 
amount of work involved translates fairly directly into both cost and time 
how long are you willing to go with nothing?

David Lang
--
David Carter Email: [EMAIL PROTECTED]
University Computing Service,Phone: (01223) 334502
New Museums Site, Pembroke Street,   Fax:   (01223) 334679
Cambridge UK. CB2 3QH.
---
Cyrus Home Page: http://asg.web.cmu.edu/cyrus
Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
--
There are two ways of constructing a software design. One way is to make it so simple 
that there are obviously no deficiencies. And the other way is to make it so 
complicated that there are no obvious deficiencies.
 -- C.A.R. Hoare
---
Cyrus Home Page: http://asg.web.cmu.edu/cyrus
Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Funding Cyrus High Availability

[David Lang]

please don't misunderstand my posts. it's not that I don't think that 
active/active/active is possible, it's just that I think it's far more 
complicated.

assiming that the simplest method would cost ~$3000 to code I would make a 
wild guess that the ballpark figures would be

1. active/passive without automatic failover $3k
2. active/passive with automatic failover (limited to two nodes or withing 
a murder cluster) $4k

3. active/passive with updates pushed to the master $5k
4. #3 with auto failover (failover not limited to two nodes or a single 
murder cluster) $7k

5. active/active (limited to a single geographic location) $10k
6. active/active/active (no limits) $30k
in addition to automaticly re-merge things after a split-brin has happened 
would probably be another $5k

now this doesn't mean that all ofs must be done in this funded project. I 
believe that people would end up going from #1 or #3 to #2 or #4 by 
individuals coding the required pieces and sharing them (#4 has as much of 
a jump over #3 becouse of all the different senrios that are involved, 
each one is individually simple) however #5 and #6 are significantly more 
difficult and I would not expent them to just happen (they are also much 
more intrusinve to the code so there is some possibility of them not 
getting merged into the core code quickly)

David Lang
-- There are two ways of constructing a software design. One way is to 
make it so simple that there are obviously no deficiencies. And the other 
way is to make it so complicated that there are no obvious deficiencies.
 -- C.A.R. Hoare

P.S. #1-4 all could qualify as the first way, #5 and #6 are both 
complicated enough to start with that it is really hard to keep them out 
of the second way
---
Cyrus Home Page: http://asg.web.cmu.edu/cyrus
Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Funding Cyrus High Availability

[David Lang]

On Sun, 19 Sep 2004, David Carter wrote:
On Sun, 19 Sep 2004, David Lang wrote:
5. Active/Active
designate one of the boxes as primary and identify all items in the 
datastore that absolutly must not be subject to race conditions between 
the two boxes (message UUID for example). In addition to implementing the 
replication needed for #1 modify all functions that need to update these 
critical pieces of data to update them on the master and let the master 
update the other box.
We may be talking at cross purposes (and its entirely likely that I've
got the wrong end of the stick!), but I consider active-active to be
the case where there is no primary: users can make changes to either
system, and if the two systems lose touch with each other they have
to resolve their differences when contact is reestablished.
UUIDs aren't a problem (each machine in a cluster owns its own fraction of 
the address space). Message UIDs are a big problem. I guess in the case of 
conflict, you could bump the UIDvalidity value on a mailbox and reassign UIDs 
for all the messages, using timestamps determine the eventual ordering of 
messages. Now that I think about it, maybe that's not a totally absurd idea. 
It would involve a lot of work though.
the problem is that when they are both up you have to have one of them 
allocate the message UID's or you have to change the UIDVALIDITY for every 
new message that arrives.

here is the problem.
  you have a new message created on both servers at the same time. how do 
you allocate the UID without any possibility of stepping on each other?

the only way to do this is to have some sort of locking so that only one 
machine at a time can allocate UID's. you can shuffle this responsibility 
back and forth between machines, but there's a significant amount of 
overhead in doing this so the useual answer is just to have one machine 
issue the numbers and the other ask the first for a number when it needs 
it.

changing UIDVALIDITY while recovering from  a split-brain is probably 
going to be needed.

but as you say it's a lot of work (which is why I'm advocating the simpler 
options get released first :-)

 Pro:
  best use of available hardware as the load is split almost evenly 
between the boxes.

best availability becouse if there is a failure half of the clients won't 
see it at all
Actually this is what I do right now by having two live mailstores. Half the 
mailboxes on each system are active, the remainder are passive.
right, but what this would allow is sharing the load on individual 
mailboxes

useually this won't matter, but I could see it for shared mailboxes
David Lang
--
David Carter Email: [EMAIL PROTECTED]
University Computing Service,Phone: (01223) 334502
New Museums Site, Pembroke Street,   Fax:   (01223) 334679
Cambridge UK. CB2 3QH.
---
Cyrus Home Page: http://asg.web.cmu.edu/cyrus
Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
--
There are two ways of constructing a software design. One way is to make it so simple 
that there are obviously no deficiencies. And the other way is to make it so 
complicated that there are no obvious deficiencies.
 -- C.A.R. Hoare
---
Cyrus Home Page: http://asg.web.cmu.edu/cyrus
Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Funding Cyrus High Availability

[David Lang]

 software
  Con:
   increased complexity
   runs the risk of breaking some deployment senerios in an attempt to 
simplify others

5. Active/Active
  designate one of the boxes as primary and identify all items in the 
datastore that absolutly must not be subject to race conditions between 
the two boxes (message UUID for example). In addition to implementing the 
replication needed for #1 modify all functions that need to update these 
critical pieces of data to update them on the master and let the master 
update the other box.

  Pro:
   best use of available hardware as the load is split almost evenly 
between the boxes.

   best availability becouse if there is a failure half of the clients 
won't see it at all

  Con:
   significantly more complex then the other options.
   behavior during a failure is less obvious
   split-brain recovery is not straightforward and if automatic failover 
is active the sysadmin will have no option to have things degraded 
slightly while a problem is fixed

   depending on the implementation this may be very sensitive to network 
latency between the machines and could be very suitable for working with 
machines in the same datacenter, but worthless for machines thousands of 
miles apart.

6. active/active/active/...
  Take #5 and extend the idea to more then a pair of boxes. this makes the 
updates more complex to propogate (they now need to be sent to every other 
machine in the cluster)

  Pro:
   better load balancing then #5
   allows for the ability to have a HA pair in a primary location and a 
backup in a remote location (i.e. your main HQ has two boxes, but your 
disaster recovery center has one as well)

  Con:
   the complexity goes up significantly when you shift from 2 to n boxes 
in a cluster.

   the bandwidth required for updates increases by a factor of roughly n!
   significantly more split-brain senerios become possible and need to be 
accounted for.


-
while #6 is the ideal option to have it can get very complex
personally I would like to see #1 (with a sample daemon or two to provide 
basic functionality and leave the doors open for more creative uses) 
followed by #3 while people try and figure out all the problems with #5 
and #6

there are a lot of senerios that are possible with #1 or #3 that are not 
possible with #5 and very little of the work needed to release #1 and #3 
as supported options is not work that needs to be done towards #5/6 anyway 
(the pieces need to be identified in the code and hooks put in place in 
the code at those locations. the details of the hooks will differ slightly

David Lang
 -- 
There are two ways of constructing a software design. One way is to make it so simple that there are obviously no deficiencies. And the other way is to make it so complicated that there are no obvious deficiencies.
 -- C.A.R. Hoare
---
Cyrus Home Page: http://asg.web.cmu.edu/cyrus
Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

RE: Funding Cyrus High Availability

[David Lang]

On Fri, 17 Sep 2004 [EMAIL PROTECTED] wrote:
From: David Lang [mailto:[EMAIL PROTECTED]

Mike, one of the problems with this is that different databases have
different interfaces and capabilities.
if you design it to work on Oracle then if you try to make it work on
MySQL there are going to be quite a few things you need to change.
--snip
another issue in all this is the maintainance of the resulting code. If
this code can be used in many different situations then more people will
use it (probably including CMU) and it will be maintained as a
side effect
of any other changes. however if it's tailored towards a very narrow
situation then only the people who have that particular problem will use
it and it's likly to have issues with new changes.
I'd actually figured something like ODBC would be used, with prepared
statements.  /shrug.  Abstract the whole interface issue.
unfortunantly there are a few problems with this
to start with ODBC is not readily available on all platforms.
secondly ODBC can't cover up the fact that different database engines have 
vastly differeing capabilities. if you don't use any of these capabilities 
then you don't run into this pitfall, but if you want to you will.

I really wish that ODBC did live up to it's hype, but in practice only the 
most trivial database users can transparently switch from database to 
database by changing the ODBC config

David Lang
--
There are two ways of constructing a software design. One way is to make it so simple 
that there are obviously no deficiencies. And the other way is to make it so 
complicated that there are no obvious deficiencies.
 -- C.A.R. Hoare
---
Cyrus Home Page: http://asg.web.cmu.edu/cyrus
Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

RE: Funding Cyrus High Availability

[David Lang]

On Fri, 17 Sep 2004 [EMAIL PROTECTED] wrote:
My biggest question here is, simply, why recreate what's already
out there?
There are a number of projects (LVM, PVFS) which do this kind of
replication/distribution/virtulization for filesystems.
There are a number of databases which have active/active clustering
(mysql, DB2, Oracle, et al) and master/slave.
Personally, I would LOVE to see a full RDBMS-backed system.  You
define your database(s) in the config file ... and that is all.
All configuration options are stored on the central RDBMS.  All
mailboxes are stored there.  You can then rely 100% on the RDBMS
systems for clustering/failover/scalability/backing up ... all
datastorage domain problems which they have already addressed/solved.

The other advantages would be very nice integration with other
applications which can query against databases. (ex: postfix directly
supports mysql lookups.)
But then, I can't afford to really help with this myself so take
my thoughts with a big "hope" pill.  =D
Mike, one of the problems with this is that different databases have 
different interfaces and capabilities.

if you design it to work on Oracle then if you try to make it work on 
MySQL there are going to be quite a few things you need to change.

if you start on MySQL and then port to Oracle then you either ignore a 
large chunk of Oracle functionality that you could use or you end up 
having to re-write a bunch of stuff to take advantage of it.

I also would love this option (I would choose postgres as the back-end) 
but this is significantly more complicated then a master->slave 
replication modification to Cyrus.

As such it would cost more to get written and you would have fewer people 
willing to pay for any particular version.

another issue in all this is the maintainance of the resulting code. If 
this code can be used in many different situations then more people will 
use it (probably including CMU) and it will be maintained as a side effect 
of any other changes. however if it's tailored towards a very narrow 
situation then only the people who have that particular problem will use 
it and it's likly to have issues with new changes.

David Lang
--
There are two ways of constructing a software design. One way is to make it so simple 
that there are obviously no deficiencies. And the other way is to make it so 
complicated that there are no obvious deficiencies.
 -- C.A.R. Hoare
---
Cyrus Home Page: http://asg.web.cmu.edu/cyrus
Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Funding Cyrus High Availability

[David Lang]

On Fri, 17 Sep 2004, Ken Murchison wrote:
David Lang wrote:
On Thu, 16 Sep 2004, Ken Murchison wrote:
Question:   Are people looking at this as both redundancy and 
performance, or just redundance?

for performance we already have murder, what we currently lack is 
redundancy. once we have redundancy then the next enhancement is going to 
be to teach murder about it so that it can failover to the backup box(s) 
as needed, but for now simply having the full data at the backup location 
would be so far ahead of where we are now that the need to reconfigure 
murder for a failover is realitivly trivial by comparison.

Actually what I was really asking, is are people looking for an active-active 
config and an active-passive config?

I think that everyone would love to have the active-active option, the 
problem I have with this is that the active-passive config will solve many 
peoples problems and I believe that is will be far simpler to do so I 
don't want the ideal goal of active-active to end up side tracking the 
huge progress that would be achieved by active-passive.

active-active also requires significantly different choices if the nodes 
are seperated by significant distances. I'd hate to end up with an 
active-active solution that works only with the machines all local and 
still have no solution to the disaster recovery senerio.

David Lang
--
There are two ways of constructing a software design. One way is to make it so simple 
that there are obviously no deficiencies. And the other way is to make it so 
complicated that there are no obvious deficiencies.
 -- C.A.R. Hoare
---
Cyrus Home Page: http://asg.web.cmu.edu/cyrus
Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Funding Cyrus High Availability

[David Lang]

On Fri, 17 Sep 2004, Paul Dekkers wrote:
Date: Fri, 17 Sep 2004 08:25:26 +0200
From: Paul Dekkers <[EMAIL PROTECTED]>
To: [EMAIL PROTECTED]
Cc: [EMAIL PROTECTED]
Subject: Re: Funding Cyrus High Availability
Hi,
Eric S. Pulley wrote:
Question:   Are people looking at this as both redundancy and
performance, or just redundance?
Cyrus performs pretty well already. Background redundancy would be 
awesome. Especially if we had control over when the syncing process 
occurred either via time interval or date/time.
I would say not at an interval but as soon as there is an action performed on 
one mailbox, the other one would be pushed to do something. I believe that is 
called rolling replication.

I would not be really happy with a interval synchronisation. It would make it 
harder to use both platforms at the same time, and that is what I want as 
well. So there is a little-bit of load-balancing involved, but more and more 
_availability_.

Being able to use both platforms at the same time maybe implies that there is 
either no master/slave role or that this is auto-elected between the two and 
that this role is floating...
right, but there are already tools freely available on most platforms to 
do the election and changing of the role (by switching between config 
files and restarting the master) what is currently lacking is any ability 
to do the master/slave role. once we have that it's just a little 
scripting to tie just about any failover software in to make it automatic.

one thing we need to watch out for here is that we don't set an 
impossible/unreasonable goal. don't try to solve every problem and add 
every availablity feater you can imagine all at once. instead let's look 
at the building blocks that are needed and identify what's currently not 
available.

currently we have murder which will spread the load across multiple 
machines.

currently we have many tools available to detect a server failure and run 
local scripts to reconfigure machines (HACMP on AIX, hearbeat for Linux, 
*BSD, Solaris, etc)

what we currently do not have is any ability to have one mailstore updated 
to match changes in another one.

once we have that ability there are many things that can be built by 
glueing togeather existing code. once we have a bit of experiance with 
people actually useing these features it will then be obvious which 
features need better integration with Cyrus and which make sense to remain 
seperate.

I also would not be really satisfied with interval synchronisation as the 
only choice.

I think we need something where the primary mailstore pushes a record of 
it's changes to the secondary mailstore

This can then be tweaked in several directions.
1. locking can be added so that the primary doesn't complete it's command 
until the secondary says it has a permanent record of the change 
(two-phase commit or a reasonable facimily of such)

2. batch up the changes until they hit some threshold (size or time or 
combination) and then send a batch of changes all at once

3. recongnise it's own changes to gain the ability to push updates in both 
directions at the same time (true two-phase commit with bi-directional 
replication, some horribile performance pathalogical cases, but attractive 
in some cases)

or other varients
but these all share the same common need
the ability for the master to output all it's changes and the ability for 
a slave to read such changes and update itself to match

the nice thing is that with IMAP much of the data needed is already 
output, you could do a first approximation of this with a client that 
opened a seperate connection to every folder on the primary server and 
just sat watching for server messages and whenever it saw an update send 
the matching command to the slave (fetching the full data as needed to get 
all the info). this obviously won't scale to any reasonalbe size, but this 
means that most of what's needed is already identified so the core could 
be just a common output of the exisitng messages with a little more data 
(mailbox and folder in most cases, message contents in a few)

let's get these small, but critical pieces done and then we can grow and 
experiment from there.

David Lang
Paul
---
Cyrus Home Page: http://asg.web.cmu.edu/cyrus
Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
--
There are two ways of constructing a software design. One way is to make it so simple 
that there are obviously no deficiencies. And the other way is to make it so 
complicated that there are no obvious deficiencies.
 -- C.A.R. Hoare
---
Cyrus Home Page: http://asg.web.cmu.edu/cyrus
Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html