Re: Ghost emails

2007-06-08 Thread Giuseppe Ravasio
Alle 22:11, martedì 5 giugno 2007, Florian Gleixner ha scritto:
 As far as i understand cyrus imapd, it does not delete the message
 instantly. It marks the message and the /usr/lib/cyrus/bin/cyr_expire
 job deletes the message some days later. The delete job is configured in
 /etc/cyrus.conf
   # this is only necessary if using duplicate delivery suppression
   delprune  cmd=cyr_expire -E 3 at=0400
 deletes duplicte messages older than 3 days every day at 4am.

 But i did not yet fully understand the entire process.

In the logs i see thoose lines every times a message is delivered (and i see 
this one, and every other message in my inbox!)...
Jun  8 09:34:44 thot lmtpunix[10927]: accepted connection
Jun  8 09:34:44 thot lmtpunix[10927]: lmtp connection preauth'd as postman
Jun  8 09:34:44 thot lmtpunix[10927]: duplicate_check: 
[EMAIL PROTECTED] user.gravasio0
Jun  8 09:34:44 thot lmtpunix[10927]: mystore: starting txn 2147490592
Jun  8 09:34:44 thot lmtpunix[10927]: mystore: committing txn 2147490592
Jun  8 09:34:44 thot lmtpunix[10927]: duplicate_mark: 
[EMAIL PROTECTED] user.gravasio
1181288084 2913
Jun  8 09:34:44 thot lmtpunix[10927]: mystore: starting txn 2147490593
Jun  8 09:34:44 thot lmtpunix[10927]: mystore: committing txn 2147490593
Jun  8 09:34:44 thot lmtpunix[10927]: duplicate_mark: 
[EMAIL PROTECTED] [EMAIL PROTECTED]   
1181288084 0

I think that the duplicate_mark is only a Debug message meaning that this ID 
is committed to the duplicate check DB.

I'm wrong?
Anyone could explain how duplicate check works? and/or how to read thoose log 
lines?

Beppe

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Cyrus with a NFS storage. random DBERROR

2007-06-08 Thread Dmitriy Kirhlarov
Michael Menge wrote:
 Hi,

 after the problem with the wiki was solved, i added a summery about
 CyrusCluster 
 http://cyrusimap.web.cmu.edu/twiki/bin/view/Cyrus/CyrusCluster .
Could you, please, describe more detailed problems with replication:
CyrusReplication: ... The replication is asynchrony so you might lose 
some mails.

I test this functionality and doesn't find problem.
If sync_client lost connection to sync_server (link down, firewalls 
drops tcp sessions, etc) I just run 'sync_client -u username' for fixing 
problem. It's enough.


WBR.
Dmitriy

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html


Re: Cyrus with a NFS storage. random DBERROR

2007-06-08 Thread Michael Menge

Hi,

i havent used the replication my selfe, so the information is only based on
what i have read on this list.

The sync_client discovers all changes on the mailboxes queues them and send
them to the server. In case of a system crash ther might be changes  
that are still queued and not send to the server.


Quoting Dmitriy Kirhlarov [EMAIL PROTECTED]:


Michael Menge wrote:

Hi,

after the problem with the wiki was solved, i added a summery about
CyrusCluster
http://cyrusimap.web.cmu.edu/twiki/bin/view/Cyrus/CyrusCluster .

Could you, please, describe more detailed problems with replication:
CyrusReplication: ... The replication is asynchrony so you might lose
some mails.

I test this functionality and doesn't find problem.
If sync_client lost connection to sync_server (link down, firewalls
drops tcp sessions, etc) I just run 'sync_client -u username' for fixing
problem. It's enough.


WBR.
Dmitriy

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html






M.Menge Tel.: (49) 7071/29-70316
Universitaet Tuebingen  Fax.: (49) 7071/29-5912
Zentrum fuer Datenverarbeitung  mail:  
[EMAIL PROTECTED]

Waechterstrasse 76
72074 Tuebingen


smime.p7s
Description: S/MIME krytographische Unterschrift

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Cyrus with a NFS storage. random DBERROR

2007-06-08 Thread Dmitriy Kirhlarov
Michael Menge wrote:
 Hi,

 i havent used the replication my selfe, so the information is only 
 based on
 what i have read on this list.

 The sync_client discovers all changes on the mailboxes queues them and 
 send
 them to the server. In case of a system crash ther might be changes 
 that are still queued and not send to the server.
It can be fixed with manualy running 'sync_client -f 
not_finished_logfile' option or 'sync_client -u user', if logfile is lose.

Paul describe more interesting situation.

I think will be good add little more details to twiki for this topic.

WBR.
Dmitriy

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html


Re: Cyrus with a NFS storage. random DBERROR

2007-06-08 Thread Paul Dekkers
Hi,

Dmitriy Kirhlarov wrote:
 Michael Menge wrote:
   
 after the problem with the wiki was solved, i added a summery about
 CyrusCluster 
 http://cyrusimap.web.cmu.edu/twiki/bin/view/Cyrus/CyrusCluster .
 
 Could you, please, describe more detailed problems with replication:
 CyrusReplication: ... The replication is asynchrony so you might lose 
 some mails.

 I test this functionality and doesn't find problem.
 If sync_client lost connection to sync_server (link down, firewalls 
 drops tcp sessions, etc) I just run 'sync_client -u username' for fixing 
 problem. It's enough.
   

... but asynchronous. It is possible that a message is delivered to your
master, not yet (or not properly) synchronized to the backup, and the
master fails (beyond repair). You can replace the master with the backup
server, but you'll lose the unsynchronized messages (or other recent
transactions).
And when folders are not properly synchronized for some reason (that's
why you need to check frequently with for instance make_md5*) you can
lose more than just the most recent messages.

Not necessarily a big problem, but just something to be aware of. It's
not that your transaction is finished after the synchronization is done
(would be slower, but more reliable): the synchronization is done
asynchronously based on a log.

While NFS (more reliable storage) might be an option for us now I
figured out that it works with -o nolock or seperate metadata, I still
think I'd add replication. It's very nice to have synchronization in the
application, in a way that even allows for geographic separation. (And
right now I'd trust it more than shared storage.)

Paul

* I'm still not fully using make_md5 myself. Still need to write a
script, that walks through the files, and only compares the messages
that are in both folders. If I run make_md5, it's never working on a
folder on both servers at the same time, so there's always a gap. So the
files aren't always equal, I think. Oh well.


Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html


Re: Ghost emails

2007-06-08 Thread Scott M. Likens
Looks like one of those is running a sieve script, what is your current
script look like?


On Fri, 8 Jun 2007 09:40:47 +0200
Giuseppe Ravasio [EMAIL PROTECTED] wrote:

 Alle 22:11, martedì 5 giugno 2007, Florian Gleixner ha scritto:
  As far as i understand cyrus imapd, it does not delete the message
  instantly. It marks the message and
  the /usr/lib/cyrus/bin/cyr_expire job deletes the message some days
  later. The delete job is configured in /etc/cyrus.conf
# this is only necessary if using duplicate delivery suppression
delprune  cmd=cyr_expire -E 3 at=0400
  deletes duplicte messages older than 3 days every day at 4am.
 
  But i did not yet fully understand the entire process.
 
 In the logs i see thoose lines every times a message is delivered
 (and i see this one, and every other message in my inbox!)...
 Jun  8 09:34:44 thot lmtpunix[10927]: accepted connection
 Jun  8 09:34:44 thot lmtpunix[10927]: lmtp connection preauth'd as
 postman Jun  8 09:34:44 thot lmtpunix[10927]: duplicate_check: 
 [EMAIL PROTECTED]
 user.gravasio0 Jun  8 09:34:44 thot lmtpunix[10927]: mystore:
 starting txn 2147490592 Jun  8 09:34:44 thot lmtpunix[10927]:
 mystore: committing txn 2147490592 Jun  8 09:34:44 thot
 lmtpunix[10927]: duplicate_mark:
 [EMAIL PROTECTED] user.gravasio
 1181288084 2913 Jun  8 09:34:44 thot lmtpunix[10927]: mystore:
 starting txn 2147490593 Jun  8 09:34:44 thot lmtpunix[10927]:
 mystore: committing txn 2147490593 Jun  8 09:34:44 thot
 lmtpunix[10927]: duplicate_mark:
 [EMAIL PROTECTED] [EMAIL PROTECTED]
 1181288084 0
 
 I think that the duplicate_mark is only a Debug message meaning that
 this ID is committed to the duplicate check DB.
 
 I'm wrong?
 Anyone could explain how duplicate check works? and/or how to read
 thoose log lines?
 
 Beppe
 
 Cyrus Home Page: http://cyrusimap.web.cmu.edu/
 Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
 List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
 
 !DSPAM:46695f5273191822916521!
 
 


-- 
What does one want when one is engaged in the sexual act?
That everything around you give you its utter attention
Think only of you, care only for you...
Every man wants to be a tyrant when he fornicates


Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html


Re: Ghost emails

2007-06-08 Thread Giuseppe Ravasio
Alle 15:57, venerdì 8 giugno 2007, Scott M. Likens ha scritto:
 Looks like one of those is running a sieve script, what is your current
 script look like?

Yes... 
my courrent sieve script is something like:
require 
[fileinto,reject,vacation,imapflags,relational,comparator-i;ascii-numeric,regex,notify];
if
header :contains X-Spam-Flag YES
{
fileinto INBOX/Z_Spam;
stop;
}

the sieve script of the user with gost emails is:

if address :all :comparator i;ascii-casemap :is 
[From, Sender, Resent-From] [[EMAIL PROTECTED], [EMAIL PROTECTED]]  {
discard;
stop;
}

if address :all :comparator i;ascii-casemap :is 
[From, Sender, Resent-From [[EMAIL PROTECTED], [EMAIL PROTECTED]]  {
discard;
stop;
}

Beppe

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html


Re: Ghost emails

2007-06-08 Thread Scott M. Likens
Attached is my current sieve script, you'll differences, modify it for
yours, and see if that helps for you.

Scott

On Fri, 8 Jun 2007 16:09:45 +0200
Giuseppe Ravasio [EMAIL PROTECTED] wrote:

 Alle 15:57, venerdì 8 giugno 2007, Scott M. Likens ha scritto:
  Looks like one of those is running a sieve script, what is your
  current script look like?
 
 Yes... 
 my courrent sieve script is something like:
 require 
 [fileinto,reject,vacation,imapflags,relational,comparator-i;ascii-numeric,regex,notify];
 if
 header :contains X-Spam-Flag YES
 {
 fileinto INBOX/Z_Spam;
 stop;
 }
 
 the sieve script of the user with gost emails is:
 
 if address :all :comparator i;ascii-casemap :is 
 [From, Sender, Resent-From] [[EMAIL PROTECTED],
 [EMAIL PROTECTED]]  { discard;
 stop;
 }
 
 if address :all :comparator i;ascii-casemap :is 
 [From, Sender, Resent-From [[EMAIL PROTECTED], [EMAIL PROTECTED]]
 { discard;
 stop;
 }
 
 Beppe
 
 Cyrus Home Page: http://cyrusimap.web.cmu.edu/
 Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
 List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
 
 !DSPAM:46696a2180703413410412!
 
 




sieve
Description: Binary data

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Cyrus with a NFS storage. random DBERROR

2007-06-08 Thread Paul Dekkers
Dmitriy Kirhlarov wrote:
 Michael Menge wrote:
   
 i havent used the replication my selfe, so the information is only 
 based on
 what i have read on this list.

 The sync_client discovers all changes on the mailboxes queues them and 
 send
 them to the server. In case of a system crash ther might be changes 
 that are still queued and not send to the server.
 
 It can be fixed with manualy running 'sync_client -f 
 not_finished_logfile' option or 'sync_client -u user', if logfile is lose.
   

Which reminds me... isn't it strange that an unfinished logfile is
removed when the cyrus master (or was it the sync_client -r) is
restarted? Would make sense to me if the file is renamed / stored for
later running through sync_client -f. (Or that sync_client -r reads this
file too before it starts rolling.)

Perhaps I'm wrong.

 Paul describe more interesting situation.

 I think will be good add little more details to twiki for this topic.
   

Paul


Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html


Writeup on Cyrus authentication config

2007-06-08 Thread Torsten Schlabach
Hi all!

In order to organize my own thoughts on the subject and as I found that 
the information is not that easy to grasp, I started writing an article 
to try and make it a bit clearer.

I would appreciate if any of the more senior folks here could take the 
time to read

http://cyrusimap.web.cmu.edu/twiki/bin/view/Cyrus/CyrusAuthentication

and comment or correct.

I am especially keen on that last section when it comes to LDAP.

A lot of what I have written is a bit based on guesswork an conclusion 
and it would be nice if someone could confirm or deny.

Regards,
Torsten

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html


Move sasldb2 between two servers

2007-06-08 Thread Martin Schweizer
Hello

I ask this question a long time ago but had no time until now. Here is
a snip of the conversation:

[snip]
...

An entry in sasldb contains 3 or maybe 4 parts. Username, Realm, Password (and
Type: the userPassword). If your users uses only a Username, without
@domain-Part, the Hostname of the Server is used for this key. This is
servername in imapd.conf for Cyrus-Imapd.

Either create entries with:

# saslpasswd -cu domain.tld username

And tell your users to use [EMAIL PROTECTED] as Username. This should work on
both servers, then.

Or if you (have already) create(d) entries with:

# saslpasswd -c username

The Hostname of the Server is used and either your Users use
[EMAIL PROTECTED] as username or you have to change something
between the Source and Destination (Backup) Server. The easiest is to change
the servername (imapd.conf) of the Backup-Server to be equal the other. Then
Users can use username on both servers.

  the Hostname of the Source-Server. Or change sasldb after copying. Or...

 ... what you mean with change sasldb after copying? Do something by hand?

If the databasetype is bdb, you can use db_dump/db_load

# db_dump -p /etc/sasldb2 | sed 's/host1\.domain\.tld/host2.domain.tld/' |
db_load /etc/sasldb2_new

This is only an example, if you really want to use something like that, you
should work at least on the regexp in sed.

Or a little Perl-Script.

[snip]

I have two cyrus mail server with exact the same setup which runs in a
master/slave syncroniation. It works perfect. The problem is I need to
update both sasldb2 files if I get a new user. As you can read above
there are some solutions. In my environment the

# saslpasswd -cu domain.tld username

is the best way because I have only about 30 users. I tested the above
but could not authentication correct. What do I wrong?

Kind regards,

-- 
Martin Schweizer
[EMAIL PROTECTED]
Tel.: +41 32 512 48 54 (VoIP)
Fax: +1 619 3300587

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html


Re: Cyrus with a NFS storage. random DBERROR

2007-06-08 Thread Nik Conwell

On Jun 8, 2007, at 11:36 AM, Paul Dekkers wrote:

 Dmitriy Kirhlarov wrote:

 Which reminds me... isn't it strange that an unfinished logfile is
 removed when the cyrus master (or was it the sync_client -r) is
 restarted? Would make sense to me if the file is renamed / stored for
 later running through sync_client -f. (Or that sync_client -r reads  
 this
 file too before it starts rolling.)

I agree.  I think sync_client should process any pending log-* in the  
sync directory when it's later restarted.  (Or at least have an  
option to do that.)

Do people run sync_client in the SERVICES section rather than START?   
The install-replication docs indicate to put it in START.  If my  
replica goes away for a little while, sync_client exits and then I  
have to restart it manually and then process any pending logs.  Would  
be nice if it just started automatically and picked up where it left  
off.

-nik


Nik Conwell
Boston University


Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html


Re: Cyrus with a NFS storage. random DBERROR

2007-06-08 Thread Dmitriy Kirhlarov
Hi, list.

Nik Conwell wrote:
 Do people run sync_client in the SERVICES section rather than START?   
 The install-replication docs indicate to put it in START.  If my  
 replica goes away for a little while, sync_client exits and then I  
 have to restart it manually and then process any pending logs.  Would  
 be nice if it just started automatically and picked up where it left  
 off.

It doesn't work with ldap ptloader:
http://lists.andrew.cmu.edu/pipermail/cyrus-devel/2007-April/000293.html


WBR.
Dmitriy

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html


Re: Writeup on Cyrus authentication config

2007-06-08 Thread Dmitriy Kirhlarov
Hi, list

Torsten Schlabach wrote:
 http://cyrusimap.web.cmu.edu/twiki/bin/view/Cyrus/CyrusAuthentication
 
 and comment or correct.
 
 I am especially keen on that last section when it comes to LDAP.
 
 A lot of what I have written is a bit based on guesswork an conclusion 
 and it would be nice if someone could confirm or deny.


I'm using only saslauthd authentication. This part looks fine.
With saslauthd also possible build authorization
saslauthd.conf:
...
ldap_group_attr: uniqueMember
ldap_group_dn: cn=imap,ou=mail,o=domain
ldap_group_match_method: attr
...

I'm not sure about topic, but cyrus group ACL's also can be creating 
with ldap-based groups
imapd.conf:
...
ldap_group_base: ou=cyrus,ou=mail,o=domain
ldap_group_filter: (cn=%U)
ldap_group_scope: one
ldap_member_attribute: cn
ldap_member_base: ou=cyrus,ou=mail,o=domain
ldap_member_filter: (uniqueMember=%D)
ldap_member_method: filter
...

cyradm:
lam shared/design
group:boss lrswipktecd
group:info lrswipktecd
anyone p

But user can be membered only one group! If it's not true, ptloader 
can't authenticate user (yes. user cant bind to server) with strange 
diagnose.

WBR.
Dmitriy

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html


Re: Cyrus with a NFS storage. random DBERROR

2007-06-08 Thread Wesley Craig
On 08 Jun 2007, at 06:52, Paul Dekkers wrote:
 * I'm still not fully using make_md5 myself. Still need to write a
 script, that walks through the files, and only compares the messages
 that are in both folders. If I run make_md5, it's never working on a
 folder on both servers at the same time, so there's always a gap.  
 So the
 files aren't always equal, I think. Oh well.

I don't have something to consume make_md5 data, yet, either.  My  
plan is to note the difference between the replica and the primary.   
On a subsequent run, if those differences aren't gone, then they  
would be included in a report.

:wes

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html


Re: Cyrus with a NFS storage. random DBERROR

2007-06-08 Thread Wesley Craig
I run it directly, outside of master.  That way when it crashes, it  
can be easily restarted.  I have a script that checks that it's  
running, that the log file isn't too big, and that there are no log- 
PID files that are too old.  If anything like that happens, it pages  
someone.

:wes

On 08 Jun 2007, at 11:48, Nik Conwell wrote:
 Do people run sync_client in the SERVICES section rather than START?
 The install-replication docs indicate to put it in START.  If my
 replica goes away for a little while, sync_client exits and then I
 have to restart it manually and then process any pending logs.  Would
 be nice if it just started automatically and picked up where it left
 off.

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html


Re: Cyrus with a NFS storage. random DBERROR

2007-06-08 Thread Rob Mueller
 I don't have something to consume make_md5 data, yet, either.  My
 plan is to note the difference between the replica and the primary.
 On a subsequent run, if those differences aren't gone, then they
 would be included in a report.

Rather than make_md5, check the MD5 UUIDs patch below. Using this, we have a 
script that regularly checks both sides of a master/replica pair to check 
everything is consistent between the UUID and the computed MD5. It was this 
that let us discover the rare didn't unlink old files bug reported about 3 
months back.

---
http://cyrus.brong.fastmail.fm/

One problem we've had is the inability to easily check that the files on 
disk correspond to what was originally delivered to check for cyrus data 
corruption after either a disk problem or some other bug has caused us to be 
unsure of our data integrity.
I wanted to calculate a digest and store it somewhere in the index file, but 
messing with the file format and fixing sync to still work, etc... it all 
sounded too painful.

So - added is a new option uuidmode in imapd.conf. Set it to md5 and you 
will get UUIDs of the form: 02(first 11 bytes of the MD5 value for the 
message) which takes up the same space, but allows pretty good integrity 
checking.

Is it safe? - we calulated that with one billion messages you have a one in 
1 billion chance of a birthday collision (two random messages with the same 
UUID). They then have to get in the same MAILBOXES collection to sync_client 
to affect each other anyway. The namespace available for generated UUIDs is 
much smaller than this, since they have no collision risk - but if you had 
that many delivering you would hit the limits and start getting blank UUIDs 
anyway.

Mitigating even the above risk: you could alter sync_client to not use UUID 
for copying. It's not like it's been working anyway (see our other UUID 
related patch). As an integrity check it's much more useful.


The attached patch adds the md5 method, a random method which I've never 
tested and is almost certainly bogus, but is there for educational 
value[tm], the following FETCH responses in imapd:

FETCH UUID = 24 character hex string (02 + first 11 bytes of MD5) FETCH 
RFC822.MD5 = 32 character hex string (16 bytes of MD5) FETCH 
RFC822.FILESIZE = size of actual file on disk (via stat or mmap)

Totally non-standard of course, but way useful for our replication checking 
scripts. Embrace and extend 'r' us.

Anyone feel like writing an RFC for fetching the digest of a message via 
IMAP? If the server calculated it on delivery and cached it then you'd have 
a great way to clean up after a UIDVALIDITY change or other destabilising 
event without having to fetch every message again.

---

Rob


Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html


Re: Cyrus with a NFS storage. random DBERROR

2007-06-08 Thread Rob Mueller

I run it directly, outside of master.  That way when it crashes, it
 can be easily restarted.  I have a script that checks that it's
 running, that the log file isn't too big, and that there are no log-
 PID files that are too old.  If anything like that happens, it pages
 someone.

Ditto, we do almost exactly the same thing.

Also if we switch master/replica roles, our code looks for any incomplete 
log files after stopping the master, and runs those first to ensure that 
replication was completely up to date.

It seems anyone seriously using replication has to unfortunately do these 
things manually at the moment. Replication just isn't reliable enough, we 
see sync_client bail out quite regularly, and there's not enough logging to 
exactly pinpoint why each time. I think there's certain race conditions that 
still need ironing out, because rerunning sync_client on the same log file 
that caused a bail out usually succeeds the second time. It would be nice if 
some code was actually made part of the core cyrus distribution to make this 
all work properly, including switching master/replica roles.

Rob


Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html


Re: Cyrus with a NFS storage. random DBERROR

2007-06-08 Thread David Lang
On Sat, 9 Jun 2007, Rob Mueller wrote:

 So - added is a new option uuidmode in imapd.conf. Set it to md5 and you
 will get UUIDs of the form: 02(first 11 bytes of the MD5 value for the
 message) which takes up the same space, but allows pretty good integrity
 checking.

 Is it safe? - we calulated that with one billion messages you have a one in
 1 billion chance of a birthday collision (two random messages with the same
 UUID). They then have to get in the same MAILBOXES collection to sync_client
 to affect each other anyway. The namespace available for generated UUIDs is
 much smaller than this, since they have no collision risk - but if you had
 that many delivering you would hit the limits and start getting blank UUIDs
 anyway.

does the IMAP spec specify how large a UUID can be?

David Lang

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html