Re: GSSAPI: A token had an invalid MIC

[Wesley Craig]

On 27 Oct 2006, at 22:35, Phil Pennock wrote:

New server:
 OS/Arch: FreeBSD 6.1 / amd64
 Cyrus IMAPd: 2.3.7
 Cyrus SASL 2.1.22
 OpenSSL: 0.9.7i
 Heimdal: 0.7.2 (OS port rev _1)

Anyone any ideas or pointers, please?


Given the versions of things, I'd suspect one or both of these two  
problems:


http://people.su.se/~lha/patches/heimdal/cfx-wrap-size.txt

and

http://www.irbs.net/internet/cyrus-sasl/0609/0011.html

:wes

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Deferred email with remote protocol error in reply

[Wesley Craig]

On 27 Oct 2006, at 16:00, Henrique de Moraes Holschuh wrote:
That said, just moving off 0,1,2 is not difficult.  ... The ideal  
fix would be to have them be dynamic.


Ideally, 0, 1, and 2 should be reserved for stdin, stdout, and  
stderr.  In an environment where std* don't make sense, i.e.,  
daemons, they should still be reserved, since standard libraries  
occasionally try to use them.  I'm not all that sure that moving off  
0, 1, and 2 would be that easy.  Take main() from master/service.c,  
where the accepted socket is dup2()'d 0, 1, and 2.  imapd, pop3d,  
etc, seem to call prot_new() on the descriptors routinely.  However,  
you'd also have to catch references to getpeername( 0, ... ),  
getsockname( 0, ... ), shutdown( 0 ), etc.  In principal  
straightforward, but in practice there's a lot of code that would  
need to be reviewed & tested.


:wes

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Cyrus, clusters, GFS - HA yet again

[Marten Lehmann]

Hello,

maybe I have understood GFS wrong, but isn't it ment to stripe data of 
several servers instead of mirroring them but make it accessable from 
several servers? If one server goes down, then you can only access the 
metadata from the GFS metadata server, but not the file itself from the 
server.


Even with a cyrus murder cluster you cannot have shared mailboxes with 
different domain names (.university.xx). With Cyrus each 
mailbox belongs to a certain backend-server. To be completly independed 
you need a big SAN (or maybe GFS) with Maildirs and then you can add an 
arbitrary amount of servers for deliveries, spamfilters, virus scan, 
imap and pop3 and make snapshots and backups of the complete storage.


However, that way you will get into trouble with the calculation of 
quota at each delivery of a new messages, because it takes very long to 
sum all sizes of the messages within a mailbox as it cannot to be done 
locally on the harddisk but has to be done over the network (through the 
GFS or SAN) and if a maildir contains several hundrets or thousand 
messages it takes really long and will slow down the whole system 
because of i/o waits. Look for an smtp-server and imap/pop3 server that 
can handle maildirsizefile (Maildir++), then this problem would be 
solved. But as far as I know there are no imap/pop3 servers available 
that have the same feature set as Cyrus (indices, quota system, shared 
folders, etc.) so you have to make a choice.


Regards
Marten

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

GSSAPI: A token had an invalid MIC

[Phil Pennock]

I'm having a problem with GSSAPI on a new install of Cyrus IMAP, where
no clients are able to successfully negotiate a connection; my own
client code is reporting "A token had an invalid MIC", GSS_S_BAD_MIC,
when trying to unwrap the data in the GSSAPI context, so it's not
getting the server's token.  The client does successfully get a ticket,
etc, and this problem also occurs with imtest, which says
"Authentication failed. generic failure".

Old server where this works:
 OS/Arch: Gentoo Linux / x86
 Cyrus IMAPd: 2.2.12
 Cyrus SASL: 2.1.21 (OS portage rev -r2)
 OpenSSL: 0.9.8d
 Heimdal: 0.7.2 (OS portage rev -r3)

New server:
 OS/Arch: FreeBSD 6.1 / amd64
 Cyrus IMAPd: 2.3.7
 Cyrus SASL 2.1.22
 OpenSSL: 0.9.7i
 Heimdal: 0.7.2 (OS port rev _1)

The server's not logging any problems, or anything happening after the
TLS negotiation; same problem occurs without TLS, when nothing at all
gets logged for the connection.  If I set CYRUS_VERBOSE=15 (and confirm
that it's in the env of the master process with ps(1)) then I get
nothing more than this.

If I ktrace the cyrus services, it's accessing the correct keytab file.
The client gets a ticket; "kinit -R" to wipe all but the TGT and then
trying again confirms that there's no problem there.

The only access for the new box is IPv6, since that lets me use a single
hostname with dedicated forward and reverse DNS, on the public Internet.
Is this likely to be connected?  Other IPv6-only services are working
fine with GSSAPI (eg, OpenLDAP), so anything specific to the Kerberos
implementation and the embedded IP addresses is working.

I've rebuilt cyrus-sasl and cyrus-imapd to ensure that they were built
and linked against the correct Heimdal libraries.

Anyone any ideas or pointers, please?
Thanks,
-Phil

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Message contains invalid header

[Marten Lehmann]

Hi,


case IMAP_MESSAGE_BADHEADER:
prot_printf(pout, "554 5.6.0 Message contains invalid header\r\n");
break;


what can I do to fix these headers? I have one message (which appears to 
be spam, but that is not the point), where I cannot find any problem. No 
8bit characters, no empty message-id header, what else could be wrong 
with it?


Regards
Marten

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: recover from hard restart

[Phil Pennock]

On 2006-10-27 at 14:39 -0400, Benjamin Adams wrote:
> cyrus.cache.NEW
> cyrus.index.NEW
> 
> And their mail clients are not working correctly.
> How can I fix this without taking the server down?

Reconstruct those folders.  See reconstruct(8) -- the command cleans
this up.
-- 
"Everything has three factors: politics, money, and the right way to do it.
 In that order."  -- Gary Donahue

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: recursive reconstruct does not work?

[Jo Rhett]

Ken...?  Haloo.

On Oct 12, 2006, at 12:04 PM, Jo Rhett wrote:

Ken, any news on this?  I could swear we talked about this a few years
ago.  Sorry for the absence, got bought out and stuck with old mobx
UW-IMAP for a few years :-(

Andreas Winkelmann wrote:

Am Wednesday 04 October 2006 00:15 schrieb Jo Rhett:

Maybe you should give us some more information about your setup?

Nothing special.  Virtdomains with unixhiersep, altnamespace and
authentication via sasl auxprop-sql (mysql) out of the box FreeBSD
ports.


And show a list of these Mailboxnames with cyradm.

Sure.  Here's the mailboxes and the ktrace of a reconstruct of this
mailbox (me)

localhost> lm user/[EMAIL PROTECTED]
user/jrhett/AT&[EMAIL PROTECTED] (\HasNoChildren)
user/jrhett/American [EMAIL PROTECTED] (\HasNoChildren)
user/jrhett/[EMAIL PROTECTED] (\HasNoChildren)
user/jrhett/[EMAIL PROTECTED] (\HasNoChildren)
user/jrhett/[EMAIL PROTECTED] (\HasNoChildren)
user/jrhett/[EMAIL PROTECTED] (\HasNoChildren)
user/jrhett/[EMAIL PROTECTED] (\HasNoChildren)
user/jrhett/[EMAIL PROTECTED] (\HasNoChildren)
user/jrhett/[EMAIL PROTECTED] (\HasNoChildren)
user/jrhett/[EMAIL PROTECTED] (\HasNoChildren)
user/jrhett/lists/[EMAIL PROTECTED] (\HasNoChildren)
user/jrhett/lists/[EMAIL PROTECTED] (\HasNoChildren)
user/jrhett/lists/[EMAIL PROTECTED] (\HasNoChildren)
user/jrhett/[EMAIL PROTECTED] (\HasChildren)
user/[EMAIL PROTECTED] (\HasChildren)

Yes, seems to be a Problem with virtdomains enabled.
Maybe Ken can say something more.



--
Jo Rhett
Senior Network Engineer
Network Consonance




--
Jo Rhett
Senior Network Engineer
Network Consonance


Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: recursive reconstruct does not work?

[Ken Murchison]

Recursive reconstruct should and does work AFAIK.  Are you trying to 
reconstruct an entire domain, multiple domains, or ...   ?




Jo Rhett wrote:

Ken...?  Haloo.

On Oct 12, 2006, at 12:04 PM, Jo Rhett wrote:

Ken, any news on this?  I could swear we talked about this a few years
ago.  Sorry for the absence, got bought out and stuck with old mobx
UW-IMAP for a few years :-(

Andreas Winkelmann wrote:

Am Wednesday 04 October 2006 00:15 schrieb Jo Rhett:

Maybe you should give us some more information about your setup?

Nothing special.  Virtdomains with unixhiersep, altnamespace and
authentication via sasl auxprop-sql (mysql) out of the box FreeBSD
ports.


And show a list of these Mailboxnames with cyradm.

Sure.  Here's the mailboxes and the ktrace of a reconstruct of this
mailbox (me)

localhost> lm user/[EMAIL PROTECTED]
user/jrhett/AT&[EMAIL PROTECTED] (\HasNoChildren)
user/jrhett/American [EMAIL PROTECTED] (\HasNoChildren)
user/jrhett/[EMAIL PROTECTED] (\HasNoChildren)
user/jrhett/[EMAIL PROTECTED] (\HasNoChildren)
user/jrhett/[EMAIL PROTECTED] (\HasNoChildren)
user/jrhett/[EMAIL PROTECTED] (\HasNoChildren)
user/jrhett/[EMAIL PROTECTED] (\HasNoChildren)
user/jrhett/[EMAIL PROTECTED] (\HasNoChildren)
user/jrhett/[EMAIL PROTECTED] (\HasNoChildren)
user/jrhett/[EMAIL PROTECTED] (\HasNoChildren)
user/jrhett/lists/[EMAIL PROTECTED] (\HasNoChildren)
user/jrhett/lists/[EMAIL PROTECTED] (\HasNoChildren)
user/jrhett/lists/[EMAIL PROTECTED] (\HasNoChildren)
user/jrhett/[EMAIL PROTECTED] (\HasChildren)
user/[EMAIL PROTECTED] (\HasChildren)

Yes, seems to be a Problem with virtdomains enabled.
Maybe Ken can say something more.



--
Jo Rhett
Senior Network Engineer
Network Consonance







--
Kenneth Murchison
Systems Programmer
Project Cyrus Developer/Maintainer
Carnegie Mellon University

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: virtual domains

[Jo Rhett]

On Oct 26, 2006, at 11:00 PM, A Clockwork Orange wrote:

Cyradm can not create virtual mailbox!
Why?
localhost.strs.ru>  cm [EMAIL PROTECTED]
createmailbox: Permission denied
localhost.strs.ru>


Please try some self-help before you ask us to fix your problems for  
you.


"Permission denied." probably indicates a permissions problem on the  
mailbox spool, yes?


You may want to investigate commercial support options...

--
Jo Rhett
Senior Network Engineer
Network Consonance


Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Sieve Vacation not working (filter does work)

[Jo Rhett]

Did you enable the mailto method in imapd.conf?  Without this  
enabled, all of the functions that send mail are disabled.


--
Jo Rhett
Senior Network Engineer
Network Consonance


Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Cyrus started

[Jo Rhett]

On Oct 26, 2006, at 11:12 PM, A Clockwork Orange wrote:

Where are these error from?
I tried to set 777 for /var/imap. It did not help.


Please investigate commercial support options before you hurt  
yourself.  Seriously.


--
Jo Rhett
Senior Network Engineer
Network Consonance


Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Deferred email with remote protocol error in reply

[Henrique de Moraes Holschuh]

On Fri, 27 Oct 2006, Wesley Craig wrote:
> On 27 Oct 2006, at 15:33, Henrique de Moraes Holschuh wrote:
> >Cyrus master has issues with FDs 0,1 and 2 since ancient times.  If  
> >anything
> >in the code, anywhere, closes them and don't reopen them immediately,
> >something like syslog() or another offender will promptly take them  
> >over and
> >cause a lot of ruckus later.
> 
> That bug was fixed:
> 
>   https://bugzilla.andrew.cmu.edu/show_bug.cgi?id=2422
> 
> Of course, it probably wouldn't have been a problem is master wasn't  
> designed to use 0, 1, and 2.

Yes, I missed a "had" after "has".  2.1 still has it somewhere deep inside
the code I didn't manage to root out for Debian, for example.

That said, just moving off 0,1,2 is not difficult.  But it would not help at
all with bug #2422 (it is a problem with statically allocated fds, not with
them being 0, 1 and 2).

The ideal fix would be to have them be dynamic.

-- 
  "One disk to rule them all, One disk to find them. One disk to bring
  them all and in the darkness grind them. In the Land of Redmond
  where the shadows lie." -- The Silicon Valley Tarot
  Henrique Holschuh

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Deferred email with remote protocol error in reply

[Henrique de Moraes Holschuh]

On Fri, 27 Oct 2006, Wesley Craig wrote:
> The larger problem is that daemons shouldn't be impacted by libraries  
> that inappropriately write to stdout or stderr.  As such, lmtpd  
> should have closed stdin, stdout, stderr and reopening them with,  
> e.g., '/', O_RDONLY.  Accepted connections should not use file  
> descriptors 0, 1, or 2, since those file descriptors are frequently  
> (mis)used by underlying libraries.  This is the fatal error in inetd,  
> and apparently in cyrus master?

Cyrus master has issues with FDs 0,1 and 2 since ancient times.  If anything
in the code, anywhere, closes them and don't reopen them immediately,
something like syslog() or another offender will promptly take them over and
cause a lot of ruckus later.

-- 
  "One disk to rule them all, One disk to find them. One disk to bring
  them all and in the darkness grind them. In the Land of Redmond
  where the shadows lie." -- The Silicon Valley Tarot
  Henrique Holschuh

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Deferred email with remote protocol error in reply

[Wesley Craig]

On 27 Oct 2006, at 15:33, Henrique de Moraes Holschuh wrote:
Cyrus master has issues with FDs 0,1 and 2 since ancient times.  If  
anything

in the code, anywhere, closes them and don't reopen them immediately,
something like syslog() or another offender will promptly take them  
over and

cause a lot of ruckus later.


That bug was fixed:

https://bugzilla.andrew.cmu.edu/show_bug.cgi?id=2422

Of course, it probably wouldn't have been a problem is master wasn't  
designed to use 0, 1, and 2.


:wes



Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

recover from hard restart

[Benjamin Adams]

I had a server hard restart, and some users (looks like just the ones login) have this now:cyrus.cachecyrus.cache.NEWcyrus.headercyrus.indexcyrus.index.NEWAnd their mail clients are not working correctly.How can I fix this without taking the server down?Thanks.Ben 
Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Message contains invalid header

[Henrique de Moraes Holschuh]

On Fri, 27 Oct 2006, Marten Lehmann wrote:
> especially this appears very often. Does Cyrus actually mean the whole 
> message, or message body or header? What can I do with existing messages?

Last time I had a problem like this, I piped them all through tr -d '\000'.
Nowadays, postfix is set to reject this crap before it comes near Cyrus in
some places, and in others (where we need the messages), we strip the NULs
away shamelessly.

After all, common courtesy of not modifying data in-transit extends only to
email, and whatever something with embedded NULs is, it certainly ain't
email.  They just resemble email enough that they can (often) be converted
into email with some NUL-stripping ;-)

The same goes to messages with empty message-id headers. We strip them away
so that they become email ;p

-- 
  "One disk to rule them all, One disk to find them. One disk to bring
  them all and in the darkness grind them. In the Land of Redmond
  where the shadows lie." -- The Silicon Valley Tarot
  Henrique Holschuh

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Deferred email with remote protocol error in reply

[Wesley Craig]

On 27 Oct 2006, at 05:08, Libor Pechacek wrote:
MAIL FROM:<[EMAIL PROTECTED] >  
SIZE=15311

RCPT TO:<[EMAIL PROTECTED] >
DATA

 < 250 2.1.0 ok
 < 250 2.1.5 ok
 < 354 go ahead

.

 < 7 lockers

In this case the obvious reason for the error is the "7 lockers" line
that leaked from BDB.  It makes only small harm itself but causes the
next message to be bounced due to protocol error in case the LMTP
connection is reused.  Workaround to the bounces is simple -
"lmtp_cache_connection = no" in Postfix's main.cf.


Here's where this happens in BDB 4.3.29:

./lock/lock_deadlock.c-418- *nlockers = 0;
./lock/lock_deadlock.c-419- return (0);
./lock/lock_deadlock.c-420- }
./lock/lock_deadlock.c-421-
./lock/lock_deadlock.c-422- if (FLD_ISSET(dbenv->verbose,  
DB_VERB_DEADLOCK))
./lock/lock_deadlock.c:423: __db_msg(dbenv, "%lu  
lockers", (u_long)count);

./lock/lock_deadlock.c-424-
./lock/lock_deadlock.c-425- count += 20;
./lock/lock_deadlock.c-426- nentries = (u_int32_t)DB_ALIGN(count,  
32) / 32;

./lock/lock_deadlock.c-427-
./lock/lock_deadlock.c-428- /*

So, first thing is that the message shouldn't be written if this  
DB_VERB_DEADLOCK isn't set.  Quickly skimming the BDB code, it  
doesn't appear that this defaults on.  I'm wondering if you have:


set_verbose db_verb_deadlock

in your DB_CONFIG file?  Once it's on, I don't see a way to turn it  
off, other than calling the set_verbose method from within Cyrus.


The larger problem is that daemons shouldn't be impacted by libraries  
that inappropriately write to stdout or stderr.  As such, lmtpd  
should have closed stdin, stdout, stderr and reopening them with,  
e.g., '/', O_RDONLY.  Accepted connections should not use file  
descriptors 0, 1, or 2, since those file descriptors are frequently  
(mis)used by underlying libraries.  This is the fatal error in inetd,  
and apparently in cyrus master?


:wes

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Message contains invalid header

[Marten Lehmann]

Hello,


case IMAP_MESSAGE_CONTAINSNULL:
prot_printf(pout, "554 5.6.0 Message contains NUL characters\r\n");
break;


especially this appears very often. Does Cyrus actually mean the whole 
message, or message body or header? What can I do with existing messages?


Regards
Marten

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Message contains invalid header

[Andrew Morgan]

On Fri, 27 Oct 2006, Marten Lehmann wrote:


Hello,


But your Cyrus IMAPd has to work with these headers! It has to be able
to sort by date, search for message ID, index the body, search for any
header field! So you cannot just ignore errors in Cyrus.


yes, I understand. But I don't know which characters are problematic for 
Cyrus. I just see the message "Message containes invalid headers". So what? 
Is it an empty Message-ID? Is it a NUL-character? Is it 8bit in header where 
usually only 7bit is allowed (munge8bit is false in our setup)? There is now 
documentation on this. Do I have to be a programmer in watch through thousand 
lines of source code to find it out? Actually, I just want to use Cyrus, not 
change it.


Looking at the source code, it appears that error message is only 
generated when there is an empty Message-ID: header.  As other people have 
said, there are ways to have your MTA fix-up the Message-ID: header 
before the message is handed to Cyrus.


There are separate error messages for other conditions:

case IMAP_MESSAGE_CONTAINSNULL:
prot_printf(pout, "554 5.6.0 Message contains NUL characters\r\n");
break;

case IMAP_MESSAGE_CONTAINSNL:
prot_printf(pout, "554 5.6.0 Message contains bare newlines\r\n");
break;

case IMAP_MESSAGE_CONTAINS8BIT:
prot_printf(pout, "554 5.6.0 Message contains non-ASCII characters in 
headers\r\n");
break;

case IMAP_MESSAGE_BADHEADER:
prot_printf(pout, "554 5.6.0 Message contains invalid header\r\n");
break;



Andy

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Cyrus started

[Andrew Morgan]

On Fri, 27 Oct 2006, A Clockwork Orange wrote:


Oct 27 10:08:58 mow master[17342]: process started
Oct 27 10:08:58 mow master[4984]: about to exec 
/usr/local/libexec/cyrus-imapd/ctl_cyrusdb
Oct 27 10:08:59 mow ctl_cyrusdb[4984]: recovering cyrus databases
Oct 27 10:09:03 mow ctl_cyrusdb[4984]: done recovering cyrus databases
Oct 27 10:09:04 mow master[17342]: service not supported for ai_socktype, 
disabling sieve
Oct 27 10:09:04 mow master[17342]: ready for work
Oct 27 10:09:04 mow master[13214]: about to exec 
/usr/local/libexec/cyrus-imapd/ctl_cyrusdb
Oct 27 10:09:04 mow ctl_cyrusdb[13214]: checkpointing cyrus databases
Oct 27 10:09:04 mow ctl_cyrusdb[13214]: archiving database file: 
/var/imap/annotations.db
Oct 27 10:09:04 mow ctl_cyrusdb[13214]: DBERROR: error listing log files: 
Permission denied
Oct 27 10:09:04 mow ctl_cyrusdb[13214]: DBERROR: archive /var/imap/db: cyrusdb 
error
Oct 27 10:09:04 mow ctl_cyrusdb[13214]: archiving database file: 
/var/imap/mailboxes.db
Oct 27 10:09:04 mow ctl_cyrusdb[13214]: DBERROR: error listing log files: 
Permission denied
Oct 27 10:09:04 mow ctl_cyrusdb[13214]: DBERROR: archive /var/imap/db: cyrusdb 
error
Oct 27 10:09:04 mow ctl_cyrusdb[13214]: done checkpointing cyrus databases
Oct 27 10:09:04 mow master[17342]: process 13214 exited, status 1

Where are these error from?
I tried to set 777 for /var/imap. It did not help.


Everything under /var/imap should be owned by your cyrus user.  If you are 
running cyrus as the user 'cyrus', then you could try 'chown -R cyrus 
/var/imap/'.


Andy

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Message contains invalid header

[Marten Lehmann]

Hello,


But your Cyrus IMAPd has to work with these headers! It has to be able
to sort by date, search for message ID, index the body, search for any
header field! So you cannot just ignore errors in Cyrus.


yes, I understand. But I don't know which characters are problematic for 
Cyrus. I just see the message "Message containes invalid headers". So 
what? Is it an empty Message-ID? Is it a NUL-character? Is it 8bit in 
header where usually only 7bit is allowed (munge8bit is false in our 
setup)? There is now documentation on this. Do I have to be a programmer 
in watch through thousand lines of source code to find it out? Actually, 
I just want to use Cyrus, not change it.


Regards
Marten

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Cyrus, clusters, GFS - HA yet again

[Dave McMurtrie]

Janne Peltonen wrote:


And if further splitting of users on more servers is needed - downtime
again. Moreover, it's confusing for the users to have to determine their
correct imap server name - we haven't really had trouble with this, but
it would be nice if the users saw a unified system image.

 

If you decide not to pursue a cluster solution, Perdition would probably 
help you with this part.



Enter weirdness. The first Cyrus to be started starts with no
complaints and ends up with the correct number (as specified in
/etc/cyrus.conf) of imapd, imapd -s, pop3d, lmtp etc. processes, all in
state S, only one process at a time having a write lock on
/var/lib/imap/socket/xxx-N.lock. 

These lockfiles are used to serialize their corresponding processes 
(imap, lmtp, etc) on a per-host basis, not across cluster nodes.  As 
such, you should write these to the local filesystem and not the cluster 
filesystem.  You can accomplish this with symbolic links.


I believe you'll also need to make some minor code changes.  When 
University of Pittsburgh implemented their Cyrus cluster, they added a 
nodename config option and then used that nodename as a filename 
component along with the pid for the lmtp temporary deliver files such 
that the filenames would be unique across cluster nodes.  Without 
modification, only the pid is used as a means to make the filenames unique.


Thanks,

Dave

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Cyrus, clusters, GFS - HA yet again

[Janne Peltonen]

Hi list.

Sorry for the long post. I hope someone has time to read it and shed
some light on my concerns. This all boils down to one question: those
that have succeeded in running active-active Cyrus cluster configs, how
have you done it?

So. Some background:

I inherited a university imap system w/ abt 40k users a few weeks ago.
The system uses an old version of Cyrus; scalability has been achieved
through splitting the user-base according to their faculty, having a
separate hostname for each faculty and adding servers as needed.
Currently, we have three more-or-less independent servers, two of which
serve abt 16k users and the third, a couple thousand. This config runs
fairly well, but has its problems. For example, shared mailboxes with
users from faculties on different servers don't work - the users have to
be migrated by hand to the sorrect server. Also, if one server goes
down, all the people on that server see a break on their mail service.
And if further splitting of users on more servers is needed - downtime
again. Moreover, it's confusing for the users to have to determine their
correct imap server name - we haven't really had trouble with this, but
it would be nice if the users saw a unified system image.

As a solution, my predecessor had considered creating an active-active
cluster system, with Cyrus mailspool and config on shared GFS, load
balancing through a 'magic box' that arbitrates incoming connections
among the nodes, and each node running its own instance of Cyrus serving
all of the mailboxes. Such a setup would have the unified system image,
each of the nodes would be fully redundant, and adding new nodes would
be simple - the nodes could be more or less identical... However, my
predecessor left to complete his studies before even creating a test
environment for these ideas.

For the last few weeks, I've been reading through Cyrus documentation,
GFS documentation, RedHat cluster suite documentation, mailing list
archives, whatnot, trying to find out whether my predecessor's solution
is achievable. To no avail: I found lots of warnings against ever
running independent Cyri on the same filesystem, and counterexamples of
people having successfully run precisely such a beast for years. The
issue seemed to boil down to having completely functional and efficient
file locking and mmaping semantics - GFS should have them.  I found the
test conducted by the Italian group that showed that GFS doesn't perform
as well as its commercial counterparts in email-like situations -
however, my system is abt an order of magnitude smaller than theirs, so
I didn't consider it an issue.

So I set up my test environment: two HP Blade servers, both with their
own disks for system and two SAN-shared block devices for Cyrus; CentOS
4.4; kernel-2.6.9-42.0.2.EL; GFS-6.1.6-1; dlm-1.0.1-1; cman-1.0.11-0;
cyrus-imapd-2.2.12-3.RHEL4.1 (the revisions are those of the rpm
packages from CentOS). /var/lib/imap (config dir) and /var/spool/imap
(spooldir) are two GFS filesystems on cLVM on SAN. To keep the testing
simple, I didn't set the Cyri up as cluster services; I simply set the
cluster up and running with no services, mounted the GFS's on both nodes
and started Cyrus on both nodes.

Enter weirdness. The first Cyrus to be started starts with no
complaints and ends up with the correct number (as specified in
/etc/cyrus.conf) of imapd, imapd -s, pop3d, lmtp etc. processes, all in
state S, only one process at a time having a write lock on
/var/lib/imap/socket/xxx-N.lock. The Cyrus on the other node starts with
only two versions of each imapd etc. process (except non-secure pop3d;
all of them run); all those end up in state D. And each of them holds a
write lock on /var/lib/imap/socket/xxx-N.lock (or, more probably, tries
to acquire the lock from the equivalent process on the other node; the
process doesn't seem to give it up, though). And if I log in on the
Cyrus on node 1 while the Cyrus on node 2 is running, the imapd on node 1
complains about database corruption after I log out. I don't know if any
database is really corrupted.

It seems to me that running two Cyri on different nodes with a shared
configdir doesn't work even if we have the required mmap and locking
semantics: it appears that someone (perhaps the cyrus-master process?)
arbitrates the locks in the /socket directory, using other
means of communication than the shared filesystem. lsof'ing the master
and imapd's shows that there is a pipe between the master process and
each its child. Might it be that they communicate via it?

Now. How do you run two instances of Cyrus on the same filesystem? Is
there a config option I'm missing? Or should I just give up and start
considering Murder?

Greetings,


--Janne Peltonen
Univ of Helsinki
Imap admin

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Deferred email with remote protocol error in reply

[Libor Pechacek]

Andreas Winkelmann wrote:

> Am Friday 15 September 2006 04:52 schrieb Hardi Gunawan:
>
>> My email server experiencing a delivery problem.
>> Here's the log:
>> relay=/var/lib/imap/socket/lmtp[/var/lib/imap/socket/lmtp],
>> delay=0, status=deferred (remote protocol error in
>> reply from
>> /var/lib/imap/socket/lmtp[/var/lib/imap/socket/lmtp]
>> while sending end of data -- message may be sent more
>> than once)
>>
>
> Hmm, it seems, that Server (Cyrus lmtpd) and Client (Postfix lmtp) are out of 
> sync. I've never seen that on Unix-Sockets.
>
> Turn on verbose Logging in Postfix. master.cf:
>
> lmtp . lmtp -v
> or
> lmtp . lmtp -v -v
>
> And show the conservation.

Here is abbreviated log of a failed LMTP transaction:

 > MAIL FROM:<[EMAIL PROTECTED] > SIZE=15311
 > RCPT TO:<[EMAIL PROTECTED] >
 > DATA
 < 250 2.1.0 ok
 < 250 2.1.5 ok
 < 354 go ahead
 > .
 < 7 lockers
# Postfix reports non-LMTP response and resets the transaction
 > RSET
 < 250 2.1.5 Ok
# this is final response to DATA command, now
# are the client and server out of sync
 > MAIL FROM:<[EMAIL PROTECTED] > SIZE=15311
 > RCPT TO:<[EMAIL PROTECTED] >
 > DATA
 < 250 2.0.0 ok
 < 250 2.1.0 ok
 < 250 2.1.5 ok
# responses to RSET (delayed), MAIL FROM and RCPT TO (both OK), the first
# response is extra; Postfix interprets the last 250 response as response
# to DATA command and bounces (!) the message
 > RSET
 < 354 go ahead
# delayed 354 response to DATA :(
 > RSET

In this case the obvious reason for the error is the "7 lockers" line
that leaked from BDB.  It makes only small harm itself but causes the
next message to be bounced due to protocol error in case the LMTP
connection is reused.  Workaround to the bounces is simple -
"lmtp_cache_connection = no" in Postfix's main.cf.

I think Cyrus should keep from writing the 250 response when the reset
is already sent by client.  I thought the response was in the output
buffer at the time RSET arrives and simply clearing the buffer would do
the work but it seems to be a little bit more complicated.  Reading the
next command with prot_fgets at lmtpengine.c:1145 flushes the output
buffer so that the unfortunate response gets to the client.  I believe
Cyrus should peek input stream for RSET before flushing the output
buffer but I still haven't found the way how to implement it.

Additional info:
OS SUSE SLES10, Postfix 2.2.9, Cyrus 2.2.12, Berkeley DB 4.3.29,
delivery through local Unix socket (mailbox_transport =
lmtp:unix:/var/lib/imap/socket/lmtp), reproduce with "smtp-source -l
15000 -m 1000 -s 100 -t test -f [EMAIL PROTECTED] localhost"

Similar out-of-sync problem is described at
http://archives.neohapsis.com/archives/postfix/2005-11/1977.html

Hope, this helps.

Libor Pechacek





Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Message contains invalid header

[Daniel Eckl]

On 26.10.2006 23:57, Marten Lehmann wrote:
> exim. It can handle even emails with NUL characters and 8bit headers,
> but I guess all up2date MTAs (like postfix or sendmail) are capable of
> this.
Well, that's a bad comparison between MTA and your mail storage! A MTA
normally doesn't care about the body. It just has SMTP envelopes to work
with. MAIL FROM, RCPT TO and DATA is all it really needs (I know that
very simplified, it adds Received: lines and so on). If the message
header or body in the DATA part is seriously broken it doesn't make much
problems (in 99,9% of all cases).

But your Cyrus IMAPd has to work with these headers! It has to be able
to sort by date, search for message ID, index the body, search for any
header field! So you cannot just ignore errors in Cyrus. But perhaps you
could use a program in between your MTA and cyrus which corrects these
problems automagically. But I don't really know of any scalable solution
for that. procmail piping to a script or similar sounds very overhead
for me, but I'm not sure here...

Best,
Daniel

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Sieve Vacation not working (filter does work)

[Georg Glas]

Martin Schweizer schrieb:

> Did you use umlauts or other non 8 bit characters in the vacation text
> (that was my problem at the beginning)?
>> the script itself is really simple:
>>
>> require "vacation";
>> vacation :days 7 :addresses [ "[EMAIL PROTECTED]", "[EMAIL PROTECTED]"] "Out 
>> of
>> office ...";

no plain 7-bit as seen in the lines above

regards
Georg Glas

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html