Re: Sync failing. Possible db corruption
Problem resolved, or resolving. This version of Cyrus may be old, but it's been rock solid and compared to previous versions I've had no problems with sync_client dying. It turns out the behavior I was seeing is normal recover behavior. There are still errors I have not seen before, but they appear to be from turning on verbose logging, and the synchronization being out of step from the previous errors. As people check mailboxes, it is catching up. Mike On 9/15/20 4:20 PM, Michael Sofka wrote: I have confirmed that running sync_client on individual mailboxes works, as does running sync on the accumulated log files. But rolling replication is not working. After sync_client is restarted there is a burst of sync activity on the accumulated log, then nothing. The new log file continues to accumulate, sync_client continues to run, but is not processing anything so far as I can determine. -- -- Michael D. Sofka sof...@rpi.edu ITI Software Architect, Email, TeX, Epistemology Rensselaer Polytechnic Institute, Troy, NY. http://www.rpi.edu/~sofkam/ Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Sync failing. Possible db corruption
I have confirmed that running sync_client on individual mailboxes works, as does running sync on the accumulated log files. But rolling replication is not working. After sync_client is restarted there is a burst of sync activity on the accumulated log, then nothing. The new log file continues to accumulate, sync_client continues to run, but is not processing anything so far as I can determine. Sep 15 15:59:17 imap-be6 cyrus/sync_client[22370]: SYNCNOTICE: replica uid:7451 modseq:19935 last_updated:1600023598 internaldate:1600023598 flags:( \Seen) Sep 15 15:59:17 imap-be6 cyrus/sync_client[22370]: inefficient replication (1487 > 1484) user.jins4.Sent Sep 15 15:59:18 imap-be6 cyrus/sync_client[22370]: inefficient replication (182427 > 182409) user.newelj Sep 15 15:59:18 imap-be6 cyrus/sync_client[22370]: SYNCERROR: only exists on replica user.newelj 121180 (a27c1dffed0326fb930b072fd85f04a8c2cbdf24) Sep 15 15:59:23 imap-be6 cyrus/sync_client[22370]: inefficient replication (5598 > 5571) user.berryk3 Sep 15 15:59:23 imap-be6 cyrus/sync_client[22370]: SYNCNOTICE: record mismatch with replica: user.berryk3 more recent on master Sep 15 15:59:23 imap-be6 cyrus/sync_client[22370]: SYNCNOTICE: master uid:2614 modseq:5636 last_updated:1600191010 internaldate:1599245383 flags:(\Seen) From the previous day's log I see: Sep 15 06:25:11 imap-be6 cyrus/sync_client[29665]: MAILBOX received NO response: System I/O error Sep 15 06:25:11 imap-be6 cyrus/sync_client[29665]: do_folders(): update failed: user.chauhs2 'The remote Server(s) denied the operation' Sep 15 06:25:11 imap-be6 cyrus/sync_client[29665]: seen_db: user swankd opened /var/lib/cyrus/user/s/swankd.seen Sep 15 06:25:11 imap-be6 cyrus/sync_client[29665]: seen_db: user edena2 opened /var/lib/cyrus/user/e/edena2.seen Sep 15 06:25:11 imap-be6 cyrus/sync_client[29665]: seen_db: user searsm2 opened /var/lib/cyrus/user/s/searsm2.seen Sep 15 06:25:11 imap-be6 cyrus/sync_client[29665]: USER received NO response: System I/O error Sep 15 06:25:11 imap-be6 cyrus/sync_client[29665]: Error in do_sync(): bailing out! The remote Server(s) denied the operation Sep 15 06:25:11 imap-be6 cyrus/sync_client[29665]: Processing sync log file /var/lib/cyrus/sync/log-29665 failed: The remote Server(s) denied the operation Sep 15 06:25:11 imap-be6 cyrus/sync_client[29665]: Reprocessing sync log file /var/lib/cyrus/sync/log-29665 I have not seen a repeat of the DBERROR, and have since restarted cyrus services. So I'm not sure a ctl_cyrusdb -r is needed, or even it it would fix things. -- -- Michael D. sofkasof...@rpi.edu ITI Software Architect, Email, TeX, Epistemology Rensselaer Polytechnic Institute, Troy, NY.http://www.rpi.edu/~sofkam/ Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Sync failing. Possible db corruption
Okay, these errors may be admin error. In running the old logs I specified -u, instead of -m. I am still concerned about the DBERROR, but those did not appear after I restarted cyrus. And rolling replication is still not working. Mike On 9/15/20 10:49 AM, Michael Sofka wrote: Sep 15 10:21:18 imap-be6 cyrus/sync_client[19816]: Inbox missing on master for MAILBOX user.travi t Sep 15 10:21:18 imap-be6 cyrus/sync_client[19816]: Inbox missing on master for MAILBOX user.sutto e Sep 15 10:21:18 imap-be6 cyrus/sync_client[19816]: Inbox missing on master for MAILBOX user.maran v On the replication server I have: Sep 15 10:34:40 imap-be5 cyrus/syncserver[31550]: cannot unlink /var/lib/cyrus/user/m/MAILBOX use r.linr7.mboxkey: No such file or directory Sep 15 10:34:40 imap-be5 cyrus/syncserver[31550]: cannot unlink /var/lib/cyrus/user/m/MAILBOX use r.kjells.mboxkey: No such file or directory Sep 15 10:34:40 imap-be5 cyrus/syncserver[31550]: cannot unlink /var/lib/cyrus/user/m/MAILBOX use r.ngang.mboxkey: No such file or directory When I try to process the sync/log- files. Any pointers or help appreciated. IMAP is functioning as far as I can tell, and only the replication process is failing. Thank you, MIke -- -- Michael D. Sofka sof...@rpi.edu ITI Software Architect, Email, TeX, Epistemology Rensselaer Polytechnic Institute, Troy, NY. http://www.rpi.edu/~sofkam/ Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Sync failing. Possible db corruption
Before I proceed, I am seeking advice. This is on Cyrus IMAP Murder v2.4.18-Debian-2.4.18-3 On one of our back-end servers sync started failing. I restarted sync_server on the replication, and on the backend server, which did not work. There are the following errors in the logs Sep 15 10:04:52 imap-be6 cyrus/sync_client[29665]: Processing sync log file /var/lib/cyrus/sync/l og-29665 failed: Bad protocol Sep 15 10:04:52 imap-be6 cyrus/sync_client[29665]: DBERROR db5: BDB1581 File handles still open a t environment close Sep 15 10:04:52 imap-be6 cyrus/sync_client[29665]: DBERROR db5: BDB1582 Open file handle: /var/li b/cyrus/db/__db.001 Sep 15 10:04:52 imap-be6 cyrus/sync_client[29665]: DBERROR db5: BDB1582 Open file handle: /var/li b/cyrus/db/__db.002 Sep 15 10:04:52 imap-be6 cyrus/sync_client[29665]: DBERROR db5: BDB1582 Open file handle: /var/li b/cyrus/db/__db.003 Sep 15 10:04:52 imap-be6 cyrus/sync_client[29665]: DBERROR db5: BDB0060 PANIC: fatal region error detected; run recovery Sep 15 10:04:52 imap-be6 cyrus/sync_client[29665]: DBERROR: critical database situation ... Sep 15 10:21:18 imap-be6 cyrus/sync_client[19816]: Inbox missing on master for MAILBOX user.travi t Sep 15 10:21:18 imap-be6 cyrus/sync_client[19816]: Inbox missing on master for MAILBOX user.sutto e Sep 15 10:21:18 imap-be6 cyrus/sync_client[19816]: Inbox missing on master for MAILBOX user.maran v On the replication server I have: Sep 15 10:34:40 imap-be5 cyrus/syncserver[31550]: cannot unlink /var/lib/cyrus/user/m/MAILBOX use r.linr7.mboxkey: No such file or directory Sep 15 10:34:40 imap-be5 cyrus/syncserver[31550]: cannot unlink /var/lib/cyrus/user/m/MAILBOX use r.kjells.mboxkey: No such file or directory Sep 15 10:34:40 imap-be5 cyrus/syncserver[31550]: cannot unlink /var/lib/cyrus/user/m/MAILBOX use r.ngang.mboxkey: No such file or directory When I try to process the sync/log- files. Any pointers or help appreciated. IMAP is functioning as far as I can tell, and only the replication process is failing. Thank you, MIke -- -- Michael D. Sofka sof...@rpi.edu ITI Software Architect, Email, TeX, Epistemology Rensselaer Polytechnic Institute, Troy, NY. http://www.rpi.edu/~sofkam/ Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: seen corruption
Thank you for your reply, Question though.. If I rename the file do I still need to update db/skpstamp with ctl_cyrusdb ? Or is the rename enough? On 06/17/2011 11:10 AM, Zbierski Christophe wrote: Hello, if your user.seen is corrupted, you have 2 solutions : - rm/rename your file - update db/skipstamp file (with ctl_cyrusdb). This solution can compact seen files. reconstruct command is useless, it does not change seen files Christophe Z. -Message d'origine- De : info-cyrus-bounces+christophe.zbierski=atosorigin@lists.andrew.cmu.edu [mailto:info-cyrus-bounces+christophe.zbierski=atosorigin@lists.andrew.cmu.edu] De la part de ssureshot Envoyé : vendredi 17 juin 2011 16:38 À : info-cyrus@lists.andrew.cmu.edu Objet : seen corruption I have a user that has had a corrupted seen file two times this week. The process I have done to fix is simply kill user process's and rename the user.seen file then have the log back in and all starts working.. My question, is this all I need to do? Do I need to run a reconstruct on their mailbox? Any special steps? Any input on what could possibly be causing this? Corrupt email client? Thank you Aaron Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ Ce message et les pièces jointes sont confidentiels et réservés à l'usage exclusif de ses destinataires. Il peut également être protégé par le secret professionnel. Si vous recevez ce message par erreur, merci d'en avertir immédiatement l'expéditeur et de le détruire. L'intégrité du message ne pouvant être assurée sur Internet, la responsabilité du groupe Atos Origin ne pourra être recherchée quant au contenu de ce message. Bien que les meilleurs efforts soient faits pour maintenir cette transmission exempte de tout virus, l'expéditeur ne donne aucune garantie à cet égard et sa responsabilité ne saurait être recherchée pour tout dommage résultant d'un virus transmis. This e-mail and the documents attached are confidential and intended solely for the addressee; it may also be privileged. If you receive this e-mail in error, please notify the sender immediately and destroy it. As its integrity cannot be secured on the Internet, the Atos Origin group liability cannot be triggered for the message content. Although the sender endeavours to maintain a computer virus-free network, the sender does not warrant that this transmission is virus-free and will not be liable for any damages resulting from any virus transmitted. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
RE: seen corruption
The rm/rename is enough. -Message d'origine- De : ssureshot [mailto:ssures...@gmail.com] Envoyé : vendredi 17 juin 2011 17:18 À : Zbierski Christophe Cc : info-cyrus@lists.andrew.cmu.edu Objet : Re: seen corruption Thank you for your reply, Question though.. If I rename the file do I still need to update db/skpstamp with ctl_cyrusdb ? Or is the rename enough? On 06/17/2011 11:10 AM, Zbierski Christophe wrote: Hello, if your user.seen is corrupted, you have 2 solutions : - rm/rename your file - update db/skipstamp file (with ctl_cyrusdb). This solution can compact seen files. reconstruct command is useless, it does not change seen files Christophe Z. -Message d'origine- De : info-cyrus-bounces+christophe.zbierski=atosorigin@lists.andrew.cmu .edu [mailto:info-cyrus-bounces+christophe.zbierski=atosorigin@lists.an drew.cmu.edu] De la part de ssureshot Envoyé : vendredi 17 juin 2011 16:38 À : info-cyrus@lists.andrew.cmu.edu Objet : seen corruption I have a user that has had a corrupted seen file two times this week. The process I have done to fix is simply kill user process's and rename the user.seen file then have the log back in and all starts working.. My question, is this all I need to do? Do I need to run a reconstruct on their mailbox? Any special steps? Any input on what could possibly be causing this? Corrupt email client? Thank you Aaron Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ Ce message et les pièces jointes sont confidentiels et réservés à l'usage exclusif de ses destinataires. Il peut également être protégé par le secret professionnel. Si vous recevez ce message par erreur, merci d'en avertir immédiatement l'expéditeur et de le détruire. L'intégrité du message ne pouvant être assurée sur Internet, la responsabilité du groupe Atos Origin ne pourra être recherchée quant au contenu de ce message. Bien que les meilleurs efforts soient faits pour maintenir cette transmission exempte de tout virus, l'expéditeur ne donne aucune garantie à cet égard et sa responsabilité ne saurait être recherchée pour tout dommage résultant d'un virus transmis. This e-mail and the documents attached are confidential and intended solely for the addressee; it may also be privileged. If you receive this e-mail in error, please notify the sender immediately and destroy it. As its integrity cannot be secured on the Internet, the Atos Origin group liability cannot be triggered for the message content. Although the sender endeavours to maintain a computer virus-free network, the sender does not warrant that this transmission is virus-free and will not be liable for any damages resulting from any virus transmitted. Ce message et les pièces jointes sont confidentiels et réservés à l'usage exclusif de ses destinataires. Il peut également être protégé par le secret professionnel. Si vous recevez ce message par erreur, merci d'en avertir immédiatement l'expéditeur et de le détruire. L'intégrité du message ne pouvant être assurée sur Internet, la responsabilité du groupe Atos Origin ne pourra être recherchée quant au contenu de ce message. Bien que les meilleurs efforts soient faits pour maintenir cette transmission exempte de tout virus, l'expéditeur ne donne aucune garantie à cet égard et sa responsabilité ne saurait être recherchée pour tout dommage résultant d'un virus transmis. This e-mail and the documents attached are confidential and intended solely for the addressee; it may also be privileged. If you receive this e-mail in error, please notify the sender immediately and destroy it. As its integrity cannot be secured on the Internet, the Atos Origin group liability cannot be triggered for the message content. Although the sender endeavours to maintain a computer virus-free network, the sender does not warrant that this transmission is virus-free and will not be liable for any damages resulting from any virus transmitted. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: seen corruption
On Fri, 17 Jun 2011 10:38:07 -0400 ssureshot ssures...@gmail.com wrote: I have a user that has had a corrupted seen file two times this week. The process I have done to fix is simply kill user process's and rename the user.seen file then have the log back in and all starts working.. My question, is this all I need to do? Do I need to run a reconstruct on their mailbox? Any special steps? For those not following the IRC channel - (since we saw you on there!) Cyrus before 2.3.12 has bugs in the skiplist implementation which mean that concurrent access can corrupt any skiplist file. The recommended solution is of course to upgrade to a newer version, but if you're willing to recompile Cyrus you can probably just copy lib/cyrusdb_skiplist.c from a newer version. And in cyrus 2.4.x, the user's own seen data is stored in cyrus.index anwyay, so the seen files get a lot less IO (only shared folders) plus the fixed skiplist library! Bron. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: reconstruct caused mailboxes (skiplist) corruption?
On Fri, Nov 12, 2010 at 08:41:50AM +0100, Per olof Ljungmark wrote: File a bug with FreeBSD? http://www.freebsd.org/support/bugreports.html Already did. http://www.freebsd.org/cgi/query-pr.cgi?pr=152151 Bron. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: reconstruct caused mailboxes (skiplist) corruption?
On Fri, 12 Nov 2010, Bron Gondwana wrote:
On Thu, Nov 11, 2010 at 11:58:04PM -0200, Henrique de Moraes Holschuh wrote:
It _will_ write to stderr (aka fd 2). If we want to be safe, we make sure
fds 0-2 are sane, and we check when we open sockets/files that we did not
get fds below 3...
Bron ( a while later, fd 2 gets re-used as the mailboxes.db handle, and
hence
the mess is created )
Indeed.
We *CANNOT* afford to have any files or sockets opened with fd 0, 1 or 2. We
should core-dump immediately if that happens, I think.
How about this skanky patch (attached?) - checks fds at the start, and if it
gets fd 2, it holds it open (to /dev/null) for the life of the process, making
sure nothing else gets it. If it gets 0 or 1 it just croaks.
IMHO we should be even more paranoid. Create a xfopen() that we use
anywhere where we are not explicitly dealing with fd 0, 1 or 2. If xfopen()
detects the problem, it should dump info to syslog priority error and
request that the report is sent to us. And, of course, return -1 (or, if
you prefer, heal the fds).
That should help us find out why sometimes one of those fds just disappear.
I don't think it is just a case of truss doing something idiotic, we have
had spurious (and _rare_) problem reports where it clearly happened over the
years...
Maybe the truss issue is FreeBSD breakage, but the weirdness has happened on
Linux as well in the past.
From 5a6433511db0002227aad069ee9e92c34932879a Mon Sep 17 00:00:00 2001
From: Bron Gondwana br...@opera.com
Date: Fri, 12 Nov 2010 14:05:02 +1100
Subject: [PATCH] Protect STDERR on FreeBSD
---
lib/libcyr_cfg.c | 20
1 files changed, 20 insertions(+), 0 deletions(-)
diff --git a/lib/libcyr_cfg.c b/lib/libcyr_cfg.c
index 83a376c..d8c6986 100644
--- a/lib/libcyr_cfg.c
+++ b/lib/libcyr_cfg.c
@@ -59,6 +59,8 @@
#define CFGVAL(t,v) {(void *)(v)}
#endif
+static int protect_stderr = -1;
+
struct cyrusopt_s cyrus_options[] = {
{ CYRUSOPT_ZERO, { NULL }, CYRUS_OPT_NOTOPT },
@@ -221,10 +223,28 @@ void libcyrus_config_setswitch(enum cyrus_opt opt, int
val)
void libcyrus_init()
{
+protect_stderr = open(/dev/null, O_RDWR, 0666);
+if (protect_stderr 2) {
+ /* Ok, we're safe */
+ close(protect_stderr);
+ protect_stderr = -1;
+}
+else if (protect_stderr == 2) {
+ syslog(LOG_ERR, WARNING: Protecting stderr from dangerous re-open.
+ Are you running under broken truss on FreeBSD?);
+}
+else {
+ abort();
+}
+
cyrusdb_init();
}
void libcyrus_done()
{
cyrusdb_done();
+if (protect_stderr -1) {
+ close(protect_stderr);
+ protect_stderr = -1;
+}
}
--
One disk to rule them all, One disk to find them. One disk to bring
them all and in the darkness grind them. In the Land of Redmond
where the shadows lie. -- The Silicon Valley Tarot
Henrique Holschuh
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: reconstruct caused mailboxes (skiplist) corruption?
We saw something similar: syslog() messages 'on the wire' (imap, pop3, etcetera) when We've restarted syslog on an in-production cyrus backend. In summary, DONT DO IT (syslog stop) with cyrus runing. On 11/11/2010 07:54 PM, Bron Gondwana wrote: On Thu, Nov 11, 2010 at 02:24:47PM -0200, Henrique de Moraes Holschuh wrote: On Thu, 11 Nov 2010, Paul Dekkers wrote: Uhoh! And then I looked at mailboxes.db: It looks like part completely rewritten, including the skiplist header, and the first line now said: user.bla: System I/O error System I/O error This is something that has plagued cyrus for a long time. Can we find a way to actually keep tabs on our FDs so it cannot ever happen again, please? I recall reports of crap showing inside prot streams 10 years ago... if now it is leaking into even worse places, well... It's a standalone program. Reconstruct was running all by itself. This probably needs a redesign of master/service fd-passing protocol, and of prot streams to be fixed for good. While at it, we should switch the master/service interaction to a modern design, since the operating system worth bothering with nowadays deal sanely with the thundering herd effect, and all of them have proper socket event support (epoll-like. Would require one of the event abstraction libraries, though, so as to support linux/bsd/solaris with minimum fuss). Since that wasn't the issue - why on earth was it allowed to have fd 2 in the first place? Is Cyrus closing fd 2, or is truss closing it?? There was no issue outside truss, it was when it ran under truss that the issue happened. Here's the start of an strace of a reconstruct run on my machine: execve(/usr/cyrus/bin/reconstruct, [/usr/cyrus/bin/reconstruct, -C, /tmp/ct-slot2/etc/imapd.conf, -s], [/* 20 vars */]) = 0 brk(0) = 0x12f1000 access(/etc/ld.so.nohwcap, F_OK) = -1 ENOENT (No such file or directory) mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fceb52d8000 access(/etc/ld.so.preload, R_OK) = -1 ENOENT (No such file or directory) open(db-4.6/lib/tls/x86_64/libsasl2.so.2, O_RDONLY) = -1 ENOENT (No such file or directory) open(db-4.6/lib/tls/libsasl2.so.2, O_RDONLY) = -1 ENOENT (No such file or directory) open(db-4.6/lib/x86_64/libsasl2.so.2, O_RDONLY) = -1 ENOENT (No such file or directory) open(db-4.6/lib/libsasl2.so.2, O_RDONLY) = -1 ENOENT (No such file or directory) open(/etc/ld.so.cache, O_RDONLY) = 3 Notice the first fd allocated: 3. And here's a run under truss on FreeBSD: [r...@cyrus1 /var/imap]# sudo -u cyrus truss /usr/local/cyrus/bin/reconstruct user.foo __sysctl(0x7fffe390,0x2,0x7fffe3ac,0x7fffe3a0,0x0,0x0) = 0 (0x0) mmap(0x0,672,PROT_READ|PROT_WRITE,MAP_ANON,-1,0x0) = 34366398464 (0x80065a000) munmap(0x80065a000,672)= 0 (0x0) __sysctl(0x7fffe400,0x2,0x800763428,0x7fffe3f8,0x0,0x0) = 0 (0x0) mmap(0x0,32768,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON,-1,0x0) = 34366398464 (0x80065a000) issetugid(0x80065b015,0x800654cc4,0x80076fc50,0x80076fc20,0x6351,0x0) = 0 (0x0) open(/etc/libmap.conf,O_RDONLY,0666) ERR#2 'No such file or directory' access(/usr/lib/libsasl2.so.2,0) ERR#2 'No such file or directory' access(/usr/local/lib/libsasl2.so.2,0) = 0 (0x0) open(/usr/local/lib/libsasl2.so.2,O_RDONLY,035431400) = 2 (0x2) Note the first fd allocated: 2! The question is - why is fd 2 being allocated? Is it necessary to explicitly open stderr? The function that's scribbling all over everything is com_err, which is supposed to be a BSD error reporting library, it SHOULD know what it's doing... Bron ( a while later, fd 2 gets re-used as the mailboxes.db handle, and hence the mess is created ) Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: reconstruct caused mailboxes (skiplist) corruption?
On Fri, Nov 12, 2010 at 05:10:20PM -0200, Sergio Bruder wrote: We saw something similar: syslog() messages 'on the wire' (imap, pop3, etcetera) when We've restarted syslog on an in-production cyrus backend. In summary, DONT DO IT (syslog stop) with cyrus runing. Ooh, that's interesting actually. How old is your data? What platform? Can you please create a bugzilla entry for it, particularly if you can reproduce! Thanks, Bron ( actually using Bugzilla now! ) Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
reconstruct caused mailboxes (skiplist) corruption?
Hi, Maybe I've some more 2.4.3 badness: I just decided to restore one message, and copied it from the archive. Unfortunately, I didn't copy properly, so the ownership was root instead of cyrus. I then ran a # sudo -u cyrus /usr/local/cyrus/bin/reconstruct user.bla user.bla: System I/O error System I/O error Hmm, allright, so I ran it with a truss (like strace for FreeBSD) to give me a bit more verbosity, and I realized I should chown. But then: # chown cyrus 22003. # sudo -u cyrus /usr/local/cyrus/bin/reconstruct user.bla fatal error: can't read mailboxes file Ehm, that's bad. # sudo -u cyrus /usr/local/cyrus/bin/ctl_mboxlist -d fatal error: can't read mailboxes file Uhoh! And then I looked at mailboxes.db: It looks like part completely rewritten, including the skiplist header, and the first line now said: user.bla: System I/O error System I/O error Oops? Fortunately I had a recent copy, I just decided to make a tarball of /var/imap after realizing I probably lost (a recent backup of) seen state (related to other thread) because /var/imap is not on ZFS :-S Regards, Paul Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: reconstruct caused mailboxes (skiplist) corruption?
On Thu, 11 Nov 2010, Paul Dekkers wrote: Uhoh! And then I looked at mailboxes.db: It looks like part completely rewritten, including the skiplist header, and the first line now said: user.bla: System I/O error System I/O error This is something that has plagued cyrus for a long time. Can we find a way to actually keep tabs on our FDs so it cannot ever happen again, please? I recall reports of crap showing inside prot streams 10 years ago... if now it is leaking into even worse places, well... This probably needs a redesign of master/service fd-passing protocol, and of prot streams to be fixed for good. While at it, we should switch the master/service interaction to a modern design, since the operating system worth bothering with nowadays deal sanely with the thundering herd effect, and all of them have proper socket event support (epoll-like. Would require one of the event abstraction libraries, though, so as to support linux/bsd/solaris with minimum fuss). -- One disk to rule them all, One disk to find them. One disk to bring them all and in the darkness grind them. In the Land of Redmond where the shadows lie. -- The Silicon Valley Tarot Henrique Holschuh Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: reconstruct caused mailboxes (skiplist) corruption?
On Thu, Nov 11, 2010 at 02:24:47PM -0200, Henrique de Moraes Holschuh wrote:
On Thu, 11 Nov 2010, Paul Dekkers wrote:
Uhoh! And then I looked at mailboxes.db: It looks like part completely
rewritten, including the skiplist header, and the first line now said:
user.bla: System I/O error System I/O error
This is something that has plagued cyrus for a long time. Can we find a
way to actually keep tabs on our FDs so it cannot ever happen again,
please? I recall reports of crap showing inside prot streams 10 years
ago... if now it is leaking into even worse places, well...
Here's the ktrace/kdump output on FreeBSD:
45426 reconstruct CALL access(0x80065d000,F_OK)
45426 reconstruct NAMI /usr/local/lib/libsasl2.so.2
45426 reconstruct RET access 0
45426 reconstruct CALL open(0x80065e000,O_RDONLY,unused0x763300)
45426 reconstruct NAMI /usr/local/lib/libsasl2.so.2
45426 reconstruct RET open 3
45426 reconstruct CALL fstat(0x3,0x7fffe350)
45426 reconstruct STRU struct stat {dev=87, ino=1084119, mode=-rwxr-xr-x ,
nlink=1, uid=0, gid=0, rdev=4338624, atime=1289519226, stime=1276299162,
ctime=1289480449, birthtime=1276299162, size=114591, blksize=16384, blocks=224,
flags=0x0 }
45426 reconstruct RET fstat 0
45426 reconstruct CALL pread(0x3,0x8007622e0,0x1000,0)
45426 reconstruct GIO fd 3 read 4096 bytes
What do you know fd 3. It's almost certainly truss doing something
stupid.
We maybe be able to work around it in Cyrus - and that might actually be
worth it - but I don't think it's Cyrus' fault.
Bron.
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: reconstruct caused mailboxes (skiplist) corruption?
On Thu, Nov 11, 2010 at 10:12:31AM +0100, Paul Dekkers wrote: Hmm, allright, so I ran it with a truss (like strace for FreeBSD) to give me a bit more verbosity, and I realized I should chown. But then: # chown cyrus 22003. # sudo -u cyrus /usr/local/cyrus/bin/reconstruct user.bla fatal error: can't read mailboxes file Confirmed: bug in FreeBSD's truss. If you must truss, always run it with a specified output file, which should avoid the bug. Otherwise a random file (mailboxes.db in this case) will get random stderr junk written to it! Bron. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: reconstruct caused mailboxes (skiplist) corruption?
Sorry - I've been busy working on the specific problem rather than the overview, and I realised I kind of glossed over this bit: On Thu, Nov 11, 2010 at 02:24:47PM -0200, Henrique de Moraes Holschuh wrote: This probably needs a redesign of master/service fd-passing protocol, and of prot streams to be fixed for good. While at it, we should switch the master/service interaction to a modern design, since the operating system worth bothering with nowadays deal sanely with the thundering herd effect, and all of them have proper socket event support (epoll-like. Would require one of the event abstraction libraries, though, so as to support linux/bsd/solaris with minimum fuss). Certainly worth considering. I won't have the time to work on it for while since what we have now works fine for us. I'll be focussing my work on new features pretty soon, once 2.4.x is stable enough that I can trust that it will be reliable for people! But if you want to look at it and come up with something better for 2.5 or even further ahead, that would be fantastic. There's certainly plenty of parts of Cyrus that could do with some modernising! Thanks, Bron. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: reconstruct caused mailboxes (skiplist) corruption?
On Fri, Nov 12, 2010 at 10:33:15AM +1100, Bron Gondwana wrote: Sorry - I've been busy working on the specific problem rather than the overview, and I realised I kind of glossed over this bit: On Thu, Nov 11, 2010 at 02:24:47PM -0200, Henrique de Moraes Holschuh wrote: This probably needs a redesign of master/service fd-passing protocol, and of prot streams to be fixed for good. While at it, we should switch the master/service interaction to a modern design, since the operating system worth bothering with nowadays deal sanely with the thundering herd effect, and all of them have proper socket event support (epoll-like. Would require one of the event abstraction libraries, though, so as to support linux/bsd/solaris with minimum fuss). Certainly worth considering. I won't have the time to work on it for while since what we have now works fine for us. I'll be focussing my work on new features pretty soon, once 2.4.x is stable enough that I can trust that it will be reliable for people! But if you want to look at it and come up with something better for 2.5 or even further ahead, that would be fantastic. There's certainly plenty of parts of Cyrus that could do with some modernising! Isn't the modern design multiple threads, rather than multiple processes? That seems to me to be the right direction for Cyrus. It might even make for a simpler design. -- -Gary Mills--Unix Group--Computer and Network Services- Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: reconstruct caused mailboxes (skiplist) corruption?
On Fri, 12 Nov 2010, Bron Gondwana wrote: Since that wasn't the issue - why on earth was it allowed to have fd 2 in the first place? Is Cyrus closing fd 2, or is truss closing it?? That is the issue that caused the leaks into protstreams, AFAIK. It is always com-err writing to fd 2, and something unexpected being on fd 2. open stderr? The function that's scribbling all over everything is com_err, which is supposed to be a BSD error reporting library, it SHOULD know what it's doing... It _will_ write to stderr (aka fd 2). If we want to be safe, we make sure fds 0-2 are sane, and we check when we open sockets/files that we did not get fds below 3... Bron ( a while later, fd 2 gets re-used as the mailboxes.db handle, and hence the mess is created ) Indeed. We *CANNOT* afford to have any files or sockets opened with fd 0, 1 or 2. We should core-dump immediately if that happens, I think. -- One disk to rule them all, One disk to find them. One disk to bring them all and in the darkness grind them. In the Land of Redmond where the shadows lie. -- The Silicon Valley Tarot Henrique Holschuh Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: reconstruct caused mailboxes (skiplist) corruption?
On Thu, 11 Nov 2010, Gary Mills wrote: Isn't the modern design multiple threads, rather than multiple processes? That seems to me to be the right direction for Cyrus. It might even make for a simpler design. Ehh... not realy. Multithreading means locking, futexes, and other pains. It also means almost rewriting master, etc. It means VERY painful debugging. Modern design just means high-performance event dispatching. Cyrus can already do pre-fork, so it just needs an update on the connection handling, you don't need to go multithread. Besides, fork()-based designs are much more resilient. -- One disk to rule them all, One disk to find them. One disk to bring them all and in the darkness grind them. In the Land of Redmond where the shadows lie. -- The Silicon Valley Tarot Henrique Holschuh Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: reconstruct caused mailboxes (skiplist) corruption?
On Thu, Nov 11, 2010 at 11:58:04PM -0200, Henrique de Moraes Holschuh wrote:
It _will_ write to stderr (aka fd 2). If we want to be safe, we make sure
fds 0-2 are sane, and we check when we open sockets/files that we did not
get fds below 3...
Bron ( a while later, fd 2 gets re-used as the mailboxes.db handle, and
hence
the mess is created )
Indeed.
We *CANNOT* afford to have any files or sockets opened with fd 0, 1 or 2. We
should core-dump immediately if that happens, I think.
How about this skanky patch (attached?) - checks fds at the start, and if it
gets fd 2, it holds it open (to /dev/null) for the life of the process, making
sure nothing else gets it. If it gets 0 or 1 it just croaks.
Bron.
From 5a6433511db0002227aad069ee9e92c34932879a Mon Sep 17 00:00:00 2001
From: Bron Gondwana br...@opera.com
Date: Fri, 12 Nov 2010 14:05:02 +1100
Subject: [PATCH] Protect STDERR on FreeBSD
---
lib/libcyr_cfg.c | 20
1 files changed, 20 insertions(+), 0 deletions(-)
diff --git a/lib/libcyr_cfg.c b/lib/libcyr_cfg.c
index 83a376c..d8c6986 100644
--- a/lib/libcyr_cfg.c
+++ b/lib/libcyr_cfg.c
@@ -59,6 +59,8 @@
#define CFGVAL(t,v) {(void *)(v)}
#endif
+static int protect_stderr = -1;
+
struct cyrusopt_s cyrus_options[] = {
{ CYRUSOPT_ZERO, { NULL }, CYRUS_OPT_NOTOPT },
@@ -221,10 +223,28 @@ void libcyrus_config_setswitch(enum cyrus_opt opt, int val)
void libcyrus_init()
{
+protect_stderr = open(/dev/null, O_RDWR, 0666);
+if (protect_stderr 2) {
+ /* Ok, we're safe */
+ close(protect_stderr);
+ protect_stderr = -1;
+}
+else if (protect_stderr == 2) {
+ syslog(LOG_ERR, WARNING: Protecting stderr from dangerous re-open.
+ Are you running under broken truss on FreeBSD?);
+}
+else {
+ abort();
+}
+
cyrusdb_init();
}
void libcyrus_done()
{
cyrusdb_done();
+if (protect_stderr -1) {
+ close(protect_stderr);
+ protect_stderr = -1;
+}
}
--
1.7.2.3
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: reconstruct caused mailboxes (skiplist) corruption?
On 11/12/10 00:06, Bron Gondwana wrote: On Thu, Nov 11, 2010 at 10:12:31AM +0100, Paul Dekkers wrote: Hmm, allright, so I ran it with a truss (like strace for FreeBSD) to give me a bit more verbosity, and I realized I should chown. But then: # chown cyrus 22003. # sudo -u cyrus /usr/local/cyrus/bin/reconstruct user.bla fatal error: can't read mailboxes file Confirmed: bug in FreeBSD's truss. If you must truss, always run it with a specified output file, which should avoid the bug. Otherwise a random file (mailboxes.db in this case) will get random stderr junk written to it! File a bug with FreeBSD? http://www.freebsd.org/support/bugreports.html Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Corruption to quotas.db (skiplist)
Hi guys, I touched on this in a recent topic about XFER however we have had a few more problems with the quotas database and it is quite worrying. We are transferring our mailboxes from our old Cyrus 2.2 IMAP server to a Cyrus 2.3 IMAP server with replication. We have 25,000 mailboxes totalling around 1.2 TB. Trying to move mailboxes in parallel ended up with serious corruption to quotas.db. We saw lots of these lines: Jan 23 04:06:47 sauber.bath.ac.uk imap[4434]: [ID 602473 mail.error] IOERROR: lock_shared /opt/etc/imapd/quotas.db: Bad file number Eventually resulting in *lots* of these lines: Jan 23 08:10:32 sauber.bath.ac.uk imap[4434]: [ID 362402 mail.error] skiplist: version mismatch: /opt/etc/imapd/quotas.db has version 2.1264205870 Jan 23 08:10:32 sauber.bath.ac.uk imap[4434]: [ID 558109 mail.error] skiplist: closed while still locked Jan 23 08:10:32 sauber.bath.ac.uk imap[4434]: [ID 729713 mail.error] DBERROR: opening /opt/etc/imapd/quotas.db: cyrusdb error Jan 23 08:10:32 sauber.bath.ac.uk imap[4434]: [ID 637875 mail.error] Fatal error: can't read quotas file Users couldn't log in, mail wasn't being delivered and I couldn't even run ctl_mboxlist -d. We had to regenerate the quotas DB from scratch and reconstruct some mailboxes that were suddenly using 3,000% of their quota! Since then we have been moving mailboxes one at a time. We transferred student mailboxes for two nights and this went fine with no errors, but when we transferred some staff mailboxes we started seeing the Bad file number errors again. We ran quota -f and fixed any corrupted quotas, and the Bad file number errors stopped appearing. Is there anything more we can do to protect ourselves from these errors? Is anyone else using skiplist as their quotas.db format? I note that quotalegacy is the default database format which is what our old Cyrus 2.2 IMAP server is using. I wondered whether problems were caused due to staff leaving their PCs switched on overnight however the logs do not show a correlation between quotas that were corrupted and mailboxes that were being checked overnight. Is the skiplist format suitably reliable for this database? It certainly seems to work OK for all the other databases. Regards, Dave. David Mayo Networks/Systems Administrator University of Bath Computing Services, UK Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Cyrus upgrades and then -- DB corruption
Folks, Our next upgrade fell victim to this as well. Even though cyrus freshly created its own database, it looks like db4 stepped on itself and more of the same DBERRORs. I moved and saved the corrupted database and let cyrus recreate it again and it now seems very happy. I'll be more than willing to share the broken databases and logs. Just let me know what you'd like to see. Jack -- Jack Neely jjne...@ncsu.edu Linux Czar, OIT Campus Linux Services Office of Information Technology, NC State University GPG Fingerprint: 1917 5AC1 E828 9337 7AA4 EA6B 213B 765F 3B6A 5B89 Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Cyrus upgrades and then -- DB corruption
Folks, We are continuing our quest to upgrade our Cyrus IMAP servers from RHEL 3 i386 with Cyrus version 2.2.12 to RHEL 5 x86_64 machines running cyrus version 2.3.14. I've had 2 cases now where the upgrade appeared to go smoothly. Cyrus comes up, logs look fine, lmtp mail is flowing in, users are happy, and the logs are perfect. For a few minutes until the alians corrupt the database. Suddenly we see stuff like: Dec 5 15:02:55 xx lmtp[4724]: DBERROR: error fetching 9311515.102504011260020246443.javamail.em-bu...@na-mm2-relay.amazon.com: DB_PAGE_NOTFOUND: Requested page not found Dec 5 15:02:55 x lmtp[4724]: DBERROR: mystore: error storing 9311515.102504011260020246443.javamail.em-bu...@na-mm2-relay.amazon.com: DB_PAGE_NOTFOUND: Requested page not found Dec 5 15:03:55 xx lmtp[4724]: DBERROR db4: page 0: illegal page type or format Dec 5 15:03:55 xx lmtp[4724]: DBERROR db4: PANIC: Invalid argument Dec 5 15:03:55 xx lmtp[4724]: DBERROR: critical database situation It goes downhill from there. LMTP doesn't deliever, the IMAP port stops responding. DBERRORs scroll. I could not dump the mailboxes.db with su cyrus -c /local/cyrus/sbin/ctl_mboxlist -d -f /imap/conf/mailboxes.db /tmp/mailboxes.corrupt Just more errors. I could not move mailboxes.db out of the way and load a known good dump back in its placemore dberrors. Finally, I moved the db/ directory and created an empty db/ as well as moved the deliever.db out of the way (some lmtp errors were citing that file.) I turned her back on and she's now up and running like nothing happened. Can someone help me understand what happened here and how to avoid it? Thanks, Jack Neely -- Jack Neely jjne...@ncsu.edu Linux Czar, OIT Campus Linux Services Office of Information Technology, NC State University GPG Fingerprint: 1917 5AC1 E828 9337 7AA4 EA6B 213B 765F 3B6A 5B89 Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Cyrus upgrades and then -- DB corruption
On 12/05/2009 05:15 PM, Jack Neely wrote: Folks, We are continuing our quest to upgrade our Cyrus IMAP servers from RHEL 3 i386 with Cyrus version 2.2.12 to RHEL 5 x86_64 machines running cyrus version 2.3.14. I've had 2 cases now where the upgrade appeared to go smoothly. Cyrus comes up, logs look fine, lmtp mail is flowing in, users are happy, and the logs are perfect. For a few minutes until the alians corrupt the database. Suddenly we see stuff like: Dec 5 15:02:55 xx lmtp[4724]: DBERROR: error fetching 9311515.102504011260020246443.javamail.em-bu...@na-mm2-relay.amazon.com: DB_PAGE_NOTFOUND: Requested page not found Dec 5 15:02:55 x lmtp[4724]: DBERROR: mystore: error storing 9311515.102504011260020246443.javamail.em-bu...@na-mm2-relay.amazon.com: DB_PAGE_NOTFOUND: Requested page not found Dec 5 15:03:55 xx lmtp[4724]: DBERROR db4: page 0: illegal page type or format Dec 5 15:03:55 xx lmtp[4724]: DBERROR db4: PANIC: Invalid argument Dec 5 15:03:55 xx lmtp[4724]: DBERROR: critical database situation It goes downhill from there. LMTP doesn't deliever, the IMAP port stops responding. DBERRORs scroll. I could not dump the mailboxes.db with su cyrus -c /local/cyrus/sbin/ctl_mboxlist -d -f /imap/conf/mailboxes.db /tmp/mailboxes.corrupt Just more errors. I could not move mailboxes.db out of the way and load a known good dump back in its placemore dberrors. Finally, I moved the db/ directory and created an empty db/ as well as moved the deliever.db out of the way (some lmtp errors were citing that file.) I turned her back on and she's now up and running like nothing happened. Can someone help me understand what happened here and how to avoid it? Remove /var/imap/db/* (keep the directory) and get rid of /var/imap/db.backup* directories before starting Cyrus for the first time after upgrading to 2.3.x . At least that is what I did when upgrading from Fedora Core 2 with 2.2.x Cyrus to CentOS 5.x with Cyrus 2.3.x to avoid those errors. Got them when testing the upgrade on a separate machine. Thanks, Jack Neely Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Cyrus upgrades and then -- DB corruption
On Sat, Dec 05, 2009 at 06:35:43PM -0400, Patrick Boutilier wrote: On 12/05/2009 05:15 PM, Jack Neely wrote: Folks, We are continuing our quest to upgrade our Cyrus IMAP servers from RHEL 3 i386 with Cyrus version 2.2.12 to RHEL 5 x86_64 machines running cyrus version 2.3.14. I've had 2 cases now where the upgrade appeared to go smoothly. Cyrus comes up, logs look fine, lmtp mail is flowing in, users are happy, and the logs are perfect. For a few minutes until the alians corrupt the database. Suddenly we see stuff like: Dec 5 15:02:55 xx lmtp[4724]: DBERROR: error fetching 9311515.102504011260020246443.javamail.em-bu...@na-mm2-relay.amazon.com: DB_PAGE_NOTFOUND: Requested page not found Dec 5 15:02:55 x lmtp[4724]: DBERROR: mystore: error storing 9311515.102504011260020246443.javamail.em-bu...@na-mm2-relay.amazon.com: DB_PAGE_NOTFOUND: Requested page not found Dec 5 15:03:55 xx lmtp[4724]: DBERROR db4: page 0: illegal page type or format Dec 5 15:03:55 xx lmtp[4724]: DBERROR db4: PANIC: Invalid argument Dec 5 15:03:55 xx lmtp[4724]: DBERROR: critical database situation It goes downhill from there. LMTP doesn't deliever, the IMAP port stops responding. DBERRORs scroll. I could not dump the mailboxes.db with su cyrus -c /local/cyrus/sbin/ctl_mboxlist -d -f /imap/conf/mailboxes.db /tmp/mailboxes.corrupt Just more errors. I could not move mailboxes.db out of the way and load a known good dump back in its placemore dberrors. Finally, I moved the db/ directory and created an empty db/ as well as moved the deliever.db out of the way (some lmtp errors were citing that file.) I turned her back on and she's now up and running like nothing happened. Can someone help me understand what happened here and how to avoid it? Remove /var/imap/db/* (keep the directory) and get rid of /var/imap/db.backup* directories before starting Cyrus for the first time after upgrading to 2.3.x . At least that is what I did when upgrading from Fedora Core 2 with 2.2.x Cyrus to CentOS 5.x with Cyrus 2.3.x to avoid those errors. Got them when testing the upgrade on a separate machine. That's just it...the old databases are never present. I dump mailboxes.db on the old server just like above, scp that to the new server on new storage, and use that to recreate a new mailboxes.db. None of the other databases are copied over in any way. Cyrus does seem to regenerate all the databases propperly...and most of the time it stays working without corruption. Most of the time. ;-) Thanks for your help! Jack Neely -- Jack Neely jjne...@ncsu.edu Linux Czar, OIT Campus Linux Services Office of Information Technology, NC State University GPG Fingerprint: 1917 5AC1 E828 9337 7AA4 EA6B 213B 765F 3B6A 5B89 Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Attachment corruption when downloading with Thunderbird...
Citerar Raymond T. Sundland raym...@sundland.com: I use Thunderbird exclusively with Cyrus and have never had issues with attachments. Am running cyrus 2.2, but I don't see why 2.3 would suddenly break that functionality. I'm running Thunderbird with Cyrus 2.3 and so far it's worked fine. Br, Ted Bron Gondwana wrote: On Mon, Apr 20, 2009 at 06:15:12PM +, Andy Fiddaman wrote: I'm running Cyrus IMAP 2.3.13 on Solaris (about to upgrade to 2.3.14) and a number of my users who use Thunderbird have reported frequent corruption of attachments. So, has anyone else had any reports of this behaviour or any reason to believe that Thunderbird does not work well with Cyrus? I'm going to enable telemetry for one of the users who has reported this and see if I can see anything relevant in the IMAP session; any suggestions of other places to look would be appreciated. I'd love to see the telemetry. Bron ( wondering if Thunderbird is fetching the encoded size and then fetching it decoded or something?? ) Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Attachment corruption when downloading with Thunderbird...
Hi, I'm running Cyrus IMAP 2.3.13 on Solaris (about to upgrade to 2.3.14) and a number of my users who use Thunderbird have reported frequent corruption of attachments. This is particularly visible for images where they are obviously truncated. The work-around they are using is to drag the message into another folder, read it then drag it back. The messages are never corrupt within Cyrus and look fine when viewed with other clients which suggests a bug within Thunderbird or an interaction issue. I found some threads on the Mozilla lists which suggests that this is a server side bug where the server reports the wrong size for message parts and the suggested workaround is to set mail.server.default.fetch_by_chunks to false within Thunderbird. (http://forums.mozillazine.org/viewtopic.php?f=29t=104) So, has anyone else had any reports of this behaviour or any reason to believe that Thunderbird does not work well with Cyrus? I'm going to enable telemetry for one of the users who has reported this and see if I can see anything relevant in the IMAP session; any suggestions of other places to look would be appreciated. Thanks, Andy Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Attachment corruption when downloading with Thunderbird...
On Mon, Apr 20, 2009 at 06:15:12PM +, Andy Fiddaman wrote: I'm running Cyrus IMAP 2.3.13 on Solaris (about to upgrade to 2.3.14) and a number of my users who use Thunderbird have reported frequent corruption of attachments. So, has anyone else had any reports of this behaviour or any reason to believe that Thunderbird does not work well with Cyrus? I'm going to enable telemetry for one of the users who has reported this and see if I can see anything relevant in the IMAP session; any suggestions of other places to look would be appreciated. I'd love to see the telemetry. Bron ( wondering if Thunderbird is fetching the encoded size and then fetching it decoded or something?? ) Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Attachment corruption when downloading with Thunderbird...
I use Thunderbird exclusively with Cyrus and have never had issues with attachments. Am running cyrus 2.2, but I don't see why 2.3 would suddenly break that functionality. Bron Gondwana wrote: On Mon, Apr 20, 2009 at 06:15:12PM +, Andy Fiddaman wrote: I'm running Cyrus IMAP 2.3.13 on Solaris (about to upgrade to 2.3.14) and a number of my users who use Thunderbird have reported frequent corruption of attachments. So, has anyone else had any reports of this behaviour or any reason to believe that Thunderbird does not work well with Cyrus? I'm going to enable telemetry for one of the users who has reported this and see if I can see anything relevant in the IMAP session; any suggestions of other places to look would be appreciated. I'd love to see the telemetry. Bron ( wondering if Thunderbird is fetching the encoded size and then fetching it decoded or something?? ) Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: murder setup - mailboxes.db corruption - trouble recovering with ctl_mboxlist
On Mon, 24 Nov 2008, Simon Matter wrote: I just wanted to follow up on this thread, rather than leaving it hanging. It seems there was a more serious issue, which ultimately lead to the failure we experienced, with our murder setup. Specifically our internal DNS servers, were having sporadic time-outs for Linux and Macintosh clients. After disabling ipv6 on both of our nameservers, it seemed the name resolution issue cleared up. The strange thing is that it has been working without any complications for the last 6 months. But nobody at Marshall can pinpoint anything that may have changed recently with regards to ipv6 networking on the internal network. What is even stranger is that the name resolution timeouts only seemed to occur on Linux and Macintosh clients, while having no harmful effects on Microsoft Windows servers and clients. I had a strange issue recently with name resolution (OT because it has nothing todo with cyrus-imapd). Resolvers on Linux and recent OSX seem to always try to query for IPv6 () records even if NO IPv6 is configured on the client (no, I think it only happens if the client program tries to resolve IPv6 like firefox in our case). The problem starts if the DNS servers in question don't reply correctly to those requests and reply with SERVFAIL for example. The resolver will then ask the next DNS server it find in resolv.conf and go on until no more servers to ask, which can take quite some time if you have 4 servers configured like in our case. In the end it turned out the company in question were running DNS loadbalancers and they simply replied with SERVFAIL for records. I've been told customers with Windows don't have that problem and I guess it's because their resolver doesn't ask for IPv6 adresses if no IPv6 is configured. I ran into an almost identical problem just yesterday with my silly home network router (D-Link DIR-655). The built-in DNS server in the router simply ignored queries. It didn't respond as all, even with SERVFAIL. Eventually the resolver library on Linux would timeout and try an A query instead. I ended up disabling the built-in DNS server and using the Comcast DNS servers directly. Anyways, I can definately understand why IPv6 might cause some issues... :) Andy Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: murder setup - mailboxes.db corruption - trouble recovering with ctl_mboxlist
I just wanted to follow up on this thread, rather than leaving it hanging. It seems there was a more serious issue, which ultimately lead to the failure we experienced, with our murder setup. Specifically our internal DNS servers, were having sporadic time-outs for Linux and Macintosh clients. After disabling ipv6 on both of our nameservers, it seemed the name resolution issue cleared up. The strange thing is that it has been working without any complications for the last 6 months. But nobody at Marshall can pinpoint anything that may have changed recently with regards to ipv6 networking on the internal network. What is even stranger is that the name resolution timeouts only seemed to occur on Linux and Macintosh clients, while having no harmful effects on Microsoft Windows servers and clients. I personally want to thank everyone who helped troubleshoot and resolve this issue. Wesley Craig at the University of Michigan, thank you for your input, steering me in the right direction, and clearing up misconceptions about perceived errors. Andrew Morgan at Oregon State University, thank you for the input on working around performance bottlenecks (due to a huge backlog of e-mail) during the re-synchronization to the front-ends. Also, a big thanks to Jeff Eaton at Carnegie Mellon, for his assistance in troubleshooting and his advice on what to do next. Andrew Morgan wrote: On Thu, 20 Nov 2008, Wolfe, Eric G wrote: We are using skiplist, I copied the mailboxes.db to frontends. If the frontends are updating, it is not apparent. I could not verify that either of them were synching from the master after the mupdate master synch. Which is why I copied them to the frontends to speed things up. I've tried this in the past for the same reasons, but it doesn't work. The only way I've been able to get the correct mailboxes.db on frontends is to let them sync with the mupdate master. And yes, this takes a while from scratch when you are using skiplist because writes are much slower with skiplist. Strange that it would just start causing problems now. We probably are seeing a cascading effect of failure, with the backlog though. Do the latest vanilla trees, have these patches included in them? The packages here: http://cyrusimap.web.cmu.edu/downloads.html#imap. I am somewhat reluctant to upgrade things in a fragile state. If these patches are included in latest releases, is 2.3.13 a fairly painless upgrade path from 2.2.12, or do we need to go with 2.2.13? I wouldn't upgrade until you can get a working system. Is there anything we can turn off in the cyrus.conf or imapd.conf, to work around this bottleneck? In other words, can we keep the MTA from knocking on the door for long enough to get everything running smoothly again? Yes, comment out lmtp in your cyrus.conf on the frontends. After you have successfully synced mailboxes.db, enable lmtp again and restart cyrus on the frontends. You probably should put some limits on lmtp as well. We use maxchild=50 here (3 frontends). Ok, because it sounds like a problem of connecting to the mupdate master port on 3905, to the unitiated. Hard to say, but take the steps above to simplify the problem. :) Andy -- Eric G. Wolfe, IT Associate, Sr. One John Marshall Drive Marshall University, Drinko Library 428k Huntington, WV 25755 Phone: 304.696.3428 Email: [EMAIL PROTECTED] Nobody's gonna believe that computers are intelligent until they start coming in late and lying about it. Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: murder setup - mailboxes.db corruption - trouble recovering with ctl_mboxlist
I just wanted to follow up on this thread, rather than leaving it hanging. It seems there was a more serious issue, which ultimately lead to the failure we experienced, with our murder setup. Specifically our internal DNS servers, were having sporadic time-outs for Linux and Macintosh clients. After disabling ipv6 on both of our nameservers, it seemed the name resolution issue cleared up. The strange thing is that it has been working without any complications for the last 6 months. But nobody at Marshall can pinpoint anything that may have changed recently with regards to ipv6 networking on the internal network. What is even stranger is that the name resolution timeouts only seemed to occur on Linux and Macintosh clients, while having no harmful effects on Microsoft Windows servers and clients. I had a strange issue recently with name resolution (OT because it has nothing todo with cyrus-imapd). Resolvers on Linux and recent OSX seem to always try to query for IPv6 () records even if NO IPv6 is configured on the client (no, I think it only happens if the client program tries to resolve IPv6 like firefox in our case). The problem starts if the DNS servers in question don't reply correctly to those requests and reply with SERVFAIL for example. The resolver will then ask the next DNS server it find in resolv.conf and go on until no more servers to ask, which can take quite some time if you have 4 servers configured like in our case. In the end it turned out the company in question were running DNS loadbalancers and they simply replied with SERVFAIL for records. I've been told customers with Windows don't have that problem and I guess it's because their resolver doesn't ask for IPv6 adresses if no IPv6 is configured. Simon I personally want to thank everyone who helped troubleshoot and resolve this issue. Wesley Craig at the University of Michigan, thank you for your input, steering me in the right direction, and clearing up misconceptions about perceived errors. Andrew Morgan at Oregon State University, thank you for the input on working around performance bottlenecks (due to a huge backlog of e-mail) during the re-synchronization to the front-ends. Also, a big thanks to Jeff Eaton at Carnegie Mellon, for his assistance in troubleshooting and his advice on what to do next. Andrew Morgan wrote: On Thu, 20 Nov 2008, Wolfe, Eric G wrote: We are using skiplist, I copied the mailboxes.db to frontends. If the frontends are updating, it is not apparent. I could not verify that either of them were synching from the master after the mupdate master synch. Which is why I copied them to the frontends to speed things up. I've tried this in the past for the same reasons, but it doesn't work. The only way I've been able to get the correct mailboxes.db on frontends is to let them sync with the mupdate master. And yes, this takes a while from scratch when you are using skiplist because writes are much slower with skiplist. Strange that it would just start causing problems now. We probably are seeing a cascading effect of failure, with the backlog though. Do the latest vanilla trees, have these patches included in them? The packages here: http://cyrusimap.web.cmu.edu/downloads.html#imap. I am somewhat reluctant to upgrade things in a fragile state. If these patches are included in latest releases, is 2.3.13 a fairly painless upgrade path from 2.2.12, or do we need to go with 2.2.13? I wouldn't upgrade until you can get a working system. Is there anything we can turn off in the cyrus.conf or imapd.conf, to work around this bottleneck? In other words, can we keep the MTA from knocking on the door for long enough to get everything running smoothly again? Yes, comment out lmtp in your cyrus.conf on the frontends. After you have successfully synced mailboxes.db, enable lmtp again and restart cyrus on the frontends. You probably should put some limits on lmtp as well. We use maxchild=50 here (3 frontends). Ok, because it sounds like a problem of connecting to the mupdate master port on 3905, to the unitiated. Hard to say, but take the steps above to simplify the problem. :) Andy -- Eric G. Wolfe, IT Associate, Sr. One John Marshall Drive Marshall University, Drinko Library 428k Huntington, WV 25755 Phone: 304.696.3428 Email: [EMAIL PROTECTED] Nobody's gonna believe that computers are intelligent until they start coming in late and lying about it. Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: murder setup - mailboxes.db corruption - trouble recovering with ctl_mboxlist
Just to follow-up on this issue. Found this: http://cyrusimap.web.cmu.edu/twiki/bin/view/Cyrus/CyrusMurderFailureModes First, I followed the Easy instructions, which was a bust. Next, I tried the Hard instructions. Four hours later the mupdate master finished synch'ing with the backends. I started up the front-ends, per the instructions. The front-ends failed to synch with the mupdate master. So in an effort to try something else. I figured if the mailboxes.db on the front-ends and the master are the same format, I could just shutdown the mupdate master; copy the mailboxes.db file over to the front-ends; and start everything up. This was also a bust. Oh, and I am getting these logs on the mupdate master. However, the number in fs.file-nr is nowhere near approaching fs.file-max. There are no ulimits on the 'cyrus' user. There was a maxfds=1024 parameter in /etc/imapd.conf. I tried restarting without this parameter, and it seemed I couldn't keep the master process running without it. If I restart the service, it will run fine for a while, but it eventually starts complaining again. So I tried quadrupling the maxfds value, and we'll see if that helps. imapd.conf (excerpt) mupdate cmd=/usr/lib64/cyrus-imapd/mupdate -m listen=3905 prefork=1 maxfds=1024 maillog (excerpt) Nov 20 07:18:54 mumailmaster mupdate[27227]: refused connection from mumailstore01 Nov 20 07:18:54 mumailmaster mupdate[27227]: warning: cannot open /etc/hosts.allow: Too many open files Additionally, I have double-checked all cyrus related service accounts and their associated passwords. Our mupdate service account is successfully authenticating on the mupdate master. I am getting a imap: kick_mupdate: can't connect to target: Connection refused on the front-ends. However, I can connect to port 3905 on the mupdate master. I have not noticed anything strange on the backends, in the logs or otherwise. I will follow-up, if I find out anything else. Again, if anyone can point us in the right direction, it would be very much appreciated. Eric G. Wolfe wrote: We have a RHEL4u7 on all 5 servers: 1 mupdate master: mumailmaster 2 backends: mumailstore01, mumailstore02 2 Postfix MTA/Cyrus proxy frontends: mumail01, mumail02 So I started getting this on my backends around 14:15 EST, at which time mail started getting deferred to the backends. I have 20,000+ per frontend deferred for delivery at the time of this e-mail. Nov 19 16:30:26 mumailstore01 ctl_cyrusdb[4672]: DBERROR db4: PANIC: Cannot allocate memory Nov 19 16:30:26 mumailstore01 ctl_cyrusdb[4672]: DBERROR: critical database situation Nov 19 16:30:26 mumailstore01 ctl_mboxlist[4673]: DBERROR db4: PANIC: fatal region error detected; run recovery Nov 19 16:30:26 mumailstore01 ctl_mboxlist[4673]: DBERROR: critical database situation Nov 19 16:30:26 mumailstore01 ctl_cyrusdb[4674]: DBERROR db4: PANIC: fatal region error detected; run recovery Nov 19 16:30:26 mumailstore01 ctl_cyrusdb[4674]: DBERROR: critical database situation I tried the following directions for recovery. http://asg.web.cmu.edu/archive/message.php?mailbox=archive.info-cyrussearchterm=skiplistmsg=32337. I made backup copies of all files deleted, renaming them $filename.corrupt. I did this on each server, recovered on the backends, and let it push the updates to the mupdate server. Manually recovered mailboxes.db on the frontends, as they did not seem to be getting updated. If I am going about this wrong, please someone point me in the right direction for documentation on murder disaster recovery. The following, while somewhat helpful, does not go into a great amount of detail: http://cyrusimap.web.cmu.edu/imapd/install-murder.html. So at this point my user agent says Unknown/Invalid partition. The partitions are correctly defined on the backend mail stores. A 'ctl_mboxlist -d' shows correct partitions, no matter which local mailboxes.db I attempt to dump. Furthermore, LMTP is still not delivering to the backends during this outage. Any helpful tips or pointers would be appreciated. Thanks, -- Eric G. Wolfe, IT Associate, Sr. One John Marshall Drive Marshall University, Drinko Library 428k Huntington, WV 25755 Phone: 304.696.3428 Email: [EMAIL PROTECTED] Who is General Failure and why is he reading my hard disk ? Microsoft spel chekar vor sail, worgs grate !! (By [EMAIL PROTECTED], Felix von Leitner) Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: murder setup - mailboxes.db corruption - trouble recovering with ctl_mboxlist
Which version of cyrus-imapd are you running? On 20 Nov 2008, at 01:23, Eric G. Wolfe wrote: I tried the following directions for recovery. http://asg.web.cmu.edu/archive/message.php?mailbox=archive.info- cyrussearchterm=skiplistmsg=32337. So you're attempting to convert from BerkeleyDB to skiplist? I made backup copies of all files deleted, renaming them $filename.corrupt. I did this on each server, recovered on the backends, and let it push the updates to the mupdate server. Did you also convert mupdate master to skiplist? Manually recovered mailboxes.db on the frontends, as they did not seem to be getting updated. Manually how? So at this point my user agent says Unknown/Invalid partition. The partitions are correctly defined on the backend mail stores. A 'ctl_mboxlist -d' shows correct partitions, no matter which local mailboxes.db I attempt to dump. Furthermore, LMTP is still not delivering to the backends during this outage. This is when the user agent talks to the a frontend? I suspect this is an artifact of how the frontends were manually updated. :wes Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: murder setup - mailboxes.db corruption - trouble recovering with ctl_mboxlist
On 20 Nov 2008, at 07:39, Eric G. Wolfe wrote: Found this: http://cyrusimap.web.cmu.edu/twiki/bin/view/Cyrus/ CyrusMurderFailureModes First, I followed the Easy instructions, which was a bust. Did you see this: https://bugzilla.andrew.cmu.edu/show_bug.cgi?id=2819 If you're not running something pretty current, Easy doesn't work. Next, I tried the Hard instructions. Four hours later the mupdate master finished synch'ing with the backends. I started up the front-ends, per the instructions. The front-ends failed to synch with the mupdate master. So, after Hard, you have a good copy on the mupdate master. In what way do the frontends fail to sync? So in an effort to try something else. I figured if the mailboxes.db on the front-ends and the master are the same format, I could just shutdown the mupdate master; copy the mailboxes.db file over to the front-ends; and start everything up. This was also a bust. Presuming you're using skiplist, you can in fact just copy mailboxes.db between mupdate master and frontends. However, when frontends start up, they insist on getting a full copy of the database from mupdate master. This can take some time, and in older versions might error out in various ways. Oh, and I am getting these logs on the mupdate master. However, the number in fs.file-nr is nowhere near approaching fs.file-max. There are no ulimits on the 'cyrus' user. There was a maxfds=1024 parameter in /etc/imapd.conf. I tried restarting without this parameter, and it seemed I couldn't keep the master process running without it. If I restart the service, it will run fine for a while, but it eventually starts complaining again. So I tried quadrupling the maxfds value, and we'll see if that helps. imapd.conf (excerpt) mupdate cmd=/usr/lib64/cyrus-imapd/mupdate -m listen=3905 prefork=1 maxfds=1024 maillog (excerpt) Nov 20 07:18:54 mumailmaster mupdate[27227]: refused connection from mumailstore01 Nov 20 07:18:54 mumailmaster mupdate[27227]: warning: cannot open /etc/hosts.allow: Too many open files The high connection rate is caused by mail delivery. Stock lmtp proxy connects to the mupdate master to get backend information, instead of referring to the local mailboxes.db. I have patches for 2.2.x cyrus, in 2.3.x cyrus, unified murder refers to mailboxes.db instead of mupdate master. The fact that lmtp proxy refers to mupdate master in any configuration is probably a bug. With a large mail backlog, plus new inbound mail, this bottleneck is a big problem. Couple that with trying to resync the frontends, and mupdate master is an even smaller bottleneck. Additionally, I have double-checked all cyrus related service accounts and their associated passwords. Our mupdate service account is successfully authenticating on the mupdate master. I am getting a imap: kick_mupdate: can't connect to target: Connection refused on the front-ends. However, I can connect to port 3905 on the mupdate master. The kick_mupdate error is just a signal that the mupdate on the frontend is in the process of resyncing. It can be ignored. :wes Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
RE: murder setup - mailboxes.db corruption - trouble recovering with ctl_mboxlist
cyrus-imapd-2.2.12-9.RHEL4 From: Wesley Craig [EMAIL PROTECTED] Sent: Thursday, November 20, 2008 10:08 AM To: Eric G.Wolfe Cc: info-cyrus@lists.andrew.cmu.edu Subject: Re: murder setup - mailboxes.db corruption - trouble recovering with ctl_mboxlist Which version of cyrus-imapd are you running? On 20 Nov 2008, at 01:23, Eric G. Wolfe wrote: I tried the following directions for recovery. http://asg.web.cmu.edu/archive/message.php?mailbox=archive.info- cyrussearchterm=skiplistmsg=32337. So you're attempting to convert from BerkeleyDB to skiplist? I made backup copies of all files deleted, renaming them $filename.corrupt. I did this on each server, recovered on the backends, and let it push the updates to the mupdate server. Did you also convert mupdate master to skiplist? Manually recovered mailboxes.db on the frontends, as they did not seem to be getting updated. Manually how? So at this point my user agent says Unknown/Invalid partition. The partitions are correctly defined on the backend mail stores. A 'ctl_mboxlist -d' shows correct partitions, no matter which local mailboxes.db I attempt to dump. Furthermore, LMTP is still not delivering to the backends during this outage. This is when the user agent talks to the a frontend? I suspect this is an artifact of how the frontends were manually updated. :wes Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
RE: murder setup - mailboxes.db corruption - trouble recovering with ctl_mboxlist
___ From: Wesley Craig [EMAIL PROTECTED] Sent: Thursday, November 20, 2008 10:27 AM To: Eric G.Wolfe Cc: info-cyrus@lists.andrew.cmu.edu Subject: Re: murder setup - mailboxes.db corruption - trouble recovering with ctl_mboxlist On 20 Nov 2008, at 07:39, Eric G. Wolfe wrote: Found this: http://cyrusimap.web.cmu.edu/twiki/bin/view/Cyrus/ CyrusMurderFailureModes First, I followed the Easy instructions, which was a bust. Did you see this: https://bugzilla.andrew.cmu.edu/show_bug.cgi?id=2819 No, but that was a problem at 2:00 AM not 6:00 AM after trying the Hard instructions. If you're not running something pretty current, Easy doesn't work. Next, I tried the Hard instructions. Four hours later the mupdate master finished synch'ing with the backends. I started up the front-ends, per the instructions. The front-ends failed to synch with the mupdate master. So, after Hard, you have a good copy on the mupdate master. In what way do the frontends fail to sync? I don't know, are the frontends' mailboxes.db supposed to stay at 144 bytes for 30 minutes or more? On the backends, at least I could monitor the mailboxes.db growing, not quickly, but progress was apparent. So in an effort to try something else. I figured if the mailboxes.db on the front-ends and the master are the same format, I could just shutdown the mupdate master; copy the mailboxes.db file over to the front-ends; and start everything up. This was also a bust. Presuming you're using skiplist, you can in fact just copy mailboxes.db between mupdate master and frontends. However, when frontends start up, they insist on getting a full copy of the database from mupdate master. This can take some time, and in older versions might error out in various ways. We are using skiplist, I copied the mailboxes.db to frontends. If the frontends are updating, it is not apparent. I could not verify that either of them were synching from the master after the mupdate master synch. Which is why I copied them to the frontends to speed things up. Oh, and I am getting these logs on the mupdate master. However, the number in fs.file-nr is nowhere near approaching fs.file-max. There are no ulimits on the 'cyrus' user. There was a maxfds=1024 parameter in /etc/imapd.conf. I tried restarting without this parameter, and it seemed I couldn't keep the master process running without it. If I restart the service, it will run fine for a while, but it eventually starts complaining again. So I tried quadrupling the maxfds value, and we'll see if that helps. imapd.conf (excerpt) mupdate cmd=/usr/lib64/cyrus-imapd/mupdate -m listen=3905 prefork=1 maxfds=1024 maillog (excerpt) Nov 20 07:18:54 mumailmaster mupdate[27227]: refused connection from mumailstore01 Nov 20 07:18:54 mumailmaster mupdate[27227]: warning: cannot open /etc/hosts.allow: Too many open files The high connection rate is caused by mail delivery. Stock lmtp proxy connects to the mupdate master to get backend information, instead of referring to the local mailboxes.db. I have patches for 2.2.x cyrus, in 2.3.x cyrus, unified murder refers to mailboxes.db instead of mupdate master. The fact that lmtp proxy refers to mupdate master in any configuration is probably a bug. Strange that it would just start causing problems now. We probably are seeing a cascading effect of failure, with the backlog though. Do the latest vanilla trees, have these patches included in them? The packages here: http://cyrusimap.web.cmu.edu/downloads.html#imap. I am somewhat reluctant to upgrade things in a fragile state. If these patches are included in latest releases, is 2.3.13 a fairly painless upgrade path from 2.2.12, or do we need to go with 2.2.13? With a large mail backlog, plus new inbound mail, this bottleneck is a big problem. Couple that with trying to resync the frontends, and mupdate master is an even smaller bottleneck. Is there anything we can turn off in the cyrus.conf or imapd.conf, to work around this bottleneck? In other words, can we keep the MTA from knocking on the door for long enough to get everything running smoothly again? Additionally, I have double-checked all cyrus related service accounts and their associated passwords. Our mupdate service account is successfully authenticating on the mupdate master. I am getting a imap: kick_mupdate: can't connect to target: Connection refused on the front-ends. However, I can connect to port 3905 on the mupdate master. The kick_mupdate error is just a signal that the mupdate on the frontend is in the process of resyncing. It can be ignored. Ok, because it sounds like a problem of connecting to the mupdate master port on 3905, to the unitiated. :wes Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing
RE: murder setup - mailboxes.db corruption - trouble recovering with ctl_mboxlist
On Thu, 20 Nov 2008, Wolfe, Eric G wrote: We are using skiplist, I copied the mailboxes.db to frontends. If the frontends are updating, it is not apparent. I could not verify that either of them were synching from the master after the mupdate master synch. Which is why I copied them to the frontends to speed things up. I've tried this in the past for the same reasons, but it doesn't work. The only way I've been able to get the correct mailboxes.db on frontends is to let them sync with the mupdate master. And yes, this takes a while from scratch when you are using skiplist because writes are much slower with skiplist. Strange that it would just start causing problems now. We probably are seeing a cascading effect of failure, with the backlog though. Do the latest vanilla trees, have these patches included in them? The packages here: http://cyrusimap.web.cmu.edu/downloads.html#imap. I am somewhat reluctant to upgrade things in a fragile state. If these patches are included in latest releases, is 2.3.13 a fairly painless upgrade path from 2.2.12, or do we need to go with 2.2.13? I wouldn't upgrade until you can get a working system. Is there anything we can turn off in the cyrus.conf or imapd.conf, to work around this bottleneck? In other words, can we keep the MTA from knocking on the door for long enough to get everything running smoothly again? Yes, comment out lmtp in your cyrus.conf on the frontends. After you have successfully synced mailboxes.db, enable lmtp again and restart cyrus on the frontends. You probably should put some limits on lmtp as well. We use maxchild=50 here (3 frontends). Ok, because it sounds like a problem of connecting to the mupdate master port on 3905, to the unitiated. Hard to say, but take the steps above to simplify the problem. :) Andy Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
murder setup - mailboxes.db corruption - trouble recovering with ctl_mboxlist
We have a RHEL4u7 on all 5 servers: 1 mupdate master: mumailmaster 2 backends: mumailstore01, mumailstore02 2 Postfix MTA/Cyrus proxy frontends: mumail01, mumail02 So I started getting this on my backends around 14:15 EST, at which time mail started getting deferred to the backends. I have 20,000+ per frontend deferred for delivery at the time of this e-mail. Nov 19 16:30:26 mumailstore01 ctl_cyrusdb[4672]: DBERROR db4: PANIC: Cannot allocate memory Nov 19 16:30:26 mumailstore01 ctl_cyrusdb[4672]: DBERROR: critical database situation Nov 19 16:30:26 mumailstore01 ctl_mboxlist[4673]: DBERROR db4: PANIC: fatal region error detected; run recovery Nov 19 16:30:26 mumailstore01 ctl_mboxlist[4673]: DBERROR: critical database situation Nov 19 16:30:26 mumailstore01 ctl_cyrusdb[4674]: DBERROR db4: PANIC: fatal region error detected; run recovery Nov 19 16:30:26 mumailstore01 ctl_cyrusdb[4674]: DBERROR: critical database situation I tried the following directions for recovery. http://asg.web.cmu.edu/archive/message.php?mailbox=archive.info-cyrussearchterm=skiplistmsg=32337. I made backup copies of all files deleted, renaming them $filename.corrupt. I did this on each server, recovered on the backends, and let it push the updates to the mupdate server. Manually recovered mailboxes.db on the frontends, as they did not seem to be getting updated. If I am going about this wrong, please someone point me in the right direction for documentation on murder disaster recovery. The following, while somewhat helpful, does not go into a great amount of detail: http://cyrusimap.web.cmu.edu/imapd/install-murder.html. So at this point my user agent says Unknown/Invalid partition. The partitions are correctly defined on the backend mail stores. A 'ctl_mboxlist -d' shows correct partitions, no matter which local mailboxes.db I attempt to dump. Furthermore, LMTP is still not delivering to the backends during this outage. Any helpful tips or pointers would be appreciated. Thanks, -- Eric G. Wolfe, IT Associate, Sr. One John Marshall Drive Marshall University, Drinko Library 428k Huntington, WV 25755 Phone: 304.696.3428 Email: [EMAIL PROTECTED] Who is General Failure and why is he reading my hard disk ? Microsoft spel chekar vor sail, worgs grate !! (By [EMAIL PROTECTED], Felix von Leitner) Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Patch to avoid mailboxesdb corruption on concurrent renames
On Fri, Apr 04, 2008 at 07:10:27AM -0400, Ken Murchison wrote: Bron Gondwana wrote: Bron ( P.S. isn't it about time for a 2.3.12? I'm getting sick of posting skiplist patches to people running the lastest and having issues! ) Yes, it probably is. Perhaps I'll make a pre-release today while I troll bugzilla for any showstoppers. That would be nice. Also, I'll check it against our patch list and see if there's anything bugfixy in there that I haven't pushed to you yet. By the way - would you prefer me to push things through Bugzilla? So far I've just been posting patches to the mailing list, but I'm happy to use whatever process is easiest for you. Bron. Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Patch to avoid mailboxesdb corruption on concurrent renames
Bron Gondwana wrote: On Fri, Apr 04, 2008 at 07:10:27AM -0400, Ken Murchison wrote: Bron Gondwana wrote: Bron ( P.S. isn't it about time for a 2.3.12? I'm getting sick of posting skiplist patches to people running the lastest and having issues! ) Yes, it probably is. Perhaps I'll make a pre-release today while I troll bugzilla for any showstoppers. That would be nice. Also, I'll check it against our patch list and see if there's anything bugfixy in there that I haven't pushed to you yet. By the way - would you prefer me to push things through Bugzilla? So far I've just been posting patches to the mailing list, but I'm happy to use whatever process is easiest for you. If you use bugzilla, its a lot less likely that I won't forget it. -- Kenneth Murchison Systems Programmer Project Cyrus Developer/Maintainer Carnegie Mellon University Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Patch to avoid mailboxesdb corruption on concurrent renames
I've put a header on the patch describing the bug. Basically,
the result code from mailbox_open_locked() wasn't being tested
sufficiently, and hence the new mailbox name would be created
in mailboxes.db even though the files were no longer available
to be copied - causing sync bailouts and IOERRORs and all sorts
of fun.
What causes this? Clients that send a RENAME or DELETE event
and then you do something else in the GUI and they open a
_separate_ connection which tries to do something else to the
source folder.
I think it is our only remaining source of bailouts! It
requires a reconstruct to fix when it happens.
Patch attached, and the perl script I used to confirm that it
existed and confirm that this patch fixed it.
Regards,
Bron ( P.S. isn't it about time for a 2.3.12? I'm getting sick
of posting skiplist patches to people running the
lastest and having issues! )
#!/usr/bin/perl -w
use IO::Socket::INET;
$| = 1;
our $ADDR = '127.0.0.1:143';
our $USER = '[EMAIL PROTECTED]';
our $PASS = 'FIXME';
my $source = prepare_source();
my %h;
foreach my $n (1..3) {
my $pid = fork();
unless ($pid) {
my $res = do_rename($source, $n);
print RES: $n = $res\n;
exit();
}
$h{$pid} = 1;
}
foreach my $pid (keys %h) {
waitpid($pid, 0);
}
dump_folders();
sub prepare_source {
my $sourcename = INBOX.source$$;
my $fh = IO::Socket::INET-new($ADDR);
$fh-getline();
$fh-print(. login $USER $PASS\r\n);
$fh-getline();
$fh-print(. delete $sourcename\r\n); # just in case!
$fh-getline();
$fh-print(. create $sourcename\r\n);
$fh-getline();
print $sourcename: ;
foreach my $n (1..1) {
my $msg = gen_msg($n);
my $len = length($msg);
$fh-print(. append $sourcename {$len}\r\n);
$fh-print($msg\r\n);
print . unless $n % 50;
}
print done\n;
$fh-print( . bye\r\n);
$fh-getline();
return $sourcename;
}
sub do_rename {
my $sourcename = shift;
my $n = shift;
my $fh = IO::Socket::INET-new($ADDR);
$fh-getline();
$fh-print(. login $USER $PASS\r\n);
$fh-getline();
$fh-print(. delete $sourcename-dest$n\r\n); # just in case!
$fh-getline();
$fh-print(. rename $sourcename $sourcename-dest$n\r\n);
my $res = $fh-getline();
$fh-print( . bye\r\n);
$fh-getline();
return $res;
}
sub gen_msg {
my $n = shift;
return qq{Subject: Message $n
From: [EMAIL PROTECTED]
To: [EMAIL PROTECTED]
This is a test message made long by shoving a whole pile of lines on it!
} . ($n\n x 10);
}
sub dump_folders {
my $fh = IO::Socket::INET-new($ADDR);
$fh-getline();
$fh-print(. login $USER $PASS\r\n);
$fh-getline();
$fh-print(TAG1 list \INBOX.*\ \*\\r\n);
$fh-getline();
while (my $res = $fh-getline()) {
last if $res =~ m{^TAG1 };
print $res;
}
$fh-print(. bye\r\n);
$fh-getline();
}
Handle concurrent mailbox renames safely
*CANDIDATE FOR UPSTREAM*
About the last cause of sync bailouts at FastMail has been
the case where a user starts renaming a folder, and then does
something else (including deletes with renames here, since we
have the deleterename option enabled)
This was caused by the mailbox_rename_copy function not
checking the return code of mailbox_open_locked properly
(probably due to it being a huge function with multiple
different styles of using the if (!r) error checking
paradigm in different places, but my can be bothered on
major refactors is low at the moment)
The side effect of this was that if the mailbox existed
going in (copy was still underway) then the old mailbox
would get deleted (noop, or corruption if you don't have
all the skiplist patches applied!) and the new mailbox name
created but with no files in place!
This patch checks the return code of mailbox_open_locked
and aborts with an IO error if the mailbox has already been
locked.
Index: cyrus-imapd-2.3.11/imap/mboxlist.c
===
--- cyrus-imapd-2.3.11.orig/imap/mboxlist.c 2008-04-04 00:59:36.0 +
+++ cyrus-imapd-2.3.11/imap/mboxlist.c 2008-04-04 02:47:46.0 +
@@ -1339,11 +1339,15 @@ int mboxlist_renamemailbox(char *oldname
if(!r) {
r = mailbox_open_locked(oldname, oldpath, oldmpath, oldacl, auth_state,
oldmailbox, 0);
- oldopen = 1;
+ if (r) {
+ goto done;
+ } else {
+ oldopen = 1;
+ }
}
/* 6. Copy mailbox */
-if (!r !(mbtype MBTYPE_REMOTE)) {
+if (!(mbtype MBTYPE_REMOTE)) {
/* Rename the actual mailbox */
r = mailbox_rename_copy(oldmailbox, newname, newpartition,
NULL, NULL, newmailbox,
Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: quota corruption
Hi again, I think I am very close to find the main reason of my weird mailbox problems. Today I tried to setup a corrupted mail account as a pop3 account and the client gave me 'Unable to lock maildrop'. I googled this error and found a reply on a mailing list which says this can be because of TCP timeout. When I saw this reply I remembered that I have changed our firewall apliance which had a static route rule to route the packages for the mail servers public IP address to another firewall apliance which has my mail server on its DMZ. I removed the old firewall apliance and installed a new one(which is netscreen), created the same static route rule and that day these weird problems started to came from the users. I guess there is a problem with the netscreens static routing because even if I request ssh to the public IP of the mail server, session drops suddenly if I keep the ssh client idle for a few seconds. If anyone has an idea it will be appreciated. When I first saw this problem I was suspicious about an outlook bug but when I saw that there are more than one users having the same problem I changed my mind because we are using cyrus on sles9 for nearly 4 years and we didnt have such problems. But anyway I will check the mailboxes with a different mail client. I am also thinking about disabling quota for all users for a short time just to solve the problem until I setup a mor recent version of cyrus on a different server. Is there a way to disable quota for all users with a single step? I'm not sure but I think a brute force method is to remove all quota files in /var/lib/imap/quota (or wherever that directory is in on your server). Of course cyrus-imapd should be stopped while doing it. If I remember correctly I did this once to fix broken quota on a server. But don't blame me if it breaks your server... Simon Thanks Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: quota corruption
Hi again, I think I am very close to find the main reason of my weird mailbox problems. Today I tried to setup a corrupted mail account as a pop3 account and the client gave me 'Unable to lock maildrop'. I googled this error and found a reply on a mailing list which says this can be because of TCP timeout. When I saw this reply I remembered that I have changed our firewall apliance which had a static route rule to route the packages for the mail servers public IP address to another firewall apliance which has my mail server on its DMZ. I removed the old firewall apliance and installed a new one(which is netscreen), created the same static route rule and that day these weird problems started to came from the users. I guess there is a problem with the netscreens static routing because even if I request ssh to the public IP of the mail server, session drops suddenly if I keep the ssh client idle for a few seconds. If anyone has an idea it will be appreciated. While I'm usually runnning Linux firewall, I once had the pleasure with Netscreen. They do some kind of DOS protection which may make it unusable to run servers behind. You have to remove those rules to make things work as expected. I don't remember in detail what was needed on the Netscreen, but you may find out yourself. At least I agree with you that you may have found the root of your problems. Simon When I first saw this problem I was suspicious about an outlook bug but when I saw that there are more than one users having the same problem I changed my mind because we are using cyrus on sles9 for nearly 4 years and we didnt have such problems. But anyway I will check the mailboxes with a different mail client. I am also thinking about disabling quota for all users for a short time just to solve the problem until I setup a mor recent version of cyrus on a different server. Is there a way to disable quota for all users with a single step? I'm not sure but I think a brute force method is to remove all quota files in /var/lib/imap/quota (or wherever that directory is in on your server). Of course cyrus-imapd should be stopped while doing it. If I remember correctly I did this once to fix broken quota on a server. But don't blame me if it breaks your server... Simon Thanks Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: quota corruption
On Thursday 27 December 2007, Erol YILDIZ wrote: I read about corrupted quota files and that I can fix this by using 'quota -f' command but on SLES9 quota command doesnt have -f parameter. su -l cyrus -s /bin/bash -c /usr/lib/cyrus/quota [-d doomain] -f user/username Possible quota of cyrus is not in /usr/lib/cyrus in SLES and user is not cyrus. -- Regards, Sergey Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: quota corruption
When I first saw this problem I was suspicious about an outlook bug but when I saw that there are more than one users having the same problem I changed my mind because we are using cyrus on sles9 for nearly 4 years and we didnt have such problems. But anyway I will check the mailboxes with a different mail client. I am also thinking about disabling quota for all users for a short time just to solve the problem until I setup a mor recent version of cyrus on a different server. Is there a way to disable quota for all users with a single step? Thanks Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: quota corruption
When I first saw this problem I was suspicious about an outlook bug but when I saw that there are more than one users having the same problem I changed my mind because we are using cyrus on sles9 for nearly 4 years and we didnt have such problems. But anyway I will check the mailboxes with a different mail client. I am also thinking about disabling quota for all users for a short time just to solve the problem until I setup a mor recent version of cyrus on a different server. Is there a way to disable quota for all users with a single step? I'm not sure but I think a brute force method is to remove all quota files in /var/lib/imap/quota (or wherever that directory is in on your server). Of course cyrus-imapd should be stopped while doing it. If I remember correctly I did this once to fix broken quota on a server. But don't blame me if it breaks your server... Simon Thanks Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: quota corruption
I could recovered the mailboxes in about six steps. By the way I have solved the cyradm problem by giving ownership of /etc/sasldb2 to cyrus and mail group. Now I can use cyradmin without any problem. Here are the steps I did below for recovering the content of the default Inbox folder. 1) I took the backup of the mailbox of the user 2) touch cyrus.index (I do this to be able to delete the mailbox, otherwise it doesnt because the mailbox is corupted) 3) from cyradmin 'dm user/gulfer^tuna' 4) from cyradmin 'cm user/gulfer^tuna' 5) Then I copied the contents of the backedup mailbox contents to the newly created 6) Then I deleted cyrus.index by 'rm /var/spool/imap/user/gulfer^tuna/cyrus.index' 7) And I recreated the cyrus.index from cyradm by 'reconstruct -f -r user/gulfer^tuna' and then the mailbox is recovered. But now some of the users still having the problems for which I have started this thread. One of the symptoms is, users can't purge deleted items, when they press purge it says the client lost connection to the server and that maybe becuase either the folder doesnt exist or this is a limitation of the imap server. Another one is when the user selects a mail to view, client says it needs to be downloaded from the server and asks the user if he wants to mark it for download. Even if the user marks it for download still cant view the mail. Now you know when I disable the quota for the user the problem was solved so that I tought it may be because of the quota files. But when I did 'quota -f' this didnt help to solve the problem. Maybe I need to delete the quota files and recreate them. Any further ideas? Thanks. -- Erol YILDIZ On Dec 28, 2007 10:46 AM, Erol YILDIZ [EMAIL PROTECTED] wrote: I have recontructed all of the users with reconstruct command and then could run quota -f without any problem but this didnt solve the connetion problems for the quota enabled accounts plus now I have a bigger problem with some of the accounts. Some of the users Inbox accounts disappeared. If I send a test mail to one of these users I receive the error below from the Mail delivery System. '[EMAIL PROTECTED] (expanded from [EMAIL PROTECTED]): host /var/lib/imap/socket/lmtp[/var/lib/imap/socket/lmtp] said: 550-Mailbox unknown. Either there is no mailbox associated with this 550-name or you do not have authorization to see it. 550 5.1.1 User unknown (in reply to RCPT TO command)' First look if the mailbox exist in cyradm or using : # ctl_mboxlist -d | grep gulfer.tuna then look if the directory exist in the file system # ls -la `mbpath user/[EMAIL PROTECTED] check for user right of the directory and of files inside. I dont understant what expanded from mean. Any idea ? How can I recover these Inbox folders? -- Thanks - Original Message - From: Alain Spineux [EMAIL PROTECTED] To: Erol YILDIZ [EMAIL PROTECTED] Cc: info-cyrus@lists.andrew.cmu.edu Sent: Thursday, December 27, 2007 11:23 PM Subject: Re: quota corruption On Dec 27, 2007 7:51 PM, Erol YILDIZ [EMAIL PROTECTED] wrote: I get the error below. a_akpinar is an old user which I have deleted but seems like its not been deleted from somewhere. How can I solve this? Dec 27 19:33:30 sles quota[14313]: IOERROR: opening /var/spool/imap/user/a_akpinar/cyrus.header: No such file or directory You can use reconstruct to rebuild these files. Then you can use the cyradm to delete the mailbox. - Original Message - From: Patrick Boutilier [EMAIL PROTECTED] To: info-cyrus@lists.andrew.cmu.edu Sent: Thursday, December 27, 2007 5:36 PM Subject: Re: quota corruption Erol YILDIZ wrote: I found it now. Its under /usr/lib/cyrus/bin but when I send quota -f command it gives 'quota: System I/O error No such file or directory'. Do you know why does it give this error? Check your log files (usually /var/log/messages or /var/log/maillog) . There should be some indication as to what file/directory it can not find. Otherwise you could strace the quota process. - Original Message - From: Simon Matter [EMAIL PROTECTED] To: Alain Spineux [EMAIL PROTECTED] Cc: Erol YILDIZ [EMAIL PROTECTED]; info-cyrus@lists.andrew.cmu.edu Sent: Thursday, December 27, 2007 4:53 PM Subject: Re: quota corruption On Dec 27, 2007 2:44 PM, Erol YILDIZ [EMAIL PROTECTED] wrote: Hi, I am using Suse linux enterprise server 9 with cyrus and for a few days having some problems. For example when a user tries to purge deleted mails, outlook says 'The folder Inbox cannot be selected. This may be because of a limitation of your IMAP server or the folder may have been deleted or moved.' and we have different similar connection problems and syronization problems. When I disable quota for the users, the problem
Re: quota corruption
-- Erol YILDIZ [EMAIL PROTECTED] is rumored to have mumbled on 29. Dezember 2007 11:44:25 +0200 regarding Re: quota corruption: But now some of the users still having the problems for which I have started this thread. One of the symptoms is, users can't purge deleted items, when they press purge it says the client lost connection to the server and that maybe becuase either the folder doesnt exist or this is a limitation of the imap server. Another one is when the user selects a mail to view, client says it needs to be downloaded from the server and asks the user if he wants to mark it for download. Even if the user marks it for download still cant view the mail. Now you know when I disable the quota for the user the problem was solved so that I tought it may be because of the quota files. But when I did 'quota -f' this didnt help to solve the problem. Maybe I need to delete the quota files and recreate them. Any further ideas? This may not be the answer you're looking for, but based on anecdotal evidence I'd say it's an Outlook bug. Here's what you could try if you're adventurous: mkdir the directory /var/lib/imap/log/username as user cyrus, replacing username with one of the users that have the problems. Wait until the problem occurs. Look at the log files created. If you can correlate a sequence of IMAP commands to the problem, it way give a hint what's wrong. And to verify that it's an Outlook issue you could switch the affected users to a different client, at least temporarily. -- Sebastian Hagedorn - RZKR-R1 (Flachbau), Zi. 18, Robert-Koch-Str. 10 Zentrum für angewandte Informatik - Universitätsweiter Service RRZK Universität zu Köln / Cologne University - Tel. +49-221-478-5587 pgpvTlNFCvFYr.pgp Description: PGP signature Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: quota corruption
I have recontructed all of the users with reconstruct command and then could run quota -f without any problem but this didnt solve the connetion problems for the quota enabled accounts plus now I have a bigger problem with some of the accounts. Some of the users Inbox accounts disappeared. If I send a test mail to one of these users I receive the error below from the Mail delivery System. '[EMAIL PROTECTED] (expanded from [EMAIL PROTECTED]): host /var/lib/imap/socket/lmtp[/var/lib/imap/socket/lmtp] said: 550-Mailbox unknown. Either there is no mailbox associated with this 550-name or you do not have authorization to see it. 550 5.1.1 User unknown (in reply to RCPT TO command)' How can I recover these Inbox folders? -- Thanks - Original Message - From: Alain Spineux [EMAIL PROTECTED] To: Erol YILDIZ [EMAIL PROTECTED] Cc: info-cyrus@lists.andrew.cmu.edu Sent: Thursday, December 27, 2007 11:23 PM Subject: Re: quota corruption On Dec 27, 2007 7:51 PM, Erol YILDIZ [EMAIL PROTECTED] wrote: I get the error below. a_akpinar is an old user which I have deleted but seems like its not been deleted from somewhere. How can I solve this? Dec 27 19:33:30 sles quota[14313]: IOERROR: opening /var/spool/imap/user/a_akpinar/cyrus.header: No such file or directory You can use reconstruct to rebuild these files. Then you can use the cyradm to delete the mailbox. - Original Message - From: Patrick Boutilier [EMAIL PROTECTED] To: info-cyrus@lists.andrew.cmu.edu Sent: Thursday, December 27, 2007 5:36 PM Subject: Re: quota corruption Erol YILDIZ wrote: I found it now. Its under /usr/lib/cyrus/bin but when I send quota -f command it gives 'quota: System I/O error No such file or directory'. Do you know why does it give this error? Check your log files (usually /var/log/messages or /var/log/maillog) . There should be some indication as to what file/directory it can not find. Otherwise you could strace the quota process. - Original Message - From: Simon Matter [EMAIL PROTECTED] To: Alain Spineux [EMAIL PROTECTED] Cc: Erol YILDIZ [EMAIL PROTECTED]; info-cyrus@lists.andrew.cmu.edu Sent: Thursday, December 27, 2007 4:53 PM Subject: Re: quota corruption On Dec 27, 2007 2:44 PM, Erol YILDIZ [EMAIL PROTECTED] wrote: Hi, I am using Suse linux enterprise server 9 with cyrus and for a few days having some problems. For example when a user tries to purge deleted mails, outlook says 'The folder Inbox cannot be selected. This may be because of a limitation of your IMAP server or the folder may have been deleted or moved.' and we have different similar connection problems and syronization problems. When I disable quota for the users, the problem ends. I read about corrupted quota files and that I can fix this by using 'quota -f' command but on SLES9 quota command doesnt have -f parameter. Are you speaking about the linux quota command or the cyrus one cyrquota ? Alain, there is space for confusion here because there is no cyrquota in the cyrus distribution, only Debian (any maybe others) rename it so. However, I don't know how SuSE does it. Simon Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html -- Alain Spineux aspineux gmail com May the sources be with you Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: quota corruption
On Dec 28, 2007 10:46 AM, Erol YILDIZ [EMAIL PROTECTED] wrote: I have recontructed all of the users with reconstruct command and then could run quota -f without any problem but this didnt solve the connetion problems for the quota enabled accounts plus now I have a bigger problem with some of the accounts. Some of the users Inbox accounts disappeared. If I send a test mail to one of these users I receive the error below from the Mail delivery System. '[EMAIL PROTECTED] (expanded from [EMAIL PROTECTED]): host /var/lib/imap/socket/lmtp[/var/lib/imap/socket/lmtp] said: 550-Mailbox unknown. Either there is no mailbox associated with this 550-name or you do not have authorization to see it. 550 5.1.1 User unknown (in reply to RCPT TO command)' First look if the mailbox exist in cyradm or using : # ctl_mboxlist -d | grep gulfer.tuna then look if the directory exist in the file system # ls -la `mbpath user/[EMAIL PROTECTED] check for user right of the directory and of files inside. I dont understant what expanded from mean. Any idea ? How can I recover these Inbox folders? -- Thanks - Original Message - From: Alain Spineux [EMAIL PROTECTED] To: Erol YILDIZ [EMAIL PROTECTED] Cc: info-cyrus@lists.andrew.cmu.edu Sent: Thursday, December 27, 2007 11:23 PM Subject: Re: quota corruption On Dec 27, 2007 7:51 PM, Erol YILDIZ [EMAIL PROTECTED] wrote: I get the error below. a_akpinar is an old user which I have deleted but seems like its not been deleted from somewhere. How can I solve this? Dec 27 19:33:30 sles quota[14313]: IOERROR: opening /var/spool/imap/user/a_akpinar/cyrus.header: No such file or directory You can use reconstruct to rebuild these files. Then you can use the cyradm to delete the mailbox. - Original Message - From: Patrick Boutilier [EMAIL PROTECTED] To: info-cyrus@lists.andrew.cmu.edu Sent: Thursday, December 27, 2007 5:36 PM Subject: Re: quota corruption Erol YILDIZ wrote: I found it now. Its under /usr/lib/cyrus/bin but when I send quota -f command it gives 'quota: System I/O error No such file or directory'. Do you know why does it give this error? Check your log files (usually /var/log/messages or /var/log/maillog) . There should be some indication as to what file/directory it can not find. Otherwise you could strace the quota process. - Original Message - From: Simon Matter [EMAIL PROTECTED] To: Alain Spineux [EMAIL PROTECTED] Cc: Erol YILDIZ [EMAIL PROTECTED]; info-cyrus@lists.andrew.cmu.edu Sent: Thursday, December 27, 2007 4:53 PM Subject: Re: quota corruption On Dec 27, 2007 2:44 PM, Erol YILDIZ [EMAIL PROTECTED] wrote: Hi, I am using Suse linux enterprise server 9 with cyrus and for a few days having some problems. For example when a user tries to purge deleted mails, outlook says 'The folder Inbox cannot be selected. This may be because of a limitation of your IMAP server or the folder may have been deleted or moved.' and we have different similar connection problems and syronization problems. When I disable quota for the users, the problem ends. I read about corrupted quota files and that I can fix this by using 'quota -f' command but on SLES9 quota command doesnt have -f parameter. Are you speaking about the linux quota command or the cyrus one cyrquota ? Alain, there is space for confusion here because there is no cyrquota in the cyrus distribution, only Debian (any maybe others) rename it so. However, I don't know how SuSE does it. Simon Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html -- Alain Spineux aspineux gmail com May the sources be with you Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html -- Alain Spineux aspineux gmail com May the sources be with you Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: quota corruption
On Dec 27, 2007 2:44 PM, Erol YILDIZ [EMAIL PROTECTED] wrote: Hi, I am using Suse linux enterprise server 9 with cyrus and for a few days having some problems. For example when a user tries to purge deleted mails, outlook says 'The folder Inbox cannot be selected. This may be because of a limitation of your IMAP server or the folder may have been deleted or moved.' and we have different similar connection problems and syronization problems. When I disable quota for the users, the problem ends. I read about corrupted quota files and that I can fix this by using 'quota -f' command but on SLES9 quota command doesnt have -f parameter. Are you speaking about the linux quota command or the cyrus one cyrquota ? Regards ? Any ideas to solve this problem? Thanks. -- Erol YILDIZ Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html -- Alain Spineux aspineux gmail com May the sources be with you Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: quota corruption
I found it now. Its under /usr/lib/cyrus/bin but when I send quota -f command it gives 'quota: System I/O error No such file or directory'. Do you know why does it give this error? - Original Message - From: Simon Matter [EMAIL PROTECTED] To: Alain Spineux [EMAIL PROTECTED] Cc: Erol YILDIZ [EMAIL PROTECTED]; info-cyrus@lists.andrew.cmu.edu Sent: Thursday, December 27, 2007 4:53 PM Subject: Re: quota corruption On Dec 27, 2007 2:44 PM, Erol YILDIZ [EMAIL PROTECTED] wrote: Hi, I am using Suse linux enterprise server 9 with cyrus and for a few days having some problems. For example when a user tries to purge deleted mails, outlook says 'The folder Inbox cannot be selected. This may be because of a limitation of your IMAP server or the folder may have been deleted or moved.' and we have different similar connection problems and syronization problems. When I disable quota for the users, the problem ends. I read about corrupted quota files and that I can fix this by using 'quota -f' command but on SLES9 quota command doesnt have -f parameter. Are you speaking about the linux quota command or the cyrus one cyrquota ? Alain, there is space for confusion here because there is no cyrquota in the cyrus distribution, only Debian (any maybe others) rename it so. However, I don't know how SuSE does it. Simon Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: quota corruption
I get the error below. a_akpinar is an old user which I have deleted but seems like its not been deleted from somewhere. How can I solve this? - Original Message - From: Patrick Boutilier [EMAIL PROTECTED] To: info-cyrus@lists.andrew.cmu.edu Sent: Thursday, December 27, 2007 5:36 PM Subject: Re: quota corruption Erol YILDIZ wrote: I found it now. Its under /usr/lib/cyrus/bin but when I send quota -f command it gives 'quota: System I/O error No such file or directory'. Do you know why does it give this error? Check your log files (usually /var/log/messages or /var/log/maillog) . There should be some indication as to what file/directory it can not find. Otherwise you could strace the quota process. - Original Message - From: Simon Matter [EMAIL PROTECTED] To: Alain Spineux [EMAIL PROTECTED] Cc: Erol YILDIZ [EMAIL PROTECTED]; info-cyrus@lists.andrew.cmu.edu Sent: Thursday, December 27, 2007 4:53 PM Subject: Re: quota corruption On Dec 27, 2007 2:44 PM, Erol YILDIZ [EMAIL PROTECTED] wrote: Hi, I am using Suse linux enterprise server 9 with cyrus and for a few days having some problems. For example when a user tries to purge deleted mails, outlook says 'The folder Inbox cannot be selected. This may be because of a limitation of your IMAP server or the folder may have been deleted or moved.' and we have different similar connection problems and syronization problems. When I disable quota for the users, the problem ends. I read about corrupted quota files and that I can fix this by using 'quota -f' command but on SLES9 quota command doesnt have -f parameter. Are you speaking about the linux quota command or the cyrus one cyrquota ? Alain, there is space for confusion here because there is no cyrquota in the cyrus distribution, only Debian (any maybe others) rename it so. However, I don't know how SuSE does it. Simon Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: quota corruption
I get the error below. a_akpinar is an old user which I have deleted but seems like its not been deleted from somewhere. How can I solve this? Dec 27 19:33:30 sles quota[14313]: IOERROR: opening /var/spool/imap/user/a_akpinar/cyrus.header: No such file or directory - Original Message - From: Patrick Boutilier [EMAIL PROTECTED] To: info-cyrus@lists.andrew.cmu.edu Sent: Thursday, December 27, 2007 5:36 PM Subject: Re: quota corruption Erol YILDIZ wrote: I found it now. Its under /usr/lib/cyrus/bin but when I send quota -f command it gives 'quota: System I/O error No such file or directory'. Do you know why does it give this error? Check your log files (usually /var/log/messages or /var/log/maillog) . There should be some indication as to what file/directory it can not find. Otherwise you could strace the quota process. - Original Message - From: Simon Matter [EMAIL PROTECTED] To: Alain Spineux [EMAIL PROTECTED] Cc: Erol YILDIZ [EMAIL PROTECTED]; info-cyrus@lists.andrew.cmu.edu Sent: Thursday, December 27, 2007 4:53 PM Subject: Re: quota corruption On Dec 27, 2007 2:44 PM, Erol YILDIZ [EMAIL PROTECTED] wrote: Hi, I am using Suse linux enterprise server 9 with cyrus and for a few days having some problems. For example when a user tries to purge deleted mails, outlook says 'The folder Inbox cannot be selected. This may be because of a limitation of your IMAP server or the folder may have been deleted or moved.' and we have different similar connection problems and syronization problems. When I disable quota for the users, the problem ends. I read about corrupted quota files and that I can fix this by using 'quota -f' command but on SLES9 quota command doesnt have -f parameter. Are you speaking about the linux quota command or the cyrus one cyrquota ? Alain, there is space for confusion here because there is no cyrquota in the cyrus distribution, only Debian (any maybe others) rename it so. However, I don't know how SuSE does it. Simon Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: quota corruption
Erol YILDIZ wrote: I get the error below. a_akpinar is an old user which I have deleted but seems like its not been deleted from somewhere. How can I solve this? What happens when you try to delete a_akpinar again? You will probably get the same I/O errors in which case just touch the needed files and try to delete a_akpinar again. touch /var/spool/imap/user/a_akpinar/cyrus.header touch /var/spool/imap/user/a_akpinar/cyrus.index touch /var/spool/imap/user/a_akpinar/cyrus.cache Dec 27 19:33:30 sles quota[14313]: IOERROR: opening /var/spool/imap/user/a_akpinar/cyrus.header: No such file or directory snip Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: quota corruption
On Dec 27, 2007 7:51 PM, Erol YILDIZ [EMAIL PROTECTED] wrote: I get the error below. a_akpinar is an old user which I have deleted but seems like its not been deleted from somewhere. How can I solve this? Dec 27 19:33:30 sles quota[14313]: IOERROR: opening /var/spool/imap/user/a_akpinar/cyrus.header: No such file or directory You can use reconstruct to rebuild these files. Then you can use the cyradm to delete the mailbox. - Original Message - From: Patrick Boutilier [EMAIL PROTECTED] To: info-cyrus@lists.andrew.cmu.edu Sent: Thursday, December 27, 2007 5:36 PM Subject: Re: quota corruption Erol YILDIZ wrote: I found it now. Its under /usr/lib/cyrus/bin but when I send quota -f command it gives 'quota: System I/O error No such file or directory'. Do you know why does it give this error? Check your log files (usually /var/log/messages or /var/log/maillog) . There should be some indication as to what file/directory it can not find. Otherwise you could strace the quota process. - Original Message - From: Simon Matter [EMAIL PROTECTED] To: Alain Spineux [EMAIL PROTECTED] Cc: Erol YILDIZ [EMAIL PROTECTED]; info-cyrus@lists.andrew.cmu.edu Sent: Thursday, December 27, 2007 4:53 PM Subject: Re: quota corruption On Dec 27, 2007 2:44 PM, Erol YILDIZ [EMAIL PROTECTED] wrote: Hi, I am using Suse linux enterprise server 9 with cyrus and for a few days having some problems. For example when a user tries to purge deleted mails, outlook says 'The folder Inbox cannot be selected. This may be because of a limitation of your IMAP server or the folder may have been deleted or moved.' and we have different similar connection problems and syronization problems. When I disable quota for the users, the problem ends. I read about corrupted quota files and that I can fix this by using 'quota -f' command but on SLES9 quota command doesnt have -f parameter. Are you speaking about the linux quota command or the cyrus one cyrquota ? Alain, there is space for confusion here because there is no cyrquota in the cyrus distribution, only Debian (any maybe others) rename it so. However, I don't know how SuSE does it. Simon Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html -- Alain Spineux aspineux gmail com May the sources be with you Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: quota corruption
I have deleted users from yast, didnt delete them from cyradm because in SLES9 you can't log in to cyradm, when you try to log in it gives segmentation fault. I think you ment to delete from cyradm which I can't do without logging in. Do have any idea how to log in cyradmin in this case? Thanks. - Original Message - From: Patrick Boutilier [EMAIL PROTECTED] To: info-cyrus@lists.andrew.cmu.edu Sent: Thursday, December 27, 2007 10:24 PM Subject: Re: quota corruption Erol YILDIZ wrote: I get the error below. a_akpinar is an old user which I have deleted but seems like its not been deleted from somewhere. How can I solve this? What happens when you try to delete a_akpinar again? You will probably get the same I/O errors in which case just touch the needed files and try to delete a_akpinar again. touch /var/spool/imap/user/a_akpinar/cyrus.header touch /var/spool/imap/user/a_akpinar/cyrus.index touch /var/spool/imap/user/a_akpinar/cyrus.cache Dec 27 19:33:30 sles quota[14313]: IOERROR: opening /var/spool/imap/user/a_akpinar/cyrus.header: No such file or directory snip Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: quota corruption
Erol YILDIZ schrieb: I have deleted users from yast, didnt delete them from cyradm because in SLES9 you can't log in to cyradm, when you try to log in it gives segmentation fault. I think you ment to delete from cyradm which I can't do without logging in. Do have any idea how to log in cyradmin in this case? Thanks. Hi Erol, I'm using Cyrus on SUSE 9.0 which may be similar to your setup, and I think I've seen this error, too, when I did not login correctly. 'cyradm' is a program not a user (at least on my system), so I can't login to it. What I do is (logged in as root): Enter su cyrus (without quotation marks) to login as user cyrus, and then enter cyradm localhost. Then I'm prompted to enter the password for user cyrus. I hope that helps you. Wieland Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: *** glibc detected *** lmtpd: double free or corruption (out): 0x00002aaaacdeaa10 *** : 2 Time(s) problem
On Wed, Sep 12, 2007 at 08:10:29PM +0200, Alain Spineux wrote: When postfix try to deliver an email, it open a LMTP connection with cyrus and cyrus, instead of speaking correct LMTP reply with garbage. BUT 2c388000-2c479000 r-xp fd:00 12452057 /lib64/libdb-4.3.so is not realy garbage this look like to be the memory areas of a process. No, it's not garbage, it's part of the error message that glibc produces when it encounters the error in the subject. - Build Cyrus with full debugging info and no optimization (-g3 -O0) - export CYRUS_VERBOSE=50 - Start Cyrus - Send a message through postfix that is known to trigger the bug - You've now 15 seconds to attach to the newly created lmtpd process with gdb - Let lmtpd continue until it crashes; post a full backtrace Now, the bactrace will contain the location where the memory corruption was _detected_, not where it had really occured, but that may still be useful. An other method would be to modify the cmd=... entry in cyrus.conf and launch lmtpd inside valgrind. You still need to rebuild Cyrus with full debugging info to get meaningful backtraces (use --num-callers=10 or more), and it will be slow, but it has higher chance to catch the real bug. Gabor -- - MTA SZTAKI Computer and Automation Research Institute Hungarian Academy of Sciences - Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: *** glibc detected *** lmtpd: double free or corruption (out): 0x00002aaaacdeaa10 *** : 2 Time(s) problem
Ali Nebi wrote: Hi all, i have some problems in a mail server and i don't know how to solve these problems. I get these errors: /usr/libexec/postfix/lmtp: bad command startup -- throttling : 588 Time(s) 198.231.4.83.rbl-plus.mail-abuse.org: RBL lookup error: Host or domain name not found. Name service error for name=198.231.4.83.rbl-plus.mail-abuse.org type=A: Host not found, try again : 2 Time(s) 3.77.214.79.rbl-plus.mail-abuse.org: RBL lookup error: Host or domain name not found. Name service error for name=3.77.214.79.rbl-plus.mail-abuse.org type=A: Host not found, try again : 2 Time(s) dict_ldap_lookup: Search error -5: Timed out : 2 Time(s) network_biopair_interop: error reading 5 bytes from the network: Connection reset by peer : 2 Time(s) non-LMTP response from hermod.iguanait.com[/var/lib/imap/socket/lmtp]: *** glibc detected *** lmtpd: double free or corruption (out): 0x2cdeaa10 *** : 2 Time(s) non-LMTP response from hermod.iguanait.com[/var/lib/imap/socket/lmtp]: *** glibc detected *** lmtpd: double free or corruption (out): 0x2cdeaa30 *** : 4 Time(s) non-LMTP response from hermod.iguanait.com[/var/lib/imap/socket/lmtp]: *** glibc detected *** lmtpd: double free or corruption (out): 0x2cdeabb0 *** : 2 Time(s) non-LMTP response from hermod.iguanait.com[/var/lib/imap/socket/lmtp]: *** glibc detected *** lmtpd: double free or corruption (out): 0x2cdeac10 *** : 2 Time(s) non-LMTP response from hermod.iguanait.com[/var/lib/imap/socket/lmtp]: *** glibc detected *** lmtpd: double free or corruption (out): 0x2cdeac50 *** : 2 Time(s) non-LMTP response from hermod.iguanait.com[/var/lib/imap/socket/lmtp]: *** glibc detected *** lmtpd: double free or corruption (out): 0x2cdeacb0 *** : 4 Time(s) non-LMTP response from hermod.iguanait.com[/var/lib/imap/socket/lmtp]: *** glibc detected *** lmtpd: double free or corruption (out): 0x2cdeacf0 *** : 2 Time(s) non-LMTP response from hermod.iguanait.com[/var/lib/imap/socket/lmtp]: *** glibc detected *** lmtpd: double free or corruption (out): 0x2cdead30 *** : 2 Time(s) non-LMTP response from hermod.iguanait.com[/var/lib/imap/socket/lmtp]: *** glibc detected *** lmtpd: double free or corruption (out): 0x2cdead70 *** : 4 Time(s) non-LMTP response from hermod.iguanait.com[/var/lib/imap/socket/lmtp]: *** glibc detected *** lmtpd: double free or corruption (out): 0x2cdeadb0 *** : 4 Time(s) non-LMTP response from hermod.iguanait.com[/var/lib/imap/socket/lmtp]: *** glibc detected *** lmtpd: double free or corruption (out): 0x2cdeaf20 *** : 4 Time(s) non-LMTP response from hermod.iguanait.com[/var/lib/imap/socket/lmtp]: *** glibc detected *** lmtpd: double free or corruption (out): 0x2cdeb070 *** : 2 Time(s) non-LMTP response from hermod.iguanait.com[/var/lib/imap/socket/lmtp]: *** glibc detected *** lmtpd: double free or corruption (out): 0x2cdeb0c0 *** : 2 Time(s) non-LMTP response from hermod.iguanait.com[/var/lib/imap/socket/lmtp]: *** glibc detected *** lmtpd: double free or corruption (out): 0x2cdeb0d0 *** : 2 Time(s) non-LMTP response from hermod.iguanait.com[/var/lib/imap/socket/lmtp]: /lib64/libc.so.6(db.001 : 20 Time(s) non-LMTP response from hermod.iguanait.com[/var/lib/imap/socket/lmtp]: 2aac6000-2aacc000 rw-s fd:00 57246642 /var/lib/imap/db/db.002 : 20 Time(s) non-LMTP response from hermod.iguanait.com[/var/lib/imap/socket/lmtp]: 2aad3000-2ab75000 rw-s fd:00 57246649 /var/lib/imap/db/db.003 : 20 Time(s) non-LMTP response from hermod.iguanait.com[/var/lib/imap/socket/lmtp]: 2ab75000-2ab8d000 rw-s fd:00 fatal: open dictionary: expecting type:name form instead of /var/lib/imap/socket/lmtp fatal: open dictionary: expecting type:name form instead of /var/lib/imap/socket/lmtp fatal: open dictionary: expecting type:name form instead of /var/lib/imap/socket/lmtp fatal: open dictionary: expecting type:name form instead of /var/lib/imap/socket/lmtp fatal: open dictionary: expecting type:name form instead of /var/lib/imap/socket/lmtp fatal: open dictionary: expecting type:name form instead of /var/lib/imap/socket/lmtp When this happen with don't receive mails and if we send the people don't receive our mails. We have installed fedora 6,cyrus-imapd-2.3.9-6.fc6, postfix-2.3.3-2. If you need more info, please ask me, i will post the needed info. How can i solve these problems ? Is that some bug in cyrus? Thanks in advanced! Regards, Ali Nebi! Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html Hi, I've had a similar problem on a small Fedora 6 dedicated server recently
Re: *** glibc detected *** lmtpd: double free or corruption (out): 0x00002aaaacdeaa10 *** : 2 Time(s) problem
Thanks for the answers and tips. We decide to downgrade the version to the prior version. We don't have problem with it. I created a report in bugzilla system of fedora. I can't play in this server at the moment, because it is production server and this is not possible at this moment. this is the output from cat command: [EMAIL PROTECTED] log]# cat /proc/7065/smaps | grep ^[0-9] 2aaab000-2aac5000 r-xp fd:00 12451858 /lib64/ld-2.5.so 2aac5000-2aac6000 rw-p 2aac5000 00:00 0 2aad2000-2aad3000 rw-p 2aad2000 00:00 0 2acc4000-2acc5000 r--p 00019000 fd:00 12451858 /lib64/ld-2.5.so 2acc5000-2acc6000 rw-p 0001a000 fd:00 12451858 /lib64/ld-2.5.so 2acc6000-2add r-xp fd:00 6588074/usr/lib64/libnetsnmpmibs.so.10.0.1 2add-2afcf000 ---p 0010a000 fd:00 6588074/usr/lib64/libnetsnmpmibs.so.10.0.1 2afcf000-2afdb000 rw-p 00109000 fd:00 6588074/usr/lib64/libnetsnmpmibs.so.10.0.1 2afdb000-2b00d000 rw-p 2afdb000 00:00 0 2b00d000-2b04f000 r-xp fd:00 6587970/usr/lib64/libnetsnmpagent.so.10.0.1 2b04f000-2b24f000 ---p 00042000 fd:00 6587970/usr/lib64/libnetsnmpagent.so.10.0.1 2b24f000-2b251000 rw-p 00042000 fd:00 6587970/usr/lib64/libnetsnmpagent.so.10.0.1 2b251000-2b252000 rw-p 2b251000 00:00 0 2b252000-2b272000 r-xp fd:00 6588072/usr/lib64/libnetsnmphelpers.so.10.0.1 2b272000-2b472000 ---p 0002 fd:00 6588072/usr/lib64/libnetsnmphelpers.so.10.0.1 2b472000-2b473000 rw-p 0002 fd:00 6588072/usr/lib64/libnetsnmphelpers.so.10.0.1 2b473000-2b474000 rw-p 2b473000 00:00 0 2b474000-2b503000 r-xp fd:00 6587908/usr/lib64/libnetsnmp.so.10.0.1 2b503000-2b703000 ---p 0008f000 fd:00 6587908/usr/lib64/libnetsnmp.so.10.0.1 2b703000-2b707000 rw-p 0008f000 fd:00 6587908/usr/lib64/libnetsnmp.so.10.0.1 2b707000-2b73b000 rw-p 2b707000 00:00 0 2b73b000-2b73d000 r-xp fd:00 12451862 /lib64/libdl-2.5.so 2b73d000-2b93d000 ---p 2000 fd:00 12451862 /lib64/libdl-2.5.so 2b93d000-2b93e000 r--p 2000 fd:00 12451862 /lib64/libdl-2.5.so 2b93e000-2b93f000 rw-p 3000 fd:00 12451862 /lib64/libdl-2.5.so 2b93f000-2b997000 r-xp fd:00 6594770/usr/lib64/librpm-4.4.so 2b997000-2bb97000 ---p 00058000 fd:00 6594770/usr/lib64/librpm-4.4.so 2bb97000-2bb9c000 rw-p 00058000 fd:00 6594770/usr/lib64/librpm-4.4.so 2bb9c000-2bbd rw-p 2bb9c000 00:00 0 2bbd-2bc2e000 r-xp fd:00 6591613/usr/lib64/librpmio-4.4.so 2bc2e000-2be2d000 ---p 0005e000 fd:00 6591613/usr/lib64/librpmio-4.4.so 2be2d000-2be32000 rw-p 0005d000 fd:00 6591613/usr/lib64/librpmio-4.4.so 2be32000-2be55000 rw-p 2be32000 00:00 0 2be55000-2be5c000 r-xp fd:00 6599277/usr/lib64/libpopt.so.0.0.0 2be5c000-2c05c000 ---p 7000 fd:00 6599277/usr/lib64/libpopt.so.0.0.0 2c05c000-2c05d000 rw-p 7000 fd:00 6599277/usr/lib64/libpopt.so.0.0.0 2c05d000-2c071000 r-xp fd:00 6601317/usr/lib64/libz.so.1.2.3 2c071000-2c27 ---p 00014000 fd:00 6601317/usr/lib64/libz.so.1.2.3 2c27-2c271000 rw-p 00013000 fd:00 6601317/usr/lib64/libz.so.1.2.3 2c271000-2c272000 rw-p 2c271000 00:00 0 2c272000-2c397000 r-xp fd:00 12451859 /lib64/libcrypto.so.0.9.8b 2c397000-2c597000 ---p 00125000 fd:00 12451859 /lib64/libcrypto.so.0.9.8b 2c597000-2c5b6000 rw-p 00125000 fd:00 12451859 /lib64/libcrypto.so.0.9.8b 2c5b6000-2c5ba000 rw-p 2c5b6000 00:00 0 2c5ba000-2c63c000 r-xp fd:00 12451884 /lib64/libm-2.5.so 2c63c000-2c83b000 ---p 00082000 fd:00 12451884 /lib64/libm-2.5.so 2c83b000-2c83c000 r--p 00081000 fd:00 12451884 /lib64/libm-2.5.so 2c83c000-2c83d000 rw-p 00082000 fd:00 12451884 /lib64/libm-2.5.so 2c83d000-2c85a000 r-xp fd:00 6591615/usr/lib64/libsensors.so.3.1.1 2c85a000-2ca5a000 ---p 0001d000 fd:00 6591615/usr/lib64/libsensors.so.3.1.1 2ca5a000-2ca81000 rw-p 0001d000 fd:00 6591615
Re: *** glibc detected *** lmtpd: double free or corruption (out): 0x00002aaaacdeaa10 *** : 2 Time(s) problem
On Thu, Sep 13, 2007 at 02:32:51PM +0200, Frederic BRIAND wrote: There seems to be a trick to solve the problem, with an environment variable you must set, but I've not tested it. We moved to CentOS 5. No, you can NOT solve the problem that way. You're just digging your head in the sand pretending the problem does not exist while your data may be silently corrupted. Gabor -- - MTA SZTAKI Computer and Automation Research Institute Hungarian Academy of Sciences - Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: *** glibc detected *** lmtpd: double free or corruption (out): 0x00002aaaacdeaa10 *** : 2 Time(s) problem
Gabor Gombas wrote: On Thu, Sep 13, 2007 at 02:32:51PM +0200, Frederic BRIAND wrote: There seems to be a trick to solve the problem, with an environment variable you must set, but I've not tested it. We moved to CentOS 5. No, you can NOT solve the problem that way. You're just digging your head in the sand pretending the problem does not exist while your data may be silently corrupted. Yeah I know. That's the reason why we moved to CentOS, together with the fact that Fedora isn't a LTS distro. As I said, I've not even tried that supposed to be solution!!! FB Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: *** glibc detected *** lmtpd: double free or corruption (out): 0x00002aaaacdeaa10 *** : 2 Time(s) problem
On 9/13/07, Ali Nebi [EMAIL PROTECTED] wrote: Thanks for the answers and tips. We decide to downgrade the version to the prior version. We don't have problem with it. I created a report in bugzilla system of fedora. I can't play in this server at the moment, because it is production server and this is not possible at this moment. this is the output from cat command: [EMAIL PROTECTED] log]# cat /proc/7065/smaps | grep ^[0-9] 2aaab000-2aac5000 r-xp fd:00 12451858 /lib64/ld-2.5.so 2aac5000-2aac6000 rw-p 2aac5000 00:00 0 2aad2000-2aad3000 rw-p 2aad2000 00:00 0 2acc4000-2acc5000 r--p 00019000 fd:00 12451858 /lib64/ld-2.5.so 2acc5000-2acc6000 rw-p 0001a000 fd:00 12451858 /lib64/ld-2.5.so 2acc6000-2add r-xp fd:00 6588074/usr/lib64/libnetsnmpmibs.so.10.0.1 2add-2afcf000 ---p 0010a000 fd:00 6588074/usr/lib64/libnetsnmpmibs.so.10.0.1 2afcf000-2afdb000 rw-p 00109000 fd:00 6588074/usr/lib64/libnetsnmpmibs.so.10.0.1 2afdb000-2b00d000 rw-p 2afdb000 00:00 0 2b00d000-2b04f000 r-xp fd:00 6587970/usr/lib64/libnetsnmpagent.so.10.0.1 2b04f000-2b24f000 ---p 00042000 fd:00 6587970/usr/lib64/libnetsnmpagent.so.10.0.1 2b24f000-2b251000 rw-p 00042000 fd:00 6587970/usr/lib64/libnetsnmpagent.so.10.0.1 2b251000-2b252000 rw-p 2b251000 00:00 0 2b252000-2b272000 r-xp fd:00 6588072/usr/lib64/libnetsnmphelpers.so.10.0.1 2b272000-2b472000 ---p 0002 fd:00 6588072/usr/lib64/libnetsnmphelpers.so.10.0.1 2b472000-2b473000 rw-p 0002 fd:00 6588072/usr/lib64/libnetsnmphelpers.so.10.0.1 2b473000-2b474000 rw-p 2b473000 00:00 0 2b474000-2b503000 r-xp fd:00 6587908/usr/lib64/libnetsnmp.so.10.0.1 2b503000-2b703000 ---p 0008f000 fd:00 6587908/usr/lib64/libnetsnmp.so.10.0.1 2b703000-2b707000 rw-p 0008f000 fd:00 6587908/usr/lib64/libnetsnmp.so.10.0.1 2b707000-2b73b000 rw-p 2b707000 00:00 0 2b73b000-2b73d000 r-xp fd:00 12451862 /lib64/libdl-2.5.so 2b73d000-2b93d000 ---p 2000 fd:00 12451862 /lib64/libdl-2.5.so 2b93d000-2b93e000 r--p 2000 fd:00 12451862 /lib64/libdl-2.5.so 2b93e000-2b93f000 rw-p 3000 fd:00 12451862 /lib64/libdl-2.5.so 2b93f000-2b997000 r-xp fd:00 6594770/usr/lib64/librpm-4.4.so 2b997000-2bb97000 ---p 00058000 fd:00 6594770/usr/lib64/librpm-4.4.so 2bb97000-2bb9c000 rw-p 00058000 fd:00 6594770/usr/lib64/librpm-4.4.so 2bb9c000-2bbd rw-p 2bb9c000 00:00 0 2bbd-2bc2e000 r-xp fd:00 6591613/usr/lib64/librpmio-4.4.so 2bc2e000-2be2d000 ---p 0005e000 fd:00 6591613/usr/lib64/librpmio-4.4.so 2be2d000-2be32000 rw-p 0005d000 fd:00 6591613/usr/lib64/librpmio-4.4.so 2be32000-2be55000 rw-p 2be32000 00:00 0 2be55000-2be5c000 r-xp fd:00 6599277/usr/lib64/libpopt.so.0.0.0 2be5c000-2c05c000 ---p 7000 fd:00 6599277/usr/lib64/libpopt.so.0.0.0 2c05c000-2c05d000 rw-p 7000 fd:00 6599277/usr/lib64/libpopt.so.0.0.0 2c05d000-2c071000 r-xp fd:00 6601317/usr/lib64/libz.so.1.2.3 2c071000-2c27 ---p 00014000 fd:00 6601317/usr/lib64/libz.so.1.2.3 2c27-2c271000 rw-p 00013000 fd:00 6601317/usr/lib64/libz.so.1.2.3 2c271000-2c272000 rw-p 2c271000 00:00 0 2c272000-2c397000 r-xp fd:00 12451859 /lib64/libcrypto.so.0.9.8b 2c397000-2c597000 ---p 00125000 fd:00 12451859 /lib64/libcrypto.so.0.9.8b 2c597000-2c5b6000 rw-p 00125000 fd:00 12451859 /lib64/libcrypto.so.0.9.8b 2c5b6000-2c5ba000 rw-p 2c5b6000 00:00 0 2c5ba000-2c63c000 r-xp fd:00 12451884 /lib64/libm-2.5.so 2c63c000-2c83b000 ---p 00082000 fd:00 12451884 /lib64/libm-2.5.so 2c83b000-2c83c000 r--p 00081000 fd:00 12451884 /lib64/libm-2.5.so 2c83c000-2c83d000 rw-p 00082000 fd:00 12451884 /lib64/libm-2.5.so 2c83d000-2c85a000 r-xp fd:00 6591615/usr/lib64/libsensors.so.3.1.1 2c85a000-2ca5a000 ---p
*** glibc detected *** lmtpd: double free or corruption (out): 0x00002aaaacdeaa10 *** : 2 Time(s) problem
Hi all, i have some problems in a mail server and i don't know how to solve these problems. I get these errors: /usr/libexec/postfix/lmtp: bad command startup -- throttling : 588 Time(s) 198.231.4.83.rbl-plus.mail-abuse.org: RBL lookup error: Host or domain name not found. Name service error for name=198.231.4.83.rbl-plus.mail-abuse.org type=A: Host not found, try again : 2 Time(s) 3.77.214.79.rbl-plus.mail-abuse.org: RBL lookup error: Host or domain name not found. Name service error for name=3.77.214.79.rbl-plus.mail-abuse.org type=A: Host not found, try again : 2 Time(s) dict_ldap_lookup: Search error -5: Timed out : 2 Time(s) network_biopair_interop: error reading 5 bytes from the network: Connection reset by peer : 2 Time(s) non-LMTP response from hermod.iguanait.com[/var/lib/imap/socket/lmtp]: *** glibc detected *** lmtpd: double free or corruption (out): 0x2cdeaa10 *** : 2 Time(s) non-LMTP response from hermod.iguanait.com[/var/lib/imap/socket/lmtp]: *** glibc detected *** lmtpd: double free or corruption (out): 0x2cdeaa30 *** : 4 Time(s) non-LMTP response from hermod.iguanait.com[/var/lib/imap/socket/lmtp]: *** glibc detected *** lmtpd: double free or corruption (out): 0x2cdeabb0 *** : 2 Time(s) non-LMTP response from hermod.iguanait.com[/var/lib/imap/socket/lmtp]: *** glibc detected *** lmtpd: double free or corruption (out): 0x2cdeac10 *** : 2 Time(s) non-LMTP response from hermod.iguanait.com[/var/lib/imap/socket/lmtp]: *** glibc detected *** lmtpd: double free or corruption (out): 0x2cdeac50 *** : 2 Time(s) non-LMTP response from hermod.iguanait.com[/var/lib/imap/socket/lmtp]: *** glibc detected *** lmtpd: double free or corruption (out): 0x2cdeacb0 *** : 4 Time(s) non-LMTP response from hermod.iguanait.com[/var/lib/imap/socket/lmtp]: *** glibc detected *** lmtpd: double free or corruption (out): 0x2cdeacf0 *** : 2 Time(s) non-LMTP response from hermod.iguanait.com[/var/lib/imap/socket/lmtp]: *** glibc detected *** lmtpd: double free or corruption (out): 0x2cdead30 *** : 2 Time(s) non-LMTP response from hermod.iguanait.com[/var/lib/imap/socket/lmtp]: *** glibc detected *** lmtpd: double free or corruption (out): 0x2cdead70 *** : 4 Time(s) non-LMTP response from hermod.iguanait.com[/var/lib/imap/socket/lmtp]: *** glibc detected *** lmtpd: double free or corruption (out): 0x2cdeadb0 *** : 4 Time(s) non-LMTP response from hermod.iguanait.com[/var/lib/imap/socket/lmtp]: *** glibc detected *** lmtpd: double free or corruption (out): 0x2cdeaf20 *** : 4 Time(s) non-LMTP response from hermod.iguanait.com[/var/lib/imap/socket/lmtp]: *** glibc detected *** lmtpd: double free or corruption (out): 0x2cdeb070 *** : 2 Time(s) non-LMTP response from hermod.iguanait.com[/var/lib/imap/socket/lmtp]: *** glibc detected *** lmtpd: double free or corruption (out): 0x2cdeb0c0 *** : 2 Time(s) non-LMTP response from hermod.iguanait.com[/var/lib/imap/socket/lmtp]: *** glibc detected *** lmtpd: double free or corruption (out): 0x2cdeb0d0 *** : 2 Time(s) non-LMTP response from hermod.iguanait.com[/var/lib/imap/socket/lmtp]: /lib64/libc.so.6(db.001 : 20 Time(s) non-LMTP response from hermod.iguanait.com[/var/lib/imap/socket/lmtp]: 2aac6000-2aacc000 rw-s fd:00 57246642 /var/lib/imap/db/db.002 : 20 Time(s) non-LMTP response from hermod.iguanait.com[/var/lib/imap/socket/lmtp]: 2aad3000-2ab75000 rw-s fd:00 57246649 /var/lib/imap/db/db.003 : 20 Time(s) non-LMTP response from hermod.iguanait.com[/var/lib/imap/socket/lmtp]: 2ab75000-2ab8d000 rw-s fd:00 fatal: open dictionary: expecting type:name form instead of /var/lib/imap/socket/lmtp fatal: open dictionary: expecting type:name form instead of /var/lib/imap/socket/lmtp fatal: open dictionary: expecting type:name form instead of /var/lib/imap/socket/lmtp fatal: open dictionary: expecting type:name form instead of /var/lib/imap/socket/lmtp fatal: open dictionary: expecting type:name form instead of /var/lib/imap/socket/lmtp fatal: open dictionary: expecting type:name form instead of /var/lib/imap/socket/lmtp When this happen with don't receive mails and if we send the people don't receive our mails. We have installed fedora 6,cyrus-imapd-2.3.9-6.fc6, postfix-2.3.3-2. If you need more info, please ask me, i will post the needed info. How can i solve these problems ? Is that some bug in cyrus? Thanks in advanced! Regards, Ali Nebi! Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: *** glibc detected *** lmtpd: double free or corruption (out): 0x00002aaaacdeaa10 *** : 2 Time(s) problem
Thanks for the fast answer. This mean this is an occasional error, not a permanent one ? What do you do to solve this ? We get these messages last week only, before that we don't get this kind of messages in the logs. About our sent messages the people receive them when mail server is working, the messages are in the queue i suppose. I'm trying to understand what is the problem, i can't do anything at this moment. We get also these messages in the maillog: Sep 12 12:57:49 hermod postfix/lmtp[14429]: warning: non-LMTP response from hermod.iguanait.com[/var/lib/imap/socket/lmtp]:2c385000-2c386000 rw-p 00012000 fd:00 12452021 /lib64/libresolv-2.5.so Sep 12 12:57:49 hermod postfix/lmtp[14429]: warning: to prevent loss of mail, turn off command pipelining for /var/lib/imap/socket/lmtp with the lmtp_discard_lhlo_keyword_address_maps parameter Sep 12 12:57:49 hermod postfix/lmtp[14429]: warning: non-LMTP response from hermod.iguanait.com[/var/lib/imap/socket/lmtp]:2c386000-2c388000 rw-p 2c386000 00:00 0 Sep 12 12:57:49 hermod postfix/lmtp[14429]: warning: to prevent loss of mail, turn off command pipelining for /var/lib/imap/socket/lmtp with the lmtp_discard_lhlo_keyword_address_maps parameter Sep 12 12:57:49 hermod postfix/lmtp[14429]: warning: non-LMTP response fromhermod.iguanait.com[/var/lib/imap/socket/lmtp]:2c388000-2c479000 r-xp fd:00 12452057 /lib64/libdb-4.3.so Sep 12 12:57:49 hermod postfix/lmtp[14429]: warning: to prevent loss of mail, turn off command pipelining for /var/lib/imap/socket/lmtp with the lmtp_discard_lhlo_keyword_address_maps parameter Sep 12 12:57:49 hermod postfix/lmtp[14429]: warning: non-LMTP response from hermod.iguanait.com[/var/lib/imap/socket/lmtp]: 2c67d000-2c685000 r-xp fd:00 6590877/usr/lib64/libwrap.so.0.7.6 I increased only proccess limits to be 200. There we have running mysql only. Yes, the hardware is ok. We have 4 GB RAM, 8 GB Swap space, the machines are ok. I think that this is not related with bad RAM or something similar, because if there is some bad block memory, then probably we will receive this kind of messages from other applications too. What can i do in this situation? Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: *** glibc detected *** lmtpd: double free or corruption (out): 0x00002aaaacdeaa10 *** : 2 Time(s) problem
What I understand is this : When postfix try to deliver an email, it open a LMTP connection with cyrus and cyrus, instead of speaking correct LMTP reply with garbage. BUT 2c388000-2c479000 r-xp fd:00 12452057 /lib64/libdb-4.3.so is not realy garbage this look like to be the memory areas of a process. Try a # cat /proc//smaps | grep ^[0-9] where is any valid process PID, you will see :-) How do you stop/start cyrus ? Do you use the usual sysV startup scripts ? If so could you shuthdown cyrus, login on the console of the computer, (not a ssh or a telnet) and start cyrus using # /usr/lib/cyrus-imapd/cyrus-master -d and see if the problem append again On 9/12/07, Ali Nebi [EMAIL PROTECTED] wrote: Thanks for the fast answer. This mean this is an occasional error, not a permanent one ? What do you do to solve this ? We get these messages last week only, before that we don't get this kind of messages in the logs. About our sent messages the people receive them when mail server is working, the messages are in the queue i suppose. I'm trying to understand what is the problem, i can't do anything at this moment. We get also these messages in the maillog: Sep 12 12:57:49 hermod postfix/lmtp[14429]: warning: non-LMTP response from hermod.iguanait.com[/var/lib/imap/socket/lmtp]:2c385000-2c386000 rw-p 00012000 fd:00 12452021 /lib64/libresolv-2.5.so Sep 12 12:57:49 hermod postfix/lmtp[14429]: warning: to prevent loss of mail, turn off command pipelining for /var/lib/imap/socket/lmtp with the lmtp_discard_lhlo_keyword_address_maps parameter Sep 12 12:57:49 hermod postfix/lmtp[14429]: warning: non-LMTP response from hermod.iguanait.com[/var/lib/imap/socket/lmtp]:2c386000-2c388000 rw-p 2c386000 00:00 0 Sep 12 12:57:49 hermod postfix/lmtp[14429]: warning: to prevent loss of mail, turn off command pipelining for /var/lib/imap/socket/lmtp with the lmtp_discard_lhlo_keyword_address_maps parameter Sep 12 12:57:49 hermod postfix/lmtp[14429]: warning: non-LMTP response fromhermod.iguanait.com[/var/lib/imap/socket/lmtp]:2c388000-2c479000 r-xp fd:00 12452057 /lib64/libdb-4.3.so Sep 12 12:57:49 hermod postfix/lmtp[14429]: warning: to prevent loss of mail, turn off command pipelining for /var/lib/imap/socket/lmtp with the lmtp_discard_lhlo_keyword_address_maps parameter Sep 12 12:57:49 hermod postfix/lmtp[14429]: warning: non-LMTP response from hermod.iguanait.com[/var/lib/imap/socket/lmtp]: 2c67d000-2c685000 r-xp fd:00 6590877/usr/lib64/libwrap.so.0.7.6 I increased only proccess limits to be 200. There we have running mysql only. Yes, the hardware is ok. We have 4 GB RAM, 8 GB Swap space, the machines are ok. I think that this is not related with bad RAM or something similar, because if there is some bad block memory, then probably we will receive this kind of messages from other applications too. What can i do in this situation? Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html -- Alain Spineux aspineux gmail com May the sources be with you Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Mailbox list corruption
On Tue, Jun 26, 2007 at 08:17:15PM +0300, Janne Peltonen wrote: Hi! Sometimes I get the following while trying to restart a murder member: Jun 26 16:12:42 pcn4.mappi.helsinki.fi i10/master[21932]: exiting on SIGTERM/SIGINT Jun 26 16:24:28 pcn4.mappi.helsinki.fi i10/master[19054]: process started Jun 26 16:24:28 pcn4.mappi.helsinki.fi i10/ctl_cyrusdb[19056]: recovering cyrus databases Jun 26 16:24:39 pcn4.mappi.helsinki.fi i10/ctl_cyrusdb[19056]: DBERROR: skiplist recovery /var/lib/imap/i 10/mailboxes.db: 5BE0D50 should be ADD or DELETE Jun 26 16:24:39 pcn4.mappi.helsinki.fi i10/ctl_cyrusdb[19056]: DBERROR: opening /var/lib/imap/i10/mailbox es.db: cyrusdb error On Tue, Jun 26, 2007 at 10:10:11PM +0300, Janne Peltonen wrote: Hi. I've got a line like the following in my cyrus.conf files: mupdatepush cmd=/usr/lib/cyrus-imapd/ctl_mboxlist -C /etc/imapd.conf.i01.master -m It is meant to synchronize the backend mailbox list with the mupdate master upon startup. Sometimes, it takes horribly long to complete: one to three hours with 60 records in the mailboxes list. (Sometimes it takes a lot less time, something like 9 seconds). There are 24 nodes in my murder. Could this create locking problems on the mupdate master? Apparently, these were both the result of ill-considered mount options on the config partitions. *sigh* There was a colleague that recommended setting the 'noatime, data=writeback' options on all cyrus filesystems during migration from our old system to the new one. Oh, well... --Janne -- Janne Peltonen [EMAIL PROTECTED] Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Mailbox list corruption
On Wed, 27 Jun 2007, Janne Peltonen wrote: On Tue, Jun 26, 2007 at 08:17:15PM +0300, Janne Peltonen wrote: Hi! Sometimes I get the following while trying to restart a murder member: Jun 26 16:12:42 pcn4.mappi.helsinki.fi i10/master[21932]: exiting on SIGTERM/SIGINT Jun 26 16:24:28 pcn4.mappi.helsinki.fi i10/master[19054]: process started Jun 26 16:24:28 pcn4.mappi.helsinki.fi i10/ctl_cyrusdb[19056]: recovering cyrus databases Jun 26 16:24:39 pcn4.mappi.helsinki.fi i10/ctl_cyrusdb[19056]: DBERROR: skiplist recovery /var/lib/imap/i 10/mailboxes.db: 5BE0D50 should be ADD or DELETE Jun 26 16:24:39 pcn4.mappi.helsinki.fi i10/ctl_cyrusdb[19056]: DBERROR: opening /var/lib/imap/i10/mailbox es.db: cyrusdb error On Tue, Jun 26, 2007 at 10:10:11PM +0300, Janne Peltonen wrote: Hi. I've got a line like the following in my cyrus.conf files: mupdatepush cmd=/usr/lib/cyrus-imapd/ctl_mboxlist -C /etc/imapd.conf.i01.master -m It is meant to synchronize the backend mailbox list with the mupdate master upon startup. Sometimes, it takes horribly long to complete: one to three hours with 60 records in the mailboxes list. (Sometimes it takes a lot less time, something like 9 seconds). There are 24 nodes in my murder. Could this create locking problems on the mupdate master? Apparently, these were both the result of ill-considered mount options on the config partitions. *sigh* There was a colleague that recommended setting the 'noatime, data=writeback' options on all cyrus filesystems during migration from our old system to the new one. Oh, well... I'm running Cyrus 2.2.13 in a murder configuration. My entire Cyrus directory (spool and config) in on an ext3 partition with the following mount options: /dev/sdb1 on /private type ext3 (rw,noatime,data=ordered) You shouldn't have any trouble with the noatime option (it gives a good performance boost). data=writeback gives higher performance, but may allow corruption of files if you experience a system crash. If you didn't have a system crash or unclean umount of the filesystem, then I am at a loss for why you would experience corruption. Andy Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Mailbox list corruption
On Wed, Jun 27, 2007 at 10:15:08AM -0700, Andrew Morgan wrote: I'm running Cyrus 2.2.13 in a murder configuration. My entire Cyrus directory (spool and config) in on an ext3 partition with the following mount options: /dev/sdb1 on /private type ext3 (rw,noatime,data=ordered) You shouldn't have any trouble with the noatime option (it gives a good performance boost). data=writeback gives higher performance, but may allow corruption of files if you experience a system crash. If you didn't have a system crash or unclean umount of the filesystem, then I am at a loss for why you would experience corruption. There /were/ unclean umounts, which probably corrupted the lists in the first place. I'm also at a loss for why it takes so long to do ctl_mboxlist and friends with the given options - without them, it takes only a couple of minutes. As does the mupdate slave synchronization thereafter (I'm using the unified config with 2.3.8, invoca.ch rpm). --Janne -- Janne Peltonen [EMAIL PROTECTED] Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Mailbox list corruption
Hi! Sometimes I get the following while trying to restart a murder member: Jun 26 16:12:42 pcn4.mappi.helsinki.fi i10/master[21932]: exiting on SIGTERM/SIGINT Jun 26 16:24:28 pcn4.mappi.helsinki.fi i10/master[19054]: process started Jun 26 16:24:28 pcn4.mappi.helsinki.fi i10/ctl_cyrusdb[19056]: recovering cyrus databases Jun 26 16:24:39 pcn4.mappi.helsinki.fi i10/ctl_cyrusdb[19056]: DBERROR: skiplist recovery /var/lib/imap/i 10/mailboxes.db: 5BE0D50 should be ADD or DELETE Jun 26 16:24:39 pcn4.mappi.helsinki.fi i10/ctl_cyrusdb[19056]: DBERROR: opening /var/lib/imap/i10/mailbox es.db: cyrusdb error Jun 26 16:24:39 pcn4.mappi.helsinki.fi i10/master[19054]: process 19056 exited, status 75 Jun 26 16:24:46 pcn4.mappi.helsinki.fi i10/idled[19922]: DBERROR: skiplist recovery /var/lib/imap/i10/mai lboxes.db: 5BE0D50 should be ADD or DELETE Jun 26 16:24:46 pcn4.mappi.helsinki.fi i10/idled[19922]: DBERROR: opening /var/lib/imap/i10/mailboxes.db: cyrusdb error Jun 26 16:24:46 pcn4.mappi.helsinki.fi i10/idled[19922]: can't read mailboxes file Jun 26 16:24:46 pcn4.mappi.helsinki.fi i10/idled[19922]: exiting Jun 26 16:24:46 pcn4.mappi.helsinki.fi i10/master[19054]: process 19922 exited, status 75 Jun 26 16:24:52 pcn4.mappi.helsinki.fi i10/ctl_mboxlist[22067]: DBERROR: skiplist recovery /var/lib/imap/ i10/mailboxes.db: 5BE0D50 should be ADD or DELETE Jun 26 16:24:52 pcn4.mappi.helsinki.fi i10/ctl_mboxlist[22067]: DBERROR: opening /var/lib/imap/i10/mailbo xes.db: cyrusdb error Jun 26 16:24:52 pcn4.mappi.helsinki.fi i10/master[19054]: process 22067 exited, status 75 Jun 26 16:24:52 pcn4.mappi.helsinki.fi i10/tls_prune[24771]: skiplist: recovered /var/lib/imap/i10/tls_se ssions.db (8 records, 423060 bytes) in 0 seconds Jun 26 16:24:52 pcn4.mappi.helsinki.fi i10/tls_prune[24771]: tls_prune: purged 0 out of 8 entries Jun 26 16:24:59 pcn4.mappi.helsinki.fi i10/sync_client[24772]: DBERROR: skiplist recovery /var/lib/imap/i 10/mailboxes.db: 5BE0D50 should be ADD or DELETE Jun 26 16:24:59 pcn4.mappi.helsinki.fi i10/sync_client[24772]: DBERROR: opening /var/lib/imap/i10/mailbox es.db: cyrusdb error Jun 26 16:24:59 pcn4.mappi.helsinki.fi i10/master[19054]: process 24772 exited, status 75 Jun 26 16:24:59 pcn4.mappi.helsinki.fi i10/master[19054]: ready for work Jun 26 16:24:59 pcn4.mappi.helsinki.fi i10/ctl_cyrusdb[24902]: checkpointing cyrus databases Jun 26 16:25:00 pcn4.mappi.helsinki.fi i10/ctl_cyrusdb[24902]: done checkpointing cyrus databases Jun 26 16:25:02 pcn4.mappi.helsinki.fi i10/lmtp[24908]: skiplist: recovered /var/lib/imap/i10/deliver.db (0 records, 144 bytes) in 3 seconds Jun 26 16:25:05 pcn4.mappi.helsinki.fi i10/pop3[24905]: DBERROR: skiplist recovery /var/lib/imap/i10/mail boxes.db: 5BE0D50 should be ADD or DELETE Jun 26 16:25:06 pcn4.mappi.helsinki.fi i10/pop3[24905]: DBERROR: opening /var/lib/imap/i10/mailboxes.db: cyrusdb error Jun 26 16:25:06 pcn4.mappi.helsinki.fi i10/pop3[24905]: Fatal error: can't read mailboxes file Jun 26 16:25:06 pcn4.mappi.helsinki.fi i10/master[19054]: service pop3 pid 24905 in READY state: terminat ed abnormally Jun 26 16:25:17 pcn4.mappi.helsinki.fi i10/lmtp[25043]: DBERROR: skiplist recovery /var/lib/imap/i10/mail boxes.db: 5BE0D50 should be ADD or DELETE Jun 26 16:25:17 pcn4.mappi.helsinki.fi i10/lmtp[25043]: DBERROR: opening /var/lib/imap/i10/mailboxes.db: cyrusdb error Jun 26 16:25:17 pcn4.mappi.helsinki.fi i10/lmtp[25043]: FATAL: can't read mailboxes file Jun 26 16:25:17 pcn4.mappi.helsinki.fi i10/master[19054]: service lmtp pid 25043 in READY state: terminat ed abnormally Jun 26 16:25:29 pcn4.mappi.helsinki.fi i10/imap[24924]: DBERROR: skiplist recovery /var/lib/imap/i10/mail boxes.db: 5BE0D50 should be ADD or DELETE Jun 26 16:25:29 pcn4.mappi.helsinki.fi i10/imap[24924]: DBERROR: opening /var/lib/imap/i10/mailboxes.db: cyrusdb error Jun 26 16:25:29 pcn4.mappi.helsinki.fi i10/imap[24924]: Fatal error: can't read mailboxes file Jun 26 16:25:29 pcn4.mappi.helsinki.fi i10/master[19054]: service imap pid 24924 in READY state: terminat ed abnormally etc. ad nauseam. I'm using the unified murder config. Any insight would be appreciated. --Janne -- Janne Peltonen [EMAIL PROTECTED] Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: corruption when using sieve vacation message?
Hi Edwin, It's strange that this corrupts the duplicate DB, and I don't know why it could do that. Vacation won't work without a sendmail given, though. Maybe the problem will disappear if vacation works fine. Try correcting that by specifying the path to sendmail in imapd.conf - like sendmail: /usr/sbin/sendmail Baltasar -- Baltasar Cevc _ FORMER 03 GmbH _ infanteriestraße 19 haus 6 eg _ D-80797 muenchen _ http://www.former03.de PGP.sig Description: This is a digitally signed message part Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
corruption when using sieve vacation message?
using cyrus imap 2.2.12. We've been using cyrus imap on many servers for serveral years without problems, however this is the first time we tried to use Sieve. Any ideas as to how this corruption might happen? Many thanks! Edwin Eefting www.syn-3.nl Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: lots of corruption of delived.db file
Well the problem is that during the recovery the delivery from my postfix servers is stopped. and it can be very annoying when the recover last around one hour users are complaning that mail didn't be delivered in time Andrew Morgan a écrit : What makes you think the log messages indicate corruption? Those are pretty standard messages if you log at DEBUG level. Andy On Tue, 13 Mar 2007, Eric Doutreleau wrote: Peter P. Benac a écrit : I have seen it. Just curious did you update your system lately. I updated mine yesterday for the new Timezone changes and have had trouble since then. Nothing like waiting till the last minute, but I had to do it. Regards, Pete Peter P. Benac, CCNA Emacolet Networking Services, Inc Providing Network and Systems Project Management and Installation and Web Hosting. Phone: 919-618-2557 Web: http://www.emacolet.com Need quick reliable Systems or Network Management advice visit http://www.nmsusers.org To have principles... First have courage.. With principles comes integrity!!! -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Eric Doutreleau Sent: Monday, March 12, 2007 10:43 To: info-cyrus@lists.andrew.cmu.edu Subject: lots of corruption of delived.db file I'm using cyrus2.3.8 on a RHEL4 machine and from time to time i got massive corruption of deliverd.db file for example today Mar 12 15:27:13 pasargades lmtp[31407]: skiplist: checkpointed /var/lib/imap/deliver.db (1282 records, 123104 bytes) in 5 seconds Mar 12 15:27:18 pasargades lmtp[30946]: skiplist: recovered /var/lib/imap/deliver.db (1282 records, 123104 bytes) in 5 seconds Mar 12 15:27:24 pasargades lmtp[30948]: skiplist: checkpointed /var/lib/imap/deliver.db (1289 records, 123800 bytes) in 6 seconds Mar 12 15:27:28 pasargades lmtp[31407]: skiplist: recovered /var/lib/imap/deliver.db (1289 records, 123800 bytes) in 4 seconds Mar 12 15:27:35 pasargades lmtp[30981]: skiplist: checkpointed /var/lib/imap/deliver.db (1300 records, 124788 bytes) in 7 seconds Mar 12 15:27:40 pasargades lmtp[31677]: skiplist: recovered /var/lib/imap/deliver.db (1300 records, 124788 bytes) in 5 seconds Mar 12 15:28:20 pasargades lmtp[31679]: skiplist: checkpointed /var/lib/imap/deliver.db (2541 records, 244756 bytes) in 11 seconds Mar 12 15:28:26 pasargades lmtp[31675]: skiplist: recovered /var/lib/imap/deliver.db (2541 records, 244756 bytes) in 6 seconds Mar 12 15:28:38 pasargades lmtp[30947]: skiplist: checkpointed /var/lib/imap/deliver.db (2578 records, 248632 bytes) in 11 seconds Mar 12 15:28:45 pasargades lmtp[30951]: skiplist: recovered /var/lib/imap/deliver.db (2578 records, 248632 bytes) in 7 seconds Mar 12 15:29:01 pasargades lmtp[31676]: skiplist: checkpointed /var/lib/imap/deliver.db (2622 records, 253440 bytes) in 15 seconds Mar 12 15:29:08 pasargades lmtp[31678]: skiplist: recovered /var/lib/imap/deliver.db (2622 records, 253440 bytes) in 7 seconds Mar 12 15:29:22 pasargades lmtp[31462]: skiplist: checkpointed /var/lib/imap/deliver.db (2699 records, 260880 bytes) in 12 seconds Mar 12 15:29:31 pasargades lmtp[31678]: skiplist: recovered /var/lib/imap/deliver.db (2699 records, 260880 bytes) in 9 seconds Mar 12 15:29:43 pasargades lmtp[30952]: skiplist: checkpointed /var/lib/imap/deliver.db (2706 records, 261556 bytes) in 12 seconds Mar 12 15:29:49 pasargades lmtp[31679]: skiplist: recovered /var/lib/imap/deliver.db (2706 records, 261556 bytes) in 6 seconds Mar 12 15:30:03 pasargades lmtp[30950]: skiplist: checkpointed /var/lib/imap/deliver.db (2720 records, 263040 bytes) in 14 seconds Mar 12 15:30:11 pasargades lmtp[30981]: skiplist: recovered /var/lib/imap/deliver.db (2720 records, 263040 bytes) in 8 seconds cyrus recover by itself but the problem is that when the deliver.db grows it take about one hour to recover and the delivery of mail is stopped. Has someone already seen that and have an idea on how to solve the problem. thank in advance for any help Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html No i didn't update my timezone. I always had these kind of problem but the intensity is variable Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: lots of corruption of delived.db file
Peter P. Benac a écrit : I have seen it. Just curious did you update your system lately. I updated mine yesterday for the new Timezone changes and have had trouble since then. Nothing like waiting till the last minute, but I had to do it. Regards, Pete Peter P. Benac, CCNA Emacolet Networking Services, Inc Providing Network and Systems Project Management and Installation and Web Hosting. Phone: 919-618-2557 Web: http://www.emacolet.com Need quick reliable Systems or Network Management advice visit http://www.nmsusers.org To have principles... First have courage.. With principles comes integrity!!! -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Eric Doutreleau Sent: Monday, March 12, 2007 10:43 To: info-cyrus@lists.andrew.cmu.edu Subject: lots of corruption of delived.db file I'm using cyrus2.3.8 on a RHEL4 machine and from time to time i got massive corruption of deliverd.db file for example today Mar 12 15:27:13 pasargades lmtp[31407]: skiplist: checkpointed /var/lib/imap/deliver.db (1282 records, 123104 bytes) in 5 seconds Mar 12 15:27:18 pasargades lmtp[30946]: skiplist: recovered /var/lib/imap/deliver.db (1282 records, 123104 bytes) in 5 seconds Mar 12 15:27:24 pasargades lmtp[30948]: skiplist: checkpointed /var/lib/imap/deliver.db (1289 records, 123800 bytes) in 6 seconds Mar 12 15:27:28 pasargades lmtp[31407]: skiplist: recovered /var/lib/imap/deliver.db (1289 records, 123800 bytes) in 4 seconds Mar 12 15:27:35 pasargades lmtp[30981]: skiplist: checkpointed /var/lib/imap/deliver.db (1300 records, 124788 bytes) in 7 seconds Mar 12 15:27:40 pasargades lmtp[31677]: skiplist: recovered /var/lib/imap/deliver.db (1300 records, 124788 bytes) in 5 seconds Mar 12 15:28:20 pasargades lmtp[31679]: skiplist: checkpointed /var/lib/imap/deliver.db (2541 records, 244756 bytes) in 11 seconds Mar 12 15:28:26 pasargades lmtp[31675]: skiplist: recovered /var/lib/imap/deliver.db (2541 records, 244756 bytes) in 6 seconds Mar 12 15:28:38 pasargades lmtp[30947]: skiplist: checkpointed /var/lib/imap/deliver.db (2578 records, 248632 bytes) in 11 seconds Mar 12 15:28:45 pasargades lmtp[30951]: skiplist: recovered /var/lib/imap/deliver.db (2578 records, 248632 bytes) in 7 seconds Mar 12 15:29:01 pasargades lmtp[31676]: skiplist: checkpointed /var/lib/imap/deliver.db (2622 records, 253440 bytes) in 15 seconds Mar 12 15:29:08 pasargades lmtp[31678]: skiplist: recovered /var/lib/imap/deliver.db (2622 records, 253440 bytes) in 7 seconds Mar 12 15:29:22 pasargades lmtp[31462]: skiplist: checkpointed /var/lib/imap/deliver.db (2699 records, 260880 bytes) in 12 seconds Mar 12 15:29:31 pasargades lmtp[31678]: skiplist: recovered /var/lib/imap/deliver.db (2699 records, 260880 bytes) in 9 seconds Mar 12 15:29:43 pasargades lmtp[30952]: skiplist: checkpointed /var/lib/imap/deliver.db (2706 records, 261556 bytes) in 12 seconds Mar 12 15:29:49 pasargades lmtp[31679]: skiplist: recovered /var/lib/imap/deliver.db (2706 records, 261556 bytes) in 6 seconds Mar 12 15:30:03 pasargades lmtp[30950]: skiplist: checkpointed /var/lib/imap/deliver.db (2720 records, 263040 bytes) in 14 seconds Mar 12 15:30:11 pasargades lmtp[30981]: skiplist: recovered /var/lib/imap/deliver.db (2720 records, 263040 bytes) in 8 seconds cyrus recover by itself but the problem is that when the deliver.db grows it take about one hour to recover and the delivery of mail is stopped. Has someone already seen that and have an idea on how to solve the problem. thank in advance for any help Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html No i didn't update my timezone. I always had these kind of problem but the intensity is variable Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: lots of corruption of delived.db file
What makes you think the log messages indicate corruption? Those are pretty standard messages if you log at DEBUG level. Andy On Tue, 13 Mar 2007, Eric Doutreleau wrote: Peter P. Benac a écrit : I have seen it. Just curious did you update your system lately. I updated mine yesterday for the new Timezone changes and have had trouble since then. Nothing like waiting till the last minute, but I had to do it. Regards, Pete Peter P. Benac, CCNA Emacolet Networking Services, Inc Providing Network and Systems Project Management and Installation and Web Hosting. Phone: 919-618-2557 Web: http://www.emacolet.com Need quick reliable Systems or Network Management advice visit http://www.nmsusers.org To have principles... First have courage.. With principles comes integrity!!! -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Eric Doutreleau Sent: Monday, March 12, 2007 10:43 To: info-cyrus@lists.andrew.cmu.edu Subject: lots of corruption of delived.db file I'm using cyrus2.3.8 on a RHEL4 machine and from time to time i got massive corruption of deliverd.db file for example today Mar 12 15:27:13 pasargades lmtp[31407]: skiplist: checkpointed /var/lib/imap/deliver.db (1282 records, 123104 bytes) in 5 seconds Mar 12 15:27:18 pasargades lmtp[30946]: skiplist: recovered /var/lib/imap/deliver.db (1282 records, 123104 bytes) in 5 seconds Mar 12 15:27:24 pasargades lmtp[30948]: skiplist: checkpointed /var/lib/imap/deliver.db (1289 records, 123800 bytes) in 6 seconds Mar 12 15:27:28 pasargades lmtp[31407]: skiplist: recovered /var/lib/imap/deliver.db (1289 records, 123800 bytes) in 4 seconds Mar 12 15:27:35 pasargades lmtp[30981]: skiplist: checkpointed /var/lib/imap/deliver.db (1300 records, 124788 bytes) in 7 seconds Mar 12 15:27:40 pasargades lmtp[31677]: skiplist: recovered /var/lib/imap/deliver.db (1300 records, 124788 bytes) in 5 seconds Mar 12 15:28:20 pasargades lmtp[31679]: skiplist: checkpointed /var/lib/imap/deliver.db (2541 records, 244756 bytes) in 11 seconds Mar 12 15:28:26 pasargades lmtp[31675]: skiplist: recovered /var/lib/imap/deliver.db (2541 records, 244756 bytes) in 6 seconds Mar 12 15:28:38 pasargades lmtp[30947]: skiplist: checkpointed /var/lib/imap/deliver.db (2578 records, 248632 bytes) in 11 seconds Mar 12 15:28:45 pasargades lmtp[30951]: skiplist: recovered /var/lib/imap/deliver.db (2578 records, 248632 bytes) in 7 seconds Mar 12 15:29:01 pasargades lmtp[31676]: skiplist: checkpointed /var/lib/imap/deliver.db (2622 records, 253440 bytes) in 15 seconds Mar 12 15:29:08 pasargades lmtp[31678]: skiplist: recovered /var/lib/imap/deliver.db (2622 records, 253440 bytes) in 7 seconds Mar 12 15:29:22 pasargades lmtp[31462]: skiplist: checkpointed /var/lib/imap/deliver.db (2699 records, 260880 bytes) in 12 seconds Mar 12 15:29:31 pasargades lmtp[31678]: skiplist: recovered /var/lib/imap/deliver.db (2699 records, 260880 bytes) in 9 seconds Mar 12 15:29:43 pasargades lmtp[30952]: skiplist: checkpointed /var/lib/imap/deliver.db (2706 records, 261556 bytes) in 12 seconds Mar 12 15:29:49 pasargades lmtp[31679]: skiplist: recovered /var/lib/imap/deliver.db (2706 records, 261556 bytes) in 6 seconds Mar 12 15:30:03 pasargades lmtp[30950]: skiplist: checkpointed /var/lib/imap/deliver.db (2720 records, 263040 bytes) in 14 seconds Mar 12 15:30:11 pasargades lmtp[30981]: skiplist: recovered /var/lib/imap/deliver.db (2720 records, 263040 bytes) in 8 seconds cyrus recover by itself but the problem is that when the deliver.db grows it take about one hour to recover and the delivery of mail is stopped. Has someone already seen that and have an idea on how to solve the problem. thank in advance for any help Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html No i didn't update my timezone. I always had these kind of problem but the intensity is variable Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
lots of corruption of delived.db file
I'm using cyrus2.3.8 on a RHEL4 machine and from time to time i got massive corruption of deliverd.db file for example today Mar 12 15:27:13 pasargades lmtp[31407]: skiplist: checkpointed /var/lib/imap/deliver.db (1282 records, 123104 bytes) in 5 seconds Mar 12 15:27:18 pasargades lmtp[30946]: skiplist: recovered /var/lib/imap/deliver.db (1282 records, 123104 bytes) in 5 seconds Mar 12 15:27:24 pasargades lmtp[30948]: skiplist: checkpointed /var/lib/imap/deliver.db (1289 records, 123800 bytes) in 6 seconds Mar 12 15:27:28 pasargades lmtp[31407]: skiplist: recovered /var/lib/imap/deliver.db (1289 records, 123800 bytes) in 4 seconds Mar 12 15:27:35 pasargades lmtp[30981]: skiplist: checkpointed /var/lib/imap/deliver.db (1300 records, 124788 bytes) in 7 seconds Mar 12 15:27:40 pasargades lmtp[31677]: skiplist: recovered /var/lib/imap/deliver.db (1300 records, 124788 bytes) in 5 seconds Mar 12 15:28:20 pasargades lmtp[31679]: skiplist: checkpointed /var/lib/imap/deliver.db (2541 records, 244756 bytes) in 11 seconds Mar 12 15:28:26 pasargades lmtp[31675]: skiplist: recovered /var/lib/imap/deliver.db (2541 records, 244756 bytes) in 6 seconds Mar 12 15:28:38 pasargades lmtp[30947]: skiplist: checkpointed /var/lib/imap/deliver.db (2578 records, 248632 bytes) in 11 seconds Mar 12 15:28:45 pasargades lmtp[30951]: skiplist: recovered /var/lib/imap/deliver.db (2578 records, 248632 bytes) in 7 seconds Mar 12 15:29:01 pasargades lmtp[31676]: skiplist: checkpointed /var/lib/imap/deliver.db (2622 records, 253440 bytes) in 15 seconds Mar 12 15:29:08 pasargades lmtp[31678]: skiplist: recovered /var/lib/imap/deliver.db (2622 records, 253440 bytes) in 7 seconds Mar 12 15:29:22 pasargades lmtp[31462]: skiplist: checkpointed /var/lib/imap/deliver.db (2699 records, 260880 bytes) in 12 seconds Mar 12 15:29:31 pasargades lmtp[31678]: skiplist: recovered /var/lib/imap/deliver.db (2699 records, 260880 bytes) in 9 seconds Mar 12 15:29:43 pasargades lmtp[30952]: skiplist: checkpointed /var/lib/imap/deliver.db (2706 records, 261556 bytes) in 12 seconds Mar 12 15:29:49 pasargades lmtp[31679]: skiplist: recovered /var/lib/imap/deliver.db (2706 records, 261556 bytes) in 6 seconds Mar 12 15:30:03 pasargades lmtp[30950]: skiplist: checkpointed /var/lib/imap/deliver.db (2720 records, 263040 bytes) in 14 seconds Mar 12 15:30:11 pasargades lmtp[30981]: skiplist: recovered /var/lib/imap/deliver.db (2720 records, 263040 bytes) in 8 seconds cyrus recover by itself but the problem is that when the deliver.db grows it take about one hour to recover and the delivery of mail is stopped. Has someone already seen that and have an idea on how to solve the problem. thank in advance for any help Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
RE: lots of corruption of delived.db file
I have seen it. Just curious did you update your system lately. I updated mine yesterday for the new Timezone changes and have had trouble since then. Nothing like waiting till the last minute, but I had to do it. Regards, Pete Peter P. Benac, CCNA Emacolet Networking Services, Inc Providing Network and Systems Project Management and Installation and Web Hosting. Phone: 919-618-2557 Web: http://www.emacolet.com Need quick reliable Systems or Network Management advice visit http://www.nmsusers.org To have principles... First have courage.. With principles comes integrity!!! -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Eric Doutreleau Sent: Monday, March 12, 2007 10:43 To: info-cyrus@lists.andrew.cmu.edu Subject: lots of corruption of delived.db file I'm using cyrus2.3.8 on a RHEL4 machine and from time to time i got massive corruption of deliverd.db file for example today Mar 12 15:27:13 pasargades lmtp[31407]: skiplist: checkpointed /var/lib/imap/deliver.db (1282 records, 123104 bytes) in 5 seconds Mar 12 15:27:18 pasargades lmtp[30946]: skiplist: recovered /var/lib/imap/deliver.db (1282 records, 123104 bytes) in 5 seconds Mar 12 15:27:24 pasargades lmtp[30948]: skiplist: checkpointed /var/lib/imap/deliver.db (1289 records, 123800 bytes) in 6 seconds Mar 12 15:27:28 pasargades lmtp[31407]: skiplist: recovered /var/lib/imap/deliver.db (1289 records, 123800 bytes) in 4 seconds Mar 12 15:27:35 pasargades lmtp[30981]: skiplist: checkpointed /var/lib/imap/deliver.db (1300 records, 124788 bytes) in 7 seconds Mar 12 15:27:40 pasargades lmtp[31677]: skiplist: recovered /var/lib/imap/deliver.db (1300 records, 124788 bytes) in 5 seconds Mar 12 15:28:20 pasargades lmtp[31679]: skiplist: checkpointed /var/lib/imap/deliver.db (2541 records, 244756 bytes) in 11 seconds Mar 12 15:28:26 pasargades lmtp[31675]: skiplist: recovered /var/lib/imap/deliver.db (2541 records, 244756 bytes) in 6 seconds Mar 12 15:28:38 pasargades lmtp[30947]: skiplist: checkpointed /var/lib/imap/deliver.db (2578 records, 248632 bytes) in 11 seconds Mar 12 15:28:45 pasargades lmtp[30951]: skiplist: recovered /var/lib/imap/deliver.db (2578 records, 248632 bytes) in 7 seconds Mar 12 15:29:01 pasargades lmtp[31676]: skiplist: checkpointed /var/lib/imap/deliver.db (2622 records, 253440 bytes) in 15 seconds Mar 12 15:29:08 pasargades lmtp[31678]: skiplist: recovered /var/lib/imap/deliver.db (2622 records, 253440 bytes) in 7 seconds Mar 12 15:29:22 pasargades lmtp[31462]: skiplist: checkpointed /var/lib/imap/deliver.db (2699 records, 260880 bytes) in 12 seconds Mar 12 15:29:31 pasargades lmtp[31678]: skiplist: recovered /var/lib/imap/deliver.db (2699 records, 260880 bytes) in 9 seconds Mar 12 15:29:43 pasargades lmtp[30952]: skiplist: checkpointed /var/lib/imap/deliver.db (2706 records, 261556 bytes) in 12 seconds Mar 12 15:29:49 pasargades lmtp[31679]: skiplist: recovered /var/lib/imap/deliver.db (2706 records, 261556 bytes) in 6 seconds Mar 12 15:30:03 pasargades lmtp[30950]: skiplist: checkpointed /var/lib/imap/deliver.db (2720 records, 263040 bytes) in 14 seconds Mar 12 15:30:11 pasargades lmtp[30981]: skiplist: recovered /var/lib/imap/deliver.db (2720 records, 263040 bytes) in 8 seconds cyrus recover by itself but the problem is that when the deliver.db grows it take about one hour to recover and the delivery of mail is stopped. Has someone already seen that and have an idea on how to solve the problem. thank in advance for any help Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Potential replica message file corruption/replacement
On Fri, 16 Feb 2007, Bron Gondwana wrote: Looks innocent, doesn't it... Mea culpa (and a definite Argh, how did I miss _that_ when it was pointed out to me yesterday). I would advise anyone who has been using replication for any length of time to undertake an audit of the files on their replicas to ensure that none of them have been replaced by this, because if you need to fail over you could present users with emails that are not their own. A simple size check will find almost all cases, compare what the imapd returns for rfc822.size with the size of the file on disk. If you want to get fancy - compute the sha1 or similar of the file at each end and compare that. This incident underlines the need for automated sanity checks. People shouldn't just blindly trust the replication system. I generate (and constantly regenerate) checksums for message bodies and cache entries. On four occasions this has picked up oddities which in hindsight were obviously this bug. -- David Carter Email: [EMAIL PROTECTED] University Computing Service,Phone: (01223) 334502 New Museums Site, Pembroke Street, Fax: (01223) 334679 Cambridge UK. CB2 3QH. Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Potential replica message file corruption/replacement
Looks innocent, doesn't it... On Thu, Feb 15, 2007 at 09:57:01AM -0500, [EMAIL PROTECTED] wrote: Update of /afs/andrew/system/cvs/src/cyrus/imap In directory unix36.andrew.cmu.edu:/var/tmp/cvs-serv18314 Modified Files: sync_support.c Log Message: unlink() the file in the stage dir before trying to create a new one (Bron Gondwana) --- links to diffs follow --- http://bugzilla.andrew.cmu.edu/cgi-bin/cvsweb.cgi/src/cyrus/imap/sync_support.c.diff?r1=1.2r2=1.3 ... here's what our cyrus patches page has to say about it: Our automated check scripts compare message bodies between master and replica at some very low probability. Imagine our shock when it found that two messages with exactly the same index meta data had totally different message bodies on disk, and the one on the replica didn't even belong to the same user. A more thorough audit turned up many more of these files, and we scratched our heads for days until we realised: a) we used to have lots of sync_client crashes during the early replication rollout. b) after a crash, the sync_server left temporary files sitting around. c) these files were hardlinked in some cases to files in users' folders - depending where the crash happened. If (and believe me, with a dozen or so separate cyrus instances per machine each with many thousands of users, it happens) another sync_server happened to reuse the same PID at some point in the future - say a month later - it would proceed to open those files in mode 'w+', writing the new file not only to the staging directory BUT ALSO OVERWRITING A FILE PUT INTO A MAILBOX A MONTH AGO. No wonder we couldn't see any pattern in the causes of this problem. This attached patch attempts to unlink the stage file and aborts before writing if it can't unlink and the file exists. I would advise anyone who has been using replication for any length of time to undertake an audit of the files on their replicas to ensure that none of them have been replaced by this, because if you need to fail over you could present users with emails that are not their own. A simple size check will find almost all cases, compare what the imapd returns for rfc822.size with the size of the file on disk. If you want to get fancy - compute the sha1 or similar of the file at each end and compare that. NOTE: a reconstruct of the folder may have caused the index to have the wrong value, in which case you're left with content analysis, which is never fun. Bron. Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
HELP! A storage failure resulted in data corruption
Greetings. We had a storage failure recently. A user's data were corrupted. The user client keeps terminated abnormally.Below are the error messages.I also noticed two .NEW files created in the user's INBOX. Can anybody gave me any suggestion? Nov 23 08:58:14 cms3 imap[19511]: seen_db: user USERNAMEopened /var/lib/imap/user/k/USERNAME.seen Nov 23 08:58:14 cms3 imap[19511]: open: user USERNAME opened Trash Nov 23 08:58:15 cms3 imap[19511]: open: user USERNAME opened INBOX Nov 23 08:58:15 cms3 master[3354]: process 19511 exited, signaled to death by 7 Nov 23 08:58:15 cms3 master[3354]: service imap pid 19511 in BUSY state: terminated abnormally -rw--- 1 cyrus mail 413140 Nov 21 16:20 cyrus.cache -rw--- 1 cyrus mail 409600 Nov 23 10:44 cyrus.cache.NEW -rw--- 1 cyrus mail183 Nov 23 10:38 cyrus.header -rw--- 1 cyrus mail 10336 Nov 21 16:20 cyrus.index -rw--- 1 cyrus mail 8192 Nov 23 10:44 cyrus.index.NEW Thanks -- Kai Wang System Services Information Technologies, University of Calgary, 2500 University Drive, N.W., Calgary, Alberta, Canada T2N 1N4 Phone (403) 220-2423, Fax (403) 282-9361 Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: HELP! A storage failure resulted in data corruption
Greetings. We had a storage failure recently. A user's data were corrupted. The user client keeps terminated abnormally.Below are the error messages.I also noticed two .NEW files created in the user's INBOX. Can anybody gave me any suggestion? Did you try to reconstruct the mailbox in question, something like: su - cyrus -c /usr/lib/cyrus-imapd/reconstruct -r user.USERNAME If that helps you may also reconstruct the whole storage with: su - cyrus -c /usr/lib/cyrus-imapd/reconstruct Let us know whether it fixes the problem. Regards, Simon Nov 23 08:58:14 cms3 imap[19511]: seen_db: user USERNAMEopened /var/lib/imap/user/k/USERNAME.seen Nov 23 08:58:14 cms3 imap[19511]: open: user USERNAME opened Trash Nov 23 08:58:15 cms3 imap[19511]: open: user USERNAME opened INBOX Nov 23 08:58:15 cms3 master[3354]: process 19511 exited, signaled to death by 7 Nov 23 08:58:15 cms3 master[3354]: service imap pid 19511 in BUSY state: terminated abnormally -rw--- 1 cyrus mail 413140 Nov 21 16:20 cyrus.cache -rw--- 1 cyrus mail 409600 Nov 23 10:44 cyrus.cache.NEW -rw--- 1 cyrus mail183 Nov 23 10:38 cyrus.header -rw--- 1 cyrus mail 10336 Nov 21 16:20 cyrus.index -rw--- 1 cyrus mail 8192 Nov 23 10:44 cyrus.index.NEW Thanks -- Kai Wang System Services Information Technologies, University of Calgary, 2500 University Drive, N.W., Calgary, Alberta, Canada T2N 1N4 Phone (403) 220-2423, Fax (403) 282-9361 Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Db corruption error
Hi all!My company is dealing with a big problem of db corruption that happens randomly every 1-2 days in correspondence to ctl_cyrusdb launch in order to checkpoint the db.Here is an extract of the log file: error startcheckpointing cyrus databasesarchiving database file: /var/mail/imap_config/annotations.dbDBERROR db4: DB_LOGC-get: LSN 1/1083033: invalid log record headerDBERROR: error listing log files: DB_NOTFOUND: No matching key/data DBERROR: archive /var/mail/imap_config/db: cyrusdb errorarchiving database file: /var/mail/imap_config/mailboxes.dbDBERROR db4: DB_LOGC-get: LSN 1/1083033: invalid log record headerts 2 timesDBERROR db4: DB_LOGC-get: LSN 1/1083303: invalid log record header done checkpointing cyrus databasesDBERROR db4: DB_LOGC-get: LSN 1/1083303: invalid log record header--- error endHere is reassumed our setup:Sendmail/Cyrus Imap server are installed on an HP-UX V. 11 Itanium server Sendmail version is 8.11 and Cyrus-imap/Cyrus-sasl come from Iexpress depot in version 2.2.12.We use saslauthd to autenticate via pam imap/pop3 users over a distributed directory service based on a Linux Openldap Server. Pam_ldap and hpldapux client are used with success also to authenticate samba-cifs users.Thanks you! Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Avoiding deliver.db corruption
If I hacked our init script to delete deliver.db before starting Cyrus IMAPD, what adverse consequences would there be, if any? We recently were bitten by deliver.db corruption when our mail server went down ungracefully. Thanks in advance. Karl Boyken -- Karl Boyken, system administrator [EMAIL PROTECTED] 303A MLH, Dept. of Comp. Sci. http://www.cs.uiowa.edu/~boyken/ The U. of Iowa, Iowa City, IA 52242 319-335-2730 (voice) 319-335-3668 (fax) smime.p7s Description: S/MIME Cryptographic Signature Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Avoiding deliver.db corruption
basically duplicate suppression might miss a few messages (since you've deleted what it does its comparison against). Also, I think it resets any state associated with vacation. e.g. Someone who has already gotten a vacation auto-response, may get another one after the deliver.db is deleted and if they send another message to the address using vacation. -Patrick On Apr 19, 2006, at 1:12 PM, Karl Boyken wrote: If I hacked our init script to delete deliver.db before starting Cyrus IMAPD, what adverse consequences would there be, if any? We recently were bitten by deliver.db corruption when our mail server went down ungracefully. Thanks in advance. Karl Boyken -- Karl Boyken, system administrator [EMAIL PROTECTED] 303A MLH, Dept. of Comp. Sci. http://www.cs.uiowa.edu/~boyken/ The U. of Iowa, Iowa City, IA 52242 319-335-2730 (voice) 319-335-3668 (fax) Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: frequent mupdate master mailboxes.db corruption, anyone else?
Sergio Devojno Bruder wrote: João Assad wrote: Sergio Devojno Bruder wrote: AHA: Sep 21 09:08:49 mupdate mupdate[17026]: IOERROR: mapping /var/lib/imap/mailboxes.db file: Cannot allocate memory Sep 21 09:08:49 mupdate mupdate[17026]: failed to mmap /var/lib/imap/mailboxes.db file I remember Joao Assad had the same problem, no? -- Sergio Devojno Bruder Sorry , I missed the original post. Is the original poster using Fedora or RHEL ? CentOS 3.0 (migrating today to CentOS 4.1 x86-64 bits), ie, RHEL. -- Sergio Devojno Bruder I belive I found a mmap bug on Fedora Core 2 and RHEL. I never got a confirmation that it is indeed a bug, since at the time support for fedora core 2 ended and the fedora devs decided to move the bug to devel. Anyway, I wrote a patch that changes the way cyrus use mmap. Instead of doing a munmap and a new mmap when needed, my patch changes it so it calls mremap instead. It has been working here ever since, never had the corruption again. You can find the patch and somewhat detailed information about the problem in the end of the bugzilla report here - https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=152548 . a copy of the patch is also posted on cyrus bugzilla - http://bugzilla.andrew.cmu.edu/show_bug.cgi?id=2640 I hope that helps you. and please review my code because as it's stated in the bugzilla report Im not a very good C programmer ;-) Best regards, João Assad Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: frequent mupdate master mailboxes.db corruption, anyone else?
Sergio Devojno Bruder wrote: Henrique de Moraes Holschuh wrote: On Thu, 15 Sep 2005, Sergio Devojno Bruder wrote: I'm fighting with frequent corruptions of my mupdate master server (high volume, currently 3.9M mailboxes) with Cyrus 2.2.10. First things first: Triple check your system RAM. AHA: Sep 21 09:08:49 mupdate mupdate[17026]: IOERROR: mapping /var/lib/imap/mailboxes.db file: Cannot allocate memory Sep 21 09:08:49 mupdate mupdate[17026]: failed to mmap /var/lib/imap/mailboxes.db file Sep 21 09:08:49 mupdate master[5866]: process 17026 exited, status 75 Sep 21 09:08:49 mupdate master[5866]: service mupdate pid 17026 in READY state: terminated abnormally I remember Joao Assad had the same problem, no? -- Sergio Devojno Bruder Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html Sorry , I missed the original post. Is the original poster using Fedora or RHEL ? Regards -- - João Assad - ParPerfeito Comunicação LTDA - http://www.parperfeito.com.br/ Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: frequent mupdate master mailboxes.db corruption, anyone else?
Henrique de Moraes Holschuh wrote: On Thu, 15 Sep 2005, Sergio Devojno Bruder wrote: I'm fighting with frequent corruptions of my mupdate master server (high volume, currently 3.9M mailboxes) with Cyrus 2.2.10. First things first: Triple check your system RAM. AHA: Sep 21 09:08:49 mupdate mupdate[17026]: IOERROR: mapping /var/lib/imap/mailboxes.db file: Cannot allocate memory Sep 21 09:08:49 mupdate mupdate[17026]: failed to mmap /var/lib/imap/mailboxes.db file Sep 21 09:08:49 mupdate master[5866]: process 17026 exited, status 75 Sep 21 09:08:49 mupdate master[5866]: service mupdate pid 17026 in READY state: terminated abnormally I remember Joao Assad had the same problem, no? -- Sergio Devojno Bruder Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: frequent mupdate master mailboxes.db corruption, anyone else?
Henrique de Moraes Holschuh wrote: On Thu, 15 Sep 2005, Sergio Devojno Bruder wrote: I'm fighting with frequent corruptions of my mupdate master server (high volume, currently 3.9M mailboxes) with Cyrus 2.2.10. First things first: Triple check your system RAM. 4G of ECC with no log of error, I dont think that in this case the RAM is the guilty. Other factor: this corruptions appeared with the volume of use, this smells 'race' badly. Example of the simptom: Sep 15 21:17:14 mupdate mupdate[1608]: DBERROR: skiplist recovery /var/lib/imap/mailboxes.db: 14D67FD8 should be ADD or DELETE When this happens mupdate get tired of create threads to the point of 'cant create another thread'. Whe are using CentOS 3.0 (if kernel/glibc/library versions can mean something). (Our backends are CentOS 4.1 64 bits, our mupdate is CentOS 3.0 32 bits). Any more relevant infos? -- Sergio Devojno Bruder Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
frequent mupdate master mailboxes.db corruption, anyone else?
I'm fighting with frequent corruptions of my mupdate master server (high volume, currently 3.9M mailboxes) with Cyrus 2.2.10. We've tried switch away from skiplist to berkley DB, same result. This problem is resolved somewhere? IE, there is hope in trying 2.3 branch or something like that? -- Sergio Devojno Bruder Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: frequent mupdate master mailboxes.db corruption, anyone else?
On Thu, 15 Sep 2005, Sergio Devojno Bruder wrote: I'm fighting with frequent corruptions of my mupdate master server (high volume, currently 3.9M mailboxes) with Cyrus 2.2.10. First things first: Triple check your system RAM. -- One disk to rule them all, One disk to find them. One disk to bring them all and in the darkness grind them. In the Land of Redmond where the shadows lie. -- The Silicon Valley Tarot Henrique Holschuh Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: cyrus-murder problems with database corruption in the frontend/master
Derrick J Brashear wrote: is there a munmap for every mmap when I start cyrus, both mupdate master and slave comes up, so I get cyrus/mupdate[2678]: MMAP at map_refresh: mmap(0, 408084480, PROT_READ, MAP_SHARED, 8, 0) = 1098067968 cyrus/mupdate[2679]: MMAP at map_refresh: mmap(0, 408084480, PROT_READ, MAP_SHARED, 8, 0) = 1098067968 The thing that got my attention is , cyrus have been running for a few hours now and every now and then I get this: cyrus/mupdate[2679]: MUNMAP at map_refresh: munmap(1098067968, 408084480) = 0 cyrus/mupdate[2679]: MMAP at map_refresh: mmap(0, 408100864, PROT_READ, MAP_SHARED, 8, 0) = 1718439936 mupdate PID 2679 called something that requested a map_refresh .. which decided to do a munmap and then mmap. That happens quite often, and I dont see anything wrong with that. On the other hand, mupdate PID 2678, after doing that initial map_refresh / mmap never munmaped or mmaped again. --- Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: cyrus-murder problems with database corruption in the frontend/master
João Assad wrote: yet another gdb backtrace of the mmap problem http://www.gazzag.com/gdb.output2.gz regards Hey Derrick, did you find anything usefull in this last backtrace ? Regards --- Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: cyrus-murder problems with database corruption in the frontend/master
yet another gdb backtrace of the mmap problem http://www.gazzag.com/gdb.output2.gz regards --- Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: cyrus-murder problems with database corruption in the frontend/master
got a full gdb backtrace of the mmap crash. Its 400k so you can download it from: http://www.gazzag.com/gdb.output.gz regards --- Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: cyrus-murder problems with database corruption in the frontend/master
On Mon, 04 Apr 2005, João Assad wrote: The following options seem to have a direct impact on how fast I run out of resources (obviously) . The more I increase them, the faster I get the mmap error. *mupdate_workers_start mupdate_workers_minspare mupdate_workers_maxspare mupdate_workers_max Just thought of something. Please set the vm.overcommit_memory syscall to 2 (it is available in /proc/sys, I think. But the right way is to use /etc/sysctl.conf and sysctl). Make *really* sure you have enough swap when you do that. You will *really* need it. Some look on /proc/meminfo (especially on CommitLimit and Committed_AS) might shed some light if that is the problem. -- One disk to rule them all, One disk to find them. One disk to bring them all and in the darkness grind them. In the Land of Redmond where the shadows lie. -- The Silicon Valley Tarot Henrique Holschuh --- Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: cyrus-murder problems with database corruption in the frontend/master
Henrique de Moraes Holschuh wrote: Just thought of something. Please set the vm.overcommit_memory syscall to 2 (it is available in /proc/sys, I think. But the right way is to use /etc/sysctl.conf and sysctl). Make *really* sure you have enough swap when you do that. You will *really* need it. Some look on /proc/meminfo (especially on CommitLimit and Committed_AS) might shed some light if that is the problem. Yeah I thought that might help too, I did that yesterday after the last corruption I also set the overcommit_ratio to 100 So far everything is running smooth. but I also recuced the mupdate_workers configs, which usually makes cyrus last longer. Im getting anxious here, I already have 643M worth of straces since I started it. So far no ENOMEMs . My current meminfo MemTotal: 2074268 kB MemFree:277240 kB Buffers: 93244 kB Cached:1437800 kB SwapCached: 3216 kB Active:1247724 kB Inactive: 402988 kB HighTotal: 1178756 kB HighFree: 153984 kB LowTotal: 895512 kB LowFree:123256 kB SwapTotal: 2096472 kB SwapFree: 2093068 kB Dirty:1448 kB Writeback: 0 kB Mapped: 136136 kB Slab:46308 kB CommitLimit: 4170740 kB Committed_AS: 1494736 kB PageTables: 90924 kB VmallocTotal: 114680 kB VmallocUsed: 2072 kB VmallocChunk: 112208 kB HugePages_Total: 0 HugePages_Free: 0 Hugepagesize: 2048 kB --- Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: cyrus-murder problems with database corruption in the frontend/master
Out of curiosity.. everytime cyrus needs to mmap/munmap something. it uses its mmap encapsulation, which is map_refresh and map_free. So every once in a while I get this on strace. 14:27:02.148097 munmap(0x54968000, 373915648) = 0 14:27:02.148765 mmap2(NULL, 373932032, PROT_READ, MAP_SHARED, 8, 0) = 0x54964000 which is a map_free followed by a map_refresh. Now this... 14:27:31.041808 mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x54963000 14:27:31.042150 munmap(0x54963000, 4096) = 0 Happens way much more. Its not cyrus doing it directly because its not going through it's mmap encapsulation, so it can only be some lib used by cyrus. What is it ? --- Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
