Re: Squatter crash with statusdb

[Andy Fiddaman]

On Mon, 24 Jun 2013, Simon Matter wrote:
;
; Or, could you check this bug here
; http://bugzilla.cyrusimap.org/show_bug.cgi?id=3757
;
; The patch below was the fix, could you verify if it also fixes your issue?

Yes that patch fixes my issue too. I did search bugzilla but didn't
see that one because it is marked as resolved. I didn't see it fixed
in git either (http://git.cyrusimap.org/cyrus-imapd/tree/imap/statuscache_db.c)
but I see now it's fixed in the 2.4 branch only.

Thanks,

Andy


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Squatter crash with statusdb

[Andy Fiddaman]

FWIW, this gets it working again:

--- cyrus-imapd-2.4.17.dist/imap/statuscache_db.c   2013-06-24
10:10:08.219203100 +
+++ cyrus-imapd-2.4.17/imap/statuscache_db.c2013-06-24
10:10:20.537711377 +
@@ -152,7 +152,7 @@
 len = strlcpy(key, mailboxname, sizeof(key));
 key[len++] = '%';
 key[len++] = '%';
-len += strlcpy(key + len, userid, sizeof(key) - len);
+len += strlcpy(key + len, userid ? userid : "cyrus", sizeof(key) -
len);

 *keylen = len;


On Mon, 24 Jun 2013, Andy Fiddaman wrote:

;
; Hi,
;
; I've just upgraded my Cyrus installation to 2.4.17 and squatter is
; crashing in statuscache_buildkey() because userid is NULL.
;
; I'm not sure what the best fix for this is. Should squatter even be using
; the statuscache or should it populate "cyrus" as the username when
; initialising the index, or something else?
;
; Thanks,
;
; Andy
;
; Program received signal SIGSEGV, Segmentation fault.
; [Switching to Thread 1 (LWP 1)]
; 0xfd7ffe3dccb0 in .strlenalign16_loop () from /lib/64/libc.so.1
; (gdb) where
; #0  0xfd7ffe3dccb0 in .strlenalign16_loop () from /lib/64/libc.so.1
; #1  0xfd7ffe414149 in strlcpy () from /lib/64/libc.so.1
; #2  0x004610ac in statuscache_buildkey (
; mailboxname=0x5ab8b0 "example.net!user.silo", userid=0x0,
; keylen=0xfd7fffdfe0cc) at statuscache_db.c:155
; #3  0x0046169a in statuscache_update_txn (
; mboxname=0x5ab8b0 "example.net!user.silo", sdata=0xfd7fffdfe290,
; tidptr=0xfd7fffdfe218) at statuscache_db.c:326
; #4  0x004619ad in statuscache_invalidate (
; mboxname=0x5ab8b0 "example.net!user.silo", sdata=0xfd7fffdfe290)
; at statuscache_db.c:425
; #5  0x00434099 in mailbox_unlock_index (mailbox=0x5db998,
; sdata=0xfd7fffdfe290) at mailbox.c:1637
; #6  0x00422feb in index_unlock (state=0x5af6a0) at index.c:1232
; #7  0x00420c98 in index_open (name=0x5ac960
; "example.net!user.silo",
; init=0x0, stateptr=0xfd7fffdfec18) at index.c:246
; #8  0x00420395 in index_me (name=0x5ac960 "example.net!user.silo",
; matchlen=20, maycreate=0, rock=0xfd7fffdffcac) at squatter.c:594
; #9  0x004208f9 in main (argc=3, argv=0xfd7fffdffcf8)
; at squatter.c:745
;
;
; (gdb) frame 7
; #7  0x00420c98 in index_open (name=0x5ac960
; "example.net!user.silo",
; init=0x0, stateptr=0xfd7fffdfec18) at index.c:246
; 246 in index.c
; (gdb) print *state
; $4 = {mailbox = 0x5db998, num_records = 6933, oldexists = 0, exists =
; 6933,
;   last_uid = 6934, highestmodseq = 1606, delayed_modseq = 0, map =
; 0x5dd740,
;   mapsize = 7168, internalseen = 0, skipped_expunge = 0, seen_dirty = 0,
;   keepingseen = 0, examining = 0, myrights = 0, numrecent = 0,
;   numunseen = 6933, firstnotseen = 1, flagname = {0x0 },
;   userid = 0x0, out = 0x0, qresync = 0, authstate = 0x0}
;
; 
; Cyrus Home Page: http://www.cyrusimap.org/
; List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
; To Unsubscribe:
; https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
;

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Squatter crash with statusdb

[Andy Fiddaman]

Hi,

I've just upgraded my Cyrus installation to 2.4.17 and squatter is
crashing in statuscache_buildkey() because userid is NULL.

I'm not sure what the best fix for this is. Should squatter even be using
the statuscache or should it populate "cyrus" as the username when
initialising the index, or something else?

Thanks,

Andy

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 1 (LWP 1)]
0xfd7ffe3dccb0 in .strlenalign16_loop () from /lib/64/libc.so.1
(gdb) where
#0  0xfd7ffe3dccb0 in .strlenalign16_loop () from /lib/64/libc.so.1
#1  0xfd7ffe414149 in strlcpy () from /lib/64/libc.so.1
#2  0x004610ac in statuscache_buildkey (
mailboxname=0x5ab8b0 "example.net!user.silo", userid=0x0,
keylen=0xfd7fffdfe0cc) at statuscache_db.c:155
#3  0x0046169a in statuscache_update_txn (
mboxname=0x5ab8b0 "example.net!user.silo", sdata=0xfd7fffdfe290,
tidptr=0xfd7fffdfe218) at statuscache_db.c:326
#4  0x004619ad in statuscache_invalidate (
mboxname=0x5ab8b0 "example.net!user.silo", sdata=0xfd7fffdfe290)
at statuscache_db.c:425
#5  0x00434099 in mailbox_unlock_index (mailbox=0x5db998,
sdata=0xfd7fffdfe290) at mailbox.c:1637
#6  0x00422feb in index_unlock (state=0x5af6a0) at index.c:1232
#7  0x00420c98 in index_open (name=0x5ac960
"example.net!user.silo",
init=0x0, stateptr=0xfd7fffdfec18) at index.c:246
#8  0x00420395 in index_me (name=0x5ac960 "example.net!user.silo",
matchlen=20, maycreate=0, rock=0xfd7fffdffcac) at squatter.c:594
#9  0x004208f9 in main (argc=3, argv=0xfd7fffdffcf8)
at squatter.c:745


(gdb) frame 7
#7  0x00420c98 in index_open (name=0x5ac960
"example.net!user.silo",
init=0x0, stateptr=0xfd7fffdfec18) at index.c:246
246 in index.c
(gdb) print *state
$4 = {mailbox = 0x5db998, num_records = 6933, oldexists = 0, exists =
6933,
  last_uid = 6934, highestmodseq = 1606, delayed_modseq = 0, map =
0x5dd740,
  mapsize = 7168, internalseen = 0, skipped_expunge = 0, seen_dirty = 0,
  keepingseen = 0, examining = 0, myrights = 0, numrecent = 0,
  numunseen = 6933, firstnotseen = 1, flagname = {0x0 },
  userid = 0x0, out = 0x0, qresync = 0, authstate = 0x0}


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: 2.4.2 on Solaris - Crashes in mailbox_unlock_index

[Andy Fiddaman]

On Mon, 1 Nov 2010, Bron Gondwana wrote:
;
; Here's your patch :)

Works perfectly, thanks :)

Andy

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/

Re: 2.4.2 on Solaris - Crashes in mailbox_unlock_index

[Andy Fiddaman]

On Sun, 31 Oct 2010, Bron Gondwana wrote:

; On Sat, Oct 30, 2010 at 11:19:14PM +, Andy Fiddaman wrote:
; > On Sun, 31 Oct 2010, Bron Gondwana wrote:
; > ;
; > ; I don't suppose the stacktrace went any further up than that?  I'm
; > ; more interested in the call-site of mailbox_close, because that's
; > ; where a dirty mailbox will be being closed.
; >
; > Here are a couple:
;
; Ok - that's all I needed.  This is a bug.  I'll push a fix
; to master straight away, and it will be in 2.4.3.

Thanks, superb support as always. I'll apply the patch and look at rolling
out 2.4.2 to production this week then go to 2.4.3 when it's out.

Andy

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/

Re: 2.4.2 on Solaris - Crashes in mailbox_unlock_index

[Andy Fiddaman]

On Sun, 31 Oct 2010, Bron Gondwana wrote:
;
; I don't suppose the stacktrace went any further up than that?  I'm
; more interested in the call-site of mailbox_close, because that's
; where a dirty mailbox will be being closed.

Here are a couple:

#0  0xfd7ffef41e4a in _lwp_kill () from /lib/64/libc.so.1
#1  0xfd7ffef3a753 in thr_kill () from /lib/64/libc.so.1
#2  0xfd7ffeee6f89 in raise () from /lib/64/libc.so.1
#3  0xfd7ffeec5f00 in abort () from /lib/64/libc.so.1
#4  0x0041a8d9 in mailbox_unlock_index (mailbox=0x58e3e8,
sdata=0x0)
at mailbox.c:1544
#5  0x004192e2 in mailbox_close (mailboxptr=0xfd7fffdfe7a8)
at mailbox.c:1003
#6  0x00416ca3 in expire (
name=0xfd7fffdfec80 "domain.com!user.postmaster",
matchlen=31,
maycreate=1, rock=0xfd7fffdffb20) at cyr_expire.c:244
#7  0x00424490 in find_cb (rockp=0xfd7fffdff9a0,
key=0xfd7ffec501a0 "domain.com!user.postmaster", keylen=31,
data=0xfd7ffec501c4 "0 default
postmas...@domain.com\tlrswipcda\tda@domain.com\tlrs\...@domain.com\tlrs\tjwho@domain.com\tlrs\t",
datalen=118) at mboxlist.c:1697
#8  0x00458c19 in myforeach (db=0x58d7b0,
prefix=0xfd7fffdff190 "*", prefixlen=0, goodp=0x423c31 ,
cb=0x424140 , rock=0xfd7fffdff9a0, tidptr=0x0)
at cyrusdb_skiplist.c:1122
#9  0x00424dc3 in mboxlist_findall (namespace=0x0,
pattern=0x58b190 "", isadmin=1, userid=0x0, auth_state=0x0,
proc=0x4167ed , rock=0xfd7fffdffb20) at mboxlist.c:1889
#10 0x00417424 in main (argc=4, argv=0xfd7fffdffc58)
at cyr_expire.c:440



#0  0xfd7ffef41e4a in _lwp_kill () from /lib/64/libc.so.1
#1  0xfd7ffef3a753 in thr_kill () from /lib/64/libc.so.1
#2  0xfd7ffeee6f89 in raise () from /lib/64/libc.so.1
#3  0xfd7ffeec5f00 in abort () from /lib/64/libc.so.1
#4  0x0041a1fb in mailbox_unlock_index (mailbox=0x58cae8,
sdata=0x0)
at mailbox.c:1544
#5  0x00418c04 in mailbox_close (mailboxptr=0xfd7fffdfdd80)
at mailbox.c:1003
#6  0x00416c1f in purge_me (
name=0xfd7fffdfddf0 "domain.org!user.reci12",
matchlen=36, maycreate=1) at ipurge.c:265
#7  0x00423db2 in find_cb (rockp=0xfd7fffdfeb10,
key=0xfd7ffed89638 "domain.org!user.reci12",
keylen=36,
data=0xfd7ffed89660 "0 default
rec...@domain.org\tlrswipcda\t", datalen=52) at
mboxlist.c:1697
#8  0x00457899 in myforeach (db=0x58af40,
prefix=0xfd7fffdfe300 "domain.org!user.reci%",
prefixlen=34, goodp=0x423553 , cb=0x423a62 ,
rock=0xfd7fffdfeb10, tidptr=0x0) at cyrusdb_skiplist.c:1122
#9  0x004246e5 in mboxlist_findall (namespace=0x5736e0,
pattern=0x58e190 "user.reci", isadmin=1, userid=0x0, auth_state=0x0,
proc=0x416ad1 , rock=0x0) at mboxlist.c:1889
#10 0x004169e4 in main (argc=5, argv=0xfd7fffdffc38)
at ipurge.c:202


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/

2.4.2 on Solaris - Crashes in mailbox_unlock_index

[Andy Fiddaman]

Hi,

I've just upgraded to Cyrus 2.4 from 2.3 on Solaris 10 and I'm seeing lots
of crashes with various utilities in mailbox_unlock_index().

(gdb) where
#0  0xfd7ffef41e4a in _lwp_kill () from /lib/64/libc.so.1
#1  0xfd7ffef3a753 in thr_kill () from /lib/64/libc.so.1
#2  0xfd7ffeee6f89 in raise () from /lib/64/libc.so.1
#3  0xfd7ffeec5f00 in abort () from /lib/64/libc.so.1
#4  0x0041a1fb in mailbox_unlock_index (mailbox=0x58cae8,
sdata=0x0)
at mailbox.c:1544
#5  0x00418c04 in mailbox_close (mailboxptr=0xfd7fffdfdd70)
at mailbox.c:1003

Just after the upgrade, I was seeing this during attempted index upgrades
(from lmtpd) and reconstruct was failing with the same problem. I removed
the cyrus.* files and then reconstruct completed correctly, and subsequent
reconstructs were fine too.

Everything now seems to be working correctly from the user's perspective,
but I'm still getting these crashes with ipurge and cyr_expire.

Is anyone else seeing this?

Thanks,

Andy

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/

Re: Got cyrus to compile but now it's not working.

[Andy Fiddaman]

On Thu, 28 Oct 2010, Frank Pittel wrote:
;
; One last question. Does anyone have any smf methods for "master" and 
"saslauthd"?

Here's a manifest that we use for master and sync client. It creates two
services:

# svcs cyrus
STATE  STIMEFMRI
online Oct_13   svc:/network/cyrus:server
online Oct_13   svc:/network/cyrus:sync

We keep the sync client separate so that we can control it without
affecting the main server.

As attached, it assumes the server is installed in /opt/cyrus and the data
is in /data/cyrus so you may have to change them.

Andy




















 















Cyrus IMAP Server


















Cyrus IMAP Sync Client









Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/

Re: Cyrus IMAPd 2.4.0 Released

[Andy Fiddaman]

On Wed, 13 Oct 2010, Frank Pittel wrote:

; Thanks for your build script. I'll give it a try when I get home. Some of the 
lacations
; of things like sasl and burkley db are in different locations but that 
shouldn't be
; an issue. I forgot to mention that I'm running sol-10 on x86.

btw, now that all databases are skiplist by default, you can get away
without compiling against Berkeley DB at all - I certainly plan to once
I've done the initial database conversion. In that case, just use
--without-bdb in the configure script.


Andy

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/

Re: Cyrus IMAPd 2.4.0 Released

[Andy Fiddaman]

On Tue, 12 Oct 2010, Frank Pittel wrote:
; > >
; > > Any chance of getting a package for solaris 10? I've been trying to build 
2.3.16 for over
; > > a week and still haven't gotten it to compile.
; >
; > I really don't know much about Solaris 10 packaging.  I do have a Solaris
; > box though... I should give it a go.  Do they usually get compiled with
; > gcc, or with the Sun compiler?
;
; If I can talk you into it I'd be happy with a tar ball and instructions on 
how/where to
; install the files. Let me know if you can get it to compile let me know what 
your
; configure options were and what the dependencies are and I'll compile it and 
put it into
; a package.

I run Cyrus on Solaris 10, SPARC and x86, no (real) problems.

I compile everything 64-bit so there are a couple of tweaks to find the
right libraries and I've separately packaged and installed Berkeley DB in
/opt/bdb, and SASL in /opt/CYRsasl2. If you want some packages let me know
and I'll make them available. I haven't tried 2.4.0 yet although I plan to
sometime in the next month, at least in our staging environment. Some of
the changes look pretty significant and I'm wary of any 64-bit alignment
problems that might be lurking there that will show up on SPARC.

Here's my configure script, should it help.

CC='gcc -m64' export CC
CFLAGS="-m64 -g -fPIC" \
LDFLAGS="-static-libgcc -L/usr/sfw/lib/64 -R/usr/sfw/lib/64" \
./configure \
--prefix=/opt/cyrus \
--with-cyrus-prefix=/opt/cyrus \
--sysconfdir=/opt/cyrus/etc \
--with-openssl=/usr/sfw \
--with-sasl=/opt/CYRsasl2 \
--with-bdb=/opt/bdb \
--with-bdb-incdir=/opt/bdb/include \
--with-bdb-libdir=/opt/bdb/lib \
--enable-replication \
--enable-murder \
--enable-idled \
--with-perl=/opt/perl/bin/perl \
--disable-gssapi \
--without-snmp

# Prevent attempted pod2man
f=Makefile
cp $f $f~
sed '
s/man et /et /
s/man com_err\/et/com_err\/et/
' < $f~ > $f

# Fix 64-bit library paths.
find . -name Makefile | while read f; do
cp $f $f~
sed < $f~ > $f '
/\/usr\/sfw\/lib /s/sfw\/lib/&\/64/g
'
done

exit 0



Andy

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/

Attachment corruption when downloading with Thunderbird...

[Andy Fiddaman]

Hi,

I'm running Cyrus IMAP 2.3.13 on Solaris (about to upgrade to 2.3.14) and a
number of my users who use Thunderbird have reported frequent corruption
of attachments.

This is particularly visible for images where they are obviously truncated.
The work-around they are using is to drag the message into another folder,
read it then drag it back.

The messages are never corrupt within Cyrus and look fine when viewed
with other clients which suggests a bug within Thunderbird or an interaction
issue.

I found some threads on the Mozilla lists which suggests that this is a
server side bug where the server reports the wrong size for message
parts and the suggested workaround is to set
mail.server.default.fetch_by_chunks to false within Thunderbird.

 (http://forums.mozillazine.org/viewtopic.php?f=29&t=104)

So, has anyone else had any reports of this behaviour or any reason to
believe that Thunderbird does not work well with Cyrus? I'm going to
enable telemetry for one of the users who has reported this and see
if I can see anything relevant in the IMAP session; any suggestions of
other places to look would be appreciated.

Thanks,

Andy


Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: [OT] m$ activesync with cyrus backend

[Andy Fiddaman]

On Tue, 16 Sep 2008, lartc wrote:
;
; is there a way to mimic exchange server in the linux environment where i
; can keep my cyrus/postfix/amavis install -- is there a linux app that
; speaks "exchange"?

Z-Push (http://z-push.sourceforge.net/) does the job nicely. I use that
for my iPhone with Cyrus as the backend.

HTH.

Andy

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: sync_server crashing..

[Andy Fiddaman]

On Mon, 19 May 2008, Bron Gondwana wrote:
; > On Sun, 18 May 2008, Bron Gondwana wrote:
; >
; > ; You have a corrupted cache file.  Various things could have caused
; > this,
; > ; it isn´t easy to know what it was.
;
; Hmm.. by corrupted cache file it could actually be the cache base pointers
; from the cyrus.index that are corrupted.  One cause was delayed expunge
; and reconstruct, but David Carter wrote some patches which got into 2.3.12
; to fix that, so new reconstructs will be fine.

Six days since the reconstruct and no crashes so it looks good, thanks
again.

Andy
Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: sync_server crashing..

[Andy Fiddaman]

On Sun, 18 May 2008, Bron Gondwana wrote:

; You have a corrupted cache file.  Various things could have caused this,
; it isn´t easy to know what it was.

Thanks.

I can't find the corrupted mailbox so I've run reconstruct on everything,
rsyncd the master to replica again and I'll see if the problem recurs (you
can tell it isn't a massive mailstore!)

I tried to find the mailbox with the problem by writing a quick program to
scan through each cache file and it didn't detect any errors. I also ran
mbexamine on every mailbox with no problems so I don't know where the
corruption, if any, was.

Keeping my fingers crossed anyway,

A.
Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

sync_server crashing..

[Andy Fiddaman]

Hi,

I'm hoping that someone who is familiar with the replication code can
help me with a problem I'm seeing with Cyrus 2.3.12.

I have a two server replicated setup and sync_server is occasionally
crashing. Once it's crashed once it keeps on crashing until I completely
reset replication by snapshotting the master, removing the sync logs
and rsyncing the snapshot to the replica.

The crash is happening in sync_cacheitem_size()

Core was generated by `sync_server'.
Program terminated with signal 11, Segmentation fault.
[New process 71345]
#0  0x000100020204 in sync_cacheitem_size (
cacheitem=0x97db9528 )
at sync_commit.c:330
330 cacheitem = CACHE_ITEM_NEXT(cacheitem);
(gdb) print cacheitem
$1 = 0x97db9528 
(gdb) print cacheitembegin
$3 = 0x778b22bc " Prestige\" ((\"del philippe\" NIL \"rangely\"
\"domain.net\")) ((\"del philippe\" NIL \"rangely\" \"domain.net\")) ((\"del
philippe\" NIL \"rangely\" \"domain.net\")) ((NIL NIL \"jagood\"
\"domain.co.uk\")) NIL NIL NIL \"<[EMAIL PROTECTED]>\")"

I've added some debug lines to the function to print out the item length
and item text for each cache item traversed. When it crashes I get something
like the following:

itemlen=407
item=[("Thu, 19 May 2016 15:20:46 +0800" "=?GB2312?B?c2FsZXPXqs/6ytuyvw==?=" 
(("=?GB2312?B?06rP+teovNIttPLU7MTjtcTNu8bGwaY=?=" NIL "osjdhk" "gokceada.com")) 
(("=?GB2312?B?06rP+teovNIttPLU7MTjtcTNu8bGwaY=?=" NIL "osjdhk" "gokceada.com")) 
(("=?GB2312?B?06rP+teovNIttPLU7MTjtcTNu8bGwaY=?=" NIL "osjdhk" "gokceada.com")) 
((NIL NIL "sales" "somedomai.co.uk")) NIL NIL NIL "<[EMAIL PROTECTED]>")]
itemlen=77
item=[("TEXT" "PLAIN" ("CHARSET" "GB2312") NIL NIL "7BIT" 5267 179 NIL NIL NIL 
NIL)]
itemlen=61
item=[("TEXT" "PLAIN" ("CHARSET" "GB2312") NIL NIL "7BIT" 5267 179)]
itemlen=48
item=[]
itemlen=166
item=[Message-Id: <[EMAIL PROTECTED]>^M
itemlen=56
item=[\350\220\245\351\224\200\344\270\223\345\256\266-\346\211\223\351\200\240\344\275\240\347\232\204\347\252\201\347\240\264\345\212\233
 <[EMAIL PROTECTED]>]
itemlen=23
item=[<[EMAIL PROTECTED]>]
itemlen=0
item=[]
itemlen=0
item=[]
itemlen=23
item=[{17}^M
itemlen=468
item=[("Thu, 19 May 2016 17:28:11 +0800" ...

over 8000 more lines, then:

itemlen=1313426464
item=[NIL "7BIT" 475 16 NIL NIL NIL NIL)("TEXT" "HTML" ("CHARSET" 
"Windows-1252") NIL NIL "QUOTED-PRINTABLE" 1760 55 NIL NIL NIL NIL) 
"ALTERNATIVE" ("BOUNDARY" "=_NextPart_000_0006_01C8A94A.CC9C2B80") NIL NIL 
NIL)]

This last itemlen pushes the pointer out of the allocated memory and
causes the crash.

Any ideas on whether these entries look right and where I should look next
to debug it?

Thanks,

Andy


Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Backscatter solutions

[Andy Fiddaman]

On Fri, 9 May 2008, Mike Cathey wrote:
; On May 9, 2008, at 3:05 AM, Andy Fiddaman wrote:
; > Pretty much the only way to stop this is to use something like BATV to
; > tweak your envelope sender address outbound. That still doesn't stop
; > everything as out-of-office replies are usually sent from a real address.
;
; BATV changes the from address of outbound messages.  How well do mailing lists
; deal with users that send messages from a different address each time?  Is
; there a nice workaround for this?

It only changes the envelope address, leaving the From: message header
intact. In my experience mailing lists validate the From: header not the
return path so no problems.

I haven't found any problems in the past year I've been using it.

HTH.

A.

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Backscatter solutions

[Andy Fiddaman]

; Marc Grober wrote:
; > I am getting pounded by backscatter as a result of one of my addresses
; > being used by some major spammers. Are there any solutions available to
; > address all the Delivery failure and bounce notices.  I would at least
; > like to be able to sort between such responses from mail I am actually
; > sending and the backscatter. I have looked through headers and nothing
; > seems an obvious candidate.

Pretty much the only way to stop this is to use something like BATV to
tweak your envelope sender address outbound. That still doesn't stop
everything as out-of-office replies are usually sent from a real address.

http://tools.ietf.org/html/draft-levine-mass-batv-02
http://sourceforge.net/projects/batv-milter/

A.


Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Cyrus IMAPd 2.3.12 Released

[Andy Fiddaman]

On Wed, 23 Apr 2008, Thomas Robers wrote:

; Ken Murchison schrieb:
; > Can you try this patch:
; >
; > 
https://bugzilla.andrew.cmu.edu/cgi-bin/cvsweb.cgi/src/cyrus/lib/libconfig.c.diff?r1=1.18;r2=1.18.2.1
;
; I tried the patch. Compiling is ok, but the master segfaults when I try to 
start:
; 347 if (buf[len-1] == '\\') {

That's the line that the patch changes, and that's the old version of the
line, so something didn't take with your patching.

A.

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Cyrus IMAPd 2.3.12 Released (fwd)

[Andy Fiddaman]

; On Wed, 23 Apr 2008, Thomas Robers - TuTech Innovation GmbH wrote:
; ; Hello,
; ;
; ; I'm getting a segmentation fault when I try to start the master process.
; ; Compiling went fine on Debian etch (2.6.18-5-xen-amd64) with gcc version
; ; (GCC) 4.1.2 20061115 (prerelease) (Debian 4.1.1-21)
;
; Take a look at http://bugzilla.andrew.cmu.edu/show_bug.cgi?id=3059 and try
; the patch in there. It looks like the same problem.

Or as a workaround, remove all blank lines from the config file or use
#
instead.

A.


Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Cyrus IMAPd 2.3.12 Released

[Andy Fiddaman]

On Wed, 23 Apr 2008, Thomas Robers - TuTech Innovation GmbH wrote:
; Hello,
;
; I'm getting a segmentation fault when I try to start the master process.
; Compiling went fine on Debian etch (2.6.18-5-xen-amd64) with gcc version
; (GCC) 4.1.2 20061115 (prerelease) (Debian 4.1.1-21)

Take a look at http://bugzilla.andrew.cmu.edu/show_bug.cgi?id=3059 and try
the patch in there. It looks like the same problem.

A.

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Restricting admin logins

[Andy Fiddaman]

At the moment we patch the Cyrus IMAP server source so that administrators
(admins in the config file) can only log in from certain IP addresses.

I was wondering if there is a better way to do this or whether some means
of achieving this is planned for future releases?

We have two patches we currently use:

1/ adds a new flag to imapd which means 'do not allow administrative
logins' and we use this flag on the instance on port 143 but not on
another instance on port 144. Port 144 can then be firewalled to everyone
apart from our administrative clients.

2/ adds a new config file option 'admins_ip' which is a list of subnets
from which administrative logins are permitted.

If it would be of benefit I would be happy to clean up either of these and
submit it to the developers.

Thanks,

Andy

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Setting the location of proc independent of configdir

[Andy Fiddaman]

On Thu, 15 Nov 2007, Ian G Batten wrote:

; I was interested to see someone suggesting putting proc into tmpfs.
; That's slightly painful if /var/imap is in ZFS: the order in which
; mounts take place means you can't just put /var/imap/proc tmpfs into /
; etc/vfstab if /var/imap is coming in through ZFS.  A glance at the
; source code says that proc is hardwired to be configdir/proc.  Would
; it be worth considering making procdir: a configuration option, so
; that it can easily be put somewhere which isn't under configdir:?
; Indeed, for a lot of people, just shoving it in /tmp/cyrusproc would
; be a good option.

Yes please. I have the same problem on Solaris with ZFS so I've resorted
to setting proc/ as not mount-at-boot and modifying my Cyrus SMF manifest
to mount it prior to starting Cyrus. It works but it's messy!

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: tools/migrate-metadata problem

[Andy Fiddaman]

On Mon, 8 Oct 2007, Rudy Gevaert wrote:
;
; I tried your patch and it doesn't really recurse here.  It does recurse if I
; match on subdir !~ /^\./  You are matching on every character at the beginning
; of the line.

Yes, I spotted the missing backslash when I used the script on my
replica..

Hopefully some variant of what we've discussed can be used to fix the
script in 2.3.10 in any case.

Andy

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

tools/migrate-metadata problem

[Andy Fiddaman]

I've just migrated my metadata files to a separate partition on faster
disks and had a problem with the migrate-metadata script. I use
virtdomains and when the script processes directories it skips any
containing a dot. Since all of my domain directories contain a dot, none
of them were processed.

I made this change to get it to work (this is with 2.3.9), it could be
done in one match, this was just a quick fix.

--- cyrus-imapd-2.3.9/tools/migrate-metadata2006-11-30
17:11:25.0 +
+++ cyrus-imapd-2.3.9/tools/migrate-metadata.new2007-10-05
14:06:54.842848647 +
@@ -160,7 +160,7 @@
  ouch "couldn't delete $part$subpath";
}
}
-   elsif ($subdir =~ /^[^\.]+$/s) {
+   elsif (-d "$part$subpath" && $subdir !~ /^./ && $subdir !~ /\.$/) {
# no dot in name, so this is a subdir, process it
mkdir($mpart . $subpath, 0700) ||
  ouch "couldn't create $mpart$subpath";


Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Quick Utilities for getting into user mail file

[Andy Fiddaman]

On Wed, 19 Sep 2007, Rick Kunkel wrote:

; Currently, I'm modifying a pinerc file and running PC-Pine to do this if
; necessary.  Are there any other tools available for this kind of quick and
; dirty mailbox manipulation?  Modifying config files on a per-user basis is
; kind of a pain, and running webmail can be slow with huge mailboxes.

If you set up ACLs on the users' mailboxes to allow you access then you
can create a folder collection in pine pointing at "Other Users/" and
navigate from there.

Pine also has a 'g' command which lets you enter a mailbox path to access
which is possibly more convenient than editting .pinerc, although the
mailbox paths are a bit long..

Andy

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Crash in perl XS module.

[Andy Fiddaman]

I'm seeing a crash whenever a perl process exits that uses the Cyrus
IMAP.pm perl module.

The cleanup function in the IMAP.xs checks the reference count post
decrement rather than pre, therefore it double frees memory and ends
up calling imclient_destroy() twice for the same object.

The following is from the unpatched version save enabling the debug
lines that are already in IMAP.xs but commented out.

localhost> ^D
!DESTROY 100602e60 1
closing
!DESTROY 100602e60 1
closing

closing means that imclient_close() is being called.

Core was generated by `/opt/perl/bin/perl -I/opt/cyrus/lib/site_perl
-MCyrus::IMAP::Shell -e shell --'.
Program terminated with signal 10, Bus error.
#0 0x7e00bf44 in imclient_eof (imclient=0x1006037f0) at
imclient.c:966
966 (*cmdcb->proc)(imclient, cmdcb->rock, &reply);
(gdb) where
#0 0x7e00bf44 in imclient_eof (imclient=0x1006037f0) at
imclient.c:966
#1 0x7e009588 in imclient_close (imclient=0x1006037f0)
at imclient.c:309
#2 0x7e0085a0 in XS_Cyrus__IMAP_DESTROY ()
from /opt/cyrus/lib/site_perl/5.8.8/sun4-solaris/auto/Cyrus/IMAP/IMAP.so

(gdb) print cmdcb
$1 = (struct imclient_cmdcallback *) 0x11

(gdb) print *imclient
$2 = {fd = 0, servername = 0x0, flags = 1,
outbuf = "\000xI\020\000\000\000\001\000R\tP\000\000\000\001\000`.P", '\0'
,
"\001\000`<\200\000\000\000\001\000$$TH\000\000\000\001\000`A\200", '\0'
, "\002\000\000\000\000\000\000\000\001\000R\v\220",
'\0' , "\002\000\000\000A\000\000\000\001\000R\v", '\0'
, "\002\000\000\000\000\000\000\000\001\000R\023\220",
'\0' , "\001\000`G ", '\0' ,
"\001\000`J ~\000\212I\000\000\000\001\000aETH", '\0' , "\001\000`7TH", '\0' , "c", '\0' ...,
outptr = 0x2d7573657224 ,
outleft = 17,
outstart = 0x4c4f422729207b09 ,
replybuf = 0x1006047f0 "",
replystart = 0x6465722064656275 ,
replyliteralleft = 16, replylen = 2314894345321193533,
alloc_replybuf = 4301278864, state = 0x2d617574687a, maxplain = 0,
gensym = 341158362702717, readytag = 0, readytxt = 0x0,
cmdcallback = 0x11, callback_num = 1848073326, callback_alloc = 574294560,
callback = 0x2d6e6f746c73, interact_results = 0x647375622061,
saslconn = 0x11, saslcompleted = 2032171901, tls_ctx = 0x2d73657276696365,
tls_conn = 0x205c26416464, tls_on = 0}



With the following patch it works fine:


--- cyrus-imapd-2.3.9.dist/perl/imap/IMAP.xs2006-11-30 17:11:23.0 
+
+++ cyrus-imapd-2.3.9/perl/imap/IMAP.xs 2007-09-05 14:52:23.800979396 +
@@ -320,7 +320,7 @@
struct xscb *nx;
 CODE:
 /* fprintf(stderr, "!DESTROY %p %d\n", client, client->cnt); */
-   if (!--client->cnt) {
+   if (!client->cnt--) {
 /* printf("closing\n"); */
  imclient_close(client->imclient);
  while (client->cb) {


as can be seen with this debug output:

localhost> ^D
!DESTROY 100602e60 1
!DESTROY 100602e60 0
closing


Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: 64-bit alignment problems.

[Andy Fiddaman]

On Mon, 17 Sep 2007, Ken Murchison wrote:
;
; I didn't like the way that the macro looked in the code, so I just typedef'd
; it instead.  Here is my current patch.  If it works, I'll commit it.

Seems fine, no crashes with my test suite.

Thanks!

(for the other crash bug in the perl XS, would you rather I addressed it
via the list or another route?)

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: 64-bit alignment problems.

[Andy Fiddaman]

On Mon, 17 Sep 2007, Ken Murchison wrote:
; Good point.  I'm wondering if we need the struct at all, and can just use a
; union.

You're right, I've just tested the following and it works fine:

#define ALIGNBUF(buf, size) \
union { \
bit64 align8; \
char buf[(size)]; \
} align8buf; \
char *buf = align8buf.buf



Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: 64-bit alignment problems.

[Andy Fiddaman]

On Fri, 14 Sep 2007, Ken Murchison wrote:
; This patch tries to force the entire buffer to be aligned, rather than
; aligning each 64-bit field individually.  Don't know if this will work or not.

The concept seems to work fine. There are some other places that
the same trick needs to be done - mailbox_expunge() and restore_expunged()
for example.

I adapted it slightly (see attachment) to use a union to conserve heap
space and a macro given the number of places it was proving necessary.
I also tried using the GNUC extension of specifying alignment but
it doesn't seem to work at all for me so I commented it out.

The following two bits aren't required - the pointer doesn't need to be
aligned, just the memory to which it points. That is already
done by malloc(). Also I don't think that anonymous unions are standard C.

-unsigned char *buf  = NULL;
+struct {
+#ifdef HAVE_LONG_LONG_INT
+   bit64 align8;
+#endif
+   unsigned char *buf;
+} align8buf;
+unsigned char *buf = align8buf.buf = NULL;

&

-const char *index_base;
+union {
+#ifdef HAVE_LONG_LONG_INT
+   bit64 align8;
+#endif
+   const char *index_base;
+};


I haven't had much time to look at split_attribs() properly so I just
left that the way my previous patch worked for now.

Andydiff -ur cyrus-imapd-2.3.9.dist/imap/annotate.c 
cyrus-imapd-2.3.9/imap/annotate.c
--- cyrus-imapd-2.3.9.dist/imap/annotate.c  2007-08-15 17:20:55.0 
+
+++ cyrus-imapd-2.3.9/imap/annotate.c   2007-09-15 19:20:10.749460641 +
@@ -310,7 +310,8 @@
 
 /* xxx use datalen? */
 /* xxx sanity check the data? */
-attrib->size = (size_t) ntohl(*(unsigned long *) data);
+memcpy(&tmp, data, sizeof(unsigned long));
+attrib->size = (size_t) ntohl(tmp);
 data += sizeof(unsigned long); /* skip to value */
 
 attrib->value = data;
diff -ur cyrus-imapd-2.3.9.dist/imap/mailbox.c cyrus-imapd-2.3.9/imap/mailbox.c
--- cyrus-imapd-2.3.9.dist/imap/mailbox.c   2007-07-20 14:21:57.0 
+
+++ cyrus-imapd-2.3.9/imap/mailbox.c2007-09-16 09:01:24.869639877 +
@@ -1283,7 +1283,7 @@
  */
 int mailbox_write_index_header(struct mailbox *mailbox)
 {
-char buf[INDEX_HEADER_SIZE];
+ALIGNBUF(buf, INDEX_HEADER_SIZE);
 unsigned long header_size = sizeof(buf);
 int n;
 
@@ -1388,7 +1388,7 @@
   int sync)
 {
 int n;
-char buf[INDEX_RECORD_SIZE];
+ALIGNBUF(buf, INDEX_RECORD_SIZE);
 
 mailbox_index_record_to_buf(record, buf);
 
@@ -1522,8 +1522,8 @@
 unsigned long exists;
 unsigned msgno;
 bit32 oldstart_offset, oldrecord_size, recsize_diff;
-char buf[INDEX_HEADER_SIZE > INDEX_RECORD_SIZE ?
-INDEX_HEADER_SIZE : INDEX_RECORD_SIZE];
+ALIGNBUF(buf, INDEX_HEADER_SIZE > INDEX_RECORD_SIZE ?
+INDEX_HEADER_SIZE : INDEX_RECORD_SIZE);
 char *bufp;
 int quota_offset = 0;
 int calculate_flagcounts = 0;
@@ -1851,8 +1851,8 @@
   mailbox_decideproc_t *decideproc, void *deciderock,
   int expunge_flags)
 {
-char buf[INDEX_HEADER_SIZE > INDEX_RECORD_SIZE ?
-INDEX_HEADER_SIZE : INDEX_RECORD_SIZE];
+ALIGNBUF(buf, INDEX_HEADER_SIZE > INDEX_RECORD_SIZE ?
+INDEX_HEADER_SIZE : INDEX_RECORD_SIZE);
 unsigned msgno;
 unsigned newexpunged;
 unsigned newexists;
@@ -2034,8 +2034,8 @@
 unsigned newanswered;
 unsigned newflagged; 
 
-char buf[INDEX_HEADER_SIZE > INDEX_RECORD_SIZE ?
-INDEX_HEADER_SIZE : INDEX_RECORD_SIZE];
+ALIGNBUF(buf, INDEX_HEADER_SIZE > INDEX_RECORD_SIZE ?
+INDEX_HEADER_SIZE : INDEX_RECORD_SIZE);
 unsigned msgno;
 struct stat sbuf;
 struct txn *tid = NULL;
diff -ur cyrus-imapd-2.3.9.dist/imap/mailbox.h cyrus-imapd-2.3.9/imap/mailbox.h
--- cyrus-imapd-2.3.9.dist/imap/mailbox.h   2006-11-30 17:11:19.0 
+
+++ cyrus-imapd-2.3.9/imap/mailbox.h2007-09-16 18:00:00.082289457 +
@@ -74,6 +74,23 @@
 #define MODSEQ_FMT "%lu"
 #endif
 
+#ifdef HAVE_LONG_LONG_INT
+#if 0 /* __GNUC__ */
+#define ALIGNBUF(buf, size) char buf[(size)] __attribute__ ((aligned (8)))
+#else
+#define ALIGNBUF(buf, size) \
+   struct { \
+   union { \
+   bit64 align8; \
+   char buf[(size)]; \
+   } u; \
+   } align8buf; \
+   char *buf = align8buf.u.buf
+#endif
+#else
+#define ALIGNBUF(buf, size) char buf[(size)]
+#endif
+
 #define MAX_MAILBOX_NAME 490
 #define MAX_MAILBOX_PATH 4096
 
diff -ur cyrus-imapd-2.3.9.dist/imap/reconstruct.c 
cyrus-imapd-2.3.9/imap/reconstruct.c
--- cyrus-imapd-2.3.9.dist/imap/reconstruct.c   2007-04-03 13:01:12.0 
+
+++ cyrus-imapd-2.3.9/imap/reconstruct.c2007-09-16 09:01:51.717884602 
+
@@ -439,8 +439,8 @@
 char newfnamebuf[MAX_MAILBOX_PATH+1];
 struct stat sbuf;
 
-char buf[((INDEX_HEADER_SIZE > INDEX_RECORD_SIZE) ?
- INDEX_HEADER_SIZE : INDEX_REC

Re: 64-bit alignment problems.

[Andy Fiddaman]

On Fri, 14 Sep 2007, Ken Murchison wrote:
;
; Before I commit your patch, would you be willing to see if the attached
; alternate patch works?
;
; This patch tries to force the entire buffer to be aligned, rather than
; aligning each 64-bit field individually.  Don't know if this will work or not.

No problem, I'll try it over the weekend. It looks very plausible. Since
only OFFSET_HIGHESTMODSEQ_64 is unaligned within the buffer your patch
should have far less speed impact than mine.

I'll still need to patch annotate.c I imagine but I'll try without first.

Andy

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Cyrus lagging accepting IMAP connections

[Andy Fiddaman]

On Tue, 11 Sep 2007, Rick Kunkel wrote:

; # telnet mail 143
; Trying xxx.xxx.xxx.xxx...
; Connected to mail.
; Escape character is '^]'.
;
; And then it just kinda sits.  Sometimes, after 30 seconds or so
;
; * OK mail Cyrus IMAP4 v2.2.13-Debian-2.2.13-10 server ready

Generally this kind of delay is due to one of the following things:

o DNS problem
  The reverse DNS lookup is timing out for the client, or the server
  interface (if using virtualdomains).

o Insufficient entropy for sasl
  When a lot of users are connecting, sasl can block because it hasn't
  enough entropy - recompile SASL to use /dev/urandom instead of
  /dev/random.

o Linux 2.6 kernel
  Some people have reported this with Linux 2.6 and solved it by reverting
  to 2.4, I haven't seen any details on exactly what the problem is
  though..

HTH,

A.

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: 64-bit alignment problems.

[Andy Fiddaman]

On Fri, 7 Sep 2007, Ken Murchison wrote:

; We have been running 2.3.x code on 64-bit sparcv9 kernels (Solaris 8 on Sun
; Fire V240) without any problem.  What is your hardware and OS?

The test server is a Sun V210 running
SunOS xxx 5.10 Generic_125100-10 sun4u sparc SUNW,Sun-Fire-V210

(Solaris 10u2 fully patched as of ~4 weeks ago)

I've compiled Cyrus as a 64-bit application:

# file /opt/cyrus/bin/imapd
/opt/cyrus/bin/imapd:   ELF 64-bit MSB executable SPARCV9 Version 1, 
dynamically linked, not stripped

CFLAGS="-g -fPIC" ./configure \
--prefix=/opt/cyrus \
--with-cyrus-prefix=/opt/cyrus \
--sysconfdir=/opt/cyrus/etc \
--with-openssl=/opt/openssl \
--with-sasl=/opt/CYRsasl2 \
--with-bdb=/opt/bdb \
--with-bdb-incdir=/opt/bdb/include \
--with-bdb-libdir=/opt/bdb/lib \
--enable-replication \
--enable-murder \
--with-perl=/opt/perl/bin/perl

Without the patches I get a lot of Bus errors in various processes,
examples below - as you can see they're all related to an unaligned 64-bit
dereference.

Program terminated with signal 10, Bus error.
#0 0x000100060138 in mailbox_index_record_to_buf (
record=0x7fffdb20, buf=0x7fffd9ec "") at mailbox.c:1372
1372 *((bit64 *)(buf+OFFSET_MODSEQ_64)) = htonll(record->modseq);

Program terminated with signal 10, Bus error.
#0 0x000100062b78 in mailbox_expunge (mailbox=0x100270558, decideproc=0,
deciderock=0x0, flags=0) at mailbox.c:2194
2194 *((bit64 *)(buf+OFFSET_QUOTA_MAILBOX_USED64)) = htonll(0);

Program terminated with signal 10, Bus error.
#0 0x00019e38 in restore_expunged (mailbox=0x7fffe618,
msgs=0x100202900, eexists=1, expunge_index_base=0x7a10 "",
numrestored=0x7fffd584, unsetdeleted=0) at unexpunge.c:307
307 newquotaused =

Program terminated with signal 10, Bus error.
#0 0x00010002af64 in split_attribs (data=0x7c017014 "",
datalen=30, attrib=0x7fffeed0) at annotate.c:313
313 attrib->size = (size_t) ntohl(*(unsigned long *) data);

etc. etc.


Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: 64-bit alignment problems.

[Andy Fiddaman]

On Tue, 4 Sep 2007, Andy Fiddaman wrote:
; I'm finally getting around to upgrading my 2.3.7 system to 2.3.9. When I
; put in 2.3.7 I had to patch a few things to make it work properly on
; 64-bit SPARC, mainly unaligned dereferences causing BUS faults due to
; the SPARC MMU.

Sorry to follow up my own post but I've finished testing cyrus 2.3.9 on
64-bit Solaris SPARC and I now have a stable server as far as I can tell
(we have a fairly intensive test suite...)

In stock 2.3.9 there are quite a few 64-bit alignment problems which cause
BUS errors and a problem with the Perl module XS that causes the imclient
structure to be double freed and therefore crashes Perl most of the time.

I've put patches for these up at http://www.fiddaman.net/cyrus/ please
feel free to use them (or something like them) as you wish. It would be
good to have these fixes rolled into the main Cyrus release.

Thanks,

Andy


Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

64-bit alignment problems.

[Andy Fiddaman]

I'm finally getting around to upgrading my 2.3.7 system to 2.3.9. When I
put in 2.3.7 I had to patch a few things to make it work properly on
64-bit SPARC, mainly unaligned dereferences causing BUS faults due to
the SPARC MMU.

Thankfully 2.3.9 has most of these sorted, mainly through using align_htonll()
and friends to force aligned accesses, but at least one place still
causes a crash for me.

Here's the patch, could it be included in the next release? I did send
the patches to [EMAIL PROTECTED] at the time but never got a
response, nor did they get fixed in subsequent releases so apologies if
anyone feels this is the wrong forum for this.

--- cyrus-imapd-2.3.9.dist/imap/annotate.c  2007-08-15 17:20:55.0 
+
+++ cyrus-imapd-2.3.9/imap/annotate.c   2007-09-04 15:01:48.066105997 +
@@ -310,7 +310,8 @@

 /* xxx use datalen? */
 /* xxx sanity check the data? */
-attrib->size = (size_t) ntohl(*(unsigned long *) data);
+memcpy(&tmp, data, sizeof(unsigned long));
+attrib->size = (size_t) ntohl(tmp);
 data += sizeof(unsigned long); /* skip to value */

 attrib->value = data;

and a backtrace from cyr_expire..

Program terminated with signal 10, Bus error.
#0  0x00010002af64 in split_attribs (data=0x7c017014 "",
datalen=30, attrib=0x7fffeed0) at annotate.c:313
313 attrib->size = (size_t) ntohl(*(unsigned long *) data);
(gdb) where
#0  0x00010002af64 in split_attribs (data=0x7c017014 "",
datalen=30, attrib=0x7fffeed0) at annotate.c:313
#1  0x00010002e094 in annotatemore_lookup (
mboxname=0x7fffeef5 "blah.com!user.fred",
entry=0x100061990 "/vendor/cmu/cyrus-imapd/expire", userid=0x1000619b0 "",
attrib=0x7fffeed0) at annotate.c:1349

Thanks,

Andy

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Replication and single instance store

[Andy Fiddaman]

On Tue, 4 Sep 2007, Bron Gondwana wrote:
; Does this mean there is no way to get single-instance-store on a replica
; if you're rebuilding it from scratch?

I've always used rsync to initialise the replica since that will
preserve the hard links - so rsync followed by shutdown, rsync
again and remove sync logs, then start it all up.

I run this on the replica to re-instate it.

for d in config data sieve; do
echo "* Syncing /$d/"
rsync \
--verbose \
--recursive \
--perms --owner --group --times \
--links --hard-links \
--delete \
--human-readable --human-readable \
--exclude=.zfs \
master:$cyrusroot/$d/ $cyrusroot/$d
done

A.


Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: using address-extensions for storing in special folders

[Andy Fiddaman]

On Tue, 7 Aug 2007, Stefan Palme wrote:
; My question is: is there a way to configure the IMAP server, so that
; mails to [EMAIL PROTECTED] are automatically stored
; in the INBOX/internal folder of user "kleiner"?

Yes, the IMAP server does this natively. You just need to make sure that
INBOX/internal has ACL 'anyone p' and it should work. Without that ACL the
messages will end up in INBOX.

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: optimization for delivery to multiple recipients

[Andy Fiddaman]

On Sat, 7 Jul 2007, Alain Spineux wrote:

; Hi
;
; When I make a copy of an email, (using imap command "copy") I can see that
; cyrus just created a new link on the file system.
;
; But when delivering the same mail to multiple user, cyrus create
; multiple identical copy
; of the mail, why ?

If you put this in your imapd.conf, Cyrus should create hard links for
messages delivered to multiple recipients.

singleinstancestore: true

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Header rewriting within existing mailstore - Disposition-Notification-To

[Andy Fiddaman]

On Thu, 5 Jul 2007, Bob Bob wrote:
;
; I don't know enough about Cyrus internal workings to know if this will
; cause any index or other issue. I have seen Disposition lines inside the
; cyrus.cache file for example. I have used the tools to recover corrupt
; stores in the past but don't remember the details. I'd like to think the
; processing I do wouldnt cause that much of a breakage though!

You'll need to run a reconstruct on the mailboxes after making the change.
As you noted there are references in the cache file and also the size of
each message file is in there so that will need updating.

A.

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Cyrus & ZFS performance

[Andy Fiddaman]

On Tue, 3 Jul 2007, Dale Ghent wrote:

; On Jul 3, 2007, at 7:05 PM, Vincent Fox wrote:
;
; > We currently have 27,323 accounts on it, and it is performing
; > very well, with atime=off and compression=on.
;
; Sorry for the double reply, but by the way, what sort of compression
; ratio are you seeing on your ZFS filesystems?

% zfs get compressratio data/cyrus
NAME PROPERTY   VALUE  SOURCE
data/cyrus   compressratio  1.23x  -


(It did get a bit silly last week with this, I'm sure they're not supposed
to get this big.

cyrus% (56) du -h cyrus.expunge
  60K   cyrus.expunge
cyrus% (57) ls -l cyrus.expunge
-rw---   1 cyrusmail320G Jun 11 14:45 cyrus.expunge

)

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: ZFS compression?

[Andy Fiddaman]

On Tue, 24 Apr 2007, Vincent Fox wrote:

; Has anyone attempted using ZFS compression on mail spools?

Yes, we use it here on some servers. The servers were originally IO bound
but since enabling compression (and delayed expunge...) they are coping
fine. The CPU utilisation went up a bit as a result but the server's still
nowhere near being busy.

A.

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Recomendations for a 15000 Cyrus Mailboxes

[Andy Fiddaman]

On Tue, 10 Apr 2007, Timo Schoeler wrote:
;
; Best: ZFS on Solaris ;)

Or now on MacOSX (Leopard) or on FreeBSD (7.0)!

(although Solaris is likely to remain the most stable implementation for
the moment)

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: unexpunge and subfolders

[Andy Fiddaman]

On Tue, 17 Oct 2006, Leena Heino wrote:

; If user has not deleted the whole subfolder only some of the messages then is
; there any way to restore those messages with unexpunge?
;
; It seems that if any subfolders is specified as an argument then then unexpuge
; segfaults eg.: unexpunge -a user.foo.subfolder
;
; (gdb) bt
; #0  0xfedb455c in strlen () from /usr/lib/libc.so.1
; #1  0xfee07058 in _doprnt () from /usr/lib/libc.so.1
; #2  0xfee090c8 in vsnprintf () from /usr/lib/libc.so.1
; #3  0xfedd62a4 in vsyslog () from /usr/lib/libc.so.1
; #4  0xfedd5de0 in syslog () from /usr/lib/libc.so.1
; #5  0x000178e8 in main (argc=4, argv=0xffbffe5c) at unexpunge.c:507
;
; unexpunge.c:507
;505  if (r || expunge_fd == -1) {
;506  /* mailbox corrupt/nonexistent -- skip it */
;507  syslog(LOG_WARNING, "unable to open/lock mailbox %s",
; argv[optind]);
;508  if (doclose) mailbox_close(&mailbox);
;509  return 0;
;510  }


Well, the crash is due to a bug in unexpunge.c, line 507 should read:
   syslog(LOG_WARNING, "unable to open/lock mailbox %s", argv[optind -1]);

(I reported this to the cyrus bugs email address a while ago).

However, if this is fixed then you will just get told
'unable to open/lock mailbox user.foo.subfolder'. I have virtdomains enabled
and use:

unexpunge -l fiddaman.net\!user.xxx.mailing_lists.cyrus

for example, which works fine. Not sure what you do for the default domain or
with virtdomains disabled, maybe try '\!user.foo.subfolder' ?

HTH.

Andy


Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: master / slave replication (sync_server / sync_client)

[Andy Fiddaman]

On Wed, 4 Oct 2006, Martin Schweizer wrote:

; Hello Andy
;
; Thank you for the hint. I change it as you desribed.
;
; Now after waiting some time I see no replication on both site. Is the
; firewall the problem (only open 2005/tcp)? Or what do I wrong again?
;
; Regards,
; Martin

Try telnet from the master to the slave on port 2005 - this checks that
syncserver is running ok (2005 is all you need through the firewall):

# telnet slave 2005
Trying xx.xxx.xxx.xxx...
Connected to slave
Escape character is '^]'.
* SASL CRAM-MD5
* OK mailstore.x.xxx Cyrus sync server v2.3.7

If it isn't running (connection refused) then make sure you're logging
debug messages (local6.debug in syslog.conf) and look to see what's wrong
- make sure you put csync in /etc/services

Try running sync_client manually on the master with verbose flag:
(as your Cyrus user)

% sync_client -r -v

Hopefully that will give a clue.

Andy


Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: master / slave replication (sync_server / sync_client)

[Andy Fiddaman]

On Wed, 4 Oct 2006, Martin Schweizer wrote:

; Hello
;
; I use cyrus imapd 2.3.7 and have to servers 192.168.10.6 and
; 192.168.20.3. The subnet is separated by a firewall (port 2005/tcp is
; open for both sites). I see no action anywhere (also not in the log).
; What do I wrong?

The master server needs to run sync_client in the START block and the slave
needs to run syncserver in the SERVICES block.

You have them both in the SERVICES block and the master is running syncserver
instead of sync_client (i.e. you have them reversed).

See http://cyrusimap.web.cmu.edu/imapd/install-replication.html

Andy


Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Can't get cyrus imapd 2.2.12 to work on a sparc

[Andy Fiddaman]

On Sun, 13 Nov 2005, Matthijs Mohlmann wrote:

; Hi,
;
; Andy Fiddaman wrote:
; > On Sat, 12 Nov 2005, Matthijs Mohlmann wrote:
; >
; Thanks a lot, that did the trick. Where can I read more about that bug ?

It's been discussed on this list before and others, try:

http://lists.freebsd.org/pipermail/freebsd-sparc64/2005-January/002658.html
http://article.gmane.org/gmane.mail.imap.cyrus/16062

You might also see problems with skiplist databases - I got crashes using
annotations until I set the database type to Berkeley
(annotation_db: berkeley) and with the perl module (sometimes crashes on
exit).

Otherwise it seems rock solid in a 64-bit environment.

A.


Cyrus Home Page: http://asg.web.cmu.edu/cyrus
Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Can't get cyrus imapd 2.2.12 to work on a sparc

[Andy Fiddaman]

On Sat, 12 Nov 2005, Matthijs Mohlmann wrote:

; Hi,
;
; I'm running FreeBSD 6.0-RELEASE on a Ultra Sparc (500Mhz with 512Mb RAM).
;
; I have setup cyrus with postfix and saslauthd. All components are
; completely working except cyrus.
;
; What I discovered is that I can create mailboxes in cyrus and can see
; them with my imap client. But when I try to deliver a mail to the lmtp
; socket / lmtp tcp socket or try to use cyrus/bin/deliver then it gives me:
;
; Nov 12 18:10:13 hermes lmtpunix[95552]: DBERROR: invalid
; berkeley_cachesize value, using internal default

If you've compiled Cyrus as 64-bit, then there is a known bug which
prevents settings from getting their default values. I had to specifically
define quite a few in my imapd.conf to get it to work properly (this
would explain why it works fine on your i386 box).

Add these lines to your imapd.conf and try again..

berkeley_cachesize: 512
berkeley_locks_max: 5

If you're using sieve, you'll also need:

sieve_maxscripts: 5
sieve_maxscriptsize: 32

for the same reason.

Andy


Cyrus Home Page: http://asg.web.cmu.edu/cyrus
Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Squatter BUS error.

[Andy Fiddaman]

I am running Cyrus IMAP 2.2.6 as a 64-bit binary on Solaris 9. I don't
know whether this is officially supported but it is running perfectly
apart from one small problem; that squatter crashes when indexing
mailboxes due to an address alignment error.

The crash is being caused by the ntohl() call in annotate.c:split_attribs()
(Note that on this architecture, an unsigned long is 64-bits)

attrib->size = (size_t) ntohl(*(unsigned long *) data);

Changing this line to

attrib->size = (size_t) ntohl(*(uint32_t *) data);

stops the crashes but I don't know enough about what's going on here to
know if this is a good fix.

If anyone has any ideas about the proper way to fix this that would be
appreciated.

Thanks,

Andy

12588/1:Incurred fault #5, FLTACCESS  %pc = 0x100030D30
12588/1:  siginfo: SIGBUS BUS_ADRALN addr=0x7CF00334
12588/1:Received signal #10, SIGBUS [default]
12588/1:  siginfo: SIGBUS BUS_ADRALN addr=0x7CF00334

(gdb) where
#0  split_attribs (
data=0x7cf00334 ,
datalen=32, attrib=0x7fffce30) at sys/byteorder.h:139
#1  0x0001000329e4 in annotatemore_lookup (
mboxname=0x7cf00334 ,
entry=0x20 , userid=0x7fffce30 "",
attrib=0x7fffce30) at annotate.c:1199
#2  0x00018bd8 in index_me (
name=0x7fffe1b0 "mydomainx.co.uk!user.xx", matchlen=23,
maycreate=1, rock=0x7fffe0e0) at squatter.c:319
#3  0x00010002284c in find_cb (rockp=0x7fffe940,
key=0x7ce004a4 ,
keylen=23,
data=0x7ce004c0 ,
datalen=62) at mboxlist.c:1853
#4  0x000100044ab4 in myforeach (db=0x1001e0e30,
prefix=0x7fffe560 "*", prefixlen=0, goodp=0x10002238c ,
cb=0x1000226f8 , rock=0x7fffe940, tid=0x0)
at sys/byteorder.h:141
#5  0x000100022f2c in mboxlist_findall (namespace=0x1001d2f28,
pattern=0x7fffea80 "*", isadmin=1, userid=0x0, auth_state=0x0,
proc=0x184ac , rock=0x7fffe940) at mboxlist.c:2032
#6  0x00019150 in main (argc=2, argv=0x7b58)
at squatter.c:529

(gdb) frame 0
#0  split_attribs (
data=0x7cf00334 ,
datalen=32, attrib=0x7fffce50) at sys/byteorder.h:139
139 {

(gdb) list
134
135 /* Convert a network long to a host long.  */
136
137 static __inline__ unsigned long
138 ntohl (unsigned long __arg)
139 {
140   return htonl (__arg);
141 }

---
Cyrus Home Page: http://asg.web.cmu.edu/cyrus
Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html