Re: service imap pid nnn in BUSY state: terminated abnormally
On 09 Oct 2010, at 00:04, Michael D. Sofka wrote: > Is the old format updated on xfer? It is updated on xfer -- when the mtimes of the message files are set to the INTERNALDATE. :wes Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: service imap pid nnn in BUSY state: terminated abnormally
Great. One final question on this topic. I currently have a warm-backup server (2.2.12) that I rsync to each night. As mailboxes are moved to the new 2.3.16 server the rsynced index files would be the wrong version. I can either rsync just the mail messages to the warm backup, and then reconstruct the index files. Or, I can take a slight detour, and upgrade the warm backup server to 2.3.16. >From what you say, I should be able to continue to rsync from the current >2.2.12 server to an upgraded backup server. If the backup should be needed, >Cyrus will upgrade the index files. Is that right? I'm not sure which approach I'll take. The goal is to eventually have a replication server. But that can't be the current backup server since it has insufficient disk space going forward. Mike "Bron Gondwana" wrote: >On Sat, Oct 09, 2010 at 12:04:23AM -0400, Michael D. Sofka wrote: >> Bron, Thanks for the answer. A one way xfer in fine. I'm just testing the >> new server, and the migration process. >> >> Is the old format updated on xfer? The account I moved is working fine. > >Cyrus will automatically update the index format when you first open >a mailbox using a newer version of any Cyrus program (imapd, pop3d, >lmtpd, whatever). This applies both when upgrading a mailbox "in place" >or when the files have been copied to a new server via XFER. The >in-place upgrade should be able to handle index files from all earlier >versions of Cyrus, not just the most recent - so you can upgrade all the >way from 2.2.12 to 2.3.16, even though it's about 3 revisions of index >later! > >The mailbox will be fine on the new server - you just can't transfer >it back. > >Bron. > -- Michael D. Sofka Sr. Systems Programmer Communications & Middleware Technologies Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: service imap pid nnn in BUSY state: terminated abnormally
On 10/9/2010 7:23 AM, Michael D. Sofka wrote: > > I can either rsync just the mail messages to the warm backup, and then > reconstruct the index files. Or, I can take a slight detour, and upgrade the > warm backup server to 2.3.16. > This is something I've been wondering, too. The problem with not copying the indexes is that presumably you lose all the metadata. And if you rsync /var/spool/cyrus/mail it would seem to me that you run the risk of the index being out of sync with the mail spool; i.e. the only way to safely use rsync is stop cyrus rsync restart cyrus Which isn't practical for some environments. imapsync seems to work fairly well and doesn't suffer from any race conditions like this, however requires that you have each user's imap password. The solution that I'm considering is creating a dummy user called "mail-backup" or something like this and giving this user write permission on every mail account. This way a single user could back up all the imap user folders on the server. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: service imap pid nnn in BUSY state: terminated abnormally
> On 10/9/2010 7:23 AM, Michael D. Sofka wrote: >> >> I can either rsync just the mail messages to the warm backup, and then >> reconstruct the index files. Or, I can take a slight detour, and upgrade >> the warm backup server to 2.3.16. >> > > This is something I've been wondering, too. The problem with not > copying the indexes is that presumably you lose all the metadata. And > if you rsync /var/spool/cyrus/mail it would seem to me that you run the > risk of the index being out of sync with the mail spool; i.e. the only > way to safely use rsync is > > stop cyrus > rsync > restart cyrus > > Which isn't practical for some environments. > > imapsync seems to work fairly well and doesn't suffer from any race > conditions like this, however requires that you have each user's imap > password. The solution that I'm considering is creating a dummy user Fortunately there is no need to do so :) All you need is defining a user in the "proxyservers" option in imapd.conf and use this one to authenticate with imapsync. It works very well. Simon Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: service imap pid nnn in BUSY state: terminated abnormally
Imapsync with the proxy user is a third option during migration. I tested imapsync previously and it worked. But rsync was much faster. As to index consistency, I run the backups from an lvm snapshot. Mike "Simon Matter" wrote: >> On 10/9/2010 7:23 AM, Michael D. Sofka wrote: >>> >>> I can either rsync just the mail messages to the warm backup, and then >>> reconstruct the index files. Or, I can take a slight detour, and upgrade >>> the warm backup server to 2.3.16. >>> >> >> This is something I've been wondering, too. The problem with not >> copying the indexes is that presumably you lose all the metadata. And >> if you rsync /var/spool/cyrus/mail it would seem to me that you run the >> risk of the index being out of sync with the mail spool; i.e. the only >> way to safely use rsync is >> >> stop cyrus >> rsync >> restart cyrus >> >> Which isn't practical for some environments. >> >> imapsync seems to work fairly well and doesn't suffer from any race >> conditions like this, however requires that you have each user's imap >> password. The solution that I'm considering is creating a dummy user > >Fortunately there is no need to do so :) >All you need is defining a user in the "proxyservers" option in imapd.conf >and use this one to authenticate with imapsync. It works very well. > >Simon > > >Cyrus Home Page: http://www.cyrusimap.org/ >List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ > -- Michael D. Sofka Sr. Systems Programmer Communications & Middleware Technologies Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: packages for solaris 10 x86
On Fri, Oct 08, 2010 at 08:06:33PM -0700, Andrew Morgan wrote: > On Fri, 8 Oct 2010, Frank Pittel wrote: > > >Now I'm trying to compile cyrus-imapd and am running into the following > >error: > > > >Sigh!! there has to be an easier way. Anyone have an idea of how > >to get past this? I'm not a programmer. > > It seems to be failing when it tries to link in the snmp libraries. > Personally, I've never used the snmp features of Cyrus. You could > try disabling them: > > make clean > ./configure --without-snmp --whatever-other-options-you-use > make > That helped but I then I ran into more troubles. I'm starting to wonder if it'll work when I do get it compiled! The errors I'm getting now is: gcc -L/usr/local/BerkeleyDB.4.7/lib -R/usr/local/BerkeleyDB.4.7/lib -o imapd \ ../master/service.o pushstats.o imapd.o proxy.o imap_proxy.o index.o version.o mutex_fake.o \ libimap.a ../lib/libcyrus.a ../lib/libcyrus_min.a -L/opt/sasl/lib -R/opt/sasl/lib -lsasl2 -lgss -lresolv -lresolv -lssl -lcrypto -lresolv -lresolv -lsocket -lnsl -L/usr/local/BerkeleyDB.4.7/lib -R/usr/local/BerkeleyDB.4.7/lib -ldb-4.7 -lz -lmd -lrt ../com_err/et/libcom_err.a Undefined first referenced symbol in file krb5_free_principal ../lib/libcyrus.a(auth_krb5.o) krb5_realm_compare ../lib/libcyrus.a(auth_krb5.o) krb5_build_principal../lib/libcyrus.a(auth_krb5.o) krb5_get_default_realm ../lib/libcyrus.a(auth_krb5.o) krb5_parse_name ../lib/libcyrus.a(auth_krb5.o) krb5_init_context ../lib/libcyrus.a(auth_krb5.o) krb5_free_context ../lib/libcyrus.a(auth_krb5.o) krb5_unparse_name ../lib/libcyrus.a(auth_krb5.o) ld: fatal: Symbol referencing errors. No output written to imapd collect2: ld returned 1 exit status *** Error code 1 make: Fatal error: Command failed for target `imapd' Current working directory /src/cyrus-imapd-2.3.16/imap *** Error code 1 The following command caused the error: for d in man com_err/et lib sieve master imap imtest perl timsieved notifyd; \ do \ (cd $d; echo "### Making" all "in" `pwd`; \ make DESTDIR= all) || exit 1; \ done make: Fatal error: Command failed for target `all' Has anyone been able to get cyrus-imap to compile under solaris-10? If I sound frustrated it's because I am. :-( Frank Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: packages for solaris 10 x86
Am 09.10.10 19:51, schrieb Frank Pittel:
> krb5_free_principal ../lib/libcyrus.a(auth_krb5.o)
> krb5_realm_compare ../lib/libcyrus.a(auth_krb5.o)
> krb5_build_principal../lib/libcyrus.a(auth_krb5.o)
> krb5_get_default_realm ../lib/libcyrus.a(auth_krb5.o)
> krb5_parse_name ../lib/libcyrus.a(auth_krb5.o)
> krb5_init_context ../lib/libcyrus.a(auth_krb5.o)
> krb5_free_context ../lib/libcyrus.a(auth_krb5.o)
> krb5_unparse_name ../lib/libcyrus.a(auth_krb5.o)
GSSAPI needs Kerberos Libs. You don't have them.
> Has anyone been able to get cyrus-imap to compile under solaris-10? If I
> sound frustrated
> it's because I am. :-(
Sure.
For proper 64 bit code to be built I used
export CC=gcc
export CXX=g++
export CFLAGS="-m64 -O3 -march=k8 "
export CXXFLAGS="-m64 -O3 -march=k8"
export CPPFLAGS="-m64 -I/usr/local/test/64/include"
export LDFLAGS="-m64 -Wl,-64 -L/usr/local/test/64/lib
-Wl,-R/usr/local/test/64/lib -L/usr/local/lib/amd64
-Wl,-R/usr/local/lib/amd64"
(We have Opteron servers here).
On my test system here the build configure was:
./configure --prefix=/usr/local/test/64 --with-openssl
--with-ldap=/usr/local/test/64 --with-sasl=/usr/local/test/64
--with-bdb=/usr/local/test/64 --enable-netscapehack
--with-cyrus-prefix=/usr/local/test/64/cyrus
--sysconfdir=/usr/local/test/64/etc/cyrus --enable-listext
--with-snmp=no --disable-snmp --with-idle=idled --enable-idled
--enable-replication --enable-gssapi=no
used libraries by imapd (as an example):
bash-3.00# ldd /usr/local/test/64/cyrus/bin/imapd
libsasl2.so.2 => /usr/local/test/64/lib/libsasl2.so.2
libssl.so.0.9.8 => /usr/local/test/64/lib/libssl.so.0.9.8
libcrypto.so.0.9.8 =>/usr/local/test/64/lib/libcrypto.so.0.9.8
libresolv.so.2 =>/lib/64/libresolv.so.2
libsocket.so.1 =>/lib/64/libsocket.so.1
libnsl.so.1 => /lib/64/libnsl.so.1
libdb-4.5.so => /usr/local/test/64/lib/libdb-4.5.so
libz.so.1 => /usr/lib/64/libz.so.1
libmd.so.1 =>/lib/64/libmd.so.1
librt.so.1 =>/lib/64/librt.so.1
libc.so.1 => /lib/64/libc.so.1
libdl.so.1 =>/lib/64/libdl.so.1
libmp.so.2 =>/lib/64/libmp.so.2
libscf.so.1 => /lib/64/libscf.so.1
libpthread.so.1 => /lib/64/libpthread.so.1
libgcc_s.so.1 => /usr/local/lib/amd64/libgcc_s.so.1
libaio.so.1 => /lib/64/libaio.so.1
libdoor.so.1 => /lib/64/libdoor.so.1
libuutil.so.1 => /lib/64/libuutil.so.1
libgen.so.1 => /lib/64/libgen.so.1
libm.so.2 => /lib/64/libm.so.2
Due to some problems with Sun's OpenSSL package I used my own one. SASL
is also linked to that.
--
Pascal Gienger Jabber/XMPP/Mail: pascal.gien...@uni-konstanz.de
University of Konstanz, IT Services Department ("Rechenzentrum")
Electronic Communications and Web Services
Building V, Room V404, Phone +49 7531 88 5048, Fax +49 7531 88 3739
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: packages for solaris 10 x86
On Sat, 9 Oct 2010, Frank Pittel wrote: > That helped but I then I ran into more troubles. I'm starting to wonder if > it'll work > when I do get it compiled! The errors I'm getting now is: > > > gcc -L/usr/local/BerkeleyDB.4.7/lib -R/usr/local/BerkeleyDB.4.7/lib -o imapd \ > ../master/service.o pushstats.o imapd.o proxy.o imap_proxy.o index.o version.o > mutex_fake.o \ > libimap.a ../lib/libcyrus.a ../lib/libcyrus_min.a -L/opt/sasl/lib > -R/opt/sasl/lib > -lsasl2 -lgss -lresolv -lresolv -lssl -lcrypto -lresolv -lresolv -lsocket > -lnsl > -L/usr/local/BerkeleyDB.4.7/lib -R/usr/local/BerkeleyDB.4.7/lib -ldb-4.7 -lz > -lmd -lrt > ../com_err/et/libcom_err.a > Undefined first referenced > symbol in file > krb5_free_principal ../lib/libcyrus.a(auth_krb5.o) > krb5_realm_compare ../lib/libcyrus.a(auth_krb5.o) > krb5_build_principal../lib/libcyrus.a(auth_krb5.o) > krb5_get_default_realm ../lib/libcyrus.a(auth_krb5.o) > krb5_parse_name ../lib/libcyrus.a(auth_krb5.o) > krb5_init_context ../lib/libcyrus.a(auth_krb5.o) > krb5_free_context ../lib/libcyrus.a(auth_krb5.o) > krb5_unparse_name ../lib/libcyrus.a(auth_krb5.o) > ld: fatal: Symbol referencing errors. No output written to imapd > collect2: ld returned 1 exit status > *** Error code 1 > make: Fatal error: Command failed for target `imapd' > Current working directory /src/cyrus-imapd-2.3.16/imap > *** Error code 1 > The following command caused the error: > for d in man com_err/et lib sieve master imap imtest perl timsieved > notifyd; \ > do \ >(cd $d; echo "### Making" all "in" `pwd`; \ >make DESTDIR= all) || exit 1; \ > done > make: Fatal error: Command failed for target `all' > > > Has anyone been able to get cyrus-imap to compile under solaris-10? If I > sound frustrated it's because I am. :-( Add "--without-krb" and see if it gets you further. Obviously, there are some problems with the build environment or the configure script detection logic for Solaris 10. However, I know several folks from this mailing list are running Solaris 10. Hopefully they will chime in. :) Andy Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: service imap pid nnn in BUSY state: terminated abnormally
On Sat, Oct 09, 2010 at 09:59:44AM -0500, Patrick Goetz wrote: > On 10/9/2010 7:23 AM, Michael D. Sofka wrote: > > > > I can either rsync just the mail messages to the warm backup, and then > > reconstruct the index files. Or, I can take a slight detour, and upgrade > > the warm backup server to 2.3.16. > > > > This is something I've been wondering, too. The problem with not > copying the indexes is that presumably you lose all the metadata. And > if you rsync /var/spool/cyrus/mail it would seem to me that you run the > risk of the index being out of sync with the mail spool; i.e. the only > way to safely use rsync is > > stop cyrus > rsync > restart cyrus > > Which isn't practical for some environments. If you have delayed expunge it should be safe to rsync the metadata first and then the spool afterwards. It's a bit racy of course. Better is to fcntl lock the cyrus.header and cyrus.index files (or lock the foldername.lock file in 2.4 of course... when we release that in OMG 2 days!) But best of all is to not fiddle with the underlying filesystem at all. I want to replace our old and creaky backup system pretty soon (it doesn't grok namelock yet - though it is safe due to the use of index exlusive locks - just inefficient). The replacement will use a cyrus tool that can do a full or partial backup of either a mailbox or an entire user. This will also replace mbdump/undump for XFER (murder mailbox moves) and be identical in syntax to the replication subsystem. I haven't had time to make it all work just how I want in Cyrus 2.4, but it's sitting on my TODO list and slowly rising up it, to the point where it gets written on whiteboards a LOT (including discussion of how to implement it). FastMail/Opera has 3 new programmers who started in the last week - I met them all during interviews, but I'm currently travelling (sitting in Newark airport right now waiting for my flight upstate - 16 hours from Oslo to Syracuse!) on my way to meet Ken and Dave. We'll have more resources to throw at Cyrus internals pretty soon. Some of it specific to our environment of course - but we like to simplify the bits we have to deal with, and that has benefits for everyone. Bron. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: service imap pid nnn in BUSY state: terminated abnormally
On Sat, Oct 09, 2010 at 08:23:15AM -0400, Michael D. Sofka wrote: > Great. One final question on this topic. > > I currently have a warm-backup server (2.2.12) that I rsync to each night. As > mailboxes are moved to the new 2.3.16 server the rsynced index files would be > the wrong version. > > I can either rsync just the mail messages to the warm backup, and then > reconstruct the index files. Or, I can take a slight detour, and upgrade the > warm backup server to 2.3.16. > > From what you say, I should be able to continue to rsync from the current > 2.2.12 server to an upgraded backup server. If the backup should be needed, > Cyrus will upgrade the index files. Is that right? Yes - I would recommend upgrading the backup servers to 2.3.16, since then you will be able to use the index files without a reconstruct. The files from the 2.2.12 server will be upgraded on first use. Of course, then you can't copy them BACK to the original server again. Any reason why you don't just upgrade them all to 2.3.16? Interestingly, the index files are still valuable even if you stay with 2.2.12, because reconstruct will be able to preserve the flag information even from a newer version (within reason - I'm not sure how far back you can go before incompatible changes cause weirdness - but the past few releases should be OK). If you're going to reconstruct anyway, you may as well keep the index files around. > I'm not sure which approach I'll take. The goal is to eventually have a > replication server. But that can't be the current backup server since it has > insufficient disk space going forward. If it has space for backups, it has space for replication (assuming you're using the replication as the backup) - replication is pretty much a smart rsync with automated triggering based on changes. Bron. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: service imap pid nnn in BUSY state: terminated abnormally
On Sat, Oct 09, 2010 at 07:30:36AM -0400, Wesley Craig wrote: > On 09 Oct 2010, at 00:04, Michael D. Sofka wrote: > > Is the old format updated on xfer? > > It is updated on xfer -- when the mtimes of the message files are set to the > INTERNALDATE. Yes, file mtimes get set to INTERNALDATE as much as possible (and vice versa when reconstructing with a missing index - hence the reason for doing it) XFER itself is a super-dumb protocol (I should know, I just rewrote it to work correctly with namelocking in 2.4, and I have one more patch outstanding to discuss with Ken on Monday...) All it does is collects up ALL the message files and meta files and copies them across the wire as-is. No consistency checking, no upgrades. They will be upgraded at the remote end when the mailbox is next opened. I'm planning to replace it with something better... ... and when I say replace, I probably mean "co-locate". To the point that an incoming XFER will cause an automatic upgrade at the point of the XFER, and an outbound XFER will probably actually generate a 2.2.x compatible index file. I don't see the need to support older than that. Cyrus will also advertise a new capability for replication-based transfers which will have much more integrity checking, and will be used if both ends are recent enough. Bron. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: service imap pid nnn in BUSY state: terminated abnormally
On 09 Oct 2010, at 20:31, Bron Gondwana wrote: > Yes, file mtimes get set to INTERNALDATE as much as possible (and vice > versa when reconstructing with a missing index - hence the reason for > doing it) Perhaps you're misunderstanding me... In 2.3.recent, the second to last step of undump (xfer) is to open the mailbox (which will cause it to be upgraded), retrieve the INTERNALDATE, and set the mtime accordingly. In short, xfer in 2.3.recent updates the mailbox, immediately, not when the mailbox is next opened. No idea what 2.4 might do... :wes Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: service imap pid nnn in BUSY state: terminated abnormally
On Sat, Oct 09, 2010 at 10:24:41PM -0400, Wesley Craig wrote: > On 09 Oct 2010, at 20:31, Bron Gondwana wrote: > > Yes, file mtimes get set to INTERNALDATE as much as possible (and vice > > versa when reconstructing with a missing index - hence the reason for > > doing it) > > Perhaps you're misunderstanding me... In 2.3.recent, the second to last step > of undump (xfer) is to open the mailbox (which will cause it to be upgraded), > retrieve the INTERNALDATE, and set the mtime accordingly. In short, xfer in > 2.3.recent updates the mailbox, immediately, not when the mailbox is next > opened. No idea what 2.4 might do... Oh yeah - you're right. That's missing in 2.4 and I think it's because I planned to replace it with replication stuff which uses the append logic which does that on the fly. But I haven't done it yet. Would make sense to add that back, at least for now. Thanks for pointing it out :) Bron. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: service imap pid nnn in BUSY state: terminated abnormally
On 09 Oct 2010, at 23:12, Bron Gondwana wrote: > Oh yeah - you're right. That's missing in 2.4 and I think it's because I > planned to replace it with replication stuff which uses the append logic > which does that on the fly. But I haven't done it yet. Would make sense > to add that back, at least for now. > > Thanks for pointing it out :) No problem. I'm not happy with the 2.3.x xfer-INTERNALDATE-upgrade behavior, so better behavior in 2.4 would be welcome. :wes Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
