Re: competition
> On Wed, Sep 22, 2010 at 04:10:15PM +1000, Bron Gondwana wrote:
>
>> Now - BDB database SHOULD be upgradable. I want to find a BDB expert
>> to help me with that - (a) detecting that an upgrade is necessary, and
>> (b) doing the upgrade.
>
> It was quite some time ago I last upgraded a Cyrus instance, but my
> upgrade script did the following:
>
> - Run the _old_ version of db_recover. Just shutting down Cyrus is not
> enough, without this step db_upgrade can fail.
>
> - Run the _new_ version of db_upgrade.
>
> - Run "db_checkpoint -1" to force the log to be rotated.
>
> I think it won't be hard to implement steps 2 & 3 inside Cyrus, and I
> don't remember those steps taking a significant amount of time, but I
> don't know how it would be possible to implement step 1. However if the
> BDB version change does not bring a change to the on-disk DB log format,
> then it may be possible to skip steps 1 & 3.
Unfortunately that's not enough in many situations. I did uncountable
tests with different BDB versions and came up with the following method in
our RPMs:
...
# make sure our Berkeley databases are in a sane state
# wait for db_checkpoint to end successfully or kill it after a timeout
db_checkpoint -v -1 -h ${imap_prefix}/db &
DB_CHECK_PID=$!
CNT=0
while [ $CNT -lt 60 ]; do
if ! kill -0 $DB_CHECK_PID > /dev/null 2>&1; then
break
fi
sleep 1
let CNT+=1
done
if kill -0 $DB_CHECK_PID > /dev/null 2>&1; then
kill -USR1 $DB_CHECK_PID > /dev/null 2>&1
sleep 1
kill -KILL $DB_CHECK_PID > /dev/null 2>&1
wait $DB_CHECK_PID > /dev/null 2>&1
fi
# do a normal recovery
db_recover -v -h ${imap_prefix}/db
RETVAL=$?
if [ $RETVAL -ne 0 ]; then
# try a catastrophic recovery instead of normal recovery
db_recover -v -c -h ${imap_prefix}/db
RETVAL=$?
ERRVAL=$(( $ERRVAL + $RETVAL ))
if [ $RETVAL -ne 0 ]; then
echo "ERROR: catastrophic recovery of Berkeley databases failed"
fi
fi
if [ "$EXPORT" = "export" ]; then
# convert all db files to portable format for migration
# TODO: quota_db, we don't touch it for now
cvt_file ${imap_prefix}/deliver.db "skiplist"
cvt_file ${imap_prefix}/mailboxes.db "skiplist"
cvt_file ${imap_prefix}/tls_sessions.db "skiplist"
cvt_file ${imap_prefix}/annotations.db "skiplist"
cvt_file ${imap_prefix}/ptclient/ptscache.db "skiplist"
cvt_file ${imap_prefix}/statuscache.db "skiplist"
cvt_file ${imap_prefix}/user_deny.db "flat"
rm -vf ${imap_prefix}/db/log.*
rm -vf ${imap_prefix}/db/__db.*
else
...
This all is done not for upgrades of BDB but for simple cases where BDB is
in a bad state for some reasons.
Now, the db_upgrade may be good for those situations where it works. But
I'm quite sure it doesn't work between some major versions of BDB. And
that's exactly what we had to care in the RPMs, so you can run RHEL2.1 on
i386 and "upgrade" it to RHEL5 on x86_64 and it will just work.
It works, but I'd be very happy to get rid of this stuff, really!
Simon
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: competition
> Kolab Systems is thinking of such SQL databases for integration > purposes, where the performance penalty now lies within having to use the > IMAP protocol to gain access to the underlying metadata (seen status, > message indexes) in distributed groupware environments where Cyrus > itself is not the only component that needs access to such information > (but would still remain authoritative, of course, and would be the only > component with write access to most tables). > > While not necessarily the best approach, it seems worth exploring. Yes, this is one of the big reasons why we use RDB backends in many of our products. The downside is performance, of course. But the upside is very big when you need it -- ability to keep one instance of important metadata and allow multiple pieces of software on multiple servers to access it. Most of the time, this metadata is read-often-update-rarely, so we don't need huge write performance. But if we have to synchronise the data across multiple servers and use local files like SQLite/BerkeleyDB/gdbm, we land up building entire synchronisation layers just to ship updates around from server to server. This is not Cyrus-specific -- any metadata which needs sharing across multiple servers seems to be easier to build with a network-accessible database, and an RDB is the commonest of them (SQLite being an exception). Basically, memcached is "better than" BerkeleyDB hashmaps in these situations. Shuvam Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: competition
On Thu, 23 Sep 2010, Shuvam Misra wrote: > > > given the issues with BDB. Is it worth embedding a copy of > > > BDB into the Cyrus distribution rather than using the OS one? I > > > > That way lies madness. > > > > BDB is one of those things where arcane blackmagic skills are needed to keep > > it working on all arches. It uses scary crap to be fast and archive high > > performance with large concurrency, and I've seen it break OpenLDAP in very > > nasty arch-specific ways in the past (OpenLDAP is to BDB what Cyrus is to > > mmap() ;-) ). > > > > And it will break the world if you get symbol versioning wrong. Lots of > > libraries (SASL *and* glibc included!) might decide to shadow-dynamic-link > > BDB with your application, and all sort of non-funny crap can happen if the > > wrong linker magic is applied. You *really* don't want to go there. > > I was a strong advocate of bundling DB libraries, etc, with Cyrus. The > points you've made here are very interesting. I didn't know many of these > things. I'm re-thinking whether bundling is such a good idea now. Thanks. Bundling is always a pain to get exactly right. Bundled libs are also a security hazard. Bundled libs *will* be disabled by any Linux distro worth being called that, but even disabled they are not harmless. They will waste resources of the security teams while acessing if an embedded copy of whatever lib needs to be updated with a security fix. Bundling also complicate the build system. Complicated build systems are a VERY BAD idea on most projects, where nobody is really paying attention to them. -- "One disk to rule them all, One disk to find them. One disk to bring them all and in the darkness grind them. In the Land of Redmond where the shadows lie." -- The Silicon Valley Tarot Henrique Holschuh Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: competition
> > given the issues with BDB. Is it worth embedding a copy of > > BDB into the Cyrus distribution rather than using the OS one? I > > That way lies madness. > > BDB is one of those things where arcane blackmagic skills are needed to keep > it working on all arches. It uses scary crap to be fast and archive high > performance with large concurrency, and I've seen it break OpenLDAP in very > nasty arch-specific ways in the past (OpenLDAP is to BDB what Cyrus is to > mmap() ;-) ). > > And it will break the world if you get symbol versioning wrong. Lots of > libraries (SASL *and* glibc included!) might decide to shadow-dynamic-link > BDB with your application, and all sort of non-funny crap can happen if the > wrong linker magic is applied. You *really* don't want to go there. I was a strong advocate of bundling DB libraries, etc, with Cyrus. The points you've made here are very interesting. I didn't know many of these things. I'm re-thinking whether bundling is such a good idea now. Thanks. Shuvam Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: competition
On Wed, Sep 22, 2010 at 11:24:04PM +0200, Gabor Gombas wrote: > On Wed, Sep 22, 2010 at 04:10:15PM +1000, Bron Gondwana wrote: > > > Now - BDB database SHOULD be upgradable. I want to find a BDB expert > > to help me with that - (a) detecting that an upgrade is necessary, and > > (b) doing the upgrade. > > It was quite some time ago I last upgraded a Cyrus instance, but my > upgrade script did the following: > > - Run the _old_ version of db_recover. Just shutting down Cyrus is not > enough, without this step db_upgrade can fail. > > - Run the _new_ version of db_upgrade. > > - Run "db_checkpoint -1" to force the log to be rotated. > > I think it won't be hard to implement steps 2 & 3 inside Cyrus, and I > don't remember those steps taking a significant amount of time, but I > don't know how it would be possible to implement step 1. However if the > BDB version change does not bring a change to the on-disk DB log format, > then it may be possible to skip steps 1 & 3. Ok - so the upshot is that we need to implement a "shutdown" path, either in master.c or in ctl_cyrusdb - and we need to make sure it gets run during the usual shutdown path. Bron. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: competition
On Wed, Sep 22, 2010 at 04:10:15PM +1000, Bron Gondwana wrote: > Now - BDB database SHOULD be upgradable. I want to find a BDB expert > to help me with that - (a) detecting that an upgrade is necessary, and > (b) doing the upgrade. It was quite some time ago I last upgraded a Cyrus instance, but my upgrade script did the following: - Run the _old_ version of db_recover. Just shutting down Cyrus is not enough, without this step db_upgrade can fail. - Run the _new_ version of db_upgrade. - Run "db_checkpoint -1" to force the log to be rotated. I think it won't be hard to implement steps 2 & 3 inside Cyrus, and I don't remember those steps taking a significant amount of time, but I don't know how it would be possible to implement step 1. However if the BDB version change does not bring a change to the on-disk DB log format, then it may be possible to skip steps 1 & 3. Gabor Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: competition
On 22 Sep 2010, at 10:46, Dave McMurtrie wrote: > Considering the state of Cyrus' interoperability with BDB and all the > recent fixes to skiplist, would it make sense to at least not make BDB a > default backend from now on? Yes, and "sane defaults" was to be one of the themes of the 2.4 release. Very pertinent if you're interested in making Cyrus IMAPd welcoming to new people... :wes Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: competition
> The big downside to using an SQL database is the enormous temptation to > point all the Cyrus servers at the same Database server and lose the > redundancy and scalability inherent in a multi node or Murder setup. But the SQL world has this figured out, at least for reads. For situations where DB *writes* need to be scaled, which the SQL world doesn't really have figured out, there is always the large list of up-and-coming noSQL solutions and they seem to fit Cyrus' use case pretty well. Cyrus' single-murder-master situation is a write bottleneck that could be alleviated with such a solution. John -- John Madden Sr UNIX Systems Engineer Ivy Tech Community College of Indiana jmad...@ivytech.edu Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: Cyrus Muder: determine backend of a mailbox
On Wed, 22 Sep 2010, Torsten Schlabach wrote:
> Dear list!
>
> We are running a Cyrus Murder configuration.
>
> In the mailboxlist (which I can get using ctl_mboxlist, I can see on
> which backend each mailbox resides physically.
>
> Is there any way to extract that information through the IMAP protocol?
>
> What I mean is, I can issue a command:
>
> list "Mail" "*"
>
> but the output will not list the backend that the mailbox / folder sits
> on.
>
> Are there any options I could use for that?
>
> Or am I on the wrong path and I should achieve this in a different way.
>
> I hope ctl_mboxlist is not the only way to access this information.
Here is a chunk of perl code which uses IMAP to get the mailbox
location/backend:
##
# Locate which backend a mailbox is on
##
sub locatemailbox {
my ($server, $authuser, $authpw, $mailbox) = @_;
my $location = "";
use Mail::IMAPClient;
my $imap = Mail::IMAPClient->new(
Server => $server,
User => $authuser,
Password => $authpw,
Ssl => 1,
);
if (! $imap) {
die("Cannot connect to mail server '$server' to locate mailbox
- $!");
}
my @results = $imap->tag_and_run(qq/GETANNOTATION $mailbox "*"
"value.shared"/);
$imap->logout;
foreach my $r (@results) {
$r =~ s/\r//g;
$r =~ s/\n//g;
if ($r =~ /\/vendor\/cmu\/cyrus-imapd\/server"
\("value.shared""(.*)"\)$/) {
$location = $1;
}
}
if ($location eq '') {
die("Cannot locate mailbox '$mailbox'.\n");
}
return $location;
}
$mailbox is something like "user.morgan". Basically, you need to look at
the mailbox annotations to get this. There are other interesting things
in the annotations. Here are the annotations on my own mailbox:
condstore: false
duplicatedeliver: false
lastpop:
lastupdate: 22-Sep-2010 10:01:36 -0700
partition: p4
pop3newuidl: true
server: cyrus-be3.onid.oregonstate.edu
sharedseen: false
size: 67386044
Andy
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Cyrus Muder: determine backend of a mailbox
Dear list! We are running a Cyrus Murder configuration. In the mailboxlist (which I can get using ctl_mboxlist, I can see on which backend each mailbox resides physically. Is there any way to extract that information through the IMAP protocol? What I mean is, I can issue a command: list "Mail" "*" but the output will not list the backend that the mailbox / folder sits on. Are there any options I could use for that? Or am I on the wrong path and I should achieve this in a different way. I hope ctl_mboxlist is not the only way to access this information. Regards, Torsten Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: competition
Hi, > Kolab Systems is thinking of such SQL databases for integration purposes, > where the performance penalty now lies within having to use the IMAP > protocol to gain access to the underlying metadata (seen status, message > indexes) in distributed groupware environments where Cyrus itself is not > the only component that needs access to such information (but would still > remain authoritative, of course, and would be the only component with write > access to most tables). > > While not necessarily the best approach, it seems worth exploring. What makes you think the query parsing and other overheads of SQL will be faster than IMAP? I'd be interested in any numbers that you might have to support the effort. The big downside to using an SQL database is the enormous temptation to point all the Cyrus servers at the same Database server and lose the redundancy and scalability inherent in a multi node or Murder setup. Regards, @ndy -- andy...@ashurst.eu.org http://www.ashurst.eu.org/ 0x7EBA75FF Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: competition
Kolab Systems is thinking of such SQL databases for integration purposes, where the performance penalty now lies within having to use the IMAP protocol to gain access to the underlying metadata (seen status, message indexes) in distributed groupware environments where Cyrus itself is not the only component that needs access to such information (but would still remain authoritative, of course, and would be the only component with write access to most tables). While not necessarily the best approach, it seems worth exploring. Kind regards, Jeroen van Meeuwen -From his Android "André Schild" wrote: > Am 22.09.2010 16:17, schrieb Lucas Zinato Carraro: >> For me it would be very interesting a option to save cyrus tables >>in a traditional database. ( mysql, postgresql, etc... ) >Beside "interesting" what would you get for a real benefit from this ? >They are ver verly likely to be slower. > >André > > >Cyrus Home Page: http://www.cyrusimap.org/ >List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: competition
On Wed, 22 Sep 2010, Bron Gondwana wrote: > Now - BDB database SHOULD be upgradable. I want to find a BDB expert > to help me with that - (a) detecting that an upgrade is necessary, and > (b) doing the upgrade. All I know is that there used to be an API call to upgrade the db environment, which basically did exactly what the db_upgrade utility does. It was easy to find before Oracle bought SleepyCat and screwed over with the documentation, nowadays I don't even know where the API docs are. If it is still exists, you could probably upgrade all DBs the first time they're accessed (if the db->upgrade() call is cheap enough when there is no work to be done, something I don't know). > So the only thing really left to do is making sure we can upgrade > BDB databases automatically. Sure downgrades will suck, but that's > not quite so scary. Downgrades can only be done manually anyway, by dump+restore. However, the file formats don't change as much as the API/ABI anyway, often two different BDB releases use the same file format. > given the issues with BDB. Is it worth embedding a copy of > BDB into the Cyrus distribution rather than using the OS one? I That way lies madness. BDB is one of those things where arcane blackmagic skills are needed to keep it working on all arches. It uses scary crap to be fast and archive high performance with large concurrency, and I've seen it break OpenLDAP in very nasty arch-specific ways in the past (OpenLDAP is to BDB what Cyrus is to mmap() ;-) ). And it will break the world if you get symbol versioning wrong. Lots of libraries (SASL *and* glibc included!) might decide to shadow-dynamic-link BDB with your application, and all sort of non-funny crap can happen if the wrong linker magic is applied. You *really* don't want to go there. -- "One disk to rule them all, One disk to find them. One disk to bring them all and in the darkness grind them. In the Land of Redmond where the shadows lie." -- The Silicon Valley Tarot Henrique Holschuh Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
RE : competition
> We wanted to use it for the user_deny database so we could insert a row > into one database table that every host has access to. This way we > didn't need to come up with a way to update the local user_deny across > each frontend server. Such database provides the same benefit to the tlscache DB. Our frontend servers share a Mysql DB (HEAP) to cache TLS > Also, we were interested in testing SQLite for things like > tls_sessions.db and deliver.db because we were tired of BDB breaking our > service. Ce message et les pièces jointes sont confidentiels et réservés à l'usage exclusif de ses destinataires. Il peut également être protégé par le secret professionnel. Si vous recevez ce message par erreur, merci d'en avertir immédiatement l'expéditeur et de le détruire. L'intégrité du message ne pouvant être assurée sur Internet, la responsabilité du groupe Atos Origin ne pourra être recherchée quant au contenu de ce message. Bien que les meilleurs efforts soient faits pour maintenir cette transmission exempte de tout virus, l'expéditeur ne donne aucune garantie à cet égard et sa responsabilité ne saurait être recherchée pour tout dommage résultant d'un virus transmis. This e-mail and the documents attached are confidential and intended solely for the addressee; it may also be privileged. If you receive this e-mail in error, please notify the sender immediately and destroy it. As its integrity cannot be secured on the Internet, the Atos Origin group liability cannot be triggered for the message content. Although the sender endeavours to maintain a computer virus-free network, the sender does not warrant that this transmission is virus-free and will not be liable for any damages resulting from any virus transmitted. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: competition
On 09/22/2010 10:52 AM, André Schild wrote: >Am 22.09.2010 16:17, schrieb Lucas Zinato Carraro: >>For me it would be very interesting a option to save cyrus tables >> in a traditional database. ( mysql, postgresql, etc... ) > Beside "interesting" what would you get for a real benefit from this ? > They are ver verly likely to be slower. We wanted to use it for the user_deny database so we could insert a row into one database table that every host has access to. This way we didn't need to come up with a way to update the local user_deny across each frontend server. Also, we were interested in testing SQLite for things like tls_sessions.db and deliver.db because we were tired of BDB breaking our service. Thanks, Dave -- Dave McMurtrie, SPE Email Systems Team Leader Carnegie Mellon University, Computing Services Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: competition
Am 22.09.2010 16:17, schrieb Lucas Zinato Carraro: > For me it would be very interesting a option to save cyrus tables >in a traditional database. ( mysql, postgresql, etc... ) Beside "interesting" what would you get for a real benefit from this ? They are ver verly likely to be slower. André Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: competition
On 09/22/2010 10:17 AM, Lucas Zinato Carraro wrote: > For me it would be very interesting a option to save cyrus tables >in a traditional database. ( mysql, postgresql, etc... ) In 2.3.13 (I think) and newer, there is the option of using an SQL backend. It hasn't been widely used and tested yet, though. Along the lines of Simon's question, other than the fact that some of the Cyrus database defaults happen to be BDB, is anyone using BDB with Cyrus because they really want to? Considering the state of Cyrus' interoperability with BDB and all the recent fixes to skiplist, would it make sense to at least not make BDB a default backend from now on? Thanks, Dave -- Dave McMurtrie, SPE Email Systems Team Leader Carnegie Mellon University, Computing Services Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: competition
For me it would be very interesting a option to save cyrus tables in a traditional database. ( mysql, postgresql, etc... ) Zinato > > On Wed, Sep 22, 2010 at 11:06 AM, Simon Matter wrote: >>> On Wednesday 22 September 2010 15:29:20 Kenneth Marshall wrote: On Wed, Sep 22, 2010 at 10:48:49PM +0930, Daniel O'Connor wrote: > On 22/09/2010, at 22:33, Kenneth Marshall wrote: > >> On a bad shutdown it requires admin intervention very frequently which > >> is pretty tedious. > >> > >> And yes, upgrading it is also a PITA. > > > > That is why we moved to skiplist. The server would require manual > > intervention to even restart after certain types of system events. > > It is hard to explain why mail was unavailable for 30+mins. when > > the server rebooted in 2mins. BDB would need to be much, much > > easier to manage and be more robust for us to use. > > Yep, I plan to migrate to skiplist some day :) > > Now if only I could use skiplist with openldap ;) Sadly, OpenLDAP does need the performance of BDB. >>> >>> With current "LDIF" exports of OpenLDAP, the downtime caused by a corrupt >>> BDB >>> database is "acceptable". >>> >>> Is a similar method possible with cyrus? >>> Eg. a simple command that exports all BDB-database-files to a simple >>> format >>> that can easily be re-imported into Cyrus when the DBD is corrupted? >> >> That's exactly what our RPM does. It converts all BDB databases to >> skiplist on shutdown. On startup, it detects the format of all database >> files and converts it back to what is configured if needed. >> >> As Bron told us, with 2.4 the magic on startup is done inside Cyrus. My >> question remains, why can't we get rid of BDB completely? >> >> Simon >> >> >> Cyrus Home Page: http://www.cyrusimap.org/ >> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ >> > Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: competition
> On Wednesday 22 September 2010 15:29:20 Kenneth Marshall wrote: >> On Wed, Sep 22, 2010 at 10:48:49PM +0930, Daniel O'Connor wrote: >> > On 22/09/2010, at 22:33, Kenneth Marshall wrote: >> > >> On a bad shutdown it requires admin intervention very frequently >> which >> > >> is pretty tedious. >> > >> >> > >> And yes, upgrading it is also a PITA. >> > > >> > > That is why we moved to skiplist. The server would require manual >> > > intervention to even restart after certain types of system events. >> > > It is hard to explain why mail was unavailable for 30+mins. when >> > > the server rebooted in 2mins. BDB would need to be much, much >> > > easier to manage and be more robust for us to use. >> > >> > Yep, I plan to migrate to skiplist some day :) >> > >> > Now if only I could use skiplist with openldap ;) >> >> Sadly, OpenLDAP does need the performance of BDB. > > With current "LDIF" exports of OpenLDAP, the downtime caused by a corrupt > BDB > database is "acceptable". > > Is a similar method possible with cyrus? > Eg. a simple command that exports all BDB-database-files to a simple > format > that can easily be re-imported into Cyrus when the DBD is corrupted? That's exactly what our RPM does. It converts all BDB databases to skiplist on shutdown. On startup, it detects the format of all database files and converts it back to what is configured if needed. As Bron told us, with 2.4 the magic on startup is done inside Cyrus. My question remains, why can't we get rid of BDB completely? Simon Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: competition
On Wednesday 22 September 2010 15:29:20 Kenneth Marshall wrote: > On Wed, Sep 22, 2010 at 10:48:49PM +0930, Daniel O'Connor wrote: > > On 22/09/2010, at 22:33, Kenneth Marshall wrote: > > >> On a bad shutdown it requires admin intervention very frequently which > > >> is pretty tedious. > > >> > > >> And yes, upgrading it is also a PITA. > > > > > > That is why we moved to skiplist. The server would require manual > > > intervention to even restart after certain types of system events. > > > It is hard to explain why mail was unavailable for 30+mins. when > > > the server rebooted in 2mins. BDB would need to be much, much > > > easier to manage and be more robust for us to use. > > > > Yep, I plan to migrate to skiplist some day :) > > > > Now if only I could use skiplist with openldap ;) > > Sadly, OpenLDAP does need the performance of BDB. With current "LDIF" exports of OpenLDAP, the downtime caused by a corrupt BDB database is "acceptable". Is a similar method possible with cyrus? Eg. a simple command that exports all BDB-database-files to a simple format that can easily be re-imported into Cyrus when the DBD is corrupted? -- Joost Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: Berkeley DB: private copy?
Just so you know, the policies of most Linux distributions forbid this kind of bundling, so unless it's easily disabled (and preferably defaults to using system libraries) the distributions will just have to patch it back out again. For example, here's the Fedora policy: http://fedoraproject.org/wiki/Packaging:No_Bundled_Libraries - J< Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: competition
On Wed, 2010-09-22 at 15:04 +0200, Simon Matter wrote: > > On Wed, 2010-09-22 at 14:44 +0200, J. Roeleveld wrote: > >> On Wednesday 22 September 2010 13:47:26 Jeffrey T Eaton wrote: > >> > >> I am probably missing some info here, but > >> > And, as Bron has said, there's something wrong with the way Cyrus uses > >> BDB. > >> > I've never been able to understand BDB well enough to figure it out > >> > myself, nor have I ever found anyone who can help. For what its > >> worth, I > >> > solved the problem by not using BDB at all on the Cyrus systems I > >> used. > >> If it is possible to not use BDB, and BDB causes problems with upgrades, > >> why > >> is BDB still used then? > > BDB is wicked fast and scales well. At least that is the typical > > argument in defense of BDB. And given the stellar performance one sees > > from OpenLDAP I'm prone to believing it. > Fine, that may all be true. But I never ever heard someone going from BDB > to skiplist only and coming back because he had any performance issues. True; and we moved to Skiplist, and no BDB, a long time ago. I was just answering the question "why is BDB still used then?". It may be some degree faster, but IMO isn't worth the agony of figuring out BDB issues. > Maybe it doesn't matter so much with Cyrus and we should try to find out > what is "Poetry and Truth"? If BDB is not really required for Cyrus it > could be made optional for those who really want it. > BTW, it's even worse than Bron said. You not only get errors in the logs > even without using any BDB, you can also end up with a broken BDB > environment which prevents Cyrus from starting up :) Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: competition
On Wed, Sep 22, 2010 at 10:48:49PM +0930, Daniel O'Connor wrote: > > On 22/09/2010, at 22:33, Kenneth Marshall wrote: > >> On a bad shutdown it requires admin intervention very frequently which is > >> pretty tedious. > >> > >> And yes, upgrading it is also a PITA. > >> > > > > That is why we moved to skiplist. The server would require manual > > intervention to even restart after certain types of system events. > > It is hard to explain why mail was unavailable for 30+mins. when > > the server rebooted in 2mins. BDB would need to be much, much > > easier to manage and be more robust for us to use. > > Yep, I plan to migrate to skiplist some day :) > > Now if only I could use skiplist with openldap ;) > Sadly, OpenLDAP does need the performance of BDB. Cheers, Ken Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: competition
On 22/09/2010, at 22:33, Kenneth Marshall wrote: >> On a bad shutdown it requires admin intervention very frequently which is >> pretty tedious. >> >> And yes, upgrading it is also a PITA. >> > > That is why we moved to skiplist. The server would require manual > intervention to even restart after certain types of system events. > It is hard to explain why mail was unavailable for 30+mins. when > the server rebooted in 2mins. BDB would need to be much, much > easier to manage and be more robust for us to use. Yep, I plan to migrate to skiplist some day :) Now if only I could use skiplist with openldap ;) -- Daniel O'Connor software and network engineer for Genesis Software - http://www.gsoft.com.au "The nice thing about standards is that there are so many of them to choose from." -- Andrew Tanenbaum GPG Fingerprint - 5596 B766 97C0 0E94 4347 295E E593 DC20 7B3F CE8C Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: competition
On Wed, Sep 22, 2010 at 10:27:11PM +0930, Daniel O'Connor wrote: > > On 22/09/2010, at 22:17, Adam Tauno Williams wrote: > > > On Wed, 2010-09-22 at 14:44 +0200, J. Roeleveld wrote: > >> On Wednesday 22 September 2010 13:47:26 Jeffrey T Eaton wrote: > >> > >> I am probably missing some info here, but > >>> And, as Bron has said, there's something wrong with the way Cyrus uses > >>> BDB. > >>> I've never been able to understand BDB well enough to figure it out > >>> myself, nor have I ever found anyone who can help. For what its worth, I > >>> solved the problem by not using BDB at all on the Cyrus systems I used. > >> If it is possible to not use BDB, and BDB causes problems with upgrades, > >> why > >> is BDB still used then? > > > > BDB is wicked fast and scales well. At least that is the typical > > argument in defense of BDB. And given the stellar performance one sees > > from OpenLDAP I'm prone to believing it. > > I'm not sure it's Cyrus, I see issues with OpenLDAP and BDB too. > > On a bad shutdown it requires admin intervention very frequently which is > pretty tedious. > > And yes, upgrading it is also a PITA. > That is why we moved to skiplist. The server would require manual intervention to even restart after certain types of system events. It is hard to explain why mail was unavailable for 30+mins. when the server rebooted in 2mins. BDB would need to be much, much easier to manage and be more robust for us to use. Cheers, Ken Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: competition
> On Wed, 2010-09-22 at 14:44 +0200, J. Roeleveld wrote: >> On Wednesday 22 September 2010 13:47:26 Jeffrey T Eaton wrote: >> >> I am probably missing some info here, but >> > And, as Bron has said, there's something wrong with the way Cyrus uses >> BDB. >> > I've never been able to understand BDB well enough to figure it out >> > myself, nor have I ever found anyone who can help. For what its >> worth, I >> > solved the problem by not using BDB at all on the Cyrus systems I >> used. >> If it is possible to not use BDB, and BDB causes problems with upgrades, >> why >> is BDB still used then? > > BDB is wicked fast and scales well. At least that is the typical > argument in defense of BDB. And given the stellar performance one sees > from OpenLDAP I'm prone to believing it. Fine, that may all be true. But I never ever heard someone going from BDB to skiplist only and coming back because he had any performance issues. Maybe it doesn't matter so much with Cyrus and we should try to find out what is "Poetry and Truth"? If BDB is not really required for Cyrus it could be made optional for those who really want it. BTW, it's even worse than Bron said. You not only get errors in the logs even without using any BDB, you can also end up with a broken BDB environment which prevents Cyrus from starting up :) Simon Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: competition
On 22/09/2010, at 22:17, Adam Tauno Williams wrote: > On Wed, 2010-09-22 at 14:44 +0200, J. Roeleveld wrote: >> On Wednesday 22 September 2010 13:47:26 Jeffrey T Eaton wrote: >> >> I am probably missing some info here, but >>> And, as Bron has said, there's something wrong with the way Cyrus uses BDB. >>> I've never been able to understand BDB well enough to figure it out >>> myself, nor have I ever found anyone who can help. For what its worth, I >>> solved the problem by not using BDB at all on the Cyrus systems I used. >> If it is possible to not use BDB, and BDB causes problems with upgrades, why >> is BDB still used then? > > BDB is wicked fast and scales well. At least that is the typical > argument in defense of BDB. And given the stellar performance one sees > from OpenLDAP I'm prone to believing it. I'm not sure it's Cyrus, I see issues with OpenLDAP and BDB too. On a bad shutdown it requires admin intervention very frequently which is pretty tedious. And yes, upgrading it is also a PITA. -- Daniel O'Connor software and network engineer for Genesis Software - http://www.gsoft.com.au "The nice thing about standards is that there are so many of them to choose from." -- Andrew Tanenbaum GPG Fingerprint - 5596 B766 97C0 0E94 4347 295E E593 DC20 7B3F CE8C Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: Berkeley DB: private copy?
> On Wed, Sep 22, 2010 at 04:43:23PM +0530, Shuvam Misra wrote: >> I'm open to criticism, but I strongly feel Cyrus should carry its own >> version of some of these critical system libraries, specially those ones >> which have caused so much compatibility grief in its history. I know >> this >> is considered 'bad taste', but the sysadmin who just wants a stable >> Cyrus >> server probably cares less about taste than stability. > > It's quite justifiable in the case of BDB. Unfortunately, not so much in > the case of SASL which is the other usual suspect. SASL tends to be > deeply integrated into the rest of the system. > > The other interesting one is openssl - but I don't think we have many > problems with it. Not having openssl kind of sucks with 2.4 though. > Would suck less if we put our own crc32 and sha1 implementations in > the code. > >> BTW, a basic question: why do we face more problems with Berkeley DB >> than >> with other file formats? > > Environments mainly. It does this funky "environment" thing which makes > it very susceptible to any errors in open/close counts between processes. > It's really not as robust in a multi-processor shared environment as > one could hope. The other big problem is the different on disk format of the different BDB version and that not all newer BDB versions can upgrade any older file. > > And we're probably using it wrong. Except I don't know how, or what > to change. Skiplist is nicely self-contained and relatively bug-free > these days. We use pure skiplist at FastMail, and we still get > scads of DBERROR messages when we start and stop due to BDB environment > counts going negative. We do also skiplist only for years now - and everybody using our RPMs in default configuration does so - and I'm very pleased how it works. That's why I think it should be possible to build Cyrus completely without BDB support. It could be an optional compile time option. Simon Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: competition
On Wed, 2010-09-22 at 14:44 +0200, J. Roeleveld wrote: > On Wednesday 22 September 2010 13:47:26 Jeffrey T Eaton wrote: > > I am probably missing some info here, but > > And, as Bron has said, there's something wrong with the way Cyrus uses BDB. > > I've never been able to understand BDB well enough to figure it out > > myself, nor have I ever found anyone who can help. For what its worth, I > > solved the problem by not using BDB at all on the Cyrus systems I used. > If it is possible to not use BDB, and BDB causes problems with upgrades, why > is BDB still used then? BDB is wicked fast and scales well. At least that is the typical argument in defense of BDB. And given the stellar performance one sees from OpenLDAP I'm prone to believing it. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: competition
On Wednesday 22 September 2010 13:47:26 Jeffrey T Eaton wrote: I am probably missing some info here, but > And, as Bron has said, there's something wrong with the way Cyrus uses BDB. > I've never been able to understand BDB well enough to figure it out > myself, nor have I ever found anyone who can help. For what its worth, I > solved the problem by not using BDB at all on the Cyrus systems I used. If it is possible to not use BDB, and BDB causes problems with upgrades, why is BDB still used then? -- Joost Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: competition
> > On Sep 22, 2010, at 7:57 AM, Sebastian Hagedorn wrote: > >> --On 22. September 2010 07:47:26 -0400 Jeffrey T Eaton >> wrote: >> >>> All of that said, I believe that, in general, you can safely upgrade >>> BDB. >>> If you have a Cyrus installation using BDB X, you can drop in a new >>> Cyrus >>> using BDB Y, as long as everything is shut down in between. You can't >>> go >>> back without effort, but upgrades should work. >> >> I'm pretty sure that's not true. *If* an application uses BDB's >> environment features (not all do!), you *cannot* simply upgrade, unless >> the application is programmed to perform the upgrade automatically. >> Otherwise you have to either use db_upgrade to convert the environment >> to the current version, or you use db_dump and db_load. There's a >> db_upgrade() call in the library that could be used, but it might not >> work in all cases: >> >> Errors >> The DB->upgrade() method may fail and return one of the following >> non-zero errors: >> >> DB_OLD_VERSION - The database cannot be upgraded by this version of the >> Berkeley DB software. >> -- >>.:.Sebastian Hagedorn - RZKR-R1 (Gebäude 52), Zimmer 18.:. >>.:.Regionales Rechenzentrum (RRZK).:. >> .:.Universität zu Köln / Cologne University - â +49-221-478-5587.:. > > Hmm. I thought that I had updated BDB in the past. I probably just nuked > the databases between updates, since we only used them for transient data > (tls_sessions.db, deliver.db, ptscache,db). You can upgrade between some versions but not all. > > The rest of my point stands; changing the BDB version is a problem in and > of itself, not changing the Cyrus version. I know of no reason why one > couldn't update a Cyrus 2.2 installation to 2.3. Well, I never said it's a problem of Cyrus, it's a problem of BDB. Just ask the OpenLDAP folks - and those doing distribution packages for it. The only way I see for Cyrus is to stay away from it. Simon Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: competition
On Sep 22, 2010, at 7:57 AM, Sebastian Hagedorn wrote: > --On 22. September 2010 07:47:26 -0400 Jeffrey T Eaton wrote: > >> All of that said, I believe that, in general, you can safely upgrade BDB. >> If you have a Cyrus installation using BDB X, you can drop in a new Cyrus >> using BDB Y, as long as everything is shut down in between. You can't go >> back without effort, but upgrades should work. > > I'm pretty sure that's not true. *If* an application uses BDB's environment > features (not all do!), you *cannot* simply upgrade, unless the application > is programmed to perform the upgrade automatically. Otherwise you have to > either use db_upgrade to convert the environment to the current version, or > you use db_dump and db_load. There's a db_upgrade() call in the library that > could be used, but it might not work in all cases: > > Errors > The DB->upgrade() method may fail and return one of the following non-zero > errors: > > DB_OLD_VERSION - The database cannot be upgraded by this version of the > Berkeley DB software. > -- >.:.Sebastian Hagedorn - RZKR-R1 (Gebäude 52), Zimmer 18.:. >.:.Regionales Rechenzentrum (RRZK).:. > .:.Universität zu Köln / Cologne University - ✆ +49-221-478-5587.:. Hmm. I thought that I had updated BDB in the past. I probably just nuked the databases between updates, since we only used them for transient data (tls_sessions.db, deliver.db, ptscache,db). The rest of my point stands; changing the BDB version is a problem in and of itself, not changing the Cyrus version. I know of no reason why one couldn't update a Cyrus 2.2 installation to 2.3. -jeaton Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: Berkeley DB: private copy?
On Wed, Sep 22, 2010 at 04:43:23PM +0530, Shuvam Misra wrote: > I'm open to criticism, but I strongly feel Cyrus should carry its own > version of some of these critical system libraries, specially those ones > which have caused so much compatibility grief in its history. I know this > is considered 'bad taste', but the sysadmin who just wants a stable Cyrus > server probably cares less about taste than stability. It's quite justifiable in the case of BDB. Unfortunately, not so much in the case of SASL which is the other usual suspect. SASL tends to be deeply integrated into the rest of the system. The other interesting one is openssl - but I don't think we have many problems with it. Not having openssl kind of sucks with 2.4 though. Would suck less if we put our own crc32 and sha1 implementations in the code. > BTW, a basic question: why do we face more problems with Berkeley DB than > with other file formats? Environments mainly. It does this funky "environment" thing which makes it very susceptible to any errors in open/close counts between processes. It's really not as robust in a multi-processor shared environment as one could hope. And we're probably using it wrong. Except I don't know how, or what to change. Skiplist is nicely self-contained and relatively bug-free these days. We use pure skiplist at FastMail, and we still get scads of DBERROR messages when we start and stop due to BDB environment counts going negative. Bron. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: competition
--On 22. September 2010 07:47:26 -0400 Jeffrey T Eaton wrote: All of that said, I believe that, in general, you can safely upgrade BDB. If you have a Cyrus installation using BDB X, you can drop in a new Cyrus using BDB Y, as long as everything is shut down in between. You can't go back without effort, but upgrades should work. I'm pretty sure that's not true. *If* an application uses BDB's environment features (not all do!), you *cannot* simply upgrade, unless the application is programmed to perform the upgrade automatically. Otherwise you have to either use db_upgrade to convert the environment to the current version, or you use db_dump and db_load. There's a db_upgrade() call in the library that could be used, but it might not work in all cases: Errors The DB->upgrade() method may fail and return one of the following non-zero errors: DB_OLD_VERSION - The database cannot be upgraded by this version of the Berkeley DB software. -- .:.Sebastian Hagedorn - RZKR-R1 (Gebäude 52), Zimmer 18.:. .:.Regionales Rechenzentrum (RRZK).:. .:.Universität zu Köln / Cologne University - ✆ +49-221-478-5587.:. p7s98D0p7NYL6.p7s Description: S/MIME cryptographic signature Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: competition
On Sep 22, 2010, at 1:50 AM, Simon Matter wrote: >> >>> Debian is still stuck on 2.2 and there seems to be no progress in that >>> area. >>> >>> The main problem they apparently have, is the migration path for the >>> various >>> DB files from 2.2 to 2.3. >>> (The 2.3 version itself works fine as .deb packages) >> >> What "migration path"? Cyrus 2.3 supports all of the same database >> backends that Cyrus 2.2 did. > > I don't know the Debian packages well but from a quick look I did long > time ago it seems like those packages leave quite some work to do by the > user of the packages. That also means there are more ways for a user to > break things which may stop them making the 2.3 packages go in easily. > >> >> To the best of my knowledge, you can drop in Cyrus 2.3 binaries, and >> with the same config files as 2.2 used, everything will just work. >> You can't easily go back, because I believe that 2.3 will update >> cyrus.index files to a format which 2.2 doesn't recognize, but that >> shouldn't prevent anyone from upgrading. > > IIRC it's a little bit more complicated. Beside BDB, there are also some > steps to do at upgrade, like compiling sieve scripts, possibly converting > its encoding. To the best of my knowledge, the complied Sieve format has not changed since 2.2. > With BDB backends you have to make sure everything fits > correctly. Binaries need to be linked against the correct version of BDB > and also the on disk BDB files need to be of the same version. Now, think > you want to do a distribution upgrade which comes with the latest greatest > BDB release and new Cyrus binaries which are using them, but your spool is > still on the old release... We all know how this ends. This would be a problem anytime you upgrade BDB, independent of the version of Cyrus. If you are rolling out a new distribution with a new version of BDB, you will need to address this problem even if you don't upgrade Cyrus. (Unless of course, you continue to ship a Cyrus built against the _old_ BDB, in which case the argument is irrelevant). > The only solution - you can call it dirty workaround - I found for our > RedHat EL RPMs was to convert all BDB databases to skiplist on Cyrus > shutdown and convert them back on startup. That way the spool is always in > a state which can be migrated once Cyrus is shutdown. > Of course getting rid of any BDB in the default configuration of the RPM > binaries has helped much to prevent BDB mess. All of that said, I believe that, in general, you can safely upgrade BDB. If you have a Cyrus installation using BDB X, you can drop in a new Cyrus using BDB Y, as long as everything is shut down in between. You can't go back without effort, but upgrades should work. And, as Bron has said, there's something wrong with the way Cyrus uses BDB. I've never been able to understand BDB well enough to figure it out myself, nor have I ever found anyone who can help. For what its worth, I solved the problem by not using BDB at all on the Cyrus systems I used. -jeaton Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Berkeley DB: private copy?
> given the issues with BDB. Is it worth embedding a copy of > BDB into the Cyrus distribution rather than using the OS one? I > know it's generally considered bad taste, but it sure makes > keeping in sync easier! IMHO, yes, most certainly. Cyrus is a large and complex system, and its maintenance would automatically become much simpler if it reduced its dependencies on other OS-specific libraries, but carried its own. OpenOffice (on my Ubuntu 9.04 laptop) has 265 .so files in its /usr/lib/openoffice/basis3.1/program directory. How many of these are exclusive to OpenOffice and how many are private versions of non-OOo libraries? (I tried verifying, but could not arrive at a clear picture.) Firefox and Thunderbird both use SQLite, AFAIK. Does anyone care whether the version of SQLite used by my Ubuntu-bundled Firefox is exactly identical to the latest version of SQLite bundled by the Ubuntu distro? I'm open to criticism, but I strongly feel Cyrus should carry its own version of some of these critical system libraries, specially those ones which have caused so much compatibility grief in its history. I know this is considered 'bad taste', but the sysadmin who just wants a stable Cyrus server probably cares less about taste than stability. BTW, a basic question: why do we face more problems with Berkeley DB than with other file formats? Shuvam Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: competition
On Wed, Sep 22, 2010 at 10:13:20AM +0200, J. Roeleveld wrote: > On Wednesday 22 September 2010 09:01:33 Bron Gondwana wrote: > > On Wed, Sep 22, 2010 at 08:02:35AM +0200, Simon Matter wrote: > > > > > Bron ( really trying to make Cyrus newbie-friendly as well as advanced-site > >friendly. I also want auto-recompilation of sieve scripts, and > >auto-fixing of broken .cache files ) > > Now that you mention it, what would be needed to do to "auto-recompile" sieve- > scripts? > If that would work, I could edit the file directly in the respective > location, > rather then having to play with the sieve management tool or a MUA with sieve- > support. :) Hmm... I take it all back. We did this a while back, and it's the path to great sadness. Directly editing the file in place means we can't auto-replicate it, can't add CRC checks on the contents, etc. The files on disk are internal format files, not to be fiddled with directly. Use an API. Now the lack of a good command to copy a file into place, enable it, etc. I'm sold on that. A command line tool to install a new sieve file for a user would be handy. But no direct editing of files on disk please. Bron. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: Repos [Was: competition]
Adam Tauno Williams wrote: > On Tue, 2010-09-21 at 12:25 +0200, Syren Baran wrote: > > Am Dienstag, den 21.09.2010, 11:48 +0200 schrieb André Schild: > > > Am 21.09.2010 11:35, schrieb Simon Matter: > > > >> I don't know, where this bad karma is coming from - I'm still happy > > > >> with > > > > > > > > I guess it's simply because for many years there were no clean > > > > packages for the most used operating systems. > > > > > > Debian is still stuck on 2.2 and there seems to be no progress in that > > > area. > > > > Most people like to use versions from repositories. For obvious reasons. > > Might make sense for cyrus to host their own repository. If it's just a > > simple entry in sources.list(.d) more people would use the recent > > version. > > The OBS supports just about every distro there is; why not host there? It'll build *something* for all distributions, but it does definitely not *support* all distributions. In fact, as a premier packager for Fedora + derivatives including but not limited to Red Hat Enterprise Linux and CentOS, and add-on repositories such as Extra Packages for Enterprise Linux as well as RPM Fusion, I can tell you I would hate to have to work with and around OBS; The OBS builds based on what SUSE thinks is an upstream version of RPM, while it's not. Undoubtedly, the build tools they use for APT-based distributions are much closer to what is in the actual distributions. Either way, in relation to my domain, RPM-based systems, as such, it is simply fundamentally flawed. The results stretch from allowed packaging foo that would never be accepted by any actual distribution, to retraceable steps during builds resulting in non-reproducible build products nonetheless. If I were to draw an analogy it would have to be; fixing your car with only a hammer If you want to build native packages for a distribution, I very, very strongly recommend to use the methods of such distribution (whether in your own repositories or not), stick to their specific packaging guidelines, and use their packaging efforts as well as your own to move both forward. For APT- based systems, this means git/svn/cvs/hg-buildpackage w/ pbuilder/cowbuilder (if not distributed), for RPM-based distributions this means dist-git and Koji. Kind regards, -- Jeroen van Meeuwen Senior Engineer, Kolab Systems AG e: vanmeeu...@kolabsys.com t: +316 42 801 403 w: http://www.kolabsys.com pgp: 9342 BF08 Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: competition
On Wednesday 22 September 2010 09:01:33 Bron Gondwana wrote: > On Wed, Sep 22, 2010 at 08:02:35AM +0200, Simon Matter wrote: > Bron ( really trying to make Cyrus newbie-friendly as well as advanced-site >friendly. I also want auto-recompilation of sieve scripts, and >auto-fixing of broken .cache files ) Now that you mention it, what would be needed to do to "auto-recompile" sieve- scripts? If that would work, I could edit the file directly in the respective location, rather then having to play with the sieve management tool or a MUA with sieve- support. :) -- Joost Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: competition
On Wed, 22 Sep 2010 09:12 +0200, "Sebastian Hagedorn" wrote: > --On 22. September 2010 16:10:15 +1000 Bron Gondwana > wrote: > > > Now - BDB database SHOULD be upgradable. I want to find a BDB expert > > to help me with that - (a) detecting that an upgrade is necessary, and > > (b) doing the upgrade. > > I wouldn't exactly call myself an expert, but I think I understand the > basics of BDB operation. The biggest problem is that there may be > multiple > version of the BDB utilities installed. So you need to know which BDB > version Cyrus was linked against and somehow figure out what the > corresponding tools are on the box you're on. The tool are named db_stat, > db_verify, db_upgrade etc., but it's not unusual to have db41_stat, > db42_stat etc . > The "file" utility is pretty useful, because it tells you the version of > a > BDB file. Unfortunately that version has nothing to to with the version > number of the BDB libraries themselves. Instead they look like this: > > Berkeley DB (Hash, version 8, native byte-order) Yeah, I can get that just by reading the value as byte 12 in the file. I pulled a copy of the BDB source and grabbed their magic numbers already. The thing is - I don't want to be calling out to the tools - I want to be doing it in_process with a Cyrus tool that's already compiled against the correct version of BDB. > I suppose one could do this: > - at startup, create a new BDB hash file using a Cyrus binary > - check its version number using "file" > - compare that to the existing BDB files > - if they're different you need to perform an upgrade Oooh. Actually, that's a point. Create a temporary file of each type and pull the version field from it, then check each BDB file's version field. Hmm. That's quite viable actually. I like it :) > The upgrade itself *should* be doable using db_upgrade (using the version > that goes with the libraries Cyrus is linked against), although I think > there are cases where you have to use db_dump followed by db_load. The > problem with the latter is that you need the dump from the *old* version > of > the libraries! http://download.oracle.com/docs/cd/E17076_02/html/upgrading/index.html seems to have good information on "what's changed" in each version. On the topic of upgrades: http://download.oracle.com/docs/cd/E17076_02/html/programmer_reference/am_upgrade.html "We do guarantee that any archived database may be upgraded using a current Berkeley DB software release and the DB->upgrade() method, and there is no need to step-wise upgrade the database using intermediate releases of Berkeley DB. Sites should consider archiving appropriate copies of their application or application sources if they may need to access archived databases without first upgrading them." > > given the issues with BDB. Is it worth embedding a copy of > > BDB into the Cyrus distribution rather than using the OS one? I > > know it's generally considered bad taste, but it sure makes > > keeping in sync easier! > > That would certainly help with the issues I described above, but only if > you either also include the utilities or have your own binaries that > fulfill their function. No need I don't think if we do the little trick above to figure out BDB version and upgrade. Ideally, I'd like to not be using BDB because the incompatibilities are a killer. If we could just say "use a backwards compatible format" to BDB and have it readable by any 3.x + version, I'd be a very happy camper. Bron. -- Bron Gondwana br...@fastmail.fm Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: competition
--On 22. September 2010 16:10:15 +1000 Bron Gondwana wrote: Now - BDB database SHOULD be upgradable. I want to find a BDB expert to help me with that - (a) detecting that an upgrade is necessary, and (b) doing the upgrade. I wouldn't exactly call myself an expert, but I think I understand the basics of BDB operation. The biggest problem is that there may be multiple version of the BDB utilities installed. So you need to know which BDB version Cyrus was linked against and somehow figure out what the corresponding tools are on the box you're on. The tool are named db_stat, db_verify, db_upgrade etc., but it's not unusual to have db41_stat, db42_stat etc . The "file" utility is pretty useful, because it tells you the version of a BDB file. Unfortunately that version has nothing to to with the version number of the BDB libraries themselves. Instead they look like this: Berkeley DB (Hash, version 8, native byte-order) I suppose one could do this: - at startup, create a new BDB hash file using a Cyrus binary - check its version number using "file" - compare that to the existing BDB files - if they're different you need to perform an upgrade The upgrade itself *should* be doable using db_upgrade (using the version that goes with the libraries Cyrus is linked against), although I think there are cases where you have to use db_dump followed by db_load. The problem with the latter is that you need the dump from the *old* version of the libraries! given the issues with BDB. Is it worth embedding a copy of BDB into the Cyrus distribution rather than using the OS one? I know it's generally considered bad taste, but it sure makes keeping in sync easier! That would certainly help with the issues I described above, but only if you either also include the utilities or have your own binaries that fulfill their function. -- .:.Sebastian Hagedorn - RZKR-R1 (Gebäude 52), Zimmer 18.:. .:.Regionales Rechenzentrum (RRZK).:. .:.Universität zu Köln / Cologne University - ✆ +49-221-478-5587.:. p7sOFjStOGfiA.p7s Description: S/MIME cryptographic signature Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: competition
On Wed, Sep 22, 2010 at 08:02:35AM +0200, Simon Matter wrote: > Documentation is one thing, and dependencies like BDB another. But there > is something else I guess, for servers which are not dedicated mail > server, it would be really nice if one could install Cyrus and it just > works for every user having an account on the system. That can be done > with the autocreate patches but they are also not included in Cyrus. In > fact, Cyrus could have two main modes, one for dedicated, sealed mail > servers and one for "allround" servers with user accounts. We're planning to include the autocreate patches in 2.4 as soon as we make sure they work nicely with virtdomains and murder. Making it work nicely with user accounts would be very handy. Bron ( really trying to make Cyrus newbie-friendly as well as advanced-site friendly. I also want auto-recompilation of sieve scripts, and auto-fixing of broken .cache files ) Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: competition
Am 21.09.2010 23:15, schrieb Jeffrey T Eaton: >> Debian is still stuck on 2.2 and there seems to be no progress in that area. >> >> The main problem they apparently have, is the migration path for the various >> DB files from 2.2 to 2.3. >> (The 2.3 version itself works fine as .deb packages) > What "migration path"? Cyrus 2.3 supports all of the same database > backends that Cyrus 2.2 did. > > To the best of my knowledge, you can drop in Cyrus 2.3 binaries, and > with the same config files as 2.2 used, everything will just work. > You can't easily go back, because I believe that 2.3 will update > cyrus.index files to a format which 2.2 doesn't recognize, but that > shouldn't prevent anyone from upgrading. From a cyrus point of view this might be true, but in real upgrades most of the time also the DB version changes and here is one of the pitfalls (Not only for cyrus) Here the current statement from the debian guys. http://lists.alioth.debian.org/pipermail/pkg-cyrus-imapd-debian-devel/2010-August/003427.html André Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
