Re: competition

[Jeroen van Meeuwen (Kolab Systems)]

Adam Tauno Williams wrote:
> On Wed, 2010-09-22 at 18:18 +0200, Jeroen van Meeuwen (Kolab Systems)
> wrote:
> >  The scenario is integration, not extension of Cyrus -which in and of
> >  itself works
> > 
> > perfecly fine and reliable for us. We're not seeking to improve Cyrus'
> > performance with *SQL db backends.
> 
> But, this assumes the data that Cyrus stores in that DB would be useful
> outside the context of the Cyrus' processes.  I've lightly played with
> the idea, and not gone any further - the data available isn't really
> very useful.
> 

Well, for one, our ActiveSync implementation wants the following information;

- List of (subscribed) IMAP folders,
- Annotations, per IMAP folder,
- Current status of the contents of such IMAP folder, such as new messages or 
deleted messages, in comparison to what the client currently holds,
- Message contents.

While connecting through the IMAP server and have Cyrus hand over the answers, 
and correlate such information on the side of the 3rd party application is 
perfectly feasible, I think it may be more efficient to correlate the 
requested information from a database directly, as opposed to attempting to 
cache the results handed over by Cyrus by each 3rd party application.

> > Imagine the following scenario; a client polls 3rd party application for
> > a list of mailboxes and the content's status very regularly -basically
> > what it's interested in knowing is whether anything changed.
> 
> Doesn't condstore solve this issue inexpensively?  [maybe I
> misunderstand condstore].  I thought it was equivalent to WebDAV/CalDAV
> ctags (which are mightily nice).
> 

I'm not sure whether the IMAP server's capabilities with regards to 
modification sequences has anything to do with this thread, but maybe I'm 
misunderstanding the IMAP CONDSTORE extension.

> > Each 3rd party app will seek to cache the
> > results somehow, for improved performance interacting with its clients
> > and as to not continuously put load on the Cyrus server.
> 
> Which is what WebDAV/CalDAV ctags are for.
> 

The WebDAV/CalDAV scenario doesn't really fly with mailboxes. For one, 
mailboxes tend to have plenty more folders and plenty more messages.

The question is not how the 3rd party app *can* get the needed information, 
the question is how many 3rd party apps can be integrated *most efficiently* 
(both in terms of performance/lower overhead, as well as architecture and 3rd 
party app's design -DYI cache for each 3rd party app?).

Kind regards,

-- 
Jeroen van Meeuwen
Senior Engineer, Kolab Systems AG

e: vanmeeu...@kolabsys.com
t: +316 42 801 403
w: http://www.kolabsys.com

pgp: 9342 BF08

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/

Re: competition

[Simon Matter]

> On Mon, Sep 27, 2010 at 09:38:08AM -0400, Jeff Eaton wrote:
>>
>> > Better to just use an internal DB codebase (like skiplists) that has
>> > nothing to do with Sleepycat.  But then someone has to write and
>> maintain
>> > this code.
>> >
>> > I think the best compromise I've heard yet is to use something like
>> > skiplists by default and make the use of libdb an optional feature
>> like the
>> > use of mysql.
>>
>> I'd be in favor of "deprecating" the use of BerkeleyDB in Cyrus.  Making
>> at least the default for new installations to be all-skiplist would be a
>> good step in that direction, as well as encouraging people to move away
>> from BDB where possible.
>
> Just committed this change to dev/2.4.0-alpha1.  Along with the code
> that automagically converts databases on ctl_cyrusdb -r, this should
> make most sites that are running the default layout transparently
> start running all-skiplist without even realising.
>
>> Ideally, I'd suggest making the default behavior of "./configure; make;
>> make install" to _not_ link in BDB at all unless --with-bdb is
>> explicitly
>> given.  As I recall, you have to jump through a few hoops to get a build
>> of Cyrus to not link in the BDB libraries, even if you're not using
>> them.
>
> I haven't tried this.  Maybe I should.

Oh yes please, would be great being able to build completely without BDB.
Even better if it would be possible to still build cvt_cyrusdb with BDB
support so one can convert things. Or, if database backends could be
compiled as loadable plugins, just imagine :)

Simon


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/

Re: competition

[Bron Gondwana]

On Mon, Sep 27, 2010 at 09:38:08AM -0400, Jeff Eaton wrote:
> 
> > Better to just use an internal DB codebase (like skiplists) that has
> > nothing to do with Sleepycat.  But then someone has to write and maintain
> > this code.
> >
> > I think the best compromise I've heard yet is to use something like
> > skiplists by default and make the use of libdb an optional feature like the
> > use of mysql.
> 
> I'd be in favor of "deprecating" the use of BerkeleyDB in Cyrus.  Making
> at least the default for new installations to be all-skiplist would be a
> good step in that direction, as well as encouraging people to move away
> from BDB where possible.

Just committed this change to dev/2.4.0-alpha1.  Along with the code
that automagically converts databases on ctl_cyrusdb -r, this should
make most sites that are running the default layout transparently
start running all-skiplist without even realising.
 
> Ideally, I'd suggest making the default behavior of "./configure; make;
> make install" to _not_ link in BDB at all unless --with-bdb is explicitly
> given.  As I recall, you have to jump through a few hoops to get a build
> of Cyrus to not link in the BDB libraries, even if you're not using them.

I haven't tried this.  Maybe I should.

> All of that said, it's probably worth directing future Cyrus DB backend
> work away from BDB and more to improving SQLite or other integration.

What about Hamster, or any of the other similar things?  What about
memcached for tls, pts and statuscache.  Worth looking at.

By the way - I'm travelling for the next few weeks, including meeting Ken
and Dave in Buffalo mid next month.  We have some new programmers starting
with FastMail/Opera in Melbourne while I'm away, and I'm hoping to dedicate
a large part of one of them's time to improving Cyrus.  My shortlist of
tasks to look at while I'm away is:

1) figure out what's up with BDB
2) seen db cleaner.  Remove stale entries from seen databases (including
   entries for user's own mailboxes now that the owner's seen is absorbed
   back into the cyrus.index)

Apart from these two things, I suspect most of the work we'll do on Cyrus
over the next few months will be quite "tactical" in nature.  Targetted
towards making cross-datacentre replication better, and working towards
supporting "conversations" (cross folder threads) in an efficient way.

And there'll be a fair bit of work on the supporting code that manages
replication from the outside too.  Possibly pulling some back into a C
daemon within Cyrus that can track replication status and monitor how
the replication is running.  Particularly of interest here is safe
failover, and knowing when a replica is "complete" - so if you bring up
a new replica, knowing when all the users have been replicated and
rolling replication is close enough that you can shut down and switch.

Bron.

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/

Re: competition

[Jeff Eaton]

> Better to just use an internal DB codebase (like skiplists) that has
> nothing to do with Sleepycat.  But then someone has to write and maintain
> this code.
>
> I think the best compromise I've heard yet is to use something like
> skiplists by default and make the use of libdb an optional feature like the
> use of mysql.

I'd be in favor of "deprecating" the use of BerkeleyDB in Cyrus.  Making
at least the default for new installations to be all-skiplist would be a
good step in that direction, as well as encouraging people to move away
from BDB where possible.

Ideally, I'd suggest making the default behavior of "./configure; make;
make install" to _not_ link in BDB at all unless --with-bdb is explicitly
given.  As I recall, you have to jump through a few hoops to get a build
of Cyrus to not link in the BDB libraries, even if you're not using them.

All of that said, it's probably worth directing future Cyrus DB backend
work away from BDB and more to improving SQLite or other integration.

-jeaton


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/

Re: competition

[Adam Tauno Williams]

On Wed, 2010-09-22 at 18:18 +0200, Jeroen van Meeuwen (Kolab Systems)
wrote:
> >> Kolab Systems is thinking of such SQL databases for integration purposes,
> > > where the performance penalty now lies within having to use the IMAP
> > > protocol to gain access to the underlying metadata (seen status, message
> > > indexes) in distributed groupware environments where Cyrus itself is not
> > > the only component that needs access to such information (but would still
> > > remain authoritative, of course, and would be the only component with 
> >write
> > > access to most tables).
> >> While not necessarily the best approach, it seems worth exploring.
> >What makes you think the query parsing and other overheads of SQL will 
> >be faster than IMAP?
> >I'd be interested in any numbers that you might have to support the effort.
>  The scenario is integration, not extension of Cyrus -which in and of itself 
> works 
> perfecly fine and reliable for us. We're not seeking to improve Cyrus'
> performance with *SQL db backends.

But, this assumes the data that Cyrus stores in that DB would be useful
outside the context of the Cyrus' processes.  I've lightly played with
the idea, and not gone any further - the data available isn't really
very useful.

> Imagine the following scenario; a client polls 3rd party application for a 
> list of
> mailboxes and the content's status very regularly -basically what it's 
> interested 
> in knowing is whether anything changed.

Doesn't condstore solve this issue inexpensively?  [maybe I
misunderstand condstore].  I thought it was equivalent to WebDAV/CalDAV
ctags (which are mightily nice).

> Each 3rd party app will seek to cache the 
> results somehow, for improved performance interacting with its clients and as 
> to not continuously put load on the Cyrus server.

Which is what WebDAV/CalDAV ctags are for.


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/

Re: competition

[Shuvam Misra]

> > For situations where we need just random access, not sequential, can we
> > use GDBM? Is that library better than Berkeley DB?
>   ^
> 
> G => GNU => GPL.  Licencing issues I suspect.  We're BSD licence,
> not GPL.

Yes, you're quite right, I just checked. Till your comment, I had assumed
that GDBM would "of course, obviously" be offered under the LGPL. But
it seems that the free software activism has brought even these small
libraries under GPL, much like MySQL client libraries.

No further questions. :)

Shuvam

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/

Re: competition

[Jeroen van Meeuwen (Kolab Systems)]

"Andy Bennett"  wrote:

>Hi,
>
>> Kolab Systems is thinking of such SQL databases for integration purposes,
> > where the performance penalty now lies within having to use the IMAP
> > protocol to gain access to the underlying metadata (seen status, message
> > indexes) in distributed groupware environments where Cyrus itself is not
> > the only component that needs access to such information (but would still
> > remain authoritative, of course, and would be the only component with 
>write
> > access to most tables).
>> 
>> While not necessarily the best approach, it seems worth exploring.
>
>What makes you think the query parsing and other overheads of SQL will 
>be faster than IMAP?
>I'd be interested in any numbers that you might have to support the effort.
>

The scenario is integration, not extension of Cyrus -which in and of itself 
works perfecly fine and reliable for us. We're not seeking to improve Cyrus' 
performance with *SQL db backends.

Imagine the following scenario; a client polls 3rd party application for a list 
of mailboxes and the content's status very regularly -basically what it's 
interested in knowing is whether anything changed. Each 3rd party app will seek 
to cache the results somehow, for improved performance interacting with its 
clients and as to not continuously put load on the Cyrus server.

Our idea is to prevent that caching from needing to happen, and needlessly be 
duplicated technology across 3rd party apps, by using what Cyrus would consider 
it's live data in SQL as the 3rd party app's cache.

Now, I don't have any numbers to compare while there is no Cyrus SQL db backend 
for the relevant databases. I'm just thinking that a single database query -if 
it could provide accurate status info- can be more efficient -to the Cyrus 
server(s) itself as well as the 3rd party app- then folderlist, iterate, 
request status info, parse, only to ultimately throw back at the client there's 
no changes -which is the result most of the time. It'd also eliminate 
duplicating attempts to cache folderlist and status results somehow, and would 
ultimately improve the scalability of such 3rd party apps as part of the infra 
they require to be shared..., since its "cache" is in SQL, etc. etc.

>The big downside to using an SQL database is the enormous temptation to 
>point all the Cyrus servers at the same Database server and lose the 
>redundancy and scalability inherent in a multi node or Murder setup.
>

One would set up the database backend server infrastructure just as much 
conform to H/A and L/B requirements as the rest of the Cyrus/groupware 
infrastructure, not unlike how you would set up a sustainable database backend 
server infrastructure in any other type of critical environment.

-- Jeroen

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/

Re: competition

[Bron Gondwana]

On Sat, Sep 25, 2010 at 01:46:10PM +0530, Shuvam Misra wrote:
> For situations where we need just random access, not sequential, can we
> use GDBM? Is that library better than Berkeley DB?
  ^

G => GNU => GPL.  Licencing issues I suspect.  We're BSD licence,
not GPL.

Bron.

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/

Re: competition

[Shuvam Misra]

> On 9/22/2010 10:20 PM, Shuvam Misra wrote:
> > I was a strong advocate of bundling DB libraries, etc, with Cyrus. The
> > points you've made here are very interesting. I didn't know many of these
> > things. I'm re-thinking whether bundling is such a good idea now. Thanks.
> 
> There's a lot to be said for the Mac OS X approach.  However, as someone
> already pointed out, most distributions won't allow this (for many
> reasons), so it's pointless to even talk about going there.
> 
> Better to just use an internal DB codebase (like skiplists) that has
> nothing to do with Sleepycat.  But then someone has to write and
> maintain this code.
> 
> I think the best compromise I've heard yet is to use something like
> skiplists by default and make the use of libdb an optional feature like
> the use of mysql.

For situations where we need just random access, not sequential, can we
use GDBM? Is that library better than Berkeley DB?

Shuvam

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/

Re: competition

[Patrick Goetz]

On 9/22/2010 10:20 PM, Shuvam Misra wrote:
> I was a strong advocate of bundling DB libraries, etc, with Cyrus. The
> points you've made here are very interesting. I didn't know many of these
> things. I'm re-thinking whether bundling is such a good idea now. Thanks.
> 

There's a lot to be said for the Mac OS X approach.  However, as someone
already pointed out, most distributions won't allow this (for many
reasons), so it's pointless to even talk about going there.

Better to just use an internal DB codebase (like skiplists) that has
nothing to do with Sleepycat.  But then someone has to write and
maintain this code.

I think the best compromise I've heard yet is to use something like
skiplists by default and make the use of libdb an optional feature like
the use of mysql.

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/

Re: competition

[Simon Matter]

> On Wed, Sep 22, 2010 at 04:10:15PM +1000, Bron Gondwana wrote:
>
>> Now - BDB database SHOULD be upgradable.  I want to find a BDB expert
>> to help me with that - (a) detecting that an upgrade is necessary, and
>> (b) doing the upgrade.
>
> It was quite some time ago I last upgraded a Cyrus instance, but my
> upgrade script did the following:
>
> - Run the _old_ version of db_recover. Just shutting down Cyrus is not
>   enough, without this step db_upgrade can fail.
>
> - Run the _new_ version of db_upgrade.
>
> - Run "db_checkpoint -1" to force the log to be rotated.
>
> I think it won't be hard to implement steps 2 & 3 inside Cyrus, and I
> don't remember those steps taking a significant amount of time, but I
> don't know how it would be possible to implement step 1. However if the
> BDB version change does not bring a change to the on-disk DB log format,
> then it may be possible to skip steps 1 & 3.

Unfortunately that's not enough in many situations. I did uncountable
tests with different BDB versions and came up with the following method in
our RPMs:

...
# make sure our Berkeley databases are in a sane state
# wait for db_checkpoint to end successfully or kill it after a timeout
db_checkpoint -v -1 -h ${imap_prefix}/db &
DB_CHECK_PID=$!
CNT=0
while [ $CNT -lt 60 ]; do
  if ! kill -0 $DB_CHECK_PID > /dev/null 2>&1; then
break
  fi
  sleep 1
  let CNT+=1
done
if kill -0 $DB_CHECK_PID > /dev/null 2>&1; then
  kill -USR1 $DB_CHECK_PID > /dev/null 2>&1
  sleep 1
  kill -KILL $DB_CHECK_PID > /dev/null 2>&1
  wait $DB_CHECK_PID > /dev/null 2>&1
fi

# do a normal recovery
db_recover -v -h ${imap_prefix}/db
RETVAL=$?
if [ $RETVAL -ne 0 ]; then
  # try a catastrophic recovery instead of normal recovery
  db_recover -v -c -h ${imap_prefix}/db
  RETVAL=$?
  ERRVAL=$(( $ERRVAL + $RETVAL ))
  if [ $RETVAL -ne 0 ]; then
echo "ERROR: catastrophic recovery of Berkeley databases failed"
  fi
fi

if [ "$EXPORT" = "export" ]; then
  # convert all db files to portable format for migration
  # TODO: quota_db, we don't touch it for now
  cvt_file ${imap_prefix}/deliver.db   "skiplist"
  cvt_file ${imap_prefix}/mailboxes.db "skiplist"
  cvt_file ${imap_prefix}/tls_sessions.db  "skiplist"
  cvt_file ${imap_prefix}/annotations.db   "skiplist"
  cvt_file ${imap_prefix}/ptclient/ptscache.db "skiplist"
  cvt_file ${imap_prefix}/statuscache.db   "skiplist"
  cvt_file ${imap_prefix}/user_deny.db "flat"
  rm -vf ${imap_prefix}/db/log.*
  rm -vf ${imap_prefix}/db/__db.*
else
...


This all is done not for upgrades of BDB but for simple cases where BDB is
in a bad state for some reasons.

Now, the db_upgrade may be good for those situations where it works. But
I'm quite sure it doesn't work between some major versions of BDB. And
that's exactly what we had to care in the RPMs, so you can run RHEL2.1 on
i386 and "upgrade" it to RHEL5 on x86_64 and it will just work.

It works, but I'd be very happy to get rid of this stuff, really!

Simon


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/

Re: competition

[Shuvam Misra]

> Kolab Systems is thinking of such SQL databases for integration
> purposes, where the performance penalty now lies within having to use the
> IMAP protocol to gain access to the underlying metadata (seen status,
> message indexes) in distributed groupware environments where Cyrus
> itself is not the only component that needs access to such information
> (but would still remain authoritative, of course, and would be the only
> component with write access to most tables).
> 
> While not necessarily the best approach, it seems worth exploring.

Yes, this is one of the big reasons why we use RDB backends in many of
our products. The downside is performance, of course. But the upside is
very big when you need it -- ability to keep one instance of important
metadata and allow multiple pieces of software on multiple servers to
access it. Most of the time, this metadata is read-often-update-rarely,
so we don't need huge write performance. But if we have to
synchronise the data across multiple servers and use local files like
SQLite/BerkeleyDB/gdbm, we land up building entire synchronisation layers
just to ship updates around from server to server.

This is not Cyrus-specific -- any metadata which needs sharing across
multiple servers seems to be easier to build with a network-accessible
database, and an RDB is the commonest of them (SQLite being an exception).
Basically, memcached is "better than" BerkeleyDB hashmaps in these
situations.

Shuvam

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/

Re: competition

[Henrique de Moraes Holschuh]

On Thu, 23 Sep 2010, Shuvam Misra wrote:
> > >  given the issues with BDB.  Is it worth embedding a copy of
> > > BDB into the Cyrus distribution rather than using the OS one?  I
> > 
> > That way lies madness.
> > 
> > BDB is one of those things where arcane blackmagic skills are needed to keep
> > it working on all arches.  It uses scary crap to be fast and archive high
> > performance with large concurrency, and I've seen it break OpenLDAP in very
> > nasty arch-specific ways in the past (OpenLDAP is to BDB what Cyrus is to
> > mmap() ;-) ).
> > 
> > And it will break the world if you get symbol versioning wrong.  Lots of
> > libraries (SASL *and* glibc included!) might decide to shadow-dynamic-link
> > BDB with your application, and all sort of non-funny crap can happen if the
> > wrong linker magic is applied.  You *really* don't want to go there.
> 
> I was a strong advocate of bundling DB libraries, etc, with Cyrus. The
> points you've made here are very interesting. I didn't know many of these
> things. I'm re-thinking whether bundling is such a good idea now. Thanks.

Bundling is always a pain to get exactly right.

Bundled libs are also a security hazard.  Bundled libs *will* be disabled by
any Linux distro worth being called that, but even disabled they are not
harmless.  They will waste resources of the security teams while acessing if
an embedded copy of whatever lib needs to be updated with a security fix.

Bundling also complicate the build system.  Complicated build systems are a
VERY BAD idea on most projects, where nobody is really paying attention to
them.

-- 
  "One disk to rule them all, One disk to find them. One disk to bring
  them all and in the darkness grind them. In the Land of Redmond
  where the shadows lie." -- The Silicon Valley Tarot
  Henrique Holschuh

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/

Re: competition

[Shuvam Misra]

> >  given the issues with BDB.  Is it worth embedding a copy of
> > BDB into the Cyrus distribution rather than using the OS one?  I
> 
> That way lies madness.
> 
> BDB is one of those things where arcane blackmagic skills are needed to keep
> it working on all arches.  It uses scary crap to be fast and archive high
> performance with large concurrency, and I've seen it break OpenLDAP in very
> nasty arch-specific ways in the past (OpenLDAP is to BDB what Cyrus is to
> mmap() ;-) ).
> 
> And it will break the world if you get symbol versioning wrong.  Lots of
> libraries (SASL *and* glibc included!) might decide to shadow-dynamic-link
> BDB with your application, and all sort of non-funny crap can happen if the
> wrong linker magic is applied.  You *really* don't want to go there.

I was a strong advocate of bundling DB libraries, etc, with Cyrus. The
points you've made here are very interesting. I didn't know many of these
things. I'm re-thinking whether bundling is such a good idea now. Thanks.

Shuvam

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/

Re: competition

[Bron Gondwana]

On Wed, Sep 22, 2010 at 11:24:04PM +0200, Gabor Gombas wrote:
> On Wed, Sep 22, 2010 at 04:10:15PM +1000, Bron Gondwana wrote:
> 
> > Now - BDB database SHOULD be upgradable.  I want to find a BDB expert
> > to help me with that - (a) detecting that an upgrade is necessary, and
> > (b) doing the upgrade.
> 
> It was quite some time ago I last upgraded a Cyrus instance, but my
> upgrade script did the following:
> 
> - Run the _old_ version of db_recover. Just shutting down Cyrus is not
>   enough, without this step db_upgrade can fail.
> 
> - Run the _new_ version of db_upgrade.
> 
> - Run "db_checkpoint -1" to force the log to be rotated.
> 
> I think it won't be hard to implement steps 2 & 3 inside Cyrus, and I
> don't remember those steps taking a significant amount of time, but I
> don't know how it would be possible to implement step 1. However if the
> BDB version change does not bring a change to the on-disk DB log format,
> then it may be possible to skip steps 1 & 3.

Ok - so the upshot is that we need to implement a "shutdown" path, either
in master.c or in ctl_cyrusdb - and we need to make sure it gets run
during the usual shutdown path.

Bron.

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/

Re: competition

[Gabor Gombas]

On Wed, Sep 22, 2010 at 04:10:15PM +1000, Bron Gondwana wrote:

> Now - BDB database SHOULD be upgradable.  I want to find a BDB expert
> to help me with that - (a) detecting that an upgrade is necessary, and
> (b) doing the upgrade.

It was quite some time ago I last upgraded a Cyrus instance, but my
upgrade script did the following:

- Run the _old_ version of db_recover. Just shutting down Cyrus is not
  enough, without this step db_upgrade can fail.

- Run the _new_ version of db_upgrade.

- Run "db_checkpoint -1" to force the log to be rotated.

I think it won't be hard to implement steps 2 & 3 inside Cyrus, and I
don't remember those steps taking a significant amount of time, but I
don't know how it would be possible to implement step 1. However if the
BDB version change does not bring a change to the on-disk DB log format,
then it may be possible to skip steps 1 & 3.

Gabor

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/

Re: competition

[Wesley Craig]

On 22 Sep 2010, at 10:46, Dave McMurtrie wrote:
> Considering the state of Cyrus' interoperability with BDB and all the 
> recent fixes to skiplist, would it make sense to at least not make BDB a 
> default backend from now on?

Yes, and "sane defaults" was to be one of the themes of the 2.4 release.  Very 
pertinent if you're interested in making Cyrus IMAPd welcoming to new people...

:wes

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/

Re: competition

[John Madden]

> The big downside to using an SQL database is the enormous temptation to
> point all the Cyrus servers at the same Database server and lose the
> redundancy and scalability inherent in a multi node or Murder setup.

But the SQL world has this figured out, at least for reads.  For 
situations where DB *writes* need to be scaled, which the SQL world 
doesn't really have figured out, there is always the large list of 
up-and-coming noSQL solutions and they seem to fit Cyrus' use case 
pretty well.  Cyrus' single-murder-master situation is a write 
bottleneck that could be alleviated with such a solution.

John



-- 
John Madden
Sr UNIX Systems Engineer
Ivy Tech Community College of Indiana
jmad...@ivytech.edu

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/

Re: competition

[Andy Bennett]

Hi,

> Kolab Systems is thinking of such SQL databases for integration purposes,
 > where the performance penalty now lies within having to use the IMAP
 > protocol to gain access to the underlying metadata (seen status, message
 > indexes) in distributed groupware environments where Cyrus itself is not
 > the only component that needs access to such information (but would still
 > remain authoritative, of course, and would be the only component with 
write
 > access to most tables).
> 
> While not necessarily the best approach, it seems worth exploring.

What makes you think the query parsing and other overheads of SQL will 
be faster than IMAP?
I'd be interested in any numbers that you might have to support the effort.

The big downside to using an SQL database is the enormous temptation to 
point all the Cyrus servers at the same Database server and lose the 
redundancy and scalability inherent in a multi node or Murder setup.





Regards,
@ndy

-- 
andy...@ashurst.eu.org
http://www.ashurst.eu.org/
0x7EBA75FF


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/

Re: competition

[Jeroen van Meeuwen (Kolab Systems)]

Kolab Systems is thinking of such SQL databases for integration purposes, where 
the performance penalty now lies within having to use the IMAP protocol to gain 
access to the underlying metadata (seen status, message indexes) in distributed 
groupware environments where Cyrus itself is not the only component that needs 
access to such information (but would still remain authoritative, of course, 
and would be the only component with write access to most tables).

While not necessarily the best approach, it seems worth exploring.

Kind regards,

Jeroen van Meeuwen
-From his Android

"André Schild"  wrote:

>  Am 22.09.2010 16:17, schrieb Lucas Zinato Carraro:
>>   For me it would be very interesting a option to save cyrus tables
>>in a traditional database. ( mysql, postgresql, etc... )
>Beside "interesting" what would you get for a real benefit from this ?
>They are ver verly likely to be slower.
>
>André
>
>
>Cyrus Home Page: http://www.cyrusimap.org/
>List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/

Re: competition

[Henrique de Moraes Holschuh]

On Wed, 22 Sep 2010, Bron Gondwana wrote:
> Now - BDB database SHOULD be upgradable.  I want to find a BDB expert
> to help me with that - (a) detecting that an upgrade is necessary, and
> (b) doing the upgrade.

All I know is that there used to be an API call to upgrade the db
environment, which basically did exactly what the db_upgrade utility does.
It was easy to find before Oracle bought SleepyCat and screwed over with the
documentation, nowadays I don't even know where the API docs are.

If it is still exists, you could probably upgrade all DBs the first time
they're accessed (if the db->upgrade() call is cheap enough when there is no
work to be done, something I don't know).

> So the only thing really left to do is making sure we can upgrade
> BDB databases automatically.  Sure downgrades will suck, but that's
> not quite so scary.

Downgrades can only be done manually anyway, by dump+restore.  However, the
file formats don't change as much as the API/ABI anyway, often two different
BDB releases use the same file format.

>  given the issues with BDB.  Is it worth embedding a copy of
> BDB into the Cyrus distribution rather than using the OS one?  I

That way lies madness.

BDB is one of those things where arcane blackmagic skills are needed to keep
it working on all arches.  It uses scary crap to be fast and archive high
performance with large concurrency, and I've seen it break OpenLDAP in very
nasty arch-specific ways in the past (OpenLDAP is to BDB what Cyrus is to
mmap() ;-) ).

And it will break the world if you get symbol versioning wrong.  Lots of
libraries (SASL *and* glibc included!) might decide to shadow-dynamic-link
BDB with your application, and all sort of non-funny crap can happen if the
wrong linker magic is applied.  You *really* don't want to go there.

-- 
  "One disk to rule them all, One disk to find them. One disk to bring
  them all and in the darkness grind them. In the Land of Redmond
  where the shadows lie." -- The Silicon Valley Tarot
  Henrique Holschuh

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/

RE : competition

[Michel Sébastien]

> We wanted to use it for the user_deny database so we could insert a row
> into one database table that every host has access to.  This way we
> didn't need to come up with a way to update the local user_deny across
> each frontend server.

Such database provides the same benefit to the tlscache DB. Our frontend 
servers share a Mysql DB (HEAP) to cache TLS
> Also, we were interested in testing SQLite for things like
> tls_sessions.db and deliver.db because we were tired of BDB breaking our
> service.


Ce message et les pièces jointes sont confidentiels et réservés à l'usage 
exclusif de ses destinataires. Il peut également être protégé par le secret 
professionnel. Si vous recevez ce message par erreur, merci d'en avertir 
immédiatement l'expéditeur et de le détruire. L'intégrité du message ne pouvant 
être assurée sur Internet, la responsabilité du groupe Atos Origin ne pourra 
être recherchée quant au contenu de ce message. Bien que les meilleurs efforts 
soient faits pour maintenir cette transmission exempte de tout virus, 
l'expéditeur ne donne aucune garantie à cet égard et sa responsabilité ne 
saurait être recherchée pour tout dommage résultant d'un virus transmis.

This e-mail and the documents attached are confidential and intended solely for 
the addressee; it may also be privileged. If you receive this e-mail in error, 
please notify the sender immediately and destroy it. As its integrity cannot be 
secured on the Internet, the Atos Origin group liability cannot be triggered 
for the message content. Although the sender endeavours to maintain a computer 
virus-free network, the sender does not warrant that this transmission is 
virus-free and will not be liable for any damages resulting from any virus 
transmitted.


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/

Re: competition

[Dave McMurtrie]

On 09/22/2010 10:52 AM, André Schild wrote:
>Am 22.09.2010 16:17, schrieb Lucas Zinato Carraro:
>>For me it would be very interesting a option to save cyrus tables
>> in a traditional database. ( mysql, postgresql, etc... )
> Beside "interesting" what would you get for a real benefit from this ?
> They are ver verly likely to be slower.

We wanted to use it for the user_deny database so we could insert a row 
into one database table that every host has access to.  This way we 
didn't need to come up with a way to update the local user_deny across 
each frontend server.

Also, we were interested in testing SQLite for things like 
tls_sessions.db and deliver.db because we were tired of BDB breaking our 
service.

Thanks,

Dave
-- 
Dave McMurtrie, SPE
Email Systems Team Leader
Carnegie Mellon University,
Computing Services


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/

Re: competition

[André Schild]

  Am 22.09.2010 16:17, schrieb Lucas Zinato Carraro:
>   For me it would be very interesting a option to save cyrus tables
>in a traditional database. ( mysql, postgresql, etc... )
Beside "interesting" what would you get for a real benefit from this ?
They are ver verly likely to be slower.

André


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/

Re: competition

[Dave McMurtrie]

On 09/22/2010 10:17 AM, Lucas Zinato Carraro wrote:
>   For me it would be very interesting a option to save cyrus tables
>in a traditional database. ( mysql, postgresql, etc... )

In 2.3.13 (I think) and newer, there is the option of using an SQL 
backend.  It hasn't been widely used and tested yet, though.

Along the lines of Simon's question, other than the fact that some of 
the Cyrus database defaults happen to be BDB, is anyone using BDB with 
Cyrus because they really want to?

Considering the state of Cyrus' interoperability with BDB and all the 
recent fixes to skiplist, would it make sense to at least not make BDB a 
default backend from now on?

Thanks,

Dave
-- 
Dave McMurtrie, SPE
Email Systems Team Leader
Carnegie Mellon University,
Computing Services

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/

Re: competition

[Lucas Zinato Carraro]

 For me it would be very interesting a option to save cyrus tables
  in a traditional database. ( mysql, postgresql, etc... )

 Zinato

>
> On Wed, Sep 22, 2010 at 11:06 AM, Simon Matter  wrote:
>>> On Wednesday 22 September 2010 15:29:20 Kenneth Marshall wrote:
 On Wed, Sep 22, 2010 at 10:48:49PM +0930, Daniel O'Connor wrote:
 > On 22/09/2010, at 22:33, Kenneth Marshall wrote:
 > >> On a bad shutdown it requires admin intervention very frequently
 which
 > >> is pretty tedious.
 > >>
 > >> And yes, upgrading it is also a PITA.
 > >
 > > That is why we moved to skiplist. The server would require manual
 > > intervention to even restart after certain types of system events.
 > > It is hard to explain why mail was unavailable for 30+mins. when
 > > the server rebooted in 2mins. BDB would need to be much, much
 > > easier to manage and be more robust for us to use.
 >
 > Yep, I plan to migrate to skiplist some day :)
 >
 > Now if only I could use skiplist with openldap ;)

 Sadly, OpenLDAP does need the performance of BDB.
>>>
>>> With current "LDIF" exports of OpenLDAP, the downtime caused by a corrupt
>>> BDB
>>> database is "acceptable".
>>>
>>> Is a similar method possible with cyrus?
>>> Eg. a simple command that exports all BDB-database-files to a simple
>>> format
>>> that can easily be re-imported into Cyrus when the DBD is corrupted?
>>
>> That's exactly what our RPM does. It converts all BDB databases to
>> skiplist on shutdown. On startup, it detects the format of all database
>> files and converts it back to what is configured if needed.
>>
>> As Bron told us, with 2.4 the magic on startup is done inside Cyrus. My
>> question remains, why can't we get rid of BDB completely?
>>
>> Simon
>>
>> 
>> Cyrus Home Page: http://www.cyrusimap.org/
>> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
>>
>

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/

Re: competition

[Simon Matter]

> On Wednesday 22 September 2010 15:29:20 Kenneth Marshall wrote:
>> On Wed, Sep 22, 2010 at 10:48:49PM +0930, Daniel O'Connor wrote:
>> > On 22/09/2010, at 22:33, Kenneth Marshall wrote:
>> > >> On a bad shutdown it requires admin intervention very frequently
>> which
>> > >> is pretty tedious.
>> > >>
>> > >> And yes, upgrading it is also a PITA.
>> > >
>> > > That is why we moved to skiplist. The server would require manual
>> > > intervention to even restart after certain types of system events.
>> > > It is hard to explain why mail was unavailable for 30+mins. when
>> > > the server rebooted in 2mins. BDB would need to be much, much
>> > > easier to manage and be more robust for us to use.
>> >
>> > Yep, I plan to migrate to skiplist some day :)
>> >
>> > Now if only I could use skiplist with openldap ;)
>>
>> Sadly, OpenLDAP does need the performance of BDB.
>
> With current "LDIF" exports of OpenLDAP, the downtime caused by a corrupt
> BDB
> database is "acceptable".
>
> Is a similar method possible with cyrus?
> Eg. a simple command that exports all BDB-database-files to a simple
> format
> that can easily be re-imported into Cyrus when the DBD is corrupted?

That's exactly what our RPM does. It converts all BDB databases to
skiplist on shutdown. On startup, it detects the format of all database
files and converts it back to what is configured if needed.

As Bron told us, with 2.4 the magic on startup is done inside Cyrus. My
question remains, why can't we get rid of BDB completely?

Simon


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/

Re: competition

[J. Roeleveld]

On Wednesday 22 September 2010 15:29:20 Kenneth Marshall wrote:
> On Wed, Sep 22, 2010 at 10:48:49PM +0930, Daniel O'Connor wrote:
> > On 22/09/2010, at 22:33, Kenneth Marshall wrote:
> > >> On a bad shutdown it requires admin intervention very frequently which
> > >> is pretty tedious.
> > >> 
> > >> And yes, upgrading it is also a PITA.
> > > 
> > > That is why we moved to skiplist. The server would require manual
> > > intervention to even restart after certain types of system events.
> > > It is hard to explain why mail was unavailable for 30+mins. when
> > > the server rebooted in 2mins. BDB would need to be much, much
> > > easier to manage and be more robust for us to use.
> > 
> > Yep, I plan to migrate to skiplist some day :)
> > 
> > Now if only I could use skiplist with openldap ;)
> 
> Sadly, OpenLDAP does need the performance of BDB.

With current "LDIF" exports of OpenLDAP, the downtime caused by a corrupt BDB 
database is "acceptable".

Is a similar method possible with cyrus? 
Eg. a simple command that exports all BDB-database-files to a simple format 
that can easily be re-imported into Cyrus when the DBD is corrupted?

--
Joost

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/

Re: competition

[Adam Tauno Williams]

On Wed, 2010-09-22 at 15:04 +0200, Simon Matter wrote:
> > On Wed, 2010-09-22 at 14:44 +0200, J. Roeleveld wrote:
> >> On Wednesday 22 September 2010 13:47:26 Jeffrey T Eaton wrote:
> >> 
> >> I am probably missing some info here, but
> >> > And, as Bron has said, there's something wrong with the way Cyrus uses
> >> BDB.
> >> >  I've never been able to understand BDB well enough to figure it out
> >> > myself, nor have I ever found anyone who can help.  For what its
> >> worth, I
> >> > solved the problem by not using BDB at all on the Cyrus systems I
> >> used.
> >> If it is possible to not use BDB, and BDB causes problems with upgrades,
> >> why
> >> is BDB still used then?
> > BDB is wicked fast and scales well.  At least that is the typical
> > argument in defense of BDB.  And given the stellar performance one sees
> > from OpenLDAP I'm prone to believing it.
> Fine, that may all be true. But I never ever heard someone going from BDB
> to skiplist only and coming back because he had any performance issues.

True; and we moved to Skiplist, and no BDB, a long time ago.

I was just answering the question "why is BDB still used then?".  It may
be some degree faster, but IMO isn't worth the agony of figuring out BDB
issues.

> Maybe it doesn't matter so much with Cyrus and we should try to find out
> what is "Poetry and Truth"? If BDB is not really required for Cyrus it
> could be made optional for those who really want it.
> BTW, it's even worse than Bron said. You not only get errors in the logs
> even without using any BDB, you can also end up with a broken BDB
> environment which prevents Cyrus from starting up :)


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/

Re: competition

[Kenneth Marshall]

On Wed, Sep 22, 2010 at 10:48:49PM +0930, Daniel O'Connor wrote:
> 
> On 22/09/2010, at 22:33, Kenneth Marshall wrote:
> >> On a bad shutdown it requires admin intervention very frequently which is 
> >> pretty tedious.
> >> 
> >> And yes, upgrading it is also a PITA.
> >> 
> > 
> > That is why we moved to skiplist. The server would require manual
> > intervention to even restart after certain types of system events.
> > It is hard to explain why mail was unavailable for 30+mins. when
> > the server rebooted in 2mins. BDB would need to be much, much
> > easier to manage and be more robust for us to use.
> 
> Yep, I plan to migrate to skiplist some day :)
> 
> Now if only I could use skiplist with openldap ;)
> 
Sadly, OpenLDAP does need the performance of BDB.

Cheers,
Ken

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/

Re: competition

[Daniel O'Connor]

On 22/09/2010, at 22:33, Kenneth Marshall wrote:
>> On a bad shutdown it requires admin intervention very frequently which is 
>> pretty tedious.
>> 
>> And yes, upgrading it is also a PITA.
>> 
> 
> That is why we moved to skiplist. The server would require manual
> intervention to even restart after certain types of system events.
> It is hard to explain why mail was unavailable for 30+mins. when
> the server rebooted in 2mins. BDB would need to be much, much
> easier to manage and be more robust for us to use.

Yep, I plan to migrate to skiplist some day :)

Now if only I could use skiplist with openldap ;)

--
Daniel O'Connor software and network engineer
for Genesis Software - http://www.gsoft.com.au
"The nice thing about standards is that there
are so many of them to choose from."
  -- Andrew Tanenbaum
GPG Fingerprint - 5596 B766 97C0 0E94 4347 295E E593 DC20 7B3F CE8C







Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/

Re: competition

[Kenneth Marshall]

On Wed, Sep 22, 2010 at 10:27:11PM +0930, Daniel O'Connor wrote:
> 
> On 22/09/2010, at 22:17, Adam Tauno Williams wrote:
> 
> > On Wed, 2010-09-22 at 14:44 +0200, J. Roeleveld wrote:
> >> On Wednesday 22 September 2010 13:47:26 Jeffrey T Eaton wrote:
> >> 
> >> I am probably missing some info here, but
> >>> And, as Bron has said, there's something wrong with the way Cyrus uses 
> >>> BDB.
> >>> I've never been able to understand BDB well enough to figure it out
> >>> myself, nor have I ever found anyone who can help.  For what its worth, I
> >>> solved the problem by not using BDB at all on the Cyrus systems I used.
> >> If it is possible to not use BDB, and BDB causes problems with upgrades, 
> >> why 
> >> is BDB still used then?
> > 
> > BDB is wicked fast and scales well.  At least that is the typical
> > argument in defense of BDB.  And given the stellar performance one sees
> > from OpenLDAP I'm prone to believing it.
> 
> I'm not sure it's Cyrus, I see issues with OpenLDAP and BDB too.
> 
> On a bad shutdown it requires admin intervention very frequently which is 
> pretty tedious.
> 
> And yes, upgrading it is also a PITA.
> 

That is why we moved to skiplist. The server would require manual
intervention to even restart after certain types of system events.
It is hard to explain why mail was unavailable for 30+mins. when
the server rebooted in 2mins. BDB would need to be much, much
easier to manage and be more robust for us to use.

Cheers,
Ken

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/

Re: competition

[Simon Matter]

> On Wed, 2010-09-22 at 14:44 +0200, J. Roeleveld wrote:
>> On Wednesday 22 September 2010 13:47:26 Jeffrey T Eaton wrote:
>> 
>> I am probably missing some info here, but
>> > And, as Bron has said, there's something wrong with the way Cyrus uses
>> BDB.
>> >  I've never been able to understand BDB well enough to figure it out
>> > myself, nor have I ever found anyone who can help.  For what its
>> worth, I
>> > solved the problem by not using BDB at all on the Cyrus systems I
>> used.
>> If it is possible to not use BDB, and BDB causes problems with upgrades,
>> why
>> is BDB still used then?
>
> BDB is wicked fast and scales well.  At least that is the typical
> argument in defense of BDB.  And given the stellar performance one sees
> from OpenLDAP I'm prone to believing it.

Fine, that may all be true. But I never ever heard someone going from BDB
to skiplist only and coming back because he had any performance issues.
Maybe it doesn't matter so much with Cyrus and we should try to find out
what is "Poetry and Truth"? If BDB is not really required for Cyrus it
could be made optional for those who really want it.

BTW, it's even worse than Bron said. You not only get errors in the logs
even without using any BDB, you can also end up with a broken BDB
environment which prevents Cyrus from starting up :)

Simon


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/

Re: competition

[Daniel O'Connor]

On 22/09/2010, at 22:17, Adam Tauno Williams wrote:

> On Wed, 2010-09-22 at 14:44 +0200, J. Roeleveld wrote:
>> On Wednesday 22 September 2010 13:47:26 Jeffrey T Eaton wrote:
>> 
>> I am probably missing some info here, but
>>> And, as Bron has said, there's something wrong with the way Cyrus uses BDB.
>>> I've never been able to understand BDB well enough to figure it out
>>> myself, nor have I ever found anyone who can help.  For what its worth, I
>>> solved the problem by not using BDB at all on the Cyrus systems I used.
>> If it is possible to not use BDB, and BDB causes problems with upgrades, why 
>> is BDB still used then?
> 
> BDB is wicked fast and scales well.  At least that is the typical
> argument in defense of BDB.  And given the stellar performance one sees
> from OpenLDAP I'm prone to believing it.

I'm not sure it's Cyrus, I see issues with OpenLDAP and BDB too.

On a bad shutdown it requires admin intervention very frequently which is 
pretty tedious.

And yes, upgrading it is also a PITA.

--
Daniel O'Connor software and network engineer
for Genesis Software - http://www.gsoft.com.au
"The nice thing about standards is that there
are so many of them to choose from."
  -- Andrew Tanenbaum
GPG Fingerprint - 5596 B766 97C0 0E94 4347 295E E593 DC20 7B3F CE8C







Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/

Re: competition

[Adam Tauno Williams]

On Wed, 2010-09-22 at 14:44 +0200, J. Roeleveld wrote:
> On Wednesday 22 September 2010 13:47:26 Jeffrey T Eaton wrote:
> 
> I am probably missing some info here, but
> > And, as Bron has said, there's something wrong with the way Cyrus uses BDB.
> >  I've never been able to understand BDB well enough to figure it out
> > myself, nor have I ever found anyone who can help.  For what its worth, I
> > solved the problem by not using BDB at all on the Cyrus systems I used.
> If it is possible to not use BDB, and BDB causes problems with upgrades, why 
> is BDB still used then?

BDB is wicked fast and scales well.  At least that is the typical
argument in defense of BDB.  And given the stellar performance one sees
from OpenLDAP I'm prone to believing it.


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/

Re: competition

[J. Roeleveld]

On Wednesday 22 September 2010 13:47:26 Jeffrey T Eaton wrote:



I am probably missing some info here, but

> And, as Bron has said, there's something wrong with the way Cyrus uses BDB.
>  I've never been able to understand BDB well enough to figure it out
> myself, nor have I ever found anyone who can help.  For what its worth, I
> solved the problem by not using BDB at all on the Cyrus systems I used.

If it is possible to not use BDB, and BDB causes problems with upgrades, why 
is BDB still used then?

--
Joost

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/

Re: competition

[Simon Matter]

>
> On Sep 22, 2010, at 7:57 AM, Sebastian Hagedorn wrote:
>
>> --On 22. September 2010 07:47:26 -0400 Jeffrey T Eaton 
>> wrote:
>>
>>> All of that said, I believe that, in general, you can safely upgrade
>>> BDB.
>>> If you have a Cyrus installation using BDB X, you can drop in a new
>>> Cyrus
>>> using BDB Y, as long as everything is shut down in between.  You can't
>>> go
>>> back without effort, but upgrades should work.
>>
>> I'm pretty sure that's not true. *If* an application uses BDB's
>> environment features (not all do!), you *cannot* simply upgrade, unless
>> the application is programmed to perform the upgrade automatically.
>> Otherwise you have to either use db_upgrade to convert the environment
>> to the current version, or you use db_dump and db_load. There's a
>> db_upgrade() call in the library that could be used, but it might not
>> work in all cases:
>>
>> Errors
>> The DB->upgrade() method may fail and return one of the following
>> non-zero errors:
>>
>> DB_OLD_VERSION - The database cannot be upgraded by this version of the
>> Berkeley DB software.
>> --
>>.:.Sebastian Hagedorn - RZKR-R1 (Gebäude 52), Zimmer 18.:.
>>.:.Regionales Rechenzentrum (RRZK).:.
>> .:.Universität zu Köln / Cologne University - ✆ +49-221-478-5587.:.
>
> Hmm.  I thought that I had updated BDB in the past.  I probably just nuked
> the databases between updates, since we only used them for transient data
> (tls_sessions.db, deliver.db, ptscache,db).

You can upgrade between some versions but not all.

>
> The rest of my point stands;  changing the BDB version is a problem in and
> of itself, not changing the Cyrus version. I know of no reason why one
> couldn't update a Cyrus 2.2 installation to 2.3.

Well, I never said it's a problem of Cyrus, it's a problem of BDB. Just
ask the OpenLDAP folks - and those doing distribution packages for it.

The only way I see for Cyrus is to stay away from it.

Simon


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/

Re: competition

[Jeffrey T Eaton]

On Sep 22, 2010, at 7:57 AM, Sebastian Hagedorn wrote:

> --On 22. September 2010 07:47:26 -0400 Jeffrey T Eaton  wrote:
> 
>> All of that said, I believe that, in general, you can safely upgrade BDB.
>> If you have a Cyrus installation using BDB X, you can drop in a new Cyrus
>> using BDB Y, as long as everything is shut down in between.  You can't go
>> back without effort, but upgrades should work.
> 
> I'm pretty sure that's not true. *If* an application uses BDB's environment 
> features (not all do!), you *cannot* simply upgrade, unless the application 
> is programmed to perform the upgrade automatically. Otherwise you have to 
> either use db_upgrade to convert the environment to the current version, or 
> you use db_dump and db_load. There's a db_upgrade() call in the library that 
> could be used, but it might not work in all cases:
> 
> Errors
> The DB->upgrade() method may fail and return one of the following non-zero 
> errors:
> 
> DB_OLD_VERSION - The database cannot be upgraded by this version of the 
> Berkeley DB software.
> -- 
>.:.Sebastian Hagedorn - RZKR-R1 (Gebäude 52), Zimmer 18.:.
>.:.Regionales Rechenzentrum (RRZK).:.
> .:.Universität zu Köln / Cologne University - ✆ +49-221-478-5587.:.

Hmm.  I thought that I had updated BDB in the past.  I probably just nuked the 
databases between updates, since we only used them for transient data 
(tls_sessions.db, deliver.db, ptscache,db).

The rest of my point stands;  changing the BDB version is a problem in and of 
itself, not changing the Cyrus version. I know of no reason why one couldn't 
update a Cyrus 2.2 installation to 2.3.

-jeaton

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/

Re: competition

[Sebastian Hagedorn]

--On 22. September 2010 07:47:26 -0400 Jeffrey T Eaton  
wrote:



All of that said, I believe that, in general, you can safely upgrade BDB.
If you have a Cyrus installation using BDB X, you can drop in a new Cyrus
using BDB Y, as long as everything is shut down in between.  You can't go
back without effort, but upgrades should work.


I'm pretty sure that's not true. *If* an application uses BDB's environment 
features (not all do!), you *cannot* simply upgrade, unless the application 
is programmed to perform the upgrade automatically. Otherwise you have to 
either use db_upgrade to convert the environment to the current version, or 
you use db_dump and db_load. There's a db_upgrade() call in the library 
that could be used, but it might not work in all cases:


Errors
The DB->upgrade() method may fail and return one of the following non-zero 
errors:


DB_OLD_VERSION - The database cannot be upgraded by this version of the 
Berkeley DB software.

--
.:.Sebastian Hagedorn - RZKR-R1 (Gebäude 52), Zimmer 18.:.
.:.Regionales Rechenzentrum (RRZK).:.
.:.Universität zu Köln / Cologne University - ✆ +49-221-478-5587.:.

p7s98D0p7NYL6.p7s
Description: S/MIME cryptographic signature

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/

Re: competition

[Jeffrey T Eaton]

On Sep 22, 2010, at 1:50 AM, Simon Matter wrote:

>> 
>>> Debian is still stuck on 2.2 and there seems to be no progress in that
>>> area.
>>> 
>>> The main problem they apparently have, is the migration path for the
>>> various
>>> DB files from 2.2 to 2.3.
>>> (The 2.3 version itself works fine as .deb packages)
>> 
>> What "migration path"?  Cyrus 2.3 supports all of the same database
>> backends that Cyrus 2.2 did.
> 
> I don't know the Debian packages well but from a quick look I did long
> time ago it seems like those packages leave quite some work to do by the
> user of the packages. That also means there are more ways for a user to
> break things which may stop them making the 2.3 packages go in easily.
> 
>> 
>> To the best of my knowledge, you can drop in Cyrus 2.3 binaries, and
>> with the same config files as 2.2 used, everything will just work.
>> You can't easily go back, because I believe that 2.3 will update
>> cyrus.index files to a format which 2.2 doesn't recognize, but that
>> shouldn't prevent anyone from upgrading.
> 
> IIRC it's a little bit more complicated. Beside BDB, there are also some
> steps to do at upgrade, like compiling sieve scripts, possibly converting
> its encoding.

To the best of my knowledge, the complied Sieve format has not changed since 
2.2.  

> With BDB backends you have to make sure everything fits
> correctly. Binaries need to be linked against the correct version of BDB
> and also the on disk BDB files need to be of the same version. Now, think
> you want to do a distribution upgrade which comes with the latest greatest
> BDB release and new Cyrus binaries which are using them, but your spool is
> still on the old release... We all know how this ends.

This would be a problem anytime you upgrade BDB, independent of the version of 
Cyrus.  If you are rolling out a new distribution  with a new version of BDB, 
you will need to address this problem even if you don't upgrade Cyrus.  (Unless 
of course, you continue to ship a Cyrus built against the _old_ BDB, in which 
case the argument is irrelevant).

> The only solution - you can call it dirty workaround - I found for our
> RedHat EL RPMs was to convert all BDB databases to skiplist on Cyrus
> shutdown and convert them back on startup. That way the spool is always in
> a state which can be migrated once Cyrus is shutdown.
> Of course getting rid of any BDB in the default configuration of the RPM
> binaries has helped much to prevent BDB mess.


All of that said, I believe that, in general, you can safely upgrade BDB.  If 
you have a Cyrus installation using BDB X, you can drop in a new Cyrus using 
BDB Y, as long as everything is shut down in between.  You can't go back 
without effort, but upgrades should work.

And, as Bron has said, there's something wrong with the way Cyrus uses BDB.  
I've never been able to understand BDB well enough to figure it out myself, nor 
have I ever found anyone who can help.  For what its worth, I solved the 
problem by not using BDB at all on the Cyrus systems I used.

-jeaton

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/

Re: competition

[Bron Gondwana]

On Wed, Sep 22, 2010 at 10:13:20AM +0200, J. Roeleveld wrote:
> On Wednesday 22 September 2010 09:01:33 Bron Gondwana wrote:
> > On Wed, Sep 22, 2010 at 08:02:35AM +0200, Simon Matter wrote:
> 
> 
> 
> > Bron ( really trying to make Cyrus newbie-friendly as well as advanced-site
> >friendly.  I also want auto-recompilation of sieve scripts, and
> >auto-fixing of broken .cache files )
> 
> Now that you mention it, what would be needed to do to "auto-recompile" sieve-
> scripts?
> If that would work, I could edit the file directly in the respective 
> location, 
> rather then having to play with the sieve management tool or a MUA with sieve-
> support. :)

Hmm...

I take it all back. We did this a while back, and it's the path to great
sadness.  Directly editing the file in place means we can't auto-replicate
it, can't add CRC checks on the contents, etc.

The files on disk are internal format files, not to be fiddled with
directly.  Use an API.  Now the lack of a good command to copy a file
into place, enable it, etc.  I'm sold on that.  A command line tool
to install a new sieve file for a user would be handy.

But no direct editing of files on disk please.

Bron.

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/

Re: competition

[J. Roeleveld]

On Wednesday 22 September 2010 09:01:33 Bron Gondwana wrote:
> On Wed, Sep 22, 2010 at 08:02:35AM +0200, Simon Matter wrote:



> Bron ( really trying to make Cyrus newbie-friendly as well as advanced-site
>friendly.  I also want auto-recompilation of sieve scripts, and
>auto-fixing of broken .cache files )

Now that you mention it, what would be needed to do to "auto-recompile" sieve-
scripts?
If that would work, I could edit the file directly in the respective location, 
rather then having to play with the sieve management tool or a MUA with sieve-
support. :)

--
Joost

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/

Re: competition

[Bron Gondwana]

On Wed, 22 Sep 2010 09:12 +0200, "Sebastian Hagedorn"  
wrote:
> --On 22. September 2010 16:10:15 +1000 Bron Gondwana  
> wrote:
> 
> > Now - BDB database SHOULD be upgradable.  I want to find a BDB expert
> > to help me with that - (a) detecting that an upgrade is necessary, and
> > (b) doing the upgrade.
> 
> I wouldn't exactly call myself an expert, but I think I understand the 
> basics of BDB operation. The biggest problem is that there may be
> multiple 
> version of the BDB utilities installed. So you need to know which BDB 
> version Cyrus was linked against and somehow figure out what the 
> corresponding tools are on the box you're on. The tool are named db_stat, 
> db_verify, db_upgrade etc., but it's not unusual to have db41_stat, 
> db42_stat etc .
> The "file" utility is pretty useful, because it tells you the version of
> a 
> BDB file. Unfortunately that version has nothing to to with the version 
> number of the BDB libraries themselves. Instead they look like this:
> 
> Berkeley DB (Hash, version 8, native byte-order)

Yeah, I can get that just by reading the value as byte 12 in the file.  I 
pulled a copy of the BDB source and grabbed their magic numbers already.

The thing is - I don't want to be calling out to the tools - I want to be doing 
it in_process with a Cyrus tool that's already compiled against the correct 
version of BDB.

> I suppose one could do this:
> - at startup, create a new BDB hash file using a Cyrus binary
> - check its version number using "file"
> - compare that to the existing BDB files
> - if they're different you need to perform an upgrade

Oooh.  Actually, that's a point.  Create a temporary file of each type and pull 
the version field from it, then check each BDB file's version field.  Hmm. 
That's quite viable actually.  I like it :)

> The upgrade itself *should* be doable using db_upgrade (using the version 
> that goes with the libraries Cyrus is linked against), although I think 
> there are cases where you have to use db_dump followed by db_load. The 
> problem with the latter is that you need the dump from the *old* version
> of 
> the libraries!

http://download.oracle.com/docs/cd/E17076_02/html/upgrading/index.html seems to 
have good information on "what's changed" in each version.

On the topic of upgrades:
http://download.oracle.com/docs/cd/E17076_02/html/programmer_reference/am_upgrade.html

"We do guarantee that any archived database may be upgraded using a current 
Berkeley DB software release and the DB->upgrade() method, and there is no need 
to step-wise upgrade the database using intermediate releases of Berkeley DB. 
Sites should consider archiving appropriate copies of their application or 
application sources if they may need to access archived databases without first 
upgrading them."

> >  given the issues with BDB.  Is it worth embedding a copy of
> > BDB into the Cyrus distribution rather than using the OS one?  I
> > know it's generally considered bad taste, but it sure makes
> > keeping in sync easier!
> 
> That would certainly help with the issues I described above, but only if 
> you either also include the utilities or have your own binaries that 
> fulfill their function.

No need I don't think if we do the little trick above to figure out BDB version 
and upgrade.  Ideally, I'd like to not be using BDB because the 
incompatibilities are a killer.  If we could just say "use a backwards 
compatible format" to BDB and have it readable by any 3.x + version, I'd be a 
very happy camper.

Bron.
-- 
  Bron Gondwana
  br...@fastmail.fm


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/

Re: competition

[Sebastian Hagedorn]

--On 22. September 2010 16:10:15 +1000 Bron Gondwana  
wrote:



Now - BDB database SHOULD be upgradable.  I want to find a BDB expert
to help me with that - (a) detecting that an upgrade is necessary, and
(b) doing the upgrade.


I wouldn't exactly call myself an expert, but I think I understand the 
basics of BDB operation. The biggest problem is that there may be multiple 
version of the BDB utilities installed. So you need to know which BDB 
version Cyrus was linked against and somehow figure out what the 
corresponding tools are on the box you're on. The tool are named db_stat, 
db_verify, db_upgrade etc., but it's not unusual to have db41_stat, 
db42_stat etc .
The "file" utility is pretty useful, because it tells you the version of a 
BDB file. Unfortunately that version has nothing to to with the version 
number of the BDB libraries themselves. Instead they look like this:


Berkeley DB (Hash, version 8, native byte-order)

I suppose one could do this:
- at startup, create a new BDB hash file using a Cyrus binary
- check its version number using "file"
- compare that to the existing BDB files
- if they're different you need to perform an upgrade

The upgrade itself *should* be doable using db_upgrade (using the version 
that goes with the libraries Cyrus is linked against), although I think 
there are cases where you have to use db_dump followed by db_load. The 
problem with the latter is that you need the dump from the *old* version of 
the libraries!



 given the issues with BDB.  Is it worth embedding a copy of
BDB into the Cyrus distribution rather than using the OS one?  I
know it's generally considered bad taste, but it sure makes
keeping in sync easier!


That would certainly help with the issues I described above, but only if 
you either also include the utilities or have your own binaries that 
fulfill their function.

--
.:.Sebastian Hagedorn - RZKR-R1 (Gebäude 52), Zimmer 18.:.
.:.Regionales Rechenzentrum (RRZK).:.
.:.Universität zu Köln / Cologne University - ✆ +49-221-478-5587.:.

p7sOFjStOGfiA.p7s
Description: S/MIME cryptographic signature

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/

Re: competition

[Bron Gondwana]

On Wed, Sep 22, 2010 at 08:02:35AM +0200, Simon Matter wrote:
> Documentation is one thing, and dependencies like BDB another. But there
> is something else I guess, for servers which are not dedicated mail
> server, it would be really nice if one could install Cyrus and it just
> works for every user having an account on the system. That can be done
> with the autocreate patches but they are also not included in Cyrus. In
> fact, Cyrus could have two main modes, one for dedicated, sealed mail
> servers and one for "allround" servers with user accounts.

We're planning to include the autocreate patches in 2.4 as soon as we make
sure they work nicely with virtdomains and murder.  Making it work nicely
with user accounts would be very handy.

Bron ( really trying to make Cyrus newbie-friendly as well as advanced-site
   friendly.  I also want auto-recompilation of sieve scripts, and
   auto-fixing of broken .cache files )

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/

Re: competition

[André Schild]

  Am 21.09.2010 23:15, schrieb Jeffrey T Eaton:
>> Debian is still stuck on 2.2 and there seems to be no progress in that area.
>>
>> The main problem they apparently have, is the migration path for the various
>> DB files from 2.2 to 2.3.
>> (The 2.3 version itself works fine as .deb packages)
> What "migration path"?  Cyrus 2.3 supports all of the same database
> backends that Cyrus 2.2 did.
>
> To the best of my knowledge, you can drop in Cyrus 2.3 binaries, and
> with the same config files as 2.2 used, everything will just work.
> You can't easily go back, because I believe that 2.3 will update
> cyrus.index files to a format which 2.2 doesn't recognize, but that
> shouldn't prevent anyone from upgrading.
 From a cyrus point of view this might be true,
but in real upgrades most of the time also the DB version changes
and here is one of the pitfalls (Not only for cyrus)

Here the current statement from the debian guys.

http://lists.alioth.debian.org/pipermail/pkg-cyrus-imapd-debian-devel/2010-August/003427.html

André

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/

Re: competition

[Bron Gondwana]

On Wed, Sep 22, 2010 at 07:50:25AM +0200, Simon Matter wrote:
> >
> >> Debian is still stuck on 2.2 and there seems to be no progress in that
> >> area.
> >>
> >> The main problem they apparently have, is the migration path for the
> >> various
> >> DB files from 2.2 to 2.3.
> >> (The 2.3 version itself works fine as .deb packages)
> >
> > What "migration path"?  Cyrus 2.3 supports all of the same database
> > backends that Cyrus 2.2 did.
> 
> I don't know the Debian packages well but from a quick look I did long
> time ago it seems like those packages leave quite some work to do by the
> user of the packages. That also means there are more ways for a user to
> break things which may stop them making the 2.3 packages go in easily.
> 
> >
> > To the best of my knowledge, you can drop in Cyrus 2.3 binaries, and
> > with the same config files as 2.2 used, everything will just work.
> > You can't easily go back, because I believe that 2.3 will update
> > cyrus.index files to a format which 2.2 doesn't recognize, but that
> > shouldn't prevent anyone from upgrading.
> 
> IIRC it's a little bit more complicated. Beside BDB, there are also some
> steps to do at upgrade, like compiling sieve scripts, possibly converting
> its encoding. With BDB backends you have to make sure everything fits
> correctly. Binaries need to be linked against the correct version of BDB
> and also the on disk BDB files need to be of the same version. Now, think
> you want to do a distribution upgrade which comes with the latest greatest
> BDB release and new Cyrus binaries which are using them, but your spool is
> still on the old release... We all know how this ends.
> The only solution - you can call it dirty workaround - I found for our
> RedHat EL RPMs was to convert all BDB databases to skiplist on Cyrus
> shutdown and convert them back on startup. That way the spool is always in
> a state which can be migrated once Cyrus is shutdown.
> Of course getting rid of any BDB in the default configuration of the RPM
> binaries has helped much to prevent BDB mess.

Now - BDB database SHOULD be upgradable.  I want to find a BDB expert
to help me with that - (a) detecting that an upgrade is necessary, and
(b) doing the upgrade.

I've already changed ctl_cyrusdb -r (run on startup) to automatically
convert between skiplist, bdb and bdb-hash formats based on reading
the magic at the start of the file on disk.

So the only thing really left to do is making sure we can upgrade
BDB databases automatically.  Sure downgrades will suck, but that's
not quite so scary.


 given the issues with BDB.  Is it worth embedding a copy of
BDB into the Cyrus distribution rather than using the OS one?  I
know it's generally considered bad taste, but it sure makes
keeping in sync easier!

Bron ( the downside being security issue tracking, of course )

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/

Re: competition

[Simon Matter]

> On 9/20/2010 8:59 AM, Marc Patermann wrote:
>> And still, if someone asks a mailing list (not here certainly) how to
>> start with IMAPd, many people shout, to go with dovecot and not using
>> Cyrus.
>
> Hi -
>
> A little late to this thread, but here are a couple of modest
> observations:
>
> 1.
> I have cyrus and dovecot installations (with postfix smtp).  dovecot
> doesn't support an lmtp transport, which I finally decided was a deal
> killer.  In our (~1000 user) dovecot system, we're using procmail to
> ferry messages from the smtp server to dovecot deliver.  One very
> disturbing recent finding is that with some regularity messages are not
> being delivered with no notification to the sender -- they just get
> dropped. We think it's an ldap authentication problem, but it doesn't
> really matter what the cause is:  mail should either be delivered or
> someone should find out that it wasn't withough having to snoop around
> in log files.   Procmail is *supposed* to send this stuff back up the
> pipe when something goes wrong, but it's not happening.  Using lmtp is
> clean and simple and affords the administrator a huge amount of
> flexibility when using postfix:
>
>   mailbox_transport = lmtp:unix:/var/run/cyrus/socket/lmtp
>
> This doesn't work with dovecot; you have to use the mailbox_command.

I think dovecot now has LMTP support in the latest (dev?) version.

> That said, after spending the summer on the cyrus-devel list, I have a
> lot of confidence in the work that particularly Bron Gondwana has been
> doing and think that with -- with an infusion of some clear
> documentation, a project I'm more than happy to contribute to -- cyrus
> can probably become the default open source imap server.

Documentation is one thing, and dependencies like BDB another. But there
is something else I guess, for servers which are not dedicated mail
server, it would be really nice if one could install Cyrus and it just
works for every user having an account on the system. That can be done
with the autocreate patches but they are also not included in Cyrus. In
fact, Cyrus could have two main modes, one for dedicated, sealed mail
servers and one for "allround" servers with user accounts.

Simon


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/

Re: competition

[Simon Matter]

>
>> Debian is still stuck on 2.2 and there seems to be no progress in that
>> area.
>>
>> The main problem they apparently have, is the migration path for the
>> various
>> DB files from 2.2 to 2.3.
>> (The 2.3 version itself works fine as .deb packages)
>
> What "migration path"?  Cyrus 2.3 supports all of the same database
> backends that Cyrus 2.2 did.

I don't know the Debian packages well but from a quick look I did long
time ago it seems like those packages leave quite some work to do by the
user of the packages. That also means there are more ways for a user to
break things which may stop them making the 2.3 packages go in easily.

>
> To the best of my knowledge, you can drop in Cyrus 2.3 binaries, and
> with the same config files as 2.2 used, everything will just work.
> You can't easily go back, because I believe that 2.3 will update
> cyrus.index files to a format which 2.2 doesn't recognize, but that
> shouldn't prevent anyone from upgrading.

IIRC it's a little bit more complicated. Beside BDB, there are also some
steps to do at upgrade, like compiling sieve scripts, possibly converting
its encoding. With BDB backends you have to make sure everything fits
correctly. Binaries need to be linked against the correct version of BDB
and also the on disk BDB files need to be of the same version. Now, think
you want to do a distribution upgrade which comes with the latest greatest
BDB release and new Cyrus binaries which are using them, but your spool is
still on the old release... We all know how this ends.
The only solution - you can call it dirty workaround - I found for our
RedHat EL RPMs was to convert all BDB databases to skiplist on Cyrus
shutdown and convert them back on startup. That way the spool is always in
a state which can be migrated once Cyrus is shutdown.
Of course getting rid of any BDB in the default configuration of the RPM
binaries has helped much to prevent BDB mess.

Simon


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/

Re: competition

[Patrick Goetz]

On 9/20/2010 8:59 AM, Marc Patermann wrote:
> And still, if someone asks a mailing list (not here certainly) how to 
> start with IMAPd, many people shout, to go with dovecot and not using Cyrus.

Hi -

A little late to this thread, but here are a couple of modest observations:

1.
I have cyrus and dovecot installations (with postfix smtp).  dovecot
doesn't support an lmtp transport, which I finally decided was a deal
killer.  In our (~1000 user) dovecot system, we're using procmail to
ferry messages from the smtp server to dovecot deliver.  One very
disturbing recent finding is that with some regularity messages are not
being delivered with no notification to the sender -- they just get
dropped. We think it's an ldap authentication problem, but it doesn't
really matter what the cause is:  mail should either be delivered or
someone should find out that it wasn't withough having to snoop around
in log files.   Procmail is *supposed* to send this stuff back up the
pipe when something goes wrong, but it's not happening.  Using lmtp is
clean and simple and affords the administrator a huge amount of
flexibility when using postfix:

  mailbox_transport = lmtp:unix:/var/run/cyrus/socket/lmtp

This doesn't work with dovecot; you have to use the mailbox_command.

2.
I hate Maildir, and Maildir+ is a kludge.  I have some users whose mail
folders are nested multiple levels deep, and when push comes to shove
it's nice to be able to use the file system to examine mail messages
easily.  dovecot will eventually support some new format called dbox,
but when I asked him about it, the dovecot developer told me that it's
not production quality yet.  The simple filesystem mail message
interface is another thing I like about cyrus.

3.
So, why the sudden popularity of dovecot?  It could come down to
documentation.  The cyrus documentation currently is beyond terrible.
Dovecot has an excellent wiki which covers an awful lot of use cases.
Further, the dovecot developer (I think there's only one) is a nice guy
whom you can frequently find on IRC.  I've learned a lot about debugging
imap problems directly from him.

That said, after spending the summer on the cyrus-devel list, I have a
lot of confidence in the work that particularly Bron Gondwana has been
doing and think that with -- with an infusion of some clear
documentation, a project I'm more than happy to contribute to -- cyrus
can probably become the default open source imap server.



Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/

Re: competition

[Jeffrey T Eaton]

> Debian is still stuck on 2.2 and there seems to be no progress in that area.
> 
> The main problem they apparently have, is the migration path for the various
> DB files from 2.2 to 2.3.
> (The 2.3 version itself works fine as .deb packages)

What "migration path"?  Cyrus 2.3 supports all of the same database
backends that Cyrus 2.2 did.

To the best of my knowledge, you can drop in Cyrus 2.3 binaries, and
with the same config files as 2.2 used, everything will just work.
You can't easily go back, because I believe that 2.3 will update
cyrus.index files to a format which 2.2 doesn't recognize, but that
shouldn't prevent anyone from upgrading.

-jeaton


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/

Re: competition

[Adam Tauno Williams]

On Tue, 2010-09-21 at 14:14 +0200, Marc Patermann wrote:
> Hi,
> Adam Tauno Williams schrieb am 21.09.2010 12:04 Uhr:
> > On Tue, 2010-09-21 at 11:44 +0200, Marc Patermann wrote:
> >> André Schild schrieb am 21.09.2010 10:40 Uhr:
> >>> Am 21.09.2010 09:31, schrieb Pascal Gienger:
>  I begin to be tired from this "dovecot is much more besser, you
>   HAVE TO USE IT", why don't you migrate, ... ...?
> >> Yes, I'm too. But this is what you see in forums and mailing list.
> >> And my post is intended to see what the Cyrus community can do to
> >> /explain/ the product any better.
> > "Be there" is the only thing I can think of.  The only way to counter
> >  advocates is to be one.
> Yes, "Be there" - but "Be prepared" and have something to point to: "See
> it is easy and all there, dude!"
> >>> One important thing is the documentation of the imap server, and
> >>>  there cyrus could offer more (Just my opinion)
> >> The new web site is a good start. We should start a best practice
> >> section in the wiki. - How do I install Cyrus on Debian/Ubuntu/...?
> > Ugh, please no.  Distro specific directions are lame, pointless, and 
> > unmaintainable [beyond possibly packages-are-here].
> Yes, you're right. But in many cases it just point to somewhere else,
> which might be better than having nothing.
> For Ubuntu there is a howto: http://wiki.ubuntuusers.de/Cyrus_IMAPD and
> what do you get? 2.2.13 - uargh:
> http://packages.ubuntu.com/lucid/cyrus-imapd-2.2
> That may not be, what we want.

It is an Ubuntu wiki - *of course* it is out of date.  And probably
wrong/bad in the first place.

> > Once the packages are installed setting up is essentially the same
> > everywhere.
> Yes. But "install" is always first. If you do not get the user to get it
> installed, everything else is useless.
> 
> If you were a newbie and most people were trying to talk you away from
> Cyrus and there is no deb or rpm, would you try to compile the software 
> yourself?

There are RPMs.  The first Bing result for "cyrus-imapd rpm".

So, while I understand your point, I don't think the situation is as
dire as building-from-source.

> >> - From single server to multi server? - How to start with
> >> partitions and why? - Where to get latest releases (rpm/deb) when
> >> is not in my distriburtion (simon matter etc.)? - Everyone is
> >> talking about "Push-Mail" - how to do it with Cyrus?
> > It just happens if you your client supports IMAP idle.
> _I_ know that. :)
> It is just one example what I think user could search for. And "Push 
> mail" IMHO was one of the email buzz words of the last few years.
> > K9 on the Android seems to work very well with Cyrus in the 'push'
> > mode.  Although I've noticed users usually turn it off within a
> > couple of weeks - really, nobody wants push-mail once they have it.
> I disagree.

That IMAP idle works, or that people don't actually want push?

> >> - What if I want "Groupware" later, can I still use Cyrus? (link to
> >> projects supporting Cyrus, OX etc)
> > Shameless self promotion -I have a chapter on Cyrus in WMOGAG 
> > (OpenGroupware Administrator's Guide) 
> >  
> > I've gotten quite a few compliments on it.
> You could put your "Shameless self promotion" in the wiki then. :)

It certainly isn't an 'official' Cyrus project document, so I'm hesitant
to do so.



Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/

Re: competition

[Claus]

Hi,

> [...]
> The new web site is a good start.
> We should start a best practice section in the wiki.
> - How do I install Cyrus on Debian/Ubuntu/...?
> - From single server to multi server?
> - How to start with partitions and why?
> - Where to get latest releases (rpm/deb) when is not in my
>distriburtion (simon matter etc.)?

I think this may also be a problem: Where to get up to date packages?

In Debian they are still on Version 2.2.13. And Ubuntu probably also. And as 
Squeeze will be finished sooner or later, they will still have the 2.2.13 
packages in their repositories.

Having a repository with the latest 2.3 and then the 2.4 cyrus versions would 
be a good thing.

This helps in getting a working cyrus server for not only newbie users.

But I know that keeping a seperate repository is quite a lot of work.

Best regards,

Claus

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/

Re: competition

[Marc Patermann]

Hi,

Adam Tauno Williams schrieb am 21.09.2010 12:04 Uhr:
> On Tue, 2010-09-21 at 11:44 +0200, Marc Patermann wrote:
>> André Schild schrieb am 21.09.2010 10:40 Uhr:
>>> Am 21.09.2010 09:31, schrieb Pascal Gienger:

 I begin to be tired from this "dovecot is much more besser, you
  HAVE TO USE IT", why don't you migrate, ... ...?
>> Yes, I'm too. But this is what you see in forums and mailing list.
>> And my post is intended to see what the Cyrus community can do to
>> /explain/ the product any better.
> "Be there" is the only thing I can think of.  The only way to counter
>  advocates is to be one.
Yes, "Be there" - but "Be prepared" and have something to point to: "See
it is easy and all there, dude!"

>>> One important thing is the documentation of the imap server, and
>>>  there cyrus could offer more (Just my opinion)
>> The new web site is a good start. We should start a best practice
>> section in the wiki. - How do I install Cyrus on Debian/Ubuntu/...?
>> 
> 
> Ugh, please no.  Distro specific directions are lame, pointless, and 
> unmaintainable [beyond possibly packages-are-here].
Yes, you're right. But in many cases it just point to somewhere else,
which might be better than having nothing.
For Ubuntu there is a howto: http://wiki.ubuntuusers.de/Cyrus_IMAPD and
what do you get? 2.2.13 - uargh:
http://packages.ubuntu.com/lucid/cyrus-imapd-2.2
That may not be, what we want.

> Once the packages are installed setting up is essentially the same
> everywhere.
Yes. But "install" is always first. If you do not get the user to get it
installed, everything else is useless.

If you were a newbie and most people were trying to talk you away from
Cyrus and there is no deb or rpm, would you try to compile the software 
yourself?

>> - From single server to multi server? - How to start with
>> partitions and why? - Where to get latest releases (rpm/deb) when
>> is not in my distriburtion (simon matter etc.)? - Everyone is
>> talking about "Push-Mail" - how to do it with Cyrus?
> It just happens if you your client supports IMAP idle.
_I_ know that. :)
It is just one example what I think user could search for. And "Push 
mail" IMHO was one of the email buzz words of the last few years.

> K9 on the Android seems to work very well with Cyrus in the 'push'
> mode.  Although I've noticed users usually turn it off within a
> couple of weeks - really, nobody wants push-mail once they have it.
I disagree.

>> - What if I want "Groupware" later, can I still use Cyrus? (link to
>> projects supporting Cyrus, OX etc)
> Shameless self promotion -I have a chapter on Cyrus in WMOGAG 
> (OpenGroupware Administrator's Guide) 
>  
> I've gotten quite a few compliments on it.
You could put your "Shameless self promotion" in the wiki then. :)


Marc

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/

Re: competition

[J. Roeleveld]

On Tuesday 21 September 2010 11:44:45 Marc Patermann wrote:
> Hi,
> 
> André Schild schrieb am 21.09.2010 10:40 Uhr:
> > Am 21.09.2010 09:31, schrieb Pascal Gienger:



> > One important thing is the documentation of the imap server, and
> > there cyrus could offer more (Just my opinion)
> 
> The new web site is a good start.
> We should start a best practice section in the wiki.
> - How do I install Cyrus on Debian/Ubuntu/...?
> - From single server to multi server?
> - How to start with partitions and why?
> - Where to get latest releases (rpm/deb) when is not in my
>distriburtion (simon matter etc.)?
> - Everyone is talking about "Push-Mail" - how to do it with Cyrus?
> - What if I want "Groupware" later, can I still use Cyrus?
>(link to projects supporting Cyrus, OX etc)
> - I want to have virtual users in LDAP/RDBMS - how to do it?
> There may be much more basic task which may be well know to us but would
> be interesting to newbies to know.

Don't forget one of the most important details (which I think should be listed 
right after "how to install / configure"):
- How do I backup and restore my email?
(eg. what needs to be backed up and what can be "rebuild")

> Just my 2 Cent

And mine, makes 4 cents :)

--
Joost

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/

Re: Re: competition

[Syren Baran]

Am Dienstag, den 21.09.2010, 11:48 +0200 schrieb André Schild:
> Am 21.09.2010 11:35, schrieb Simon Matter:
> >> I don't know, where this bad karma is coming from - I'm still happy with
> > I guess it's simply because for many years there were no clean packages
> > for the most used operating systems.
> >
> Debian is still stuck on 2.2 and there seems to be no progress in that area.

Most people like to use versions from repositories. For obvious reasons.

Might make sense for cyrus to host their own repository. If it's just a
simple entry in sources.list(.d) more people would use the recent
version.

Greetings,
Syren Baran

+++

Niederlassung Deutschland:
BIT-HOUSE Ltd.
internet professionals
Erthalstr. 17
63739 Aschaffenburg
Deutschland

Amtsgericht:Aschaffenburg
HRB-Nr.:9136
St/Nr.: 204/104/60603
USt-Id-Nr.: DE814408164
Gechäftsführer: Thomas Witzel

+++

Telefon:+49.6021.8622-680
VoIP:   +49.6021.8622-680
Fax:+49.6021.8622-676
E-Mail: m...@bit-house.com
Homepage: www.bit-house.com

+++

Hauptsitz:
BIT-HOUSE Ltd.
69 Great Hampton Street
Birmingham
West Midlands
United Kingdom

Registergericht: Companies House of Cardiff
Registernummer:  05325636
Registereintrag: United Kingdom
Director: Thomas Witzel

+++
<>
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/

Re: competition

[Adam Tauno Williams]

On Tue, 2010-09-21 at 11:44 +0200, Marc Patermann wrote:
> Hi,
> André Schild schrieb am 21.09.2010 10:40 Uhr:
> > Am 21.09.2010 09:31, schrieb Pascal Gienger:
> >> I begin to be tired from this "dovecot is much more besser, you 
> >> HAVE TO USE IT", why don't you migrate, ... ...?
> Yes, I'm too. But this is what you see in forums and mailing list. And
> my post is intended to see what the Cyrus community can do to /explain/ 
> the product any better.

"Be there" is the only thing I can think of.  The only way to counter
advocates is to be one.

> > 2. Newbies, which do not know cyrus, they just ask for advice and a 
> > simple starter experience with the imap server. In the last month 
> > there has been much posts of type "Dovecot is the 
> > greates/simples/best/etc. " So the new users are very likely to go to
> >  dovecot instead of cyrus, because of "advices" from other internet 
> > users.
> This is my experience, too.

Yep.  Also, for some reason, many people have the opinion that Cyrus is
'arcane' in some way.  I honestly have no idea where that comes from
except perhaps from the dreadful state of the documentation.  But over
the years I've been able to fairly easily figure out what I need to
know.

> > One important thing is the documentation of the imap server, and 
> > there cyrus could offer more (Just my opinion)
> The new web site is a good start.
> We should start a best practice section in the wiki.
> - How do I install Cyrus on Debian/Ubuntu/...?

Ugh, please no.  Distro specific directions are lame, pointless, and
unmaintainable [beyond possibly packages-are-here].  Once the packages
are installed setting up is essentially the same everywhere.

> - From single server to multi server?
> - How to start with partitions and why?
> - Where to get latest releases (rpm/deb) when is not in my
>distriburtion (simon matter etc.)?
> - Everyone is talking about "Push-Mail" - how to do it with Cyrus?

It just happens if you your client supports IMAP idle.  K9 on the
Android seems to work very well with Cyrus in the 'push' mode.  Although
I've noticed users usually turn it off within a couple of weeks -
really, nobody wants push-mail once they have it.

> - What if I want "Groupware" later, can I still use Cyrus?
>(link to projects supporting Cyrus, OX etc)

Shameless self promotion -I have a chapter on Cyrus in WMOGAG
(OpenGroupware Administrator's Guide)

I've gotten quite a few compliments on it.

> - I want to have virtual users in LDAP/RDBMS - how to do it?
> There may be much more basic task which may be well know to us but would 
> be interesting to newbies to know.
> IMHO the "What is Cyrus?" page should be worked over. Why do we not talk 
> about who is using Cyrus IMAPd and how - "Success Stories" (fastmail.fm?)?

-- 
Adam Tauno Williams  LPIC-1, Novell CLA

OpenGroupware, Cyrus IMAPd, Postfix, OpenLDAP, Samba


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/

Re: competition

[André Schild]

  Am 21.09.2010 11:35, schrieb Simon Matter:
>> I don't know, where this bad karma is coming from - I'm still happy with
> I guess it's simply because for many years there were no clean packages
> for the most used operating systems.
>
Debian is still stuck on 2.2 and there seems to be no progress in that area.

The main problem they apparently have, is the migration path for the various
DB files from 2.2 to 2.3.
(The 2.3 version itself works fine as .deb packages)

André

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/

Re: competition

[Marc Patermann]

Hi,

André Schild schrieb am 21.09.2010 10:40 Uhr:
> Am 21.09.2010 09:31, schrieb Pascal Gienger:

>> I begin to be tired from this "dovecot is much more besser, you 
>> HAVE TO USE IT", why don't you migrate, ... ...?
Yes, I'm too. But this is what you see in forums and mailing list. And
my post is intended to see what the Cyrus community can do to /explain/ 
the product any better.

> 2. Newbies, which do not know cyrus, they just ask for advice and a 
> simple starter experience with the imap server. In the last month 
> there has been much posts of type "Dovecot is the 
> greates/simples/best/etc. " So the new users are very likely to go to
>  dovecot instead of cyrus, because of "advices" from other internet 
> users.
This is my experience, too.

> One important thing is the documentation of the imap server, and 
> there cyrus could offer more (Just my opinion)
The new web site is a good start.
We should start a best practice section in the wiki.
- How do I install Cyrus on Debian/Ubuntu/...?
- From single server to multi server?
- How to start with partitions and why?
- Where to get latest releases (rpm/deb) when is not in my
   distriburtion (simon matter etc.)?
- Everyone is talking about "Push-Mail" - how to do it with Cyrus?
- What if I want "Groupware" later, can I still use Cyrus?
   (link to projects supporting Cyrus, OX etc)
- I want to have virtual users in LDAP/RDBMS - how to do it?
There may be much more basic task which may be well know to us but would 
be interesting to newbies to know.

IMHO the "What is Cyrus?" page should be worked over. Why do we not talk 
about who is using Cyrus IMAPd and how - "Success Stories" (fastmail.fm?)?


Just my 2 Cent


Marc

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/

Re: competition

[Simon Matter]

> I don't know, where this bad karma is coming from - I'm still happy with

I guess it's simply because for many years there were no clean packages
for the most used operating systems.

Simon


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/

RE: competition

[Michel Sébastien]

>> where does Cyrus IMAPd stand today?
>> When I was starting to think about moving to a open source mail system
>> (migrating away from Lotus Domino btw.), there ware Cyrus IMAPd, Courier
>> and UW-IMAP I think.
>> Cyrus was the only "full flavored" IMAP server with active development.
>> We were going the 2.2. path, while 2.3 seemed to "fresh". So there was
>> development.
>> On the other side there were still many people complaining about Cyrus
>> being too complex and too unstable with all the BDB fiddlings.
>>
>> Then dovecot emerged and quickly evolved. I don't know why,

> I begin to be tired from this "dovecot is much more besser, you HAVE TO USE 
> IT", why don't you migrate, ... ...?
>
> For me, this is a typical open source crowd behaviour. If one product tends 
> to have better results, change immediately on the faster running > train 
> (which is believed to run faster) and to dump the reliable solution 
> immediately, beginning to do advocacy in all forums, newsgroups and > 
> discussion groups to tell people that the new product is even better than 
> sliced bread.

Personally I like the competition, it encourage the continuous improvement of 
open source software. Dovecot has some good ideas or design solutions (see this 
rackspace document about Dovecot. It indicates that Apple did the switch from 
Cyrus to Dovecot  http://www.dovecot.org/talks/macsysadmin-2009-09-16.pdf )

We run Cyrus for years and are happy with it. We host nearly 30 million 
mailboxes on our biggest platform. But begin to have mailboxes with hundreds of 
thousands of emails and it becomes difficult to balance the storage. I don't 
know what is really sharding in dovecot, but I would like to have a virtual 
storage in Cyrus (emails localized in an IMAP folder stored on several storage 
nodes).

Sébastien


Ce message et les pièces jointes sont confidentiels et réservés à l'usage 
exclusif de ses destinataires. Il peut également être protégé par le secret 
professionnel. Si vous recevez ce message par erreur, merci d'en avertir 
immédiatement l'expéditeur et de le détruire. L'intégrité du message ne pouvant 
être assurée sur Internet, la responsabilité du groupe Atos Origin ne pourra 
être recherchée quant au contenu de ce message. Bien que les meilleurs efforts 
soient faits pour maintenir cette transmission exempte de tout virus, 
l'expéditeur ne donne aucune garantie à cet égard et sa responsabilité ne 
saurait être recherchée pour tout dommage résultant d'un virus transmis.

This e-mail and the documents attached are confidential and intended solely for 
the addressee; it may also be privileged. If you receive this e-mail in error, 
please notify the sender immediately and destroy it. As its integrity cannot be 
secured on the Internet, the Atos Origin group liability cannot be triggered 
for the message content. Although the sender endeavours to maintain a computer 
virus-free network, the sender does not warrant that this transmission is 
virus-free and will not be liable for any damages resulting from any virus 
transmitted.


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/

Re: competition

[André Schild]

Am 21.09.2010 09:31, schrieb Pascal Gienger:
> I begin to be tired from this "dovecot is much more besser, you HAVE TO USE 
> IT", why don't you migrate, ... ...?

I think we have two different cases to consider:

1. Experienced users, running a cyrus installation for several months/years.
 Here I see no "danger" that they switch over to another imap 
project, unless
 that project provides better "overall experience" for the them.
 Running cyrus since ~8 years now, we don't intend to switch to 
another project.

2. Newbies, which do not know cyrus, they just ask for advice and a 
simple starter
 experience with the imap server.
 In the last month there has been much posts of type "Dovecot is the 
greates/simples/best/etc. "
 So the new users are very likely to go to dovecot instead of cyrus, 
because of "advices"
 from other internet users.

So I think we should take care, that cyrus is not loosing new users, 
because the
userbase will shrink over the years, compared to dovecot.

One important thing is the documentation of the imap server,
and there cyrus could offer more (Just my opinion)

Also the SASL stuff is not first-user friendly (But very powerfull when 
understood)
So helping new users in this area would probably also help for getting 
cyrus "recommendations" for new users.


André


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/

Re: competition

[Pascal Gienger]

Le 20 sept. 2010 à 15:59, Marc Patermann a écrit :

> Hi,
> 
> where does Cyrus IMAPd stand today?
> When I was starting to think about moving to a open source mail system 
> (migrating away from Lotus Domino btw.), there ware Cyrus IMAPd, Courier 
> and UW-IMAP I think.
> Cyrus was the only "full flavored" IMAP server with active development. 
> We were going the 2.2. path, while 2.3 seemed to "fresh". So there was 
> development.
> On the other side there were still many people complaining about Cyrus 
> being too complex and too unstable with all the BDB fiddlings.
> 
> Then dovecot emerged and quickly evolved. I don't know why,

I begin to be tired from this "dovecot is much more besser, you HAVE TO USE 
IT", why don't you migrate, ... ...?

For me, this is a typical open source crowd behaviour. If one product tends to 
have better results, change immediately on the faster running train (which is 
believed to run faster) and to dump the reliable solution immediately, 
beginning to do advocacy in all forums, newsgroups and discussion groups to 
tell people that the new product is even better than sliced bread.

Today it is dovecot, tomorrow it can be .

This kind of advocacy kills capacities, kills work time (because you have to 
tell to your boss why it is not a good idea to dump the existing installation 
only because another linux guy plotted out some coloured powerpoint slides 
stating that dovecot is the way to go) and focusses on the wrong side of the 
story.

Cyrus has been proven to be reliable for over ten years. The security record 
for Cyrus is quite high. It runs perfectly on our infrastructure. It is still 
actively developed. Until now, it does not hurt any performance barrier and 
staff is trained with it.
Just to change because numbers seem to be better or an IMHO arrogant founder 
offers "1000 euros if you can hack my server" (which is like being in a 
kindergarden from my point of view) is not a viable option. Image the cost of 
training staff to a new system. And even a "easy migration path" often turns 
out to get downtime. Downtime - for nothing you get "in plus" - no service will 
be "better" - it is far worse: as your staff isn't experienced with dovecot you 
likely will do more errors in administering ist.

As for a new installation, that can be another case. 

We are very satisfied with the performance and flexibility Cyrus IMAP gives to 
us. There's no need to change apart from being "in head with the open source 
advocacy croud".

-- 
Pascal Gienger Jabber/XMPP/Mail: pascal.gien...@uni-konstanz.de
University of Konstanz, IT Services Department ("Rechenzentrum")
Electronic Communications and Web Services
Building V, Room V404, Phone +49 7531 88 5048, Fax +49 7531 88 3739

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/

Re: competition

[David Lang]

On Mon, 20 Sep 2010, Andrew Morgan wrote:

> On Mon, 20 Sep 2010, Vincent Fox wrote:
>
>> Umm, what?  We run Cyrus IMAP server with no Murder
>> for 20K+ people.  Murder may be a feature but it's not a
>> deployment requirement.
>>
>> We used Perdition, originally just thrown up to provide a
>> transparent bridge as we migrated from Uwash to Cyrus.
>> But as things moved along, we ended up with Perdition
>> and multiple Cyrus backends and no good reason to switch
>> over to a Murder.
>
> I end up granting myself rights to various users' mailboxes to investigate
> when we see one of our users sending out spam.  It usually turns out that
> they have been phished recently.  Once I grant myself rights to their
> mailbox, I see the mailbox in my regular email client (Alpine) in the
> "Other Users" hierarchy.
>
> Does this work without Murder?

it definantly works with a single back-end. In the case of a multi-server 
setup, 
it depends on if your murder substatute will show you the folders across 
back-ends or not in a listing.

you can always connect directly to the back-end (bypassing murder or it's 
replacement) to see all "other users" for that back-end.

David Lang

>  How do you investigate users' mailboxes in
> cases like this?
>
>   Andy
> 
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
>

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/

Re: competition

[Vincent Fox]

  On 09/20/2010 04:23 PM, Andrew Morgan wrote:
> I end up granting myself rights to various users' mailboxes to 
> investigate when we see one of our users sending out spam.  It usually 
> turns out that they have been phished recently.  Once I grant myself 
> rights to their mailbox, I see the mailbox in my regular email client 
> (Alpine) in the "Other Users" hierarchy.
>
> Does this work without Murder?  How do you investigate users' 
> mailboxes in cases like this?
>

I just login to the backend and look at their mailbox directly.  
Admittedly clunkier but the number of times where I've had to poke 
directly is small.





Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/

Re: competition

[Andrew Morgan]

On Mon, 20 Sep 2010, Vincent Fox wrote:

> Umm, what?  We run Cyrus IMAP server with no Murder
> for 20K+ people.  Murder may be a feature but it's not a
> deployment requirement.
>
> We used Perdition, originally just thrown up to provide a
> transparent bridge as we migrated from Uwash to Cyrus.
> But as things moved along, we ended up with Perdition
> and multiple Cyrus backends and no good reason to switch
> over to a Murder.

I end up granting myself rights to various users' mailboxes to investigate 
when we see one of our users sending out spam.  It usually turns out that 
they have been phished recently.  Once I grant myself rights to their 
mailbox, I see the mailbox in my regular email client (Alpine) in the 
"Other Users" hierarchy.

Does this work without Murder?  How do you investigate users' mailboxes in 
cases like this?

Andy

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/

Re: competition

[Vincent Fox]

  On 09/20/2010 06:59 AM, Marc Patermann wrote:
> But where does Cyrus IMAPd stand today?
> It may be Murder/Aggregator - but how to get the people, when on first
> contact, where they just need a simple IMAP server, they are pointed to
> other product, which they then stay with?
Umm, what?  We run Cyrus IMAP server with no Murder
for 20K+ people.  Murder may be a feature but it's not a
deployment requirement.

We used Perdition, originally just thrown up to provide a
transparent bridge as we migrated from Uwash to Cyrus.
But as things moved along, we ended up with Perdition
and multiple Cyrus backends and no good reason to switch
over to a Murder.



Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/