Re: [Dovecot] mail spool filesystem

[Seth Mattinen]

On 8/17/11 7:42 AM, Adrian Ulrich wrote:
>> I read that XFS is a good choice, but is not 
>> too reliable...
> 
> Are you using Maildir or MBOX?
> 
> In any case: XFS would be my last choice:
> 
> XFS is nice if you are working with large files (> 2GB), but
> for E-Mail i'd stick with ext3 (or maybe even reiser3)
> as it works very well with small files.
> 

I'd have to disagree. This is completely anecdotal, but I originally
deployed ext3 on all of my mail servers (Dovecot maildir) and spools
(Postfix) until they started exhibiting loading issues when busy.
Reformatting into XFS resolved the problem with no other changes. I
didn't have time to do any comparisons or gather statistics since it was
an emergency situation and this was before ext4, but XFS has performed
flawlessly for me.

~Seth

Re: [Dovecot] Question about "slow" storage but fast cpus, plenty of ram and dovecot

[Seth Mattinen]

On 12/12/2010 00:49, Stan Hoeppner wrote:
> 
> Since Javier is looking for ways to decrease I/O load on the SAN, not
> necessarily increase Dovecot performance, I think putting the index
> files on a ramdisk is best thing to try first.  It may not be a silver
> bullet.  If he's still got spare memory to add to this guest, doing both
> would be better.   Using a ramdisk for the index files will instantly
> remove all index I/O from the SAN.  More of Dovecot's IMAP I/O is to the
> index files than mail files isn't it?  So by moving the index files to
> ramdisk you should pretty much instantly remove half your SAN I/O load.
>  This is assuming that Javier currently stores his index files on a SAN LUN.
> 


Speaking of ramdisk/SSD, has anyone tried a PCIe SSD for indexes?

~Seth

Re: [Dovecot] nfs director

[Seth Mattinen]

On 8/31/2010 10:22, Ariel Biener wrote:
> 
> Oh, and Timo, I don't think we are just "a couple of NFS users". Maildir
> and NFS are not as uncommon as
> you'd think, even in very large installations.
> 


NFS with maildir has been the gold standard for a long time, whether the
NFS server be  boxes or a huge hardware
appliance. It's certainly not uncommon.

~Seth

Re: [Dovecot] Maildir over NFS

[Seth Mattinen]

On 8/9/10 5:31 AM, Bernd Petrovitsch wrote:
> On Mon, 2010-08-09 at 12:18 +0200, Edgar Fuß wrote:
>>> The NFS server sits in user space.
>> Oops? I don't know what Linux does, but with BSD, it has always been 
>> in-kernel.
> 
> Historically there was a user-space NFS-daemon (and can very probably be
> found via Google today).
> Actually there are stories about people using it because if you export
> filesystems via a user-space NFS daemon, you can change the mounting
> below without affecting the clients for NFS-v3.
> 
> But the kernel has a NFS-server since years and all (somewhat common)
> distributions use it per default.
> 


Debian still gives you a choice with packages nfs-user-server and
nfs-kernel-server.

~Seth

Re: [Dovecot] Maildir over NFS

[Seth Mattinen]

On 8/6/2010 12:31, CJ Keist wrote:
> 
> So question I have for the dovecot team, does running Maildir over NFS
> work well?  Or would you recommend that all user mail folders be stored
> locally on the mail server when using Maildir?
> 
> We have about 3400 users, doing about 30k mail deliveries daily.  Some
> users have 10's of thousands of mail messages in hundreds of mail folders.
> 

Maildir (and Dovecot in general) will work fine with NFS.

http://wiki.dovecot.org/NFS

~Seth

Re: [Dovecot] dovecot evaluation on a 30 gb mailbox

[Seth Mattinen]

On 6/23/2010 15:41, Stan Hoeppner wrote:
> Seth Mattinen put forth on 6/23/2010 4:30 PM:
> 
>> When using SquirrelMail, make sure these settings are 'true':
>>
>> Allow server thread sort : true
>> Allow server-side sorting: true
>>
>> Otherwise performance will suffer because it will do these things itself
>> rather than letting the server do them.
> 
> Is this still true if both SM and Dovecot are on the same box?  I.e. is this a
> CPU bandwidth issue or a network bandwidth issue?  If these settings solve a
> network b/w issue then they're irrelevant when both SM and Dovecot are on the
> same host.
> 


Found it in the docs:

http://squirrelmail.org/docs/admin/admin-6.html#ss6.3

I would very strongly suggest that IMAP SORT be enabled if the server
supports it, which of course Dovecot does. IMAP THREAD is the other one.

~Seth

Re: [Dovecot] dovecot evaluation on a 30 gb mailbox

[Seth Mattinen]

On 6/23/2010 15:41, Stan Hoeppner wrote:
> Seth Mattinen put forth on 6/23/2010 4:30 PM:
> 
>> When using SquirrelMail, make sure these settings are 'true':
>>
>> Allow server thread sort : true
>> Allow server-side sorting: true
>>
>> Otherwise performance will suffer because it will do these things itself
>> rather than letting the server do them.
> 
> Is this still true if both SM and Dovecot are on the same box?  I.e. is this a
> CPU bandwidth issue or a network bandwidth issue?  If these settings solve a
> network b/w issue then they're irrelevant when both SM and Dovecot are on the
> same host.
> 


It's more along the lines of SM will not use commands like SORT and
attempt to do things itself because your server does not implement it.
For example, I believe SurgeMail does not implement IMAP SORT so you'd
have to set it to 'false'. Dovecot does, and is much better at executing
SORT than having squirrel simulate it.

~Seth

Re: [Dovecot] dovecot evaluation on a 30 gb mailbox

[Seth Mattinen]

On 6/23/2010 12:27, Stan Hoeppner wrote:
> A.L.E.C put forth on 6/23/2010 6:51 AM:
>> On 23.06.2010 13:22, Rajesh M wrote:
>>
>>> did the tests once again on this large email box
>>
>> First of all, are you using one client app or many? Make sure it's not
>> the client issue. You should also enable some debug to see what commands
>> are sent to IMAP server and what is the response and what is response
>> time. For example, there's imap_debug option in Roundcube.
> 
> I think he's actually using Squirrelmail guys.  I previously mentioned
> performance I get with Roundcube+Dovecot+mbox strictly as a comparison.
> That's how RC sneaked into the thread.
> 


When using SquirrelMail, make sure these settings are 'true':

Allow server thread sort : true
Allow server-side sorting: true

Otherwise performance will suffer because it will do these things itself
rather than letting the server do them.

~Seth

Re: [Dovecot] Sieve GUI

[Seth Mattinen]

On 6/1/10 10:24 AM, Phil Howard wrote:
> On Tue, Jun 1, 2010 at 13:07, Gerhard Waldemair  wrote:
>>> Is there à GUI for sieve.
>>>
>>> I don't have installed Squrielmail, but won't like to manage the Rules per
>>> ssh
> 
> I would hope there would be a way to put Sieve scripts in place via
> IMAP itself.  Otherwise I won't be able to do that on my server (since
> there won't be system user accounts for mail users ... it's "one
> system user" virtual here).


One word: managesieve

~Seth

Re: [Dovecot] Could use some help setting Dovecot up for the first time

[Seth Mattinen]

On 3/7/2010 12:35, joe wrote:
> That's what i thought too but for the life of me i cannot figure out
> what is causing it. I figured i must have missed something along the
> way. I'm on my 5th or so time installing everything (within a jail), so
> hopefully i get it right this time!
> 
> Wish me luck, and if anyone out there might have an idea of how to
> resolve this issue (it happens everytime i install everything) i would
> love to hear it!
> 

Are you running postfix in chroot? If you are, I'd suggest turning it off.

~Seth

Re: [Dovecot] Anyone successfully setup Continous Backup of mailboxes using rsync ?

[Seth Mattinen]

On 1/28/10 12:07 AM, CoolAtt NNA wrote:
> 
> Hi All..
> 
> Am working on continous backup of mailboxes using rsync(for e.g by running 
> rsync every 2 min)
> 
> Things gets more complicated when users create Subfolders in INBOX , SENT , 
> etc..

How?


> If anyone among you did that plz guide & advise.
> 

Sure:

rsync -e 'ssh -ax -i mykey' \
-a --delete --stats \
mailserver.host:/var/mail/vmail \
/localbackup


~Seth

Re: [Dovecot] Released Pigeonhole Sieve v0.1.14 and ManageSieve v0.11.10 for Dovecot v1.2.9

[Seth Mattinen]

Stephan Bosch wrote:
> Seth Mattinen wrote:
>> Stephan Bosch wrote:
>>> Hello Dovecot users,
>>>
>>> Due to personal matters I haven't had time to do anything in the last
>>> few weeks. Before that I provided some pre-releases and since nobody
>>> indicated problems with those, I release them definitively now. I hope
>>> the shit storm has blown over, so that I can dedicate more of my time
>>> towards Pigeonhole development.
>>>
>>> The Sieve release involves a large architectural change, but also fixes
>>> quite a few small problems. The ManageSieve release is necessary to
>>> match Sieve API changes.
>>>
>>
>> Is there any plan to add quota enforcement to managesieve?
>>
> Yes, it's on the TODO list.
> 

Yes, it's been on the TODO for a long time, but how much of a priority
is it?

~Seth

Re: [Dovecot] Released Pigeonhole Sieve v0.1.14 and ManageSieve v0.11.10 for Dovecot v1.2.9

[Seth Mattinen]

Stephan Bosch wrote:
> Hello Dovecot users,
> 
> Due to personal matters I haven't had time to do anything in the last
> few weeks. Before that I provided some pre-releases and since nobody
> indicated problems with those, I release them definitively now. I hope
> the shit storm has blown over, so that I can dedicate more of my time
> towards Pigeonhole development.
> 
> The Sieve release involves a large architectural change, but also fixes
> quite a few small problems. The ManageSieve release is necessary to
> match Sieve API changes.
> 

Is there any plan to add quota enforcement to managesieve?

~Seth

Re: [Dovecot] Vacation message with Sieve

[Seth Mattinen]

Rene Bakkum wrote:


I don't like catch-all addresses either, and I have almost never used 
them myself. But our customers seems to like them sometimes, so I can't 
ignore their wishes.


You can't do it with Sieve 'vacation'. Listing the exact address is 
required.


~Seth

Re: [Dovecot] [OT] DRBD

[Seth Mattinen]

Rodolfo Gonzalez Gonzalez wrote:

Hello all,

has someone worked with DRBD (http://www.drbd.org) for HA of mail 
storage? if so, does it have stability issues? comments and experiences 
are thanked :)




Look for thread "Dovecot + DRBD/GFS mailstore" starting 6/5/2009 on this 
list.


~Seth

Re: [Dovecot] [OT] preferred clients

[Seth Mattinen]

Thomas Harold wrote:
> On 11/20/2009 1:51 PM, Thomas Berezansky wrote:
>> Personally, I am using Horde (http://www.horde.org/) at work (this
>> address) and Outlook 2007 at home (largely due to getting freaky
>> attachments I need to open on a regular basis that only Microsoft mail
>> clients seem to be able to open properly).
> 
> Out of morbid curiosity... how good is Outlook 2007 at IMAP?  I've had
> experience with Outlook 2003 and the 2GB PST limit was a deal breaker
> for me.  I'm curious whether Outlook is getting better or worse at IMAP.

They finally added the ability to set a "sent items" folder on the IMAP
server rather than using stupid tricks to copy it over, so that's a
plus. The fact that it can't "delete" by moving a copy to a "deleted
items" folder still bugs me enough that I won't use it. No idea about
the PST size thing though, never got that far.

~Seth

Re: [Dovecot] Web-Interface for Dovecot-Sieve?

[Seth Mattinen]

Steffen Kaiser wrote:
> On Wed, 18 Nov 2009, Seth Mattinen wrote:
> 
>>>>> is there anywhere a web-interface for managing sieve-filters with
>>>>> dovecot?
>>>
>>>> Beware that dovecot managesieve does not have any kind of security to
>>>> prevent abuse if you open it to the outside world.
>>>
>>> Huh?
>>> It has the same security as Dovecot itself: authentification with TLS.
> 
>> The last time I checked dovecot managesieve has a denial of service
>> potential of no limit to how much disk space it will let sieve consume.
> 
> OK, but this is not related to "outside", you need a password to fill
> the space and take the system down.
> 

So? That doesn't mean every logged in connection will be well behaved.

Even a well meaning user could use a managesieve tool with a bug that
brings your server down. Until dovecot managesieve figures out how to
add some very basic DOS protection I wouldn't open it up to end users. I
haven't looked at the code (too busy) but i can't imagine it would be an
impossible task to add a fixed size per script (i.e. a couple megs) and
maximum number of allowed scripts (like 50).

~Seth

Re: [Dovecot] Web-Interface for Dovecot-Sieve?

[Seth Mattinen]

Steffen Kaiser wrote:
> On Tue, 17 Nov 2009, Seth Mattinen wrote:
> 
>>> is there anywhere a web-interface for managing sieve-filters with
>>> dovecot?
> 
>> Beware that dovecot managesieve does not have any kind of security to
>> prevent abuse if you open it to the outside world.
> 
> Huh?
> It has the same security as Dovecot itself: authentification with TLS.
> 


The last time I checked dovecot managesieve has a denial of service
potential of no limit to how much disk space it will let sieve consume.

~Seth

Re: [Dovecot] Web-Interface for Dovecot-Sieve?

[Seth Mattinen]

Andrew Schulman wrote:
> 
> Someone please correct me if I'm wrong, but IIRC the managesieve interfaces
> of Roundcube and Thunderbird only allow you to choose between existing
> sieve scripts that are already available through the managesieve server.
> They *don't* provide an interface for constructing sieve scripts in the
> first place-- you still have to write them by hand.
> 

That's correct for the Thunderbird plugin; it won't help you write them.
I don't know about roundcube.

~Seth

Re: [Dovecot] plugin cmusieve not found

[Seth Mattinen]

juan carlos wrote:
> # ls -l /usr/lib/dovecot/modules/lda
> lrwxrwxrwx 1 root root 22 nov 12 21:20 lib01_acl_plugin.so
> lrwxrwxrwx 1 root root 24 nov 12 21:20 lib10_quota_plugin.so
> lrwxrwxrwx 1 root root 24 nov 12 21:20 lib11_trash_plugin.so
> lrwxrwxrwx 1 root root 29 nov 12 21:20 lib20_autocreate_plugin.so
> lrwxrwxrwx 1 root root 26 nov 12 21:20 lib20_convert_plugin.so
> lrwxrwxrwx 1 root root 25 nov 12 21:20 lib20_expire_plugin.so
> lrwxrwxrwx 1 root root 22 nov 12 21:20 lib20_fts_plugin.so
> lrwxrwxrwx 1 root root 29 nov 12 21:20 lib20_listescape_plugin.so
> lrwxrwxrwx 1 root root 27 nov 12 21:20 lib20_mail_log_plugin.so
> lrwxrwxrwx 1 root root 26 nov 12 21:20 lib20_virtual_plugin.so
> lrwxrwxrwx 1 root root 28 nov 12 21:20 lib21_fts_squat_plugin.so
> -rw-r--r-- 1 root root 3663976 nov 15 23:04 lib90_sieve_plugin.a
> -rwxr-xr-x 1 root root1029 nov 15 23:04 lib90_sieve_plugin.la
> -rwxr-xr-x 1 root root 1777465 nov 15 23:04 lib90_sieve_plugin.so
> 

It's right there at the bottom of your list.

~Seth

Re: [Dovecot] Web-Interface for Dovecot-Sieve?

[Seth Mattinen]

Daniel Spannbauer wrote:
> Hello,
> 
> is there anywhere a web-interface for managing sieve-filters with dovecot?
> 

Beware that dovecot managesieve does not have any kind of security to
prevent abuse if you open it to the outside world.

~Seth

Re: [Dovecot] Problem with stored procedure

[Seth Mattinen]

Fcuk F wrote:

Hi  Timo/all

I am trying to use stored procedure for password query when it try to
connect i get authenticated only for the first user after that for
subsequent user it gives me this error.

Password query failed: Commands out of sync; you can't run this command now



That message is from MySQL. If your stored procedure returns more than 
one dataset you'll get that error.


~Seth

Re: [Dovecot] HA Dovecot Config?

[Seth Mattinen]

Eric Rostetter wrote:
> 
> For IMAP to be truely HA, you will need shared storage of some sort.
> 
> 
> You can "emulate" a SAN with something like DRBD if budget doesn't allow a
> real SAN (that is what I do).
> 

I can vouch for DRBD too. It works quite well.

~Seth

Re: [Dovecot] simple steps with sieve

[Seth Mattinen]

Gavin Hamill wrote:
> On Mon, 2009-10-19 at 17:21 +0200, Pascal Volk wrote:
> 
>> You are missing the 'addresses:' list in your vacation script. See
>> http://wiki.dovecot.org/LDA/Sieve#Vacation_auto-reply-1 for more details.
> 
> Interesting; thank you.
> 
> I can confirm that it works fine when I specify the exact address -
> however is it possible to use a wildcard, and further, assume the
> wildcard implicitly? 


It's not supposed to blindly autorespond to everything that crosses its
path, i.e. mailing lists not directly addressed to you. Thus you have to
specify the list of addresses.

~Seth

Re: [Dovecot] Binary locations

[Seth Mattinen]

Ed W wrote:
> Timo Sirainen wrote:
>> Where do you think the following binaries should be installed to? The
>> possible locations are:
>>
>>  - bin/
>>  - sbin/
>>   
> 
> Should these not be /usr/bin and /usr/sbin ?
> 
> I believe that /bin and /sbin are for system essentials and /usr/bin for
> package installed binaries?
> 

Timo did not put a leading / in there - I assume that means add your
$PREFIX to those paths.

~Seth

Re: [Dovecot] Question about the pop3 feature "leave messages on server for a certain period of time"

[Seth Mattinen]

Δημήτριος Καραπιπέρης wrote:
> O/H Timo Sirainen έγραψε:
>> On Fri, 2009-09-11 at 11:57 +0300, Δημήτριος Καραπιπέρης wrote:
>>  
>>> I am missing something on the the pop3 "leave messages" rationale.
>>>
>>> Although the UIDL feature solves for the MUA the problem  "whch mails
>>> should be downloaded",
>>> how the duration that these mails should be kept on server, say 10
>>> days for one MUA and 20 days for another MUA for the same account, is
>>> resolved on the server?
>>> 
>>
>> I guess you could configure Dovecot to delete mails older than n days
>> with expire plugin: http://wiki.dovecot.org/Plugins/Expire
>>
>> But I've no idea if it's really a good idea to do that. Some clients
>> might freak out. Oh and expire plugin would delete messages regardless
>> of whether client has ever even downloaded them.
>>
>>   
> Hi there
> 
> Definitely imap4 is more sophisticated protocol offering many useful
> features. By no means did I try to compare pop3 versus imap.
> I am using imap , I have imap servers configured and I suggest to
> everyone to use imap.
> I just had this question regarding multiple MUA's configuration on the
> option "leave messages on server" on the same account, and if there
> was an option to cease the mess that different time period setting (on
> DELE command) can cause..
> 

No. Whichever deletes first deletes the message for good.

~Seth

Re: [Dovecot] log for each user

[Seth Mattinen]

Thomas G. Lau wrote:
> Dear Pascal,
> 
> rawlog does not include timestamp on the log, how is it possible to be
> done, also, is it possible to combine in, out log into one? so that we
> could see the whole picture
> 

Syslog.

~Seth

Re: [Dovecot] Scalability plans: Abstract out filesystem and make it someone else's problem

[Seth Mattinen]

Ed W wrote:
> 
> Actually I use maildir, but apart from delete performance which is
> usually rare, mailbox seems better for nearly all use patterns
> 
> Seems like if it were possible to "solve" delete performance then
> mailbox becomes the preferred choice for many requirements (also lets
> solve the backup problem where the whole file changes every day)
> 


I think dbox's hybrid mbox/maildir approach would combine the best of
both, but I haven't looked at dbox progress lately.

~Seth

Re: [Dovecot] Dovecot Versions and Debian

[Seth Mattinen]

Mario Antonio wrote:

Seth,

How do you deal with new important patches?

Do you patch the source and then rebuild the packages? Is it safe or 
better just wait until SID release the new source?




I just wait for it to show up in sid. You could also apply patches 
yourself and rebuild, either way works.


~Seth

Re: [Dovecot] Scalability plans: Abstract out filesystem and make it someone else's problem

[Seth Mattinen]

Eric Jon Rostetter wrote:
> Quoting Seth Mattinen :
> 
>> Queue directories and clusters don't
>> mix well, but a read-heavy maildir/dbox environment shouldn't suffer the
>> same problem.
> 
> Why don't queue directories and clusters mix well?  Is this a performance
> issue only, or something worse?
> 

It depends on the locking scheme used by the filesystem. Working queue
directories (the ones where stuff comes and goes rapidly) is best suited
for a local FS anyway.

~Seth

Re: [Dovecot] Scalability plans: Abstract out filesystem and make it someone else's problem

[Seth Mattinen]

Timo Sirainen wrote:
> On Aug 11, 2009, at 2:16 AM, Seth Mattinen wrote:
> 
>>> Show me a clustered filesystem that can guarantee that each file is
>>> stored in at least 3 different data centers and can scale linearly by
>>> simply adding more servers (let's say at least up to thousands).
>>
>> Easy, AFS. It is known to support tens of thousands of clients [1] and
>> it's not exactly new. Like supporting the quirks of NFS, the quirks of a
>> clustered filesystem could be found and dealt with, too.
> 
> I was more thinking about thousands of servers, not clients. Each server
> should contribute to the amount of storage you have. Buying huge
> storages is more expensive. Also it would be nice if you could just keep
> plugging in more servers to get more storage space, disk I/O and CPU and
> the system would just automatically reconfigure itself to take advantage
> of those. I can't really see any of that happening easily with AFS.

While that would be fancy, I don't think that level of integration would
be compatible with abstracting the filesystem per the original plan, so
I didn't consider it. I just considered robust, site independent,
scalable storage as you asked for. ;)

OpenAFS is worth a read, at least, to see what it offers and ideas you
could incorporate.
http://www.dementia.org/twiki/bin/view/AFSLore/GeneralFAQ

It focuses on "users" but you can pretend a user is really "server
running Dovecot". AFS also uses Kerberos. That alone would probably
disqualify its use for the purposes of simple Dovecot replication. I
picked on AFS because it closely matches what you were looking for in scale.


>> Key/value databases are hardly a magic bullet for redundancy. You don't
>> get 3 copies in different datacenters by simply switching to a
>> database-style storage.
> 
> Some (several?) of them can be somewhat easily configured to support
> that. (That's what their web pages say, anyway.)

Well, so can a global filesystem designed to do precisely that at the
block level. No advantage here.


>>> Clustered filesystems are also complex. They're much more complex than
>>> what Dovecot really requires.
>>
>> I mention it because you stated wanting to outsource the storage
>> portion. The complexity of whatever database engine you choose or
>> supporting a clustered filesystem (like NFS) is a wash since you're not
>> maintaining either one personally.
> 
> I also want something that's cheap and easy to scale. Sure, people who
> already have NFS/AFS/etc. systems can keep using Dovecot with the
> filesystem backends, but I don't think it's the cheapest or easiest
> choice. There's a reason why e.g. Amazon S3 isn't running on top of them.

S3 isn't really a fair comparison. There's Google FS too, but they're
both purpose built systems.

Now, keep in mind, I have not personally used AFS with Dovecot. My point
is to not dismiss building on a clustered file system just because it's
old and lacks sex appeal compared to the backend that Facebook uses.
UUCP is ancient too, but it still blows away stupid SMTP tricks many
people see as modern for disconnected endpoints.

~Seth

Re: [Dovecot] Scalability plans: Abstract out filesystem and make it someone else's problem

[Seth Mattinen]

Robert Schetterer wrote:
> Timo Sirainen schrieb:
>> On Aug 11, 2009, at 12:41 AM, Seth Mattinen wrote:
>>
>>>> Nothing forces you to switch from maildir, if you're happy with it :)
>>>> But if you want to support millions of users, it's simpler to distribute
>>>> the storage and disk I/O evenly across hundreds of servers using a
>>>> database that was designed for it. And by databases I mean here some of
>>>> those key/value-like databases, not SQL. (What's a good collective name
>>>> for those dbs anyway? BASE and NoSQL are a couple names I've seen.)
>>>>
>>>
>>> Why is a database a better choice than a clustered filesystem?
>> Show me a clustered filesystem that can guarantee that each file is
>> stored in at least 3 different data centers and can scale linearly by
>> simply adding more servers (let's say at least up to thousands).
>>
>> Clustered filesystems are also complex. They're much more complex than
>> what Dovecot really requires.
>>
> 
> i like the idea of sql based mail services
> whatever your  choice is, use of cluster file systems stays ever,
> but with databased setups it should much more easy to
> have redudant mailstores, i have all possible stuff quota, acl etc in a
> database yet, incl spamassassin, greylisting, webmail the only thing
> which is left ,is the mail store, it would be great if there would be
> the possibility to have that, if there are no big disadvantages
> like poor performance etc with it
> 
> there is http://www.dbmail.org/
> has sombody ever used it ?
> so it can be compared


It wouldn't be an SQL database - it's not really suitable for this kind
of thing at the scale Timo is proposing.

~Seth

Re: [Dovecot] Scalability plans: Abstract out filesystem and make it someone else's problem

[Seth Mattinen]

Timo Sirainen wrote:
> On Aug 11, 2009, at 12:41 AM, Seth Mattinen wrote:
> 
>>> Nothing forces you to switch from maildir, if you're happy with it :)
>>> But if you want to support millions of users, it's simpler to distribute
>>> the storage and disk I/O evenly across hundreds of servers using a
>>> database that was designed for it. And by databases I mean here some of
>>> those key/value-like databases, not SQL. (What's a good collective name
>>> for those dbs anyway? BASE and NoSQL are a couple names I've seen.)
>>>
>>
>>
>> Why is a database a better choice than a clustered filesystem?
> 
> Show me a clustered filesystem that can guarantee that each file is
> stored in at least 3 different data centers and can scale linearly by
> simply adding more servers (let's say at least up to thousands).

Easy, AFS. It is known to support tens of thousands of clients [1] and
it's not exactly new. Like supporting the quirks of NFS, the quirks of a
clustered filesystem could be found and dealt with, too.

Key/value databases are hardly a magic bullet for redundancy. You don't
get 3 copies in different datacenters by simply switching to a
database-style storage.

[1]
http://www-conf.slac.stanford.edu/AFSBestPractices/Slides/MorganStanley.pdf


> Clustered filesystems are also complex. They're much more complex than
> what Dovecot really requires.
> 

I mention it because you stated wanting to outsource the storage
portion. The complexity of whatever database engine you choose or
supporting a clustered filesystem (like NFS) is a wash since you're not
maintaining either one personally.

~Seth

Re: [Dovecot] Scalability plans: Abstract out filesystem and make it someone else's problem

[Seth Mattinen]

Curtis Maloney wrote:
> Seth Mattinen wrote:
>> Ick, some people (myself included) hate the idea of storing mail in a
>> database versus simple and almost impossible to screw up plain text
>> files of maildir. Cyrus already does the whole mail-in-database thing.
> 
> Why do you think 'maildir' isn't a database?
> 
> Or to you does 'database' only mean "SQL database"?
> 

Please, don't put words in my mouth. I'm not stupid.

~Seth

Re: [Dovecot] Scalability plans: Abstract out filesystem and make it someone else's problem

[Seth Mattinen]

Timo Sirainen wrote:
> On Mon, 2009-08-10 at 14:33 -0700, Seth Mattinen wrote:
>> Timo Sirainen wrote:
>>> This is something I figured out a few months ago, mainly because this
>>> one guy at work (hi, Stu) kept telling me my multi-master replication
>>> plan sucked and we should use some existing scalable database. (I guess
>>> it didn't go exactly like that, but that's the result anyway.)
>>>
>> Ick, some people (myself included) hate the idea of storing mail in a 
>> database versus simple and almost impossible to screw up plain text 
>> files of maildir. 
> 
> Nothing forces you to switch from maildir, if you're happy with it :)
> But if you want to support millions of users, it's simpler to distribute
> the storage and disk I/O evenly across hundreds of servers using a
> database that was designed for it. And by databases I mean here some of
> those key/value-like databases, not SQL. (What's a good collective name
> for those dbs anyway? BASE and NoSQL are a couple names I've seen.)
> 


Why is a database a better choice than a clustered filesystem? It seems
that you're adding a huge layer of complexity (a database) for something
that's already solved (clusters). Queue directories and clusters don't
mix well, but a read-heavy maildir/dbox environment shouldn't suffer the
same problem.

~Seth

Re: [Dovecot] Scalability plans: Abstract out filesystem and make it someone else's problem

[Seth Mattinen]

Timo Sirainen wrote:

This is something I figured out a few months ago, mainly because this
one guy at work (hi, Stu) kept telling me my multi-master replication
plan sucked and we should use some existing scalable database. (I guess
it didn't go exactly like that, but that's the result anyway.)



Ick, some people (myself included) hate the idea of storing mail in a 
database versus simple and almost impossible to screw up plain text 
files of maildir. Cyrus already does the whole mail-in-database thing.


~Seth

Re: [Dovecot] High Availability strategies

[Seth Mattinen]

Ed W wrote:
> I think there is a whitepaper from Dovecot's current main sponsor
> (before they were called Rackspace) which described their architecture
> using pairs of servers and DRBD between them.  Each server is moderately
> loaded and active.  If one server fails then half the users don't notice
> and the other half get switched over to what is really their slave server.
> 
> Seemed like an elegant architecture...
> 

It's stupidly simple but it works. If you have two spare systems to make
a lab out of you'll be surprised how easy it really is. The only thing I
hate about it is the idle slave hardware just because I feed bad about
powerful machines sitting around waiting for something to happen.

~Seth

Re: [Dovecot] Dovecot Versions and Debian

[Seth Mattinen]

Mario Antonio wrote:
> If you want to run Dovecot on Debian Lenny for a Production System (with
> Fast Upgrade System for Security Patches),
> What would you recommend?
> Stick to the current port with Dovecot Version 1.0.15 (1:1.0.15-2.3)
> Use Backports Version 1.1.13 (1:1.1.13-2~bpo50+1)

> Do you find any compelling reason for not using version 1.0.15 ?
> (Besides of ACL limitations ...)
> 

It's ancient history. There are far too many improvements to list. The
first time you have a question with 1.0.15 we'll tell you to upgrade
before you'll get any help. I rebuild the package from sid:

1. Download source packages (orig, dsc, and diff files)
2. apt-get build-dep 
3. dpkg-source -x 
4. cd into directory created by dpkg-source -x
5. fakeroot dpkg-buildpackage -uc -b

And it'll make the four dovecot packages.

~Seth

Re: [Dovecot] Several errors

[Seth Mattinen]

FraNL wrote:

And how do I implement that patch?



Grab the source, do a "patch -P0 < file" and recompile.

~Seth

Re: [Dovecot] High Availability strategies

[Seth Mattinen]

Joseba Torre wrote:
> Hi,
> 
> we have a medium setup (8000 pop and imap users using almost every 
> available client, 800GB of stored mails using maildir on a Celerra NFS 
> server, with index files on local disks, and procmail for local 
> delivery), being served by a Dell PowerEdge 2850 (2GB RAM and dual P4 
> Xeon 3,2GHz).
> 
> Our current not-so-high availability setup is based on a similar 
> server with the same setup and a easy but manual process to switch 
> from one server to another.

If you don't care about keeping an active/standby setup and you're happy
with what you've currently got going on, you can easily automate the
process with heartbeat.

~Seth

Re: [Dovecot] RFC: Different quota for dovecot and deliver

[Seth Mattinen]

Philipp Marek wrote:
> Hello Timo!
> 
> On Montag, 20. Juli 2009, Timo Sirainen wrote:
>> On Jul 20, 2009, at 1:56 AM, Philipp Marek wrote:
>>> Are placeholders like "%s", "%n" etc. listed anywhere?
>> http://wiki.dovecot.org/Variables
> Thank you very much for your quick answer!
> 
> BTW: Google gives
>   Your search - link:http://wiki.dovecot.org/Variables - did not 
>   match any documents.
> 
> so how about linking to that page in the Wiki ;-) ?
> 

It is: http://wiki.dovecot.org/MainConfig

And others. Google is not always right.

~Seth

Re: [Dovecot] dsync - one or two ways?

[Seth Mattinen]

Timo Sirainen wrote:

On Fri, 2009-07-17 at 12:06 -0700, Seth Mattinen wrote:

So is this targeted as a kind of replication solution then?


Either continuous replication or just one-off user moves/conversions
without locking the mailbox and without losing changes done during the
move/conversion.



Cool. I just built a master/slave redundant dovecot last night with DRBD 
and heartbeat and this sounds easier. ;)


~Seth

Re: [Dovecot] dsync - one or two ways?

[Seth Mattinen]

Timo Sirainen wrote:
> On Fri, 2009-07-17 at 11:59 -0700, Seth Mattinen wrote:
>>> Now, the question is: Does anyone want dsync to only sync changes from
>>> source to destination, instead of doing a full two-way sync? I think in
>>> typical cases where you'd think you would want only one-way sync are
>>> also the cases where there's no changes coming the other way in any
>>> case.
>>>
>> Yes; moving mail from an old source (for example, my old home dovecot
>> server which still occasionally gets mail, but otherwise I don't really
>> use it). I wouldn't want all my current mail sent to it for this purpose.
> 
> dsync probably wouldn't be such a good idea for this use case anyway.
> dsync assumes that an IMAP client has accessed the mailboxes in both
> source and destination, so if there are any UID conflicts (and in the
> case you describe there would be UID conflicts all the time) it gives
> new UIDs for the conflicting messages.
> 


So is this targeted as a kind of replication solution then?

~Seth

Re: [Dovecot] dsync - one or two ways?

[Seth Mattinen]

Timo Sirainen wrote:
> dsync in Dovecot v2.0 tree is a new utility for syncing a mailbox in two
> locations. Some things it can be used for:
> 
>  - Initially transfer a mailbox to another server via SSH
>  - A faster sync done to an existing mailbox, sending only changes
>  - A superfast sync based on modification sequences.
>  - Source and destination mailboxes can use different formats
> (convert-tool will be history)
> 
> dsync can handle all kinds of conflicts in mailboxes, handle mailbox
> deletions, renames, etc. So it's safe to sync even if both source and
> destination mailboxes have had all kinds of changes.
> 
> Now, the question is: Does anyone want dsync to only sync changes from
> source to destination, instead of doing a full two-way sync? I think in
> typical cases where you'd think you would want only one-way sync are
> also the cases where there's no changes coming the other way in any
> case.
> 

Yes; moving mail from an old source (for example, my old home dovecot
server which still occasionally gets mail, but otherwise I don't really
use it). I wouldn't want all my current mail sent to it for this purpose.

~Seth

Re: [Dovecot] E-Mail Encryption

[Seth Mattinen]

Tom Hendrikx wrote:
> Thomas schreef:
>> Arkadiusz Miskiewicz wrote:
>>> On Wednesday 15 of July 2009, Patrick Domack wrote:
 The only benefit this would being, is email being saved on the server
 would be encrypted. Otherwise it offers no protection.

 I guess if you paranoid that the system admin might read your emails,
 but then, he can just as easily read them as they come in or out of
 the system.
>>> Actually such encryption is interesting as a protection in case when
>>> someone steals server hardware/disks.
>> It could be a feature. Why not.
>> But I'd say that's might be a better idea to encrypt the filesystem.
>> But... why not if you have time to code it :)
>>
>> Cheers,
>> Thomas
> 
> When you have to worry about unauthorized persons having physical access
> to your hardware, you're solving the wrong problem. Encryption would add
> only false security because the person could also pop some sniffer
> device onto your NIC connection that reads wire traffic...
> 
> The "de/encryption in deliver" concept is interesting, but imho not much
> use in real life. hard disk encryptoin would be much easier though (i.e.
> off-the-shelve). But I think these tin foil hat ideas get a little
> off-topic:)
> 

Encrypting with a public key is completely reasonable, but for proper
security, the decryption should only take place on the client's trusted
workstation with their private key.

~Seth

Re: [Dovecot] Possible fix to an old Thunderbird related problem

[Seth Mattinen]

Marc Perkel wrote:
> 
> 
> Charles Marcus wrote:
>> On 7/12/2009 12:23 PM, Marc Perkel wrote:
>>  
>>> I spoke too soon. Thunderbird is still broken. And it's really
>>> frustrating.
>>> 
>>
>> I'm sure Timo and others (myself included) would like more details...
>>
>> *What* is broken?
>>
>> Dovecot config details (dovecot -n output and other info that isn't
>> provided by that output)?
>>
>> Logs exhibiting the problem?
>>
>>   
> 
> I'm not sure where the problem is. Thunderbird always starts out fine.
> But after running for some time the message count of unread message will
> increase on folder with new messages but when you click on the folders
> the new messages don't appear. Or messages appear but seem to be empty
> messages. When selecting to empty the trash - the trash doesn't empty,
> ot the count of messages in trash will drop down somewhat.
> 
> Restarting Dovecot or restarting Thunderbird always solves the problem
> for a while. I noticed the problem got a lot worse recently and the last
> change was upgrading to 1.2.1. But it's been a serious annoyance for
> over a year.
> 
> Nothing strange in the logs to report. Wish I had more information.
> 

I see this every now and then with Thunderbird as well; just today, in
fact. Nothing strange (other than this) to report and no errors in the
logs. I'm on 1.1.17.

I know I'm absolutely no help, but it's not just you. ;)

~Seth

Re: [Dovecot] Directory Layout Performance

[Seth Mattinen]

to...@tuxteam.de wrote:
> On Wed, Jul 01, 2009 at 11:48:04AM -0700, Seth Mattinen wrote:
>> Mario Antonio Garcia wrote:
>>> From a performance perspective:
> 
> [...]
> 
>> I use a filesystem that handles this better than ext3 such as XFS or Reiser.
> 
> Ext3 should be fine for huge directories these days (given mount option
> dir_index it uses hashes for directory lists, but that should be the default
> in newer installations). You can find out whether it's on with "tune2fs 
> ".
> 

Not sure about maildir, but I experienced horrible performance with ext3
on a very busy postfix queue a few years ago. The dir_index was on, it
was on a battery-backed hardware RAID, but it just couldn't handle it
until I reformatted /var to XFS in the middle of the day out of
desparation. Yeah, I know Reiser should have better performance with
lots of small files, but I had a scary reiser-ate-itself experience
once. As far as huge mbox files, XFS should have an advantage there.

I don't have any numbers beyond ext3 "mail is backed up, people are
complaining, and the load average is through the roof" vs. XFS "it's
easily keeping up with the queue, everyone is happy, and the load
average is sane". ;)

~Seth

Re: [Dovecot] Directory Layout Performance

[Seth Mattinen]

Mario Antonio Garcia wrote:
> From a performance perspective:
> Which Directory layout is better:
> 1. All mailboxes are stored in a single directory and prefixed with a dot
> or
> 2. Maildirs using physical directories, such as: Maildir/folder/subfolder/
> 
> It looks like the second option -- File Sytem Layout (LAYOUT=fs) --
> could spread the load in a better way ..
> I do not expect a user  reaching maximum number of subfolders in linux ,
> but what about those heavy users with a lot of sub-folders:
> Is there a significant performance gain by choosing a specific 
> Directory Layout ...
> 
> Any Comments?
> 

I use a filesystem that handles this better than ext3 such as XFS or Reiser.

~Seth

Re: [Dovecot] quota.

[Seth Mattinen]

Mauro wrote:

2009/6/17 Mauro :

Yes, for now I've set a global quota.

Perhaps I have to delete maildirsize file for all the accounts?

I guess you could try that.


It works now, thank you.
Perhaps maildirsize files format were corrupted.
I am disappointed that the new debian stable has dovecot 1.0 rather
than dovecot 1.1.



You can find 1.1 in backports as apparently the next release will 
finally have 1.1 (right around the time 1.2 becomes the standard), or 
rebuild the package (I'll post the commands to do it if anyone is 
interested) from unstable.


~Seth

Re: [Dovecot] Lost sub-mailboxes - not showing after upgrading to Dovecot from Courier-IMAP

[Seth Mattinen]

James Brown wrote:
> Last night we migrated from Courier-IMAP to Dovecot 1.2RC5 and all seems
> to work fairly well, but I seem to have lost all my mailboxes apart from
> INBOX. The maildirs are all on the HD in the correct place
> (/usr/local/virtual/bordo.com.au/jlbrown):
> 

> 
> Any idea how to get them back?
> 

Subscribe to them; that's all.

~Seth

Re: [Dovecot] Help please- Post - Login Script--- Migration

[Seth Mattinen]

Ashraf wrote:
> Seth,  I have tried to use most of them but whichever i use its creating a
> folder and then its creating the dovecot-uidlist and rest of the files.
> please advice.
> 

The variable ${HOME} - in your script, not in the mail_executable
setting - should give you what you want.

~Seth

Re: [Dovecot] Help please- Post - Login Script--- Migration

[Seth Mattinen]

Ashraf wrote:
> Once the script is executed its executing the binary as a last step.
> Reference http://wiki.dovecot.org/Migration/Courier
> 
>  You can also convert each user as they log in for the first time, using
> PostLoginScripting  with a
> script something like:
> 
> #!/bin/sh
> # WARNING: Be sure to use mail_drop_priv_before_exec=yes,
> # otherwise the files are created as root!
> 
> courier-dovecot-migrate.pl --quiet --to-dovecot --convert ~/Maildir
> # This is for imap, create a similar script for pop3 too
> exec /usr/local/libexec/dovecot/imap
> 
> in place of ~/Maildir i want to pass the home directory of users which is
> passed as mail variable (Query from MySql).
> 
> Appreciate your response.
> 

According to those links you need to use the various environment
variables. Why have you chosen not to do that?

~Seth

Re: [Dovecot] Help please- Post - Login Script--- Migration

[Seth Mattinen]

Ashraf wrote:
> Hi all
>   I am in the process of migrating Courier to Dovecot, am using
> courier-dovecot-migrate.plit
> works fine. I want to migrate each user as they login using post-login
> script (As i have huge no of users). We are using NFS storage,  users are
> authenticated using mysql, User home directories as passed using variable
> mail ( usign mysql query).
> 
>I am trying to execute postlogin script which does the conversion but
> unable to pass the user home directories .
> 
> 
> --*/etc/dovecot.conf
> 
> mail_executable = /usr/libexec/dovecot/postloginscript.sh %h
> ---*
> %h   for home directory.
> 
> 
> ---*postloginscript.sh
> 
> #!/bin/sh
> /usr/libexec/dovecot/courier-dovecot-migrate.pl --quiet --to-dovecot
> --recursive --convert $1
> # This is for  pop3
> exec /usr/libexec/dovecot/pop3
> *
> 
> 
> When i pass %h as argument its creating  "%h" directory and writing
> dovecot-uidlist and rest the files in %h directory. Its not interpreting the
> %h and not passing the home directory. Its just passing as it is.
> 


I could be wrong, but I don't think you can do that. The mail_executable
setting is for the location of the binary.

~Seth

Re: [Dovecot] Clustering dovecot?

[Seth Mattinen]

dove...@corwyn.net wrote:
> At 02:00 PM 5/27/2009, Seth Mattinen wrote:
>> You're going to need something in front of the two servers to abstract
>> the connections (another poster recommended ultramonkey) if you want
>> them both to be serviced by the same IP address. My suggestion was to
>> use active/standby, where the standby doesn't "up" its mail IP unless it
>> detects the primary is dead, typically through some kind of heartbeat;
>> not separate clients across the two servers.
> 
> I'm not sure what you mean by "client"; do you mean instances of dovecot?
> 
> wouldn't that be two server instances, each running it's own copy of
> dovecot?
> 

Yes, two totally separate servers, either running shared storage (still
a common failure point, unless you have redundancy there too) or one
that waits in the wings in standby mode until the primary fails using
DBRD or something else to keep the mail stores in sync.

Their website explains it all better than I can:
http://www.drbd.org/

~Seth

Re: [Dovecot] Clustering dovecot?

[Seth Mattinen]

dove...@corwyn.net wrote:
> At 01:17 PM 5/27/2009, Seth Mattinen wrote:
>> You're probably looking for Dovecot replication, which as far as I'm
>> aware, has not made itself known in a functional form yet. I really want
>> to see master/master replication.
> 
> I at least don't think I am - I think I'd rather run dovecot with two
> front-end servers clustered to a backend SAN. If I"m running two
> separate servers then all of my clients (which i can point to only 1 IP)
> I think woudl go down if "their" server went  down.


You're going to need something in front of the two servers to abstract
the connections (another poster recommended ultramonkey) if you want
them both to be serviced by the same IP address. My suggestion was to
use active/standby, where the standby doesn't "up" its mail IP unless it
detects the primary is dead, typically through some kind of heartbeat;
not separate clients across the two servers.

~Seth

Re: [Dovecot] Clustering dovecot?

[Seth Mattinen]

dove...@corwyn.net wrote:
> 
> 
> 
> Currently I have a mail server running postfix and dovecot with users in
> mysql. I have Outlook (blich) clients checking IMAP mail on the server.
> I'd like to build some fault-tolerance into that system in case I have
> hardware failure.
> 
> I've read through: http://wiki.dovecot.org/MailLocation/SharedDisk but I
> feel I need more information. My basic understanding is that I'll need 2
> clustered systems running with shared disk space and shared IP. 
> Performance/load-balancing isn't an issue for me, it's just
> fault-tolerance that I'm worried about.
> 
> I found a nice long doc for Cyrus IMAP
> (http://cyrusimap.web.cmu.edu//ag.html), anything similar for dovecot?
> 

You're probably looking for Dovecot replication, which as far as I'm
aware, has not made itself known in a functional form yet. I really want
to see master/master replication.

I was working on coming up with something myself in an active/standby
setup using DRBD. If the heartbeat dies, DRBD on the standby fires up
its Dovecot and takes over. Vice-versa for the primary recovering: shut
down standby and start primary. It's not totally transparent because it
would drop all active connections during the switch. In practice, I find
that nobody notices unless it happens multiple times in a row or they
happened to be in the middle of a long IMAP command.

~Seth

Re: [Dovecot] offlineimap with dovecot and sieve

[Seth Mattinen]

Dieter Faulbaum wrote:
> I have a dumb question, is it possible to use this combination:
> 
> Synchronising a remote IMAP-server and a local dovecot-IMAP-server by
> offlineimap (with both types = IMAP), which works very well.
> But it seems, that my sieve-scripts on the local dovecot server are not
> recognised by this combination.
> Is there a good debugging way for this?
> 

If "deliver" doesn't run the sieve scripts won't run. It's not part of
IMAP. Simple as that.

~Seth

Re: [Dovecot] How to run Dovecot as IMAP Proxy?

[Seth Mattinen]

V S Rao wrote:
> Thanks for the many responses and views.
> 
> I have taken RH support for my mail server only and so have to ensure they 
> support me. So am going with their recommended version which is 0.99.x.
> 
> Now on the webmail I have migrated to RHEL 5.3 and the dovecot being used 
> there is 1.0.7.x. So that should support proxying, right? 
> 
> BTW, originally I did not go to Redhat. I moved from uw-imap to Dovecot 
> 1.1.14 on the mail server for POP3. The problem of POP3 timeouts continued 
> and I could not find any reason why POP3 was timing out when 400+ concurrent 
> IMAP sessions were working fine. Moreover other network services such as 
> SMTP, Telnet, SSH etc were working fine. Had the box installed behind a IPS 
> box to see if there were any DoS attacks, but had to rule that out. As a last 
> resort, paid RH for support and they made me downgrade Dovecot from 1.1.14 to 
> 0.99.x. Surprisingly the moment I downgraded dovecot, the problem shifted to 
> IMAP & remained ever since. So naturally they told me uw-imap was the problem 
> and made me move to Dovecot 0.99.x for IMAP as well, but no improvement. 
> 
> But the surprising thing, as I mentioned in my original post, IMAP works fine 
> on command line, with clients such as Outlook, THunderbird etc., but Webmail 
> is very slow that its almost not available. 
> 
> BTW webmail was also running on RHEL 4.0. Seeing that the problem existed in 
> webmail alone I upgraded webmail to RHEL 5.3 with all the latest RH supported 
> packages on a new hardware. 
> 
> Any ideas to help me debug this issue would be of great help as I am stuck on 
> this issue for almost a month now. 
> 
> Oh, btw regarding stats, I don't have a measure, but when webmail was working 
> the login was well within 5 seconds. Now it takes forever (more than a min) 
> or timeouts most of the time. Even after login, any request such as opening a 
> mail or changing to a mail folder was well within 5 seconds. Now again it 
> takes forever or timeouts most of the time. 
> 

Use imapproxyd (UP-ImapProxy) if you want a caching proxy for webmail.
It specifically deals with the "webmail constantly logging in" issue.
>From the Debian man page:

"ImapProxy was written to compensate for webmail clients that are unable
 to maintain persistent connections  to an IMAP server. Most webmail
clients need to log in to an IMAP server for nearly every single
transaction; This behaviour can cause tragic performance problems on the
IMAP server.  ImapProxy tries to deal with this  problem by  leaving
server  connections open for a short time after a webmail client logs
out. When the webmail client connects again, ImapProxy will determine if
there is a cached connection available and reuse it if possible."

Dovecot proxy quite simply won't do what it looks like you want to do,
and is irrelevant on your webmail server.

~Seth

Re: [Dovecot] How to run Dovecot as IMAP Proxy?

[Seth Mattinen]

Timo Sirainen wrote:
> On Thu, 2009-05-21 at 11:47 -0700, Seth Mattinen wrote:
>> Yes, that too. My point was not to judge Dovecot's performance based on
>> a 0.99 version. ;) I've successfully used several versions of dovecot
>> 1.1.x with imapproxyd and IMAP-only webmail clients.
> 
> Do you have any numbers about how much faster it works with imapproxyd
> vs. without?
> 


I've never run it without. I just did it because I'd rather collapse the
10 distinct logins a typical webmail client will make for every click
into a single login.

~Seth

Re: [Dovecot] How to run Dovecot as IMAP Proxy?

[Seth Mattinen]

Scott Silva wrote:
> on 5-21-2009 11:33 AM Seth Mattinen spake the following:
>> V S Rao wrote:
>>> Hi Timo,
>>>
>>> Thanks for the response. Apologize, but my responses are going to be a bit 
>>> lengthy.
>>>
>>>> I have migrated from uw-imap to Dovecot for POP3 & IMAP service. I run
>>>> webmail using squirrelmail. When running uw-imap I used to run
>>>> up.imapproxyd on the webmail server for faster responses. After
>>>> migrating to Dovecot, I find that up.imapproxyd does not work well
>>>> with dovecot. 
>>>> Why not?
>>> Here are my observations. I have around 6000+ mailboxes and roughly the 
>>> same number of users. Earlier the mail server (running sendmail + uw-imap 
>>> for POP3 & IMAP) would have around 80 pop3 connections (peak) and around 
>>> 300 IMAP connections, concurrent. 
>>> (ps -aef | grep imap | wc -l or ps -aef | grep pop3 | wc -l). There have 
>>> been cases where I have observed upto 500 concurrent IMAP sessions.
>>>
>>> The IMAP connections are from a webmail server running Apache with 
>>> Squirrelmail. The observation was that response was slow & so based on the 
>>> suggestion on Squirrelmail for performance improvement we have installed 
>>> up-imapproxyd on the squirrelmail machine. There was a significant 
>>> improvement in the response times for the users, because of caching.
>>>
>>> After that I encountered some strange problems of POP3 timing out for users 
>>> (earlier I did post that problem in this forum). I opened a ticket with 
>>> Redhat and naturally they refused to support me with uw-imap running. So 
>>> switched to Dovecot 0.99.x (I run the server on RHEL 4.0 and that is the 
>>> max version supported by Redhat for that version). Ever since I did that 
>>> POP3 works fine but now webmail is almost not available to the users. 
>>> People usually get "connection dropped by IMAP server". However the IMAP 
>>> server seems to work fine. Checked through manual "telnet 
>>>  143 & also through other client such as outlook 
>>> & Thunderbird. 
>>>
>>
>> I realize you're using 0.99 because it's "supported" by RedHat, but in
>> reality it's absolutely ancient history. There have been far too many
>> performance enhancements/fixes between then and now to even begin to list.
>>
>> ~Seth
>>
> Besides, he didn't go to RedHat for support anyway. He came here. And here
> says start with at least the latest 1.0 version, or maybe even 1.1. You can
> get it from atrpms.net if you want an rpm.
> 


Yes, that too. My point was not to judge Dovecot's performance based on
a 0.99 version. ;) I've successfully used several versions of dovecot
1.1.x with imapproxyd and IMAP-only webmail clients.

~Seth

Re: [Dovecot] How to run Dovecot as IMAP Proxy?

[Seth Mattinen]

V S Rao wrote:
> Hi Timo,
> 
> Thanks for the response. Apologize, but my responses are going to be a bit 
> lengthy.
> 
>> I have migrated from uw-imap to Dovecot for POP3 & IMAP service. I run
>> webmail using squirrelmail. When running uw-imap I used to run
>> up.imapproxyd on the webmail server for faster responses. After
>> migrating to Dovecot, I find that up.imapproxyd does not work well
>> with dovecot. 
> 
>> Why not?
> 
> Here are my observations. I have around 6000+ mailboxes and roughly the same 
> number of users. Earlier the mail server (running sendmail + uw-imap for POP3 
> & IMAP) would have around 80 pop3 connections (peak) and around 300 IMAP 
> connections, concurrent. 
> (ps -aef | grep imap | wc -l or ps -aef | grep pop3 | wc -l). There have been 
> cases where I have observed upto 500 concurrent IMAP sessions.
> 
> The IMAP connections are from a webmail server running Apache with 
> Squirrelmail. The observation was that response was slow & so based on the 
> suggestion on Squirrelmail for performance improvement we have installed 
> up-imapproxyd on the squirrelmail machine. There was a significant 
> improvement in the response times for the users, because of caching.
> 
> After that I encountered some strange problems of POP3 timing out for users 
> (earlier I did post that problem in this forum). I opened a ticket with 
> Redhat and naturally they refused to support me with uw-imap running. So 
> switched to Dovecot 0.99.x (I run the server on RHEL 4.0 and that is the max 
> version supported by Redhat for that version). Ever since I did that POP3 
> works fine but now webmail is almost not available to the users. People 
> usually get "connection dropped by IMAP server". However the IMAP server 
> seems to work fine. Checked through manual "telnet  
> 143 & also through other client such as outlook & Thunderbird. 
> 


I realize you're using 0.99 because it's "supported" by RedHat, but in
reality it's absolutely ancient history. There have been far too many
performance enhancements/fixes between then and now to even begin to list.

~Seth

Re: [Dovecot] ext3/ext4 performance issue

[Seth Mattinen]

Matthijs Kooijman wrote:
> Hi Scott,
> 
>> One could always run the mailstore on LVM and then you could snapshot the
>> mount and then fsck it while still technically in use. It would probably slow
>> down the filesystem, but it is still live.
> Uhm, and then you have a nice and fsck'd snapshot, but your live filesystem
> will still be untouched? Or do you propose replacing the mailstore with a
> (readonly?) versioned mounted from the lvm snapshot? I'm not aware of anything
> that can swap filesystems like this online (ie, with files open) and that
> would also mean that while fsck'ing, you can only read mail, not receive, move
> or delete mail?
> 

Changes made to a snapshot will never been seen by the fs it was
snapshotted from and vice-versa. It uses copy on write which will get
progressively slower as the snapshot and the origional diverge until you
end up with a performance disaster.

http://www.nikhef.nl/~dennisvd/lvmcrap.html

~Seth

[Dovecot] Filesystems (was Re: Migration questions...)

[Seth Mattinen]

Richard Hobbs wrote:
>
> Trouble is... i've been googling this as well, just now, and loads of
> people say XFS has the better performance, but loads of other people say
> ReiserFS has the better performance.

It starts to become a religious argument at some point.


> We have battery backed up RAID controllers too, in this new system, and
> the systems are UPSd, so on that basis i'm still none the wiser! lol
>
> I appreciate your experience with XFS is a positive one, but even the
> dovecot web site says XFS might now be a good choice...
>
> http://wiki.dovecot.org/MailboxFormat/Maildir
>
> What a tough decision! I know it probably won't make too much difference
> in my situation, but i want this to be a very long-term solution, so
> want to do it right first time!
>
> Any other opinions on XFS vs Reiserfs with Dovecot maildir?
>
> Thanks again!
>
> Richard.
>


If you have time to play "tweak the filesystem":

http://everything2.com/index.pl?node_id=1479435

It's a good comparison of etx3, xfs, and reiser from a performance
standpoint. Also try this if you want to pull the plug or randomly
reboot and see what happens:

http://brad.livejournal.com/2116715.html

~Seth

Re: [Dovecot] Migration questions...

[Seth Mattinen]

Richard Hobbs wrote:
> Seth Mattinen wrote:
>> Phillip Macey wrote:
>>> On 14/05/2009 5:11 PM, Steffen Kaiser wrote:
>>>> On Wed, 13 May 2009, Richard Hobbs wrote:
>>>>> The main complaint we have from users is that their IMAP Inbox, with
>>>>> 5000 emails in it takes ages to appear, and no amount of coaxing will
>>>>> convince them to split their inbox into multiple folders.
>>>> Oh, we serve Maildir via Dovecot IMAP and 5000 messages per folder are
>>>> a wimp. Problems start if the user:
>>> We are having some performancec issues on our server at the moment - all
>>> I can put it down to is the large size of some maildirs. Eg. `ls -ld
>>> Maildir/cur` might show a directory >20Mb in size for some of our users
>>> (20-30k emails).
>>> (Performance issues == everything is running ok then all of a sudden
>>> load avg goes through the roof, system cpu time goes crazy. Reading mail
>>> grinds to a halt. Then everything recovers just as suddenly and the load
>>> avg gradually returns to normal levels)
>> Are you using ext3 by chance? Vanilla ext3 without directory indexing
>> (or whatever it's called) *hates* directories with a lot of files - like
>> maildir. Personally, I use XFS, which doesn't suffer from this problem
>> since it uses b-trees instead of a table(!) like ext3 does.
> 
> This raises another question for me actually...
> 
> We will have one volume for indexes and another volume for data (using
> maildir). We will be using the latest stable Debian Linux distro.
> 
> Any opinions on the best filesystem to use? We would normally go
> ReiserFS for large volumes, and ext3 for small volumes because of the
> unlimited inodes in reiserfs, but i understand that support for that is
> beginning to disappear given that Hans Reiser got into a bit of trouble
> a few years ago.
> 
> Anyway... that would leave ext3, but is there a better choice i could
> make in this instance? We do want performance, of course, but also
> complete reliability and resilience when it comes to system crashes
> etc... we do *not* want data corruption, and ext3 we know has a very
> good journalling and data recovery methods. Well... they're very mature,
> anyway.
> 

I used to use ext3, ran into its horrible performance even with
directory indexing enabled, went to XFS and never looked back. All of my
systems are Debian stable. Reiser3 is part of the kernel so I wouldn't
worry about that; Namesys considered it complete and stopped work on it
long before the whole murder thing. Both Reiser3 and XFS have worse
reputations than ext3, however, I've seen ext3 filesystems hosed beyond
repair, too. All my XFS filesystems have battery-backed cache
controllers, so it hasn't happened to me yet, hopefully never. ;) One
catch with XFS (such as with LVM) to keep in mind is you can't ever
shrink it, only grow.

ext3 is mature but IMHO completely unsuitable for a busy mail server or
any situation where you'll have a bajillion of files in one directory.
The exact point at which ext3 will screw you over obviously depends on
many factors. But when it happens it'll probably be painful to reformat
to something better.

~Seth

Re: [Dovecot] Migration questions...

[Seth Mattinen]

Richard Hobbs wrote:
> Curtis Maloney wrote:
>> Phillip Macey wrote:
 Oh, we serve Maildir via Dovecot IMAP and 5000 messages per folder
 are a wimp. Problems start if the user:
>>> We are having some performancec issues on our server at the moment -
>>> all I can put it down to is the large size of some maildirs. Eg. `ls
>>> -ld Maildir/cur` might show a directory >20Mb in size for some of our
>>> users (20-30k emails).
>>> (Performance issues == everything is running ok then all of a sudden
>>> load avg goes through the roof, system cpu time goes crazy. Reading
>>> mail grinds to a halt. Then everything recovers just as suddenly and
>>> the load avg gradually returns to normal levels)
>> At first glance this sounds like a large folder is being indexed... are
>> you using Dovecot deliver (which updates indices on deliver)?
> 
> This raises an interesting question for me actually... given that we've
> now decided dovecot and maildir is the way forward for us, which
> delivery method should we use in exim? exim can support maildir,
> (right?) and so can dovecot, so should i use dovecot's "deliver"
> mechanism, or exim's own internal mechanism?
> 

Only dovecot 'deliver' will update the index on delivery.

~Seth

Re: [Dovecot] Migration questions...

[Seth Mattinen]

Phillip Macey wrote:
> On 14/05/2009 5:11 PM, Steffen Kaiser wrote:
>> On Wed, 13 May 2009, Richard Hobbs wrote:
>>> The main complaint we have from users is that their IMAP Inbox, with
>>> 5000 emails in it takes ages to appear, and no amount of coaxing will
>>> convince them to split their inbox into multiple folders.
>>
>> Oh, we serve Maildir via Dovecot IMAP and 5000 messages per folder are
>> a wimp. Problems start if the user:
> We are having some performancec issues on our server at the moment - all
> I can put it down to is the large size of some maildirs. Eg. `ls -ld
> Maildir/cur` might show a directory >20Mb in size for some of our users
> (20-30k emails).
> (Performance issues == everything is running ok then all of a sudden
> load avg goes through the roof, system cpu time goes crazy. Reading mail
> grinds to a halt. Then everything recovers just as suddenly and the load
> avg gradually returns to normal levels)

Are you using ext3 by chance? Vanilla ext3 without directory indexing
(or whatever it's called) *hates* directories with a lot of files - like
maildir. Personally, I use XFS, which doesn't suffer from this problem
since it uses b-trees instead of a table(!) like ext3 does.

~Seth

Re: [Dovecot] Migration questions...

[Seth Mattinen]

Richard Hobbs wrote:
> 
> That's also good to know... i like to do a job right instead of relying
> on faster hardware, as i'm sure you all do too, but it's good to know
> that if i make one or two "non-optimal" choices along the way, it'll
> probably be lightning fast anyway!
> 
> The main complaint we have from users is that their IMAP Inbox, with
> 5000 emails in it takes ages to appear, and no amount of coaxing will
> convince them to split their inbox into multiple folders.
> 

Most of my mailing list folders that I work with on a daily basis like
NANOG (over 20,000) and cisco-nsp (over 35,000) are no slower than
folders with a handful of messages in them. It's not a private server
either, it's the same one my customers use. If you're seeing a slowdown
at 5k messages, either your server is woefully underpowered or something
isn't quite right.

~Seth

Re: [Dovecot] Unable To Send Mail

[Seth Mattinen]

Carlos Williams wrote:
> On Tue, May 5, 2009 at 3:27 PM, Seth Mattinen  wrote:
>> Dovecot does not speak SMTP to other mail servers, unless you mean "send
>> mail" in a different sense than I'm used to.
>>
>> ~Seth
> 
> I mean that I can send (SMPT) via Telnet which Postfix does fine.
> However when I configure Thunderbird mail client to use the IMAP
> (Dovecot) settings and attempt to compose a new message, the user
> (recipient) never receives the message. I am thinking as well as users
> of the Postfix list that this issue is related to my Dovecot config
> file not being properly set up. I could be wrong...


Thunderbird is connecting via SMTP to your Postfix installation to send;
Dovecot only does the POP3/IMAP half.

~Seth

Re: [Dovecot] Unable To Send Mail

[Seth Mattinen]

Carlos Williams wrote:
> On Tue, May 5, 2009 at 3:21 PM, Seth Mattinen  wrote:
>> Try the postfix user list.
>>
>> http://www.postfix.org/lists.html
> 
> I did and was told this appears to be a Dovecot config issues since
> the MTA (Postfix) is sending mail just fine via Telnet. What makes you
> suggest this to be a MTA issue?

Dovecot does not speak SMTP to other mail servers, unless you mean "send
mail" in a different sense than I'm used to.

~Seth

Re: [Dovecot] Unable To Send Mail

[Seth Mattinen]

Try the postfix user list.

http://www.postfix.org/lists.html

Re: [Dovecot] Dovecot -> Gmail (via POP Mail Fetcher)

[Seth Mattinen]

Neil wrote:
> imapsync is failing for me, with very odd errors; mostly "failure to append
> message to folder", or some such...  And unlike most of the failures with
> imapsync, it didn't just error out (and give me the message with "N messages
> failed."), it just flat out crashes (it's outputting as if it's running, and
> then suddenly drops me back to a prompt).
> I kind of gave up on it for Gmail; but if you want to take a look at the
> errors, I'd be happy to toss you the log.
> 

Yeah, gmail doesn't support folders. Their IMAP support is misleading.
Try dumping it all to INBOX.

~Seth

Re: [Dovecot] Dovecot -> Gmail (via POP Mail Fetcher)

[Seth Mattinen]

Rick Romero wrote:
> 
> Try imapsync - it seems to work fairly well with Gmail, except folders
> are screwy.
> freshmeat.net/projects/imapsync/
> 

That's because gmail doesn't implement folders, they use "labels".

~Seth

Re: [Dovecot] For the record: Postfix+Spamassassin+ClamAV+Dovecot

[Seth Mattinen]

James Butler wrote:
> Postfix 2.5.5
> SpamAssassin 3.2.5 (under Perl 5.10.0)
> ClamAV 0.95.1
> Dovecot 1.2.rc2
> 
> works fine on Fedora 10.
> 
> Installed Dovecot and ClamAV from source and everything else using yum.
> 
> I'm using the ClamAV plugin for Spamassassin:
>  http://wiki.apache.org/spamassassin/ClamAVPlugin
> 
> I'm calling Spamassassin with:
> 
> /etc/postfix/main.cf:
> mailbox_command = /usr/bin/spamc -f -e /usr/local/libexec/dovecot/deliver
> 
> Postfix hands off to Spamassassin, which processes ALL mail (not just
> attachments) through the ClamAV plugin before parsing for spam, and then
> hands the whole mess off to Dovecot for 'deliver' to handle.
> 
> How simple is that?
> 
> Since ClamAV scanns all mail, it might be too processor-intensive for
> really large mail systems, but it is working great for our 120+ user
> system with lots of spam coming in. If you're using Procmail or some other
> preprocessor that can hand off to a pipe, then you could skip the plugin
> and pipe messages over a certain size (i.e. >1024) to clamd, instead.
> 

SpamAssassin is more of a CPU hog than Clam is, although that depends if
you're scanning large files or not. If my mail systems aren't falling
over while running Clam and SA (although I don't use the plugin, I scan
the stream with clamd before SA) the average system should be fine too.

~Seth

Re: [Dovecot] multiple sql servers - loadbalancing and failover

[Seth Mattinen]

Arkadiusz Miskiewicz wrote:
> On Friday 27 of March 2009, Seth Mattinen wrote:
>> Arkadiusz Miskiewicz wrote:
>>> On Friday 27 of March 2009, Xueron Nee wrote:
>>>> You can try mysql-proxy:
>>>>
>>>> http://dev.mysql.com/downloads/mysql-proxy/index.html
>>> This introduces single point of failure.
>> http://forge.mysql.com/wiki/MySQL_Proxy#Load_Balancing_.26_Failover
> 
> If I get this right if mysql proxy fails I still get nothing. Hm, I would 
> have 
> to run mysql proxy on every machine that has dovecot installed. Then it would 
> make some sense.
> 
> Ok, that should work.


I used to do it that way (using sqlrelay) before switching to hardware
load balancers. It's not really much a single point of failure since
it's running on the same machine; it's no different than Dovecot having
do the work itself. Simpler than creating a HA cluster, too.

~Seth

Re: [Dovecot] multiple sql servers - loadbalancing and failover

[Seth Mattinen]

Arkadiusz Miskiewicz wrote:
> On Friday 27 of March 2009, Xueron Nee wrote:
>> You can try mysql-proxy:
>>
>> http://dev.mysql.com/downloads/mysql-proxy/index.html
> 
> This introduces single point of failure.
> 

Plus use of heartbeat, HA, etc.

Re: [Dovecot] multiple sql servers - loadbalancing and failover

[Seth Mattinen]

Arkadiusz Miskiewicz wrote:
> On Friday 27 of March 2009, Xueron Nee wrote:
>> You can try mysql-proxy:
>>
>> http://dev.mysql.com/downloads/mysql-proxy/index.html
> 
> This introduces single point of failure.
> 

http://forge.mysql.com/wiki/MySQL_Proxy#Load_Balancing_.26_Failover

Re: [Dovecot] Preventing IMAP folder deletion?

[Seth Mattinen]

Seth Mattinen wrote:
> dove...@corwyn.net wrote:
>>
>> Hi!
>>
>> I'm running postfix with dovecot using mysql for user accounts, with
>> Outlook being used as the IMAP mail client.  I'm using a mail delimiter
>> ("+") and mail automatically goes to a Spam folder as part of spam
>> filtering with amavisd and spam assassin. It's configured so that
>> folders won't automatically create; if the folder isn't there mail
>> delivers to the inbox (so that spam doesn't create extra folders).
>>
>> the problem I ran into today is that a user deleted the Spam folder.
>> While recreating it isn't that hard, is there a way I can prevent them
>> from being able to delete specific folders in IMAP?
>>
>> In squirrelmail I can create "system" folders that the user can't change
>> or delete, but that's client based. Since I don't see that ability in
>> Outlook, can I somehow configure dovecot so the user can't delete
>> specified folders?
>>
> 
> Never tried it personally, but what about an ACL denying the "folder can
>  be deleted" attribute?
> 
> http://wiki.dovecot.org/ACL
> 

I meant "mailbox can be deleted" attribute "x" or "delete".

~Seth

Re: [Dovecot] Preventing IMAP folder deletion?

[Seth Mattinen]

dove...@corwyn.net wrote:
> 
> 
> Hi!
> 
> I'm running postfix with dovecot using mysql for user accounts, with
> Outlook being used as the IMAP mail client.  I'm using a mail delimiter
> ("+") and mail automatically goes to a Spam folder as part of spam
> filtering with amavisd and spam assassin. It's configured so that
> folders won't automatically create; if the folder isn't there mail
> delivers to the inbox (so that spam doesn't create extra folders).
> 
> the problem I ran into today is that a user deleted the Spam folder.
> While recreating it isn't that hard, is there a way I can prevent them
> from being able to delete specific folders in IMAP?
> 
> In squirrelmail I can create "system" folders that the user can't change
> or delete, but that's client based. Since I don't see that ability in
> Outlook, can I somehow configure dovecot so the user can't delete
> specified folders?
> 

Never tried it personally, but what about an ACL denying the "folder can
 be deleted" attribute?

http://wiki.dovecot.org/ACL

~Seth

Re: [Dovecot] dovecot with IMAP and POP3 mixed

[Seth Mattinen]

Bill Cole wrote:
> Andreas Meyer wrote, On 3/21/09 11:16 AM:
>> Hello!
>>
>> I tried to setup dovecot to work with IMAP and POP3 but when I connect
>> with Squirrelmail, I get an error saying CREATE "INBOX.Sent"
>> reason: Mailbox doesn't allow inferior mailboxes
> 
> That's for user7, right?
> 
> The cause of that error is SquirrelMail (using IMAP) is trying to create
> a mailbox as a child of INBOX, but since user7 is defined as using mbox
> format, that fails. If you use mbox storage, an IMAP folder can have
> either messages or subfolders but never both, because when using mbox an
> IMAP folder containing messages is a file, while an IMAP folder
> containing subfolders is a directory. See
> http://wiki.dovecot.org/MailboxFormat/mbox for more info.
> 
> There *should* be some way to make SquirrelMail stop trying to create
> its folders under INBOX, but I have no idea if there actually *is* one.
> Some IMAP servers only allow the creation of folders as children of
> INBOX, and as a result some (broken) IMAP clients only know how to
> create folders that way. You can accommodate such software in Dovecot by
> using namespaces.
> 

Run the squirrelmail configure program and change the "default folder
prefix" to blank under "folder defaults".

~Seth

Re: [Dovecot] Dovecot LDA and "undeliverable" emails

[Seth Mattinen]

Kat Wisnosky wrote:
> Timo wrote:
> 
>> So Postfix delivers bounces to everyone by calling Dovecot LDA
>> separately for each bounce? Looks like a Postfix configuration problem
>> to me.
> 
> I should have been more specific... because we'd also changed domain
> names at the time we moved over to using Dovecot as the LDA, I
> temporarily disabled automatically bouncing mail addressed to invalid
> users because I figured that we'd have at least one who decided that
> they didn't want the standard formation of email addresses...  What I'd
> wanted to happen was for Postfix to accept the message(s) and then for
> them to be delivered to the postmaster address so that I could address
> the issue.
> 
> I'm still unsure as to why, when Dovecot couldn't find the user in the
> tables, it copied the message to every user of the system - that seems
> like odd behavior, which indicates that even though the system is
> "working", I need to fine-tune something.
> 

Dovecot didn't do that, Postfix executed those "deliver" instances.

~Seth

Re: [Dovecot] General approach question - am I doing this the right way?

[Seth Mattinen]

Alan Bates wrote:
> Sure - I looked at sieve, but as i understood it, i'm bypassing any LDA
> stage, using getmail to deliver direct to the maildir.
> 
> i suppose i could use getmail to deliver to a holding maildir, then use an
> LDA to filter and deliver that mail...
> 

Or procmail:

:0:
* ^X-Spam-Status: Yes
Maildir/.Junk/

Never used getmail personally so I have no idea what it can do.

~Seth

Re: [Dovecot] General approach question - am I doing this the right way?

[Seth Mattinen]

Alan Bates wrote:
> Quick outline of my current working setup:
> 
> I'm running dovecot 1.0.5 as an IMAP mail server on my ubuntu home server,
> which i connect to with my phone and several other machines to check my mail
> etc.
> 
> i use getmail to collect mail from my various pop accounts. getmail filters
> the mail using spamassassin via a Filter_external, before moving it to my
> Maildir on the home server.
> 
> Then when I connect to the home server from my various machines, IMAP
> manages the status of all the mail, so unread messages are common etc.
> 
> All works very nicely. However I'd like to fix a minor annoyance...
> 
> ...Spamassassin marks mail as SPAM but the mail client has to do the job of
> moving it to the Junk folder. I'd like the mail to be filtered on the home
> server on receipt.
> 

Sieve can do that.

http://wiki.dovecot.org/LDA/Sieve

~Seth

Re: [Dovecot] I'm the beginner and I want some help

[Seth Mattinen]

Wei Da wrote:
> Hello,
> I'm a software programmer and I work for a software corporation. We worked
> on dovecot for a while. We want to add some new functions to dovecot. For
> example, the mail stored on Mail-Server is plaintext, so we want to make it
> encrypted. As the documentation was not too much, and read the source code
> was too difficult for em. Would you give me some advice if I make this
> specialty as a plugin?
> Thank you!
> 

Sounds like you want to write a new storage engine. See the "Cydir"
storage source code for an example. Wiki page:
http://wiki.dovecot.org/MailboxFormat/Cydir

~Seth

Re: [Dovecot] problems with dotlock

[Seth Mattinen]

Mark Hedges wrote:
> 
> On Fri, 27 Feb 2009, Timo Sirainen wrote:
>> OK, so core dumps are enabled, but for some reason they
>> don't get written. There are really only two possibilities
>> then:
>>
>> a) You don't really have mail_drop_priv_before_exec=yes.
>> You could verify this with dovecot -n.
> 
> [r...@anubis etc]# /usr/local/sbin/dovecot -n | grep drop
> mail_drop_priv_before_exec: yes
> 
>> b) Kernel doesn't want to write the core to /tmp/core or
>> before changing that it didn't want to write it to user's
>> home directory.
> 
> [r...@anubis etc]# grep -i core
> /boot/config-2.6.18-92.1.22.el5
> CONFIG_ELF_CORE=y
> # Core Netfilter Configuration
> CONFIG_MLX4_CORE=m
> CONFIG_SERIAL_CORE=y
> CONFIG_SERIAL_CORE_CONSOLE=y
> # CONFIG_I2C_OCORES is not set
> # CONFIG_I2C_DEBUG_CORE is not set
> CONFIG_PROC_KCORE=y
> CONFIG_PROC_VMCORE=y
> 
> Is that informative?  I would not be surprised if the kernel
> is buggy.  It also indefinitely holds onto network
> connections in CLOSE_WAIT state, never times them out, and
> after some list research it seems there's no option to
> control that, could be wrong, but I gave up.
> 

Totally unrelated to your Dovecot issues, but are you using an Intel
card with that kernel? Specifically the e1000 driver is buggy enough in
that version that I gave up and use at least kernel 2.6.20.3 with e1000
cards. I wouldn't be surprised if there were other network related issues.

~Seth

Re: [Dovecot] exim4 + dovecot + SquirrelMail virtualhosted on Debian

[Seth Mattinen]

Tim Legg wrote:
> Hello,
> 
> I have a Debian machine that is virtual-hosting 6 domains with exim4.  I 
> finally got exim4 working well enough where I can send and receive mail using 
> the mail command on my machine.  I would eventually like to have SquirrelMail 
> (which requires IMAP) working on my website.  For whatever reason, exim4 
> doesn't seem to support IMAP.  I found out that Dovecot is a prominent IMAP 
> server.

Exim and its counterparts (Postfix, Sendmail, etc.) is an SMTP server.


> So I installed the most current Dovecot package (1.0.rc15) in Debian.  
> Something is broken though...
> 
> As I log into SquirrelMail, I get an error message "Error connecting to IMAP 
> server: localhost. 111 : Connection refused".  To investigate, I decided to 
> telnet localhost on port 143 and found out that yes, the connection is in 
> fact refused.
> 
> So what do I need to do in order to have dovecot listen on port 143?
> 
> Also, I noticed when I restart dovecot...
> 
> debian:/etc/dovecot# /etc/init.d/dovecot restart
> Restarting mail server: dovecotError: Can't use mail executable 
> /usr/lib/dovecot/imap: No such file or directory
> 
> I hope this isn't a serious problem.
> 

You probably didn't install the dovecot-imap package.

~Seth

Re: [Dovecot] GUI/WUI for creating (common) sieve scripts?

[Seth Mattinen]

Peter Lindgren wrote:
> Roderick A. Anderson skrev:
>> Anyone aware of a Web User Interface or GUI to allow users to create
>> simple/common-type sieve scripts?
>> So before I go invent this wheel I'd like to know what others are doing?
> 
> For Thunderbird, there's a plugin (I haven't tested):
> 
> http://sieve.mozdev.org/
> 

It doesn't do hand-holding, it's just an interface to managesieve.

~Seth

Re: [Dovecot] unkillable imap process(es) with high CPU-usage

[Seth Mattinen]

Timo Sirainen wrote:

On Sun, 2009-02-15 at 12:10 -0700, Gene Steinberg wrote:

This is sounding similar to the problem I have with my setup:

1. High CPU usage.

2. Can't kill IMAP.


kill -9 doesn't work for imap processes? You didn't mention this before.


I'm using CentOS 5.2 64-bit version with the latest cPanel.

So what am I missing, other than the problem nobody else is having is  
clearly something they ARE having?


If you can't kill -9 a process, it means the kernel is buggy. At least
2.6.27 was buggy and it was fixed in 2.6.27.10.




Or it's I/O locked on the filesystem i.e. NFS server went away or 
something else. What state ate the un-killable processes in? (Z, D, S, etc.)


~Seth

Re: [Dovecot] Production server migration from mbox to maildir ( need advice )

[Seth Mattinen]

Proskurin Kirill wrote:
> Frank Bonnet wrote:
>>> I use Debian amd64 with maildir. It works. Nothing really to say about
>>> it. It's never given me any trouble and nobody ever complains about
>>> performance. Easy to restore specific messages from backups when
>>> customers accidentally delete messages.
>>>
>>> ~Seth
>>
>> Thank you Seth :-)
> 
> But with your storage you must think about backup strategies.
> Rsync + maildir = many time
> 

The other option is LVM snapshots or a filesystem that has snapshots
like ZFS. Just don't use ext3 with snapshots on a busy server - it's a
horrible performer because it's block-based. Extent-based ones like xfs
perform much better and you can "freeze" an xfs filesystem so the
snapshot is consistent.

This is about MySQL but it has some good information:
http://www.mysqlperformanceblog.com/2009/02/05/disaster-lvm-performance-in-snapshot-mode/

And this Ars article describes the basic snapshot procedure for those
unfamiliar with it:
http://arstechnica.com/open-source/news/2004/10/linux-20041013.ars

~Seth

Re: [Dovecot] Production server migration from mbox to maildir ( need advice )

[Seth Mattinen]

Frank Bonnet wrote:
> Hello
> 
> I'm going to change the raid array of my mailhub in fews days/weeks and I
> wonder if it would be a good idea to take the opportunity of this migration
> to also change the storage method from mailbox to maildir.
> 
> I've never used maildir on my production server only on small satellites.
> 
> So I really need advice of admins that use maildir format on prod servers.
> 
> My server has approx 3500 users/mailboxes that is approx 450 Gb of data.
> 
> The OS is Debian etch amd64 but could change also during the migration
> Opensuse X86_64, CAOS-NSA, FreeBSD 64 are some options.
> 
> Any infos/experiences welcome, I'll summarize answers.
> 
> Thanks a lot.


I use Debian amd64 with maildir. It works. Nothing really to say about
it. It's never given me any trouble and nobody ever complains about
performance. Easy to restore specific messages from backups when
customers accidentally delete messages.

~Seth

Re: [Dovecot] delivering to folder based on recipient_delimiter?

[Seth Mattinen]

dove...@corwyn.net wrote:
> 
> 
> 
> I'm currently running postfix + dovecot, with virtual users
> 
> In postfix I have set recipient_delimiter = + so that postfix accepts
> mail for u...@example.com, user+...@example.com, and
> user+s...@example.com all deliver to inbox.
> 
> what I would like is to have email directed to user+...@example.com
> delivered to the IMAP folder foo  (and, in the context of amavisd, have
> amavis configured for recipient_delimited, and add
> addr_extension_spam_maps add +spam to each mail identified as spam, and
> thus the mail would go to the spam folder on delivery.
> 
> The postfix guys say I should be able to do this. can someone point me
> to how to implement this with dovecot? (as I don't see how to even start).
> 

I use Sieve to do this. There's probably other ways, too.

http://wiki.dovecot.org/LDA

~Seth

Re: [Dovecot] dbox redesign

[Seth Mattinen]

Timo Sirainen wrote:
> This is about how to implement multiple msgs/file dbox format. The
> current v1.1's one msg/file design would stay pretty much the same and
> it would be compatible with this new design.
> 

Out of curiosity, what's the advantage to going to multiple messages per
file? Wouldn't this have the same problems as mbox?

~Seth

Re: [Dovecot] Replication status

[Seth Mattinen]

Timo Sirainen wrote:
> On Sat, 2009-01-31 at 03:49 -0800, Asheesh Laroia wrote:
>> I remember last year we discussed Dovecot replication a bit on this list, 
>> e.g. http://www.dovecot.org/list/dovecot/2008-May/030446.html .
>>
>> Has there been any development related to this, and can I help at all? The 
>> more I think about the hacks I'm employing to synchronize mail, the more I 
>> realize I want this. So hopefully I can put either some money or some time 
>> where my mouth is.
> 
> Replication is still on my TODO list and I've a bit of code for the
> replication plugin implemented, but it's not very far. Anyway since I
> now started working for Mailtrust they'll decide what the priorities are
> for new Dovecot features. I can't say for sure if/when I'll start coding
> the replication again, but my current understanding is that they're
> interested in it, but some other things are more important to get done
> first.
> 
> So throwing money at me won't help, but nothing of course stops you from
> writing the code :) I can send you the code I have so far if you want
> it.


It's a shame replication isn't a priority - that would be a killer feature.

Something easy I've considered is a transaction model similar to how
MySQL does replication by adding sync calls (delete x, change flag on y,
z is new, etc.) in strategic locations to the maildir storage engine.

~Seth

Re: [Dovecot] pop3_lock_session question

[Seth Mattinen]

Timo Sirainen wrote:
> On Wed, 2009-02-04 at 14:55 -0800, Seth Mattinen wrote:
>> Timo Sirainen wrote:
>>> On Wed, 2009-02-04 at 13:23 -0800, Seth Mattinen wrote:
>>>> Like imapproxy which holds the connection for subsequent requests to
>>>> avoid the short-lived HTTP connection issue. I certainly recommend it
>>>> for squirrelmail installations since squirrelmail can't IDLE the 
>>>> connection.
>>> I've heard imapproxy doesn't help all that much with Dovecot. Do you (or
>>> anyone) have any actual statistics?
>>>
>> It does prevent spawning a separate IMAP process 
> 
> Yes, but I think it's not all that much extra work.

I've never benchmarked it, but I do hate seeing it cycle through
connect/auth/do nothing/disconnect because someone is clicking refresh
as fast as they possibly can for some unknown reason.


>> and running it through AUTH (which executes an SQL call in my case) 
> 
> Enabling auth cache would avoid it.
> 

True, but there's a small risk someone could get locked out of their
mail box for a short time, which isn't acceptable in my environment.

~Seth

Re: [Dovecot] pop3_lock_session question

[Seth Mattinen]

Timo Sirainen wrote:
> On Wed, 2009-02-04 at 13:23 -0800, Seth Mattinen wrote:
>> Like imapproxy which holds the connection for subsequent requests to
>> avoid the short-lived HTTP connection issue. I certainly recommend it
>> for squirrelmail installations since squirrelmail can't IDLE the connection.
> 
> I've heard imapproxy doesn't help all that much with Dovecot. Do you (or
> anyone) have any actual statistics?
> 

It does prevent spawning a separate IMAP process and running it through
AUTH (which executes an SQL call in my case) every time a webmail user
clicks on multiple things like a rabid squirrel with attention deficit
disorder. No hard numbers though.

~Seth

Re: [Dovecot] pop3_lock_session question

[Seth Mattinen]

Justin Krejci wrote:
>>> Could it be some (older?) webmail clients that use pop3 instead of imap?
>> I wouldn't expect a webmail client to hold a pop3 connection open
>> across multiple web requests.  We have standard webmail clients
>> available for customer use, but they use IMAP.  With the frequency
>> we're seeing this problem, I'd expect it's more likely to be something
>> newer or more commonly used.
> 
> Presuming you've been able to identify which users this is affecting I would
> suspect you could go back to those users and determine what clients they are
> connecting with and then interested parties (dovecot devs?) could perform
> further investigation in a lab or whatever to determine what is going on.
> Maybe the client(s) is/are just whacky or there is a bug somewhere.
> 
> You can also track down the source IP addresses which may give you an idea
> as to the client as well. If it is a RIM subnet then you may be able to
> assume it's a blackberry. If the PTR record for the IP is
> webmail.somecompany.com then you can probably contact the company and
> discuss with them. Etc. Some companies may have a proxy or something that is
> attempting to hold the connections open for faster response times maybe
> geared for slow link connections. People do a lot of "interesting" things
> from time to time.
> 

Like imapproxy which holds the connection for subsequent requests to
avoid the short-lived HTTP connection issue. I certainly recommend it
for squirrelmail installations since squirrelmail can't IDLE the connection.

~Seth

Re: [Dovecot] Dovecot optimisation

[Seth Mattinen]

Proskurin Kirill wrote:

Steffen Kaiser пишет:


Does these problems occure all the time, e.g. if you restart the 
Dovecot demon and there are just a bunch of users on it.
After restart - it work for some time, but it is just because no all 
connect to it again.


Do you have a file descriptor limit for the demon process?

System limit? 512mb. It not run on this.


Do you have some "security" stuff running, a BSD-equivalent of SELinux 
or AppArmour?

Nope.

As I sad before - i just increase *all* thing what can be a bottleneck 
twice. And seems to "fix" it.
But it makes me think what it is real will be good if dovecot can say in 
debuging loging what he reach some limits from his config.
It will make finding a bottleneck such easy.. heh. I just what to know - 
were was a problem, but seam to i don`t have any tool for this.


ps ax | grep imap-login | wc -l

~Seth

Re: [Dovecot] Dovecot discards mail over quota

[Seth Mattinen]

Charles Marcus wrote:

On 1/20/2009, Seth Mattinen (se...@rollernet.us) wrote:
It's simply a fact these days that people find backscatter and 
misdirected bounces annoying. In many cases they end up at a forged 
address that had nothing to do with the original transaction. A well 
behaved receiver will make decisions during the SMTP transactions 
with appropriate response codes. It should never accept and generate 
a message later.


Well, its also about the fact that if one engages in such behavior, one
will quickly find one's mailserver/IP address on numerous blacklists
that are a pain to get off of, IF you can get off of them.




Merely a reinforcement by the people who find backscatter unacceptable 
directed at people who can't take a hint. ;)


~Seth

Re: [Dovecot] Dovecot discards mail over quota

[Seth Mattinen]

It's simply a fact these days that people find backscatter and 
misdirected bounces annoying. In many cases they end up at a forged 
address that had nothing to do with the original transaction. A well 
behaved receiver will make decisions during the SMTP transactions with 
appropriate response codes. It should never accept and generate a 
message later.


It's not a matter of mincing what an RFC says or does not say, it's 
about not being a dick on the internet and respecting your virtual 
neighbors. It's like if you were to go through your postal mail box, 
separate the junk mail and throw it into your neighbor's yard.


~Seth

Re: [Dovecot] Dovecot optimisation

[Seth Mattinen]

Proskurin Kirill wrote:

Timo Sirainen wrote:

On Jan 20, 2009, at 6:05 AM, Proskurin Kirill wrote:

But dovecot -n don`t show things like a login_max_processes_count and 
other thing what I think must be a bootle neck.


It doesn't show it, because you're using the default value for it. 
Anyway if you think login processes is the bottleneck, you can try if 
setting login_process_per_connection=no helps. 
http://wiki.dovecot.org/LoginProcess




Problem what I don`t know were is a bottle neck and I don`t know how to 
find it. I turn on mail_debug and all other but it not show anything 
what I look.


I just increes all things what may be was a bottle neck and... seams to 
fix it. But I really want to know what it is was.





Setting "login_process_per_connection = no" is much more scalable for 
large environments. Obviously there is a slight security trade off. You 
can confirm if it's a login bottleneck by looking to see if you have the 
maximum number of "imap-login" processes (default is 128) rather than 
just arbitrarily increasing everything.


~Seth

Re: [Dovecot] deliver rejection bounces

[Seth Mattinen]

Charles Marcus wrote:

On 1/19/2009 11:51 AM, Seth Mattinen wrote:

http://vda.sourceforge.net/


Just realize using this patch will void the postfix warranty (you won't
get much help on the postfix list)...



I've never personally used it so i can't vouch for it (I prefer using a 
pre-queue interface) but just pointing it out.


~Seth