Re: [Dovecot] Design: Adding checksums to index files
On 08/05/13 14:47, Timo Sirainen wrote: I've been planning on adding these for years. Maybe it's about time soon. I guess they could be added already to v2.2, but enabled only by a new setting because it requires file format changes that old Dovecots can't then read. I could probably patch v2.1 also so it is able to at least read the new format without failing. For v2.3 this new format could then be made the default. What would these solve? Pointing out errors in dovecot, operating system, or faulty hardware? Modern file/storage systems checksum the data all the way to the platters.
Re: [Dovecot] Passing data safely in password_key?
On 08/02/2013 02:32 PM, Timo Sirainen wrote: On Mon, 2013-07-29 at 09:22 +0200, Attila Nagy wrote: On 07/28/13 13:49, Attila Nagy wrote: Hi, I would like to convert my custom POP/IMAP proxy to Dovecot's. In this proxy I do more than giving back user name, password and the host and I need extra information. Luckily all of them are available as variables, but more than one comes as user input (like user name and cleartext password) and I'm not sure how to pass them safely. Obviously I would need a separator, which is guaranteed not to show up either in user name and the cleartext password. Should I use escape (%E) here, or is there a better way? Just for the record, this is what I use currently: password_key = dovecot/passdb^MAuth-User: %u^MAuth-Pass: %w^MAuth-Protocol: %s^M Client-IP: %r^M I have no idea what you're talking about. What is password_key? The password that is being sent to the backend IMAP/POP3 server? RTFM? ;) http://wiki2.dovecot.org/AuthDatabase/Dict?highlight=%28password_key%29
Re: [Dovecot] Passing data safely in password_key?
On 07/28/13 13:49, Attila Nagy wrote: Hi, I would like to convert my custom POP/IMAP proxy to Dovecot's. In this proxy I do more than giving back user name, password and the host and I need extra information. Luckily all of them are available as variables, but more than one comes as user input (like user name and cleartext password) and I'm not sure how to pass them safely. Obviously I would need a separator, which is guaranteed not to show up either in user name and the cleartext password. Should I use escape (%E) here, or is there a better way? Just for the record, this is what I use currently: password_key = dovecot/passdb^MAuth-User: %u^MAuth-Pass: %w^MAuth-Protocol: %s^M Client-IP: %r^M
[Dovecot] Passing data safely in password_key?
Hi, I would like to convert my custom POP/IMAP proxy to Dovecot's. In this proxy I do more than giving back user name, password and the host and I need extra information. Luckily all of them are available as variables, but more than one comes as user input (like user name and cleartext password) and I'm not sure how to pass them safely. Obviously I would need a separator, which is guaranteed not to show up either in user name and the cleartext password. Should I use escape (%E) here, or is there a better way?
Re: [Dovecot] How the does "new" autocreate method works?
On 05/23/13 14:08, Timo Sirainen wrote:
On 23.5.2013, at 15.06, Attila Nagy wrote:
On 05/23/13 14:01, Timo Sirainen wrote:
On 23.5.2013, at 14.26, Attila Nagy wrote:
I'm trying to migrate from the deprecated autocreate plugin to the mailbox {
auto }setting without success.
What do I forget, or misunderstand?
I deliver mails via LMTP and log in on IMAP, neither of them create the folders
other than the inbox itself.
The new method is creating the folders lazily to disk. They will be visible in
IMAP session, but they won't be actually created to disk until the folder is
opened.
Your config looks correct to me.
Exactly what I see, but I thought this was an error. Could you please clarify
this somewhere appropriate?
BTW, this is a problem for us, because we have a custom software accessing the
maildir, which won't see these until created.
Would it be possible to set the laziness of this process and provide the
possibility to create the folders on disk?
This changed, because the previous behavior was unnecessarily accessing the
disk all the time at each login. I wasn't really planning on adding the old
behavior back anymore. Maybe you could create the folders when the user is
created?
Very good point, will do.
Thanks.
Re: [Dovecot] How the does "new" autocreate method works?
On 05/23/13 14:01, Timo Sirainen wrote:
On 23.5.2013, at 14.26, Attila Nagy wrote:
I'm trying to migrate from the deprecated autocreate plugin to the mailbox {
auto }setting without success.
What do I forget, or misunderstand?
I deliver mails via LMTP and log in on IMAP, neither of them create the folders
other than the inbox itself.
The new method is creating the folders lazily to disk. They will be visible in
IMAP session, but they won't be actually created to disk until the folder is
opened.
Your config looks correct to me.
Exactly what I see, but I thought this was an error. Could you please
clarify this somewhere appropriate?
BTW, this is a problem for us, because we have a custom software
accessing the maildir, which won't see these until created.
Would it be possible to set the laziness of this process and provide the
possibility to create the folders on disk?
Thanks!
[Dovecot] How the does "new" autocreate method works?
Hi,
I'm trying to migrate from the deprecated autocreate plugin to the
mailbox { auto }setting without success.
What do I forget, or misunderstand?
I deliver mails via LMTP and log in on IMAP, neither of them create the
folders other than the inbox itself.
# doveconf -n
# 2.2.2: /usr/local/etc/dovecot/dovecot.conf
# OS: FreeBSD 9.1-STABLE amd64
auth_cache_negative_ttl = 0
auth_cache_size = 100 M
default_process_limit = 1000
default_vsz_limit = 1 G
disable_plaintext_auth = no
import_environment = LD_PRELOAD
info_log_path = syslog
lda_mailbox_autocreate = yes
lda_mailbox_autosubscribe = yes
lmtp_save_to_detail_mailbox = yes
log_path = /var/log/dovecot-errors.log
mail_gid = 999
mail_location = maildir:~/Maildir
mail_plugins = " quota"
mail_temp_dir = /data/tmp
mail_uid = 999
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope
encoded-character vacation subaddress comparator-i;ascii-numeric
relational regex imap4flags copy include variables body enotify
environment mailbox date ihave
namespace inbox {
inbox = yes
location =
mailbox Drafts {
auto = subscribe
special_use = \Drafts
}
mailbox Junk {
special_use = \Junk
}
mailbox Sent {
auto = subscribe
special_use = \Sent
}
mailbox "Sent Messages" {
special_use = \Sent
}
mailbox Spam {
auto = subscribe
special_use = \Junk
}
mailbox Trash {
auto = subscribe
special_use = \Trash
}
prefix =
}
passdb {
args = /usr/local/etc/dovecot/master-users
driver = passwd-file
master = yes
pass = yes
}
passdb {
args = /usr/local/etc/dovecot/dovecot-ldap.conf.ext
driver = ldap
}
plugin {
mail_log_events = delete mailbox_delete
mail_log_fields = uid box msgid size flags vsize from subject
quota = maildir:User quota
quota_warning = storage=95%% quota-warning 95 %h
quota_warning2 = storage=80%% quota-warning 80 %h
recipient_delimiter = +
sieve = ~/.dovecot.sieve
sieve_dir = ~/sieve
}
protocols = pop3 imap lmtp
service auth {
unix_listener auth-userdb {
mode = 0600
user = qmailldap
}
}
service lmtp {
inet_listener lmtp {
port = 24
}
user = qmailldap
}
service managesieve-login {
inet_listener sieve {
port = 4190
}
}
service quota-warning {
executable = script /usr/local/quota-warning/quota-warning.sh
unix_listener quota-warning {
user = qmailldap
}
user = qmailldap
}
ssl = no
userdb {
driver = prefetch
}
userdb {
args = /usr/local/etc/dovecot/dovecot-ldap.conf.ext
driver = ldap
}
userdb {
args = /usr/local/etc/dovecot/dovecot-ldap-catchall.conf.ext
driver = ldap
}
verbose_proctitle = yes
protocol lmtp {
mail_plugins = " quota mail_log notify sieve"
}
protocol imap {
mail_plugins = " quota imap_quota mail_log notify"
}
Thanks,
Re: [Dovecot] Non-standard fields?
On 01/04/13 01:01, Timo Sirainen wrote: - are there any plans to utilize these? No. Other Maildir tools would probably break if they were used. Dovecot's extra maildir metadata is placed to dovecot-uidlist file. That's why I'm asking, I don't want to write a tool, which will break with a future Dovecot version. But if you won't introduce these, it's fine. :) Thanks,
[Dovecot] Non-standard fields?
Hi, Non-standard fields are mentioned here: http://wiki2.dovecot.org/MailboxFormat/Maildir and they are stated as currently not used. Questions: - am I right that if they will be used, they will be key=value pairs, like fields in the base filename? Like: 1035478339.27041_118.foo.org,S=1000,W=1030:2,S,X=12,A=something - or are they supposed to be flags, like: 1035478339.27041_118.foo.org,S=1000,W=1030:2,S,ABCD - are there any plans to utilize these? Thanks,
Re: [Dovecot] dsync redesign
On 03/23/12 22:25, Timo Sirainen wrote: In case anyone is interested in reading (and maybe helping!) with a dsync redesign that's intended to fix all of its current problems, here are some possibly incoherent ramblings about it: http://dovecot.org/tmp/dsync-redesign.txt and even if you don't understand that, here's another document disguising as an algorithm class problem :) If anyone has thoughts on how to solve it, would be great: http://dovecot.org/tmp/dsync-redesign-problem.txt It only deals with saving new messages, not expunges/flag changes/etc, but those should be much simpler. Well, dsync is a very useful tool, but with continuous replication it tries to solve a problem which should be handled -at least partially- elsewhere. Storing stuff in plain file systems and duplicating them to another one just doesn't scale. I personally think that Dovecot could gain much more if the amount of work going into fixing or improving dsync would go into making Dovecot to (be able of) use a high scale, distributed storage backend. I know it's much harder, because there are several major differences compared to the "low latency" and consistency problem free local file systems, but its fruits are also sweeter for the long term. :) It would bring Dovecot into the class of open source mail servers where there are currently no contenders. BTW, for the previous question in this topic (are there any nosql dbs supporting application-level conflict resolution?), there are similar solutions (like CouchDB, but having some experiences with it, I wouldn't recommend it for massive mail storage -at least the plain CouchDB product), but I guess you would be better off with designing a schema which doesn't need it at the first time. For example, messages are immutable, so you won't face this issue in this area. And for metadata, maybe the solution is not to store "digested" snapshots of the current metadata (folders, flags, message links for folders etc), but to store the changes happening on the user's mailbox and occasionally aggregate them into a last known good and consistent state. Also, there are other interesting ideas, maybe with real single instance store (splitting mime parts? Storing attachments in plain binary form? This always brings up the question of whether the mail server should modify the mails, can be pretty bad for encrypted/signed stuff). And of course there is always the problem of designing a good, consistent method which is also efficient.
Re: [Dovecot] dsync replication available for testing
On 03/05/12 13:48, Timo Sirainen wrote: On 5.3.2012, at 14.15, Attila Nagy wrote: On 03/04/12 11:44, Timo Sirainen wrote: In dovecot-2.1 hg you can now test dsync-based replication. Everything isn't finished yet, but it appears to work and I've enabled it for my @dovecot.fi mails. Some issues: Do you plan to make it more performant in the future? I mean calling doveadm (and ssh) for every change -even when they are aggregated- seems to be very resource intensive, it won't keep up on a machine with a lot of modifications happening every seconds. Sure the idea is to improve the performance :) There are two ways: 1) Use longer running SSH sessions which dsync more than one user at a time. 2) Use TCP connections instead of SSH. Don't forget about connection pooling to get concurrency. :) There's already concurrency. replication_max_conns (default 10) specifies how many dsyncs can be running concurrently. Good to hear. BTW, despite being somewhat harder to implement, I personally like native connections better. Native = TCP? It's not difficult, probably a few lines of more code since doveadm server can already listening for TCP connections. It doesn't support SSL though. Yes. For large installations there may be some backend channel already (SSL tunnels, IPSec etc), so it seems to be OK. It would be good to have constantly running daemons on both sides to eliminate the high startup/teardown costs. The process startup/teardown costs are pretty low. I'll need to improve dsync's performance at some point though. Actually I pretty much redesigned the whole dsync already, but I'll probably leave that to v2.2. The current design can still be improved. It depends. For a moderately loaded server I get this: # time ssh root@be02 "echo 1" I meant doveadm/dsync costs, ssh startup is rather slow. I see. Running from network makes this worse slightly. Long running processes with long running connections rule. :) Yes, dsync seems to need some optimizations too. :) I've tried previously on one pair of our servers with a higher level of concurrency (8-16 or so, I can't remember), and it couldn't keep up with the changes. The method was similar to yours: - an external library wrote modified user ids to a file - in an endless loop a script picked up those (moved the file) and started parallel dsyncs (on ssh) The runs were longer and longer... dsync doesn't currently take enough advantage of modseqs and send only the changed data. Hm. What is your estimate about the performance capability of the current "best" replication scheme available in Dovecot? I know it's hard to tell, because there are a lot of parameters, but do you think it's good for a real world environment with (10-1000*x :) thousands of users, and a lot of changes? BTW, it would even better to have something scalable as Cassandra, so Dovecout wouldn't have to worry about replication and (read/write) scalability. BTW, we modify the maildirs externally, so this adds a lot of inefficiency here... Definitely doesn't help. I know, we are working on this. :)
Re: [Dovecot] dsync replication available for testing
On 03/05/12 11:08, Timo Sirainen wrote: On 5.3.2012, at 9.25, Attila Nagy wrote: On 03/04/12 11:44, Timo Sirainen wrote: In dovecot-2.1 hg you can now test dsync-based replication. Everything isn't finished yet, but it appears to work and I've enabled it for my @dovecot.fi mails. Some issues: Do you plan to make it more performant in the future? I mean calling doveadm (and ssh) for every change -even when they are aggregated- seems to be very resource intensive, it won't keep up on a machine with a lot of modifications happening every seconds. Sure the idea is to improve the performance :) There are two ways: 1) Use longer running SSH sessions which dsync more than one user at a time. 2) Use TCP connections instead of SSH. Don't forget about connection pooling to get concurrency. :) BTW, despite being somewhat harder to implement, I personally like native connections better. It would be good to have constantly running daemons on both sides to eliminate the high startup/teardown costs. The process startup/teardown costs are pretty low. I'll need to improve dsync's performance at some point though. Actually I pretty much redesigned the whole dsync already, but I'll probably leave that to v2.2. The current design can still be improved. It depends. For a moderately loaded server I get this: # time ssh root@be02 "echo 1" 1 0.000u 0.009s 0:00.30 0.0% 0+0k 0+0io 0pf+0w ICMP echo RTT is 0.878 ms. So the ssh connection adds ~29 ms overhead to each sync request. Yes, dsync seems to need some optimizations too. :) I've tried previously on one pair of our servers with a higher level of concurrency (8-16 or so, I can't remember), and it couldn't keep up with the changes. The method was similar to yours: - an external library wrote modified user ids to a file - in an endless loop a script picked up those (moved the file) and started parallel dsyncs (on ssh) The runs were longer and longer... BTW, we modify the maildirs externally, so this adds a lot of inefficiency here...
Re: [Dovecot] dsync replication available for testing
Hi, On 03/04/12 11:44, Timo Sirainen wrote: In dovecot-2.1 hg you can now test dsync-based replication. Everything isn't finished yet, but it appears to work and I've enabled it for my @dovecot.fi mails. Some issues: - public namespace isn't replicated at all - shared namespace is replicated, but not private mail flags - I've only tested SSH replication setup now, not director replication setup (and director setup is still missing many things) - SSH replication setup uses aggregator process, which isn't really necessary and can probably be avoided in future Do you plan to make it more performant in the future? I mean calling doveadm (and ssh) for every change -even when they are aggregated- seems to be very resource intensive, it won't keep up on a machine with a lot of modifications happening every seconds. It would be good to have constantly running daemons on both sides to eliminate the high startup/teardown costs. (if I understand things correctly) Thanks for working on this.
Re: [Dovecot] FreeBSD maintainer ?
On 02/26/12 08:22, Frank Bonnet wrote: Does the FreeBSD Dovecot's port maintainer read this mailing-list ? If you read it, you may know the answer (depending on which port do you use).
[Dovecot] assertion failed in mail-index.c
Hi,
I have this:
Jan 04 15:55:21 pop3(jfm47): Panic: file mail-index.c: line 257
(mail_index_keyword_lookup_or_create): assertion failed: (*keyword != '\0')
Jan 04 15:55:21 master: Error: service(pop3): child 3391 killed with
signal 6 (core not dumped - set service pop3 { drop_priv_before_exec=yes })
I don't know why this happened, but wouldn't be the "self healing" (seen
in the wiki I think :) kick in here?
I mean it's even better to completely remove the index than dying and
make the mailbox inaccessible.
Thanks,
Re: [Dovecot] doveadm + dsync merging
Hi,
On 12/29/2011 01:35 PM, Timo Sirainen wrote:
doveadm already supports some nice things, such as being able to remotely
launch a doveadm command via TCP socket. It also supports executing a command
for all users or to some specific users using a wildcard. dsync could use these
features, so I merged dsync and doveadm into same binary for v2.1.
I'll still install "dsync" symlink pointing to "doveadm", and running that way
it should be fully backwards compatible with the old dsync binary and its parameters.
I'm mainly now wondering about the command naming for running dsync via
doveadm. Any suggestions?
a) Use "doveadm dsync" prefix, and otherwise keep the names same:
dsync mirror -> doveadm dsync mirror
dsync backup -> doveadm dsync backup
dsync server -> doveadm dsync server (for running dsync remotely via ssh/etc.)
b) Don't have the dsync prefix:
dsync mirror -> doveadm mirror
dsync backup -> doveadm backup
dsync server -> doveadm dsync-server (could be hidden from the doveadm
commands list)
c) Either a) or b), but rename "mirror" to "sync" or "dsync" or "replicate"?
d) Something else?
Slightly different, but it would be good to have a persistently running
daemon which could operate both in server and client mode.
In server mode it would listen on a TCP socket. In client mode it would
accept source and target information via a control socket. The target IP
address and port would be the daemon's listening socket.
Something like this on the server side:
service dsync {
process_limit = 8
client_limit = 8
inet_listener dsync {
port =
}
Then doveadm sync on the "client) could first connect to the local
server (client), which then connects to the remote service on the
server. Eg.:
doveadm sync [-C ] [-m ] [-u ] [-frRv]
mirror | [@]
where user@host should specify the remote user (mailbox user) and host
should read like 1.1.1.1:1234 (IP address|hostname and port where the
dsync service listens. Or a separate port option to allow easier parsing.
Having the client in a persistent setup would allow faster syncs for
repeated invocations. It would be good to have a simple API to trigger
the sync (a simple text protocol on a unix socket, or something) from
outside programs, to avoid calling doveadm.
The next thing would be to follow dovecot logs and do a sync/async
replication. :)
Re: [Dovecot] Maildir "locking"
On 09/15/11 11:43, Timo Sirainen wrote:
On Thu, 2011-09-15 at 11:37 +0200, Attila Nagy wrote:
So you have a proxy that decides what backend server the connections are
redirected to? How about you do it completely without locking with
dsync? Moving between servers works basically the same as converting a
mailbox format, with the difference of "changing mail_location" you
"change backend server".
http://wiki2.dovecot.org/Tools/Dsync#example_converting
Yes, there is a proxy in front of the servers. Is dsync usable with 3rd
party maildir programs? (not only Dovecot uses these mailboxes)
The problems with 3rd party maildir programs come if during the move
they:
- Expunge last message(s) from mailbox. (dsync can't know if it should
add or expunge them, so it plays it safe and adds them back)
- Delete a mailbox. (dsync can't know if it should add or delete it, so
again it just adds it back.)
- Remove subscriptions. (again pretty much the same reason.)
It's probably quite unlikely that they do any of this during the move.
You could even reduce the window by doing:
1. dsync backup
2. dsync backup
3. switch to new server
4. kill all existing connections
5. dsync mirror
The 3-5 steps probably take only a few seconds. The "dsync backup" then
guarantees that the destination server will look exactly like the source
server. ("dsync mirror" is used in step 5, because between steps 3-4
either server can get changes.)
OK, thanks for the info, I will try it out.
Re: [Dovecot] Maildir "locking"
On 09/15/11 10:39, Timo Sirainen wrote: On Thu, 2011-09-15 at 10:25 +0200, Attila Nagy wrote: What is the best way to do this? If there is no such thing currently, would it be hard to implement the sticky bit checking on the root? dovecot-uidlist.lock basically does this. Dovecot comes with maildirlock utility to properly create it. How long would your locks be? They are assumed stale after 2 minutes if you don't update the mtime. Readers will block and if they're still locked after 2 minutes they'll abort (if mtime has been changed). There's also mail_max_lock_timeout setting that changes this wait (you could e.g. lower it only with lmtp). Well, basically "forever" in the sense that I would like to move the mailbox to a different machine, So you have a proxy that decides what backend server the connections are redirected to? How about you do it completely without locking with dsync? Moving between servers works basically the same as converting a mailbox format, with the difference of "changing mail_location" you "change backend server". http://wiki2.dovecot.org/Tools/Dsync#example_converting Yes, there is a proxy in front of the servers. Is dsync usable with 3rd party maildir programs? (not only Dovecot uses these mailboxes)
Re: [Dovecot] Maildir "locking"
On 09/15/11 10:19, Timo Sirainen wrote: On Wed, 2011-09-14 at 13:32 +0200, Attila Nagy wrote: Hello, I'm looking for the alternative of qmail's chmod -t (sticky bit on the maildir root) for Dovecot. What I'm trying to achieve with this lock: - Dovecot lmtp should give back a temporary error (so the email will be deferred and re-delivered later) - all other Dovecot daemons (pop, imap) should work as usual, but should not alter maildir contents (they can modify their own files, like indexes, logs etc) What is the best way to do this? If there is no such thing currently, would it be hard to implement the sticky bit checking on the root? dovecot-uidlist.lock basically does this. Dovecot comes with maildirlock utility to properly create it. How long would your locks be? They are assumed stale after 2 minutes if you don't update the mtime. Readers will block and if they're still locked after 2 minutes they'll abort (if mtime has been changed). There's also mail_max_lock_timeout setting that changes this wait (you could e.g. lower it only with lmtp). Well, basically "forever" in the sense that I would like to move the mailbox to a different machine, so if lmtp waits for the lock to disappear and that happens when the mailbox is deleted, and it will do the delivery, it's a bad thing. Before Dovecot, we've had the following process of mailbox moving: - set the sticky bit on the maildir, so qmail won't deliver into it (will give back 4XX) - start to sync/copy the mailbox to the other machine - if it's over, remove the directory on the source machine So what I'm looking for is a lock method, which makes the mailbox read only, so every modification should "soft" fail (no 500 errors on lmtp). What would be the best for this (moving mailboxes between machines)? BTW, the process can be time consuming, even tens of minutes (lots of mails).
[Dovecot] Maildir "locking"
Hello, I'm looking for the alternative of qmail's chmod -t (sticky bit on the maildir root) for Dovecot. What I'm trying to achieve with this lock: - Dovecot lmtp should give back a temporary error (so the email will be deferred and re-delivered later) - all other Dovecot daemons (pop, imap) should work as usual, but should not alter maildir contents (they can modify their own files, like indexes, logs etc) What is the best way to do this? If there is no such thing currently, would it be hard to implement the sticky bit checking on the root? Thanks,
Re: [Dovecot] Converting CLIENT_MAIL_DATA_MAX_INMEMORY_SIZE to a configurable?
Hi, Sorry for the late answer... On 06/13/11 15:40, Timo Sirainen wrote: On Thu, 2011-06-09 at 20:56 +0200, Attila Nagy wrote: Hi, Currently Dovecot's LMTPd writes incoming emails to mail_temp_dir if it's bigger than 128k. But I would like to spare those unnecessary operations (creating a file, deleting it, writing into it, reading from it, checking whether there is free space and if not, rejecting (temporarily) the message). Memory is cheap, disk IO is not. :) And BTW, on a lot of systems, /tmp is a memory file system already, so there is absolute no need for this. If there's not enough disk space, nowadays the message is read fully into memory instead of tempfailing. Well, that doesn't seem to be the case (or maybe it's caused by other stuff, like pigeonhole?). Dovecot 2.0.13, with a temp dir capable of holding <64k: Filesystem SizeUsed Avail Capacity Mounted on tmpfs64k4.0k 60k 6% /data/tmp Sending a message of 60k succeeds: smtp-source -d -f from@from -l 6 -m 1 -s 1 -S test -t to@to -L -v dovecot:24 /var/tmp/smtp-source: name_mask: all /var/tmp/smtp-source: smtp_stream_setup: maxtime=300 enable_deadline=0 /var/tmp/smtp-source: vstream_tweak_tcp: TCP_MAXSEG 1448 /var/tmp/smtp-source: <<< 220 dovecot Dovecot LMTP ready /var/tmp/smtp-source: LHLO me /var/tmp/smtp-source: <<< 250-dovecot /var/tmp/smtp-source: <<< 250-8BITMIME /var/tmp/smtp-source: <<< 250-ENHANCEDSTATUSCODES /var/tmp/smtp-source: <<< 250 PIPELINING /var/tmp/smtp-source: MAIL FROM: /var/tmp/smtp-source: <<< 250 2.1.0 OK /var/tmp/smtp-source: RCPT TO: /var/tmp/smtp-source: <<< 250 2.1.5 OK /var/tmp/smtp-source: DATA /var/tmp/smtp-source: <<< 354 OK /var/tmp/smtp-source: . /var/tmp/smtp-source: <<< 250 2.0.0 id Saved /var/tmp/smtp-source: QUIT /var/tmp/smtp-source: <<< 221 2.0.0 Client quit While with a bigger message: smtp-source -d -f from@from -l 20 -m 1 -s 1 -S test -t to@to -L -v dovecot:24 /var/tmp/smtp-source: name_mask: all /var/tmp/smtp-source: smtp_stream_setup: maxtime=300 enable_deadline=0 /var/tmp/smtp-source: vstream_tweak_tcp: TCP_MAXSEG 1448 /var/tmp/smtp-source: <<< 220 dovecot Dovecot LMTP ready /var/tmp/smtp-source: LHLO me /var/tmp/smtp-source: <<< 250-dovecot /var/tmp/smtp-source: <<< 250-8BITMIME /var/tmp/smtp-source: <<< 250-ENHANCEDSTATUSCODES /var/tmp/smtp-source: <<< 250 PIPELINING /var/tmp/smtp-source: MAIL FROM: /var/tmp/smtp-source: <<< 250 2.1.0 OK /var/tmp/smtp-source: RCPT TO: /var/tmp/smtp-source: <<< 250 2.1.5 OK /var/tmp/smtp-source: DATA /var/tmp/smtp-source: <<< 354 OK /var/tmp/smtp-source: . /var/tmp/smtp-source: <<< 451 4.3.0 Temporary internal failure /var/tmp/smtp-source: fatal: end of data rejected: 451 4.3.0 Temporary internal failure When I give a bigger tmp filesystem to it, it accepts the message. Also are you sure that writing to the file actually produces disk I/O? It depends. On a tmpfs file system, it is possible, if there is not enough memory and the system must page. Pretty bad condition. Of course this is mostly the same with no temporary files (holding the emails in memory). Well, mostly, because you don't duplicate all e-mails in memory. And if emails come and go in the range of some hundred Mbps, this can count. Also, a file in tmpfs possibly requires more memory than the same message in an efficient memory structure (a c string for example, which has only a small metadata, compared to tmpfs). If the tmp directory is not a tmpfs, it depends on whether you commit the written bits (I guess you don't fsync it, why would you :) and whether the file system wants to write them. There are file systems, which can't handle blocks belonging to different files independently with fsync. So if you fsync a small file, and you have written 3 GB to the temporary dir (let's assume they are on the same FS), which you will delete in the next second and you haven't fsynced them, 3 GB plus the small file will be written (to the log). Of course you can (and will) separate the temporary file system, which alleviates this problem. But even then it will be possible that the bits will written, for example because the file system's "commit time" has come and see the above, it may write out a lot of stuff. Even if /tmp isn't a memory filesystem, I think there's a good chance that the file will be gone before any disk writes have a chance to start. Can you see some measurable disk I/O change by changing this value? I can't really measure it now, because I don't have a separate disk pool for temporary files (because nothing uses /tmp, so it would be useless, all resources are delegated to the main pool) and I use tmpfs. But even it's just a few IOPS and some wasted CPU cycles, why wouldn't I set that? :) I think it would be nice to have this as a configurable option, so there would be no need to rebuild every time.
[Dovecot] Converting CLIENT_MAIL_DATA_MAX_INMEMORY_SIZE to a configurable?
Hi,
Currently Dovecot's LMTPd writes incoming emails to mail_temp_dir if
it's bigger than 128k. But I would like to spare those unnecessary
operations (creating a file, deleting it, writing into it, reading from
it, checking whether there is free space and if not, rejecting
(temporarily) the message). Memory is cheap, disk IO is not. :)
And BTW, on a lot of systems, /tmp is a memory file system already, so
there is absolute no need for this.
I only fired two greps so far before writing this mail, in the hope that
I can spare writing, testing and sending a patch, which will be either
rejected, or rewritten. :)
So, am I right that the following constant would be needed to be
converted into a configurable setting and the task is done?
static int
client_input_add(struct client *client, const unsigned char *data,
size_t size)
{
if (client->state.mail_data->used + size <=
CLIENT_MAIL_DATA_MAX_INMEMORY_SIZE &&
client->state.mail_data_output == NULL) {
buffer_append(client->state.mail_data, data, size);
return 0;
} else {
return client_input_add_file(client, data, size);
}
}
It could be defaulted to 128k, but the user could set it "unlimited" (0
or -1, depending on the author's mood, 0 and/or -1 being unlimited, or 0
being 0, meaning don't even store a bit -doesn't really make sense to me).
LMTP is mostly protected from the outside world, so I don't see too much
DoS potential here (absolutely not more than in the tmpfs case).
Thanks,
Re: [Dovecot] lda_mailbox_autocreate does not work for lmtp?
On 06/08/11 12:11, Attila Nagy wrote: [a lot of things] Oh crap, it turned out that some binary junk crept into the LMTP sequence I tried with copy-paste...
[Dovecot] lda_mailbox_autocreate does not work for lmtp?
Hi,
I try to deliver into specific folders with the "plus addressing", namely:
rcpt to:
This works only if the folder exists.
If it does not, I get the following error:
rcpt to:
501 5.5.4 Unsupported options
example-config/conf.d/20-lmtp.conf says:
# When recipient address includes the detail (e.g. user+detail), try to save
# the mail to the detail mailbox. See also recipient_delimiter and
# lda_mailbox_autocreate settings.
But it seems it does not work (or I am missing something).
Current config (I've also tried to include autocreate plugin into lmtp,
without any success) is below:
# 2.0.13: /usr/local/etc/dovecot/dovecot.conf
# OS: FreeBSD 8.2-STABLE amd64
auth_cache_negative_ttl = 0
auth_cache_size = 100 M
auth_cache_ttl = 1 days
disable_plaintext_auth = no
info_log_path = syslog
lda_mailbox_autocreate = yes
lmtp_save_to_detail_mailbox = yes
log_path = /var/log/dovecot-errors.log
mail_fsync = never
mail_gid = 999
mail_location = maildir:~/Maildir
mail_plugins = " quota"
mail_uid = 999
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope
encoded-character vacation subaddress comparator-i;ascii-numeric
relational regex imap4flags copy include variables body enotify
environment mailbox date
passdb {
args = /usr/local/etc/dovecot/master-users
driver = passwd-file
master = yes
pass = yes
}
passdb {
args = /usr/local/etc/dovecot/dovecot-ldap.conf.ext
driver = ldap
}
plugin {
autocreate = INBOX.Trash
autocreate2 = INBOX.Drafts
autocreate3 = INBOX.Sent
autocreate4 = INBOX.Spam
autosubscribe = INBOX.Trash
autosubscribe2 = INBOX.Drafts
autosubscribe3 = INBOX.Sent
autosubscribe4 = INBOX.Spam
mail_log_events = delete mailbox_delete
mail_log_fields = uid box msgid size flags vsize from subject
quota = maildir:User quota
recipient_delimiter = +
sieve = ~/.dovecot.sieve
sieve_dir = ~/sieve
}
protocols = pop3 imap lmtp
service anvil {
client_limit = 8192
}
service auth {
client_limit = 8192
unix_listener auth-userdb {
mode = 0600
user = qmailldap
}
}
service imap-login {
client_limit = 1000
process_limit = 100
process_min_avail = 8
service_count = 0
}
service imap {
client_limit = 8
process_limit = 2048
process_min_avail = 16
service_count = 0
}
service lmtp {
inet_listener lmtp {
port = 24
}
user = qmailldap
}
service managesieve-login {
inet_listener sieve {
port = 4190
}
}
service pop3-login {
client_limit = 1000
process_limit = 100
process_min_avail = 8
service_count = 0
}
service pop3 {
client_limit = 8
process_limit = 2048
process_min_avail = 32
service_count = 0
}
ssl = no
userdb {
driver = prefetch
}
userdb {
args = /usr/local/etc/dovecot/dovecot-ldap.conf.ext
driver = ldap
}
verbose_proctitle = yes
protocol lmtp {
mail_plugins = " quota autocreate mail_log notify sieve"
}
protocol imap {
mail_max_userip_connections = 1024
mail_plugins = " quota imap_quota autocreate mail_log notify"
}
protocol pop3 {
mail_max_userip_connections = 1024
mail_plugins = " quota autocreate"
}
Re: [Dovecot] POP3 error
On 03/08/2011 10:37 PM, Robert Schetterer wrote: Am 08.03.2011 21:38, schrieb Attila Nagy: On 03/08/2011 06:51 PM, Charles Marcus wrote: On 2011-03-08 12:42 PM, Thierry de Montaudry wrote: On 08 Mar 2011, at 19:37, Charles Marcus wrote: So... if the httpd process is the one consuming all of the CPU, doesn't it stand to reason that it might be something to do with one of your web apps, and not dovecot? But then why was it fine with 1.1.13, which never had once this problem in 2 years? or is 2.0.9 slower, or consuming more resources to create the problem? You don't see how it might be possible that 2.0.x does something that 1.1.x didn't do that your webmail app might not like, without it being a dovecot bug? I'm not saying it is or it isn't, but I'd look there first - see if an update is available for your webmail app... since you were running an ancient version of dovecot, maybe you're also running an ancient version of it too? I can see similar problems (subject: "Restarting dovecot-auth stops authentication"), on a different OS, and nothing common in the webmail area. I think this is clearly related to Dovecot. It handles load very badly (well, it seems at least on common OS settings), doesn't just slow down, but starts to refuse clients. It seems to be obvious that the interprocess socket communication is where it fails, so this is what needs to be investigated. Sadly, doing this on a machine, which cries for a deep breath already is not always easy. you might upgrade to the latest 2.x code as it might possible your using more stuff then you had in older versions, after all there was a long performance thread on this list , look for it in archives I'm running the latest 2.x code (well, sort of, I haven't upgraded to 2.0.10, because of the LDAP bug, so I have both .9 and .11), I've never run 1.x on these machines. I've run qmail and courier. They are pretty different in their architecture, where these kind of stuff (unix socket communication between persisently running daemons) is not touched, so there can't be a problem, where for example five thousand connections are made in the same moment to a single socket/process. There there will be five thousand forks/execs, which won't fail with connection refused, they will be served as fast as the machine can handle them (modulo available memory/file descriptors/etc of course).
Re: [Dovecot] POP3 error
On 03/08/2011 09:58 PM, Charles Marcus wrote: I think this is clearly related to Dovecot. It handles load very badly Whoa, pardner, fyi, there are many, many installations humming along smoothly. No offense. It may be more correct to say situations, where the OS can't deliver prompt resources to Dovecot, like saturated disk IO and similar stuff. I can't see such problems with moderate load, and maybe there aren't so many installations, which handle a lot of traffic. I don't know. I don't think it's a bug, currently to me it seems to be a tuning/configuration issue. But maybe it's a common design related issue, which is not yet fully explored. (well, it seems at least on common OS settings), doesn't just slow down, but starts to refuse clients. Maybe there is a bug somewhere that only becomes evident under certain circumstances, but it is also possibly due to config problems caused by... Sure.
Re: [Dovecot] POP3 error
On 03/08/2011 06:51 PM, Charles Marcus wrote: On 2011-03-08 12:42 PM, Thierry de Montaudry wrote: On 08 Mar 2011, at 19:37, Charles Marcus wrote: So... if the httpd process is the one consuming all of the CPU, doesn't it stand to reason that it might be something to do with one of your web apps, and not dovecot? But then why was it fine with 1.1.13, which never had once this problem in 2 years? or is 2.0.9 slower, or consuming more resources to create the problem? You don't see how it might be possible that 2.0.x does something that 1.1.x didn't do that your webmail app might not like, without it being a dovecot bug? I'm not saying it is or it isn't, but I'd look there first - see if an update is available for your webmail app... since you were running an ancient version of dovecot, maybe you're also running an ancient version of it too? I can see similar problems (subject: "Restarting dovecot-auth stops authentication"), on a different OS, and nothing common in the webmail area. I think this is clearly related to Dovecot. It handles load very badly (well, it seems at least on common OS settings), doesn't just slow down, but starts to refuse clients. It seems to be obvious that the interprocess socket communication is where it fails, so this is what needs to be investigated. Sadly, doing this on a machine, which cries for a deep breath already is not always easy.
[Dovecot] Dovecot doesn't show any e-mails, while there are a lot in the maildir
Hi,
I've recently seen the following issue. A user logs in with pop3, and
sees zero e-mails. The maildir contains emails both in cur and new.
The mail delivery is done with qmail, and this user have had no logins
before, so it has only the inbox, trash and sent (while dovecot is
configured to create drafts and spam folders too). Because of this, the
user doesn't have any dovecot indexes either.
So, he logs in, and sees nothing. The strange thing is Dovecot doesn't
create any of the above directories nor indexes. After moving the
mailbox to another server, the user can log in, the additional folders
get created and also the indexes appear. pop3 shows the correct amount
of emails.
Moving the mailbox back to the original server shows the same issue: no
emails at all.
Restarting dovecot on the original server resolves the issue.
This is a pretty loaded machine, so I couldn't trace the processes. I
know that this isn't too much, but that's what I have.
The only thing I could identify in the error log is:
Feb 17 12:45:58 pop3(varszr@asdfg): Warning: Created dotlock file's
timestamp is different than current time (1297943148 vs 1297943086):
/home/varszr@asdfg/Maildir/subscriptions
Which may happened due to a restart and maybe a clock setting. But that
was yesterday.
Any ideas?
# 2.0.8: /usr/local/etc/dovecot/dovecot.conf
# OS: FreeBSD 8.2-PRERELEASE amd64
auth_cache_negative_ttl = 0
auth_cache_size = 100 M
auth_cache_ttl = 1 days
disable_plaintext_auth = no
info_log_path = syslog
log_path = /var/log/dovecot-errors.log
mail_fsync = never
mail_gid = 999
mail_location = maildir:~/Maildir
mail_plugins = " quota"
mail_uid = 999
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope
encoded-character vacation subaddress comparator-i;ascii-numeric
relational regex imap4flags copy include variables body enotify
environment mailbox date
passdb {
args = /usr/local/etc/dovecot/dovecot-ldap.conf.ext
driver = ldap
}
plugin {
autocreate = INBOX.Trash
autocreate2 = INBOX.Drafts
autocreate3 = INBOX.Sent
autocreate4 = INBOX.Spam
autosubscribe = INBOX.Trash
autosubscribe2 = INBOX.Drafts
autosubscribe3 = INBOX.Sent
autosubscribe4 = INBOX.Spam
mail_log_events = delete mailbox_delete
mail_log_fields = uid box msgid size flags vsize from subject
quota = maildir:User quota
}
protocols = pop3 imap lmtp
service anvil {
client_limit = 8192
}
service auth {
client_limit = 8192
unix_listener auth-userdb {
mode = 0600
user = qmailldap
}
}
service imap-login {
client_limit = 1000
process_limit = 100
process_min_avail = 8
service_count = 0
}
service imap {
client_limit = 1
process_limit = 2048
process_min_avail = 16
service_count = 0
}
service lmtp {
inet_listener lmtp {
port = 24
}
user = qmailldap
}
service pop3-login {
client_limit = 1000
process_limit = 100
process_min_avail = 8
service_count = 0
}
service pop3 {
client_limit = 1
process_limit = 2048
process_min_avail = 32
service_count = 0
}
ssl = no
userdb {
driver = prefetch
}
userdb {
args = /usr/local/etc/dovecot/dovecot-ldap.conf.ext
driver = ldap
}
protocol lmtp {
mail_plugins = " quota mail_log notify"
}
protocol imap {
mail_max_userip_connections = 1024
mail_plugins = " quota imap_quota autocreate mail_log notify"
}
protocol pop3 {
mail_max_userip_connections = 1024
mail_plugins = " quota autocreate"
}
Re: [Dovecot] RFC: grouped (f)sync
On 01/05/2011 11:38 PM, Timo Sirainen wrote: On 6.1.2011, at 0.27, Attila Nagy wrote: With this, 10 syncs would happen in every second from Dovecot LDA processes, meaning if a client connects in t=0 it will immediately got the response 250, if a client connects in t=0.05, it will get the response in 50 ms (in an ideal world, where syncing does not take time), and the committed blocks could accumulate for a maximum of 100 ms. In a busy system (where this setting would make sense), it means it would be possible to write more data with less IOPS needed. I guess this could work. Although earlier I thought about delaying fsyncs so that when saving/copying multiple mails in a single transaction with maildir it would delay about 10 or so close()s and then fsync() them all at the same time at the end. This ended up being slower (but I only tested it with a single user - maybe in real world setups it might have worked better). What filesystem was used for this test? If that writes only the involved FD's data with an fsync, the effect is pretty much the same when you issue fsync real time, or serialize them into nearly the same time: the file system will write small amounts of data and issue a flush after each fsync. On a file system, which writes all the dirty data for an fsync (like ZFS does), it may work better, altough only the first fsync would be necessary, with the others you will only risk that other data got into the caches and you make the solution useless with that. That's why I wrote in this case you would need to use sync() instead of fsync(), so this would make this file system independent. Many sync() man pages write these: FreeBSD: BUGS The sync() system call may return before the buffers are completely flushed. Linux: BUGS According to the standard specification (e.g., POSIX.1-2001), sync() schedules the writes, but may return before the actual writing is done. However, since version 1.3.20 Linux does actually wait. (This still does not guarantee data integrity: modern disks have large caches.) But I think the same warning will stand against fsync too. Otherwise, I guess a little experimenting and reading would be needed here. I think for this setting it would be OK to assume some technical knowledge on the users and say: you should only turn this on, if you have a file system, which flushes all dirty buffers for a single fsync for the entire file system. Then you would delay fsyncs for a list of FDs, and issue only one for the list, instead of one for each of the list elements. Or just issue a single sync(). I can see two problems: 1. there is no call for committing a lot of file descriptors in one transaction, so instead of fsync() for each of the modified FDs, a sync() would be needed. sync() writes all buffers to stable storage, which is bad if you have a mixed workload, where there are a lot of non-fsynced data, or other heavy fsync users. But modern file systems, like ZFS will write those back too, so there an fsync(fd) is -AFAIK- mostly equivalent with a sync(pool on which fd is). sync() of course is system wide, so if you have other file systems, those will be synced as well. (this setting isn't for everybody) 2. in a multiprocess environment this would need coordination, so instead of doing fsyncs in distinct processes, there would be one process needed, which does the sync and returns OK for the others, so they can notify the client about the commit to the stable storage. It's possible for you to send the fds to another process via UNIX socket that does fsync() on them. I was also hoping for using lib-fs for at least some mailbox formats at some point (either some modified dbox, or a new one), and for that it would be even easier to add an fsync plugin that does this kind of fsync-transfer to another process. Yes, but as above stated, I don't think it will help, because on a file system, which writes only the given FD's data, it's the same, nothing gained, and on a file system, which flushes all dirty buffers, you will possibly have some dirty buffers between those fsyncs, so it will be the same, IOPS will be the limiting factor.
[Dovecot] RFC: grouped (f)sync
Hello, Currently Dovecot (and any other application, which cares about e-mail delivery) does at least one fsync per mail delivery. Given that hard disk drives have a very limited IOPS, this effectively limits the maximum mail delivery performance to a very low value, under utilizing the available storage IO capacity. Calculating with an average mail size of 50 kB and an average consumer HDD with 120 IOPS, the theoretical mail delivery performance will be 50 kB*120 IOPS=5.85 MBps. But if we could write 500 kB with every transaction, the delivery speed would be nearly 10 times as well. Dovecot have two process models: separate processes for each client connection and an async in-process multiplexing method. This works for each one, albeit the timing is somewhat different. So here's the idea: instead of fsyncing immediately in the LDA (lmtpd) every time when the client says "\r\n.\r\n" after the DATA phase, let's introduce a user settable timer (let's call that sync_delay from now on) and only sync in every sync_delay seconds. This would introduce an up to sync_delay seconds delay in lmtpd returning "250 Ok" to the client, but that's generally not a problem, because in high traffic setups there is a great amount of concurrency, so you could use a lot of client connections easily. Take an example setting of sync_delay = 100 ms. With this, 10 syncs would happen in every second from Dovecot LDA processes, meaning if a client connects in t=0 it will immediately got the response 250, if a client connects in t=0.05, it will get the response in 50 ms (in an ideal world, where syncing does not take time), and the committed blocks could accumulate for a maximum of 100 ms. In a busy system (where this setting would make sense), it means it would be possible to write more data with less IOPS needed. I can see two problems: 1. there is no call for committing a lot of file descriptors in one transaction, so instead of fsync() for each of the modified FDs, a sync() would be needed. sync() writes all buffers to stable storage, which is bad if you have a mixed workload, where there are a lot of non-fsynced data, or other heavy fsync users. But modern file systems, like ZFS will write those back too, so there an fsync(fd) is -AFAIK- mostly equivalent with a sync(pool on which fd is). sync() of course is system wide, so if you have other file systems, those will be synced as well. (this setting isn't for everybody) 2. in a multiprocess environment this would need coordination, so instead of doing fsyncs in distinct processes, there would be one process needed, which does the sync and returns OK for the others, so they can notify the client about the commit to the stable storage. Any opinions on this?
Re: [Dovecot] Restarting dovecot-auth stops authentication
On 12/21/2010 01:31 PM, Timo Sirainen wrote:
On 21.12.2010, at 13.51, Attila Nagy wrote:
On 11/18/2010 06:45 PM, Timo Sirainen wrote:
On Wed, 2010-11-17 at 21:17 +0100, Attila Nagy wrote:
pop3-login: Error: net_connect_unix(pop3) failed: Connection refused
Right. This is the main problem. So the question becomes why is the
connection being refused.
I would really love to solve this now. :)
What connects here to what? pop3-login to the pop3 service?
Yes. For each login.
service imap {
client_limit = 8
This definitely doesn't help. I'm not sure if setting client_limit=1 fixes this
problem entirely or not, but try that first.
It seems it does, at least I haven't got any connection refused errors
since I set client_limit to 1 on the imap and pop3 services.
BTW, the difference is about +10k open file handles, and 2-3 (judging
from munin graphs, which aren't so precise) times "active" memory usage
(this is on FreeBSD).
Are there any chances to solve this, or should I forget running in this
mode until you implement async IO?
Thanks,
Re: [Dovecot] Restarting dovecot-auth stops authentication
On 12/21/2010 06:41 PM, Timo Sirainen wrote:
On 21.12.2010, at 19.37, Attila Nagy wrote:
service imap-login {
client_limit = 8
Why imap-login with this small client_limit? The default should be ok (1000).
Because I think that Dovecot's processes block on IO and not just on distinct
IO operations, but larger tasks, like opening a maildir with a lot of e-mails
without indexes.
*-login processes do no disk I/O.
Oh, I've had problem with authentication previously, and that stuck.
Removed, thanks for noticing.
Am I wrong? Or partly wrong, because it uses blocking IO, but it can multiplex
them, so while one user struggles with the file system to build indexes of his
maildir, an other client in the same process can happily do POP/IMAP stuff?
The rationale is to spread IO (and users) amongst processes, because the OS can
schedule them concurrently, but don't have too many processes, because that
eats a lot of memory and other resources.
This applies to imap/pop3 service, and that's why only client_limit=1 works
well with them for now.
Apart from these errors, it works fine. It would be interesting to see a
response time statistics for both settings, to see how worse it gets
with raising client_limit.
Re: [Dovecot] Restarting dovecot-auth stops authentication
On 12/21/2010 01:31 PM, Timo Sirainen wrote:
On 21.12.2010, at 13.51, Attila Nagy wrote:
On 11/18/2010 06:45 PM, Timo Sirainen wrote:
On Wed, 2010-11-17 at 21:17 +0100, Attila Nagy wrote:
pop3-login: Error: net_connect_unix(pop3) failed: Connection refused
Right. This is the main problem. So the question becomes why is the
connection being refused.
I would really love to solve this now. :)
What connects here to what? pop3-login to the pop3 service?
Yes. For each login.
Yes, that's clear.
service imap-login {
client_limit = 8
Why imap-login with this small client_limit? The default should be ok (1000).
Because I think that Dovecot's processes block on IO and not just on
distinct IO operations, but larger tasks, like opening a maildir with a
lot of e-mails without indexes.
Am I wrong? Or partly wrong, because it uses blocking IO, but it can
multiplex them, so while one user struggles with the file system to
build indexes of his maildir, an other client in the same process can
happily do POP/IMAP stuff?
The rationale is to spread IO (and users) amongst processes, because the
OS can schedule them concurrently, but don't have too many processes,
because that eats a lot of memory and other resources.
service imap {
client_limit = 8
This definitely doesn't help. I'm not sure if setting client_limit=1 fixes this
problem entirely or not, but try that first.
I've already done experiments with that, but had to switch context and
forgot the results. Will do again.
Re: [Dovecot] Restarting dovecot-auth stops authentication
On 11/18/2010 06:45 PM, Timo Sirainen wrote:
On Wed, 2010-11-17 at 21:17 +0100, Attila Nagy wrote:
pop3-login: Error: net_connect_unix(pop3) failed: Connection refused
Right. This is the main problem. So the question becomes why is the
connection being refused.
I would really love to solve this now. :)
What connects here to what? pop3-login to the pop3 service?
My current config is:
# 2.0.8: /usr/local/etc/dovecot/dovecot.conf
# OS: FreeBSD 8.2-PRERELEASE amd64
auth_cache_negative_ttl = 0
auth_cache_size = 100 M
auth_cache_ttl = 1 days
default_process_limit = 2000
disable_plaintext_auth = no
info_log_path = syslog
log_path = /var/log/dovecot-errors.log
mail_fsync = never
mail_gid = 999
mail_location = maildir:~/Maildir
mail_plugins = " quota"
mail_uid = 999
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope
encoded-character vacation subaddress comparator-i;ascii-numeric
relational regex imap4flags copy include variables body enotify
environment mailbox date
passdb {
args = /usr/local/etc/dovecot/dovecot-ldap.conf.ext
driver = ldap
}
plugin {
autocreate = INBOX.Trash
autocreate2 = INBOX.Drafts
autocreate3 = INBOX.Sent
autocreate4 = INBOX.Spam
autosubscribe = INBOX.Trash
autosubscribe2 = INBOX.Drafts
autosubscribe3 = INBOX.Sent
autosubscribe4 = INBOX.Spam
mail_log_events = delete undelete expunge copy mailbox_delete
mailbox_rename flag_change save mailbox_create
mail_log_fields = uid box msgid size flags vsize from subject
quota = maildir:User quota
}
protocols = pop3 imap lmtp
service anvil {
client_limit = 8192
}
service auth {
client_limit = 8192
unix_listener auth-userdb {
mode = 0600
user = qmailldap
}
}
service imap-login {
client_limit = 8
process_min_avail = 16
service_count = 0
}
service imap {
client_limit = 8
process_min_avail = 16
service_count = 0
}
service lmtp {
inet_listener lmtp {
port = 24
}
user = qmailldap
}
service pop3-login {
client_limit = 8
process_min_avail = 16
service_count = 0
}
service pop3 {
client_limit = 8
process_min_avail = 32
service_count = 0
}
ssl = no
userdb {
driver = prefetch
}
userdb {
args = /usr/local/etc/dovecot/dovecot-ldap.conf.ext
driver = ldap
}
protocol lmtp {
mail_plugins = " quota mail_log notify"
}
protocol imap {
mail_max_userip_connections = 1024
mail_plugins = " quota imap_quota autocreate"
}
protocol pop3 {
mail_max_userip_connections = 1024
mail_plugins = " quota autocreate"
}
I've raised pop3's process_min_avail, but I still get these errors.
There is nothing more in the error log.
I've checked, auth's start time is yesterday, so it wasn't restarted. I
guess what remains is the resource limit (client_limit maybe?).
If that happens, there should be a log entry. You can grep process_limit
and client_limit from logs.
Nothing with those.
How does dovecot logs timed out LDAP lookups?
"Connection appears to be hanging, reconnecting"
I usually log errors to a different log file. Normally that file should
be empty, so you can easily see all the errors and warnings that could
be causing problems.
log_path = /var/log/dovecot-errors.log
info_log_path = /var/log/dovecot-info.log
Done that, but I only have these in the errors.log:
Dec 21 12:41:06 pop3-login: Error: net_connect_unix(pop3) failed:
Connection refused
Dec 21 12:41:06 pop3-login: Error: net_connect_unix(pop3) failed:
Connection refused
Dec 21 12:43:29 pop3-login: Error: net_connect_unix(pop3) failed:
Connection refused
Dec 21 12:43:29 pop3-login: Error: net_connect_unix(pop3) failed:
Connection refused
Dec 21 12:43:29 pop3-login: Error: net_connect_unix(pop3) failed:
Connection refused
Dec 21 12:43:29 pop3-login: Error: net_connect_unix(pop3) failed:
Connection refused
Dec 21 12:43:29 pop3-login: Error: net_connect_unix(pop3) failed:
Connection refused
Dec 21 12:43:29 pop3-login: Error: net_connect_unix(pop3) failed:
Connection refused
Dec 21 12:43:29 pop3-login: Error: net_connect_unix(pop3) failed:
Connection refused
Dec 21 12:43:29 pop3-login: Error: net_connect_unix(pop3) failed:
Connection refused
Dec 21 12:43:29 pop3-login: Error: net_connect_unix(pop3) failed:
Connection refused
Dec 21 12:43:29 pop3-login: Error: net_connect_unix(pop3) failed:
Connection refused
Dec 21 12:43:29 pop3-login: Error: net_connect_unix(pop3) failed:
Connection refused
Dec 21 12:43:29 pop3-login: Error: net_connect_unix(pop3) failed:
Connection refused
Dec 21 12:43:29 pop3-login: Error: net_connect_unix(pop3) failed:
Connection refused
Dec 21 12:43:29 pop3-login: Error: net_connect_unix(pop3) failed:
Connection refused
Dec 21 12:43:29 pop3-login: Error: net_connect_unix(pop3) failed:
Connection refused
Dec 21 12:43:29 pop3-login: Error: net_connect_unix(pop3) failed:
Connection refused
Dec 21 12:43:29 pop3-login: Error: net_connect_unix(pop3) failed:
Connecti
Re: [Dovecot] Restarting dovecot-auth stops authentication
On 11/16/10 14:40, Attila Nagy wrote: The Dovecot wiki states that Dovecot's master restarts all died processes, which is good for availability. But when I kill dovecot/auth (to simulate an error condition which happened on a machine), the authentication fails with: Nov 16 14:32:40 be dovecot: imap: Error: net_connect_unix(auth-master) failed: No such file or directory Of course I forgot to tell it's 2.0.6. BTW, sending SIGUSR2 to dovecot/auth doesn't lot anything, while sending SIGHUP logs the "clearing cache" message. The wiki says on USR2 it should log cache statistics.
Re: [Dovecot] Restarting dovecot-auth stops authentication
On 11/17/2010 06:55 PM, Timo Sirainen wrote: On Wed, 2010-11-17 at 13:45 +0100, Attila Nagy wrote: Nov 17 11:42:10 be dovecot: pop3-login: Internal login failure (auth failed, 1 attempts): user=, method=PLAIN, rip=172.28.16.20, lip=172.16.253.13 Something else really should have been logged just before this. An error or a warning. There were a few situations where it might not have, which are fixed by this patch: http://hg.dovecot.org/dovecot-2.0/rev/aec1f1614028 I'm sorry, I was in a rush and I have a lot of logs. pop3-login: Error: net_connect_unix(pop3) failed: Connection refused This precedes all the above log entries. I've checked, auth's start time is yesterday, so it wasn't restarted. I guess what remains is the resource limit (client_limit maybe?). The strange thing is according to tcpdump, dovecot does very few LDAP lookups, and they are answered in a reasonable time, but I haven't done a strict correlation with the above errors. What's the best way to find out what causes this connection refused batches once or twice a day? How does dovecot logs timed out LDAP lookups?
Re: [Dovecot] Restarting dovecot-auth stops authentication
On 11/16/10 18:29, Timo Sirainen wrote:
On Tue, 2010-11-16 at 14:52 +0100, Attila Nagy wrote:
Nov 16 14:32:40 be dovecot: imap: Error: net_connect_unix(auth-master)
failed: No such file or directory
Of course I forgot to tell it's 2.0.6.
2.0.7 fixed this.
Thanks, I've upgraded to it.
BTW, I have these in batches:
Nov 17 11:42:10 be dovecot: pop3-login: Internal login failure (auth
failed, 1 attempts): user=, method=PLAIN, rip=172.28.16.20,
lip=172.16.253.13
Nov 17 11:42:10 be dovecot: pop3-login: Internal login failure (auth
failed, 1 attempts): user=, method=PLAIN, rip=172.28.16.20,
lip=172.16.253.13
[...]
22 from this in the same second, then nothing for hours. This time this
wasn't because the auth process disappeared.
I suspected LDAP errors, but Dovecot is so effective in LDAP caching
that there are no 22 LDAP queries in the same second. How could I figure
out what causes these errors? I don't see any more verbosity in the
source code in the place, where this comes from, and I have pretty much
connections, so doing a verbose log for days isn't an option...
Config:
# 2.0.7: /usr/local/etc/dovecot/dovecot.conf
# OS: FreeBSD 8.1-STABLE amd64
auth_cache_negative_ttl = 0
auth_cache_size = 100 M
auth_cache_ttl = 1 days
default_process_limit = 2000
disable_plaintext_auth = no
mail_fsync = never
mail_gid = 999
mail_location = maildir:~/Maildir
mail_plugins = " quota"
mail_uid = 999
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope
encoded-character vacation subaddress comparator-i;ascii-numeric
relational regex imap4flags copy include variables body enotify
environment mailbox date
passdb {
args = /usr/local/etc/dovecot/dovecot-ldap.conf.ext
driver = ldap
}
plugin {
autocreate = INBOX.Trash
autocreate2 = INBOX.Drafts
autocreate3 = INBOX.Sent
autocreate4 = INBOX.Spam
autosubscribe = INBOX.Trash
autosubscribe2 = INBOX.Drafts
autosubscribe3 = INBOX.Sent
autosubscribe4 = INBOX.Spam
mail_log_events = delete undelete expunge copy mailbox_delete
mailbox_rename flag_change save mailbox_create
mail_log_fields = uid box msgid size flags vsize from subject
quota = maildir:User quota
}
protocols = pop3 imap lmtp
service anvil {
client_limit = 8192
}
service auth {
client_limit = 8192
unix_listener auth-userdb {
mode = 0600
user = qmailldap
}
}
service imap-login {
client_limit = 8
process_min_avail = 16
service_count = 0
vsz_limit = 64 M
}
service imap {
client_limit = 8
process_min_avail = 16
service_count = 0
}
service lmtp {
inet_listener lmtp {
port = 24
}
user = qmailldap
}
service pop3-login {
client_limit = 8
process_min_avail = 16
service_count = 0
}
service pop3 {
client_limit = 8
process_min_avail = 16
service_count = 0
}
ssl = no
userdb {
driver = prefetch
}
userdb {
args = /usr/local/etc/dovecot/dovecot-ldap.conf.ext
driver = ldap
}
protocol lmtp {
mail_plugins = " quota mail_log notify"
}
protocol imap {
mail_max_userip_connections = 1024
mail_plugins = " quota imap_quota autocreate"
}
protocol pop3 {
mail_max_userip_connections = 1024
mail_plugins = " quota autocreate"
}
but the process' size barely grows, regardless the large number of
connections and users:
dovecot 21600 0.9 0.0 32304 14604 ?? S 9:24PM 6:06.91
dovecot/auth
BTW, sending SIGUSR2 to dovecot/auth doesn't lot anything, while sending
SIGHUP logs the "clearing cache" message. The wiki says on USR2 it
should log cache statistics.
Works here:
Nov 16 17:26:25 auth: Info: Authentication cache hits 0/2 (0%)
Nov 16 17:26:25 auth: Info: Authentication cache inserts: positive: 2 95B,
negative: 0 0B
So .. Since SIGHUP works, I don't really know. They should be using
exactly the same code right next to each others. I guess something could
disable SIGUSR2 somewhere somehow. What passdb/userdb do you use?
LDAP.
procstat -i says it's OK:
PID COMM SIG FLAGS
21600 auth HUP --C
21600 auth INT --C
21600 auth QUIT ---
21600 auth ILL ---
21600 auth TRAP ---
21600 auth ABRT ---
[...]
21600 auth USR1 ---
21600 auth USR2 --C
[Dovecot] Restarting dovecot-auth stops authentication
Hi, The Dovecot wiki states that Dovecot's master restarts all died processes, which is good for availability. But when I kill dovecot/auth (to simulate an error condition which happened on a machine), the authentication fails with: Nov 16 14:32:40 be dovecot: imap: Error: net_connect_unix(auth-master) failed: No such file or directory It seems -albeit it gets restarted- dovecot/auth doesn't re-create its socket file. Before: # ls /var/run/dovecot/ anvil auth-worker doveadm-server anvil-auth-penalty config dovecot.conf auth-client dictempty auth-login director-admin lmtp auth-master director-userdb login auth-userdb dns-client master.pid # ps auwx | grep dovecot/auth dovecot 87455 0.0 0.0 20024 3832 ?? S 2:34PM 0:00.01 dovecot/auth # rm /var/run/dovecot/auth-master; kill 87455 # ps auwx | grep dovecot/auth dovecot 88815 0.0 0.0 20024 3776 ?? S 2:36PM 0:00.01 dovecot/auth # ls /var/run/dovecot/ anvil config dovecot.conf anvil-auth-penalty dictempty auth-client director-admin lmtp auth-login director-userdb login auth-userdb dns-client master.pid auth-worker doveadm-server I've deleted the auth-master socket because if I don't, you can't see that it's not re-created. :) Is this a normal behaviour? I understand that killing dovecot/auth is not, but Dovecot could survive this easily, if recreating and re-using the new socket file would work. And loosing dovecot/auth happens sometimes (I don't yet now why). Thanks,
Re: [Dovecot] Dovecot occasionally stops with Fatal: kevent(): Invalid argument
On 11/14/2010 07:12 PM, Timo Sirainen wrote: On 14.11.2010, at 18.11, Timo Sirainen wrote: On 14.11.2010, at 18.11, Timo Sirainen wrote: On 14.11.2010, at 17.52, Attila Nagy wrote: FreeBSD/amd64 8-STABLE, Dovecot 2.0.6, stops in random times with: Nov 14 18:42:14 be dovecot: master: Fatal: kevent(): Invalid argument Apply the attached patch. Of course I forgot that. And it was still wrong. :( OK, now. Compiling and distributing the bits. I'll let you know if it crashed again. Thanks,
[Dovecot] Dovecot occasionally stops with Fatal: kevent(): Invalid argument
Hi, FreeBSD/amd64 8-STABLE, Dovecot 2.0.6, stops in random times with: Nov 14 18:42:14 be dovecot: master: Fatal: kevent(): Invalid argument Any ideas? Thanks,
Re: [Dovecot] v2.0.5 released
On 10/04/2010 04:57 PM, Timo Sirainen wrote: On Mon, 2010-10-04 at 15:57 +0200, Attila Nagy wrote: On 10/04/10 15:48, Timo Sirainen wrote: On Mon, 2010-10-04 at 15:27 +0200, Attila Nagy wrote: It seems there is an FD leak in this version. The number of open file descriptors quickly raise after upgrading from 2.0.4. In which process? Sorry, I had to quickly downgrade, dovecot ate more than 100k descriptors in some minutes... I just thought that there weren't too many changes between 4 and 5 and it's trivial from a diff where those descriptors can leak (maybe the pop optimization, that's what really spins here). Here: http://hg.dovecot.org/dovecot-2.0/rev/2b8b2875af26 I guess it's because you had autocreate plugin enabled and autosubscribe entries. It shouldn't have been writing anything to logs then though, I'll fix that too. Thanks, I will try this. I guess it doesn't deserve a tarball re-rolling.
Re: [Dovecot] v2.0.5 released
On 10/04/10 15:48, Timo Sirainen wrote:
On Mon, 2010-10-04 at 15:27 +0200, Attila Nagy wrote:
It seems there is an FD leak in this version. The number of open file
descriptors quickly raise after upgrading from 2.0.4.
In which process?
Sorry, I had to quickly downgrade, dovecot ate more than 100k
descriptors in some minutes... I just thought that there weren't too
many changes between 4 and 5 and it's trivial from a diff where those
descriptors can leak (maybe the pop optimization, that's what really
spins here).
Of course I will look after it in a more quiet time.
service imap {
client_limit = 8
This might not be the best idea. Connections can sometimes block on
locks and such for a while and it then causes the other connections in
the same process to hang.
I was worried about it too, but so far it seems to handle the load
nicely. BTW, it would be nice to have some basic or advanced statistics
about service times, it would be great to monitor that, and would also
help to size those settings in a real system (finding the balance
between in-process parallelism and the number of worker processes).
Re: [Dovecot] v2.0.5 released
Hi,
It seems there is an FD leak in this version. The number of open file
descriptors quickly raise after upgrading from 2.0.4. (I haven't got
more time to look after the issue, it may be obvious from the diffs)
# dovecot -n (after the downgrade)
# 2.0.4: /usr/local/etc/dovecot/dovecot.conf
# OS: FreeBSD 8.1-STABLE amd64
auth_cache_size = 104857600
auth_cache_ttl = 86400 s
disable_plaintext_auth = no
mail_fsync = never
mail_gid = 999
mail_location = maildir:~/Maildir
mail_plugins = $mail_plugins quota
mail_uid = 999
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope
encoded-character vacation subaddress comparator-i;ascii-numeric
relational regex imap4flags copy include variables body enotify
environment mailbox date
passdb {
args = /usr/local/etc/dovecot/dovecot-ldap.conf.ext
driver = ldap
}
plugin {
autocreate = INBOX.Trash
autocreate2 = INBOX.Drafts
autocreate3 = INBOX.Sent
autocreate4 = INBOX.Spam
autosubscribe = INBOX.Trash
autosubscribe2 = INBOX.Drafts
autosubscribe3 = INBOX.Sent
autosubscribe4 = INBOX.Spam
mail_log_events = delete undelete expunge copy mailbox_delete
mailbox_rename flag_change save mailbox_create
mail_log_fields = uid box msgid size flags vsize from subject
quota = maildir:User quota
}
protocols = pop3 imap lmtp
service auth {
unix_listener auth-userdb {
mode = 0600
user = qmailldap
}
}
service imap-login {
process_min_avail = 16
service_count = 0
}
service imap {
client_limit = 8
process_min_avail = 16
service_count = 0
}
service lmtp {
inet_listener lmtp {
port = 24
}
user = qmailldap
}
service pop3-login {
process_min_avail = 16
service_count = 0
}
service pop3 {
client_limit = 8
process_min_avail = 16
service_count = 0
}
ssl = no
userdb {
driver = prefetch
}
userdb {
args = /usr/local/etc/dovecot/dovecot-ldap.conf.ext
driver = ldap
}
protocol lmtp {
mail_plugins = $mail_plugins mail_log notify
}
protocol imap {
mail_max_userip_connections = 1024
mail_plugins = $mail_plugins imap_quota autocreate
}
protocol pop3 {
mail_max_userip_connections = 1024
mail_plugins = $mail_plugins autocreate
}
On 10/01/10 23:30, Timo Sirainen wrote:
http://dovecot.org/releases/2.0/dovecot-2.0.5.tar.gz
http://dovecot.org/releases/2.0/dovecot-2.0.5.tar.gz.sig
See the "ACL handling bugs" message for more details about the ACL
merging bug.
* acl: Fixed the logic of merging multiple ACL entries. Now it works as
documented, while previously it could have done slightly different
things depending on the order of the entries.
* virtual: Allow opening virtual mailboxes that refer to non-existing
mailboxes. It seems that the benefits of this outweigh the lack of
error message when typoing a mailbox name.
+ Added some disk I/O optimizations to Maildir and index code. They're
especially helpful with short-lived connections like POP3.
+ pop3: Added pop3_fast_size_lookups setting.
- doveconf sometimes failed with complaining about missing ssl_key
setting, causing e.g. dovecot-lda to fail.
- lda: If there's an error in configuration, doveconf didn't exit with
EX_TEMPFAIL as it should have.
- sdbox: Fixed memory leak when copying messages with hard links.
- zlib + sdbox combination didn't work
- zlib: Fixed several crashes, which mainly showed up with mbox.
- quota: Don't crash if user has quota disabled, but plugin loaded.
- doveadm fetch uid was actually returning sequence, not uid.
- v2.0.4's subscription listing ignored (and logged a warning about)
subscriptions=no namespaces' entries in some configurations.
(So listing shared mailboxes' subscriptions could have been broken.)
- acl: Fixed crashing when sometimes listing shared mailboxes via
dict proxy.
Re: [Dovecot] mail_fsync=never doesn't work?
On 09/28/2010 07:06 PM, Timo Sirainen wrote: On Tue, 2010-09-28 at 19:03 +0200, Attila Nagy wrote: On 09/28/2010 05:09 PM, Timo Sirainen wrote: On Tue, 2010-09-28 at 08:29 +0200, Attila Nagy wrote: Sadly, I'm one of them. Currently during the provisioning it's not guaranteed that a user will have a mailbox in the file system, so autocreating it during pop logins is a bless. :) But POP3 doesn't use anything except INBOX, so why do you need to autocreate them there? I'd think it's useful only with IMAP. You mean the four folders, or the inbox itself? For the inbox itself: if there is no such directory, the login will fail and the user will look for errors. INBOX is always auto-created. In earlier versions there were a couple of bugs that caused it to fail though. I didn't know that, but see later... For the other folders: we have a crappy webmail, which messes directly with the maildir, and that also doesn't autocreate the mailboxes, so if the user only logs in via pop and webmail, and I only create inbox, he will miss trash, drafts and sent... Weird, I know. :) Oh well. That also means then that my future autocreate plugin won't work with your webmail, because it doesn't physically create the mailboxes. I hope we can switch to an IMAP based webmail until Dovecot 2.1 is released. :) And I would be more than happy to ditch maildir too for a distributed, replicated, highly available storage.
Re: [Dovecot] mail_fsync=never doesn't work?
On 09/28/2010 05:09 PM, Timo Sirainen wrote: On Tue, 2010-09-28 at 08:29 +0200, Attila Nagy wrote: Sadly, I'm one of them. Currently during the provisioning it's not guaranteed that a user will have a mailbox in the file system, so autocreating it during pop logins is a bless. :) But POP3 doesn't use anything except INBOX, so why do you need to autocreate them there? I'd think it's useful only with IMAP. You mean the four folders, or the inbox itself? For the inbox itself: if there is no such directory, the login will fail and the user will look for errors. For the other folders: we have a crappy webmail, which messes directly with the maildir, and that also doesn't autocreate the mailboxes, so if the user only logs in via pop and webmail, and I only create inbox, he will miss trash, drafts and sent... Weird, I know. :)
Re: [Dovecot] mail_fsync=never doesn't work?
On 09/27/10 23:26, Timo Sirainen wrote: Thanks! But shouldn't "Avoid fsyncing subscriptions file when it doesn't change or if mail_fsync=never." be "Avoid re-writing subscriptions file when it doesn't change and fsyncing it if mail_fsync=never."? I mean (I haven't read the code context) if you know that it does not change, why write? Ah, but I don't know :) The possibilities are: a) Open a temp file and start writing new subscriptions to it. If we notice that the subscription is there already, abort and delete the temp file. This is how it works now. I see, thanks. That's because you have 4 autosubscribe entries. The current code doesn't allow doing multiple subscription changes at once. But anyway, again, the problem here is the autocreate plugin. I hated even adding it to included plugins, but too many people just wanted it. Sadly, I'm one of them. Currently during the provisioning it's not guaranteed that a user will have a mailbox in the file system, so autocreating it during pop logins is a bless. :) But I will keep this in mind and will remove it as soon as I can. (v2.1 way of handling it would be to have kind of virtual mailbox names that are always listed, but actually physically created only when it's opened the first time. That way you could also change them and if user has never actually accessed a removed mailbox it would automatically also disappear.) Sounds good.
Re: [Dovecot] mail_fsync=never doesn't work?
On 09/27/2010 04:52 PM, Timo Sirainen wrote:
On Mon, 2010-09-27 at 16:31 +0200, Attila Nagy wrote:
doveconf -n:
mail_plugins = $mail_plugins quota
One annoying problem about v2.0's doveconf -n output is that it hides
when using multiple var=$var lines.. Did you have more plugins than just
quota enabled?
Yes:
./20-lmtp.conf: mail_plugins = $mail_plugins mail_log notify
./10-mail.conf:mail_plugins = $mail_plugins quota
./20-pop3.conf: mail_plugins = $mail_plugins autocreate
./20-imap.conf: mail_plugins = $mail_plugins imap_quota autocreate
Should I set mail_fsync for only pop and imap instances? Currently I
don't use Dovecot's lmtp or deliver, but will, so there I can't allow
disabling fsync. I don't like setting this to never, but after a
migration from courier, the machine couldn't handle the load with fsync,
so this is a must.
Yeah, either set it to only protocol imap/pop3 {} or alternatively put
mail_fsync=no to protocol lda/lmtp {}.
That's what I want to do, when I'll start to use LMTP.
BTW, just curious, what is the rationale of the two fstats so close
together?
It's because of how the code works internally. It would be too much
trouble to avoid the second call and would make the code uglier. fstat()
is cheap anyway.
Yes, 42 or 24 us according to the trace.
I've searched for some more examples and each of them were like this
one. It seems that Dovecot copies the subscriptions file to this
temporary file and then fsyncs it. And the question also arises: why?
Fixed: http://hg.dovecot.org/dovecot-2.0/rev/4959db811d29
Thanks! But shouldn't "Avoid fsyncing subscriptions file when it doesn't
change or if mail_fsync=never." be "Avoid re-writing subscriptions file
when it doesn't change and fsyncing it if mail_fsync=never."?
I mean (I haven't read the code context) if you know that it does not
change, why write?
Also http://hg.dovecot.org/dovecot-2.0/rev/432208994270 avoids multiple
write() syscalls. Although it still does writes to temp file even when
it's not necessary, but that's again annoyingly too much trouble to
prevent..
Hmm. It may be trouble, but inspecting a user's pop3 login (no fsync
now) starting with the first appearance of subscription.lock, ending
with the last I see more weird stuff:
62900 pop3 1285613658.763692 NAMI
"/home/hm01/user2/Maildir/subscriptions.lock"
62900 pop3 1285613658.763777 RET lstat -1 errno 2 No such file
or directory
[...]
62900 pop3 1285613658.783839 CALL unlink(0x8010b0a40)
62900 pop3 1285613658.783865 NAMI
"/home/hm01/user2/Maildir/subscriptions.lock"
62900 pop3 1285613658.784045 RET unlink 0
I see four(!) rewrites of the subscriptions file between the two. And
also there is 20 ms in between, during that nothing happened, just
creating a temporary file, linking it to subscriptions.lock, reading
subscriptions, writing its conents to the temporary file, then deleting
both the temporary file and subscriptions.lock and this is four times.
(BTW, this is 20ms when virtually nobody uses the server, during the
peak I think it will be more)
Do you see the same, or is this just my config?
Even one unnecessary read&write (which makes a lot more file system
operations, so stresses the kernel in case of a lot of logins) can be
significant with a thousand of logins per second, not talking about four. :)
And more specifically: why in a pop3 server?
Do you have autocreate plugin enabled with autosubscribe plugin
settings? You didn't show your full doveconf output so I have to guess..
Sorry. Yes. Here's the full config:
# 2.0.4: /usr/local/etc/dovecot/dovecot.conf
# OS: FreeBSD 8.1-STABLE amd64
auth_cache_size = 104857600
auth_cache_ttl = 86400 s
disable_plaintext_auth = no
mail_fsync = never
mail_gid = 999
mail_location = maildir:~/Maildir
mail_plugins = $mail_plugins quota
mail_uid = 999
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope
encoded-character vacation subaddress comparator-i;ascii-numeric
relational regex imap4flags copy include variables body enotify
environment mailbox date
passdb {
args = /usr/local/etc/dovecot/dovecot-ldap.conf.ext
driver = ldap
}
plugin {
autocreate = INBOX.Trash
autocreate2 = INBOX.Drafts
autocreate3 = INBOX.Sent
autocreate4 = INBOX.Spam
autosubscribe = INBOX.Trash
autosubscribe2 = INBOX.Drafts
autosubscribe3 = INBOX.Sent
autosubscribe4 = INBOX.Spam
mail_log_events = delete undelete expunge copy mailbox_delete
mailbox_rename flag_change save mailbox_create
mail_log_fields = uid box msgid size flags vsize from subject
quota = maildir:User quota
}
protocols = pop3 imap lmtp
service auth {
unix_listener auth-userdb {
mode = 0600
user = qmailldap
}
}
service imap-login {
process_min_avail = 16
service_count = 0
}
service imap {
client_limit =
Re: [Dovecot] mail_fsync=never doesn't work?
On 09/27/10 15:48, Timo Sirainen wrote:
On Mon, 2010-09-27 at 10:09 +0200, Attila Nagy wrote:
I've tried to set the above in Dovecot 2.0.3, but according to ktrace
(FreeBSD) it still fsync()s a lot (pop3 processes for example).
Is this switch useful at all?
I did a quick test and didn't see any fsyncs. Can you find out what file
it's fsyncing? doveconf -n output could also be useful.
doveconf -n:
# 2.0.4: /usr/local/etc/dovecot/dovecot.conf
# OS: FreeBSD 8.1-STABLE amd64
auth_cache_size = 104857600
auth_cache_ttl = 86400 s
disable_plaintext_auth = no
mail_fsync = never
mail_gid = 999
mail_location = maildir:~/Maildir
mail_plugins = $mail_plugins quota
[...]
Should I set mail_fsync for only pop and imap instances? Currently I
don't use Dovecot's lmtp or deliver, but will, so there I can't allow
disabling fsync. I don't like setting this to never, but after a
migration from courier, the machine couldn't handle the load with fsync,
so this is a must.
The files:
69524 pop3 1285595867.816137 CALL
open(0x801004f08,O_RDWR|O_CREAT|O_EXCL,S_IRUSR|S_IWUSR|S_IRGRP|S_IWGRP|S_IROTH|S_IWOTH)
69524 pop3 1285595867.816153 NAMI
"/home/hm01/16/12/user1/Maildir/temp.hm01.69524.579c9846c8ec2b26"
69524 pop3 1285595867.816238 RET open 9
[...]
69524 pop3 1285595867.816596 CALL fstat(0x9,0x7fffe4a0)
69524 pop3 1285595867.816608 STRU struct stat {dev=2188960719,
ino=263529548, mode=-rw--- , nlink=1, uid=999, gid=999, rdev=0,
atime=1285595867.815852103, stime=1285595867.815852103,
ctime=1285595867.815852103, birthtime=1285595867.815852103, size=0,
blksize=131072, blocks=1, flags=0x0 }
69524 pop3 1285595867.816620 RET fstat 0
[...]
69524 pop3 1285595867.822932 CALL fstat(0x9,0x7fffe570)
69524 pop3 1285595867.822955 STRU struct stat {dev=2188960719,
ino=263529548, mode=-rw--- , nlink=1, uid=999, gid=999, rdev=0,
atime=1285595867.815852103, stime=1285595867.815852103,
ctime=1285595867.815852103, birthtime=1285595867.815852103, size=0,
blksize=131072, blocks=1, flags=0x0 }
69524 pop3 1285595867.822974 RET fstat 0
[...]
69524 pop3 1285595867.823065 CALL write(0x9,0x8016d1200,0x5)
69524 pop3 1285595867.823116 GIO fd 9 wrote 5 bytes
"Trash"
69524 pop3 1285595867.823134 RET write 5
69524 pop3 1285595867.823148 CALL write(0x9,0x8006f5be3,0x1)
69524 pop3 1285595867.823173 GIO fd 9 wrote 1 byte
"
"
69524 pop3 1285595867.823186 RET write 1
69524 pop3 1285595867.823198 CALL write(0x9,0x8016d1206,0x6)
69524 pop3 1285595867.823220 GIO fd 9 wrote 6 bytes
"Drafts"
[...]
69524 pop3 1285595867.82 CALL fsync(0x9)
69524 pop3 1285595868.119406 RET fsync 0
[...]
69524 pop3 1285595868.129605 CALL close(0x9)
69524 pop3 1285595868.129674 RET close 0
BTW, just curious, what is the rationale of the two fstats so close
together?
I've searched for some more examples and each of them were like this
one. It seems that Dovecot copies the subscriptions file to this
temporary file and then fsyncs it. And the question also arises: why?
And more specifically: why in a pop3 server?
Thanks,
[Dovecot] mail_fsync=never doesn't work?
Hello, I've tried to set the above in Dovecot 2.0.3, but according to ktrace (FreeBSD) it still fsync()s a lot (pop3 processes for example). Is this switch useful at all?
[Dovecot] xz compression support
Hello, Just wondering: Dovecot has gzip/bzip2 compression/decompression support. According to this: http://stephane.lesimple.fr/wiki/blog/lzop_vs_compress_vs_gzip_vs_bzip2_vs_lzma_vs_lzma2-xz_benchmark_reloaded gzip and bzip2 aren't really useful together. While gzip can offer speed and a reasonable amount of compression ratio, bzip2 is slow, especially on decompression, which Dovecot does more frequently than compression (1 vs many). So: is it planned to include xz (liblzma) support? It would be useful especially for compressing old mails in the background (due to the resource intensity of the compression phase). Thanks,
Re: [Dovecot] 2.0.3 doesn't start without SSL
On 09/21/2010 05:28 PM, Timo Sirainen wrote: On Tue, 2010-09-21 at 17:21 +0200, Attila Nagy wrote: I've upgraded from 2.0.1 to 2.0.3 and while the former starts, the latter doesn't (the config is the same). It bails with: Sep 21 17:15:32 locsocs dovecot: ssl-params: Fatal: Dovecot built without SSL support Sep 21 17:15:32 locsocs dovecot: master: Error: service(ssl-params): command startup failed, throttling I have ssl=off. Yeah, it's a bug that it logs that with ssl=no. But it shouldn't prevent Dovecot from starting up and working (and I just tested, it doesn't). Damn, you are right. :) Thanks,
[Dovecot] 2.0.3 doesn't start without SSL
Hi, I've upgraded from 2.0.1 to 2.0.3 and while the former starts, the latter doesn't (the config is the same). It bails with: Sep 21 17:15:32 locsocs dovecot: ssl-params: Fatal: Dovecot built without SSL support Sep 21 17:15:32 locsocs dovecot: master: Error: service(ssl-params): command startup failed, throttling I have ssl=off.
[Dovecot] Best way to migrate from qmail-ldap's autoreply?
Hi, qmail-ldap uses four LDAP attributes for handling autoreplies in qmail-local: - deliverymode (to see whether the autoreply should be send) - mailreplytext, which is a base64 encoded multiline string* - mailreplystart and mailreplystop, unix dates I'm wondering how this could be handled with Dovecot (2.0) in the most easier way. All I could find out so far: - patching the code - writing a plugin (it it possible to get these LDAP attributes in the LMTPd with a single userdb-lookup and get it from the plugin during a new mail event? Any examples?) - conditionally provide these values to a sieve script and process them accordingly (the base64 string and maybe the dates seem to be hard for the first glimpse) the mad stuff: - using an LD_PRELOAD library, which emulates per user sieve scripts according to LDAP lookups (gives a lot more flexibility) - an LMTP proxy/server, which does all this and forwards the traffic to the Dovecot LMTPd/deliver process * a sample mailreplytext looks like this: '''%HEADER% Content-type: text/plain; charset="utf-8" távol vagyok''' (of course this is in UTF-8).
Re: [Dovecot] Modifying the underlying maildir externally (webmail, replication)
Tony Rutherford wrote: Timo Sirainen wrote: On 20.1.2010, at 22.21, Attila Nagy wrote: After running through http://wiki.dovecot.org/IndexFiles I'm not sure how well would Dovecot work with other programs modifying the maildirs (adding, deleting, moving messages, folders etc). The "Main index" section says "The index file is synchronized against mailbox only if the syncing information changes.", where syncing information consists or cur and new directories' timestamps. Does that mean I am safe there? Yes. The worst that can happen is that Dovecot doesn't see external changes for 2 seconds. And that's only if your filesystem doesn't support sub-second timestamps. Are the above right, and can Dovecot use its indexes and caches safely with others using the same maildirs? Yes. I've only recently added maildir_very_dirty_syncs=yes that improves performance but makes it work less safely when other programs modify the maildir. Although there is kind of a potential problem if other programs modify the maildir without locking. http://wiki.dovecot.org/MailboxFormat/Maildir#Locking but that isn't unique to Dovecot. That would cause problems with all programs accessing maildir. Dovecot just logs an error about it, instead of silently giving broken information to IMAP clients. We have the exact same configuration, and we had similar concerns. I'm happy to say that we (so far) have been pleasantly surprised by how well Dovecot handles this situation and keeps its index files in synch while other 3rd parties (web, etc.) are changing the Maildirs. It seems very reliable, and we haven't seen any problems. Great to hear that, thanks for sharing!
[Dovecot] Modifying the underlying maildir externally (webmail, replication)
Hello, I have a setup where there are multiple maildir users, one of them being the pop/imap server itself, the others are for example a maildir capable webmail and message replication. I would like to evaluate Dovecot, but I wonder how would its binary indexes and logs fit into this picture. After running through http://wiki.dovecot.org/IndexFiles I'm not sure how well would Dovecot work with other programs modifying the maildirs (adding, deleting, moving messages, folders etc). The "Main index" section says "The index file is synchronized against mailbox only if the syncing information changes.", where syncing information consists or cur and new directories' timestamps. Does that mean I am safe there? The cache file is hopefully just a cache, so it won't contain information from messages which are inserted or moved by an external program, and I assume Dovecot fetches the message lists and other information from the main index, which seems to be OK. The transaction log is for the main index, so if the latter is OK, the former should be OK too. http://wiki.dovecot.org/MailboxFormat/Maildir talks about MUAs and there nothing scary can be found, except for some temporary conditions, where the changes won't propagate to Dovecot in real time, but it's OK. Are the above right, and can Dovecot use its indexes and caches safely with others using the same maildirs? Thanks,
