Batching quota clone updates
Hi, I like the quota count+clone configuration. However, I'm wondering if the cloning process could batch its updates into chunks, or spread them out over time? The clone updates happen whenever the quota is updated, typically this is fine, but when a tsunami of spam comes resulting in delivery to every user on the system, then can be quite overwhelming for a sql server to do an update for every user. Thanks! -- micah
Re: LDA-based vacation responses bouncing
> Thanks for any suggestions, Looks like my virtual_mailbox_domains setting in postfix was not doing the proper lookup, so it thought my domain was local. I've fixed that, so this is no longer an issue.
LDA-based vacation responses bouncing
I've got dovecot vacataion auto-responses working via sieve, but when
the responses are sent, they are sent as if they are to be locally
delivered, which causes them to bounce for domains that I am not local
for. How can I get dovecot's sieve vacation response to check the domain
is local or not?
Here is an example of what happens:
The LDA notices it needs to send a vacation response:
2019-05-14T10:50:47.827466-07:00 brant dovecot:
lda(u...@example.org)<14817>: sieve:
msgid=<20190514135045.012486@localhost>: sent vacation response to
but that response was sent via the postfix pipe through the dovecot lda
transport... because that is local, and the domain it is sending to is
not local, it bounces:
2019-05-14T10:50:47.92-07:00 brant postfix/pipe[14821]: C998381539:
to=, relay=dovecot, delay=0.11, delays=0.04/0.01/0/0.06,
dsn=5.1.1, status=bounced (user unknown)
Postfix is set to do this:
dovecot unix- n n - - pipe
flags=DRhu user=mail argv=/usr/bin/spamc --connect-retries=1 -H -d 10.0.1.92
-s 1024 -t 100 -u ${recipient} -e /usr/lib/dovecot/dovecot-lda -f ${sender}
-d ${recipient}
require ["fileinto", "vacation"];
vacation
:days 2
:subject "Helllo"
"Hi";
Here is my doveconf -n:
# 2.3.4.1 (f79e8e7e4): /etc/dovecot/dovecot.conf
# Pigeonhole version 0.5.4 ()
# OS: Linux 4.9.0-8-amd64 x86_64 Debian 9.8
auth_worker_max_count = 120
default_client_limit = 6144
default_process_limit = 256
default_vsz_limit = 512 M
dict {
expire = mysql:/etc/dovecot/dovecot-dict-sql.conf
quota = mysql:/etc/dovecot/dovecot-dict-sql.conf
}
disable_plaintext_auth = no
first_valid_gid = 8
first_valid_uid = 8
last_valid_gid = 8
last_valid_uid = 8
listen = *
login_greeting = howdy, ready.
login_log_format_elements = user=<%u> method=%m rip=%r lip=%l mpid=%e %c %k
mail_location = mdbox:~/mdbox
mail_plugins = expire quota zlib
maildir_very_dirty_syncs = yes
namespace {
inbox = yes
location =
prefix =
separator = .
}
namespace {
alias_for =
hidden = yes
inbox = no
list = no
location =
prefix = INBOX.
separator = .
}
passdb {
args = /etc/dovecot/dovecot-sql.conf
driver = sql
}
plugin {
expire = Trash
expire2 = Trash/*
expire3 = Spam
expire_dict = proxy::expire
quota = dict:Your mail quota::proxy::quota
quota_rule = *:bytes=100663296
quota_rule2 = Trash:bytes=+20%%
quota_rule3 = Spam:bytes=+10%%
quota_rule4 = INBOX.Trash:bytes=+20%%
quota_rule5 = INBOX.Spam:bytes=+10%%
quota_rule6 = INBOX.restored_from_backups:bytes=+500%%
sieve = ~/.dovecot.sieve
sieve_after = /var/lib/dovecot/sieve/after.sieve
sieve_before = /var/lib/dovecot/sieve/default.sieve
sieve_dir = ~/sieve
zlib_save = gz
zlib_save_level = 6
}
protocols = " imap pop3"
service anvil {
unix_listener anvil {
group = dovecot
mode = 0660
user = root
}
}
service auth {
client_limit = 6144
unix_listener auth-userdb {
group = mail
mode = 0600
user = mail
}
}
service dict {
unix_listener dict {
group = mail
mode = 0600
user = mail
}
}
service imap-login {
process_min_avail = 10
service_count = 0
vsz_limit = 512 M
}
service imap-postlogin {
executable = script-login /usr/local/sbin/postlogin_imap
}
service imap {
executable = imap imap-postlogin
process_limit = 6144
}
service pop3-login {
process_min_avail = 10
service_count = 0
vsz_limit = 512 M
}
service pop3-postlogin {
executable = script-login /usr/local/sbin/postlogin_pop
}
service pop3 {
executable = pop3 pop3-postlogin
}
service stats {
unix_listener stats-writer {
group = mail
mode = 0660
}
}
ssl_cert =
Benchmarking auth
Hello, I'm attempting to use imaptest to test auths/sec on my imap server, to simulate the number I'm getting now on a new server. Based on my readings of dovecot stats outputs, my current machine is doing somewhere between 11.6 and 196 logins/sec at its busiest, and I want to make sure the new server can handle that amount. With imaptest I attempt to simulate this by just simulating login/logout speed, but I'm not sure I understand the results and where things are being blocked. For testing purposes, I set: mail_max_userip_connections = 6000 mail_max_userip_connections = 6000 auth_worker_max_count = 240 If you have some additional suggestions for what I can also tweak here to improve this, I would like to know. I then ran imaptest with: imaptest clients=196 user=test pass=testpw host=127.0.0.1 port=14300 - select=0 seed=123 secs=300 It doesn't seem like I can simulate *only* authentications/logins. I have to pair them with logouts. It makes me wonder how I can simulate the amount of load on my current live system on this system. How do I read the output? I don't really understand what is being shown here: Logi Logo 100% 100% 00 100/100 [51%] does this mean that it tried 100 logins and 100 logouts and 51% succeeded? What are the first two columns? 77 104/121 [61%] 66 131/139 [70%] 33 137/148 (84 stalled >3s) [75%] 84 stalled for more than 3 seconds, how does this relate to the other values presented? 55 155/163 (79 stalled >3s) [83%] 11 166/166 (107 stalled >3s) [84%] 22 168/172 (128 stalled >3s) [87%] 65 179/190 (140 stalled >3s) [96%] 22 22 182/196 (136 stalled >3s) 43 195/196 (133 stalled >3s) 5845 129 ms/cmd avg I really wonder if these first two columns are ms/cmds? It then continues: Logi Logo 100% 100% 10 11 195/196 (131 stalled >3s) 78 194/196 (150 stalled >3s) 33 194/196 (171 stalled >3s) 109 192/196 (166 stalled >3s) 22 22 182/196 (156 stalled >3s) 89 193/196 (153 stalled >3s) - 2 stalled for 16 secs in command: 1 LOGIN "test" "testpw" - 7 stalled for 16 secs in command: 1 LOGIN "test" "testpw" - 8 stalled for 16 secs in command: 1 LOGIN "test" "testpw" - 9 stalled for 16 secs in command: 1 LOGIN "test" "testpw" - 11 stalled for 16 secs in command: 1 LOGIN "test" "testpw" - 12 stalled for 16 secs in command: 1 LOGIN "test" "testpw" - 14 stalled for 16 secs in command: 1 LOGIN "test" "testpw" - 22 stalled for 16 secs in command: 1 LOGIN "test" "testpw" - 25 stalled for 16 secs in command: 1 LOGIN "test" "testpw" ... it starts to build up like this, until I hit control-c twice: - 100 stalled for 17 secs in command: 1 LOGIN "test" "testpw" ^CInfo: Received second SIGINT - stopping immediately 11347 118 ms/cmd avg Totals: Logi Logo 100% 100% 130 134 What are these totals? Thanks! micah
making a plugin encrypt index data
Hello, I'm encrypting mail on disk using a plugin[0], but the index files are not encrypted (specifically the dovecot.index.cache can be read). I want to do is encrypt index on disk, so I'm looking for how a plugin can achieve that by hooking into the right locations. Is that easily possible in a plugin? I can turn off those indexes by passing INDEX=MEMORY, but that isn't possible if I use sdbox/mdbox. thanks for any suggestions! micah 0. https://0xacab.org/riseuplabs/tofu-scrambler
setting service_count results in Permission denied
The load on our machines has gotten too high, so I started to see log
messages that directed me to look at this page for config changes:
https://pad.riseup.net/redirect#http%3A//wiki2.dovecot.org/SocketUnavailable
One of the things I wanted to do on that page was to set the:
service imap { service_count = 100 }
but when I added that to the service imap block, and restarted dovecot,
I see this in the logs: Error:
net_connect_unix(/run/dovecot/imap-postlogin) failed: Permission denied
I dont understand how this impacts permissions. Its clearly this
setting, because when I remove it, things work again. I found nothing
else in the logs indicating a configuration problem.
This is what the config block looked like:
service imap {
# Most of the memory goes to mmap()ing files. You may need to increase this
# limit if you have huge mailboxes.
#vsz_limit = 256M
# Max. number of IMAP processes (connections)
process_limit = 4096
service_count = 100
executable = imap imap-postlogin
}
service imap-postlogin {
# the script process runs as the user specified here (v2.0.14+):
#user = $default_internal_user
executable = script-login /usr/local/sbin/postlogin_imap
# this UNIX socket listener must use the same name as given to imap executable
unix_listener imap-postlogin {
}
}
This is with dovecot 2.2.13, thanks for any help you can provide!
micah
Couldn't set transaction log view (seq 9..13)
Hello, I'm using 2.2.13 on Debian stable, and I perform regular dsync backups. Typically if there is an error, dsync can recover from it the next time it is run, but I've been encountering a new error that doesn't seem to self-heal: # /usr/bin/dsync -u gae backup ssh -i /root/.ssh/id_rsa backmaildir@internal.example /usr/bin/dsync -u micah dsync-remote(micah): Error: /srv/maildirbackups/m/micah/daily.1/mailboxes/INBOX/dbox-Mails/dovecot.index: Couldn't set transaction log view (seq 9..13) dsync-remote(micah): Error: Failed to read transaction log for mailbox INBOX Re-running the dsync never fixes the issue. I'm happy to provide any additional information if so required to help track down this problem! micah
Limit confusion
Hi,
On one of my dovecot servers, i've got a limit problem... and I'm a
little confused about which knobs I should adjust to solve it.
I've been getting both of these:
Error: net_connect_unix(imap) failed: Resource temporarily unavailable -
http://wiki2.dovecot.org/SocketUnavailableess
This one caused me to go read that wiki page, but none of those seem to
apply to me:
. i'm running 2.2.9
. i dont have service imap { client_limit }, although I do have service auth (
client_limit = 1024 }
. my dovecot/auth process shows: dovecot/auth [91 wait, 0 passdb, 0 userdb]
. it doesn't appear the master process is using very much cpu (in fact
the system is mostly low load in general)
. I have service_count = 0 set for imap-login for the 'high performance mode'
dovecot: master: Warning: service(imap): process_limit (2560) reached,
client connections are being dropped
The second is because I have process_limit of 2560 for the imap
service... I can raise that to 4096, but I suspect there are other
limits I need to juggle?
Thanks a lot for any suggestions!
I have ulimit -n 2048 set for the dovecot processes (in
/etc/default/dovecot) and the following is my dovecot -n:
# 2.2.9: /etc/dovecot/dovecot.conf
auth_verbose = yes
default_process_limit = 256
default_vsz_limit = 512 M
dict {
expire = mysql:/etc/dovecot/dovecot-dict-sql.conf
quota = mysql:/etc/dovecot/dovecot-dict-sql.conf
}
disable_plaintext_auth = no
first_valid_gid = 8
first_valid_uid = 8
last_valid_gid = 8
last_valid_uid = 8
listen = *
login_greeting = howdy, ready.
mail_location = mdbox:~/mdbox:INDEX=/srv/dovecot_indexes/%d/%1n/%n
mail_plugins = expire quota zlib
maildir_very_dirty_syncs = yes
namespace {
inbox = yes
location =
prefix =
separator = .
}
namespace {
alias_for =
hidden = yes
inbox = no
list = no
location =
prefix = INBOX.
separator = .
}
passdb {
args = /etc/dovecot/dovecot-sql.conf
driver = sql
}
plugin {
antispam_allow_append_to_spam = yes
antispam_backend = pipe
antispam_debug_target = syslog
antispam_pipe_program = /usr/local/bin/train_spam
antispam_pipe_program_args = -d;10.0.1.109
antispam_pipe_program_notspam_arg = ham
antispam_pipe_program_spam_arg = spam
antispam_pipe_tmpdir = /var/tmp
antispam_signature = X-Spam-Flag
antispam_signature_missing = error
antispam_spam_pattern_ignorecase = SPAM
antispam_trash_pattern = trash;Trash;Deleted *
expire = Trash
expire2 = Trash/*
expire3 = Spam
expire_dict = proxy::expire
quota = dict:Your mail quota::proxy::quota
quota_rule = *:bytes=100663296
quota_rule2 = Trash:bytes=+20%%
quota_rule3 = Spam:bytes=+10%%
quota_rule4 = INBOX.Trash:bytes=+20%%
quota_rule5 = INBOX.Spam:bytes=+10%%
sieve = ~/.dovecot.sieve
sieve_after = /var/lib/dovecot/sieve/after.sieve
sieve_before = /var/lib/dovecot/sieve/default.sieve
sieve_dir = ~/sieve
zlib_save = gz
zlib_save_level = 6
}
protocols = imap pop3
service anvil {
unix_listener anvil {
group = dovecot
mode = 0660
user = root
}
}
service auth {
client_limit = 1024
unix_listener auth-userdb {
group = mail
mode = 0600
user = mail
}
}
service dict {
unix_listener dict {
group = mail
mode = 0600
user = mail
}
}
service imap-login {
process_min_avail = 10
service_count = 0
vsz_limit = 512 M
}
service imap-postlogin {
executable = script-login /usr/local/sbin/postlogin_imap
}
service imap {
executable = imap imap-postlogin
process_limit = 4096
}
service pop3-login {
process_min_avail = 10
service_count = 0
vsz_limit = 512 M
}
service pop3-postlogin {
executable = script-login /usr/local/sbin/postlogin_pop
}
service pop3 {
executable = pop3 pop3-postlogin
}
service stats {
fifo_listener stats-mail {
mode = 0600
user = mail
}
}
ssl_cert = /etc/certs/wildcard/cert.pem
ssl_cipher_list = HIGH:-ADH:-aNULL
ssl_key = /etc/certs/wildcard/key.pem
userdb {
args = /etc/dovecot/dovecot-sql.conf
driver = sql
}
verbose_proctitle = yes
protocol lda {
mail_plugins = expire quota zlib sieve
plugin {
quota_exceeded_message = Sorry, your message cannot be delivered to that
person because their mailbox is full. If you can contact them another way, you
may wish to tell them of this problem.
quota_rule7 = INBOX:bytes=+10%%
}
}
protocol imap {
imap_client_workarounds = delay-newmail tb-extra-mailbox-sep
mail_max_userip_connections = 10
mail_plugins = expire quota zlib imap_quota imap_zlib antispam
plugin {
quota_exceeded_message = You are over quota. To avoid losing mail,
immediately empty your Trash and Sent folders and delete emails with large
attachments.
}
}
protocol pop3 {
mail_plugins = expire quota zlib
pop3_client_workarounds = outlook-no-nuls oe-ns-eoh
pop3_uidl_format = %g
}
[Dovecot] default client_limit
I recently upgraded my dovecot from 2.1 to 2.2, and when I started, I
received this message:
doveconf: Warning: service auth { client_limit=1000 } is lower than required
under max. load (1024)
Searching through my configs, I do not have 1024 set anywhere.
In order to stop this I set client_limit=1024 in my auth{} block... this
seemed odd that the defaults disagreed with each other.
However, it made me realize that my system could probably handle more,
if it were properly tuned, and the different limit settings at their
defaults are probably smaller than they could be for my server and the
load it is under. So, I'm wondering how I can accurately set some of
these values?
The different possible things that can be set are:
default_process_limit (for some reason I have this set to 256)
default_client_limit (default)
default_vsz_limit (I have this set to 512M)
for the imap-login service, I have:
service_count = 0
process_min_avail = 10
vsz_limit = 512M
for service pop3-login, these are set:
service_count = 0
process_min_avail = 10
vsz_limit = 512M
for service imap,
process_limit = 2048
service auth, I've just set the client_limit=1024
I've also got a ulimit -n set to 2048 and in my sysctl.conf, I have the
following:
fs.inotify.max_user_instances = 2048
because I was seeing this in the logs:
Warning: Inotify instance limit for user 8 (UID mail) exceeded, disabling.
Increase /proc/sys/fs/inotify/max_user_instances
What I am looking for is a few things:
. is there a formula for setting a proper value for some of these?
. will dovecot warn me when I've reached a limit, so I can adjust
these?
thanks for any suggestions or improvements, I'm pretty sure this system
can handle a lot more than it is doing right now, I just am unclear on
how to tune it up to do that.
micah
[Dovecot] doveadm mailbox create error codes
Hello, I'm trying to find out what the different 'doveadm mailbox create' error codes are. I've been looking through the source, but I can't seem to find them. I did determine that '65' happens when Error: Can't create mailbox x: Mailbox already exists, but I've also received error 67, and the more mysterious error code 16640. Thanks for any pointers! micah
[Dovecot] Migrating from dovecot 2.1 to 2.2
Due to the incompatibilities with the dsync protocol version between 2.1 and 2.2, I am unable to use dsync to move users to a new machine where we have 2.2.5 setup. This wouldn't normally be a problem, as I could just rsync the user's mdbox and dovecot would happily do the right thing. However, on the new system I've placed the indexes into a faster storage medium, where on the older system the indexes were included in their mail directory. I can rsync over the entire thing, including their old indexes, and dovecot will try to do the right thing and build new indexes, but what happens is all mail is marked as unread, and I get some old index cruft left around in the directories. How can I preserve the seen/unseen flags in such a migration? Thanks! micah
[Dovecot] More fine-grained connection limitations?
There is the mail_max_userip_connections setting, which helps for dealing with number of connections at any given time, but I'm looking for something that will help deal with users who configure their mail clients to connect too frequently. For example, I've seen users who configured their clients to check (IMAP) mail every 3 seconds. This is far too frequent and puts unnecessary load on the server. Even once per minute seems excessive to me. It would be really great if there were a way to tarpit those users to slow them down, or send them an imap message saying they are connecting to often and connections have been disabled for the next 5 minutes or something. Micah ps - what happens to a user when they hit mail_mx_userip_connections? Further connections are just denied, or dropped?
[Dovecot] Dovecot antispam plugin for 2.2
Hello, I recently was looking to update to dovecot 2.2 and I have been happily using the dovecot-antispam plugin in 2.1, but found that there have been some changes in dovecot that make this version not compile against 2.2. I'm wondering if Johannes you have any plans to update it, or if others have managed to figure out what is necessary to get it to work? The particular piece that I am currently stuck on is that the dict_init() in dovecot now requires more arguments than in previous versions, it now requires this: int dict_init(const char *uri, enum dict_data_type value_type, const char *username, const char *base_dir, struct dict **dict_r, const char **error_r); but previously, we only needed four arguments. Now we need to know what to supply for '**dict_r' and '**error_r'. Any advise/help would be appreciated: make[1]: Entering directory `/home/micah/debian/dovecot-antispam/dovecot-antispam-2.0+20120225' cc -DHAVE_CONFIG_H -I/usr/include/dovecot/ -I/usr/include/dovecot/src/ -I/usr/include/dovecot/src/lib/ -I/usr/include/dovecot/src/lib-storage/ -I/usr/include/dovecot/src/lib-mail/ -I/usr/include/dovecot/src/lib-imap/ -I/usr/include/dovecot/src/lib-dict/ -I/usr/include/dovecot/src/lib-index/ -I/usr/include/dovecot/src/imap/ -o dovecot-version dovecot-version.c ./dovecot-version dovecot-version.h CC antispam-storage.o In file included from antispam-storage-2.2.c:20:0, from antispam-storage.c:3: antispam-plugin.h: In function ‘string_dict_init’: antispam-plugin.h:288:2: error: too few arguments to function ‘dict_init’ return dict_init(uri, DICT_DATA_TYPE_STRING, username, NULL); ^ In file included from antispam-plugin.h:13:0, from antispam-storage-2.2.c:20, from antispam-storage.c:3: /usr/include/dovecot/dict.h:34:5: note: declared here int dict_init(const char *uri, enum dict_data_type value_type, ^ micah
Re: [Dovecot] DRAC plugin for Dovecot-2.x
Hello, USUDA Hisashi us...@designet.co.jp writes: Hello All, I released the DRAC plugin for dovecot-2.x. http://sourceforge.jp/projects/dovecot2-drac/ It's based the plugin for dovecot-1.1: http://dovecot.org/patches/1.1/drac.c I recently tried to get your DRAC plugin to work for 2.2.5, but it fails to compile as it is looking for network.h include file, and that doesn't appear to exist any longer in 2.2. Do you have plans to make a newer version of this plugin? thanks, micah
Re: [Dovecot] dsync timeout?
Sean Kamath kam...@moltingpenguin.com writes: On Jan 30, 2013, at 3:46 PM, micah anderson mi...@riseup.net wrote: Seems that only the above process was still around and no other dsync processes. I have three machines that all have this happening it seems. I wonder if there is a ssh configuration option I could set to make these die off. If the ssh process isn't sending anything, and just waiting for read()s, and keepalives are turned off, the SSH session might never know the remote side is long gone. . . This time I managed to capture a process that was stuck and look at it from the server side, and the client side: on the server: 2000 19470 0.0 0.0 7512 3816 ?Ss Feb05 0:01 /usr/bin/dsync dsync-server -E -u foo # strace -s 1024 -F -p 19470 Process 19470 attached - interrupt to quit write(2, dsync-remote(foo): Error: mdbox /srv/maildirbackups/foo/daily.1/storage: Duplicate GUID 96860517f68aa94f8b5197f19f0b in m.41:682501 and m.37:653225\n, 167 on the client: root 19001 0.0 0.0 41308 1600 ?SFeb05 0:00 ssh -i /root/.ssh/backmaildir_id_rsa backmaildir@hoopoe-pn /usr/bin/dsync -u foo server # strace -s 1024 -F -p 19001 Process 19001 attached - interrupt to quit select(8, [4], [], NULL, NULL interestingly, now that I've been watching this more, the same users keep getting wedged. When I attempt to do a dsync of that user by hand, I get this: dsync-local(foo): Error: Unexpected reply from server: 13 d2a100118c45d24f760f97f19f0b3561128 \Recent 1353980259 I tried one of the other users that was stuck, and it gave me: dsync-remote(bar): Error: Corrupted dbox file /srv/maildirbackups/bar/daily.1/storage/m.130 (around offset=22532): msg header has bad magic value This looks like there is something corrupted with the dbox for the user on the client side, is there something I can do to repair those? If any data were transmitted, it would discover the remote side is turned off. One thing I am doing is using a ssh controlmaster socket, and if I kill the process on the client's side, the server side process also dies. micah
Re: [Dovecot] dsync timeout?
Sean Kamath kam...@moltingpenguin.com writes: On Jan 30, 2013, at 3:46 PM, micah anderson mi...@riseup.net wrote: Seems that only the above process was still around and no other dsync processes. I have three machines that all have this happening it seems. I wonder if there is a ssh configuration option I could set to make these die off. If the ssh process isn't sending anything, and just waiting for read()s, and keepalives are turned off, the SSH session might never know the remote side is long gone. . . If any data were transmitted, it would discover the remote side is turned off. See man ssh_config and the option TCPKeepAlive. BTW: Since it's not on the command line, it's likely in /etc/ssh_config or /etc/ssh/ssh_config. Or ~/.ssh/config. In /etc/ssh/sshd_config on the server I'm sending to, TCPKeepAlive yes is set. The default on this system, according to the man page, seems to be to have TCPKeepAlive set. Perhaps I should set ServerAliveInterval? micah
[Dovecot] dsync timeout?
I'm using dsync for a regular backup. The backup system flocks so that two cannot run at the same time, which is generally a good thing. The problem is that it seems like dsync sometimes goes off into the weeds and never comes back, leaving a process running and doing nothing forever, hogging the lock and causing my backups never to run again. I just finally figured out that was what was causing the backups not to run on this system was this process: root 17836 0.0 0.0 40888 1600 ?S 2012 0:00 ssh -i /root/.ssh/backmaildir_id_rsa backmaildir@arg /usr/bin/dsync -u foobar server yeah, that has been running since 2012 :( root:/tmp# strace -p 17836 Process 17836 attached - interrupt to quit select(8, [4], [], NULL, NULL very exciting... There doesn't seem to be a timeout in dsync, but perhaps there should be? At this point my only option is to write a cronjob that will look for dsync processes that are over a certain amount of time old and then kill them, after I do that I will need to take a shower because that is a very dirty solution :P thanks for any ideas, or help! micah --
[Dovecot] Error: Internal quota calculation error
Hello, I'm using 2.1.7 with seive and mysql quotas. We had an outage the other day where the database server where quotas are stored was not available for a short period of time. In dovecot land, the following types of errors occured in that scenario: Oct 26 22:19:01 grosbeak dovecot: lda(exam...@riseup.net): Error: Internal quota calculation error Oct 26 22:19:01 grosbeak dovecot: lda(exam...@riseup.net): Error: sieve: msgid=20122132765181x.abcce...@example.com: failed to store into mailbox 'Trash': Internal error occurred. Refer to server log for more information. [2012-10-26 22:19:01] Oct 26 22:19:01 grosbeak dovecot: lda(exam...@riseup.net): Error: sieve: script /maildir/e/example/.dovecot.sieve failed with unsuccessful implicit keep (user logfile /maildir/e/example/.dovecot.sieve.log may reveal additional details) I expect that there would be quota calculation errors as dovecot could not reach the database server, but what worried me was the 'failed to store into mailbox' message from sieve. The 'Trash' mailbox in this particular seive script is the correct location for the message to be filed into, but the worrisome message is the 'failed with unsuccessful implicit keep'. I looked through all the message-ids that reported this error and I found that the messages were properly delivered in the end, so this might be some issue interacting between sieve, dovecot and quota and just causing a scary message that can be ignored? thanks for any information you can provide, dovecot is great! micah -- pgpwF2qsdIGgP.pgp Description: PGP signature
Re: [Dovecot] mailboxes missing after upgrade from 2.1.4 to 2.1.7
Micah Anderson mi...@riseup.net writes: I mentioned this on the #dovecot irc channel, but I thought I would post here so I can provide more details. The basic problem is that when I upgraded from 2.1.4 to 2.1.7, some users no longer are able to see their folders in pine/alpine. The folders are actually there, and are subscribed (according to doveadm) and its possible to do operations on those folders, if you know the name (for example, opening the folder works fine if you know the name). When I downgrade to 2.1.4 again, the folders appear again, so its clearly something that changed between the two versions. I looked through the changelog, and found a few possibile culprits, I think pine/alpine use imapc, so that could be related: I was able to solve this by changing the alpine folder-collection configuration to *not* have /INBOX.[] at the end and instead have just []! micah
[Dovecot] mailboxes missing after upgrade from 2.1.4 to 2.1.7
Hi all,
I mentioned this on the #dovecot irc channel, but I thought I would post
here so I can provide more details.
The basic problem is that when I upgraded from 2.1.4 to 2.1.7, some
users no longer are able to see their folders in pine/alpine. The
folders are actually there, and are subscribed (according to doveadm)
and its possible to do operations on those folders, if you know the
name (for example, opening the folder works fine if you know the name).
When I downgrade to 2.1.4 again, the folders appear again, so its
clearly something that changed between the two versions. I looked
through the changelog, and found a few possibile culprits, I think
pine/alpine use imapc, so that could be related:
. imapc: Use imapc_list_prefix also for listing subscriptions
. Fixed looking up parent mailbox name with different namespace vs. layout
separators.
. imapc: Don't list mailboxes that don't match list patterns.
. layout=fs: Fixed listing mailboxes with prefix=INBOX/
. LIST with mailbox {} settings incorrectly set subscribed-flags.
Here is my dovecot -n output:
# OS: Linux 2.6.32-5-amd64 x86_64 Debian 6.0.5
auth_default_realm = example.net
auth_verbose = yes
default_process_limit = 256
default_vsz_limit = 512 M
dict {
expire = mysql:/etc/dovecot/dovecot-dict-sql.conf
quota = mysql:/etc/dovecot/dovecot-dict-sql.conf
}
disable_plaintext_auth = no
first_valid_gid = 8
first_valid_uid = 8
last_valid_gid = 8
last_valid_uid = 8
listen = *
login_greeting = howdy, ready.
mail_location = mdbox:~/mdbox:INDEX=/srv/dovecot_indexes/%d/%1n/%n
mail_plugins = expire quota zlib stats
maildir_very_dirty_syncs = yes
namespace {
inbox = yes
location =
prefix =
separator = .
}
namespace {
alias_for =
hidden = yes
inbox = no
list = no
location =
prefix = INBOX.
separator = .
}
passdb {
args = /etc/dovecot/dovecot-sql.conf
driver = sql
}
plugin {
antispam_allow_append_to_spam = yes
antispam_backend = pipe
antispam_debug_target = syslog
antispam_pipe_program = /usr/local/bin/train_spam
antispam_pipe_program_args = -d;10.0.1.109
antispam_pipe_program_notspam_arg = ham
antispam_pipe_program_spam_arg = spam
antispam_pipe_tmpdir = /var/tmp
antispam_signature = X-Spam-Flag
antispam_signature_missing = error
antispam_spam_pattern_ignorecase = SPAM
antispam_trash_pattern = trash;Trash;Deleted *
expire = Trash
expire2 = Trash/*
expire3 = Spam
expire_dict = proxy::expire
quota = dict:Your mail quota::proxy::quota
quota_rule = *:bytes=100663296
quota_rule2 = Trash:bytes=+20%%
quota_rule3 = Spam:bytes=+10%%
quota_rule4 = INBOX.Trash:bytes=+20%%
quota_rule5 = INBOX.Spam:bytes=+10%%
quota_rule6 = INBOX.restored_from_backups:bytes=+500%%
sieve = ~/.dovecot.sieve
sieve_after = /var/lib/dovecot/sieve/after.sieve
sieve_before = /var/lib/dovecot/sieve/default.sieve
sieve_dir = ~/sieve
stats_refresh = 30 secs
stats_track_cmds = yes
zlib_save = gz
zlib_save_level = 6
}
postmaster_address = postmas...@example.net
protocols = imap pop3
service anvil {
unix_listener anvil {
group = dovecot
mode = 0660
user = root
}
}
service auth {
unix_listener auth-userdb {
group = mail
mode = 0600
user = mail
}
}
service dict {
unix_listener dict {
group = mail
mode = 0600
user = mail
}
}
service imap-login {
process_min_avail = 10
service_count = 0
vsz_limit = 512 M
}
service imap-postlogin {
executable = script-login /usr/local/sbin/postlogin_imap
}
service imap {
executable = imap imap-postlogin
process_limit = 2048
}
service pop3-login {
process_min_avail = 10
service_count = 0
vsz_limit = 512 M
}
service pop3-postlogin {
executable = script-login /usr/local/sbin/postlogin_pop
}
service pop3 {
executable = pop3 pop3-postlogin
}
service stats {
fifo_listener stats-mail {
mode = 0600
user = mail
}
}
ssl_cert = /etc/certs/wildcard/cert.pem
ssl_cipher_list = HIGH:-ADH:-aNULL
ssl_key = /etc/certs/wildcard/key.pem
userdb {
args = /etc/dovecot/dovecot-sql.conf
driver = sql
}
verbose_proctitle = yes
protocol lda {
mail_plugins = expire quota zlib stats sieve
plugin {
quota_exceeded_message = Sorry, your message cannot be delivered to that
person because their mailbox is full. If you can contact them another way, you
may wish to tell them of this problem.
quota_rule7 = INBOX:bytes=+10%%
}
}
protocol imap {
imap_client_workarounds = delay-newmail tb-extra-mailbox-sep
mail_max_userip_connections = 20
mail_plugins = expire quota zlib stats imap_quota imap_zlib antispam
imap_stats
plugin {
quota_exceeded_message = You are over quota. To avoid losing mail,
immediately empty your Trash and Sent folders and delete emails with large
attachments. Visit https://help.example.net/quota for more help.
}
}
protocol pop3 {
mail_plugins = expire quota zlib stats
pop3_client_workarounds = outlook-no-nuls oe-ns-eoh
pop3_uidl_format = %g
}
As requested by tss in
Re: [Dovecot] Dsync automation
Jitendra Bhaskar jeetuind...@gmail.com writes: I am using dovecot-2.1.3 on centos 5.7. Its running fine. I have two systems for mail server one is for backup. Currently I am running rsync daily. But I want to use dsync. I went through all the http://wiki2.dovecot.org/Tools/Dsync and http://blog.dovecot.org/2012/02/dovecot-clustering-with-dsync-based.htmlbut I couldn't understood how to automatize dsync. Can anybody give me any idea ho to do it ? As far as I can tell, doing a for loop around 'doveadm user \*' is the only way. That will do a dsync of all the users. On the backup server you will need to decide what to do with the data. I've been rotating it on a daily basis to try and get several daily backups, then some weekly, and monthly. I've been working on a backupninja[0] handler to do dsync backups, but its still rough. micah 0. https://labs.riseup.net/code/projects/backupninja --
[Dovecot] 2.1.3: doveadm mailbox delete lost the -s option
Looks like the recently added -s (for unsubscribing) option to doveadm mailbox delete went missing: root@vireo# /usr/bin/doveadm mailbox delete -u micahtest -s restored delete: invalid option -- 's' doveadm mailbox delete [-u user|-A] [-S socket_path] [-s] mailbox [...] although the option still is shown as valid :) micah --
Re: [Dovecot] dsync redesign
Charles Marcus cmar...@media-brokers.com writes: On 2012-03-27 11:47 AM, Micah Anderson mi...@riseup.net wrote: One would be the ability to perform *intelligent* incremental / rotated backups. I can do this now by running a dsync backup operation and then doing manual hardlinking or moving of the backup directories (daily.1, daily.2, weekly.1, monthly.1, etc.), but it would be more intelligent if this were baked into the backup process. There are already numerous tools that do this flawlessly - I've been using rsnapshot (which uses rsync) for this for years. Are you snapshotting your filesystem (using LVM, or SAN, or similar) before doing rsnapshot? Because if you aren't then rsync will not assuredly get everything in a consistent state. I don't know if Timo should be spending his time reinventing the wheel. dsync backup is already here, and it is quite useful. I'm much more interested in dsync working flawlessly to keep one or more secondary servers in sync, and leave backups to backup software. I'm not against that idea, I just have not yet found a good way to use any backup software in such a way to handle large numbers of user's mail. Although, one interesting piece that I am hopeful I'll be able to implement soon (with Timo's professional help) is the ability to easily and automatically map my rsnapshot snapshots directory to a read-only 'Backups' namespace that automatically shows the snapshots by date and time as they are produced. This way users could 'go back in time' anytime they wanted without having to call me... :) Interesting idea, would be a great one to share with the community if you decide to do so. micah
Re: [Dovecot] dsync redesign
Timo Sirainen t...@iki.fi writes: In case anyone is interested in reading (and maybe helping!) with a dsync redesign that's intended to fix all of its current problems, here are some possibly incoherent ramblings about it: thank you for opening this discussion about dsync! besides the problems I've encountered with dsync, there are a couple things that I think would be great to build into the new vision of the protocol. One would be the ability to perform *intelligent* incremental/rotated backups. I can do this now by running a dsync backup operation and then doing manual hardlinking or moving of the backup directories (daily.1, daily.2, weekly.1, monthly.1, etc.), but it would be more intelligent if this were baked into the backup process. Secondly, being able to filter out mailboxes could result in much more efficient syncing. Now there is the capability to operate on only specific mailboxes, but this doesn't scale well when I am trying to backup thousands of users and I want to omit the Spam and Trash folders from the sync. I would have to get a mailbox list of each user, and then iterate over each mailbox for each user, skipping the Spam and Trash folders, forking a new 'dsync backup' for each of their mailboxes, for each user. Lastly, there isn't a good method for restoring backups. I can reverse the backup process, onto the user's live mailbox, but that brings the user into an undesirable state (eg. their mailbox state one day ago). Better would be if their backup could be restored in such a way that the user can resolve the missing pieces manually, as they know best. thanks again for your work on this, from my position dovecot is an amazing piece of software, the only part that seems to have some issues is dsync and I applaud the effort to redesign to fix things! micah
Re: [Dovecot] dovecot 2.1.3 dsync Unexpected finish reply
Micah Anderson mi...@riseup.net writes: dsync-local(u...@example.com): Error: Unexpected finish reply: by ims-d13.mx.aol.com (8.14.1/8.14.1) with ESMTP id q2LEhqXZ017169; dsync-local(u...@example.com): Error: Unexpected reply from server:Wed, 21 Mar 2012 10:43:52 -0400 dsync-local(u...@example.com): Warning: Mailbox changes caused a desync. You may want to run dsync again. I'm also getting similar strange results with my regular dsync backup: dsync-local(u...@example.com): Error: Unexpected reply from server: 0 23bdce147b43674f8e272c449efa1242146 \Recent 1332335848 this is with 2.1.3. micah
[Dovecot] doveadm user -f index
I've configured my mail_location to have a different location for performance reasons so they aren't in the same location as the mail_location. The 'doveadm user -f home' is useful to find where a user's home directory is for various scripting purposes, but I can't seem to find a way to determine the location of the user's indexes. I can do something with the output of dovecot -a to find the mail_location and then look for a configured INDEX, but then I don't have a good way of translating the %d/%1n/%n type string formatters into their values for a user. thanks for any suggestions! micah --
[Dovecot] dovecot 2.1.3 dsync Unexpected finish reply
I've been moving users from one system to another by doing a dsync mirror operation. The first dsync mirror takes some time, because of the number of users involved, so I am doing an initial sync, and then I direct the users to the new location and do a final 'freshening' sync to get any changes that happened during the longer sync. The problem seems to be with this freshening sync, it seems quite fragile breaking with things like: dsync-local(u...@example.com): Error: msg-get failed: box=Spam uid=3034 guid=1ad456015ae9694f083b1d12b59f dsync-local(u...@example.com): Error: msg-get failed: box=INBOX uid=69096 guid=c22b541a71e4694fc9371d12b59f dsync-local(u...@example.com): Error: msg-get failed: box=INBOX uid=69097 guid=4b6d6b13d0e9694f50571d12b59f dsync-local(u...@example.com): Error: msg-get failed: box=INBOX uid=69098 guid=175b1c2e4aea694fc9711d12b59f dsync-local(u...@example.com): Error: msg-get failed: box=INBOX uid=69099 guid=bfb08c1b3bee694f133e1d12b59f dsync-local(u...@example.com): Error: msg-get failed: box=INBOX uid=69100 guid=fa5d630c17ef694fa75f1d12b59f dsync-local(u...@example.com): Error: msg-get failed: box=INBOX uid=69101 guid=7ca96011dcef694f3f041d12b59f dsync-local(u...@example.com): Error: msg-get failed: box=INBOX uid=69102 guid=ef547107eff1694ff9671d12b59f dsync-local(u...@example.com): Error: msg-get failed: box=INBOX uid=69103 guid=5597bc0519f2694f2e701d12b59f dsync-local(u...@example.com): Error: msg-get failed: box=INBOX uid=69104 guid=8336a53a54f5694fb2101d12b59f dsync-local(u...@example.com): Error: msg-get failed: box=INBOX uid=69105 guid=96169d13c8fd694f83181d12b59f dsync-local(u...@example.com): Error: msg-get failed: box=INBOX uid=69106 guid=af21a5183f036a4f26321d12b59f dsync-local(u...@example.com): Error: msg-get failed: box=INBOX uid=69107 guid=d0fde3348e036a4ff4401d12b59f dsync-local(u...@example.com): Error: msg-get failed: box=INBOX uid=69108 guid=4ce01d1a59056a4fee221d12b59f dsync-local(u...@example.com): Error: msg-get failed: box=INBOX uid=69109 guid=497f96066e056a4f32271d12b59f dsync-local(u...@example.com): Error: msg-get failed: box=INBOX uid=69110 guid=ef34f505c0066a4fc26b1d12b59f dsync-local(u...@example.com): Error: msg-get failed: box=INBOX uid=69111 guid=81adcb2c6e076a4f75111d12b59f dsync-local(u...@example.com): Error: msg-get failed: box=INBOX uid=69112 guid=a110841e8a076a4fa2151d12b59f dsync-local(u...@example.com): Error: msg-get failed: box=INBOX uid=69113 guid=60d8e70a970d6a4fae211d12b59f dsync-local(u...@example.com): Error: msg-get failed: box=INBOX uid=69114 guid=7c6cb41572106a4ff13c1d12b59f dsync-local(u...@example.com): Error: msg-get failed: box=INBOX uid=69115 guid=aaf4d32b2f126a4ff2101d12b59f dsync-local(u...@example.com): Error: msg-get failed: box=INBOX uid=69116 guid=ab52f43a58126a4ffd181d12b59f dsync-local(u...@example.com): Error: msg-get failed: box=INBOX uid=69117 guid=eb543a2179186a4fe4581d12b59f dsync-local(u...@example.com): Error: msg-get failed: box=INBOX uid=69118 guid=cd7cb408a12a6a4f27211d12b59f dsync-local(u...@example.com): Error: msg-get failed: box=INBOX uid=69119 guid=2ec02e2ef2326a4f9e111d12b59f dsync-local(u...@example.com): Error: msg-get failed: box=INBOX uid=69120 guid=e7a4552ff8336a4f7f671d12b59f dsync-local(u...@example.com): Error: msg-get failed: box=INBOX uid=69121 guid=0724b023d33a6a4f1b331d12b59f dsync-local(u...@example.com): Error: msg-get failed: box=INBOX uid=69122 guid=9985c91afe3b6a4f12711d12b59f dsync-local(u...@example.com): Error: msg-get failed: box=INBOX uid=69123 guid=9300751b913d6a4f7a401d12b59f dsync-local(u...@example.com): Error: msg-get failed: box=INBOX uid=69124 guid=822ff806ae3f6a4f293b1d12b59f dsync-local(u...@example.com): Error: msg-get failed: box=INBOX uid=69125 guid=eac8ed1f2b426a4f16421d12b59f dsync-local(u...@example.com): Error: msg-get failed: box=INBOX uid=69126 guid=4109561ae3426a4ff2671d12b59f dsync-local(u...@example.com): Error: msg-get failed: box=INBOX uid=69127 guid=30bc832e5e496a4f56361d12b59f dsync-local(u...@example.com): Error: msg-get failed: box=INBOX uid=69128 guid=c0b36410bd4b6a4f102b1d12b59f dsync-local(u...@example.com): Error: msg-get failed: box=INBOX uid=69129 guid=38a9d41a534d6a4ff4021d12b59f dsync-local(u...@example.com): Error: msg-get failed: box=INBOX uid=69130 guid=b8e84d239b4d6a4fd1101d12b59f dsync-local(u...@example.com): Error: msg-get failed: box=INBOX uid=69131 guid=f060ef22154f6a4f2b5c1d12b59f dsync-local(u...@example.com): Error: msg-get failed: box=INBOX uid=69132 guid=e2999c107c4f6a4f5a761d12b59f dsync-local(u...@example.com): Error: msg-get failed: box=INBOX uid=69133 guid=8d09280aae506a4f07351d12b59f dsync-local(u...@example.com): Error: msg-get failed: box=INBOX uid=69134 guid=43d7ec3aa6556a4f963a1d12b59f dsync-local(u...@example.com): Error:
Re: [Dovecot] Antispam plugin not compatible with Dovecot 2.1
Eugene Paskevich eug...@raptor.kiev.ua writes: On Wed, 18 Jan 2012 18:31:49 +0200, Timo Sirainen t...@iki.fi wrote: On Wed, 2012-01-18 at 18:19 +0200, Eugene Paskevich wrote: mailbox.c: In function 'antispam_save_begin': mailbox.c:138:12: error: 'struct mail_save_context' has no member named 'copying' The copying should be changed to copying_via_save. Thank you, Timo. Would #if DOVECOT_IS_GE(2,1) suffice or do I need anything more specific? Where do you expect to find such macro? ;) Hm. Perhaps I should try to add one. Heh. That's Johannes' package private macro... :) I notice that Johannes hasn't made a 2.1 version of the anti-spam plugin, Eugene were you able to build one successfully? If so, would you be willing to share your changes that were required to make it work? thanks, micah
Re: [Dovecot] dot named folders
Robert Schetterer rob...@schetterer.org writes: Am 08.03.2012 17:27, schrieb Micah Anderson: Willie Gillespie wgillespie+dove...@es2eng.com writes: On 03/07/2012 12:43 PM, Micah Anderson wrote: When a user makes a folder called 'x.y' it actually creates a folder called 'x' with a folder called 'y' inside, rather than a folder called 'x.y'. I'm guessing this has to do with an internal folder separator namespace configuration, but I'm a bit confused by how this works. Correct. Similar to how in Linux, I could create a folder mkdir test1/test2 It will create test2 inside of test1. The difference being that IMAP doesn't necessarily need the parent mailbox to exist, where Linux would throw an error if test1/ didn't exist first. So basically, as far as I know, you can't have a folder with a . in the name with the namespaces you have set up. That makes sense, however I'm not sure that I need these namespaces any longer if I no longer am using the maildir format (mdbox). In either case, it seems like the internal folder separator should not be exposed to the user like this. What is happening now is the user gets something other than they expect (a folder within a folder, instead of a folder with a dot in the name) because of some unknown internal configuration. If moving to mdbox is not enough to remove these namespace configurations that cause this, then it would be good if the user was unable to create such a folder, because it was prohibited, rather than creating something other than they expect. micah http://wiki.dovecot.org/Plugins/Listescape may help Interesting, thanks for the pointer, although I think I prefer if users are just prohibited from making a 'folder.withadot' and told that it is prohibited right away, rather than giving them a way to do it. micah --
Re: [Dovecot] mdbox + gzip and rsync
Jean-Daniel Beaubien jd.beaub...@gmail.com writes: After reading the following paragraph from the dovecot doc, I've been wondering how it would affect rsync (when combined with gzip): Expunging a message only decreases the message's refcount. The space is later freed in purge step. This is typically done in a nightly cronjob when there's less disk I/O activity. The purging first finds all files that have refcount=0 mails. Then it goes through each file and copies the refcount0 mails to other mdbox files (to the same files as where newly saved messages would also go), updates the map index and finally deletes the original file. So there is never any overwriting or file truncation. Interesting, so it would be recommended to those using mdbox format to run a 'dovadm purge -A' every night to clean up these unused files? It seems like without this, mail storage usage will just grow infinitely. It does appear that using an rsync backup process for mdbox would not be able to detect this and backups would also grow infinitely. micah --
Re: [Dovecot] dot named folders
Willie Gillespie wgillespie+dove...@es2eng.com writes: On 03/07/2012 12:43 PM, Micah Anderson wrote: When a user makes a folder called 'x.y' it actually creates a folder called 'x' with a folder called 'y' inside, rather than a folder called 'x.y'. I'm guessing this has to do with an internal folder separator namespace configuration, but I'm a bit confused by how this works. Correct. Similar to how in Linux, I could create a folder mkdir test1/test2 It will create test2 inside of test1. The difference being that IMAP doesn't necessarily need the parent mailbox to exist, where Linux would throw an error if test1/ didn't exist first. So basically, as far as I know, you can't have a folder with a . in the name with the namespaces you have set up. That makes sense, however I'm not sure that I need these namespaces any longer if I no longer am using the maildir format (mdbox). In either case, it seems like the internal folder separator should not be exposed to the user like this. What is happening now is the user gets something other than they expect (a folder within a folder, instead of a folder with a dot in the name) because of some unknown internal configuration. If moving to mdbox is not enough to remove these namespace configurations that cause this, then it would be good if the user was unable to create such a folder, because it was prohibited, rather than creating something other than they expect. micah
Re: [Dovecot] seeking advice: dovecot versions; mailbox formats.
Vincent Schut sc...@sarvision.nl writes: Debian currently has dovecot 1.2.15 in its repositories; not that much newer... No, Debian has 1.2.15 in its /stable (squeeze)/ repositories, there are newer versions available in other Debian repositories. micah
[Dovecot] dot named folders
When a user makes a folder called 'x.y' it actually creates a folder
called 'x' with a folder called 'y' inside, rather than a folder called
'x.y'. I'm guessing this has to do with an internal folder separator
namespace configuration, but I'm a bit confused by how this works.
I'm using 2.0.15 with mdbox and this is what I have configured for my
namespaces:
namespace {
separator = .
prefix =
inbox = yes
}
namespace {
separator = .
prefix = INBOX.
inbox = no
hidden = yes
list = no
}
I migrated from courier maildirs, so perhaps I no longer need some of
these now that the conversion is finished?
thanks for any suggestions, I've got my head mixed up on this issue,
micah
--
Re: [Dovecot] expunging
On Sat, 18 Feb 2012 18:52:57 +0200, Timo Sirainen t...@iki.fi wrote: On 17.2.2012, at 20.25, Micah Anderson wrote: /usr/bin/doveadm expunge -A mailbox Trash* savedbefore 21d /usr/bin/doveadm expunge -A mailbox Spam savedbefore 7d /usr/bin/doveadm expunge -A mailbox Sent savedbefore 120d It seems like it does not use the iterate query, but rather it looks into the database at the expires table and rather it iterates over every user mentioned there. This is a problem because I've got multiple dovecot machines with different sets of users on them, using the same table, so that means that its doing stat() calls on each system for every user mentioned there and failing to find the user on the filesystem (because the user is on another system). In these cases it spits out an error: doveadm(user): Info: User no longer exists, skipping Upgrade to v2.1: * Expire plugin: Only go through users listed by userdb iteration. Delete dict rows for nonexistent users, unless expire_keep_nonexistent_users=yes. Great, i'll give that version a spin, thanks! Is there a similar dict row cleanup for quota? micah
Re: [Dovecot] 2.1 upgrade docs?
Marc Perkel m...@perkel.com writes: Just wondering if there's any upgrade docs for 2.1 online? yes there are. (hint: search terms dovecot upgrade)
[Dovecot] expunging
I am having trouble scaling some regular dovecot cleanup operations on our servers. On a daily basis, I'm wanting to do this on each server, which contains its own isolated set of user storage: /usr/bin/doveadm expunge -A mailbox Trash* savedbefore 21d /usr/bin/doveadm expunge -A mailbox Spam savedbefore 7d /usr/bin/doveadm expunge -A mailbox Sent savedbefore 120d but these are a very expensive operations. For example, just doing the Spam expunge takes 30 minutes (or more, depending on the load) of heavy disk operations, on each machine it is run on. There are approximately 20k users on each machine. It seems like it does not use the iterate query, but rather it looks into the database at the expires table and rather it iterates over every user mentioned there. This is a problem because I've got multiple dovecot machines with different sets of users on them, using the same table, so that means that its doing stat() calls on each system for every user mentioned there and failing to find the user on the filesystem (because the user is on another system). In these cases it spits out an error: doveadm(user): Info: User no longer exists, skipping For users that do exist on this system, it seems to do something on the order of 15 stat() calls, at minimum. What are some things I can do to make this less of an expensive process? If I had a shared storage system that each machine used, this would cut down on the resource waste because I'd only need to run the query once, but unfortuntely, that isn't how these systems were designed. thanks for any ideas, tips etc. micah pgpcDBpKXjq4y.pgp Description: PGP signature
[Dovecot] dovecot-antispam: pipe backend with spamc failing to parse args
I'm trying to get the pipe backend to work with the new dovecot-antispam (I was very pleased to see the new version, thanks Johannes!) Before I would run the following over the maildir: spamc -d 10.0.1.109 -L ham $file 21 /dev/null now, i'm using dovecot-antispam and mdbox and have the following configured: antispam_backend = pipe antispam_signature = X-Spam-Flag antispam_signature_missing = error antispam_trash_pattern = trash;Trash;Deleted * antispam_spam_pattern_ignorecase = SPAM antispam_pipe_tmpdir = /var/tmp antispam_pipe_program_spam_arg = -L spam antispam_pipe_program_notspam_arg = -L ham antispam_pipe_program = /usr/bin/spamc antispam_pipe_program_args = -d 10.0.1.109 Now, when I move a message to Spam, dovecot-antispam detects it and attempts to run, but it fails: Dec 14 16:22:45 grosbeak [username 10.0.1.74 uid]: antispam: run program failed with exit code 64 Exit code 64 is spamc's exit code for command line usage error - but I can't see exactly the command that is being constructed by dovecot-antispam to know where the error is. Any ideas? micah -- pgpvqEZKd1stL.pgp Description: PGP signature
Re: [Dovecot] overriding userdb connection host for doveadm-user
On Thu, 08 Dec 2011 06:22:58 +0200, Timo Sirainen t...@iki.fi wrote: On Wed, 2011-12-07 at 13:02 -0500, Micah Anderson wrote: I've got my users in a replicated database setup and dovecot configured with two connect lines in dovecot-sql.conf: connect = host=127.0.0.1 port=3306 dbname=users user=user password=pass connect = host=127.0.0.2 port=3306 dbname=users user=user password=pass this works really well to help balance the load. If you actually have it like that, it should only be using 127.0.0.2, as the second connect setting overrides the first one.. What I actually have is: connect = host=127.0.0.1 port=3306 dbname=users user=user password=pass connect = host=127.0.0.1 port=3307 dbname=users user=user password=pass two different port configurations (I'm running stunnel), rather than two different host entries[0]. The right way would have been: connect = host=127.0.0.1 host=127.0.0.2 port=... So, if I just have two different ports, would the right way be: connect = host=127.0.0.1 port=3306 port=3307 dbname... ? micah 0. When I was anonymizing the db connection information in the original post, I accidentally changed the ip and decided it didn't matter for the question I was asking, so just left it. Now I see that might be confusing. pgpN0jQlsZCIa.pgp Description: PGP signature
[Dovecot] overriding userdb connection host for doveadm-user
I've got my users in a replicated database setup and dovecot configured with two connect lines in dovecot-sql.conf: connect = host=127.0.0.1 port=3306 dbname=users user=user password=pass connect = host=127.0.0.2 port=3306 dbname=users user=user password=pass this works really well to help balance the load. However, when a new user is created, the replication information sometimes has not made it to the slave database before my user creation process tries to send that new user an email about their new email account. If it hasn't and the email gets sent, then it bounces because postfix thinks the user doesn't exist yet. Replication usually happens within seconds, but could be delayed due to problems. I would like a way to see if the new user's information has been replicated yet before sending the email. I was thinking I could use 'doveadm user' to lookup the user, but I am not sure that I can specify one of the specific 'connect' lines in my configuration to check. Is there a way to override that configuration variable to do this test? I tried various incarnations of passing '-o connect=host='... but doveadm user would just ignore them. thanks for any suggestions! micah ps - yes, I can make my user creation process just 'sleep' for an arbitrary amount of time and by that time the replication should have occured, but that is a bit of a shot in the dark, and doesn't handle longer replication delays that could potentially happen. -- pgp5kLjzEjaju.pgp Description: PGP signature
Re: [Dovecot] dovecot-lda quota rule
micah anderson mi...@riseup.net writes:
On Wed, 16 Nov 2011 11:09:12 -0500, micah anderson mi...@riseup.net wrote:
On Tue, 15 Nov 2011 22:27:20 +0200, Timo Sirainen t...@iki.fi wrote:
On Thu, 2011-11-10 at 13:59 -0500, Micah Anderson wrote:
I really like the feature where you can define quota rules with percents
which trigger off of the default values[0] (so you can set the Trash to
allow for 10% more of the user's quota for example).
What I would really love in dovecot would be for the ability to
configure a quota rule for dovecot-lda. I would like to configure things
so we don't bounce emails for users until they are well over quota, the
IMAP quota plugin is a really great way to notify people that they are
over quota because it fails to write to other folders that should be
enough to get people's attention that they need to deal with things, but
bouncing is harsh.
You can give LDA-specific quota settings. Like maybe:
protocol lda {
plugin {
quota_rule2 = INBOX:storage=200%
}
}
I just tried doing this, and I started bouncing mail for everyone
because they were over quota (even when they arent), so I am clearly
missing something in how I have it configured. This is what I have:
Solved, thanks to tss on irc... My problem was the dovecot-lda
invocation was not passing the -d parameter, so no userdb quota rules
were being applied.
micah
Re: [Dovecot] dovecot-lda quota rule
On Wed, 16 Nov 2011 11:09:12 -0500, micah anderson mi...@riseup.net wrote:
On Tue, 15 Nov 2011 22:27:20 +0200, Timo Sirainen t...@iki.fi wrote:
On Thu, 2011-11-10 at 13:59 -0500, Micah Anderson wrote:
I really like the feature where you can define quota rules with percents
which trigger off of the default values[0] (so you can set the Trash to
allow for 10% more of the user's quota for example).
What I would really love in dovecot would be for the ability to
configure a quota rule for dovecot-lda. I would like to configure things
so we don't bounce emails for users until they are well over quota, the
IMAP quota plugin is a really great way to notify people that they are
over quota because it fails to write to other folders that should be
enough to get people's attention that they need to deal with things, but
bouncing is harsh.
You can give LDA-specific quota settings. Like maybe:
protocol lda {
plugin {
quota_rule2 = INBOX:storage=200%
}
}
I just tried doing this, and I started bouncing mail for everyone
because they were over quota (even when they arent), so I am clearly
missing something in how I have it configured. This is what I have:
namespace {
separator = .
prefix =
inbox = yes
}
namespace {
separator = .
prefix = INBOX.
inbox = no
hidden = yes
list = no
}
plugin {
quota = dict:Your mail quota::proxy::quota
}
dict {
quota = mysql:/etc/dovecot/dovecot-dict-sql.conf
}
plugin {
quota_exceeded_message = You are over quota. To avoid losing mail,
immediately empty your Trash and Sent folders and delete emails with large
attachments.
quota_rule = *:bytes=100663296 # in case someone doesn't have sql quota
quota_rule2 = Trash:bytes=+20%% # let trash be larger by 20%
quota_rule3 = Spam:bytes=+10%% # let spam be larger by 10%
quota_rule4 = INBOX.Trash:bytes=+20%% # repeat the above for namespaces
quota_rule5 = INBOX.Spam:bytes=+10%% # same
quota_rule6 = INBOX.restored:bytes=+500%% # allow for temporary folder
}
...
protocol lda {
# Space separated list of plugins to load (default is global mail_plugins).
mail_plugins = $mail_plugins
plugin {
quota_rule7 = INBOX:bytes=+10%%
}
}
Maybe I should have that quota_rule7 be *:bytes=+10%% ?
thanks for any suggestions for what I am missing here!
micah
pgp0AMtVVmc58.pgp
Description: PGP signature
Re: [Dovecot] Indexes to MLC-SSD
Dovecot-GDH ghandidrivesahum...@rocketfish.com writes: If I/O performance is a concern, you may be interested in ZFS and Flashcache. Specifically, ZFS' ZIL (ZFS Intent Log) and its L2ARC (Layer 2 Adaptive Read Cache) ZFS does run on Linux http://zfs-fuse.net Flashcache: https://github.com/facebook/flashcache/ That site has no information about what flashcache is.
[Dovecot] doveadm import assertion failed
Restoring a user's mailbox yesterday resulted in 'doveadm import' panic'ing with an assertion failure and giving a backtrace: /usr/bin/doveadm import -u user mdbox:/maildir/riseup.net/a/user/.daily.1/mdbox restored_from_backups/daily1 all doveadm(user): Error: Transaction log /maildir/riseup.net/a/user/.daily.1/mdbox/mailboxes/INBOX/dbox-Mails/dovecot.index.log: duplicate transac tion log sequence (3) doveadm(user): Error: Transaction log /maildir/riseup.net/a/user/.daily.1/mdbox/mailboxes/INBOX/dbox-Mails/dovecot.index.log: duplicate transac tion log sequence (3) doveadm(user): Panic: file mail-transaction-log-file.c: line 187 (mail_transaction_log_file_add_to_list): assertion failed: ((*p)-hdr.file_se q file-hdr.file_seq) doveadm(user): Error: Raw backtrace: /usr/lib/dovecot/libdovecot.so.0(+0x3fd1a) [0x7f05152bbd1a] - /usr/lib/dovecot/libdovecot.so.0(default_f atal_handler+0x32) [0x7f05152bbe02] - /usr/lib/dovecot/libdovecot.so.0(i_error+0) [0x7f051529519f] - /usr/lib/dovecot/libdovecot-storage.so .0(+0xa596a) [0x7f05159e696a] - /usr/lib/dovecot/libdovecot-storage.so.0(mail_transaction_log_file_open+0x1f8) [0x7f05159e7bc8] - /usr/lib/ dovecot/libdovecot-storage.so.0(mail_transaction_log_find_file+0xd0) [0x7f05159e46e0] - /usr/lib/dovecot/libdovecot-storage.so.0(mail_transa ction_log_view_set+0xcb) [0x7f05159e8bdb] - /usr/lib/dovecot/libdovecot-storage.so.0(mail_index_sync_map+0xbe) [0x7f05159dff6e] - /usr/lib/ dovecot/libdovecot-storage.so.0(mail_index_map+0x86) [0x7f05159d1816] - /usr/lib/dovecot/libdovecot-storage.so.0(+0x9a6da) [0x7f05159db6da] - /usr/lib/dovecot/libdovecot-storage.so.0(mail_index_sync_begin_to+0x56) [0x7f05159db956] - /usr/lib/dovecot/libdovecot-storage.so.0(mail_ index_sync_begin+0x1e) [0x7f05159dc21e] - /usr/lib/dovecot/libdovecot-storage.so.0(mdbox_sync_begin+0x121) [0x7f05159bbb41] - /usr/lib/dove cot/libdovecot-storage.so.0(mdbox_sync+0x46) [0x7f05159bc1f6] - /usr/lib/dovecot/libdovecot-storage.so.0(mdbox_storage_sync_init+0x43) [0x7f 05159bc293] - /usr/lib/dovecot/libdovecot-storage.so.0(mailbox_sync_init+0x31) [0x7f0515974031] - /usr/lib/dovecot/libdovecot-storage.so.0(mailbox_sync+0x27) [0x7f05159748b7] - /usr/bin/doveadm(doveadm_mail_iter_init+0x8b) [0x40c9db] - /usr/bin/doveadm() [0x40c284] - /usr/bin/doveadm() [0x40a673] - /usr/bin/doveadm(doveadm_mail_single_user+0x61) [0x40a731] - /usr/bin/doveadm() [0x40a95d] - /usr/bin/doveadm(doveadm_mail_try_run+0x141) [0x40ad71] - /usr/bin/doveadm(main+0x381) [0x4107e1] - /lib/libc.so.6(__libc_start_main+0xfd) [0x7f0514b2dc4d] - /usr/bin/doveadm() [0x409e59] This is with 2.0.15. Micah -- pgpF07fgRXSq0.pgp Description: PGP signature
Re: [Dovecot] dsync problems
On Tue, 15 Nov 2011 22:43:24 +0200, Timo Sirainen t...@iki.fi wrote: On Tue, 2011-11-15 at 14:24 -0500, Micah Anderson wrote: When a user renames their username, I am using dsync to copy their mail over to the new username's mail location[0]. Some of the dsyncs are failing with errors that I dont know how to work with, for example: dsync(username): Error: Trying to open a non-listed mailbox with guid=41fcd40303c8a64e43237ef44c7a dsync(username): Error: msg iteration failed: Couldn't open mailbox 41fcd40303c8a64e43237ef44c7a dsync(username): Error: Trying to open a non-listed mailbox with guid=41fcd40303c8a64e43237ef44c7a These shouldn't really happen. Something's going internally wrong with dsync. Can you reproduce this reliably somehow? Well, I dont know if I can do it reliably, but its been happening a lot. One point of information that might be useful is that these users were converted from courier maildir to mdbox, and their courier bits are still around in the source mailbox (I haven't become brave enough to remove them yet). The errors cause a non-zero exit code from dsync, which causes my rename script to bail out. What are these errors, and how can I fix them? Does a second dsync on error succeed? :) Before I tried it again, I looked at their mailboxes: # doveadm mailbox list -u username Trash_084ed82bc59ca54eb5377ef44c7a Sent Drafts INBOX_094ed82bc59ca54eb5377ef44c7a INBOX Then I tried it again, and I got an error: dsync(username): Info: INBOX: only in dest (guid=14bf0409fa08c04e68297ef44c7a) dsync(username): Error: Trying to open a non-listed mailbox with guid=14bf0409fa08c04e68297ef44c7a dsync(username): Error: msg iteration failed: Couldn't open mailbox 14bf0409fa08c04e68297ef44c7a dsync(username): Error: Trying to open a non-listed mailbox with guid=14bf0409fa08c04e68297ef44c7a and the mailboxes: # doveadm mailbox list -u username Trash_084ed82bc59ca54eb5377ef44c7a Sent Drafts INBOX_094ed82bc59ca54eb5377ef44c7a I tried it a third time, and... it worked, no error, and now their mailbox list: # doveadm mailbox list -u username Trash Sent Drafts INBOX This particular user only has one test email 0. Why use dsync instead of a simple mv operation? This seems to be necessary for two corner cases: 1. dovecot creates the new mailbox automatically when the user logs in or receives a mail, so if the user changes their mail and logs in or receives an email before the move has been done, then the mailbox is created and then a move command will fail. 2. If there has been new mail created under the new name, we can't just simply remove the stuff that is automatically created there and replace it with the old things because we could potentially be removing mail that has been delivered in the mean time. You could temporarily change the permissions for the home directory so that no new mailboxes/mails could be created during the move (e.g. 0700 root). The problem is there are a number of users on the system and all the mail is stored under /srv/mailstorage/letter/username. So if foo wants to change their username to bar -- I dont have a deterministic way of determining that bar exists yet because mail could be delivered or they could login and dovecot would create it and I can't set /srv/mailstorage/letter 0700 root or nobody would be able to receive mail. micah pgpRFlbnVOgtn.pgp Description: PGP signature
Re: [Dovecot] dovecot-lda quota rule
On Tue, 15 Nov 2011 22:27:20 +0200, Timo Sirainen t...@iki.fi wrote:
On Thu, 2011-11-10 at 13:59 -0500, Micah Anderson wrote:
I really like the feature where you can define quota rules with percents
which trigger off of the default values[0] (so you can set the Trash to
allow for 10% more of the user's quota for example).
What I would really love in dovecot would be for the ability to
configure a quota rule for dovecot-lda. I would like to configure things
so we don't bounce emails for users until they are well over quota, the
IMAP quota plugin is a really great way to notify people that they are
over quota because it fails to write to other folders that should be
enough to get people's attention that they need to deal with things, but
bouncing is harsh.
You can give LDA-specific quota settings. Like maybe:
protocol lda {
plugin {
quota_rule2 = INBOX:storage=200%
}
}
Aha! Just what I was looking for. I presume the number following
'quota_rule' must be an unique number across all quota_rules in my
config?
or even not enforcing quota at all:
protocol lda {
plugin {
quota = maildir:User quota:noenforcing
}
}
I was not aware of the noenforcing option, that answered another
question I had not asked yet, so thanks for that! :)
micah
pgpcuI0ivRtaB.pgp
Description: PGP signature
Re: [Dovecot] dovecot-lda quota rule
On Wed, 16 Nov 2011 18:27:20 +0200, Timo Sirainen t...@iki.fi wrote:
On Wed, 2011-11-16 at 11:09 -0500, micah anderson wrote:
protocol lda {
plugin {
quota_rule2 = INBOX:storage=200%
}
}
Aha! Just what I was looking for. I presume the number following
'quota_rule' must be an unique number across all quota_rules in my
config?
No only unique, but all of the numbers must exist. For example
quota_rule2 isn't used if quota_rule doesn't exist. Or quota_rule10
isn't used, unless quota_rule2..9 also exist.
Ah, interesting, good to know. I presume that 'quota_rule' stands in for
'quota_rule1' and one does not need to specify a quota_rule1 if one has
a 'quota_rule'.
micah
pgp86Wcne4iz6.pgp
Description: PGP signature
[Dovecot] dsync problems
When a user renames their username, I am using dsync to copy their mail over to the new username's mail location[0]. Some of the dsyncs are failing with errors that I dont know how to work with, for example: dsync(username): Error: Trying to open a non-listed mailbox with guid=41fcd40303c8a64e43237ef44c7a dsync(username): Error: msg iteration failed: Couldn't open mailbox 41fcd40303c8a64e43237ef44c7a dsync(username): Error: Trying to open a non-listed mailbox with guid=41fcd40303c8a64e43237ef44c7a dsync(username): Error: Trying to open a non-listed mailbox with guid=42fcd40303c8a64e43237ef44c7a dsync(username): Error: Trying to open a non-listed mailbox with guid=43fcd40303c8a64e43237ef44c7a dsync(username): Error: Trying to open a non-listed mailbox with guid=44fcd40303c8a64e43237ef44c7a ERROR: dsync failed, exit code: 256 The errors cause a non-zero exit code from dsync, which causes my rename script to bail out. What are these errors, and how can I fix them? Thanks, micah 0. Why use dsync instead of a simple mv operation? This seems to be necessary for two corner cases: 1. dovecot creates the new mailbox automatically when the user logs in or receives a mail, so if the user changes their mail and logs in or receives an email before the move has been done, then the mailbox is created and then a move command will fail. 2. If there has been new mail created under the new name, we can't just simply remove the stuff that is automatically created there and replace it with the old things because we could potentially be removing mail that has been delivered in the mean time. I'd be really interested if people had suggestions for a better mechanism, or perhaps a way to have dovecot not create the new mail location automatically. -- pgp54e2yahexo.pgp Description: PGP signature
[Dovecot] Deleting a mailbox and all its children
I'm looking at scripting a mechanism to delete a mailbox, which is easy with 'doveadm mailbox delete' -- however it gets complicated when there are children of arbitrary depth, for example, I may have this: restored.daily1.INBOX restored.daily1.Sent restored.daily1.Mystuff restored.daily1.Mystuff.foo I would like to delete the 'restored' mailbox and all its children (and unsubscribe the folders), but the 'doveadm mailbox delete' command wont delete any children. I'm surprised that there isn't a -r (recursive) option to 'doveadm mailbox delete', but perhaps I am missing the easier way to handle this? thanks for any ideas or suggestions! micah -- pgpruGvZod8bb.pgp Description: PGP signature
Re: [Dovecot] Deleting a mailbox and all its children
On Wed, 16 Nov 2011 03:15:02 +0200, Timo Sirainen t...@iki.fi wrote: On 16.11.2011, at 3.04, Micah Anderson wrote: I'm looking at scripting a mechanism to delete a mailbox, which is easy with 'doveadm mailbox delete' -- however it gets complicated when there are children of arbitrary depth, for example, I may have this: restored.daily1.INBOX restored.daily1.Sent restored.daily1.Mystuff restored.daily1.Mystuff.foo I would like to delete the 'restored' mailbox and all its children (and unsubscribe the folders), but the 'doveadm mailbox delete' command wont delete any children. I'm surprised that there isn't a -r (recursive) option to 'doveadm mailbox delete', but perhaps I am missing the easier way to handle this? Well, how about something like: doveadm mailbox list 'restored.*' | xargs -1 doveadm mailbox delete doveadm mailbox delete restored That wont work because 'restored.daily1.Mystuff.foo' wont get removed before 'restored.daily1.Mystuff' is attempted, resulting in the following: doveadm mailbox list 'restored.*' | xargs -1 doveadm mailbox delete doveadm(micah): Error: Can't delete mailbox restored.daily1.Mystuff: Mailbox has children, delete them first The problem is the subfolders need to be removed first and then walk up the tree. perhaps I can do: doveadm mailbox list 'restored.*' | sort -r | xargs -n1 doveadm mailbox delete doveadm mailbox delete restored although I'm not sure if the reverse sort is going to work in all cases. micah pgpjj3dO5EJlx.pgp Description: PGP signature
[Dovecot] dovecot-lda quota rule
I really like the feature where you can define quota rules with percents which trigger off of the default values[0] (so you can set the Trash to allow for 10% more of the user's quota for example). What I would really love in dovecot would be for the ability to configure a quota rule for dovecot-lda. I would like to configure things so we don't bounce emails for users until they are well over quota, the IMAP quota plugin is a really great way to notify people that they are over quota because it fails to write to other folders that should be enough to get people's attention that they need to deal with things, but bouncing is harsh. Is there a way to do this now that I haven't seen? thanks! micah 0. http://wiki2.dovecot.org/Quota/Configuration -- pgpaJaa1mOJFt.pgp Description: PGP signature
Re: [Dovecot] Accessing a strange mailbox
Hi, thanks for the reply! On Fri, 04 Nov 2011 21:34:03 +0200, Timo Sirainen t...@iki.fi wrote: On Fri, 2011-10-21 at 10:50 -0400, Micah Anderson wrote: I have a user who has a mailbox called: A-B It seemed to work in courier fine, they managed to create it, and there are mails in it. However, dovecot is not letting the user access it, the IMAP server gives an error. I tried to rename it but I would also get an error: # doveadm mailbox rename 'A-B' AtoB doveadm(root): Error: Can't rename mailbox A-B to AtoB: Mailbox doesn't exist: A-B Maybe it's not named that? What does it look like in filesystem? Worked fine with me: I moved the user from courier maildir to mdbox, and I've still got the maildir folders around, the filesystem shows it as: drwx-- 6 mail mail 4096 Oct 20 07:56 .A-B I just realized my mistake, my 'doveadm mailbox rename' command wasn't limited to the particular user (I was not passing -u username) so it was looking in the wrong namespace for that user. Once I specified the user, it worked properly. Micah pgpGTC8lpFFv5.pgp Description: PGP signature
[Dovecot] Accessing a strange mailbox
I have a user who has a mailbox called: A-B It seemed to work in courier fine, they managed to create it, and there are mails in it. However, dovecot is not letting the user access it, the IMAP server gives an error. I tried to rename it but I would also get an error: # doveadm mailbox rename A-\B AtoB doveadm(root): Error: Can't rename mailbox A-B to AtoB: Mailbox doesn't exist: A-B # doveadm mailbox rename A-B AtoB doveadm(root): Error: Can't rename mailbox A- to AtoB: Mailbox doesn't exist: A- # doveadm mailbox rename 'A-B' AtoB doveadm(root): Error: Can't rename mailbox A-B to AtoB: Mailbox doesn't exist: A-B # doveadm mailbox rename A-B AtoB doveadm(root): Error: Can't rename mailbox A-B to AtoB: Mailbox doesn't exist: A-B # How can I deal with this mailbox (using mdbox here)? Thanks! -- pgpVuAtT478AK.pgp Description: PGP signature
[Dovecot] Safely restoring backups
I'm trying to determine what the best way to restore mail with mdbox is. Restoring using maildir was trivial, I just used rsync --ignore-existing which wrote any mails that were removed and didn't touch things that already existed[1]. With mdbox things have become more complicated, and I haven't found a way to restore mail that doesn't result in many message duplicates. My backup setup is simple, I'm doing daily rsync backups of user's mailstores, as well as weekly backups, so I end up having on my backup server daily.1, daily.2, daily.3, daily.4, weekly.1, weekly.2... each containing the entire contents of the user's mdbox. The different restore methods I've tried are: 1. I tried rsyncing the different backup directories back to the mail storage host, and then doing: 'dsync -R backup -u $user mdbox:/path/to/to/daily.1/mdbox'[2] 'dsync -R backup -u $user mdbox:/path/to/to/daily.2/mdbox' This works ok, but it gives duplicates of every mail that already exists for every daily/weekly I dsync. This is what the rsync --ignore-existing avoided. This is particularly annoying if I restore the weekly, and multiple daily directories, because then you get a copy for every one you restore. I had thought that the individual message's GIDs would keep them from being duplicated? 2. I also tried to use 'doveadm import' in two different ways. The first way I created a 'restored_from_backups' folder and then import each of the daily.#, weekly.# mdboxes into their own subfolder within that folder, for example: 'doveadm import -u $username mdbox:/path/to/daily.1/mdbox restored_from_backups/daily1 all' 'doveadm import -u $username mdbox:/path/to/daily.2/mdbox restored_from_backups/daily2 all' ... etc. I then go through and subscribe the new folders[3] and the user ends up with a folder structure like this: restored_from_backups/ daily.1/ INBOX Trash folder1 folder2 daily.2 INBOX Trash folder1 folder2 etc. This works fine, except that it results in the user having a entire duplicate copies of their mailbox for each daily/weekly that I restore. That will quickly bring people over quota. 3. I also tried to use 'doveadm import' to import all the different directories all into the same restored_from_backups directory, so there are no subdirectories for each daily/weekly under restored_from_backups, like so: 'doveadm mailbox create -u $user -s restored_from_backups' 'doveadm -v -D import -u $user mdbox:/path/to/daily.1/mdbox restored_from_backups all' I then go through and subscribe the folders[3]. Using this method, the 'restored_from_backups' mailbox is created, and populated with the folders. The only problem with this method is the same as method #1: for every backup I restore, mails are duplicated. Is there a way I can restore things from backup and avoid duplicates? Is there another method I should try? thanks for any ideas, pointers, suggestions for improvement, etc. micah 1. this would of course bring back mails that were deleted, but that was fine as the user could deal with that. 2. yes, I know I could setup dsync on the backup server, and use dsync directly to pull the mails from there, but this is complicated in my situation due to how the backupserver works. 3. why doesn't doveadm import have a -s option to subscribe? -- pgpt0Z8N5J07x.pgp Description: PGP signature
Re: [Dovecot] doveadm mailbox subscribe wildcard
A.L.E.C a...@alec.pl writes: On 2011-10-06 23:23, Micah Anderson wrote: # doveadm mailbox list -s -u bob dovecot* # doveadm mailbox subscribe -u bob dovecot* RFC3501. Wildcards are allowed only in LIST/LSUB commands. Aha, thanks! I'll use the for loop method to achieve what I want. micah -- pgp6eY116W31a.pgp Description: PGP signature
Re: [Dovecot] doveadm mailbox list shows file system directories as mailboxes
Jürgen Obermann juergen.oberm...@hrz.uni-giessen.de writes: Hello. In our dovecvot we use mbox format with the default filesystem layout. Therefore it is not possible to have mailboxes which are subfolders of other mailboxes containing messages. The command doveadm mailbox list includes the file system directories, that contain only subfolders, as normal mailboxes in the output: archiv archiv/daemon archiv/dovecot Did you perhaps do a 'doveadm mailbox subscribe' with a wildcard and the shell expanded it? micah
[Dovecot] lazy expunge namespaces
I turned on lazy_expunge and configured a namespace following the
wiki[0] instructions for mdbox configuration:
namespace {
inbox = yes
location =
prefix =
separator = .
}
namespace {
hidden = yes
inbox = no
list = no
location =
prefix = INBOX.
separator = .
}
namespace {
location = mdbox:~/mdbox:MAILBOXDIR=expunged
prefix = EXPUNGED.
separator = .
}
This created an EXPUNGED mailbox in my client (thunderbird), which was
greyed out. When I deleted things, they got put in the EXPUNGED mailbox,
as expected, but I could not access them to try and restore them. The
server would say that the mailbox did not exist.
micah
0. http://wiki2.dovecot.org/Plugins/Lazyexpunge
--
pgpLuAxubmLRp.pgp
Description: PGP signature
Re: [Dovecot] dsync with quotas
Micah Anderson mi...@riseup.net writes: I have been working on converting people from courier maildir - dovecot mdbox and during some of the dsync runs I'm seeing the quota_exceeded_message be printed as an Error: dsync(user): Error: Can't save message to mailbox INBOX: You are over quota. To avoid losing mail, immediately empty your Trash and Sent folders and \ delete emails with large attachments. dsync(user): Info: INBOX: Couldn't keep all uids^M Its possible the user was over quota on the originating courier side, but I would still like to migrate their mail proprely to mdbox, but it seems like being over quota is inhibiting that. Are the quota calculations including both the maildir files as well as the converted mdbox files, resulting in a double counting? I'm not entirely sure if the messages above indicate that the migration failed for that user or not, so I've been manually increasing their quota, then redoing the dsync mirror until it works properly. I wonder if it would be better if I turned off quota entirely during migration so I don't run into this problem? Sorry to 'bump' this subject, but I'm still confused about this issue and nobody replied, perhaps nobody has any ideas? :) micah
[Dovecot] dsync with quotas
I have been working on converting people from courier maildir - dovecot mdbox and during some of the dsync runs I'm seeing the quota_exceeded_message be printed as an Error: dsync(user): Error: Can't save message to mailbox INBOX: You are over quota. To avoid losing mail, immediately empty your Trash and Sent folders and \ delete emails with large attachments. dsync(user): Info: INBOX: Couldn't keep all uids^M Its possible the user was over quota on the originating courier side, but I would still like to migrate their mail proprely to mdbox, but it seems like being over quota is inhibiting that. Are the quota calculations including both the maildir files as well as the converted mdbox files, resulting in a double counting? I'm not entirely sure if the messages above indicate that the migration failed for that user or not, so I've been manually increasing their quota, then redoing the dsync mirror until it works properly. I wonder if it would be better if I turned off quota entirely during migration so I don't run into this problem? thanks, micah -- pgprlwVpl83vL.pgp Description: PGP signature
Re: [Dovecot] v2.0.15 released
Simon Brereton simon.brere...@buongiorno.com writes: I'll bite.. But it's probably not your concern. When will we be able to get stable 2.x packages for Debian? Apt installs 1.2.15-7 2.x packages are available in Debian Wheezy and Sid. A backport to Squeeze (stable) will be made once the 1.x transition packages make it into Wheezy (~10 days from now). micah
Re: [Dovecot] convert maildrop filter to sieve
Aleksandar Lazic al-dove...@none.at writes: Dear list member, does anybody have a script which can convert maildrop filter to sive filter? I would also like this, but I'm afraid it may be too complicated due to the different possibilities. Moving to sieve from maildrop would be great, but in practice quite difficult to do when you have a large number of people with customized mailfilters. micah
Re: [Dovecot] moving mail out of alt storage
Timo Sirainen t...@iki.fi writes: On Wed, 2011-09-14 at 23:17 -0400, Micah Anderson wrote: I moved some mail into the alt storage: doveadm altmove -u jo...@example.com seen savedbefore 1w and now I want to move it back to the regular INBOX, but I can't see how I can do that with either 'altmove' or 'mailbox move'. Is this sdbox or mdbox? With sdbox you could simply mv the files. Or apply patch: http://hg.dovecot.org/dovecot-2.0/rev/1910c76a6cc9 This is mdbox, which is why I am not sure how to operate because I am used to individual files as is with maildir. micah
[Dovecot] movming mail out of alt storage
I moved some mail into the alt storage: doveadm altmove -u jo...@example.com seen savedbefore 1w and now I want to move it back to the regular INBOX, but I can't see how I can do that with either 'altmove' or 'mailbox move'. thanks! micah -- pgpsYHjQ0YeIm.pgp Description: PGP signature
Re: [Dovecot] quota percents
Timo Sirainen t...@iki.fi writes:
On 12.9.2011, at 19.17, Micah Anderson wrote:
However, it doesn't seem to work in practice, because I have a user that
is at 99% of quota, with nothing in the Trash who cannot move a 77KB
message into the Trash without getting the quota_exceeded message and
refusing to move it.
dovecot -n output?
namespace {
inbox = yes
location =
prefix =
separator = .
}
namespace {
hidden = yes
inbox = no
list = no
location =
prefix = INBOX.
separator = .
}
Maybe the client is using the INBOX. namespace and trying to copy to
INBOX.Trash instead of Trash? You could try adding another quota rule for
INBOX.Trash. I should probably also change the quota code to follow
alias_for directives. Then you could have only a single Trash, if you add
alias_for = to the INBOX. namespace.
ok, I made an INBOX.Trash quota rule:
90-quota.conf: quota_rule = *:bytes=24117248
90-quota.conf: quota_rule2 = Trash:bytes=+10%%
90-quota.conf: quota_rule2 = INBOX.Trash:bytes=+10%%
90-quota.conf: quota_rule3 = INBOX.Spam:bytes=+20%%
and that seemed to solve the problem!
micah
ps - its very annoying to test quota with thunderbird, it doesn't update
the information very reliably.
--
Re: [Dovecot] quota percents
Timo Sirainen t...@iki.fi writes:
On Fri, 2011-09-09 at 16:05 -0400, Micah Anderson wrote:
its incorrect because if you change it to what it suggests ('+10%') it
wont work because that wont be 10% more, rather, it is set to bytes=+10:
Well, that's a more complex problem. The %% escaping needs to be done in
dovecot.conf, but it doesn't need to be done elsewhere. If this said
that use +10%% and someone gives that in SQL user_query, it will
break. So I'm not changing the warning.
Ok, that makes sense.
However, it doesn't seem to work in practice, because I have a user that
is at 99% of quota, with nothing in the Trash who cannot move a 77KB
message into the Trash without getting the quota_exceeded message and
refusing to move it.
dovecot -n output?
NOTE: i turned off the quota plugins while this is sorted, so you wont
see them enabled below.
# 2.0.13: /etc/dovecot/dovecot.conf
# OS: Linux 2.6.32-5-amd64 x86_64 Debian 6.0.2
auth_default_realm = example.net
auth_verbose = yes
default_process_limit = 256
default_vsz_limit = 512 M
dict {
expire = mysql:/etc/dovecot/dovecot-dict-sql.conf
quota = mysql:/etc/dovecot/dovecot-dict-sql.conf
}
disable_plaintext_auth = no
first_valid_gid = 8
first_valid_uid = 8
last_valid_gid = 8
last_valid_uid = 8
listen = *
login_greeting = howdy, ready.
mail_location = mdbox:~/mdbox
mail_plugins = expire
maildir_very_dirty_syncs = yes
namespace {
inbox = yes
location =
prefix =
separator = .
}
namespace {
hidden = yes
inbox = no
list = no
location =
prefix = INBOX.
separator = .
}
passdb {
args = /etc/dovecot/dovecot-sql.conf
driver = sql
}
plugin {
expire = Trash
expire2 = Trash/*
expire3 = Spam
expire_dict = proxy::expire
quota = dict:Your mail quota::proxy::quota
quota_exceeded_message = You are over quota. To avoid losing mail,
immediately empty your Trash and Sent folders and delete emails with large
attachments.
quota_rule = *:bytes=24117248
quota_rule2 = Trash:bytes=+50%%
quota_rule3 = Spam:bytes=+20%%
}
postmaster_address = postmas...@example.net
protocols = imap pop3
service dict {
unix_listener dict {
group = mail
mode = 0600
user = mail
}
}
service imap-login {
process_min_avail = 10
service_count = 0
vsz_limit = 512 M
}
service imap-postlogin {
executable = script-login /maildir/postlogin_imap
user = $default_internal_user
}
service pop3-postlogin {
executable = script-login /maildir/postlogin_pop
user = $default_internal_user
}
ssl_cert = /etc/certs/wildcard/cert.pem
ssl_cipher_list = HIGH:-ADH:-aNULL
ssl_key = /etc/certs/wildcard/key.pem
userdb {
args = /etc/dovecot/dovecot-sql.conf
driver = sql
}
verbose_proctitle = yes
protocol imap {
imap_client_workarounds = delay-newmail tb-extra-mailbox-sep
mail_plugins = expire
}
protocol pop3 {
pop3_client_workarounds = outlook-no-nuls oe-ns-eoh
pop3_uidl_format = %g
}
[Dovecot] quota percents
I've noticed that http://wiki2.dovecot.org/Quota/Configuration is out of
date, it says:
plugin {
quota = maildir:User quota
quota_rule = *:storage=1GB
# 10% of 1GB = 100MB
quota_rule2 = Trash:storage=10%%
# 20% of 1GB = 200MB
quota_rule3 = Spam:storage=20%%
}
but if you use the '10%%' notation, dovecot complains, and *also* incorrectly:
dovecot: imap(t...@example.net): Warning: quota root mail quota rule
Trash:bytes=10%: obsolete configuration for rule 'bytes=10%' should be changed
to 'bytes=+10%'
its incorrect because if you change it to what it suggests ('+10%') it
wont work because that wont be 10% more, rather, it is set to bytes=+10:
dovecot: imap(t...@example.net): Debug: Quota rule: root=mail quota
mailbox=Trash bytes=+10 messages=0
clearly, that isn't right... so I guess it needs to be changed to
'+10%%', setting that provides:
dovecot: imap(t...@example.net): Debug: Quota rule: root=mail quota mailbox=*
bytes=2211724 messages=0
dovecot: imap(t...@example.net): Debug: Quota rule: root=mail quota
mailbox=Trash bytes=+221172 (10%) messages=0
that seems ok, its saying the Trash mailbox is set to have 221172bytes
(215 kilobytes) of quota. So, clearly
http://wiki2.dovecot.org/Quota/Configuration needs to be updated.
However, it doesn't seem to work in practice, because I have a user that
is at 99% of quota, with nothing in the Trash who cannot move a 77KB
message into the Trash without getting the quota_exceeded message and
refusing to move it.
so... how do I get this to work?
thanks!
micah
--
pgpJVKcKQpvwj.pgp
Description: PGP signature
[Dovecot] sorting order changed during migration from courier
Hello, I've been working to migrate to dovecot (debian stable: 1:1.2.15-7) from courier (debian old-stable: 4.4.0-2). I've setup everything and migrated a test user over to the new system, and ran the courier-dovecot-migrate.pl script. The user is using alpine (the free version of pine) and their sort order changed on move to dovecot. Originally it was the default in alpine, which is to sort based on arrival. If he changes the sort order in alpine to sort based on date, then things are back to how he expects it. I noticed that there are no known issues with pine (http://wiki.dovecot.org/Clients#Pine). Ideally, I would prefer if all the users who are migrated aren't presented with a different sort order than they were used to, as the support burden will be quite high. What is it about this migration that could have caused that, and more importantly, how can I fix it? I moved the mail from the original system to this system via rsync -a. The old system was an i686 machine and ext3, this new one is amd64 and ext4. thanks for any ideas! micah # dovecot -n # 1.2.15: /etc/dovecot/dovecot.conf # OS: Linux 2.6.32-5-amd64 x86_64 Debian 6.0.2 protocols: imap imaps pop3 pop3s disable_plaintext_auth: no verbose_ssl: yes login_dir: /var/run/dovecot/login login_executable(default): /usr/lib/dovecot/imap-login login_executable(imap): /usr/lib/dovecot/imap-login login_executable(pop3): /usr/lib/dovecot/pop3-login login_greeting: howdy, ready. login_processes_count: 10 login_max_processes_count: 256 first_valid_uid: 8 last_valid_uid: 8 first_valid_gid: 8 last_valid_gid: 8 mail_uid: 8 mail_gid: 8 mail_location: maildir:~/ maildir_copy_preserve_filename: yes maildir_very_dirty_syncs: yes mbox_write_locks: fcntl dotlock mail_executable(default): /maildir/postlogin_imap mail_executable(imap): /maildir/postlogin_imap mail_executable(pop3): /maildir/postlogin_pop mail_process_size: 512 mail_plugins(default): expire quota imap_quota mail_plugins(imap): expire quota imap_quota mail_plugins(pop3): expire quota mail_plugin_dir(default): /usr/lib/dovecot/modules/imap mail_plugin_dir(imap): /usr/lib/dovecot/modules/imap mail_plugin_dir(pop3): /usr/lib/dovecot/modules/pop3 imap_client_workarounds(default): tb-extra-mailbox-sep delay-newmail imap_client_workarounds(imap): tb-extra-mailbox-sep delay-newmail imap_client_workarounds(pop3): pop3_client_workarounds(default): pop3_client_workarounds(imap): pop3_client_workarounds(pop3): outlook-no-nuls oe-ns-eoh namespace: type: private separator: . prefix: INBOX. inbox: yes list: yes subscriptions: yes lda: postmaster_address: postmas...@riseup.net auth default: default_realm: riseup.net verbose: yes passdb: driver: sql args: /etc/dovecot/dovecot-sql.conf userdb: driver: prefetch dict: expire: mysql:/etc/dovecot/dovecot-dict-expire.conf -- pgpASf7RZiWzd.pgp Description: PGP signature
