[Dovecot] feature request: pipe for custom quota dict queries
Just a followup to my previous post. I appears that a pipe for quota queries via dict is not supported right now. Because of the way we group things we need more flexibility. I'd like to propose that dovecot should support quota queries just like userdb dict queries. My expertise is not C, but I might try to poke around and see if I can make something work. ...Jeff
[Dovecot] quota and dict
I have a question about using dict and quotas. I want dovecot to send
quota queries to a custom dict server over a socket. I'm doing this
because I can't do group quotas based on domain since a customer can
have each of their users associated with different domains under a
single account. I need to lookup the account ID and group based on that.
I'm worried putting everything in mysql will cause way too many writes
and lower the performance of our mysql cluster.
I have having a little trouble connecting all the dots in the config
file. In the userdb example, there is an 'args' parameter that allows a
file to specify a uri. I don't see how to do that for dict. I only see
file, mysql, and postgresql. Shouldn't I be able to use a dictionary
proxy to attach any custom program to a quota dict socket?
Tell the quota plug to proxy quota which then points to a socket:
plugin {
quota = dict:User quota::proxy::quota
}
dict {
quota = proxy:/tmp/test-socket
}
or should it be:
plugin {
quota = dict:User quota::proxy:/tmp/test-socket
}
Neither one create a socket in /tmp.
It seems like this should be possible, but I don't see an obvious way
to do it.
...Jeff
[Dovecot] accumulating orphaned processes
I'm getting orphaned processes on our mail servers. Some machines has
processes hanging around from a week ago. I'm running the current stable
of dovecot 2.1.15, but the problem has been happening for a while now. I
thought it would be good to report the issue.
I ran an strace on one of them and this is the output:
Process 21370 attached - interrupt to quit
open(/var/run/dovecot/stats-mail, O_WRONLY) = 7
write(7, DISCONNECT\ta09ade0350d81d517a530..., 44) = 44
munmap(0x7f1ad8a8d000, 266240) = 0
close(13) = 0
close(12) = 0
epoll_ctl(9, EPOLL_CTL_DEL, 11, {0, {u32=17384128, u64=17384128}}) = 0
close(11) = 0
close(7)= 0
munmap(0x7f1ada5d6000, 2106272) = 0
munmap(0x7f1ada7d9000, 2161360) = 0
munmap(0x7f1ada3cf000, 2124136) = 0
munmap(0x323c80, 2162768) = 0
munmap(0x7f1ad9fc8000, 2102976) = 0
munmap(0x7f1ada1ca000, 2114728) = 0
epoll_ctl(9, EPOLL_CTL_DEL, 4, {0, {u32=17319120, u64=17319120}}) = 0
close(4)= 0
close(8)= 0
close(9)= 0
exit_group(0) = ?
Process 21370 detached
As you can see, the process exited at that point. Any ideas?
...Jeff
Re: [Dovecot] key - object mailstore
On Fri, 2012-09-14 at 17:59 +0300, Timo Sirainen wrote: I've a whole new design for it and I was planning on implementing it for v2.2. Do you want to help coding it? :) Which storage would you want to use? The generic idea is: - only one server accesses one user simultaneously - index files are copied from object storage to local filesystem and accessed there, once in a while uploaded back to object storage - if user is accessed from two servers because of some bug/split brain/something, the changes are merged using dsync - support high latency: asynchronous reads/writes. prefetch mail bodies. With this system, would the read/write ultimately go to a normal OS file function? If it is a file function, could this be used with a system like glusterfs, ceph, etc? The other option would be to write it against a object store client library and bypass the normal file functions. ...Jeff
[Dovecot] more dsync issues
It hit another dsync snag. We have a script that executes a dsync backup command to a remote host. No write should be happening on the remote host (the backup host). The error is: dsync-local(anonym...@domain.com): Error: remote: dsync-remote(anonym...@domain.com): Error: Can't delete mailbox INBOX: INBOX can't be deleted. dsync-remote(anonym...@domain.com): Error: command MSG-SAVE failed dsync-local(anonym...@domain.com): Error: read() from worker server failed: EOF dsync-local(anonym...@domain.com): Error: Unexpected reply from server: dsync-local(anonym...@domain.com): Warning: Mailbox changes caused a desync. You may want to run dsync again. We're currently running 2.1.7 since 2.1.8 causes a problem with our webmail software. ...Jeff
Re: [Dovecot] dsync backup gets stuck... fails
On Wed, 2012-08-15 at 12:27 +0300, Timo Sirainen wrote: On 15.8.2012, at 5.41, Jeff Gustafson wrote: I have an issue related to this problem. dsync returns an error 75 when it detects the source mailbox is empty (client probably pop3'd all of their email). It also returns an error 75 when I get the timeout error. You mean this? dsync-local(tss): Fatal: dsync backup: Looks like you're trying to run backup in wrong direction. Source is empty and destination is not. That's the one! Maybe it needs a force setting, or change the detection somehow.. That would be nice. A backup script is executing the command, so it should never do it in the wrong direction. A force setting might be the simplest way to go. ...Jeff
Re: [Dovecot] dsync backup gets stuck... fails
On Tue, 2012-08-14 at 23:23 +0300, Timo Sirainen wrote: On 11.8.2012, at 0.54, Jeff Gustafson wrote: More dsync issues. We were running 2.1.7 and we updated to 2.1.9. Same problem with both versions. I'm getting an error 75 on about 40 boxes out of 1800. It is the same list of boxes every time we use 'dsync backup' to backup the server. dsync seems to stop communicating to the backup box (over ssh). strace just shows it sitting at a epoll_wait. So you can easily reproduce this by running dsync for a specific user? Yes. There is a subset of mailboxes that always time out. Once the program quits (times out?), a 'du' shows the destination is smaller (200kbyte in one case). As in, some of the mails didn't get synced? (doveadm fetch could be used to do a better comparison, file sizes don't necessarily mean anything.) True, I will dump out the mailboxes and see if it truly was incomplete. Those hangs are a little bit annoying to debug, and the whole code has been rewritten for v2.2 already in a way that should make the hangs pretty much impossible. Annoyingly v2.2 isn't ready yet.. I have found a manual work around. I use rsync to get the files over to the backup machines, then I let the backup script keep things up to date. It is not the best way to go, but at least I have backups. I suppose I can check the log and continue to rsync things over until 2.2 comes out. ...Jeff
Re: [Dovecot] dsync backup gets stuck... fails
On Tue, 2012-08-14 at 23:23 +0300, Timo Sirainen wrote: On 11.8.2012, at 0.54, Jeff Gustafson wrote: What can I do to help the developers locate the bug? Those hangs are a little bit annoying to debug, and the whole code has been rewritten for v2.2 already in a way that should make the hangs pretty much impossible. Annoyingly v2.2 isn't ready yet.. I have an issue related to this problem. dsync returns an error 75 when it detects the source mailbox is empty (client probably pop3'd all of their email). It also returns an error 75 when I get the timeout error. For not I am parsing the error to find out which is which and act accordingly. It would be much nicer if dsync returned a different error code for empty source mailboxes. ...Jeff
Re: [Dovecot] dsync backup gets stuck... fails
On Sat, 2012-08-11 at 15:50 +0200, Daniel Parthey wrote: Maybe you have run into the epoll kernel bug under RHEL/CentOS: :) Yeah... been there, done that. We found that bug within *minutes* of ksplice updating the kernel. I don't think this is an epoll thing because, if it was, customers wouldn't be able to connect to our services. I think there is something else going on. I think it is a bug, but I suppose it could be a setting somewhere. Something is timing out or getting stuck. ...Jeff
[Dovecot] dsync backup gets stuck... fails
More dsync issues. We were running 2.1.7 and we updated to 2.1.9. Same problem with both versions. I'm getting an error 75 on about 40 boxes out of 1800. It is the same list of boxes every time we use 'dsync backup' to backup the server. dsync seems to stop communicating to the backup box (over ssh). strace just shows it sitting at a epoll_wait. Once the program quits (times out?), a 'du' shows the destination is smaller (200kbyte in one case). Has anyone else seen an exit code of 75? Nothing in the documentation mentions what exit code 75 could mean. What can I do to help the developers locate the bug? ...Jeff
Re: [Dovecot] dsync backup gets stuck... fails
On Sat, 2012-08-11 at 00:56 +0200, Daniel Parthey wrote:
Hi Jeff,
Please start by following the instructions at
http://dovecot.org/bugreport.html
and post your 'doveconf -n' output in order to provide
possibly important information about your system and configs.
Storage is local hardware-based RAID (SATA drives). FS is ext4.
No core dump that I know of.
Command used (mailbox is over 2GB):
# su vmail -c dsync -u u...@domain.com backup ssh vmail@bk05 dsync -c
/etc/dovecot/dovecot.conf -u u...@domain.com
Config file:
# 2.1.9: /etc/dovecot/dovecot.conf
# OS: Linux 2.6.32-220.17.1.el6.x86_64 x86_64 CentOS release 6.2
(Final)
auth_mechanisms = plain login
default_client_limit = 15000
default_process_limit = 1
disable_plaintext_auth = no
listen = *
mail_gid = vmail
mail_location = mdbox:~/mdbox
mail_plugins = zlib quota stats
mail_uid = vmail
mmap_disable = yes
namespace {
inbox = yes
location =
prefix = INBOX.
separator = .
}
passdb {
args = /etc/dovecot/conf.d/dovecot-sql.conf.ext
driver = sql
}
plugin {
quota = dict:User noenforcing quota::file:%h/dovecot-quota
stats_refresh = 30 secs
stats_track_cmds = yes
zlib_save = gz
}
protocols = imap pop3
service auth {
client_limit = 1
unix_listener auth-userdb {
mode = 0666
}
}
service imap-postlogin {
executable = script-login /usr/bin/postlogin-imap.sh
user = $default_internal_user
}
service imap {
drop_priv_before_exec = yes
executable = imap
process_limit = 1
}
service pop3-postlogin {
executable = script-login /usr/bin/postlogin-pop.sh
user = $default_internal_user
}
service pop3 {
drop_priv_before_exec = yes
executable = pop3
process_limit = 2500
}
service stats {
fifo_listener stats-mail {
mode = 0600
user = vmail
}
}
ssl_cert = /etc/pki/dovecot/certs/dovecot.pem
ssl_key = /etc/pki/dovecot/private/dovecot.pem
userdb {
driver = prefetch
}
userdb {
args = /etc/dovecot/conf.d/dovecot-sql.conf.ext
driver = sql
}
protocol lmtp {
mail_plugins = zlib quota stats
}
protocol lda {
mail_plugins = zlib quota stats
}
protocol imap {
mail_max_userip_connections = 100
mail_plugins = zlib quota stats imap_quota imap_stats
}
protocol pop3 {
mail_max_userip_connections = 30
mail_plugins = zlib quota stats
}
[Dovecot] more dsync problems: Error: proxy client timed out
Hi all, I'm using dsync for backups. I recently upgraded to 2.1.7 due to issues with dsync mirror/backups. 2.1.7 fixed the issue with the guid conflict, but now I'm seeing this error on a couple of mailboxes: Error: proxy client timed out (waiting for output stream to flush, 0 bytes left) If there are 0 bytes left, why is it trying to flush? I'm using this command to backup the mailbox: dsync -vo mail_home=/home/users/user%domain.com backup ssh vmail@10.1.2.2 dsync -o mail_home=/home/.incoming_mail_migrations/users/user%adomain.com Any ideas? ...Jeff
[Dovecot] more info (more dsync problems)
Rerunning the dsync command right after the timeout (see previous message) gave me these errors: dsync-local(vmail): Error: remote: dsync-remote(vmail): Error: Corrupted dbox file /home/.incoming_mail_migrations/users/user%domain.com/mdbox/s torage/m.60 (around offset=807789): msg header has bad magic value dsync-local(vmail): Error: remote: dsync-remote(vmail): Error: Corrupted dbox file /home/.incoming_mail_migrations/users/user%domain.com/mdbox/s torage/m.60 (around offset=807789): msg header has bad magic value dsync-local(vmail): Error: remote: dsync-remote(vmail): Warning: mdbox /home/.incoming_mail_migrations/users/user% domain.com/mdbox/storage: Inconsistency in map index (2,75904 != 2,82524) dsync-remote(vmail): Warning: mdbox /home/.incoming_mail_migrations/users/user% domain.com/mdbox/storage: rebuilding indexes dsync-local(vmail): Error: remote: dsync-remote(vmail): Error: Corrupted dbox file /home/.incoming_mail_migrations/users/user% domain.com/mdbox/storage/m.60 (around offset=807789): msg header has bad magic value dsync-local(vmail): Error: remote: dsync-remote(vmail): Warning: dbox: Copy of the broken file saved to /home/.incoming_mail_migrations/users/user% domain.com/mdbox/storage/m.60.broken dsync-local(vmail): Error: remote: dsync-remote(vmail): Error: Corrupted dbox file /home/.incoming_mail_migrations/users/user% domain.com/mdbox/storage/m.60 (around offset=807789): EOF reading msg header (got 0/30 bytes) dsync-local(vmail): Error: remote: dsync-remote(vmail): Warning: mdbox /home/.incoming_mail_migrations/users/user% domain.com/mdbox/storage: rebuilding indexes dsync-local(vmail): Error: remote: dsync-remote(vmail): Error: mdbox /home/.incoming_mail_migrations/users/user% domain.com/mdbox/storage: Duplicate GUID 1341304555.M433317P26661.n13,S=142447,W=144391 in m.62:790375 and m.60:790375
Re: [Dovecot] dsync error: Mailboxes don't have unique GUIDs
On Sat, 2012-06-23 at 06:29 -0400, Charles Marcus wrote: snip # 2.0.13: /etc/dovecot/dovecot.conf As you are aware (since you participated in the thread discussion about this months ago), Timo is working on a total rewrite of dsync, and if memory serves, it is mainly for 2.1+, and it is not recommend to use it in earlier versions if you need reliability (ie, 2.0.x, as you are using)... I did try the 2.1.x version of dsync back in March. I found the version to be very unreliable. It would crash with many types of operations (e.g. maildir - mdbox conversions). ...Jeff
Re: [Dovecot] dsync error: Mailboxes don't have unique GUIDs
On Mon, 2012-06-25 at 19:54 +0300, Timo Sirainen wrote: On 25.6.2012, at 19.49, Charles Marcus wrote: I'd suggest you try again with 2.1.7... The rewritten dsync is in v2.2 tree. v2.1's dsync is a fixed version of v2.0's dsync. I have no idea why v2.1's dsync would be less reliable than v2.0's. It only had bugfixes. Anyway, the GUID error could very well be because of buggy mailbox listing code in v2.0, which was rewritten for v2.1. I will try the latest 2.1.x code and see what happens. dsync in 2.0.x seems to work just fine... most of the time. ...Jeff
[Dovecot] dsync error: Mailboxes don't have unique GUIDs
I'm getting an error backing up mailboxes. I'm using the mirror
command:
dsync -fvo mail_home=/home/users/bob mirror ssh vmail@10.1.4.1 dsync -o
mail_home=/home/.incoming_mail_migrations/users/bob
dsync-remote(vmail): Error: Mailboxes don't have unique GUIDs:
1ef6ee37c694894d78310581a675 is shared by INBOX and INBOX
dsync-remote(vmail): Error: command BOX-LIST failed
dsync-local(vmail): Error: Worker server's mailbox iteration failed
The mail user doesn't yet exist on the destination yet, thus the use of
the mail_home parameter.
I found a mailing list message where a person was having a similar
problem but I couldn't find confirmation that the issue was resolved.
In our case, the backup goes from maildir to mdbox format (we can't to
convert to mdbox). Things seemed to be moving along, but there are quite
a few examples of dsync failing. I think the issue happens more often
with large mailboxes ( 50GB ).
We're running version 2.0.13.
doveconf -n:
# 2.0.13: /etc/dovecot/dovecot.conf
# OS: Linux 2.6.18-274.12.1.el5 x86_64 CentOS release 5.7 (Final)
auth_mechanisms = plain login
default_client_limit = 15000
default_process_limit = 1
disable_plaintext_auth = no
listen = *
mail_gid = vmail
mail_location = maildir:~/Maildir
mail_plugins = zlib
mail_uid = vmail
mmap_disable = yes
namespace {
inbox = yes
location =
prefix = INBOX.
separator = .
}
passdb {
args = /etc/dovecot/conf.d/dovecot-sql.conf.ext
driver = sql
}
plugin {
zlib_save = gz
}
protocols = imap pop3
service auth {
client_limit = 1
unix_listener auth-userdb {
mode = 0666
}
}
service imap-postlogin {
executable = script-login /usr/bin/postlogin-imap.sh
user = $default_internal_user
}
service imap {
drop_priv_before_exec = yes
executable = imap
process_limit = 1
}
service pop3-postlogin {
executable = script-login /usr/bin/postlogin-pop.sh
user = $default_internal_user
}
}
service pop3 {
drop_priv_before_exec = yes
executable = pop3
process_limit = 2500
}
ssl_cert = /etc/pki/dovecot/certs/dovecot.pem
ssl_key = /etc/pki/dovecot/private/dovecot.pem
userdb {
driver = prefetch
}
userdb {
args = /etc/dovecot/conf.d/dovecot-sql.conf.ext
driver = sql
}
protocol lmtp {
mail_plugins = zlib
}
protocol lda {
mail_plugins = zlib
}
protocol imap {
mail_max_userip_connections = 100
mail_plugins = zlib
}
protocol pop3 {
mail_max_userip_connections = 30
mail_plugins = zlib
}
...Jeff
Re: [Dovecot] dsync redesign
On Wed, 2012-03-28 at 11:07 -0500, Stan Hoeppner wrote: Locally attached/internal/JBOD storage typically offers the best application performance per dollar spent, until you get to things like backup scenarios, where off node network throughput is very low, and your backup software may suffer performance deficiencies, as is the issue titling this thread. Shipping full or incremental file backups across ethernet is extremely inefficient, especially with very large filesystems. This is where SAN arrays with snapshot capability come in really handy. I'm a new employee at the company. I was a bit surprised they were not using iSCSI. They claim they just can't risk the extra latency. I believe that you are right. It seems to me that offloading snapshots and backups to an iSCSI SAN would improve things. The problem is that this company has been burned on storage solutions more than once and they are a little skeptical that a product can scale to what they need. There are some SAN vendor names that are a four letter word here. So far, their newest FC SAN is performing well. I think having more, small, iSCSI boxes would be a good solution. One problem I've seen with smaller iSCSI products is that feature sets like snapshotting are not the best implementation. It works, but doing any sort of automation can be painful. The snap takes place wholly within the array and is very fast, without the problems you see with host based snapshots such as with Linux LVM, where you must first freeze the filesystem, wait for the snapshot to complete, which could be a very long time with a 1TB FS. While this occurs your clients must wait or timeout while trying to access mailboxes. With a SAN array snapshot system this isn't an issue as the snap is transparent to hosts with little or no performance degradation during the snap. Two relatively inexpensive units that have such snapshot capability are: How does this work? I've always had Linux create a snapshot. Would the SAN doing a snapshot without any OS buy-in cause the filesystem to be saved in an inconsistent state? I know that ext4 is pretty good at logging, but still, wouldn't this be a problem? http://www.equallogic.com/products/default.aspx?id=10613 http://h10010.www1.hp.com/wwpc/us/en/sm/WF04a/12169-304616-241493-241493-241493.html The Equallogic units are 1/10 GbE iSCSI only IIRC, whereas the HP can be had in 8Gb FC, 1/10Gb iSCSI, or 6Gb direct attach SAS. Each offer 4 or more host/network connection ports when equipped with dual controllers. There are many other vendors with similar models/capabilities. I mention these simply because Dell/HP are very popular and many OPs are already familiar with their servers and other products. I will take a look. I might have some convincing to do. There are 3 flavors of ZFS: native Oracle Solaris, native FreeBSD, Linux FUSE. Which were you using? If the last, that would fully explain the suck. There is one more that I had never used before coming on board here: ZFSonLinux. ZFSonLinux is a real kernel level fs plugin. My understanding is that they were using it on the backup machines with the front end dovecot machines using ext4. I'm told the metadata issue is a ZFS thing and they have the same problem on Solaris/Nexenta. I've relatively new here, but I'll ask around about XFS and see if anyone had tested it in the development environment. If they'd tested it properly, and relatively recently, I would think they'd have already replaced EXT4 on your Dovecot server. Unless others factors prevented such a migration. Or unless I've misunderstood the size of your maildir workload. I don't know the entire history of things. I think they really wanted to use ZFS for everything and then fell back to ext4 because it performed well enough in the cluster. Performance becomes an issue with backups using rsync. Rsync is faster than Dovecot's native dsync by a very large margin. I know that dsync is doing more than rsync, but still, seconds compared to over five minutes? That is a significant difference. The problem is that rsync can't get a perfect backup. ...Jeff
Re: [Dovecot] Need fast Maildir to mdbox conversion
On Wed, 2012-03-28 at 09:24 +0200, Jan-Frode Myklebust wrote: Why is it a problem that dsync takes a long time, when it can be done without downtime for the users? I just started our maildir-mdbox convertion yesterday, using the attached script. I only converted a little over 1 easy accounts (accounts with simple folder names, as I expect to run into problems once we start hitting accounts with trailing dot or broken latin1/utf8 characters in the folder names). I might agree it wasn't quick, but that really doesn't matter as the only downtime for the user is that he's potentially kicked out during the userdb update. I looked over your script. I plan on doing some trial runs with it. I think the trick where you re-run the sync and then boot the user off the connection should work pretty well. I hadn't totally fleshed out the scripting on the conversion since there is a lot more I need to do with the database and configuration files first. It appears I can use your script as a starting point for our configuration. ...Jeff -jf We're hoping that converting away from Maildir will help us speed up the backup processes by reducing the number of files to process.
Re: [Dovecot] Need fast Maildir to mdbox conversion
On Tue, 2012-03-27 at 20:00 -0700, Robin wrote: I'm writing a swiss-army (C-based, no bytecode crap languages) mailbox transcoding tool, since none appear to exist. To keep it simple, I/O to/from remote mailbox (connections) are not pipelined. It won't require more than MAXEMAILSIZE's worth of RAM (if one of the directions involves a remote connection), and so far when processing MIX, Maildir, and Mbox files, it's extremely fast. This sounds interesting. If it could so [sm]dbox, it would be very, very useful to large installations. ...Jeff
Re: [Dovecot] dsync is SLOW compared to rsync
On Thu, 2012-03-29 at 02:12 +0300, Timo Sirainen wrote: On 22.3.2012, at 23.25, Jeff Gustafson wrote: [root@n24 bu]# time dsync backup -u testu...@domain.com \ mdbox:/home/bu/testuser real1m9.519s user1m7.592s sys 0m1.126s Most of the time is spent on usermode CPU code. I doubt the problem is dsync itself, most likely the problem is mdbox's saving code. Or possibly index/cache code. Try the same dsync backup for: - mbox:/tmp/mbox - mbox:/tmp/mbox:INDEX=MEMORY - sdbox:/tmp/sdbox My tests show that maildir to mdbox or sdbox backup/conversions take about the same length in time. I noticed maybe a second or two difference between mdbox and sdbox). On a 3.1GB mailbox either one took about 6 minutes. Rsync, on the other hand, took less than a minute. I will re-run the tests with a maildir to maildir backup and see how long it takes. ...Jeff
Re: [Dovecot] dsync is SLOW compared to rsync
On Thu, 2012-03-29 at 03:06 +0300, Timo Sirainen wrote: time doveadm -o mail=mdbox:/tmp/mdbox import mdbox:/path/to/real/mdbox all This tried to write to /root for some reason and failed (dovecot 2.1.3): # time doveadm -o mail=maildir:/home/bu/test.mdbox import maildir:/home/users/u...@domain.com/Maildir all doveadm(root): Error: chdir(/root/) failed: Permission denied (euid=10025(vmail) egid=10025(vmail) missing +x perm: /root, we're not in group 0(root), dir owned by 0:0 mode=0550) doveadm(root): Error: chdir(/root) failed: Permission denied doveadm(root): Error: Can't find namespace for mailbox Trash doveadm(root): Error: Can't find namespace for mailbox test Or if it's the mail reading code: time doveadm fetch -u user@domain text all /dev/null This ran quicker than a full dsync. Only 40s for 3.1GB. rsync still beat it clocking in at 16s. I ran the fetch command twice figuring the files would get cached by the OS. ...Jeff
Re: [Dovecot] dsync is SLOW compared to rsync
On Thu, 2012-03-29 at 04:07 +0300, Timo Sirainen wrote: On 29.3.2012, at 3.48, Jeff Gustafson wrote: On Thu, 2012-03-29 at 03:06 +0300, Timo Sirainen wrote: time doveadm -o mail=mdbox:/tmp/mdbox import mdbox:/path/to/real/mdbox all This tried to write to /root for some reason and failed (dovecot 2.1.3): # time doveadm -o mail=maildir:/home/bu/test.mdbox import maildir:/home/users/u...@domain.com/Maildir all doveadm(root): Error: chdir(/root/) failed: Permission denied (euid=10025(vmail) egid=10025(vmail) missing +x perm: /root, we're not in group 0(root), dir owned by 0:0 mode=0550) doveadm(root): Error: chdir(/root) failed: Permission denied doveadm(root): Error: Can't find namespace for mailbox Trash doveadm(root): Error: Can't find namespace for mailbox test Maybe -o mail_home=/tmp parameter makes it happier? Or possibly it needs -u user@domain, but I'd test that first with a test account to make sure it doesn't break the mailbox in case the userdb lookup overrides some fields. That fixed some errors, but it still is having some sort of trouble with that command: # time doveadm -o mail=maildir:/home/bu/user.mdbox import -u u...@domain.com maildir:/home/users/user%domain.com/Maildir/ all doveadm(u...@domain.com): Error: Can't find namespace for mailbox Trash doveadm(u...@domain.com): Error: Can't find namespace for mailbox test ...Jeff
Re: [Dovecot] dsync redesign
On Tue, 2012-03-27 at 15:09 -0500, Stan Hoeppner wrote: On 3/26/2012 2:34 PM, Jeff Gustafson wrote: Do you have any suggestions for a distributed replicated filesystem that works well with dovecot? I've looked into glusterfs, but the latency is way too high for lots of small files. They claim this problem is fixed in glusterfs 3.3. NFS too slow for my installation so I don't see how any of the distributed filesystems would help me. I've also tried out ZFS, but it appears to have issues with metadata look ups with directories that have tens or hundreds of thousands of files in them. For me, the best filesystem is straight up ext4 running on locally attached storage. It sounds like you're in need of a more robust and capable storage/backup solution, such as an FC/iSCSI SAN array with PIT and/or incremental snapshot capability. We do have a FC system that another department is using. The company dropped quite a bit of cash on it for a specific purpose. Our department does not have access it to. People are somewhat afraid of iSCSI around here because they believe it will add too much latency to the overall IO performance. They're a big believer in locally attached disks. Less features, but very good performance. We thought ZFS would provide us with a nice snapshot and backup system (with zfs send). We never got that far once we discovered that ZFS doesn't work very well in this context. Running rsync on it gave us terrible performance. Also, you speak of a very large maildir store, with hundreds of thousands of directories, obviously many millions of files, of 1TB total size. Thus I would assume you have many thousands of users, if not 10s of thousands. It's a bit hard to believe you're not running XFS on your storage, given your level of parallelism. You'd get much better performance using XFS vs EXT4. Especially with kernel 2.6.39 or later which includes the delayed logging patch. This patch increases metadata write throughput by a factor of 2-50+ depending on thread count, and decreases IOPS and MB/s hitting the storage by about the same factor, depending on thread count. I've relatively new here, but I'll ask around about XFS and see if anyone had tested it in the development environment. ...Jeff
[Dovecot] Need fast Maildir to mdbox conversion
I looked around the 'Net to see if there might be a custom program for offline Maildir to mdbox conversion. So far I haven't turned up anything. The problem for us is that the dsync program simply takes a lot of time to convert mailboxes. I wonder if time could be saved with a program that is optimized to convert mailboxes without the fancy locking that dsync needs to do. Does have (or seen) a tool that could do this? We're hoping that converting away from Maildir will help us speed up the backup processes by reducing the number of files to process. ...Jeff
Re: [Dovecot] dsync is SLOW compared to rsync
On Fri, 2012-03-23 at 23:12 -0700, Linda Walsh wrote: Next -- bench cp -ax, against rsync -axHAX when it has to copy 75% of the data (cp ~6-8x speed). But for file speed, 'dd' is king, as it can use large buffers (~16MB gives best results on my local Gbit network), but it misses all those pesky acls and extended attrs, not to mention file perms...*sigh* Compare that to the I/O done 4k at a time by many older utils... cp -ax: real0m3.088s user0m0.034s sys 0m3.054s rsync -axHAX real0m15.850s user0m19.314s sys 0m8.816s dsync's time was over six minutes. Each time I cleared out the destination folder. dsync is doing something that is taking much, much, much longer to do. ...Jeff
Re: [Dovecot] dsync is SLOW compared to rsync
On Sat, 2012-03-24 at 14:21 +0100, Maarten Bezemer wrote: On Fri, 23 Mar 2012, Jeff Gustafson wrote: That didn't seem to make much of a difference. On a 3.1GB backup it shaved off 5 seconds. dsync's time was over 6 minutes with or without the mail_fsync=never. rsync copied the same 3.1GB mailbox in 15 seconds. It seems to me that dsync *should* be able to be just as fast, but it currently is spending way too much time doing something. What is it? Syncing 3.1GB in 15 seconds would require a speed of more than 200MB per second. Depending on the harddisks used, that would be quite a challenge. If you use rsync to only transfer the files that changed (based on file modification time) you may or may not miss files that have changed but still have the same time stamp. I assume you didn't use the --checksum parameter to rsync, right? The destination directory was empty. I was doing a full backup. dsync does so much more than simply copy some files... I realize that. I am hoping that the extra data that dsync has available to it would improve the speed of syncing backups. My baseline testing of simply backing up a mailbox to an empty directory shows that dsync is takes way too long to backup a single mailbox. I have over a terabyte of data to backup. I'm currently using rsync and it must traverse tens of thousands of files and check the time information. It works, but I was hoping dsync would be a better solution. dsync should be able to sync faster, by gulping in the index information for each mailbox. I haven't even moved to the point of sync'ing since the baseline test of simply exporting a mailbox is so slow. ...Jeff
Re: [Dovecot] dsync redesign
On Sat, 2012-03-24 at 08:19 +0100, Attila Nagy wrote: I personally think that Dovecot could gain much more if the amount of work going into fixing or improving dsync would go into making Dovecot to (be able of) use a high scale, distributed storage backend. I know it's much harder, because there are several major differences compared to the low latency and consistency problem free local file systems, but its fruits are also sweeter for the long term. :) Do you have any suggestions for a distributed replicated filesystem that works well with dovecot? I've looked into glusterfs, but the latency is way too high for lots of small files. They claim this problem is fixed in glusterfs 3.3. NFS too slow for my installation so I don't see how any of the distributed filesystems would help me. I've also tried out ZFS, but it appears to have issues with metadata look ups with directories that have tens or hundreds of thousands of files in them. For me, the best filesystem is straight up ext4 running on locally attached storage. I think a solid, fast dsync implementation would be very useful for a large installation. ...Jeff
Re: [Dovecot] dsync is SLOW compared to rsync
On Fri, 2012-03-23 at 19:02 +0100, Christoph Bußenius wrote: Hi, maybe try dsync -o mail_fsync=never. That didn't seem to make much of a difference. On a 3.1GB backup it shaved off 5 seconds. dsync's time was over 6 minutes with or without the mail_fsync=never. rsync copied the same 3.1GB mailbox in 15 seconds. It seems to me that dsync *should* be able to be just as fast, but it currently is spending way too much time doing something. What is it? ...Jeff
[Dovecot] dsync is SLOW compared to rsync
Hi all, We are currently using snapshots and rsync to backup a large mail server to a backup mail server. I have been looking into using dsync to replace rsync in hopes that it would make backups more efficient. I decided to test the performance using a single mailbox. Unfortunately dsync seems to run much slower than rsync. Rsync was able to sync the mailbox in 2 seconds. dsync took over a minute. The test was run so that the source and destination are on the same filesystem. We would like to using the new replication system, but that doesn't seem likely since the performance of the underlying dsync is so much slower than rsync. Even with the extra work that dsync is doing I can't believe the difference in performance would be that great. I realize that dsync is actively being worked on and I hope bringing attention to performance issue will provoke some ideas on how to improve it. Here is the output of the tests using dovecot 2.1.3: [root@n24 bu]# du -hs /home/10.0.1.101/1009/users/testuser% domain.com/Maildir/ 517M/home/10.0.1.101/1009/users/testuser%domain.com/Maildir/ [root@n24 bu]# time rsync -va /home/10.0.1.101/1009/users/testuser% domain.com/Maildir/ . sending incremental file list Maildir/ Maildir/dovecot-uidlist [ ... deleted cruft ... ] Maildir/cur/1332387577.M381054P27635.n24,S=14215502,W=14448554:2, Maildir/new/ Maildir/tmp/ sent 540927820 bytes received 1222 bytes 216371616.80 bytes/sec total size is 540855755 speedup is 1.00 real0m2.677s user0m3.184s sys 0m1.513s [root@n24 bu]# time dsync backup -u testu...@domain.com \ mdbox:/home/bu/testuser real1m9.519s user1m7.592s sys 0m1.126s [root@n24 bu]# time dsync backup -u testu...@domain.com \ sdbox:/home/bu/testuser2 real1m2.164s user1m0.882s sys 0m0.993s [root@n24 bu]#
