[Dovecot] feature request: pipe for custom quota dict queries

2013-05-03 Thread Jeff Gustafson
Just a followup to my previous post. I appears that a pipe for quota queries via dict is not supported right now. Because of the way we group things we need more flexibility. I'd like to propose that dovecot should support quota queries just like userdb dict queries. My expertise

[Dovecot] quota and dict

2013-05-02 Thread Jeff Gustafson
I have a question about using dict and quotas. I want dovecot to send quota queries to a custom dict server over a socket. I'm doing this because I can't do group quotas based on domain since a customer can have each of their users associated with different domains under a single account.

[Dovecot] accumulating orphaned processes

2013-02-15 Thread Jeff Gustafson
I'm getting orphaned processes on our mail servers. Some machines has processes hanging around from a week ago. I'm running the current stable of dovecot 2.1.15, but the problem has been happening for a while now. I thought it would be good to report the issue. I ran an strace on

Re: [Dovecot] key - object mailstore

2012-09-14 Thread Jeff Gustafson
On Fri, 2012-09-14 at 17:59 +0300, Timo Sirainen wrote: I've a whole new design for it and I was planning on implementing it for v2.2. Do you want to help coding it? :) Which storage would you want to use? The generic idea is: - only one server accesses one user simultaneously - index

[Dovecot] more dsync issues

2012-08-22 Thread Jeff Gustafson
It hit another dsync snag. We have a script that executes a dsync backup command to a remote host. No write should be happening on the remote host (the backup host). The error is: dsync-local(anonym...@domain.com): Error: remote: dsync-remote(anonym...@domain.com): Error: Can't delete

Re: [Dovecot] dsync backup gets stuck... fails

2012-08-15 Thread Jeff Gustafson
On Wed, 2012-08-15 at 12:27 +0300, Timo Sirainen wrote: On 15.8.2012, at 5.41, Jeff Gustafson wrote: I have an issue related to this problem. dsync returns an error 75 when it detects the source mailbox is empty (client probably pop3'd all of their email). It also returns an error 75

Re: [Dovecot] dsync backup gets stuck... fails

2012-08-14 Thread Jeff Gustafson
On Tue, 2012-08-14 at 23:23 +0300, Timo Sirainen wrote: On 11.8.2012, at 0.54, Jeff Gustafson wrote: More dsync issues. We were running 2.1.7 and we updated to 2.1.9. Same problem with both versions. I'm getting an error 75 on about 40 boxes out of 1800. It is the same list

Re: [Dovecot] dsync backup gets stuck... fails

2012-08-14 Thread Jeff Gustafson
On Tue, 2012-08-14 at 23:23 +0300, Timo Sirainen wrote: On 11.8.2012, at 0.54, Jeff Gustafson wrote: What can I do to help the developers locate the bug? Those hangs are a little bit annoying to debug, and the whole code has been rewritten for v2.2 already in a way that should make

Re: [Dovecot] dsync backup gets stuck... fails

2012-08-13 Thread Jeff Gustafson
On Sat, 2012-08-11 at 15:50 +0200, Daniel Parthey wrote: Maybe you have run into the epoll kernel bug under RHEL/CentOS: :) Yeah... been there, done that. We found that bug within *minutes* of ksplice updating the kernel. I don't think this is an epoll thing because, if it was,

[Dovecot] dsync backup gets stuck... fails

2012-08-10 Thread Jeff Gustafson
More dsync issues. We were running 2.1.7 and we updated to 2.1.9. Same problem with both versions. I'm getting an error 75 on about 40 boxes out of 1800. It is the same list of boxes every time we use 'dsync backup' to backup the server. dsync seems to stop communicating to the

Re: [Dovecot] dsync backup gets stuck... fails

2012-08-10 Thread Jeff Gustafson
On Sat, 2012-08-11 at 00:56 +0200, Daniel Parthey wrote: Hi Jeff, Please start by following the instructions at http://dovecot.org/bugreport.html and post your 'doveconf -n' output in order to provide possibly important information about your system and configs. Storage is local

[Dovecot] more dsync problems: Error: proxy client timed out

2012-07-10 Thread Jeff Gustafson
Hi all, I'm using dsync for backups. I recently upgraded to 2.1.7 due to issues with dsync mirror/backups. 2.1.7 fixed the issue with the guid conflict, but now I'm seeing this error on a couple of mailboxes: Error: proxy client timed out (waiting for output stream to flush, 0 bytes left)

[Dovecot] more info (more dsync problems)

2012-07-10 Thread Jeff Gustafson
Rerunning the dsync command right after the timeout (see previous message) gave me these errors: dsync-local(vmail): Error: remote: dsync-remote(vmail): Error: Corrupted dbox file /home/.incoming_mail_migrations/users/user%domain.com/mdbox/s torage/m.60 (around offset=807789): msg header has bad

Re: [Dovecot] dsync error: Mailboxes don't have unique GUIDs

2012-06-25 Thread Jeff Gustafson
On Sat, 2012-06-23 at 06:29 -0400, Charles Marcus wrote: snip # 2.0.13: /etc/dovecot/dovecot.conf As you are aware (since you participated in the thread discussion about this months ago), Timo is working on a total rewrite of dsync, and if memory serves, it is mainly for 2.1+, and it

Re: [Dovecot] dsync error: Mailboxes don't have unique GUIDs

2012-06-25 Thread Jeff Gustafson
On Mon, 2012-06-25 at 19:54 +0300, Timo Sirainen wrote: On 25.6.2012, at 19.49, Charles Marcus wrote: I'd suggest you try again with 2.1.7... The rewritten dsync is in v2.2 tree. v2.1's dsync is a fixed version of v2.0's dsync. I have no idea why v2.1's dsync would be less reliable than

[Dovecot] dsync error: Mailboxes don't have unique GUIDs

2012-06-22 Thread Jeff Gustafson
I'm getting an error backing up mailboxes. I'm using the mirror command: dsync -fvo mail_home=/home/users/bob mirror ssh vmail@10.1.4.1 dsync -o mail_home=/home/.incoming_mail_migrations/users/bob dsync-remote(vmail): Error: Mailboxes don't have unique GUIDs:

Re: [Dovecot] dsync redesign

2012-03-28 Thread Jeff Gustafson
On Wed, 2012-03-28 at 11:07 -0500, Stan Hoeppner wrote: Locally attached/internal/JBOD storage typically offers the best application performance per dollar spent, until you get to things like backup scenarios, where off node network throughput is very low, and your backup software may suffer

Re: [Dovecot] Need fast Maildir to mdbox conversion

2012-03-28 Thread Jeff Gustafson
On Wed, 2012-03-28 at 09:24 +0200, Jan-Frode Myklebust wrote: Why is it a problem that dsync takes a long time, when it can be done without downtime for the users? I just started our maildir-mdbox convertion yesterday, using the attached script. I only converted a little over 1 easy

Re: [Dovecot] Need fast Maildir to mdbox conversion

2012-03-28 Thread Jeff Gustafson
On Tue, 2012-03-27 at 20:00 -0700, Robin wrote: I'm writing a swiss-army (C-based, no bytecode crap languages) mailbox transcoding tool, since none appear to exist. To keep it simple, I/O to/from remote mailbox (connections) are not pipelined. It won't require more than MAXEMAILSIZE's

Re: [Dovecot] dsync is SLOW compared to rsync

2012-03-28 Thread Jeff Gustafson
On Thu, 2012-03-29 at 02:12 +0300, Timo Sirainen wrote: On 22.3.2012, at 23.25, Jeff Gustafson wrote: [root@n24 bu]# time dsync backup -u testu...@domain.com \ mdbox:/home/bu/testuser real1m9.519s user1m7.592s sys 0m1.126s Most of the time is spent on usermode CPU

Re: [Dovecot] dsync is SLOW compared to rsync

2012-03-28 Thread Jeff Gustafson
On Thu, 2012-03-29 at 03:06 +0300, Timo Sirainen wrote: time doveadm -o mail=mdbox:/tmp/mdbox import mdbox:/path/to/real/mdbox all This tried to write to /root for some reason and failed (dovecot 2.1.3): # time doveadm -o mail=maildir:/home/bu/test.mdbox import

Re: [Dovecot] dsync is SLOW compared to rsync

2012-03-28 Thread Jeff Gustafson
On Thu, 2012-03-29 at 04:07 +0300, Timo Sirainen wrote: On 29.3.2012, at 3.48, Jeff Gustafson wrote: On Thu, 2012-03-29 at 03:06 +0300, Timo Sirainen wrote: time doveadm -o mail=mdbox:/tmp/mdbox import mdbox:/path/to/real/mdbox all This tried to write to /root for some

Re: [Dovecot] dsync redesign

2012-03-27 Thread Jeff Gustafson
On Tue, 2012-03-27 at 15:09 -0500, Stan Hoeppner wrote: On 3/26/2012 2:34 PM, Jeff Gustafson wrote: Do you have any suggestions for a distributed replicated filesystem that works well with dovecot? I've looked into glusterfs, but the latency is way too high for lots of small files

[Dovecot] Need fast Maildir to mdbox conversion

2012-03-27 Thread Jeff Gustafson
I looked around the 'Net to see if there might be a custom program for offline Maildir to mdbox conversion. So far I haven't turned up anything. The problem for us is that the dsync program simply takes a lot of time to convert mailboxes. I wonder if time could be saved with a program that

Re: [Dovecot] dsync is SLOW compared to rsync

2012-03-26 Thread Jeff Gustafson
On Fri, 2012-03-23 at 23:12 -0700, Linda Walsh wrote: Next -- bench cp -ax, against rsync -axHAX when it has to copy 75% of the data (cp ~6-8x speed). But for file speed, 'dd' is king, as it can use large buffers (~16MB gives best results on my local Gbit network), but it misses all

Re: [Dovecot] dsync is SLOW compared to rsync

2012-03-26 Thread Jeff Gustafson
On Sat, 2012-03-24 at 14:21 +0100, Maarten Bezemer wrote: On Fri, 23 Mar 2012, Jeff Gustafson wrote: That didn't seem to make much of a difference. On a 3.1GB backup it shaved off 5 seconds. dsync's time was over 6 minutes with or without the mail_fsync=never. rsync copied the same

Re: [Dovecot] dsync redesign

2012-03-26 Thread Jeff Gustafson
On Sat, 2012-03-24 at 08:19 +0100, Attila Nagy wrote: I personally think that Dovecot could gain much more if the amount of work going into fixing or improving dsync would go into making Dovecot to (be able of) use a high scale, distributed storage backend. I know it's much harder,

Re: [Dovecot] dsync is SLOW compared to rsync

2012-03-23 Thread Jeff Gustafson
On Fri, 2012-03-23 at 19:02 +0100, Christoph Bußenius wrote: Hi, maybe try dsync -o mail_fsync=never. That didn't seem to make much of a difference. On a 3.1GB backup it shaved off 5 seconds. dsync's time was over 6 minutes with or without the mail_fsync=never. rsync copied the same

[Dovecot] dsync is SLOW compared to rsync

2012-03-22 Thread Jeff Gustafson
Hi all, We are currently using snapshots and rsync to backup a large mail server to a backup mail server. I have been looking into using dsync to replace rsync in hopes that it would make backups more efficient. I decided to test the performance using a single mailbox. Unfortunately dsync