[Dovecot] feature request: pipe for custom quota dict queries

2013-05-03 Thread Jeff Gustafson
Just a followup to my previous post. I appears that a pipe for quota queries via dict is not supported right now. Because of the way we group things we need more flexibility. I'd like to propose that dovecot should support quota queries just like userdb dict queries. My expertise i

[Dovecot] quota and dict

2013-05-02 Thread Jeff Gustafson
I have a question about using dict and quotas. I want dovecot to send quota queries to a custom dict server over a socket. I'm doing this because I can't do group quotas based on domain since a customer can have each of their users associated with different domains under a single account. I

[Dovecot] accumulating orphaned processes

2013-02-15 Thread Jeff Gustafson
I'm getting orphaned processes on our mail servers. Some machines has processes hanging around from a week ago. I'm running the current stable of dovecot 2.1.15, but the problem has been happening for a while now. I thought it would be good to report the issue. I ran an strace on o

Re: [Dovecot] key -> object mailstore

2012-09-14 Thread Jeff Gustafson
On Fri, 2012-09-14 at 17:59 +0300, Timo Sirainen wrote: > I've a whole new design for it and I was planning on implementing it > for v2.2. Do you want to help coding it? :) Which storage would you > want to use? > > The generic idea is: > - only one server accesses one user simultaneously > - in

[Dovecot] more dsync issues

2012-08-22 Thread Jeff Gustafson
It hit another dsync snag. We have a script that executes a dsync backup command to a remote host. No write should be happening on the remote host (the backup host). The error is: dsync-local(anonym...@domain.com): Error: remote: dsync-remote(anonym...@domain.com): Error: Can't delete mail

Re: [Dovecot] dsync backup gets stuck... fails

2012-08-15 Thread Jeff Gustafson
On Wed, 2012-08-15 at 12:27 +0300, Timo Sirainen wrote: > On 15.8.2012, at 5.41, Jeff Gustafson wrote: > > > I have an issue related to this problem. dsync returns an error 75 when > > it detects the source mailbox is empty (client probably pop3'd all of > > the

Re: [Dovecot] dsync backup gets stuck... fails

2012-08-14 Thread Jeff Gustafson
On Tue, 2012-08-14 at 23:23 +0300, Timo Sirainen wrote: > On 11.8.2012, at 0.54, Jeff Gustafson wrote: > > What can I do to help the developers locate the bug? > > Those hangs are a little bit annoying to debug, and the whole code has > been rewritten for v2.2 already in

Re: [Dovecot] dsync backup gets stuck... fails

2012-08-14 Thread Jeff Gustafson
On Tue, 2012-08-14 at 23:23 +0300, Timo Sirainen wrote: > On 11.8.2012, at 0.54, Jeff Gustafson wrote: > > > More dsync issues. We were running 2.1.7 and we updated to 2.1.9. Same > > problem with both versions. > > I'm getting an error 75 on about 40 boxe

Re: [Dovecot] dsync backup gets stuck... fails

2012-08-14 Thread Jeff Gustafson
I ran a rsync on the mailboxes that I was having issues with. I re-ran rsync until I had a full sync with no further updates. Then I ran a dsync. dsync was able to run without issue. If I wipe out the target directory and re-run dsync, I'm back to dsync getting stuck. Running rsync

Re: [Dovecot] dsync backup gets stuck... fails

2012-08-13 Thread Jeff Gustafson
On Sat, 2012-08-11 at 15:50 +0200, Daniel Parthey wrote: > Maybe you have run into the epoll kernel bug under RHEL/CentOS: > :) Yeah... been there, done that. We found that bug within *minutes* of ksplice updating the kernel. I don't think this is an epoll thing because, if it was, custo

Re: [Dovecot] dsync backup gets stuck... fails

2012-08-10 Thread Jeff Gustafson
On Sat, 2012-08-11 at 00:56 +0200, Daniel Parthey wrote: > Hi Jeff, > Please start by following the instructions at > http://dovecot.org/bugreport.html > and post your 'doveconf -n' output in order to provide > possibly important information about your system and configs. Storage is local hardware

[Dovecot] dsync backup gets stuck... fails

2012-08-10 Thread Jeff Gustafson
More dsync issues. We were running 2.1.7 and we updated to 2.1.9. Same problem with both versions. I'm getting an error 75 on about 40 boxes out of 1800. It is the same list of boxes every time we use 'dsync backup' to backup the server. dsync seems to stop communicating to the bac

[Dovecot] more info (more dsync problems)

2012-07-10 Thread Jeff Gustafson
Rerunning the dsync command right after the timeout (see previous message) gave me these errors: dsync-local(vmail): Error: remote: dsync-remote(vmail): Error: Corrupted dbox file /home/.incoming_mail_migrations/users/user%domain.com/mdbox/s torage/m.60 (around offset=807789): msg header has bad

[Dovecot] more dsync problems: Error: proxy client timed out

2012-07-10 Thread Jeff Gustafson
Hi all, I'm using dsync for backups. I recently upgraded to 2.1.7 due to issues with dsync mirror/backups. 2.1.7 fixed the issue with the guid conflict, but now I'm seeing this error on a couple of mailboxes: Error: proxy client timed out (waiting for output stream to flush, 0 bytes left)

Re: [Dovecot] dsync error: "Mailboxes don't have unique GUIDs"

2012-06-25 Thread Jeff Gustafson
On Mon, 2012-06-25 at 19:54 +0300, Timo Sirainen wrote: > On 25.6.2012, at 19.49, Charles Marcus wrote: > > I'd suggest you try again with 2.1.7... > > The rewritten dsync is in v2.2 tree. v2.1's dsync is a fixed version > of v2.0's dsync. I have no idea why v2.1's dsync would be less > reliable t

Re: [Dovecot] dsync error: "Mailboxes don't have unique GUIDs"

2012-06-25 Thread Jeff Gustafson
On Sat, 2012-06-23 at 06:29 -0400, Charles Marcus wrote: > > > > # 2.0.13: /etc/dovecot/dovecot.conf > > As you are aware (since you participated in the thread discussion about > this months ago), Timo is working on a total rewrite of dsync, and if > memory serves, it is mainly for 2.1+, and i

[Dovecot] dsync error: "Mailboxes don't have unique GUIDs"

2012-06-22 Thread Jeff Gustafson
I'm getting an error backing up mailboxes. I'm using the mirror command: dsync -fvo mail_home=/home/users/bob mirror ssh vmail@10.1.4.1 dsync -o mail_home=/home/.incoming_mail_migrations/users/bob dsync-remote(vmail): Error: Mailboxes don't have unique GUIDs: 1ef6ee37c694894d78310581a

Re: [Dovecot] dsync is SLOW compared to rsync

2012-03-28 Thread Jeff Gustafson
On Thu, 2012-03-29 at 04:07 +0300, Timo Sirainen wrote: > On 29.3.2012, at 3.48, Jeff Gustafson wrote: > > > On Thu, 2012-03-29 at 03:06 +0300, Timo Sirainen wrote: > > > >> time doveadm -o mail=mdbox:/tmp/mdbox import mdbox:/path/to/real/mdbox "" > &g

Re: [Dovecot] dsync is SLOW compared to rsync

2012-03-28 Thread Jeff Gustafson
On Thu, 2012-03-29 at 03:06 +0300, Timo Sirainen wrote: > time doveadm -o mail=mdbox:/tmp/mdbox import mdbox:/path/to/real/mdbox "" all This tried to write to /root for some reason and failed (dovecot 2.1.3): # time doveadm -o mail=maildir:/home/bu/test.mdbox import maildir:/home/users/u

Re: [Dovecot] dsync is SLOW compared to rsync

2012-03-28 Thread Jeff Gustafson
On Thu, 2012-03-29 at 02:12 +0300, Timo Sirainen wrote: > On 22.3.2012, at 23.25, Jeff Gustafson wrote: > > > [root@n24 bu]# time dsync backup -u testu...@domain.com \ > > mdbox:/home/bu/testuser > > > > real1m9.519s > > user1m7.592s > > sys

Re: [Dovecot] Need fast Maildir to mdbox conversion

2012-03-28 Thread Jeff Gustafson
On Tue, 2012-03-27 at 20:00 -0700, Robin wrote: > I'm writing a swiss-army (C-based, no bytecode crap languages) mailbox > "transcoding" tool, since none appear to exist. To keep it simple, I/O > to/from "remote" mailbox (connections) are not pipelined. It won't > require more than MAXEMAILSI

Re: [Dovecot] Need fast Maildir to mdbox conversion

2012-03-28 Thread Jeff Gustafson
On Wed, 2012-03-28 at 09:24 +0200, Jan-Frode Myklebust wrote: > Why is it a problem that dsync takes a long time, when it can be done > without downtime for the users? > > I just started our maildir->mdbox convertion yesterday, using the > attached script. I only converted a little over 1 eas

Re: [Dovecot] dsync redesign

2012-03-28 Thread Jeff Gustafson
On Wed, 2012-03-28 at 11:07 -0500, Stan Hoeppner wrote: > Locally attached/internal/JBOD storage typically offers the best > application performance per dollar spent, until you get to things like > backup scenarios, where off node network throughput is very low, and > your backup software may suff

[Dovecot] Need fast Maildir to mdbox conversion

2012-03-27 Thread Jeff Gustafson
I looked around the 'Net to see if there might be a custom program for offline Maildir to mdbox conversion. So far I haven't turned up anything. The problem for us is that the dsync program simply takes a lot of time to convert mailboxes. I wonder if time could be saved with a program that

Re: [Dovecot] dsync redesign

2012-03-27 Thread Jeff Gustafson
On Tue, 2012-03-27 at 15:09 -0500, Stan Hoeppner wrote: > On 3/26/2012 2:34 PM, Jeff Gustafson wrote: > > > Do you have any suggestions for a distributed replicated filesystem > > that works well with dovecot? I've looked into glusterfs, but the > > latency is w

Re: [Dovecot] dsync redesign

2012-03-26 Thread Jeff Gustafson
On Sat, 2012-03-24 at 08:19 +0100, Attila Nagy wrote: > > I personally think that Dovecot could gain much more if the amount of > work going into fixing or improving dsync would go into making Dovecot > to (be able of) use a high scale, distributed storage backend. > I know it's much harder, be

Re: [Dovecot] dsync is SLOW compared to rsync

2012-03-26 Thread Jeff Gustafson
On Sat, 2012-03-24 at 14:21 +0100, Maarten Bezemer wrote: > On Fri, 23 Mar 2012, Jeff Gustafson wrote: > > > That didn't seem to make much of a difference. On a 3.1GB backup it > > shaved off 5 seconds. dsync's time was over 6 minutes with or without > > the

Re: [Dovecot] dsync is SLOW compared to rsync

2012-03-26 Thread Jeff Gustafson
On Fri, 2012-03-23 at 23:12 -0700, Linda Walsh wrote: > Next -- bench "cp -ax", against rsync -axHAX when it has to copy >75% of > the data (cp ~6-8x speed). But for file speed, 'dd' is king, as it can > use large buffers (~16MB gives best results on my local Gbit network), > but it > misses a

Re: [Dovecot] dsync is SLOW compared to rsync

2012-03-23 Thread Jeff Gustafson
On Fri, 2012-03-23 at 19:02 +0100, Christoph Bußenius wrote: > Hi, > > maybe try "dsync -o mail_fsync=never". That didn't seem to make much of a difference. On a 3.1GB backup it shaved off 5 seconds. dsync's time was over 6 minutes with or without the mail_fsync=never. rsync copied the sa

[Dovecot] dsync is SLOW compared to rsync

2012-03-22 Thread Jeff Gustafson
Hi all, We are currently using snapshots and rsync to backup a large mail server to a backup mail server. I have been looking into using dsync to replace rsync in hopes that it would make backups more efficient. I decided to test the performance using a single mailbox. Unfortunately dsync s