Re: [Dovecot] dsync on large mailbox "fails"

2012-04-24 Thread Timo Sirainen
On 24.4.2012, at 20.05, Jeff Kletsky wrote: > On 4/24/12 3:26 AM, Timo Sirainen wrote: >> On 24.4.2012, at 6.38, Jeff Kletsky wrote: >> >>> 25934 jeff 1 1030 135M 102M CPU22 128:47 100.00% >>> doveadm >> Looks like it could be in an infinite loop. To find out where: > http:/

Re: [Dovecot] dsync on large mailbox "fails"

2012-04-24 Thread Jeff Kletsky
On 4/24/12 3:26 AM, Timo Sirainen wrote: On 24.4.2012, at 6.38, Jeff Kletsky wrote: 25934 jeff 1 1030 135M 102M CPU22 128:47 100.00% doveadm Looks like it could be in an infinite loop. To find out where: 1) strace for a few seconds: strace -tt -p 25934 2>log 2) get gdb

Re: [Dovecot] dsync on large mailbox "fails"

2012-04-24 Thread Timo Sirainen
On 24.4.2012, at 6.38, Jeff Kletsky wrote: > 25934 jeff 1 1030 135M 102M CPU22 128:47 100.00% doveadm Looks like it could be in an infinite loop. To find out where: 1) strace for a few seconds: strace -tt -p 25934 2>log 2) get gdb backtrace a couple of times (2-5 should be

Re: [Dovecot] dsync on large mailbox "fails"

2012-04-23 Thread Jeff Kletsky
On 4/23/12 12:56 PM, Jeff Kletsky wrote: I've been trying to convert an existing set of mbox mail to maildir using dsync, but it seems to fail when it gets to the large boxes. $ dsync -Dv mirror maildir:~/Maildir Everything seems to work reasonably well until the first of the large boxes hits

[Dovecot] dsync on large mailbox "fails"

2012-04-23 Thread Jeff Kletsky
I've been trying to convert an existing set of mbox mail to maildir using dsync, but it seems to fail when it gets to the large boxes. $ dsync -Dv mirror maildir:~/Maildir Everything seems to work reasonably well until the first of the large boxes hits: dsync(jeff): Warning: Maildir /home/je

Re: [Dovecot] dsync: convert only mbox in /var/mail to maildir?

2012-04-20 Thread Timo Sirainen
On 20.4.2012, at 22.26, Jeff Simmons wrote: > I'm migrating an old UW pop server to a new Dovecot server. The old server > did > not allow mail to be stored on the server, so only mbox files exist in > /var/mail, with nothing in /home/$user. I'd like to copy the /var/mail > directory over to t

[Dovecot] dsync: convert only mbox in /var/mail to maildir?

2012-04-20 Thread Jeff Simmons
I'm migrating an old UW pop server to a new Dovecot server. The old server did not allow mail to be stored on the server, so only mbox files exist in /var/mail, with nothing in /home/$user. I'd like to copy the /var/mail directory over to the new server and then use dsync to convert the mbox fil

[Dovecot] Dsync test on 2.1.3

2012-04-10 Thread Jitendra Bhaskar
Hi, I am using dovecot version 2.1.3 on centos. I tested dsync from my live server to one of backup server. But I got that its working for most of the user and there is problem for few user. Some users got duplicated mail to their mailbox and I saw there is some extra folders i.e Sent Items_* , Dr

Re: [Dovecot] dsync redesign

2012-04-03 Thread Charles Marcus
On 2012-04-02 7:15 PM, Micah Anderson wrote: Charles Marcus writes: On 2012-03-27 11:47 AM, Micah Anderson wrote: One would be the ability to perform *intelligent* incremental / rotated backups. I can do this now by running a dsync backup operation and then doing manual hardlinking or moving

Re: [Dovecot] dsync redesign

2012-04-02 Thread Micah Anderson
Charles Marcus writes: > On 2012-03-27 11:47 AM, Micah Anderson wrote: >> One would be the ability to perform *intelligent* incremental / >> rotated backups. I can do this now by running a dsync backup >> operation and then doing manual hardlinking or moving of the backup >> directories (daily.1

Re: [Dovecot] Dsync automation

2012-04-02 Thread Micah Anderson
Jitendra Bhaskar writes: > I am using dovecot-2.1.3 on centos 5.7. Its running fine. I have two > systems for mail server one is for backup. Currently I am running rsync > daily. But I want to use dsync. I went through all the > http://wiki2.dovecot.org/Tools/Dsync and > http://blog.dovecot.org/2

[Dovecot] Dsync automation

2012-04-02 Thread Jitendra Bhaskar
Hi, I am using dovecot-2.1.3 on centos 5.7. Its running fine. I have two systems for mail server one is for backup. Currently I am running rsync daily. But I want to use dsync. I went through all the http://wiki2.dovecot.org/Tools/Dsync and http://blog.dovecot.org/2012/02/dovecot-clustering-with-

Re: [Dovecot] dsync redesign

2012-03-29 Thread Robin
On 3/29/2012 5:24 AM, Stan Hoeppner wrote: > This happens with a lot of "fan boys". There was so much hype > surrounding ZFS that even many logically thinking people were frothing > at the mouth waiting to get their hands on it. Then, as with many/most > things in the tech world, the goods didn't

Re: [Dovecot] dsync redesign

2012-03-29 Thread Stan Hoeppner
On 3/28/2012 3:54 PM, Jeff Gustafson wrote: > On Wed, 2012-03-28 at 11:07 -0500, Stan Hoeppner wrote: > >> Locally attached/internal/JBOD storage typically offers the best >> application performance per dollar spent, until you get to things like >> backup scenarios, where off node network throughp

Re: [Dovecot] dsync is SLOW compared to rsync

2012-03-28 Thread ncjeffgus
On Thu, 29 Mar 2012 07:04:26 +0300, Timo Sirainen wrote: On 29.3.2012, at 5.07, Jeff Gustafson wrote: That fixed some errors, but it still is having some sort of trouble with that command: # time doveadm -o mail=maildir:/home/bu/user.mdbox import -u u...@domain.com maildir:/home/users/

Re: [Dovecot] dsync is SLOW compared to rsync

2012-03-28 Thread Timo Sirainen
On 29.3.2012, at 5.07, Jeff Gustafson wrote: > That fixed some errors, but it still is having some sort of trouble > with that command: > > # time doveadm -o mail=maildir:/home/bu/user.mdbox import -u > u...@domain.com maildir:/home/users/user%domain.com/Maildir/ "" all > doveadm(u...@domai

Re: [Dovecot] dsync is SLOW compared to rsync

2012-03-28 Thread Jeff Gustafson
On Thu, 2012-03-29 at 04:07 +0300, Timo Sirainen wrote: > On 29.3.2012, at 3.48, Jeff Gustafson wrote: > > > On Thu, 2012-03-29 at 03:06 +0300, Timo Sirainen wrote: > > > >> time doveadm -o mail=mdbox:/tmp/mdbox import mdbox:/path/to/real/mdbox "" > >> all > > > > This tried to write to /ro

Re: [Dovecot] dsync is SLOW compared to rsync

2012-03-28 Thread Timo Sirainen
On 29.3.2012, at 3.48, Jeff Gustafson wrote: > On Thu, 2012-03-29 at 03:06 +0300, Timo Sirainen wrote: > >> time doveadm -o mail=mdbox:/tmp/mdbox import mdbox:/path/to/real/mdbox "" all > > This tried to write to /root for some reason and failed (dovecot > 2.1.3): > > # time doveadm -o ma

Re: [Dovecot] dsync is SLOW compared to rsync

2012-03-28 Thread Jeff Gustafson
On Thu, 2012-03-29 at 03:06 +0300, Timo Sirainen wrote: > time doveadm -o mail=mdbox:/tmp/mdbox import mdbox:/path/to/real/mdbox "" all This tried to write to /root for some reason and failed (dovecot 2.1.3): # time doveadm -o mail=maildir:/home/bu/test.mdbox import maildir:/home/users/u

Re: [Dovecot] dsync is SLOW compared to rsync

2012-03-28 Thread Timo Sirainen
On 29.3.2012, at 2.51, Jeff Gustafson wrote: >> Most of the time is spent on usermode CPU code. I doubt the problem is dsync >> itself, most likely the problem is mdbox's saving code. Or possibly >> index/cache code. Try the same dsync backup for: >> >> - mbox:/tmp/mbox >> - mbox:/tmp/mbox:INDE

Re: [Dovecot] dsync is SLOW compared to rsync

2012-03-28 Thread Jeff Gustafson
On Thu, 2012-03-29 at 02:12 +0300, Timo Sirainen wrote: > On 22.3.2012, at 23.25, Jeff Gustafson wrote: > > > [root@n24 bu]# time dsync backup -u testu...@domain.com \ > > mdbox:/home/bu/testuser > > > > real1m9.519s > > user1m7.592s > > sys 0m1.126s > > Most of the time is spent on

Re: [Dovecot] dsync is SLOW compared to rsync

2012-03-28 Thread Timo Sirainen
On 22.3.2012, at 23.25, Jeff Gustafson wrote: > [root@n24 bu]# time dsync backup -u testu...@domain.com \ > mdbox:/home/bu/testuser > > real1m9.519s > user1m7.592s > sys 0m1.126s Most of the time is spent on usermode CPU code. I doubt the problem is dsync itself, most likely the pro

Re: [Dovecot] dsync redesign

2012-03-28 Thread Timo Sirainen
On 23.3.2012, at 23.25, Timo Sirainen wrote: > and even if you don't understand that, here's another document disguising as > an algorithm class problem :) If anyone has thoughts on how to solve it, > would be great: > > http://dovecot.org/tmp/dsync-redesign-problem.txt > > It only deals with

Re: [Dovecot] dsync redesign

2012-03-28 Thread Timo Sirainen
On 27.3.2012, at 1.14, Michescu Andrei wrote: > This being said and acknowledged here are my 2 cents: > > I think that the current '1 brain / 2 workers' seems to be the correct > model. The "the client" connects to the "server" and pushes the local > changes and after retrieves the updated/new it

Re: [Dovecot] dsync redesign

2012-03-28 Thread Jeff Gustafson
On Wed, 2012-03-28 at 11:07 -0500, Stan Hoeppner wrote: > Locally attached/internal/JBOD storage typically offers the best > application performance per dollar spent, until you get to things like > backup scenarios, where off node network throughput is very low, and > your backup software may suff

Re: [Dovecot] dsync redesign

2012-03-28 Thread Stan Hoeppner
On 3/27/2012 3:57 PM, Jeff Gustafson wrote: > We do have a FC system that another department is using. The company > dropped quite a bit of cash on it for a specific purpose. Our department > does not have access it to. People are somewhat afraid of iSCSI around > here because they believe

Re: [Dovecot] dsync redesign

2012-03-27 Thread Jeff Gustafson
On Tue, 2012-03-27 at 15:09 -0500, Stan Hoeppner wrote: > On 3/26/2012 2:34 PM, Jeff Gustafson wrote: > > > Do you have any suggestions for a distributed replicated filesystem > > that works well with dovecot? I've looked into glusterfs, but the > > latency is way too high for lots of small fi

Re: [Dovecot] dsync redesign

2012-03-27 Thread Stan Hoeppner
On 3/26/2012 2:34 PM, Jeff Gustafson wrote: > Do you have any suggestions for a distributed replicated filesystem > that works well with dovecot? I've looked into glusterfs, but the > latency is way too high for lots of small files. They claim this problem > is fixed in glusterfs 3.3. NFS to

Re: [Dovecot] dsync redesign

2012-03-27 Thread Charles Marcus
On 2012-03-27 11:47 AM, Micah Anderson wrote: One would be the ability to perform *intelligent* incremental / rotated backups. I can do this now by running a dsync backup operation and then doing manual hardlinking or moving of the backup directories (daily.1, daily.2, weekly.1, monthly.1, etc.)

Re: [Dovecot] dsync redesign

2012-03-27 Thread Micah Anderson
Timo Sirainen writes: > In case anyone is interested in reading (and maybe helping!) with a dsync > redesign that's intended to fix all of its current problems, here are some > possibly incoherent ramblings about it: thank you for opening this discussion about dsync! besides the problems I've

Re: [Dovecot] dsync redesign

2012-03-26 Thread Michescu Andrei
Hello Timo, Thank you very much for planning a redesign of the dsycn and for opening this discussion. As I can see from the replies that came until now everybody misses the main point of IMAP: IMAP has been designed to work as a disconnected, high-latency data store. To make this more clear: onc

Re: [Dovecot] dsync redesign

2012-03-26 Thread Jeff Gustafson
On Sat, 2012-03-24 at 08:19 +0100, Attila Nagy wrote: > > I personally think that Dovecot could gain much more if the amount of > work going into fixing or improving dsync would go into making Dovecot > to (be able of) use a high scale, distributed storage backend. > I know it's much harder, be

Re: [Dovecot] dsync is SLOW compared to rsync

2012-03-26 Thread Jeff Gustafson
On Sat, 2012-03-24 at 14:21 +0100, Maarten Bezemer wrote: > On Fri, 23 Mar 2012, Jeff Gustafson wrote: > > > That didn't seem to make much of a difference. On a 3.1GB backup it > > shaved off 5 seconds. dsync's time was over 6 minutes with or without > > the mail_fsync=never. rsync copied the

Re: [Dovecot] dsync is SLOW compared to rsync

2012-03-26 Thread Jeff Gustafson
On Fri, 2012-03-23 at 23:12 -0700, Linda Walsh wrote: > Next -- bench "cp -ax", against rsync -axHAX when it has to copy >75% of > the data (cp ~6-8x speed). But for file speed, 'dd' is king, as it can > use large buffers (~16MB gives best results on my local Gbit network), > but it > misses a

Re: [Dovecot] dsync is SLOW compared to rsync

2012-03-25 Thread Ed W
On 24/03/2012 13:21, Maarten Bezemer wrote: On Fri, 23 Mar 2012, Jeff Gustafson wrote: That didn't seem to make much of a difference. On a 3.1GB backup it shaved off 5 seconds. dsync's time was over 6 minutes with or without the mail_fsync=never. rsync copied the same 3.1GB mailbox in 15 s

Re: [Dovecot] dsync is SLOW compared to rsync

2012-03-24 Thread Maarten Bezemer
On Fri, 23 Mar 2012, Jeff Gustafson wrote: That didn't seem to make much of a difference. On a 3.1GB backup it shaved off 5 seconds. dsync's time was over 6 minutes with or without the mail_fsync=never. rsync copied the same 3.1GB mailbox in 15 seconds. It seems to me that dsync

Re: [Dovecot] dsync redesign

2012-03-24 Thread Timo Sirainen
On 24.3.2012, at 9.19, Attila Nagy wrote: > Well, dsync is a very useful tool, but with continuous replication it tries > to solve a problem which should be handled -at least partially- elsewhere. > Storing stuff in plain file systems and duplicating them to another one just > doesn't scale. d

Re: [Dovecot] dsync redesign

2012-03-24 Thread Jan-Frode Myklebust
On Sat, Mar 24, 2012 at 08:19:48AM +0100, Attila Nagy wrote: > On 03/23/12 22:25, Timo Sirainen wrote: > > > Well, dsync is a very useful tool, but with continuous replication > it tries to solve a problem which should be handled -at least > partially- elsewhere. Storing stuff in plain file systems

Re: [Dovecot] dsync redesign

2012-03-24 Thread Attila Nagy
On 03/23/12 22:25, Timo Sirainen wrote: In case anyone is interested in reading (and maybe helping!) with a dsync redesign that's intended to fix all of its current problems, here are some possibly incoherent ramblings about it: http://dovecot.org/tmp/dsync-redesign.txt and even if you don't

Re: [Dovecot] dsync is SLOW compared to rsync

2012-03-23 Thread Linda Walsh
Jeff Gustafson wrote: On Fri, 2012-03-23 at 19:02 +0100, Christoph Bußenius wrote: Hi, maybe try "dsync -o mail_fsync=never". That didn't seem to make much of a difference. On a 3.1GB backup it shaved off 5 seconds. dsync's time was over 6 minutes with or without the mail_fsyn

[Dovecot] dsync redesign

2012-03-23 Thread Timo Sirainen
In case anyone is interested in reading (and maybe helping!) with a dsync redesign that's intended to fix all of its current problems, here are some possibly incoherent ramblings about it: http://dovecot.org/tmp/dsync-redesign.txt and even if you don't understand that, here's another document d

Re: [Dovecot] dsync is SLOW compared to rsync

2012-03-23 Thread Jeff Gustafson
On Fri, 2012-03-23 at 19:02 +0100, Christoph Bußenius wrote: > Hi, > > maybe try "dsync -o mail_fsync=never". That didn't seem to make much of a difference. On a 3.1GB backup it shaved off 5 seconds. dsync's time was over 6 minutes with or without the mail_fsync=never. rsync copied the sa

Re: [Dovecot] dsync is SLOW compared to rsync

2012-03-23 Thread Christoph Bußenius
Hi, maybe try "dsync -o mail_fsync=never". Cheers, Christoph -- Christoph Bußenius Rechnerbetriebsgruppe der Fakultäten Informatik und Mathematik Technische Universität München +49 89-289-18519 <> Raum 00.05.055 <> Boltzmannstr. 3 <> Garching

[Dovecot] dsync is SLOW compared to rsync

2012-03-22 Thread Jeff Gustafson
Hi all, We are currently using snapshots and rsync to backup a large mail server to a backup mail server. I have been looking into using dsync to replace rsync in hopes that it would make backups more efficient. I decided to test the performance using a single mailbox. Unfortunately dsync s

Re: [Dovecot] Dsync Dovecot

2012-03-20 Thread Michescu Andrei
Hello, As log as example.com resolves to 192.0.43.10 (which I suppose it is a host that you don't own) this will not work. dsync seems to be resolving example.com and trying to connect there via ssh using the current user. It is better to test on domains that you own, on on domains that don't ex

[Dovecot] Dsync Dovecot

2012-03-20 Thread Jitendra Bhaskar
Hey Frnds, Could you tell me about the error : # dsync -Dv -u jitendr...@example.com mirror jitendr...@example.com doveadm(root): Debug: Loading modules from directory: /usr/local/lib/dovecot/doveadm doveadm(root): Debug: Skipping module doveadm_acl_plugin, because dlopen() failed: /usr/local/

Re: [Dovecot] Dsync Dovecot

2012-03-19 Thread Michescu Andrei
Hello Jitendra, I'm trying to do the same thing but I hit a wall as the sync seems to be doubling some emails in my case. My issues is still open with Timo (I HOPE!) As you seem to be stuck in an earlier stage here are my advices: 1) add the id_rsa to the .ssh folder to the user that runs the sy

[Dovecot] Dsync Dovecot

2012-03-19 Thread Jitendra Bhaskar
Hi guys, I am using dovecot-2.1.0 in centos 5.7, I configured dovecot with postfix in 2 system both system is having same configuration and os. I want to use dsync in mirror mode via ssh but I am not able to do it. When I followed the dsync wiki then I an unable to find the username : *dsync -u

Re: [Dovecot] dsync replication available for testing

2012-03-08 Thread Michael Grimm
Hi -- On 08.03.2012 12:35, Timo Sirainen wrote: On Thu, 2012-03-08 at 11:26 +0100, Michael Grimm wrote: You can do for example: service config { unix_listener config { user = vmail } } I will try that later. It seems to me, that whenever a larger number of mails arrive on both s

Re: [Dovecot] dsync replication available for testing

2012-03-08 Thread Timo Sirainen
essages whenever replicator is in action > like (although everything is being synced correctly): > > | mail dovecot: dsync-local(test): Error: remote: > dsync-remote(test): Info: save: box=INBOX, uid=27, > msgid=<3v2jfh5kv4z...@example.tld>, size=547, from=t...@exam

Re: [Dovecot] dsync replication available for testing

2012-03-08 Thread Michael Grimm
oveadm as vmail user (security), though. How should I do that without running into the error messages above? Now some observations regarding replicator: 1) I see a lot of error messages whenever replicator is in action like (although everything is being synced correctly): | mail dov

Re: [Dovecot] dsync replication available for testing

2012-03-05 Thread Timo Sirainen
On 5.3.2012, at 14.35, Jan-Frode Myklebust wrote: > On Mon, Mar 05, 2012 at 12:45:26PM +0200, Timo Sirainen wrote: >>> >>> So, ideally (for us), dovecot should keep a log over which accounts are >>> active (has received or checked mail), and only sync users that has been >>> active for the last $

Re: [Dovecot] dsync replication available for testing

2012-03-05 Thread Timo Sirainen
On 5.3.2012, at 15.11, Attila Nagy wrote: >> dsync doesn't currently take enough advantage of modseqs and send only the >> changed data. > Hm. What is your estimate about the performance capability of the current > "best" replication scheme available in Dovecot? > I know it's hard to tell, becau

Re: [Dovecot] dsync replication available for testing

2012-03-05 Thread Attila Nagy
On 03/05/12 13:48, Timo Sirainen wrote: On 5.3.2012, at 14.15, Attila Nagy wrote: On 03/04/12 11:44, Timo Sirainen wrote: In dovecot-2.1 hg you can now test dsync-based replication. Everything isn't finished yet, but it appears to work and I've enabled it for my @dovecot.fi mails. Some issue

Re: [Dovecot] dsync replication available for testing

2012-03-05 Thread Timo Sirainen
On 5.3.2012, at 14.15, Attila Nagy wrote: >>> On 03/04/12 11:44, Timo Sirainen wrote: In dovecot-2.1 hg you can now test dsync-based replication. Everything isn't finished yet, but it appears to work and I've enabled it for my @dovecot.fi mails. Some issues: >>> Do you plan

Re: [Dovecot] dsync replication available for testing

2012-03-05 Thread Jan-Frode Myklebust
On Mon, Mar 05, 2012 at 12:45:26PM +0200, Timo Sirainen wrote: > > > > So, ideally (for us), dovecot should keep a log over which accounts are > > active (has received or checked mail), and only sync users that has been > > active for the last $timeperiode on startup. > > Well, all of this could

Re: [Dovecot] dsync replication available for testing

2012-03-05 Thread Attila Nagy
On 03/05/12 11:08, Timo Sirainen wrote: On 5.3.2012, at 9.25, Attila Nagy wrote: On 03/04/12 11:44, Timo Sirainen wrote: In dovecot-2.1 hg you can now test dsync-based replication. Everything isn't finished yet, but it appears to work and I've enabled it for my @dovecot.fi mails. Some issues

Re: [Dovecot] dsync replication available for testing

2012-03-05 Thread Timo Sirainen
On 5.3.2012, at 10.35, Jan-Frode Myklebust wrote: >> 1) Replicator syncs all users at startup. If you can change your userdb >> iteration to return only one test user for replicator that avoids it. (You >> may be able to do protocol replicator { userdb {..} } and protocol >> !replicator { .. })

Re: [Dovecot] dsync replication available for testing

2012-03-05 Thread Timo Sirainen
On 5.3.2012, at 9.25, Attila Nagy wrote: > On 03/04/12 11:44, Timo Sirainen wrote: >> In dovecot-2.1 hg you can now test dsync-based replication. Everything isn't >> finished yet, but it appears to work and I've enabled it for my @dovecot.fi >> mails. Some issues: >> > Do you plan to make it mo

Re: [Dovecot] dsync replication available for testing

2012-03-05 Thread Jan-Frode Myklebust
On Sun, Mar 04, 2012 at 01:38:14PM +0200, Timo Sirainen wrote: > > > > Great news. I would love to test it, if I will be able to run this on a > > test > > account, only. All other users should become synced the "old way" for the > > time > > being. > > > > Would that be possible with the cu

Re: [Dovecot] dsync replication available for testing

2012-03-04 Thread Attila Nagy
Hi, On 03/04/12 11:44, Timo Sirainen wrote: In dovecot-2.1 hg you can now test dsync-based replication. Everything isn't finished yet, but it appears to work and I've enabled it for my @dovecot.fi mails. Some issues: - public namespace isn't replicated at all - shared namespace is replicate

Re: [Dovecot] dsync replication available for testing

2012-03-04 Thread Timo Sirainen
On 4.3.2012, at 23.39, Michael Grimm wrote: >> Anyway, replicator simply runs doveadm, so there's not much that can go >> wrong. So >> you could even ignore 1) and just let it sync everyone at startup. > > Does that mean that the new functionality (queue) does only run dsync > replication > the

Re: [Dovecot] dsync replication available for testing

2012-03-04 Thread Michael Grimm
Hi -- On 04.03.2012, at 12:38, Timo Sirainen wrote: > On 4.3.2012, at 13.34, Michael Grimm wrote: >> On 04.03.2012, at 11:44, Timo Sirainen wrote: >>> In dovecot-2.1 hg you can now test dsync-based replication. >> >> Great news. I would love to test it, if I will be able to run this on a test >

Re: [Dovecot] dsync replication available for testing

2012-03-04 Thread Timo Sirainen
On 4.3.2012, at 13.34, Michael Grimm wrote: > On 04.03.2012, at 11:44, Timo Sirainen wrote: > >> In dovecot-2.1 hg you can now test dsync-based replication. > > Great news. I would love to test it, if I will be able to run this on a test > account, only. All other users should become synced the

Re: [Dovecot] dsync replication available for testing

2012-03-04 Thread Michael Grimm
Hi -- On 04.03.2012, at 11:44, Timo Sirainen wrote: > In dovecot-2.1 hg you can now test dsync-based replication. Great news. I would love to test it, if I will be able to run this on a test account, only. All other users should become synced the "old way" for the time being. Would that be p

[Dovecot] dsync replication available for testing

2012-03-04 Thread Timo Sirainen
In dovecot-2.1 hg you can now test dsync-based replication. Everything isn't finished yet, but it appears to work and I've enabled it for my @dovecot.fi mails. Some issues: - public namespace isn't replicated at all - shared namespace is replicated, but not private mail flags - I've only tes

[Dovecot] dsync 2.0.x versus 2.1.x field report

2012-02-24 Thread Michael Grimm
Hi -- I used to run two mailservers with identical MX priorities (poor man's fail-over approach) with Maildir mail folders accessed by Dovecot. Synchronization has been achieved using unison, mirroring all files except dovecot index files. Once in a while some oddities could be observed, like

Re: [Dovecot] dsync, hard-links and refcounts

2012-02-21 Thread Christoph Bußenius
On 21.02.2012 13:15, Timo Sirainen wrote: And they might even become deduplicated if you do doveadm force-resync + purge. I hadn't tried that yet. Thanks for the hint, this is probably all we need. Cheers, Christoph -- Christoph Bußenius Rechnerbetriebsgruppe der Fakultäten Informatik und

Re: [Dovecot] dsync, hard-links and refcounts

2012-02-21 Thread Timo Sirainen
On 21.2.2012, at 13.55, Christoph Bußenius wrote: > Apart from the waste of space, I was wondering: > Is it okay for an mdbox to have several duplicate instances of a message with > the same GUID? Might some kind of corruption arise from this? No corruption. And they might even become deduplica

Re: [Dovecot] dsync, hard-links and refcounts

2012-02-21 Thread Christoph Bußenius
On 21.02.2012 12:04, Timo Sirainen wrote: On Tue, 2012-02-21 at 11:23 +0100, Christoph Bußenius wrote: Hi, On 15.02.2012 04:46, Timo Sirainen wrote: It worked when the GUID already existed somewhere in destination, but not if it was added only during the same session. The attached patch fixe

Re: [Dovecot] dsync, hard-links and refcounts

2012-02-21 Thread Timo Sirainen
On Tue, 2012-02-21 at 11:23 +0100, Christoph Bußenius wrote: > Hi, > > On 15.02.2012 04:46, Timo Sirainen wrote: > > > > It worked when the GUID already existed somewhere in destination, but not > > if it was added only during the same session. The attached patch fixes it. > > I'll commit it to

Re: [Dovecot] dsync, hard-links and refcounts

2012-02-21 Thread Christoph Bußenius
Hi, On 15.02.2012 04:46, Timo Sirainen wrote: It worked when the GUID already existed somewhere in destination, but not if it was added only during the same session. The attached patch fixes it. I'll commit it to v2.1 hg after I'll release v2.1.0.. sorry to bother you again, but I think the

Re: [Dovecot] dsync, hard-links and refcounts

2012-02-20 Thread Christoph Bußenius
On 15.02.2012 04:46, Timo Sirainen wrote: On 13.2.2012, at 16.40, Timo Sirainen wrote: It should deduplicate when GUIDs are the same.. I guess I'll have to look into why it's not working. The attached patch fixes it. I'll commit it to v2.1 hg after I'll release v2.1.0.. After replacing "do

Re: [Dovecot] dsync, hard-links and refcounts

2012-02-15 Thread Christoph Bußenius
On 15.02.2012 04:46, Timo Sirainen wrote: The attached patch fixes it. I'll commit it to v2.1 hg after I'll release v2.1.0.. Thanks. I guess it would be wise to upgrade our new Dovecot mail store to 2.1 before we migrate all our Courier maildir users to it... Cheers, Chnristoph -- Christo

Re: [Dovecot] dsync, hard-links and refcounts

2012-02-14 Thread Timo Sirainen
On 13.2.2012, at 16.40, Timo Sirainen wrote: > On 13.2.2012, at 16.16, Christoph Bußenius wrote: > >> when creating a copy of a mail, dovecot provides a feature that will store >> its contents only once. In maildir, this is done by means of hard links, >> while mdbox has some special refcounti

Re: [Dovecot] dsync, hard-links and refcounts

2012-02-13 Thread Christoph Bußenius
On 02/13/2012 03:40 PM, Timo Sirainen wrote: It should deduplicate when GUIDs are the same.. I guess I'll have to look into why it's not working. I’d very much appreciate that. We will have to migrate many large mailboxes, and it would be a pity to needlessly waste space. I got the same re

Re: [Dovecot] dsync, hard-links and refcounts

2012-02-13 Thread Timo Sirainen
On 13.2.2012, at 16.16, Christoph Bußenius wrote: > when creating a copy of a mail, dovecot provides a feature that will store > its contents only once. In maildir, this is done by means of hard links, > while mdbox has some special refcounting mechanism. > > My question is, how can we convert

[Dovecot] dsync, hard-links and refcounts

2012-02-13 Thread Christoph Bußenius
Hi, when creating a copy of a mail, dovecot provides a feature that will store its contents only once. In maildir, this is done by means of hard links, while mdbox has some special refcounting mechanism. My question is, how can we convert mails from maildir to mdbox without duplicating thes

Re: [Dovecot] dsync deleting too many emails (sdbox)

2012-02-09 Thread Asheesh Laroia
On Thu, 9 Feb 2012, Timo Sirainen wrote: On 9.2.2012, at 21.47, Timo Sirainen wrote: I've anyway done several fixes in v2.1. Can you try if these problems happen with it too? And in any case cleanup the dbox from the *.broken files, so that "doveadm force-resync" won't give any errors. A bi

Re: [Dovecot] dsync deleting too many emails (sdbox)

2012-02-09 Thread Timo Sirainen
On 9.2.2012, at 21.47, Timo Sirainen wrote: > I've anyway done several fixes in v2.1. Can you try if these problems > happen with it too? And in any case cleanup the dbox from the *.broken > files, so that "doveadm force-resync" won't give any errors. A bit more specifically: The last such dbox b

Re: [Dovecot] dsync deleting too many emails (sdbox)

2012-02-09 Thread Timo Sirainen
On Thu, 2012-02-02 at 14:59 -0500, Asheesh Laroia wrote: > I'm guessing this is some bad interaction with sdbox and partial file > downloads? > > I haven't read the code for this, but I would guess the dsync process isn't > being atomic > about file transfers, so it is leaving half-completed tra

Re: [Dovecot] dsync error Mailbox has children, delete them first

2012-02-07 Thread Jürgen Obermann
Am 13.12.2011 11:47, schrieb Jürgen Obermann: Hi, I use dsync to backup mailboxes from mbox format to mdbox on a remote system. The first run for a user with dsync is OK, but during the second there are lots of the following errors: dsync-remote(user): Error: Can't delete mailbox directory Exam

[Dovecot] dsync deleting too many emails (sdbox)

2012-02-02 Thread Asheesh Laroia
I'm using dsync to synchronize emails on a laptop where wifi connectivity sometimes fails in the middle of a sync. I have a shell script that runs dsync, and here is one line of it including the output of dsync: + dsync -f -m realmail mirror /home/paulproteus/projects/ssh-attach/run ssh rose.make

Re: [Dovecot] Dsync and compressed mailboxes

2012-01-13 Thread Dovecot-GDH
xes? I'm not sure if > > dsync -u $USER mirror mdbox:compressed_mdbox_path > > works, or if I have to use something else (I guess that with a running > dovecot dsync backup should work). > > Thanks.

[Dovecot] Dsync and compressed mailboxes

2012-01-13 Thread Joseba Torre
ve to use something else (I guess that with a running dovecot dsync backup should work). Thanks.

[Dovecot] dsync conversion and ldap attributes

2012-01-13 Thread Jan-Frode Myklebust
I have: mail_home = /srv/mailstore/%256RHu/%d/%n mail_location = maildir:~/:INDEX=/indexes/%1u/%1.1u/%u userdb { args = /etc/dovecot/dovecot-ldap.conf.ext driver = ldap } and the dovecot-ldap.conf.ext specifies: user_attrs

Re: [Dovecot] dsync panic

2012-01-10 Thread Jürgen Obermann
Am 20.12.2011 06:45, schrieb Timo Sirainen: On 16.12.2011, at 0.00, Jürgen Obermann wrote: Hello, when I try to convert from mbox to mdbox with dsync with one user it always panics: # /opt/local/bin/dsync -v -u userxy backup ssh root@minerva1 /opt/local/bin/dsync -v -u userxy dsync-remote(

Re: [Dovecot] Dsync fails on second sync for folders with dot in the name

2012-01-03 Thread Timo Sirainen
On 3.1.2012, at 14.54, Jan-Frode Myklebust wrote: > But isn't it a bug that users are allowed to create folders named .a.b, The folder name is "a.b", it just exists in filesystem with Maildir++ as ".a.b". > or that dovecot creates this as a folder named .a.b instead of .a/.b > when the separator

Re: [Dovecot] Dsync fails on second sync for folders with dot in the name

2012-01-03 Thread Jan-Frode Myklebust
On Tue, Jan 03, 2012 at 02:34:59PM +0200, Timo Sirainen wrote: > On Tue, 2012-01-03 at 13:12 +0100, Jan-Frode Myklebust wrote: > > dsync-remote(janfr...@tanso.net): Error: Can't delete mailbox directory > > INBOX.a: Mailbox has children, delete them first > > Oh, this happens only with dsync

Re: [Dovecot] Dsync fails on second sync for folders with dot in the name

2012-01-03 Thread Timo Sirainen
On Tue, 2012-01-03 at 13:12 +0100, Jan-Frode Myklebust wrote: > dsync-remote(janfr...@tanso.net): Error: Can't delete mailbox directory > INBOX.a: Mailbox has children, delete them first Oh, this happens only with dsync backup, and only with Maildir++ -> FS layout change. You can simply ign

Re: [Dovecot] Dsync fails on second sync for folders with dot in the name

2012-01-03 Thread Jan-Frode Myklebust
On Tue, Jan 03, 2012 at 02:00:08PM +0200, Timo Sirainen wrote: > > So here on source you have namespace separator '.' and in destination > you have separator '/'? Maybe that's the problem? Try with both having > '.' separator. I added this namespace on the destination: namespace { inbox = yes

Re: [Dovecot] Dsync fails on second sync for folders with dot in the name

2012-01-03 Thread Timo Sirainen
On Sun, 2012-01-01 at 20:59 +0100, Jan-Frode Myklebust wrote: > I'm in the processes of running our first dsync backup of all users > (from maildir to mdbox on remote server), and one problem I'm hitting > that dsync will work fine on first run for some users, and then > reliably fail whenever I

Re: [Dovecot] dsync / separator / namespace config-problem

2012-01-03 Thread Timo Sirainen
On Thu, 2011-12-29 at 21:03 +0100, Jan-Frode Myklebust wrote: > On Thu, Dec 29, 2011 at 03:49:57PM +0200, Timo Sirainen wrote: > > >> > > >> With mdbox the internal separator is '/', but it's not valid to have > > >> "INBOX." prefix then (it should be "INBOX/"). > > > > > > But how should this b

Re: [Dovecot] Dsync fails on second sync for folders with dot in the name

2012-01-02 Thread Jan-Frode Myklebust
On Mon, Jan 02, 2012 at 09:51:00AM -0500, Charles Marcus wrote: > > dovecot -n output? What are you using for the namespace hierarchy separator? I have the folder format default separator (maildir "."), but still dovecot creates directories named ".a.b". On receiving dsync server:

Re: [Dovecot] Dsync fails on second sync for folders with dot in the name

2012-01-02 Thread Charles Marcus
On 2012-01-01 2:59 PM, Jan-Frode Myklebust wrote: I'm in the processes of running our first dsync backup of all users (from maildir to mdbox on remote server), and one problem I'm hitting that dsync will work fine on first run for some users, and then reliably fail whenever I try a new run:

[Dovecot] Dsync fails on second sync for folders with dot in the name

2012-01-01 Thread Jan-Frode Myklebust
I'm in the processes of running our first dsync backup of all users (from maildir to mdbox on remote server), and one problem I'm hitting that dsync will work fine on first run for some users, and then reliably fail whenever I try a new run: $ sudo dsync -u janfr...@example.net backup ss

Re: [Dovecot] dsync / separator / namespace config-problem

2011-12-29 Thread Jan-Frode Myklebust
On Thu, Dec 29, 2011 at 03:49:57PM +0200, Timo Sirainen wrote: > >> > >> With mdbox the internal separator is '/', but it's not valid to have > >> "INBOX." prefix then (it should be "INBOX/"). > > > > But how should this be handled in the migration phase from maildir to > > mdbox then? Can we ha

Re: [Dovecot] Dsync Error Can't Delete Mailbox

2011-12-29 Thread Asai
OK, thank you very much for this wonderful application. --asai On 12/29/2011 6:11 AM, Timo Sirainen wrote: On 27.12.2011, at 17.46, Asai wrote: I'm getting this error in the dsync logs: Warning: Destination mailbox INBOX has been modified, need to recreate it before we can continue syncing

Re: [Dovecot] dsync / separator / namespace config-problem

2011-12-29 Thread Timo Sirainen
On 29.12.2011, at 15.42, Jan-Frode Myklebust wrote: > On Thu, Dec 29, 2011 at 03:10:20PM +0200, Timo Sirainen wrote: >> >> With mdbox the internal separator is '/', but it's not valid to have >> "INBOX." prefix then (it should be "INBOX/"). > > But how should this be handled in the migration ph

Re: [Dovecot] dsync / separator / namespace config-problem

2011-12-29 Thread Jan-Frode Myklebust
On Thu, Dec 29, 2011 at 03:10:20PM +0200, Timo Sirainen wrote: > > With mdbox the internal separator is '/', but it's not valid to have "INBOX." > prefix then (it should be "INBOX/"). But how should this be handled in the migration phase from maildir to mdbox then? Can we have different namespac

<    1   2   3   4   5   6   7   >