Re: [Dovecot] Same filenames on different servers?
Oh, I finally find out the problem. If Dovecot see in the IMAP mailbox two message files named equally (say in cur/ and new/ dirs) it renames one of them into new name dispute this is the same message. I'm not sure if this is intended behaviour. It is quite easy to see such situation when doing periodical rsync (SMTP put message to new/ dir, then rsync copy it to second server, then message on the first server move to cur/, and second rsync results in two files on second server). It there any workaround? Yours, Alexander Chekalin 04.01.2010 18:41, Timo Sirainen wrote: On 4.1.2010, at 15.32, Alexander Chekalin wrote: The problem is that files in Maildirs on First server are named like that: 1262610335.H798006P5447.first.domain.my,S=1962:2, 1262610402.H529761P5596.first.domain.my,S=2975:2, 1262610410.H787724P5657.first.domain.my,S=75691:2, ... and the files on Second server should be named like 1262610335.H798006P5447.second.domain.my,S=1962:2, 1262610402.H529761P5596.second.domain.my,S=2975:2, 1262610410.H787724P5657.second.domain.my,S=75691:2, No, they shouldn't. The maildir filename must be globally unique. Otherwise it doesn't matter what it is. Just keep them as they are. The main problem with rsync is going to be if two servers have the same filename, but one has renamed its flags. So there are two identical base filenames but with different flags. Then Dovecot will log a warning about duplicate message and rename one of the files.
Re: [Dovecot] Same filenames on different servers?
On 2010-01-11, Alexander Chekalin (acheka...@lazurit.com) wrote: I'm not sure if this is intended behaviour. It is quite easy to see such situation when doing periodical rsync (SMTP put message to new/ dir, then rsync copy it to second server, then message on the first server move to cur/, and second rsync results in two files on second server). It there any workaround? I would only use rsync for migrating to a new box, or for backups only... It soundslike you're using the wrong tool for the job - maybe you should be using something like imapsync? Or, please describe what you are trying to accomplish?
Re: [Dovecot] Same filenames on different servers?
On Mon, Jan 11, 2010 at 02:18:07PM +0200, Alexander Chekalin wrote: Oh, I finally find out the problem. If Dovecot see in the IMAP mailbox two message files named equally (say in cur/ and new/ dirs) it renames one of them into new name dispute this is the same message. I'm not sure if this is intended behaviour. It is quite easy to see such situation when doing periodical rsync (SMTP put message to new/ dir, then rsync copy it to second server, then message on the first server move to cur/, and second rsync results in two files on second server). It there any workaround? You could use imapsync instead: http://www.linux-france.org/prj/imapsync/ It synchronizes two IMAP mailboxes (including message flags etc) independent of the underlying implementation. Geert -- Geert Hendrickx -=- g...@telenet.be -=- PGP: 0xC4BB9E9F This e-mail was composed using 100% recycled spam messages!
Re: [Dovecot] Same filenames on different servers?
Yes, I'm trying to make some backup solution. The idea was to have two identical (well, nearly identical) servers with the same software and data on them, and while the first do its job as SMTP and POP3/IMAP4 server, the second server just get copy of first's current state (that is, copy user db and mail spool). In case of something wrong with 1st server I'll just put its IP on second one and will go on. And the problem is that rsynced mailboxes sometimes have duplicates that may confuse users. It happens when box changes between rsyncs by means of Dovecot's ordinary operations (when msg file move from new/ to cur/ on 1st server and after rsync to 2nd this may produce two files with same filename in new/ and cur/, and Dovecot for some reason renames one of it). Doing this using imapsync won't work. It is slow (hours vs minutes with rsync) so I can't run it every 5 minutes, and, moreover, it crashes on some messages, so can not rely on it in production. Yours, Alexander 11.01.2010 15:07, Charles Marcus wrote: On 2010-01-11, Alexander Chekalin (acheka...@lazurit.com) wrote: I'm not sure if this is intended behaviour. It is quite easy to see such situation when doing periodical rsync (SMTP put message to new/ dir, then rsync copy it to second server, then message on the first server move to cur/, and second rsync results in two files on second server). It there any workaround? I would only use rsync for migrating to a new box, or for backups only... It soundslike you're using the wrong tool for the job - maybe you should be using something like imapsync? Or, please describe what you are trying to accomplish?
Re: [Dovecot] Same filenames on different servers?
Hi, This is a general backup issue (files changing during backup run) and nothing Dovecot specific. You should look into something that gives you a consistent view of the data, f.i. LVM snapshots (when you're running linux). Regards, Tom Alexander Chekalin wrote: Yes, I'm trying to make some backup solution. The idea was to have two identical (well, nearly identical) servers with the same software and data on them, and while the first do its job as SMTP and POP3/IMAP4 server, the second server just get copy of first's current state (that is, copy user db and mail spool). In case of something wrong with 1st server I'll just put its IP on second one and will go on. And the problem is that rsynced mailboxes sometimes have duplicates that may confuse users. It happens when box changes between rsyncs by means of Dovecot's ordinary operations (when msg file move from new/ to cur/ on 1st server and after rsync to 2nd this may produce two files with same filename in new/ and cur/, and Dovecot for some reason renames one of it). Doing this using imapsync won't work. It is slow (hours vs minutes with rsync) so I can't run it every 5 minutes, and, moreover, it crashes on some messages, so can not rely on it in production. Yours, Alexander 11.01.2010 15:07, Charles Marcus wrote: On 2010-01-11, Alexander Chekalin (acheka...@lazurit.com) wrote: I'm not sure if this is intended behaviour. It is quite easy to see such situation when doing periodical rsync (SMTP put message to new/ dir, then rsync copy it to second server, then message on the first server move to cur/, and second rsync results in two files on second server). It there any workaround? I would only use rsync for migrating to a new box, or for backups only... It soundslike you're using the wrong tool for the job - maybe you should be using something like imapsync? Or, please describe what you are trying to accomplish?
Re: [Dovecot] Same filenames on different servers?
Hi, The idea was to have two identical (well, nearly identical) servers with the same software and data on them, and while the first do its job as SMTP and POP3/IMAP4 server, the second server just get copy of first's current state (that is, copy user db and mail spool). In case of something wrong with 1st server I'll just put its IP on second one and will go on. Use Heartbeat and DRBD for replication - all mail is synched between the servers. If first crashes, heartbeat runs second as new server for the users. As dovecot on second is only started if first is down, there are no issues about inconsistencies. best regards, Anton
Re: [Dovecot] Same filenames on different servers?
On 11.1.2010, at 16.06, Alexander Chekalin wrote: Doing this using imapsync won't work. It is slow (hours vs minutes with rsync) so I can't run it every 5 minutes, and, moreover, it crashes on some messages, so can not rely on it in production. With v2.0 dsync would be great for this..
Re: [Dovecot] Same filenames on different servers?
I see the only option I have is to use filesystem sync and not rsync. Frankly I'd rather use rsync and not play with GEOM (I'm on FreeBSD) on servers right now. The problem as I've said is not sync itself but rather Dovecot's way of treating files. You can test it yourself: go to your our cur/ dir, and copy any message to new/ dir, and then login to the mailbox with IMAP client. As soon as these messages are identical I'd expect Dovecot to remove one of it but Dovecot simple rename second message file so you'll see two identical messages in your INBOX in mail client. That's what is strange for me. I can fight it on IMAP level (compare messages and find duplicates) but this way won't look nice. Your, Alexander 11.01.2010 16:25, Anton Dollmaier пишет: Hi, The idea was to have two identical (well, nearly identical) servers with the same software and data on them, and while the first do its job as SMTP and POP3/IMAP4 server, the second server just get copy of first's current state (that is, copy user db and mail spool). In case of something wrong with 1st server I'll just put its IP on second one and will go on. Use Heartbeat and DRBD for replication - all mail is synched between the servers. If first crashes, heartbeat runs second as new server for the users. As dovecot on second is only started if first is down, there are no issues about inconsistencies. best regards, Anton
Re: [Dovecot] Same filenames on different servers?
On 11.1.2010, at 16.41, Alexander Chekalin wrote: In fact I just try to understand if Dovecot's behaviour is right and my expectations are wrong, or vise versa. I mean when server see two identical messages (files) both in cur and in new dirs it can compare these files not only by name and rename if these names are the same, but maybe check content of the messages. Feel free to write such code :) There is already a bit of checking to drop hard linked duplicates.
Re: [Dovecot] Same filenames on different servers?
I'd love to use it (it's going to be magic solution for me). But no I haven't looked at v2 yet. Is dsync is ready to use now or this is just an alpha/beta concept code? 11.01.2010 16:34, Timo Sirainen пишет: On 11.1.2010, at 16.06, Alexander Chekalin wrote: Doing this using imapsync won't work. It is slow (hours vs minutes with rsync) so I can't run it every 5 minutes, and, moreover, it crashes on some messages, so can not rely on it in production. With v2.0 dsync would be great for this..
Re: [Dovecot] Same filenames on different servers?
On 1/11/2010 9:34 AM, Timo Sirainen wrote: On 11.1.2010, at 16.06, Alexander Chekalin wrote: Doing this using imapsync won't work. It is slow (hours vs minutes with rsync) so I can't run it every 5 minutes, and, moreover, it crashes on some messages, so can not rely on it in production. With v2.0 dsync would be great for this.. How could dsync beat rsync? With just one rsync command you can replicate the whole mail store (a root folder) Can you do the same with Dsync? or Do you have to write a script that feeds Domains/Users ...? M.A.
Re: [Dovecot] Same filenames on different servers?
No of course it won't. But dsync seems to be a solution that won't deal with storage level so it will resolve conflicts easily. Personally I prefer rsync :) How could dsync beat rsync? With just one rsync command you can replicate the whole mail store (a root folder) Can you do the same with Dsync? or Do you have to write a script that feeds Domains/Users ...?
Re: [Dovecot] Same filenames on different servers?
Timo, I'd really love to but I'm really not sure in my code skills to patch such a sw as Dovecot. 11.01.2010 16:42, Timo Sirainen пишет: On 11.1.2010, at 16.41, Alexander Chekalin wrote: In fact I just try to understand if Dovecot's behaviour is right and my expectations are wrong, or vise versa. I mean when server see two identical messages (files) both in cur and in new dirs it can compare these files not only by name and rename if these names are the same, but maybe check content of the messages. Feel free to write such code :) There is already a bit of checking to drop hard linked duplicates.
Re: [Dovecot] Same filenames on different servers?
On 11.1.2010, at 16.46, Mario Antonio wrote: With v2.0 dsync would be great for this.. How could dsync beat rsync? With just one rsync command you can replicate the whole mail store (a root folder) Can you do the same with Dsync? or Do you have to write a script that feeds Domains/Users ...? If there are no changes, dsync needs to do a lot less disk IO, since it just needs to open index files. Currently you'd have to write a script to iterate through all users, but I don't see it as a huge issue. Scripting it shouldn't be too difficult, and writing code to dsync to directly support it would probably require less than 100 lines.
Re: [Dovecot] Same filenames on different servers?
v2.0 is still in beta. I should release second beta sometimes soon, but I'm a bit too busy / internetless currently. I'm anyway using the latest hg for my own mails, so it's not completely broken, but I wouldn't necessarily recomment it for larger installations.. On 11.1.2010, at 16.45, Alexander Chekalin wrote: I'd love to use it (it's going to be magic solution for me). But no I haven't looked at v2 yet. Is dsync is ready to use now or this is just an alpha/beta concept code? 11.01.2010 16:34, Timo Sirainen пишет: On 11.1.2010, at 16.06, Alexander Chekalin wrote: Doing this using imapsync won't work. It is slow (hours vs minutes with rsync) so I can't run it every 5 minutes, and, moreover, it crashes on some messages, so can not rely on it in production. With v2.0 dsync would be great for this..
Re: [Dovecot] Same filenames on different servers?
So I'd better wait for dsync? Or I can get it (without iterate code) now somehow? 11.01.2010 16:54, Timo Sirainen пишет: On 11.1.2010, at 16.46, Mario Antonio wrote: With v2.0 dsync would be great for this.. How could dsync beat rsync? With just one rsync command you can replicate the whole mail store (a root folder) Can you do the same with Dsync? or Do you have to write a script that feeds Domains/Users ...? If there are no changes, dsync needs to do a lot less disk IO, since it just needs to open index files. Currently you'd have to write a script to iterate through all users, but I don't see it as a huge issue. Scripting it shouldn't be too difficult, and writing code to dsync to directly support it would probably require less than 100 lines.
Re: [Dovecot] Same filenames on different servers?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Mon, 11 Jan 2010, Alexander Chekalin wrote: Yes, I'm trying to make some backup solution. The idea was to have two identical (well, nearly identical) servers with the same software and data on them, and while the first do its job as SMTP and POP3/IMAP4 server, the second server just get copy of first's current state (that is, copy user db and mail spool). In case of something wrong with 1st server I'll just put its IP on second one and will go on. So, where does the duplicate name comes from? This paragraphe sounds as no user accesses the secondary server by default. And the problem is that rsynced mailboxes sometimes have duplicates that may confuse users. It happens when box changes between rsyncs by means of Dovecot's ordinary operations (when msg file move from new/ to cur/ on 1st server and after rsync to 2nd this may produce two files with same filename in new/ and cur/ When Dovecot on the second server is not running - well, I guess that's true as long as nobody logs in - , you can do a rsync --delete -razH primaryServer secondaryServer so the secondary server has a complete copy of the Maildir of the primary server (well, if you have separated INDEXes or CONTROL from there, another rsync is required). If you omit the --delete, you should have plenty of duplicate messages in cur/, because any change of keywords or status will change the filename, too. Also, any deleted message would be still there. So I still wonder, where your duplicates come from. Your user won't complain that a message is back NEW, once the secondary took over, will they? I make rsync backups as well, but restore them as new folders, e.g. .Archive.Sub.Folder becomes .RESTORE.Archive.Sub.Folder, in order to avoid the problem you describe. and Dovecot for some reason renames one of it). Because Maildir is designed that way that no duplicate filename stem (part before colon) must exist. So any server may handle this situation on its own. Regards, - -- Steffen Kaiser -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux) iQEVAwUBS0tAg7+Vh58GPL/cAQIYhgf+OpvlwRZS1JcNXnCeBY7tsNunWk6zIHQD cU1I5LS8it/zLLBt2ORYkm0+u7rRxMYcigEgNsHB1m71f2BeYthlczPzYSm+A17b OmPwb9oZ6ZCJxk8jXzF6GZ39PgT6UoPHqiN0oo6RH2Xso6FYrp0/KibU96K9+uO0 3dXBbl10TdxCfPYvLgO46ounaBVBW3D/xDzKIUeW4DTWApnPuB/gjXs68lYKoZrb F/2pGQAwDGF1uMyOq1dY4Mefe05uzgA4TR/IBzYHMtBhG+YLpKqxg/sGFTTMjTPS DfMtHc0fFbyiIpJOZ7YwwRpq2hNd4tXm7Yq3PmG7eU+guxQboeBkJw== =I4Da -END PGP SIGNATURE-
Re: [Dovecot] Same filenames on different servers?
On 4.1.2010, at 15.32, Alexander Chekalin wrote: The problem is that files in Maildirs on First server are named like that: 1262610335.H798006P5447.first.domain.my,S=1962:2, 1262610402.H529761P5596.first.domain.my,S=2975:2, 1262610410.H787724P5657.first.domain.my,S=75691:2, ... and the files on Second server should be named like 1262610335.H798006P5447.second.domain.my,S=1962:2, 1262610402.H529761P5596.second.domain.my,S=2975:2, 1262610410.H787724P5657.second.domain.my,S=75691:2, No, they shouldn't. The maildir filename must be globally unique. Otherwise it doesn't matter what it is. Just keep them as they are. The main problem with rsync is going to be if two servers have the same filename, but one has renamed its flags. So there are two identical base filenames but with different flags. Then Dovecot will log a warning about duplicate message and rename one of the files.
