Re: [Dovecot] dsync replication errors
On Sun, 2013-02-17 at 12:30 +0200, Timo Sirainen wrote: > (So yeah, ideally there should be checks for detecting hostname hash > collisions..) Added to v2.2 hg: % doveconf -H dovecot%d No duplicate host hashes in dovecot0 .. dovecot9 % doveconf -H dovecot%2d No duplicate host hashes in dovecot0 .. dovecot99 % doveconf -H dovecot%02d No duplicate host hashes in dovecot00 .. dovecot99 doveconf -H without the template it attempts to detect it from the current hostname.
Re: [Dovecot] dsync replication errors
On 17.2.2013, at 22.04, Michael Grimm wrote:
> On 17.02.2013, at 11:08, Timo Sirainen wrote:
>
>> There may be some other features that require unique hostnames in
>> future. Anything where multiple Dovecot servers need to communicate
>> between each others.
>
> I'd like to come back to that issue in order to understand your statement
> cited below.
>
> First of all: whenever you referred to "hostname" in this thread you have
> been using it as a synonym for the local part [1] of a FQDN, right?
I mean what gethostname() function returns, which is what "hostname" command
usually also returns. And yes, I think it's the local part always.
> I have both servers of mine configured to use identical local parts ("test")
> but different FQDN (aka "test.domainA.tldA" and "test.domainB.tldB"). Your
> fix has been to replace "my_hostname" by "my_hostdomain()", thus using
> "test.domainA.tldA" and "test.domainB.tldB" instead of "test", right?
Yes.
>> If some day there is such generic communication between Dovecot servers
>> I'm planning on enforcing this requirement.
>
> Given that all my interpretations of your statements are correct I do have
> difficulties in understanding why a "generic communication between Dovecot
> servers" should be limited to enforcing different local parts of all Dovecot
> servers implied instead of different FQDN? That would make much more sense
> regarding uniqueness in hostnames, IMHO. Two servers like
> "dovecot.forget-about.it" and "dovecot.you-name.it" should be able to
> communicate generically, again: IMHO.
I think systems named those would belong to different clusters and wouldn't
need to communicate with each others.
I looked through the code. The hostname (without domain) are currently used for:
* maildir filenames
* temporary filenames
* authentication challenge strings in some auth mechanisms
* logging
So I think the hostname uniqueness matters mainly when using a shared
filesystem (e.g. NFS).
> BTW: I had had defined "hostname=" in dovecot.conf identically using
> completely different *but* identical FQDNs "mail.my-domain.tld" because of:
>
> | conf.d/15-lda.conf:
>
> | # Hostname to use in various parts of sent mails, eg. in Message-Id.
> | # Default is the system's real hostname.
> | #hostname =
>
> At least my_hostdomain() doesn't care about that setting, right?
Right. I updated the comment a bit:
http://hg.dovecot.org/dovecot-2.2/rev/6a67a1440e15
lda_hostname would have been a better name for the settings.
Re: [Dovecot] Dovecot-2.1.14 - pop3 processes always hangs forever - another follow-up
On Sun, 2013-02-17 at 12:24 +0100, Axel Luttgens wrote: > Le 17 févr. 2013 à 05:47, Timo Sirainen a écrit : > > > [...] > > The gethostbyname() call is used to figure out the current system's domain. > > Why it would be hanging there, I don't really know. That call anyway isn't > > fully required, you could just replace it with "hent = NULL;" > > Yes, looks like those gethostbyname() require some investigations here... > I'll have a look and provide my "findings" here. > > In the meantime, I don't really like the idea of messing with a general > purpose function, my_hostdomain(), just to avoid a local problem... > Wouldn't you have a better idea? ;-) Solution for v2.2: http://hg.dovecot.org/dovecot-2.2/rev/a6b40687c0a4
Re: [Dovecot] dsync replication errors
On 17.02.2013, at 21:04, Michael Grimm wrote: > BTW: I had had defined "hostname=" in dovecot.conf identically using > completely different *but* identical FQDNs "mail.my-domain.tld" because of: s/using completely different/using completely different to locally reported by resolver/g Regards, Michael
Re: [Dovecot] dsync replication errors
Am 17.02.2013 21:04, schrieb Michael Grimm:
> On 17.02.2013, at 11:08, Timo Sirainen wrote:
>
>> There may be some other features that require unique hostnames in
>> future. Anything where multiple Dovecot servers need to communicate
>> between each others.
>
> I'd like to come back to that issue in order to understand your statement
> cited below.
>
> First of all: whenever you referred to "hostname" in this thread you have
> been using it as a synonym for the local part [1] of a FQDN, right?
>
> I have both servers of mine configured to use identical local parts ("test")
> but different FQDN (aka "test.domainA.tldA" and "test.domainB.tldB"). Your
> fix has been to replace "my_hostname" by "my_hostdomain()", thus using
> "test.domainA.tldA" and "test.domainB.tldB" instead of "test", right?
>
>> If some day there is such generic communication between Dovecot servers
>> I'm planning on enforcing this requirement.
>
> Given that all my interpretations of your statements are correct I do have
> difficulties in understanding why a "generic communication between Dovecot
> servers" should be limited to enforcing different local parts of all Dovecot
> servers implied instead of different FQDN? That would make much more sense
> regarding uniqueness in hostnames, IMHO. Two servers like
> "dovecot.forget-about.it" and "dovecot.you-name.it" should be able to
> communicate generically, again: IMHO.
the better design would be if doveot generates some UUID at the first startup
in a /etc/dovecot/uuid.conf if the file does not exist becasue it would
make hostnames meaningless at all AND give you the option if you are
knowing what you are doing to replace a machine with a newer one by rsync
datadirs and the whole /etc/dovecot/
signature.asc
Description: OpenPGP digital signature
Re: [Dovecot] dsync replication errors
On 17.02.2013, at 11:08, Timo Sirainen wrote:
> There may be some other features that require unique hostnames in
> future. Anything where multiple Dovecot servers need to communicate
> between each others.
I'd like to come back to that issue in order to understand your statement cited
below.
First of all: whenever you referred to "hostname" in this thread you have been
using it as a synonym for the local part [1] of a FQDN, right?
I have both servers of mine configured to use identical local parts ("test")
but different FQDN (aka "test.domainA.tldA" and "test.domainB.tldB"). Your fix
has been to replace "my_hostname" by "my_hostdomain()", thus using
"test.domainA.tldA" and "test.domainB.tldB" instead of "test", right?
> If some day there is such generic communication between Dovecot servers
> I'm planning on enforcing this requirement.
Given that all my interpretations of your statements are correct I do have
difficulties in understanding why a "generic communication between Dovecot
servers" should be limited to enforcing different local parts of all Dovecot
servers implied instead of different FQDN? That would make much more sense
regarding uniqueness in hostnames, IMHO. Two servers like
"dovecot.forget-about.it" and "dovecot.you-name.it" should be able to
communicate generically, again: IMHO.
BTW: I had had defined "hostname=" in dovecot.conf identically using completely
different *but* identical FQDNs "mail.my-domain.tld" because of:
| conf.d/15-lda.conf:
| # Hostname to use in various parts of sent mails, eg. in Message-Id.
| # Default is the system's real hostname.
| #hostname =
At least my_hostdomain() doesn't care about that setting, right?
Again, I can live with mandatory different local hostname parts, but I would
love to understand why ...
With kind regards,
Michael
[1] http://en.wikipedia.org/wiki/Hostname
Re: [Dovecot] Dovecot-2.1.14 - pop3 processes always hangs forever - another follow-up
Le 17 févr. 2013 à 05:47, Timo Sirainen a écrit : > [...] > The gethostbyname() call is used to figure out the current system's domain. > Why it would be hanging there, I don't really know. That call anyway isn't > fully required, you could just replace it with "hent = NULL;" Yes, looks like those gethostbyname() require some investigations here... I'll have a look and provide my "findings" here. In the meantime, I don't really like the idea of messing with a general purpose function, my_hostdomain(), just to avoid a local problem... Wouldn't you have a better idea? ;-) And, BTW, imap processes may hang as well, for the same underlying causes. It is just a matter of selecting INBOX (up to now, I just performed a login/logout sequence). TIA, Axel
Re: [Dovecot] dsync replication errors
Am 17.02.2013 11:08, schrieb Timo Sirainen: >> What parts of Dovecot would be involved? I'm curious because my production >> mailservers use identical hostnames in their jails ever since running >> Dovecot (starting 1.x). > > Mainly that maildir filenames are used as GUIDs. If two have the same > name, they are assumed to be identical. That's why the maildir filenames > include the hostname in them, to make sure that the GUID is different > even if two mails happen to be delivered at exactly the same time with > the same PID and same size to two different servers. So pretty unlikely, > but better to be safe. :) > > There may be some other features that require unique hostnames in > future. Anything where multiple Dovecot servers need to communicate > between each others. If some day there is such generic communication > between Dovecot servers I'm planning on enforcing this requirement. Postfix is enforcing this since forever "Greeted me with my own hostname" hostnames inside a network should always be unique signature.asc Description: OpenPGP digital signature
Re: [Dovecot] Dsync converting dbox->Maildir
Ah, right. dsync preserves the dbox GUIDs by storing them into filenames instead. Anyway, if you use dict-file quota backend there's no slowness (but yes, with maildir quota backend is). On 17.2.2013, at 12.25, Anes Mukhametov wrote: > I used dsync mirror: http://wiki2.dovecot.org/Migration/MailFormat > AFAIK without S field there is big overhead with quota recalculation. > > Timo Sirainen писал 2013-02-16 20:29: >> On Wed, 2013-02-13 at 12:28 +0400, Anes Mukhametov wrote: >>> I've changed my mailbox format from dbox to Maildir. The problem is maildir >>> message filename extension has no S=/W= field after migration. >>> Using dovecot 2.1.15. >> How did you change it? dsync is the only tool that can properly do it, >> and it definitely adds the S/W fields. >> Anyway, Dovecot can handle maildir files without S/W files anyway. But >> if you really want to add them, use >> http://dovecot.org/tools/maildir-size-fix.pl with -a parameter. It adds >> only S though, no W support yet. > >
Re: [Dovecot] dsync replication errors
On 17.2.2013, at 12.19, Michael Grimm wrote: > On 17.02.2013, at 11:08, Timo Sirainen wrote: >> On Sun, 2013-02-17 at 10:44 +0100, Michael Grimm wrote: > Although even if it does, other parts of Dovecot still use only the hostname part to guarantee global uniqueness of things. So better to have unique hostnames. >>> >>> What parts of Dovecot would be involved? I'm curious because my production >>> mailservers use identical hostnames in their jails ever since running >>> Dovecot (starting 1.x). >> >> Mainly that maildir filenames are used as GUIDs. If two have the same >> name, they are assumed to be identical. That's why the maildir filenames >> include the hostname in them, to make sure that the GUID is different >> even if two mails happen to be delivered at exactly the same time with >> the same PID and same size to two different servers. So pretty unlikely, >> but better to be safe. :) > > Ok, that won't hit me for the time being because I am using mdbox. It's basically the same with mdbox, except instead of using actual hostname it's using a 32bit hash of it. (So yeah, ideally there should be checks for detecting hostname hash collisions..)
Re: [Dovecot] Dsync converting dbox->Maildir
I used dsync mirror: http://wiki2.dovecot.org/Migration/MailFormat AFAIK without S field there is big overhead with quota recalculation. Timo Sirainen писал 2013-02-16 20:29: On Wed, 2013-02-13 at 12:28 +0400, Anes Mukhametov wrote: I've changed my mailbox format from dbox to Maildir. The problem is maildir message filename extension has no S=/W= field after migration. Using dovecot 2.1.15. How did you change it? dsync is the only tool that can properly do it, and it definitely adds the S/W fields. Anyway, Dovecot can handle maildir files without S/W files anyway. But if you really want to add them, use http://dovecot.org/tools/maildir-size-fix.pl with -a parameter. It adds only S though, no W support yet.
Re: [Dovecot] dsync replication errors
On 17.02.2013, at 11:08, Timo Sirainen wrote: > On Sun, 2013-02-17 at 10:44 +0100, Michael Grimm wrote: >>> Although even if it does, other parts of Dovecot still use only the >>> hostname part to guarantee global uniqueness of things. So better to have >>> unique hostnames. >> >> What parts of Dovecot would be involved? I'm curious because my production >> mailservers use identical hostnames in their jails ever since running >> Dovecot (starting 1.x). > > Mainly that maildir filenames are used as GUIDs. If two have the same > name, they are assumed to be identical. That's why the maildir filenames > include the hostname in them, to make sure that the GUID is different > even if two mails happen to be delivered at exactly the same time with > the same PID and same size to two different servers. So pretty unlikely, > but better to be safe. :) Ok, that won't hit me for the time being because I am using mdbox. > There may be some other features that require unique hostnames in > future. Anything where multiple Dovecot servers need to communicate > between each others. If some day there is such generic communication > between Dovecot servers I'm planning on enforcing this requirement. Thanks for that clarification. Thus I will need to think about different hostnames, although that implies "no more just copying config files between both servers that imply identical hostnames at both sites" ;-) Regards, Michael
Re: [Dovecot] dsync replication errors
On Sun, 2013-02-17 at 10:44 +0100, Michael Grimm wrote: > > Although even if it does, other parts of Dovecot still use only the > > hostname part to guarantee global uniqueness of things. So better to have > > unique hostnames. > > What parts of Dovecot would be involved? I'm curious because my production > mailservers use identical hostnames in their jails ever since running Dovecot > (starting 1.x). Mainly that maildir filenames are used as GUIDs. If two have the same name, they are assumed to be identical. That's why the maildir filenames include the hostname in them, to make sure that the GUID is different even if two mails happen to be delivered at exactly the same time with the same PID and same size to two different servers. So pretty unlikely, but better to be safe. :) There may be some other features that require unique hostnames in future. Anything where multiple Dovecot servers need to communicate between each others. If some day there is such generic communication between Dovecot servers I'm planning on enforcing this requirement.
Re: [Dovecot] dsync replication errors
On 17.02.2013, at 06:23, Timo Sirainen wrote: > On 17.2.2013, at 7.06, Timo Sirainen wrote: >> On 17.2.2013, at 0.12, Michael Grimm wrote: >>> Hmm. Both jails run at distinct servers. ssh replication uses different >>> domains, though. But, both jails are named identically "test", and both >>> jails resolve to identical hostnames "test" if using "hostname". But, a >>> "hostname -f" is lacking to return "test.mx1.invalid" and >>> "test.mx2.invalid", respectively (although a "nslookup test" does). Hmm, do >>> you think I should need to provide different hostnames in both jails? >> >> That's the problem most likely. I'd guess Dovecot sees both servers as >> having "test" as the hostname and each server thinks it's the one that >> should be doing the locking and not the other. >> >> See if this helps: http://hg.dovecot.org/dovecot-2.2/rev/e7aabd79c9d5 Good news! Those identical hostnames at both servers broke replicator. Now, with v2.2.beta1 (1dd1e88ba0a2) I cannot break replicator any longer how many messages I do inject at both servers simultaneously. (Tested a couple of times up to 2000 mails at every server.) > Although even if it does, other parts of Dovecot still use only the hostname > part to guarantee global uniqueness of things. So better to have unique > hostnames. What parts of Dovecot would be involved? I'm curious because my production mailservers use identical hostnames in their jails ever since running Dovecot (starting 1.x). Thanks for the new replicator code, I really appreciate your work! And, from my point of view I will consider replicator v2.2 ready for production. With kind regards, Michael
Re: [Dovecot] dsync replication errors
On Sat, 2013-02-16 at 19:32 +0100, Oli Schacher wrote: > There seems to be an issue left when expunging a large amount of > messages from the Trash. I managed to get it twice so far by expunging > ~3k messages. I'll try to create a reproducible test script for this > scenario. I can currently only provide my "clicking around" log output. > Version is current hg, e63d1cf19ec7. > > First time it happened: > Feb 16 18:49:48 doco2 dovecot: imap(user1): Warning: Maildir > /mailstore/user1/maildir/.Trash: Expunged message reappeared, giving a new > UID (old uid=1221, file=1361035457.M728795P6220.doco1,S=2476,W=2555:2,Sa) These errors should be gone now in hg. Although there's still some mail duplication problem with maildir that doesn't log any errors about it. I'm not sure why that happens. > Feb 16 18:50:14 doco2 dovecot: doveadm: Error: dsync-remote(user1): Error: > dsync(local): Received unexpected input S != H Fixed also this error that happened on locking failure. > Feb 16 19:13:08 doco1 dovecot: doveadm: Error: dsync-remote(user1): Panic: > file mail-transaction-log-view.c: line 72 (mail_transaction_log_view_set): > assertion failed: (min_file_seq <= max_file_seq) Not sure about this one. But usually this happens only once and retry works.
