Re: Stuck here - help please

[Bernardo Reino]

On Fri, 17 Jul 2020, Gerald Galster wrote:


You could try 
https://serverfault.com/questions/939418/how-do-i-configure-doveadm-a-with-passdb

passdb {
   args = scheme=sha512-crypt /etc/mail/passwd
   driver = passwd-file
}

userdb {
   default_fields = uid=vmail gid=vmail home=/var/vmail/%d/%n
   args = /etc/mail/passwd
   driver = passwd-file
}

I've never tested this as I have my users in a mysql database.


Just wanted to quickly butt in to confirm that the above does work (my 
config is very similar) and allows for the replication to work just fine.


For reference, I have:

userdb {
  driver = passwd-file
  args = username_format=%Lu /etc/dovecot/virtual_passwd
  default_fields = uid=vmail gid=vmail home=/var/mail/%d/%n
}

passdb {
  driver = passwd-file
  args = scheme=SHA512-CRYPT username_format=%Lu /etc/dovecot/virtual_passwd
}

Cheers,
Bernardo

Re: Stuck here - help please

[Gerald Galster]

> Thank you for the details. As per your suggestion, I have made the changes to 
> dovecot.conf file. Still I don't see any replication is happening. Please see 
> the dovecot.conf file.
> 
> I do not see "/etc/dovecot/conf.d/12-replication.conf" in my servers. So I 
> had put everything  in the dovecot.conf file only. Please see the complete 
> data in it below. The below data is in

There should be other config files in /etc/dovecot/conf.d/ - if 
12-replication.conf is not there you can just create it, but putting it in 
dovecot.conf will work too.
(it is easier to locate a specific configuration this way as to search a long 
dovecot.conf)

> server A. In other server (server B) Also I have the same configuration, 
> except mail_replica line and it is pointing to the other server like, " 
> mail_replica = remote:vm...@bal3200dev001.testorg.com 
>  ". 
> 
> I have generated/configured the ssh keys also for vmail user in both servers. 
> Now When i manually ssh to the server, it is not asking for a password. 

That's good.

> userdb {
> args = uid=vmail gid=vmail home=/z1devenv/mail/virtual/%d/%n
> driver = static
> }

The replication wiki says:

Make sure that user listing is configured for your userdb, this is required by 
replication to find the list of users that are periodically replicated:
doveadm user '*'

Did you try that?

I think doveadm user '*' will not work with static userdb because no users are 
actually configured.

You could try 
https://serverfault.com/questions/939418/how-do-i-configure-doveadm-a-with-passdb

passdb {
args = scheme=sha512-crypt /etc/mail/passwd
driver = passwd-file
}

userdb {
default_fields = uid=vmail gid=vmail home=/var/vmail/%d/%n
args = /etc/mail/passwd
driver = passwd-file
}

I've never tested this as I have my users in a mysql database.

If it works you should see some output like the following from doveadm 
replicator:

# doveadm replicator status
Queued 'sync' requests0 

  
Queued 'high' requests0 

  
Queued 'low' requests 0 

  
Queued 'failed' requests  0 

  
Queued 'full resync' requests 0 

  
Waiting 'failed' requests 0 

  
Total number of known users   1234

# doveadm replicator status '*'
username  priority fast sync full sync success sync failed
l...@gcore.biznone 00:00:28  05:52:55  00:00:28 - 

 
Best regards
Gerald

Re: Stuck here - help please

[Kishore Potnuru]

Hi Gerald,

Thank you for the details. As per your suggestion, I have made the changes
to dovecot.conf file. Still I don't see any replication is happening.
Please see the dovecot.conf file.

I do not see "/etc/dovecot/conf.d/12-replication.conf" in my servers. So I
had put everything  in the dovecot.conf file only. Please see the complete
data in it below. The below data is in server A. In other server (server B)
Also I have the same configuration, except mail_replica line and it is
pointing to the other server like, " mail_replica =
remote:vm...@bal3200dev001.testorg.com ".

I have generated/configured the ssh keys also for vmail user in both
servers. Now When i manually ssh to the server, it is not asking for a
password.

=
=
disable_plaintext_auth = no

listen = *
log_path = /var/log/dovecot.log
#mail_location = maildir:/z1nfs/mail/virtual/%d/%n/Maildir/
mail_location = maildir:/z1devenv/mail/virtual/%d/%n/Maildir/
passdb {
args = /etc/dovecot/passwd
driver = passwd-file
}

pop3_uidl_format = %g
protocols = pop3 imap

#ssl = yes
#ssl_cert = 
wrote:

>
> I have done the sync manually with "doveadm sync" command. But, I have not
> configured the replication yet.
>
>
> If you don't tell dovecot where to replicate, nothing gets replicated.
>
> I am looking at the below webpage for the replication.
>
> https://wiki.dovecot.org/Replication
>
> I am using the dovecot version  "2.2.36". I am confused with what needs to
> be done after reading that page.
>
> 1. They are talking about v2.3.1 and v2.2+. Which one do I need to follow?
> Could you please give me more details on this? Providing some sample
> settings will be more helpful for me, please.
>
>
> I don't understand your confusion. You are using 2.2.36, which is v2.2+
> (meaning a version greater than 2.2).
> The documentation states you need at least 2.3.1 if you want to use the
> noreplicate feature.
> So you can't use that with 2.2.36, but as your goal is to replicate
> everything you don't need "noreplicate".
> Besides that I can't see any difference in configuring replication for
> 2.2/2.3.
>
> If you want to replicate emails with ssh you just have to follow the first
> section, the sample settings are
> right on that page. It's basically copying everything from "mail_plugins =
> $mail_plugins notify replication"
> to "replication_max_conns = 10" into a config file like
> /etc/dovecot/conf.d/12-replication.conf
>
> You only have to change the following line to match your server/ssh setup:
> mail_replica = remote:vm...@anotherhost.example.com
>
> Then generate and configure ssh keys for user vmail (passwordless
> authentication) on both servers.
>
> 2. Also, do I need to set the replication on both of my servers the same
> and as it is?
>
>
> On server A) you should configure mail_replica = remote:vmail@server_B and
> on server B) you should configure mail_replica = remote:vmail@server_A
>
> If you skip B) and new mail arrives on B) it is not immediately synced to
> A)
> In that case you would have to wait until a mail gets synced from A)
> (you remember sync is bidirectional)
>
> Best regards
> Gerald
>

Re: Stuck here - help please

[Gerald Galster]

> I have done the sync manually with "doveadm sync" command. But, I have not 
> configured the replication yet.

If you don't tell dovecot where to replicate, nothing gets replicated.

> I am looking at the below webpage for the replication. 
> 
> https://wiki.dovecot.org/Replication  
> 
> I am using the dovecot version  "2.2.36". I am confused with what needs to be 
> done after reading that page.
> 
> 1. They are talking about v2.3.1 and v2.2+. Which one do I need to follow? 
> Could you please give me more details on this? Providing some sample settings 
> will be more helpful for me, please. 

I don't understand your confusion. You are using 2.2.36, which is v2.2+ 
(meaning a version greater than 2.2).
The documentation states you need at least 2.3.1 if you want to use the 
noreplicate feature.
So you can't use that with 2.2.36, but as your goal is to replicate everything 
you don't need "noreplicate".
Besides that I can't see any difference in configuring replication for 2.2/2.3.

If you want to replicate emails with ssh you just have to follow the first 
section, the sample settings are
right on that page. It's basically copying everything from "mail_plugins = 
$mail_plugins notify replication"
to "replication_max_conns = 10" into a config file like 
/etc/dovecot/conf.d/12-replication.conf

You only have to change the following line to match your server/ssh setup:
mail_replica = remote:vm...@anotherhost.example.com

Then generate and configure ssh keys for user vmail (passwordless 
authentication) on both servers.

> 2. Also, do I need to set the replication on both of my servers the same and 
> as it is?

On server A) you should configure mail_replica = remote:vmail@server_B and
on server B) you should configure mail_replica = remote:vmail@server_A

If you skip B) and new mail arrives on B) it is not immediately synced to A)
In that case you would have to wait until a mail gets synced from A)
(you remember sync is bidirectional)

Best regards
Gerald

Re: Stuck here - help please

[Kishore Potnuru]

Thank you for the reply.

I have done the sync manually with "doveadm sync" command. But, I have not
configured the replication yet.

I am looking at the below webpage for the replication.

https://wiki.dovecot.org/Replication

I am using the dovecot version  "2.2.36". I am confused with what needs to
be done after reading that page.

1. They are talking about v2.3.1 and v2.2+. Which one do I need to follow?
Could you please give me more details on this? Providing some sample
settings will be more helpful for me, please.

2. Also, do I need to set the replication on both of my servers the same
and as it is?

Please help me in understanding this?

Thanks,
Kishore Potnuru

On Thu, Jul 16, 2020 at 6:20 PM Gerald Galster 
wrote:

> I have 2 test servers with the below configuration.
>
> ==
> Linux OS-  Red Hat Enterprise Linux Server release 7.7 (Maipo)
> Dovecot version -  2.2.36 (1f10bfa63)
> Postfix version -  2.10.1
> ==
>
> Trying to create High Availability.
>
> I have added both of the above servers behind a F5 load balancer. I have
> got a Load Balancer FQDN "intl-dev-imaptest.testorg.com". I have
> enabled/opened the ports (25/110/143/993/995) on the above  "
> intl-dev-imaptest.testorg.com".
>
> When I send 10 emails to  "intl-dev-imaptest.testorg.com", then those 10
> emails are getting distributed between the above 2 backend servers (5
> emails to each server). I see those 5 emails each in both the servers.
>
>
> You should see 10 emails on each server if replication is working: 5
> emails that were directly delivered via loadbalancer and 5 emails from the
> other server via replication.
>
> From Outlook I have configured the email address using "POP and IMAP",
> when I gave the IMAP server as  "intl-dev-imaptest.testorg.com" ,then it
> shows only 5 emails from server1 in outlook and after a few
> seconds/minutes, automatically it shows/refreshes the other 5 emails from
> server2. But I am not seeing all the 10 emails at the same time. why?
>
>
> The loadbalancer does its job, sometimes the Outlook connection is
> forwarded to server A sometimes to server B. So you just see the mails on
> the respective server. This is very bad. Your Mailclient is probably
> syncing and deleting emails everytime the connection is moved to the other
> server. As I suggested in the other thread you should at least configure
> some kind of ip stickyness when using a loadbalancer, so that your
> mailclient reaches the same backend.
>
> The purpose of replication is that two servers, operating independently,
> have the same dataset. Your servers seem to have completely distinct
> datasets, which indicates replication is not working. Did you configure
> replication?
>
>
> So I tried the sync command. When I execute sync command like below from
> server1, it reflects the same emails in other server2 also. Then I see the
> same number of emails in both the servers. Is it not possible to access the
> both servers emails at one time with the "sync" command? Do we need to run
> this on all the email boxes on both servers? don't we miss/lose any emails
> during this sync process multiple times?
>
> "doveadm sync -f -u kish...@test.testorg.com
> remote:vm...@bal3200dev002.testorg.com"
>
> Is "replication" and "sync" are same?
>
>
> Think of replication as a continous sync. This has to be done every time
> an email is delivered, which dovecot does automatically when replication is
> configured.
>
> You don't lose any emails because the replication/sync is bidirectional,
> it copies from the respective other server what's missing. Of course this
> is not instant but usually happens within seconds.
>
>
> Why are we not able to see all the emails at one time without the "sync"
> command?
>
>
> Probably because you did not configure replication?
>
>
> What is the best and easiest way to create High Availability with just 2
> servers, like emails should travel to both servers equally and if one
> server goes down also, another server should take care of the
> emails/functionality. This is my requirement.
>
>
> It seems you just have to configure replication.
>
> My current real time environment: I have around 10 email domains and each
> domain is having 10 imap emails. In total around 100 email boxes/addresses.
> We receive around 50K emails in a day to those email addresses. We are
> using the "Maildir" format in our environment. Want to move to the High
> Availability option with 2 servers.
>
>
> See my other mail, it may be better to use mdbox instead of maildir.
>
> Best regards
> Gerald
>

Re: Stuck here - help please

[Gerald Galster]

> I have 2 test servers with the below configuration.
>  
> ==
> Linux OS-  Red Hat Enterprise Linux Server release 7.7 (Maipo)
> Dovecot version -  2.2.36 (1f10bfa63)
> Postfix version -  2.10.1 
> == 
> 
> Trying to create High Availability. 
> 
> I have added both of the above servers behind a F5 load balancer. I have got 
> a Load Balancer FQDN "intl-dev-imaptest.testorg.com 
> ". I have enabled/opened the ports 
> (25/110/143/993/995) on the above  "intl-dev-imaptest.testorg.com 
> ".
> 
> When I send 10 emails to  "intl-dev-imaptest.testorg.com 
> ", then those 10 emails are getting 
> distributed between the above 2 backend servers (5 emails to each server). I 
> see those 5 emails each in both the servers.

You should see 10 emails on each server if replication is working: 5 emails 
that were directly delivered via loadbalancer and 5 emails from the other 
server via replication.

> From Outlook I have configured the email address using "POP and IMAP", when I 
> gave the IMAP server as  "intl-dev-imaptest.testorg.com 
> " ,then it shows only 5 emails from 
> server1 in outlook and after a few seconds/minutes, automatically it 
> shows/refreshes the other 5 emails from server2. But I am not seeing all the 
> 10 emails at the same time. why?

The loadbalancer does its job, sometimes the Outlook connection is forwarded to 
server A sometimes to server B. So you just see the mails on the respective 
server. This is very bad. Your Mailclient is probably syncing and deleting 
emails everytime the connection is moved to the other server. As I suggested in 
the other thread you should at least configure some kind of ip stickyness when 
using a loadbalancer, so that your mailclient reaches the same backend.

The purpose of replication is that two servers, operating independently, have 
the same dataset. Your servers seem to have completely distinct datasets, which 
indicates replication is not working. Did you configure replication?
 
> So I tried the sync command. When I execute sync command like below from 
> server1, it reflects the same emails in other server2 also. Then I see the 
> same number of emails in both the servers. Is it not possible to access the 
> both servers emails at one time with the "sync" command? Do we need to run 
> this on all the email boxes on both servers? don't we miss/lose any emails 
> during this sync process multiple times?
> 
> "doveadm sync -f -u kish...@test.testorg.com 
>  remote:vm...@bal3200dev002.testorg.com 
> "
> 
> Is "replication" and "sync" are same?

Think of replication as a continous sync. This has to be done every time an 
email is delivered, which dovecot does automatically when replication is 
configured.

You don't lose any emails because the replication/sync is bidirectional, it 
copies from the respective other server what's missing. Of course this is not 
instant but usually happens within seconds.


> Why are we not able to see all the emails at one time without the "sync" 
> command?

Probably because you did not configure replication?


> What is the best and easiest way to create High Availability with just 2 
> servers, like emails should travel to both servers equally and if one server 
> goes down also, another server should take care of the emails/functionality. 
> This is my requirement. 

It seems you just have to configure replication.

> My current real time environment: I have around 10 email domains and each 
> domain is having 10 imap emails. In total around 100 email boxes/addresses. 
> We receive around 50K emails in a day to those email addresses. We are using 
> the "Maildir" format in our environment. Want to move to the High 
> Availability option with 2 servers. 

See my other mail, it may be better to use mdbox instead of maildir.

Best regards
Gerald

Stuck here - help please

[Kishore Potnuru]

Hi All,

I have 2 test servers with the below configuration.

==
Linux OS-  Red Hat Enterprise Linux Server release 7.7 (Maipo)
Dovecot version -  2.2.36 (1f10bfa63)
Postfix version -  2.10.1
==

Trying to create High Availability.

I have added both of the above servers behind a F5 load balancer. I have
got a Load Balancer FQDN "intl-dev-imaptest.testorg.com". I have
enabled/opened the ports (25/110/143/993/995) on the above  "
intl-dev-imaptest.testorg.com".

When I send 10 emails to  "intl-dev-imaptest.testorg.com", then those 10
emails are getting distributed between the above 2 backend servers (5
emails to each server). I see those 5 emails each in both the servers.

>From Outlook I have configured the email address using "POP and IMAP", when
I gave the IMAP server as  "intl-dev-imaptest.testorg.com" ,then it shows
only 5 emails from server1 in outlook and after a few seconds/minutes,
automatically it shows/refreshes the other 5 emails from server2. But I am
not seeing all the 10 emails at the same time. why?

So I tried the sync command. When I execute sync command like below from
server1, it reflects the same emails in other server2 also. Then I see the
same number of emails in both the servers. Is it not possible to access the
both servers emails at one time with the "sync" command? Do we need to run
this on all the email boxes on both servers? don't we miss/lose any emails
during this sync process multiple times?

"doveadm sync -f -u kish...@test.testorg.com
remote:vm...@bal3200dev002.testorg.com"

Is "replication" and "sync" are same?

Why are we not able to see all the emails at one time without the "sync"
command?

What is the best and easiest way to create High Availability with just 2
servers, like emails should travel to both servers equally and if one
server goes down also, another server should take care of the
emails/functionality. This is my requirement.

My current real time environment: I have around 10 email domains and each
domain is having 10 imap emails. In total around 100 email boxes/addresses.
We receive around 50K emails in a day to those email addresses. We are
using the "Maildir" format in our environment. Want to move to the High
Availability option with 2 servers.

Please help me to fix the issue.

Thanks & Regards,
Kishore Potnuru