Re: Issue with LMTP proxying and port number

[Laz C . Peterson]

Hello Sami, yes, see below.

We run Dovecot at a different versions, mainly 2.2.10 (CentOS), 2.2.22 (Ubuntu) 
and now 2.2.36 (CentOS).  The issue is weird, because it only happened after 
the update from 2.2.10->36.  Just to understand it would be great.

I'm actually checking out the configs now ... Our SQL userdb does not specify 
port.  So I'm guessing this may be to blame?

(This was by design, though -- we don't want to specify one port for different 
client protocols.  Though, I do recall seeing some hack online using CASE in 
SQL query ...)

These servers run LMTP as a unix socket as well as a TCP port 24 serving all IP 
sources.  The internal servers are running LMTP on TCP port 24 (as well as unix 
socket, but that's irrelevant), but no LMTP comm happens between directors and 
backend mail servers after the 2.2.10->36 update on the directors with our 
config.  I do apologize that I can't get more specific than those versions ...

The backend mail servers function the same in our environment on both versions 
2.2.10 and 2.2.36.

We are good now, as we changed the config to go to the TCP port instead of unix 
socket.  But we had a good jolt of fun this morning. :-)

Would love to understand what we have done wrong, or how we misunderstood the 
configuration directives -- in either version.

Thank you Sami.

Here is the SQL:

driver = mysql
connect = host=10.1.1.110 dbname=someoneelsesmail user=notouruser 
password=chicken
default_pass_scheme = PLAIN
password_query = SELECT CONCAT(a.username,'@',d.domain) AS 
user,a.password,'captainmaster' AS master,'kentuckyfriedchicken' AS pass,'Y' AS 
proxy FROM accounts a INNER JOIN domains d ON a.domain_id=d.id WHERE 
a.username='%n' AND d.domain='%d' AND a.active=1 AND d.active=1

Here is the doveconf:

# 2.2.36 (1f10bfa63): /etc/dovecot/dovecot.conf
# Pigeonhole version 0.4.24 (124e06aa)
doveconf: Warning: service auth { client_limit=5200 } is lower than required 
under max. load (6144)
# OS: Linux 3.10.0-693.17.1.el7.centos.plus.x86_64 x86_64 CentOS Linux release 
7.6.1810 (Core)  
# Hostname: imapdir1.randomdomain.net
auth_master_user_separator = *
auth_mechanisms = plain login cram-md5
default_client_limit = 5200
default_process_limit = 1024
director_mail_servers = 10.9.5.131 10.9.5.132
director_servers = 10.9.5.151 10.9.5.152
disable_plaintext_auth = no
doveadm_port = 24245
first_valid_uid = 1000
lmtp_proxy = yes
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope encoded-character 
vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy 
include variables body enotify environment mailbox date index ihave duplicate 
mime foreverypart extracttext
mbox_write_locks = fcntl
namespace inbox {
  inbox = yes
  location = 
  mailbox Drafts {
    special_use = \Drafts
  }
  mailbox Junk {
    special_use = \Junk
  }
  mailbox Sent {
    special_use = \Sent
  }
  mailbox "Sent Messages" {
    special_use = \Sent
  }
  mailbox Trash {
    special_use = \Trash
  }
  prefix = 
}
passdb {
  args = /etc/dovecot/dovecot-sql.conf.ext
  driver = sql
}
plugin {
  quota = maildir:User quota
  quota_grace = 10%%
  quota_status_nouser = DUNNO
  quota_status_overquota = 522 5.2.2 Mailbox is full
  quota_status_success = DUNNO
  quota_warning = storage=95%% quota-warning 95 %u
  quota_warning2 = storage=80%% quota-warning 80 %u
  sieve = file:~/sieve;active=~/.dovecot.sieve
}
protocols = imap pop3 lmtp sieve
service auth {
  unix_listener auth-client {
    mode = 0660
    user = exim
  }
}
service director {
  fifo_listener login/proxy-notify {
    mode = 0600
    user = $default_login_user
  }
  inet_listener {
    port = 9090
  }
  unix_listener director-userdb {
    mode = 0600
  }
  unix_listener login/director {
    mode = 0666
  }
}
service doveadm {
  inet_listener {
    port = 24245
  }
}
service imap-login {
  executable = imap-login director
}
service ipc {
  unix_listener ipc {
    user = dovecot
  }
}
service lmtp {
  inet_listener lmtp {
    port = 24
  }
  process_min_avail = 5
}
service managesieve-login {
  executable = managesieve-login director
}
service pop3-login {
  executable = pop3-login director
}
service quota-warning {
  executable = script /etc/dovecot/quota-warning.sh
  user = dovecot
}
ssl_cert =  
wrote:
  On 31 Dec 2018, at 0.43, Laz C. Peterson  wrote: Hello 
there, Everything was working fine on Dovecot 2.2.10 (on CentOS 7), but after 
updating to version 2.2.36, our director servers are not able to proxy LMTP. We 
are sending mail from exim to the local Dovecot LMTP socket, which then used to 
send it to our internal mail servers on port 24. Besides the update, nothing 
has changed in the config files.  The configuration has been working great for 
months … Then we had a great idea of doing holiday maintenance. (Currently, we 
are working around this issue by having exim send 

Issue with LMTP proxying and port number

[Laz C . Peterson]

Hello there,

Everything was working fine on Dovecot 2.2.10 (on CentOS 7), but after updating 
to version 2.2.36, our director servers are not able to proxy LMTP.

We are sending mail from exim to the local Dovecot LMTP socket, which then used 
to send it to our internal mail servers on port 24.

Besides the update, nothing has changed in the config files.  The configuration 
has been working great for months … Then we had a great idea of doing holiday 
maintenance.

(Currently, we are working around this issue by having exim send mail to the IP 
and port, instead of the socket.)

Doing a tcpdump capture shows no TCP traffic either.  The logs seem to say that 
it is trying to go to port 0, so my guess is that it’s not getting the port 
directive from either static or sql — since the docs state that if these 
settings aren't specified, it will always use the same connection type for the 
proxy that it received the connection on.

Any thoughts?  I can share config if necessary.

~ Laz Peterson
Paravis, LLC

Re: Symbolic link for sieve script not created by dsync

[Laz C. Peterson]

Or actually, is the issue in the “sending” side of dsync?  Seems that if we do 
the sieve script on the side running Ubuntu 16.04.1, the Ubuntu 14.04.5 server 
does in fact get the symbolic link.  But not vice versa.  Can someone verify 
this to be the case?

Such a drag that the Ubuntu packages for 14.04 are so far behind …

~ Laz Peterson
Paravis, LLC

> On Aug 15, 2016, at 12:12 PM, Laz C. Peterson  wrote:
> 
> Hello Dovecot ~
> 
> I know this bug has been discussed previously for other versions of dsync, 
> pigeonhole and dovecot … But we still can’t get dsync to create the symbolic 
> link for the sieve script after it is created on a replicated server.
> 
> We are running Ubuntu 16.04.1 with the official Ubuntu dovecot packages 
> (2.2.22).  Sieve script itself is replicated, but not the symbolic link.
> 
> Any suggestions?
> 
> ~ Laz Peterson
> Paravis, LLC

Symbolic link for sieve script not created by dsync

[Laz C. Peterson]

Hello Dovecot ~

I know this bug has been discussed previously for other versions of dsync, 
pigeonhole and dovecot … But we still can’t get dsync to create the symbolic 
link for the sieve script after it is created on a replicated server.

We are running Ubuntu 16.04.1 with the official Ubuntu dovecot packages 
(2.2.22).  Sieve script itself is replicated, but not the symbolic link.

Any suggestions?

~ Laz Peterson
Paravis, LLC

Re: Slow auth

[Laz C. Peterson]

We had this issue as well - switch your primary authentication to LDAP and make 
sure it is attempting those auth sources first before any PAM sources.  You 
also don’t need to have your Dovecot server joined to the domain by doing it 
this way, which is nice.

We were previously using PAM auth through Kerberos as a method of 
authenticating from our LDAP servers.  I can’t remember the reason why we 
decided to go with Dovecot->LDAP (no mediating auth service in between), but 
the performance was significantly faster.

Or, you can also try PAM using Kerberos, instead of WInbind (or whatever you 
are using with PAM).  Just a thought.

~ Laz Peterson
Paravis, LLC

> On Jun 7, 2016, at 11:16 AM, aki.tu...@dovecot.fi wrote:
> 
> 
>> On June 7, 2016 at 9:06 PM Ranbir  wrote:
>> 
>> 
>> On Tue, 2016-06-07 at 11:45 -0500, Edgar Pettijohn wrote:
>> 
>>> You have Pam as your passdb driver.
>> 
>> Yes, because I have to. How else would I get Dovecot to authenticate
>> users against my FreeIPA server? 
>> 
>> 
>> -- 
>> Ranbir
> 
> LDAP does come into mind... IPA after all IS ldap based. It's what sssd uses 
> as well.
> 
> ---
> Aki Tuomi

Re: dovecot proxy/director and high availability design

[Laz C. Peterson]

The consensus seems to say no to RR DNS … I am going to take that into serious 
consideration.

With this proxy setup you describe, what would happen if HAProxy or Dovecot 
Proxy were to fail?

I think there is no problem with many moving parts, as long as there is a 
backup plan in case something goes awry.  My goal is slightly different, as I 
want to have HA available across datacenters without using BGP or having 
control over the IP space (so, no anycast).  Just a simple way to get the 
clients redirected to the other Dovecot server when I lose an entire datacenter 
network for whatever reason.

~ Laz Peterson
Paravis, LLC

> On Jul 20, 2015, at 5:32 PM, Chad M Stewart  wrote:
> 
> 
> Round-robin DNS last I checked can be fraught with issues.  
> 
> While doing something else I came up with this idea:  Clients --> Load 
> Balancer(HAProxy) --> Dovecot Proxy(DP) --> Dovecot Director(DD) --> MS1 / 
> MS2.
> 
> 
> When DP checks say user100 it'll find a host=DD-POD1 that returns two IPs, 
> those of the two DD that sit in front of POD1. This DD pair is the only pair 
> in the ring and only responsible for POD1.  Another pair will handle POD2.  
> When DD looks up the host value for a user it'll find the same name, but the 
> IPs returned will be different.  Instead have both IPs of the mail stores 
> returned.  
> 
> I believe this will achieve what I'm after.  HAProxy will do the load 
> balancing of the DP instances.  DP will balance the DDs, and DDs will do its 
> job well and ensure that say user300 has all of their connections sent to 
> MS1.  When I need to do maintenance on MS1 I can use the DD pair for POD1 to 
> gently move the connections to MS2, etc..   I could also make each POD a 2+1 
> cluster, so a silent but up-to-date and replicated store sits there waiting 
> should it be needed, or even a 2+2 cluster.  After all "two is one, and one 
> is none".
> 
> Not sure when I'll get time to implement/test this out, but in theory it 
> sounds reasonable. I admit its a fair amount of moving parts and areas for 
> failure but I think it maybe the balance needed to achieve the service level 
> availability I'm after while still allowing for maintenance on the systems 
> w/o clients noticing.
> 
> -Chad
> 
> 
> On Jul 20, 2015, at 1:04 PM, Laz C. Peterson  wrote:
> 
>> I’m trying to do this too.  But the goal would be simply for automatic 
>> failover to the other datacenter.  Everything is working if the server’s 
>> unique hostname is entered, but I want to do something like round robin DNS 
>> that mail clients will automatically attempt to connect to the other IP if 
>> they cannot get to the first address.  Unfortunately mail applications don’t 
>> really do this like web browsers do …
>> 
>> ~ Laz Peterson
>> Paravis, LLC
>> 
>>> On Jul 20, 2015, at 10:29 AM, Chad M Stewart  wrote:
>>> 
>>> 
>>> I'm trying to determine which dovecot components to use and how to order 
>>> them in the network path from client to mail store.
>>> 
>>> 
>>> If I have say 1,000 users, all stored in MySQL (or LDAP) and have 4 mail 
>>> stores, configured into 2, 2 node pods.
>>> 
>>> 
>>> MS1 and MS2 are pod1 and are configured with replication (dsync) and host 
>>> users 0-500.  MS3 and MS4 are pod2 and are configured with replication 
>>> between them and host users 501-1000.   Ideally the active connections in 
>>> pod1 would be split 50/50 between MS1 and MS2.  When maintenance is 
>>> performed obviously all active connections/users would be moved to the 
>>> other node in the pod and then rebalanced once maintenance is completed.  
>>> 
>>> I'm not sure if I need to use both the proxy and director, or just one or 
>>> the other? If both then what is the proper path, from a network 
>>> perspective?  I like the functionality director provides, being able to 
>>> add/remove servers on the fly and adjust connections, etc.. But from what 
>>> I've read director needs to know about all mail servers.  The problem is 
>>> that not all servers host all users.  User100 could be serviced by ms1 or 
>>> ms2, but not by ms3 or ms4.  
>>> 
>>> I'm trying to design a system that should provide as close to 99.999% 
>>> service availability as possible.
>>> 
>>> 
>>> 
>>> Thank you,
>>> Chad

Re: dovecot proxy/director and high availability design

[Laz C. Peterson]

I’m trying to do this too.  But the goal would be simply for automatic failover 
to the other datacenter.  Everything is working if the server’s unique hostname 
is entered, but I want to do something like round robin DNS that mail clients 
will automatically attempt to connect to the other IP if they cannot get to the 
first address.  Unfortunately mail applications don’t really do this like web 
browsers do …

~ Laz Peterson
Paravis, LLC

> On Jul 20, 2015, at 10:29 AM, Chad M Stewart  wrote:
> 
> 
> I'm trying to determine which dovecot components to use and how to order them 
> in the network path from client to mail store.
> 
> 
> If I have say 1,000 users, all stored in MySQL (or LDAP) and have 4 mail 
> stores, configured into 2, 2 node pods.
> 
> 
> MS1 and MS2 are pod1 and are configured with replication (dsync) and host 
> users 0-500.  MS3 and MS4 are pod2 and are configured with replication 
> between them and host users 501-1000.   Ideally the active connections in 
> pod1 would be split 50/50 between MS1 and MS2.  When maintenance is performed 
> obviously all active connections/users would be moved to the other node in 
> the pod and then rebalanced once maintenance is completed.  
> 
> I'm not sure if I need to use both the proxy and director, or just one or the 
> other? If both then what is the proper path, from a network perspective?  I 
> like the functionality director provides, being able to add/remove servers on 
> the fly and adjust connections, etc.. But from what I've read director needs 
> to know about all mail servers.  The problem is that not all servers host all 
> users.  User100 could be serviced by ms1 or ms2, but not by ms3 or ms4.  
> 
> I'm trying to design a system that should provide as close to 99.999% service 
> availability as possible.
> 
> 
> 
> Thank you,
> Chad

Re: Dovecot auth username mapping

[Laz C. Peterson]

Ahh Peter, good call on this one!



So after playing around with the order of authentication in Dovecot, you are 
correct, the PAM timeout was causing the holdup.  I guess since PAM has no way 
of looking up whether or not a user exists prior to authenticating, this is 
causing the hiccup, versus LDAP which can search for a user’s existence prior 
to the auth.  Switching these around, I notice almost *no* degradation in 
performance for PAM authentications, and the LDAP authentications run smooth as 
I would hope them to.

Awesome, so now we have our solution!  (I think.)

Gotta say, a lot of love goes out to the Dovecot community (especially Timo!) 
for all the inspiration and help that I’ve received.  Dovecot is a great app 
and this community is the backbone of it all.  Cheers to all!

Thanks again.

~ Laz Peterson
Paravis, LLC
Ph: 951.319.3240 x201

> On Jul 2, 2015, at 6:25 AM, Laz C. Peterson  wrote:
> 
> Peter,
> 
> Yes that is a possibility.  I will try disabling PAM (or switching the auth 
> order) and see if that makes a difference.  Thanks for the suggestion!
> 
> ~ Laz Peterson
> Paravis, LLC
> Ph: 951.319.3240 x201
> 
>> On Jul 1, 2015, at 11:34 PM, Peter Chiochetti  wrote:
>> 
>> Am 2015-07-02 um 01:41 schrieb Laz C. Peterson:
>>> 
>>> I did attempt to switch the PAM/Kerberos authentication to Dovecot
>>> LDAP authentication, but now performance is unbelievably slow.
>>> Any thoughts to this?
>> 
>> In case you have multiple passdb backends, it could be, that LDAP only gets 
>> its chance, after PAM did time out.
>> 
>> 
>> -- 
>> peter

Re: Dovecot auth username mapping

[Laz C. Peterson]

Peter,

Yes that is a possibility.  I will try disabling PAM (or switching the auth 
order) and see if that makes a difference.  Thanks for the suggestion!

~ Laz Peterson
Paravis, LLC
Ph: 951.319.3240 x201

> On Jul 1, 2015, at 11:34 PM, Peter Chiochetti  wrote:
> 
> Am 2015-07-02 um 01:41 schrieb Laz C. Peterson:
>> 
>> I did attempt to switch the PAM/Kerberos authentication to Dovecot
>> LDAP authentication, but now performance is unbelievably slow.
>> Any thoughts to this?
> 
> In case you have multiple passdb backends, it could be, that LDAP only gets 
> its chance, after PAM did time out.
> 
> 
> -- 
> peter

Re: Dovecot auth username mapping

[Laz C. Peterson]

It’s actually unbelievable how much slower LDAP auth is than PAM.  Does anyone 
have any suggestions how I can improve Dovecot LDAP auth?  I have tried caching 
authentications and that doesn’t help either.

~ Laz Peterson
Paravis, LLC
Ph: 951.319.3240 x201

> On Jul 1, 2015, at 4:41 PM, Laz C. Peterson  wrote:
> 
> Thank you for the response Axel.  I will look into that.
> 
> I did attempt to switch the PAM/Kerberos authentication to Dovecot LDAP 
> authentication, but now performance is unbelievably slow.  For example, with 
> PAM/Kerberos, a user can log into webmail and have all of their 
> emails/folders showing almost immediately.  When using Dovecot LDAP, it takes 
> literally 8-10 seconds to see the same thing.
> 
> I was hoping that was a possible replacement for this, but my goodness it was 
> so incredibly slow!  This would definitely be an option though, as it does 
> serve the purpose.  I just can’t figure out how to fix the performance issue. 
>  Any thoughts to this?
> 
> ~ Laz Peterson
> Paravis, LLC
> Ph: 951.319.3240 x201
> 
>> On Jul 1, 2015, at 3:24 PM, Axel Luttgens  wrote:
>> 
>> 
>>> Le 1 juil. 2015 à 04:38, Laz C. Peterson
>> 
>>> a écrit :
>>> 
>>> I have an interesting case here …
>>> 
>>> Virtual mailboxes, domain/username/aliases stored in MySQL, authentication 
>>> done using PAM.  PAM authenticates through Kerberos, which are internal 
>>> realms and not the email domains — for example, my username would be 
>>> laz@PARAVIS.LOCAL <mailto:laz@PARAVIS.LOCAL> and my email address would be 
>>> l...@paravis.net <mailto:l...@paravis.net>.
>>> 
>>> All of this works just fine.  But what I want to do is allow the users to 
>>> log in using their email address and not their full Kerberos name.  It is 
>>> becoming laborious to help the users understand the difference between 
>>> their username@LOCAL.REALM and username@email.address 
>>> <mailto:username@email.address> and why we have to have two separate 
>>> identities that mean the same thing.
>>> 
>>> I have the SQL statements to convert either the Kerberos login or the email 
>>> address to the actual Kerberos login (so they may use either).  But I 
>>> cannot seem to figure out how to get Dovecot to acknowledge this as the 
>>> mapped username.
>>> 
>>> I’m sure there has to be a way.  Any help will be greatly appreciated.  
>>> Thank you!
>> 
>> Hello Laz,
>> 
>> I fear you’ll have to resort to CheckPassword 
>> (http://wiki2.dovecot.org/AuthDatabase/CheckPassword) or something similar.
>> 
>> Indeed, your MySql database may contain everything needed to convert email 
>> addresses to kerb login (and vice-versa), but Dovecot’s PAM interface 
>> understandably just knows about a (login, password) pair, where the login is 
>> the one provided by the user wanting to log in.
>> 
>> That said, I hope to be wrong,
>> Axel

Re: Dovecot auth username mapping

[Laz C. Peterson]

Thank you for the response Axel.  I will look into that.

I did attempt to switch the PAM/Kerberos authentication to Dovecot LDAP 
authentication, but now performance is unbelievably slow.  For example, with 
PAM/Kerberos, a user can log into webmail and have all of their emails/folders 
showing almost immediately.  When using Dovecot LDAP, it takes literally 8-10 
seconds to see the same thing.

I was hoping that was a possible replacement for this, but my goodness it was 
so incredibly slow!  This would definitely be an option though, as it does 
serve the purpose.  I just can’t figure out how to fix the performance issue.  
Any thoughts to this?

~ Laz Peterson
Paravis, LLC
Ph: 951.319.3240 x201

> On Jul 1, 2015, at 3:24 PM, Axel Luttgens  wrote:
> 
> 
>> Le 1 juil. 2015 à 04:38, Laz C. Peterson
> 
>> a écrit :
>> 
>> I have an interesting case here …
>> 
>> Virtual mailboxes, domain/username/aliases stored in MySQL, authentication 
>> done using PAM.  PAM authenticates through Kerberos, which are internal 
>> realms and not the email domains — for example, my username would be 
>> laz@PARAVIS.LOCAL <mailto:laz@PARAVIS.LOCAL> and my email address would be 
>> l...@paravis.net <mailto:l...@paravis.net>.
>> 
>> All of this works just fine.  But what I want to do is allow the users to 
>> log in using their email address and not their full Kerberos name.  It is 
>> becoming laborious to help the users understand the difference between their 
>> username@LOCAL.REALM and username@email.address 
>> <mailto:username@email.address> and why we have to have two separate 
>> identities that mean the same thing.
>> 
>> I have the SQL statements to convert either the Kerberos login or the email 
>> address to the actual Kerberos login (so they may use either).  But I cannot 
>> seem to figure out how to get Dovecot to acknowledge this as the mapped 
>> username.
>> 
>> I’m sure there has to be a way.  Any help will be greatly appreciated.  
>> Thank you!
> 
> Hello Laz,
> 
> I fear you’ll have to resort to CheckPassword 
> (http://wiki2.dovecot.org/AuthDatabase/CheckPassword) or something similar.
> 
> Indeed, your MySql database may contain everything needed to convert email 
> addresses to kerb login (and vice-versa), but Dovecot’s PAM interface 
> understandably just knows about a (login, password) pair, where the login is 
> the one provided by the user wanting to log in.
> 
> That said, I hope to be wrong,
> Axel

Dovecot auth username mapping

[Laz C. Peterson]

I have an interesting case here …

Virtual mailboxes, domain/username/aliases stored in MySQL, authentication done 
using PAM.  PAM authenticates through Kerberos, which are internal realms and 
not the email domains — for example, my username would be laz@PARAVIS.LOCAL 
 and my email address would be l...@paravis.net 
.

All of this works just fine.  But what I want to do is allow the users to log 
in using their email address and not their full Kerberos name.  It is becoming 
laborious to help the users understand the difference between their 
username@LOCAL.REALM and username@email.address  
and why we have to have two separate identities that mean the same thing.

I have the SQL statements to convert either the Kerberos login or the email 
address to the actual Kerberos login (so they may use either).  But I cannot 
seem to figure out how to get Dovecot to acknowledge this as the mapped 
username.

I’m sure there has to be a way.  Any help will be greatly appreciated.  Thank 
you!

~ Laz Peterson
Paravis, LLC