Re: Dovecot Submission Proxy Auth

[Paul Hecker]

Hi,

> On 6. Apr 2018, at 18:58, Odhiambo Washington  wrote:
> 
> Hi Paul,
> 
> Care to share your config (even OFFLIST) that has successfully integrated 
> Dovecot Submission service with Exim??

here the steps I have done to integrate Dovecot submission in Exim:

- Create and set the acl_smtp_mailauth ACL:

acl_smtp_mailauth = acl_check_mailauth

acl_check_mailauth:
  accept
hosts  = <; 127.0.0.1 ; ::1
condition  = ${if eq{$interface_port}{10025}}
log_message= Will accept MAIL AUTH parameter for $authenticated_sender

  deny


- add a deny fo all connections to 10025 without MAIL AUTH parameter in 
acl_smtp_mail ACL:

  deny
condition  = ${if eq{$interface_port}{10025}}
condition  = ${if eq{$authenticated_sender}{}}
message= All connections on port $interface_port need MAIL AUTH 
sender

- in Dovecot, add the following submission parameters

submission_relay_port = 10025
submission_relay_ssl = starttls
submission_relay_ssl_verify = no

All the remaining parts of the Dovecot config is the default for submission 
protocol/service, copied either from the sources (default config) or from here:

https://wiki.dovecot.org/Submission

Feel free is you have any further questions.

Regards,
Paul


> 
> I use Exim+Dovecot (Exim4U) and wouldn't mind exploring this.
> 
> Thanks in advance.
> 
> 
> On 6 April 2018 at 19:15, Paul Hecker  wrote:
> Hi,
> 
> Thanks you very much. This did the trick!
> 
> > On 6. Apr 2018, at 15:56, Stephan Bosch  wrote:
> >
> >
> >
> > Op 6-4-2018 om 13:52 schreef Paul Hecker:
> >> Hi,
> >>
> >> Dovecot 2.3.1 (8e2f634). Could not get Dovecot to forward the (plain) 
> >> authentication to the SMTP server using submission. Reason why I need it 
> >> is sender spoofing (do not want my employees to send messages in behalf of 
> >> me).
> >>
> >> In exim I can disable sender spoofing with the authenticated user. When 
> >> sending through dovecot, exim either does not accept the email (need auth) 
> >> or relay every sender address (because relaying from localhost).
> >>
> >> Am I missing a setting or do I need any additional field in the (MySQL) 
> >> user_query/password_query to forward the password?
> >>
> >> You can find my config here:
> >>
> >> https://gist.github.com/lluuaapp/7daddf761131da47237b0f45e6bab5a8
> >
> > That would be possible using the following SMTP AUTH feature:
> >
> > https://tools.ietf.org/html/rfc4954#section-5
> >
> > Which is apparently supported by Exim: 
> > https://www.exim.org/exim-html-current/doc/html/spec_html/ch-smtp_authentication.html#SECTauthparamail
> > This requires explicit configuration, so it will not work out of the box.
> 
> Here is what I did:
> 
> I had to add the acl_smtp_mailauth to only allow this on a certain port. Then 
> I had to duplicate my code for sender spoofing for authenticated users and 
> change the $authenticated_id -> $authenticated_sender.
> 
> Besides that, I must use TLS (in my case STARTTLS) so that Dovecot actually 
> sends the MAIL AUTH parameter.
> 
> >
> > The Dovecot Submission service should support this too. It sends an AUTH 
> > parameter with the MAIL command (currently only then the username is a 
> > valid SMTP address). However, I must say, I haven't tested this recently.
> 
> I can confirm that it works (only with TLS with my current configuration, see 
> above).
> 
> >
> > I can try this in a few days. Feel free to experiment with this yourself.
> >
> > Regards,
> >
> > Stephan.
> 
> Thanks again,
> Paul
> 
> 
> 
> 
> -- 
> Best regards,
> Odhiambo WASHINGTON,
> Nairobi,KE
> +254 7 3200 0004/+254 7 2274 3223
> "Oh, the cruft."



smime.p7s
Description: S/MIME cryptographic signature

Warning message about quota.conf

[forums]

Hi, 

In my maillog I see this warning message at every reception of e-mail: 

_STATUS=SENT (DELIVERED VIA DOVECOT SERVICE (DOVECONF: WARNING:
/ETC/DOVECOT/CONF.D/90-QUOTA.CONF LINE 39: GLOBAL SETTING MAIL_PLUGINS
WON'T CHAN._ 

It is just a warning, the e-mail was delivered. 

I don't find where is the problem. 

Thank's.

Dsync from multiple IMAPs -- personal/local dovecot setup

[Kim ALLAMANDOLA]

Hello list,
new aspiring user here, if I'm not in the wright place sorry,
I do not fund any other option for now...

As from subject I'm looking for a *personal* *standalone*
dovecot setup, especially I'm looking for a DSync from multiple
IMAP accounts to a local maildir (to being compatible with notmuch).

My actual mail setup is:
 OfflineIMAP that sync different mailboxes in a personal taxonomy
  ~/mail/$mailboxName/...
 and notmuch-emacs + various scripts as MUA. Local postfix just to
 avoid emacs freeze during send, muchsync to support

It does work but is slow, buggy and sometime complicated to kept up
to date (I have many homegrow spaghetti-code scripts and snippets
in zsh, elisp, python etc stratified in years). My dream is switch
to a local Dovecot that DSync from multiple IMAPs instead of
OfflineIMAP, with sieve filters instead of homegrow scripts that
wrap notmuch search --output=files | xargs ...

For the MUA part I'd like to stick with notmuch. So my questions are:
is possible to dsync from multiple IMAPS to a maildir? if yes does
dovecot accept physical messages moves if I refile (archive etc) them
via notmuch?

Thanks!

-- Kim ALLAMANDOLA

Re: Dovecot Submission Proxy Auth

[Odhiambo Washington]

Hi Paul,

Care to share your config (even OFFLIST) that has successfully integrated
Dovecot Submission service with Exim??

I use Exim+Dovecot (Exim4U) and wouldn't mind exploring this.

Thanks in advance.


On 6 April 2018 at 19:15, Paul Hecker  wrote:

> Hi,
>
> Thanks you very much. This did the trick!
>
> > On 6. Apr 2018, at 15:56, Stephan Bosch  wrote:
> >
> >
> >
> > Op 6-4-2018 om 13:52 schreef Paul Hecker:
> >> Hi,
> >>
> >> Dovecot 2.3.1 (8e2f634). Could not get Dovecot to forward the (plain)
> authentication to the SMTP server using submission. Reason why I need it is
> sender spoofing (do not want my employees to send messages in behalf of me).
> >>
> >> In exim I can disable sender spoofing with the authenticated user. When
> sending through dovecot, exim either does not accept the email (need auth)
> or relay every sender address (because relaying from localhost).
> >>
> >> Am I missing a setting or do I need any additional field in the (MySQL)
> user_query/password_query to forward the password?
> >>
> >> You can find my config here:
> >>
> >> https://gist.github.com/lluuaapp/7daddf761131da47237b0f45e6bab5a8
> >
> > That would be possible using the following SMTP AUTH feature:
> >
> > https://tools.ietf.org/html/rfc4954#section-5
> >
> > Which is apparently supported by Exim: https://www.exim.org/exim-
> html-current/doc/html/spec_html/ch-smtp_authentication.
> html#SECTauthparamail
> > This requires explicit configuration, so it will not work out of the box.
>
> Here is what I did:
>
> I had to add the acl_smtp_mailauth to only allow this on a certain port.
> Then I had to duplicate my code for sender spoofing for authenticated users
> and change the $authenticated_id -> $authenticated_sender.
>
> Besides that, I must use TLS (in my case STARTTLS) so that Dovecot
> actually sends the MAIL AUTH parameter.
>
> >
> > The Dovecot Submission service should support this too. It sends an AUTH
> parameter with the MAIL command (currently only then the username is a
> valid SMTP address). However, I must say, I haven't tested this recently.
>
> I can confirm that it works (only with TLS with my current configuration,
> see above).
>
> >
> > I can try this in a few days. Feel free to experiment with this yourself.
> >
> > Regards,
> >
> > Stephan.
>
> Thanks again,
> Paul
>
>


-- 
Best regards,
Odhiambo WASHINGTON,
Nairobi,KE
+254 7 3200 0004/+254 7 2274 3223
"Oh, the cruft."

Re: Dovecot Submission Proxy Auth

[Paul Hecker]

Hi,

Thanks you very much. This did the trick!

> On 6. Apr 2018, at 15:56, Stephan Bosch  wrote:
> 
> 
> 
> Op 6-4-2018 om 13:52 schreef Paul Hecker:
>> Hi,
>> 
>> Dovecot 2.3.1 (8e2f634). Could not get Dovecot to forward the (plain) 
>> authentication to the SMTP server using submission. Reason why I need it is 
>> sender spoofing (do not want my employees to send messages in behalf of me).
>> 
>> In exim I can disable sender spoofing with the authenticated user. When 
>> sending through dovecot, exim either does not accept the email (need auth) 
>> or relay every sender address (because relaying from localhost).
>> 
>> Am I missing a setting or do I need any additional field in the (MySQL) 
>> user_query/password_query to forward the password?
>> 
>> You can find my config here:
>> 
>> https://gist.github.com/lluuaapp/7daddf761131da47237b0f45e6bab5a8
> 
> That would be possible using the following SMTP AUTH feature:
> 
> https://tools.ietf.org/html/rfc4954#section-5
> 
> Which is apparently supported by Exim: 
> https://www.exim.org/exim-html-current/doc/html/spec_html/ch-smtp_authentication.html#SECTauthparamail
> This requires explicit configuration, so it will not work out of the box.

Here is what I did:

I had to add the acl_smtp_mailauth to only allow this on a certain port. Then I 
had to duplicate my code for sender spoofing for authenticated users and change 
the $authenticated_id -> $authenticated_sender.

Besides that, I must use TLS (in my case STARTTLS) so that Dovecot actually 
sends the MAIL AUTH parameter.

> 
> The Dovecot Submission service should support this too. It sends an AUTH 
> parameter with the MAIL command (currently only then the username is a valid 
> SMTP address). However, I must say, I haven't tested this recently.

I can confirm that it works (only with TLS with my current configuration, see 
above).

> 
> I can try this in a few days. Feel free to experiment with this yourself.
> 
> Regards,
> 
> Stephan.

Thanks again,
Paul



smime.p7s
Description: S/MIME cryptographic signature

Re: Dovecot Submission Proxy Auth

[Stephan Bosch]

Op 6-4-2018 om 13:52 schreef Paul Hecker:

Hi,

Dovecot 2.3.1 (8e2f634). Could not get Dovecot to forward the (plain) 
authentication to the SMTP server using submission. Reason why I need it is 
sender spoofing (do not want my employees to send messages in behalf of me).

In exim I can disable sender spoofing with the authenticated user. When sending 
through dovecot, exim either does not accept the email (need auth) or relay 
every sender address (because relaying from localhost).

Am I missing a setting or do I need any additional field in the (MySQL) 
user_query/password_query to forward the password?

You can find my config here:

https://gist.github.com/lluuaapp/7daddf761131da47237b0f45e6bab5a8


That would be possible using the following SMTP AUTH feature:

https://tools.ietf.org/html/rfc4954#section-5

Which is apparently supported by Exim: 
https://www.exim.org/exim-html-current/doc/html/spec_html/ch-smtp_authentication.html#SECTauthparamail

This requires explicit configuration, so it will not work out of the box.

The Dovecot Submission service should support this too. It sends an AUTH 
parameter with the MAIL command (currently only then the username is a 
valid SMTP address). However, I must say, I haven't tested this recently.


I can try this in a few days. Feel free to experiment with this yourself.

Regards,

Stephan.

Re: 2.3.1 Replication is throwing scary errors

[Michael Grimm]

Reuben Farrelly wrote:
> From: Michael Grimm 

>> [This is Dovecot 2.3.1 at FreeBSD STABLE-11.1 running in two jails at 
>> distinct servers.]
>> I did upgrade from 2.2.35 to 2.3.1 today, and I do become pounded by error 
>> messages at server1 (and vice versa at server2) as follows:
>>  | Apr  2 17:12:18  server1.lan dovecot: doveadm: Error: 
>> dsync(server2.lan): I/O has stalled, \
>>  no activity for 600 seconds (last sent=mail_change, last 
>> recv=mail_change (EOL))
>>  | Apr  2 17:12:18  server1.lan dovecot: doveadm: Error: 
>> Timeout during state=sync_mails \
>>  (send=changes recv=mail_requests)
>>  [?]
>>  | Apr  2 18:59:03  server1.lan dovecot: doveadm: Error: 
>> dsync(server2.lan): I/O has stalled, \
>>  no activity for 600 seconds (last sent=mail, last recv=mail 
>> (EOL))
>>  | Apr  2 18:59:03  server1.lan dovecot: doveadm: Error: 
>> Timeout during state=sync_mails \
>>  (send=mails recv=recv_last_common)
>> I cannot see in my personal account any missing replications, *but* I 
>> haven't tested this thoroughly enough. I do have customers being serviced at 
>> these productive servers, *thus* I'm back to 2.2.35 until I do understand or 
>> have learned what is going on.

In my reply to this statement of mine I mentioned that I have seen those 
timeouts quite some times during the past year. Thus, I upgraded to 2.3.1 
again, and boom: after some hours I ended up in hanging processes [1] like (see 
Remko's mail in addition) ...

doveadm-server: [IP4/6  SOME/MAILBOX import:0/0] (doveadm-server)

… at server2 paired with a file like …

-rw--- 1 vmail dovecot uarch 0 Apr 3 16:52 
/home/to/USER1/.dovecot-sync.lock 

Corresponding logfile entries at server2 are like …

  Apr  3 17:10:49  server2.lan dovecot: doveadm: Error: Couldn't lock 
/home/to/USER1/.dovecot-sync.lock: \
  fcntl(/home/to/USER1/.dovecot-sync.lock, write-lock, F_SETLKW) locking 
failed: Timed out after 30 seconds \
  (WRITE lock held by pid 51110)

[1] Even stopping dovecot will not end those processes. One has to manually 
kill those before restarting dovecot.

After one day of testing 2.3.1 with a couple of those episodes of 
locking/timeout, and now missing mails depending with server your MUA will 
connect to, I went back to 2.2.35. After two days at that version I never had 
such an episode again.

> It's not just you.  This issue hit me recently, and it was impacting
> replication noticeably.  I am following git master-2.3 .
[...]
> There is also a second issue of a long standing race with replication
> occurring somewhere whereby if a mail comes in, is written to disk, is
> replicated and then deleted in short succession, it will reappear
> again to the MUA.  I suspect the mail is being replicated back from
> the remote.  A few people have reported it over the years but it's not
> reliable or consistent, so it has never been fixed.
> And lastly there has been an ongoing but seemingly minor issue
> relating to locking timing out after 30s particularly on the remote
> host that is being replicated to.  I rarely see the problem on my
> local disk where almost all of the mail comes in, it's almost always
> occurring on the replicate/remote system.

It might be time to describe our setups in order to possibly find common 
grounds that might trigger this issue you describe and Rimko and myself ran 
into as well.

Servers:Cloud Instances (both identical), around 25ms latency apart.
Intel Core Processor (Haswell, no TSX) (3092.91-MHz K8-class 
CPU)
Both servers are connected via IPsec/racoon tunnels
OS: FreeBSD 11.1-STABLE (both servers)
Filesystem: ZFS
MTA:postfix 3.4-20180401 (postfix delivers via dovecot's LMTP)
IMAP:   dovecot running in FreeBSD jails (issues with 2.3.1, fine with 
2.2.35)
Replication:unsecured tcp / master-master
MUA:mainly iOS or macOS mail.app, rarely roundcube

I believe it is worthwhile to mention here that I run a poor man's fail-over 
approach (round-robin DNS) as follows:

DNS:mail.server.tld resolves to one IP4 and one IP6 address of each 
server, thus 4 IP addresses in total

According it's MX priority one server (server1) will receive most mail, thus 
dovecot will mostly replicate mail from server1 to server2. Server2 is the one 
showing that deadlocking issues you see in your setup as well.

But connecting MUAs will hop quite frequently between server1 and server2, and 
sometimes will connect to both servers simultaneously, even mixing IP4 and IP6, 
because MUA do not follow MX priorities (IIRC). Normally I would believe that 
this shouldn't be an issue for dovecot, but let me ask dovecot's developers: 

Can those simultaneous connects and modifications of \SEEN et al. be a reason 
for my issues regarding deadlocking?

> For me it seems very unlikely that on an unloaded/idle VPS there 

Dovecot Submission Proxy Auth

[Paul Hecker]

Hi,

Dovecot 2.3.1 (8e2f634). Could not get Dovecot to forward the (plain) 
authentication to the SMTP server using submission. Reason why I need it is 
sender spoofing (do not want my employees to send messages in behalf of me).

In exim I can disable sender spoofing with the authenticated user. When sending 
through dovecot, exim either does not accept the email (need auth) or relay 
every sender address (because relaying from localhost).

Am I missing a setting or do I need any additional field in the (MySQL) 
user_query/password_query to forward the password?

You can find my config here:

https://gist.github.com/lluuaapp/7daddf761131da47237b0f45e6bab5a8

Thanks,
Paul





smime.p7s
Description: S/MIME cryptographic signature

Re: Compatibility of Submission in 2.3.1

[A. Schulze]

Stephan Bosch:

Here is the debug log (note the space between “From:" and  
">”):


Any idea which client is doing that?



older android mail app for example...
see https://marc.info/?l=postfix-users=141600120612100

Re: Compatibility of Submission in 2.3.1

[Stephan Bosch]

Op 6-4-2018 om 12:29 schreef Paul Hecker:

Hi,

Dovecot 2.3.1 (8e2f634). When using submission one client of mine can 
not send messages. Directly sending through exim works. Seems that 
dovecot is too picky or follow the RFC too precise here to be compatible.


Here is the debug log (note the space between “From:" and 
">”):


Apr  6 12:02:08 mail dovecot: submission(u...@host.com 
)<14972>: Debug: smtp-server: 
conn 22.73.xx.144:53943 [0]: Received new command: MAIL From: 
>
Apr  6 12:02:08 mail dovecot: submission(u...@host.com 
)<14972>: Debug: smtp-server: 
conn 22.73.xx.144:53943 [0]: command MAIL; 501 reply: Submitted
Apr  6 12:02:08 mail dovecot: submission(u...@host.com 
)<14972>: Debug: smtp-server: 
conn 22.73.xx.144:53943 [0]: command MAIL: Ready to reply
Apr  6 12:02:08 mail dovecot: submission(u...@host.com 
)<14972>: Debug: smtp-server: 
conn 22.73.xx.144:53943 [0]: Trigger output
Apr  6 12:02:08 mail dovecot: submission(u...@host.com 
)<14972>: Debug: smtp-server: 
conn 22.73.xx.144:53943 [0]: Timeout stop
Apr  6 12:02:08 mail dovecot: submission(u...@host.com 
)<14972>: Debug: smtp-server: 
conn 22.73.xx.144:53943 [0]: Sending replies
Apr  6 12:02:08 mail dovecot: submission(u...@host.com 
)<14972>: Debug: smtp-server: 
conn 22.73.xx.144:53943 [0]: command MAIL; 501 reply: Sent: 501 5.5.4 
Invalid FROM: Missing '<' at beginning of path
Apr  6 12:02:08 mail dovecot: submission(u...@host.com 
)<14972>: Debug: smtp-server: 
conn 22.73.xx.144:53943 [0]: command MAIL: Destroy
Apr  6 12:02:08 mail dovecot: submission(u...@host.com 
)<14972>: Debug: smtp-server: 
conn 22.73.xx.144:53943 [0]: command MAIL; 501 reply: Destroy


Is there a chance that you will incorporate a fix (be more compatible 
but less stringent)?


I can make an option for doing that (some sort of workaround setting). I 
am not a great fan of being liberal by default. That way, violations 
like that will never surface. We will discuss this internally..


Regards,

Stephan.

Compatibility of Submission in 2.3.1

[Paul Hecker]

Hi,

Dovecot 2.3.1 (8e2f634). When using submission one client of mine can not send 
messages. Directly sending through exim works. Seems that dovecot is too picky 
or follow the RFC too precise here to be compatible.

Here is the debug log (note the space between “From:" and "”):

Apr  6 12:02:08 mail dovecot: 
submission(u...@host.com)<14972>: Debug: smtp-server: conn 
22.73.xx.144:53943 [0]: Received new command: MAIL From: 
Apr  6 12:02:08 mail dovecot: 
submission(u...@host.com)<14972>: Debug: smtp-server: conn 
22.73.xx.144:53943 [0]: command MAIL; 501 reply: Submitted
Apr  6 12:02:08 mail dovecot: 
submission(u...@host.com)<14972>: Debug: smtp-server: conn 
22.73.xx.144:53943 [0]: command MAIL: Ready to reply
Apr  6 12:02:08 mail dovecot: 
submission(u...@host.com)<14972>: Debug: smtp-server: conn 
22.73.xx.144:53943 [0]: Trigger output
Apr  6 12:02:08 mail dovecot: 
submission(u...@host.com)<14972>: Debug: smtp-server: conn 
22.73.xx.144:53943 [0]: Timeout stop
Apr  6 12:02:08 mail dovecot: 
submission(u...@host.com)<14972>: Debug: smtp-server: conn 
22.73.xx.144:53943 [0]: Sending replies
Apr  6 12:02:08 mail dovecot: 
submission(u...@host.com)<14972>: Debug: smtp-server: conn 
22.73.xx.144:53943 [0]: command MAIL; 501 reply: Sent: 501 5.5.4 Invalid FROM: 
Missing '<' at beginning of path
Apr  6 12:02:08 mail dovecot: 
submission(u...@host.com)<14972>: Debug: smtp-server: conn 
22.73.xx.144:53943 [0]: command MAIL: Destroy
Apr  6 12:02:08 mail dovecot: 
submission(u...@host.com)<14972>: Debug: smtp-server: conn 
22.73.xx.144:53943 [0]: command MAIL; 501 reply: Destroy

Is there a chance that you will incorporate a fix (be more compatible but less 
stringent)?

Thanks,
Paul

smime.p7s
Description: S/MIME cryptographic signature