Submission: track authenticated_user
Hello,
We would like to use Dovecot Submission to have less queues to maintain.
The relayhost (Postfix) after Dovecot routes mail by sender_map, so
authenticated user, not the "mail from" because .
For what we've seen, we can't use receive header to retrieve this
authenticated_user.
Example of header :
Received: from mailhost ([0.0.0.0])
by submission.host with ESMTPSA
id submission-id
(envelope-from)
for; Thu, 9 Nov 2022 08:27:41 +
So we've thought to use X-client, but reading the doc seems that's not
the a good way :
/https://doc.dovecot.org/settings/core//
- submission_relay_trusted
If enabled, the relay server is trusted.
Determines whether we try to send (Postfix-specific) XCLIENT data to the
relay server (only if enabled).
But, XCLIENT for Submission seems to not transfer LOGIN :
/https://doc.dovecot.org/settings/core/
/
XCLIENT command can be used to override:
Session ID
Client IP and port (|%{rip}|,|%{rport}|)
HELO - Overrides what the client sent earlier in the EHLO command
LOGIN - Currently unused
PROTO - Currently unused
|forward_*| fields can be sent to auth process’s passdb lookup
The trust is always checked against the connecting IP address.
Except if HAProxy is used, then the original client IP address is used.
Do you know another way to inform the relayhost of submission of the
authenticated_user?
Thanks
Ismaël TANGUY
Re: Dovecot Submission - rate limiting
Le 31/03/2022 à 16:08, Aki Tuomi a écrit :
On 31/03/2022 17:00itan...@univ-brest.fr wrote:
Hello,
is there any way to rate limit submission mail (counting per user) when
using Dovecot/Submission (no Postfix)?
Using a milter seems impossible.
Using Dovecot Events seems "too" global (not per users discrimination).
Maybe rate-limiting has to be external and then include in the pass_db
query ?
Thank your for any hints.
Hi!
Dovecot submission is not an MTA, it *always* requires something behind. You
cannot use milters with it. If your MTA supports XFORWARD/XCLIENT you can use
submission_relay_trusted to allow this information to be forwarded to the MTA.
Aki
Thanks for your answer.
We find better to count at the Submission Step.
I try to use metric, but don't find yet the good parameters to see
submission events :
metric submission_counter {
filter = event=smtp_submit_finished
fields = mail_from recipients data_size
}
Submit some mails and then doveadm stats dump.
result stay null :
doveadm stats dump
metric_name field count sum min max avg median
stddev %95
submission_counter duration 0 0 0 0
0.00 0 0.000
submission_counter mail_from 0 0 0 0
0.00 0 0.000
submission_counter recipients 0 0 0 0
0.00 0 0.000
submission_counter data_size 0 0 0 0
0.00 0 0.000
Ismaël Tanguy
Dovecot Submission - rate limiting
Hello, is there any way to rate limit submission mail (counting per user) when using Dovecot/Submission (no Postfix)? Using a milter seems impossible. Using Dovecot Events seems "too" global (not per users discrimination). Maybe rate-limiting has to be external and then include in the pass_db query ? Thank your for any hints. Ismaël Tanguy
Re: Too many wait in auth process
On 8. Feb 2022, at 12.27, itan...@univ-brest.fr wrote:
service auth-worker {
client_limit = 1
idle_kill = 0
process_limit = 600
process_min_avail = 0
service_count = 1
vsz_limit = 18446744073709551615 B
}
What dovecot version is this? with 2.3.17 or later you should probably use
service_count=0 here.
That would prevent auth-worker process from dying after each authentication and
then need for new process to be spawned for each authentication.
Yes, it is 2.3.17.
I give a try, it's slighty better. There is a little fewer stalled auth
processes.
But I didn't manage to go more than 2000 clients although in production
it's more than 8000 connections.
Maybe, it's because I didn't find how to make persistent connections
with imaptest and there was too many login/logout. I use delay to make
client during around 5 seconds
So I increase this delay up to 120s, this slow down login/logout and
decrease processes stuck in wait auth queue.
I think I will go this way to simulate normal load on this server.
But that doesn't simulate a reboot of service while clients are connected.
Thank you all,
Ismaël
Re: Too many wait in auth process
Hello,
thank you for your advices and sorry to not have detailed infra
ismael> I'm currently benchmarking new hardware aimed to serve around
ismael> 70k users For now, our IMAP server have 13k users.
This doesn't help us help you. Is this a new rasperry Pi 4? Is it a
Dual CPU AMD Rzyzen with 128gb of memory and fast NVMe disks? What is
your system setup?
Sorry, I have two servers to bench :
- first one (a model like our current IMAP servers) is 18To HDD, 256Go
RAM, 8c/16th
- second (new one aimed to serve many more customers) is 24 x 14 TO (HDD
SAS), 192GB DDR4 2,6Ghz, 12c/24t - 2.4GHz/3.5GHz
OS is FreeBSD 12.2
ismael> To run imaptest, I've spwan some bench clients.
Are these tests run from remote hosts? What kind of network are you
using?
Yes, imaptest is running from kvm remote virtual machines in the same DC.
They are some networks hops between them, but few.
ismael> Each bench client can run imaptest with 1000 clients.
ismael> More than 1000 clients will load CPU of this bench client
ismael> imaptest command (command are chosen from usage stat on our other IMAP
servers):
ismael> imaptest host=x port=xxx userfile=userfile mbox=/root/dovecot-crlf
ismael> pass=s seed=123 clients=1000 select=194 uidfetch=94 noop=70
ismael> status=82 append=49 fetch=276 list=12 store=19 expunge=22
ismael> msubs=4 search=4 logout=1 delete=81 no_pipelining
ismael> With one bench client, everything runs smoothly.
ismael> # ps aux | grep dovecot | awk '{print $11,$12,$13,$14,$15,$16,$17,$18}'
| sort | uniq -c
ismael> 1 anvil: [221 connections] (anvil)
ismael> 1 auth: [13 wait, 0 passdb, 0 userdb] (auth)
ismael> 1 dovecot/config
ismael> 1 dovecot/imap
ismael> 84 dovecot/imap-login
ismael> 1 dovecot/log
ismael> 20 dovecot/pop3-login
ismael> 1 grep dovecot
ismael> 1 stats: [1307 connections] (stats)
ismael> When a second instance bench instance start imaptest, clients
ismael> of first and second instance begin to stall :
ismael> 1400 stalled for 20 secs in command: 1 LOGIN"fakeuser644@mailbench"
"password"
So how is your dovecot authentication setup? Are you using a mysql
backend? LDAP? Where is the server you're querying against? Are you
running mysql on the same server you're running dovecot on?
In production, we use a remote galera cluster.
On benchmarking, for now, I use static for passdb and a file for userdb.
Are you running multiple dovecot servers with dovecot director in
front of them to help spread the load and to offer resilience if/when
a backend server fails?
No. I'm directly benchmarking backend.
ismael> And :
ismael> # ps aux | grep dovecot | awk '{print $11,$12,$13,$14,$15,$16,$17,$18}'
| sort | uniq -c
ismael> 1 anvil: [221 connections] (anvil)
ismael> 1 auth: [1227 wait, 0 passdb, 0 userdb] (auth)
ismael> 1 dovecot/config
ismael> 1 dovecot/imap
ismael> 37 dovecot/imap-login
ismael> 1 dovecot/log
ismael> 20 dovecot/pop3-login
ismael> 1 grep dovecot
ismael> 1 stats: [680 connections] (stats)
ismael> Every auth go in wait, number of connection decreases.
ismael> Using mysql or a password file give same results.
Where is mysql located?
Remote one, but I'll go, for now, with a passwd-file to exclude
potentials DB problems at the beginning of benchmarking.
ismael> I have used different values for service_count with also no success.
Post your configuration details.
#doveconf -n
auth_cache_negative_ttl = 0
auth_cache_size = 100 M
auth_cache_ttl = 2 mins
auth_failure_delay = 5 secs
auth_master_user_separator = *
auth_username_chars =
abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ01234567890.-_@%+
auth_username_translation = %@
auth_verbose = yes
auth_worker_max_count = 500
base_dir = /var/run/dovecot/
default_client_limit = 10
disable_plaintext_auth = no
imap_idle_notify_interval = 30 secs
listen =
login_greeting = xx
login_trusted_networks = xxx
mail_gid =
mail_uid =
mailbox_list_index = no
namespace {
inbox = yes
location =
prefix = INBOX.
separator = .
type = private
}
namespace {
hidden = yes
inbox = no
list = no
location =
prefix =
separator = .
type = private
}
passdb {
args = password=#hidden_use-P_to_show#
driver = static
}
plugin {
acl = vfile
quota = maildir:User quota
}
protocols = imap pop3
service anvil {
client_limit = 97000
unix_listener anvil-auth-penalty {
mode = 00
}
}
service auth-worker {
client_limit = 1
idle_kill = 0
process_limit = 600
process_min_avail = 0
service_count = 1
vsz_limit = 18446744073709551615 B
}
service auth {
client_limit = 0
idle_kill = 0
process_limit = 1
process_min_avail = 1
service_count = 0
vsz_limit = 1000 M
}
service imap-login {
client_limit = 26000
process_min_avail = 16
service_count = 0
vsz_limit = 1 G
}
service imap {
drop_priv_before_exec = yes
process_limit = 1
}
