Re: New Dovecot version? github repo without new updates?
> On 19/07/2023 01:22 EEST Jorge Bastos wrote: > > > Hi Guys, > Just asking, i don't see a new dovecot version since last December, and the > girhub repo has no new requests/updates. > Is there a new place for dovecot versions, or it's happening something? > Thanks in advanced, > Jorge, Our github repo has been last updated 2 days ago, so not sure what you mean... We are working on new major release, so this usually causes a longer pause between releases. Aki ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
Re: Replication going away?
>> While I understand it takes effort to maintain the replication plugin, this >> is especially problematic for small active/active high-availability >> deployments. >> I guess there are lots of servers that use replication for just 50 or 100 >> mailboxes. Cloudstorage (like S3) would be overkill for these. > > Even without active/active, it's super useful for the simple > active/backup configuration which I use on my personal mail server This depends heavily on individual usage. Coming from an active/active deployment it's a major step backwards though: usually two servers are running independently in geographically dispersed datacenters. High-availabilty is achieved by a simple DNS entry that returns two ip addresses, one from each datacenter. Under normal circumstances that gives you 50/50 loadbalancing without loadbalancers, without additional components that can fail. In case one datacenter goes down, and that happens to every datacenter at some time, the other datacenter takes over - automatically, without any configuration changes. Additionally mail user agents (Outlook, Thunderbird, ...) don't need special configuration. If one ip address is unrechable they connect to the other one obtained via DNS and users can quite seemlessly send and receive email again. After the outage ceased and the other datacenter is back online again, there is nothing to do. No configuration changes, no error prone manual synchronization or promoting passive to active - it just works and heals itself. Being used to a carefree setup like that you don't want to go back. Of course there are other possibilities like nfs, glusterfs, gfs2, zfs snapshots, ceph, minio or dsync backup but they all have their own drawbacks. For small mailservers that want high availability dsync replication is quite the perfect solution. > setup (one colo box, one home server) and a small company mail > server; as such I'm pretty sad to see it go. Still, it is up > to OX where they want to put their resources. Well, it seems the dsync replication function is still there, just the replication plugin that notifies what to replicate is deprectated. Of course it's OX's decision, I'm just hoping they were not aware how useful replication is in the before mentioned scenario. Moreover I'm quite sure this kind of small-scale replication does not have any impact on customers upgrading to the new cloud architecture. Big customers will go for cloud because it scales way better and does not have replication induced performance penalties and small customers probably can't afford to upgrade because it's too pricey. > I guess losing repl probably doesn't affect larger ISP type setups > so much; it seems a bit more common to use shared storage (e.g. > maildirs on an nfs appliance or similar) in those cases if they're > actually running their own storage. > >> Do you provide dovecot pro subscriptions for such small deployments? > > Unless I misunderstood the message (and I don't think I did), repl > was removed in pro too. (I don't expect that pro is available on my > usual choice of OS anyway..). As I understood it dsync is still working. Replication configured via ssh is calling dsync under the hood, so if local storage and index/log formats don't change for single deployments, it seems to be more of a political decision. I know maintenance is not for free, that's why I suggested to think about a dovecot small/medium business edition with a more affordable price tag. Best regards, Gerald ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
New Dovecot version? github repo without new updates?
Hi Guys, Just asking, i don't see a new dovecot version since last December, and the girhub repo has no new requests/updates. Is there a new place for dovecot versions, or it's happening something? Thanks in advanced, Jorge,___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
Re: Replication going away?
On 2023-07-18, Gerald Galster wrote: > While I understand it takes effort to maintain the replication plugin, this > is especially problematic for small active/active high-availability > deployments. > I guess there are lots of servers that use replication for just 50 or 100 > mailboxes. Cloudstorage (like S3) would be overkill for these. Even without active/active, it's super useful for the simple active/backup configuration which I use on my personal mail server setup (one colo box, one home server) and a small company mail server; as such I'm pretty sad to see it go. Still, it is up to OX where they want to put their resources. I guess losing repl probably doesn't affect larger ISP type setups so much; it seems a bit more common to use shared storage (e.g. maildirs on an nfs appliance or similar) in those cases if they're actually running their own storage. > Do you provide dovecot pro subscriptions for such small deployments? Unless I misunderstood the message (and I don't think I did), repl was removed in pro too. (I don't expect that pro is available on my usual choice of OS anyway..). ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
Re: Fatal error in http-client.c when passdb is oauth2 and userdb is sql
- все Test reply. Sorry. 18.07.2023, 15:37, "Aki Tuomi via dovecot" : On 18/07/2023 15:10 EEST Евгений Бахтинwrote:Dovecot version: 2.3.20 Dear Colleagues! passdb is set to oauth2 and userdb is set to sql. On attempt of authorization a fatal error is rising. The case reproducible 100%. The following suggested configuration files allow you to reproduce the case. Thank you! Can you send auth_debug=yes and debug=yes (in oauth2 config) logs?Aki___dovecot mailing list -- dovecot@dovecot.orgTo unsubscribe send an email to dovecot-le...@dovecot.org___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
Options to track performance?
Hi there, after upgrading my dovecot on a bookworm container, I now have a weird delay when imap clients like Evolution connect the first time. Is there any performance logging configuration I could enable, to see what dovecot is doing in which timing? I suspect some timeout or delay somewhere, but unable to find it so far. Kind regards Chris ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
-d ${user}@${nexthop} or -d ${recipient}?
What's the differency between these two lines; 1- flags=DORhu user=vmail:vmail argv=/usr/local/libexec/dovecot/dovecot-lda -f ${sender} -r ${recipient} -d ${user}@${nexthop} 2- flags=DRhu user=vmail:vmail argv=/usr/local/libexec/dovecot/dovecot-lda -f ${sender} -d ${recipient} and which one should I use, based on a setup like; Dovecot, Postfix, Postfixadmin, ManageSieve and virtual users hosted in SQL? Thanks in advance. ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
Re: dovecot: lda(info@mydomain) - vacation action: discarding vacation response to mailinglist recipient?
Weird, but I think I solved it, I had in my postfix main.cf, a line like: "header_checks = regexp:/etc/postfix/list_unsub_header" to have a header like: List-Unsubscribe: mailto:i...@mydomain.com?subject=unsubscribe, and when I removed that line, the auto-reply seems worked. BUT, why's so? The sender is Gmail, List-Unsubscribe header is my own mailbox header, not the Gmail's.. Why it detects the SENDER e-mail account as "mailinglist"? ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
Re: Replication going away?
> While I understand it takes effort to maintain the replication plugin, this > is especially problematic for small active/active high-availability > deployments. > I guess there are lots of servers that use replication for just 50 or 100 > mailboxes. Cloudstorage (like S3) would be overkill for these. > > Do you provide dovecot pro subscriptions for such small deployments? I’m running such a setup for 5-10 mailboxes.. :D -- https://steven.varco.ch/ ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
Re: Replication going away?
>> Just to understand that correctly: I could setup a (cron) based process for >> doveadm sync, but no longer a setup like >> plugin { >> mail_replica = tcp:$IMAP_REPLICA_SERVER:$IMAP_REPLICA_PORT >> } >> where the cron would lead to some delay and would have to check for >> concurrent jobs? > > You can also have that too. > > doveadm sync -d > > makes it use mail_replica setting. @Aki: Is it possible to monitor actions that would have triggered replication in dovecot < 2.4, e.g. parsing logs or a lua script to imitate the previous behaviour? @Michael Slusarz: While I understand it takes effort to maintain the replication plugin, this is especially problematic for small active/active high-availability deployments. I guess there are lots of servers that use replication for just 50 or 100 mailboxes. Cloudstorage (like S3) would be overkill for these. Do you provide dovecot pro subscriptions for such small deployments? The basic replication functionality with dsync seems to be available in future versions. Would you consider releasing a "dovecot smb" version for small/medium businesses that maintains the replication plugin (without director) for a yearly subscription fee? Otherwise those affected would have to look for alternatives. The recent release of rust-written Stalwart mailserver comes to mind, that bundles a lot of functionality (smtp, pop, imap, jmap, s3, sieve, dkim/arc/spf/dmarc, dane, mta-sas, ...): https://stalw.art/blog/ Best regards, Gerald ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
Re: Fatal error in http-client.c when passdb is oauth2 and userdb is sql
I think this mail related to the problem. https://dovecot.org/mailman3/archives/list/dovecot@dovecot.org/message/X5PBA2RVC7ZDIUNMXJQZRFIDS7MZNGJH/ ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
Re: Fatal error in http-client.c when passdb is oauth2 and userdb is sql
Please, delete this wrong thread. ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
Re: Fatal error in http-client.c when passdb is oauth2 and userdb is sql
Please, delete this wrong thread. ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
Re: Fatal error in http-client.c when passdb is oauth2 and userdb is sql
Aki Tuomi wrote: > > Can you send auth_debug=yes and debug=yes (in oauth2 config) logs? This is with debug. Jul 18 15:29:14 mail01 dovecot: master: Dovecot v2.3.20 (80a5ac675d) starting up for imap (core dumps disabled) Jul 18 15:29:21 mail01 dovecot: auth: Debug: Loading modules from directory: /usr/lib/dovecot/modules/auth Jul 18 15:29:21 mail01 dovecot: auth: Debug: Module loaded: /usr/lib/dovecot/modules/auth/lib20_auth_var_expand_crypt.so Jul 18 15:29:21 mail01 dovecot: auth: Debug: Module loaded: /usr/lib/dovecot/modules/auth/libdriver_pgsql.so Jul 18 15:29:21 mail01 dovecot: auth: Debug: sqlpool(pgsql): Creating new connection Jul 18 15:29:21 mail01 dovecot: auth: Debug: Read auth token secret from /run/dovecot/auth-token-secret.dat Jul 18 15:29:21 mail01 dovecot: auth: Debug: sqlpool(pgsql): Creating new connection Jul 18 15:29:21 mail01 dovecot: auth: Panic: file http-client.c: line 646 (http_client_context_close): assertion failed: (cctx->clients_list == NULL) Jul 18 15:29:21 mail01 dovecot: auth: Error: Raw backtrace: /usr/lib/dovecot/libdovecot.so.0(backtrace_append+0x3d) [0x7f8bd1f0c85d] -> /usr/lib/dovecot/libdovecot.so.0(backtrace_get+0x1e) [0x7f8bd1f0c97e] -> /usr/lib/dovecot/libdovecot.so.0(+0x10091b) [0x7f8bd1f1991b] -> /usr/lib/dovecot/libdovecot.so.0(+0x1009b1) [0x7f8bd1f199b1] -> /usr/lib/dovecot/libdovecot.so.0(+0x54b7c) [0x7f8bd1e6db7c] -> /usr/lib/dovecot/libdovecot.so.0(+0x4d208) [0x7f8bd1e66208] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_set_current+0x75) [0x7f8bd1f2eca5] -> /usr/lib/dovecot/modules/auth/libdriver_pgsql.so(+0x4686) [0x7f8bd1b43686] -> /usr/lib/dovecot/modules/auth/libdriver_pgsql.so(+0x49b7) [0x7f8bd1b439b7] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_call_io+0x69) [0x7f8bd1f2fe59] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0x131) [0x7f8bd1f31481] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0x4c) [0x7f8bd1f2fefc] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_run+0x40) [0x7f8 bd1f30080] -> /usr/lib/dovecot/modules/auth/libdriver_pgsql.so(+0x57ec) [0x7f8bd1b447ec] -> /usr/lib/dovecot/modules/auth/libdriver_pgsql.so(+0x405b) [0x7f8bd1b4305b] -> dovecot/auth(+0x4728b) [0x649266d1628b] -> dovecot/auth(+0x36ca1) [0x649266d05ca1] -> dovecot/auth(userdb_init+0x1a) [0x649266d0370a] -> dovecot/auth(auths_init+0xc9) [0x649266ce4ce9] -> dovecot/auth(main+0x398) [0x649266ce3b78] -> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xeb) [0x7f8bd1bc709b] -> dovecot/auth(_start+0x2a) [0x649266ce3d8a] Jul 18 15:29:21 mail01 dovecot: master: Error: service(auth): command startup failed, throttling for 2.000 secs Jul 18 15:29:21 mail01 dovecot: auth: Fatal: master: service(auth): child 12988 killed with signal 6 (core dumps disabled - https://dovecot.org/bugreport.html#coredumps) ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
Re: Tons of imap-login processes despite client_limit very high
Awesome, thank you for the information Aki. :) We'll set vsz_limit and keep service_count = 0, as service_count >= 1 is not an option for us due to the high memory consumption associated with having many imap-login processes running. The new process_shutdown_filter option looks interesting, though if it shutdowns the process "after finishing the current connections", as the doc says, that may not resolve the issue in our case due to imap-login processes often having lasting connections (preventing the process to be recycled when service_count > 1 in the first place). Thanks again, very helpful! ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
Re: Replication going away?
> On 18/07/2023 15:19 EEST i...@joergschulz.de wrote: > > > Just to understand that correctly: I could setup a (cron) based process for > doveadm sync, but no longer a setup like > plugin { > mail_replica = tcp:$IMAP_REPLICA_SERVER:$IMAP_REPLICA_PORT > } > where the cron would lead to some delay and would have to check for > concurrent jobs? You can also have that too. doveadm sync -d makes it use mail_replica setting. Aki ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
Re: Fatal error in http-client.c when passdb is oauth2 and userdb is sql
Can you send auth_debug=yes and debug=yes (in oauth2 config) logs?This is with debug. Jul 18 15:29:14 mail01 dovecot: master: Dovecot v2.3.20 (80a5ac675d) starting up for imap (core dumps disabled)Jul 18 15:29:21 mail01 dovecot: auth: Debug: Loading modules from directory: /usr/lib/dovecot/modules/authJul 18 15:29:21 mail01 dovecot: auth: Debug: Module loaded: /usr/lib/dovecot/modules/auth/lib20_auth_var_expand_crypt.soJul 18 15:29:21 mail01 dovecot: auth: Debug: Module loaded: /usr/lib/dovecot/modules/auth/libdriver_pgsql.soJul 18 15:29:21 mail01 dovecot: auth: Debug: sqlpool(pgsql): Creating new connectionJul 18 15:29:21 mail01 dovecot: auth: Debug: Read auth token secret from /run/dovecot/auth-token-secret.datJul 18 15:29:21 mail01 dovecot: auth: Debug: sqlpool(pgsql): Creating new connectionJul 18 15:29:21 mail01 dovecot: auth: Panic: file http-client.c: line 646 (http_client_context_close): assertion failed: (cctx->clients_list == NULL)Jul 18 15:29:21 mail01 dovecot: auth: Error: Raw backtrace: /usr/lib/dovecot/libdovecot.so.0(backtrace_append+0x3d) [0x7f8bd1f0c85d] -> /usr/lib/dovecot/libdovecot.so.0(backtrace_get+0x1e) [0x7f8bd1f0c97e] -> /usr/lib/dovecot/libdovecot.so.0(+0x10091b) [0x7f8bd1f1991b] -> /usr/lib/dovecot/libdovecot.so.0(+0x1009b1) [0x7f8bd1f199b1] -> /usr/lib/dovecot/libdovecot.so.0(+0x54b7c) [0x7f8bd1e6db7c] -> /usr/lib/dovecot/libdovecot.so.0(+0x4d208) [0x7f8bd1e66208] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_set_current+0x75) [0x7f8bd1f2eca5] -> /usr/lib/dovecot/modules/auth/libdriver_pgsql.so(+0x4686) [0x7f8bd1b43686] -> /usr/lib/dovecot/modules/auth/libdriver_pgsql.so(+0x49b7) [0x7f8bd1b439b7] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_call_io+0x69) [0x7f8bd1f2fe59] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0x131) [0x7f8bd1f31481] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0x4c) [0x7f8bd1f2fefc] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_run+0x40) [0x7f8bd1f30080] -> /usr/lib/dovecot/modules/auth/libdriver_pgsql.so(+0x57ec) [0x7f8bd1b447ec] -> /usr/lib/dovecot/modules/auth/libdriver_pgsql.so(+0x405b) [0x7f8bd1b4305b] -> dovecot/auth(+0x4728b) [0x649266d1628b] -> dovecot/auth(+0x36ca1) [0x649266d05ca1] -> dovecot/auth(userdb_init+0x1a) [0x649266d0370a] -> dovecot/auth(auths_init+0xc9) [0x649266ce4ce9] -> dovecot/auth(main+0x398) [0x649266ce3b78] -> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xeb) [0x7f8bd1bc709b] -> dovecot/auth(_start+0x2a) [0x649266ce3d8a]Jul 18 15:29:21 mail01 dovecot: master: Error: service(auth): command startup failed, throttling for 2.000 secsJul 18 15:29:21 mail01 dovecot: auth: Fatal: master: service(auth): child 12988 killed with signal 6 (core dumps disabled - https://dovecot.org/bugreport.html#coredumps) ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
dovecot: lda(info@mydomain) - vacation action: discarding vacation response to mailinglist recipient?
Hello, I'm trying to set auto reply for my email account, through Roundcube and ManageSieve plugin (the plugin itself and its regular filtering works fine), however the auto-reply doesn't arrive to the sender's inbox, when I tested it with sending an email to myself/my server, from my Gmail account, my maillog has a line, stating; Jul 18 15:28:06 mailsrv dovecot: lda(i...@mywebsite.com)<44812>: sieve: msgid=: vacation action: discarding vacation response to mailinglist recipient Any idea would be much appreciated, thank you! ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
Re: Fatal error in http-client.c when passdb is oauth2 and userdb is sql
Can you send auth_debug=yes and debug=yes (in oauth2 config) logs?This is with debug. Jul 18 15:29:14 mail01 dovecot: master: Dovecot v2.3.20 (80a5ac675d) starting up for imap (core dumps disabled)Jul 18 15:29:21 mail01 dovecot: auth: Debug: Loading modules from directory: /usr/lib/dovecot/modules/authJul 18 15:29:21 mail01 dovecot: auth: Debug: Module loaded: /usr/lib/dovecot/modules/auth/lib20_auth_var_expand_crypt.soJul 18 15:29:21 mail01 dovecot: auth: Debug: Module loaded: /usr/lib/dovecot/modules/auth/libdriver_pgsql.soJul 18 15:29:21 mail01 dovecot: auth: Debug: sqlpool(pgsql): Creating new connectionJul 18 15:29:21 mail01 dovecot: auth: Debug: Read auth token secret from /run/dovecot/auth-token-secret.datJul 18 15:29:21 mail01 dovecot: auth: Debug: sqlpool(pgsql): Creating new connectionJul 18 15:29:21 mail01 dovecot: auth: Panic: file http-client.c: line 646 (http_client_context_close): assertion failed: (cctx->clients_list == NULL)Jul 18 15:29:21 mail01 dovecot: auth: Error: Raw backtrace: /usr/lib/dovecot/libdovecot.so.0(backtrace_append+0x3d) [0x7f8bd1f0c85d] -> /usr/lib/dovecot/libdovecot.so.0(backtrace_get+0x1e) [0x7f8bd1f0c97e] -> /usr/lib/dovecot/libdovecot.so.0(+0x10091b) [0x7f8bd1f1991b] -> /usr/lib/dovecot/libdovecot.so.0(+0x1009b1) [0x7f8bd1f199b1] -> /usr/lib/dovecot/libdovecot.so.0(+0x54b7c) [0x7f8bd1e6db7c] -> /usr/lib/dovecot/libdovecot.so.0(+0x4d208) [0x7f8bd1e66208] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_set_current+0x75) [0x7f8bd1f2eca5] -> /usr/lib/dovecot/modules/auth/libdriver_pgsql.so(+0x4686) [0x7f8bd1b43686] -> /usr/lib/dovecot/modules/auth/libdriver_pgsql.so(+0x49b7) [0x7f8bd1b439b7] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_call_io+0x69) [0x7f8bd1f2fe59] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0x131) [0x7f8bd1f31481] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0x4c) [0x7f8bd1f2fefc] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_run+0x40) [0x7f8bd1f30080] -> /usr/lib/dovecot/modules/auth/libdriver_pgsql.so(+0x57ec) [0x7f8bd1b447ec] -> /usr/lib/dovecot/modules/auth/libdriver_pgsql.so(+0x405b) [0x7f8bd1b4305b] -> dovecot/auth(+0x4728b) [0x649266d1628b] -> dovecot/auth(+0x36ca1) [0x649266d05ca1] -> dovecot/auth(userdb_init+0x1a) [0x649266d0370a] -> dovecot/auth(auths_init+0xc9) [0x649266ce4ce9] -> dovecot/auth(main+0x398) [0x649266ce3b78] -> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xeb) [0x7f8bd1bc709b] -> dovecot/auth(_start+0x2a) [0x649266ce3d8a]Jul 18 15:29:21 mail01 dovecot: master: Error: service(auth): command startup failed, throttling for 2.000 secsJul 18 15:29:21 mail01 dovecot: auth: Fatal: master: service(auth): child 12988 killed with signal 6 (core dumps disabled - https://dovecot.org/bugreport.html#coredumps) ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
Re: Replication going away?
Just to understand that correctly: I could setup a (cron) based process for doveadm sync, but no longer a setup like plugin { mail_replica = tcp:$IMAP_REPLICA_SERVER:$IMAP_REPLICA_PORT } where the cron would lead to some delay and would have to check for concurrent jobs? ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
Re: Fatal error in http-client.c when passdb is oauth2 and userdb is sql
> On 18/07/2023 15:10 EEST Евгений Бахтин wrote: > > > > Dovecot version: 2.3.20 > > Dear Colleagues! > passdb is set to oauth2 and userdb is set to sql. On attempt of authorization > a fatal error is rising. The case reproducible 100%. The following suggested > configuration files allow you to reproduce the case. > Thank you! > Can you send auth_debug=yes and debug=yes (in oauth2 config) logs? Aki ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
Fatal error in http-client.c when passdb is oauth2 and userdb is sql
Dovecot version: 2.3.20 Dear Colleagues!passdb is set to oauth2 and userdb is set to sql. On attempt of authorization a fatal error is rising. The case reproducible 100%. The following suggested configuration files allow you to reproduce the case.Thank you! dovecot.conf```protocols = imapauth_mechanisms = plain xoauth2 oauthbearerdisable_plaintext_auth = no passdb { driver = oauth2 mechanisms = xoauth2 oauthbearer args = /etc/dovecot/oauth2.conf.ext} userdb { driver = sql args = /etc/dovecot/userdb.conf.ext} mail_home=/home/test/%Lumail_location=sdbox:~/Mail namespace { inbox = yes separator = /}``` /etc/dovecot/oauth2.conf.ext```introspection_mode = postintrospection_url = https://admin:admin@authserver:9443/oauth2/introspectusername_attribute = usernameactive_attribute = activeactive_value = truetls_allow_invalid_cert = yes``` /etc/dovecot/userdb.conf.ext```driver = pgsqlconnect = host=dbhost port=5432 dbname=test user=test password=testuser_query = SELECT 'something' AS something``` Error log:```Jul 18 13:04:51 mail01 dovecot: auth: Panic: file http-client.c: line 646 (http_client_context_close): assertion failed: (cctx->clients_list == NULL)Jul 18 13:04:51 mail01 dovecot: auth: Error: Raw backtrace: /usr/lib/dovecot/libdovecot.so.0(backtrace_append+0x3d) [0x733f846a085d] -> /usr/lib/dovecot/libdovecot.so.0(backtrace_get+0x1e) [0x733f846a097e] -> /usr/lib/dovecot/libdovecot.so.0(+0x10091b) [0x733f846ad91b] -> /usr/lib/dovecot/libdovecot.so.0(+0x1009b1) [0x733f846ad9b1] -> /usr/lib/dovecot/libdovecot.so.0(+0x54b7c) [0x733f84601b7c] -> /usr/lib/dovecot/libdovecot.so.0(+0x4d208) [0x733f845fa208] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_set_current+0x75) [0x733f846c2ca5] -> /usr/lib/dovecot/modules/auth/libdriver_pgsql.so(+0x4686) [0x733f842d7686] -> /usr/lib/dovecot/modules/auth/libdriver_pgsql.so(+0x49b7) [0x733f842d79b7] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_call_io+0x69) [0x733f846c3e59] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0x131) [0x733f846c5481] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0x4c) [0x733f846c3efc] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_run+0x40) [0x733f846c4080] -> /usr/lib/dovecot/modules/auth/libdriver_pgsql.so(+0x57ec) [0x733f842d87ec] -> /usr/lib/dovecot/modules/auth/libdriver_pgsql.so(+0x405b) [0x733f842d705b] -> dovecot/auth(+0x4728b) [0x5a35cac1128b] -> dovecot/auth(+0x36ca1) [0x5a35cac00ca1] -> dovecot/auth(userdb_init+0x1a) [0x5a35cabfe70a] -> dovecot/auth(auths_init+0xc9) [0x5a35cabdfce9] -> dovecot/auth(main+0x398) [0x5a35cabdeb78] -> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xeb) [0x733f8435b09b] -> dovecot/auth(_start+0x2a) [0x5a35cabded8a]Jul 18 13:04:51 mail01 dovecot: master: Error: service(auth): command startup failed, throttling for 32.000 secsJul 18 13:04:51 mail01 dovecot: auth: Fatal: master: service(auth): child 32434 killed with signal 6 (core dumps disabled - https://dovecot.org/bugreport.html#coredumps)```___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
Re: cram-md5, sasl and lua (inheriting technical debt)
> On 18/07/2023 13:59 EEST tk...@tunenet.dk wrote: > > > Hello, > > I am exploring the posibility of migrating an exsisting setup to > postfix+dovecot. > The issue being that many clients are currently configured for cram-md5 > authentication. > I am fully aware that this is a really, really, really bad idea, but re > configuring all clients at once is not feasible with limited end user support > resources. > I have a setup running with LUA for the passdb, and everything works with > PLAIN login. > > To keep compatibility with the PLAIN login mechanism i have tried to store > {PLAIN} passwords in the DB, since proper secure password storage > is incompatible with CRAM-MD5. > > My issue is that the LUA function auth_password_verify(req, pass) not even > seems to be called for cram-md5 logins. > Reading through the documentation also seems to indicate that the callenge is > not passed to the LUA function making it impossible to compute the hash > in LUA or the function req.password_verify(req, row.password, pass). > > Is my assumption correct that cram-md5 can not work with a LUA script ? > > Kind regards, > Peter K. Hi! CRAM-MD5 only works if you return a credential, so you need to use auth_passdb_lookup to return it with {PLAIN} prefix. It's not possible to use Lua script to calculate it yourself. There is also CRAM-MD5 password scheme, but calculating that is slightly difficult, so i'd just return the PLAIN password from auth_passdb_lookup instead. Aki Aki ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
cram-md5, sasl and lua (inheriting technical debt)
Hello, I am exploring the posibility of migrating an exsisting setup to postfix+dovecot. The issue being that many clients are currently configured for cram-md5 authentication. I am fully aware that this is a really, really, really bad idea, but re configuring all clients at once is not feasible with limited end user support resources. I have a setup running with LUA for the passdb, and everything works with PLAIN login. To keep compatibility with the PLAIN login mechanism i have tried to store {PLAIN} passwords in the DB, since proper secure password storage is incompatible with CRAM-MD5. My issue is that the LUA function auth_password_verify(req, pass) not even seems to be called for cram-md5 logins. Reading through the documentation also seems to indicate that the callenge is not passed to the LUA function making it impossible to compute the hash in LUA or the function req.password_verify(req, row.password, pass). Is my assumption correct that cram-md5 can not work with a LUA script ? Kind regards, Peter K. ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
Re: Tons of imap-login processes despite client_limit very high
It seems to us that the ideal solution would be that once service_count is reached, a new process is spawned and the remaining connections are moved to that new process so that the old one can die quickly. But I suspect that's not a simple change to do. ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
Re: Tons of imap-login processes despite client_limit very high
> On 18/07/2023 12:11 EEST D D wrote: > > > After further testing we realized that it was due to service_count = 100. We > suspect that when the service count is reached, a new process is spawned, > explaining the large number of imap-login processes. > > With service_count = 0 we stick with only 4 processes (process_min_avail). > However, we're concerned with having these processes run indefinitely in case > of memory leaks. > > Any insights on that? It's unlikely that they leak memory, but in case that worries, you can do two things: configure vsz_limit for the service(s), or configure service_count for the service(s). The first one will slay the login processes, disconnecting everyone, if there is a leak, the second one will recycle the process at some point. Since 2.3.19, you can also use https://doc.dovecot.org/settings/core/#core_setting-process_shutdown_filter to shutdown the process if it's memory consumption goes too high. Aki ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
Re: Tons of imap-login processes despite client_limit very high
Thank you Joseph and Aki! You got it right, the issue was indeed with this service_count=100. With service_count=0 it works as intended (only 4 imap-login processes), though now we're concerned about possible memory leaks with this config. What you described Jospeh (https://www.mail-archive.com/dovecot%40dovecot.org/msg85850.html) is what we've observed as well. In addition, service_count > 1 + high process_limit consumes much more mermory because of all those imap-login processes handling juste a few lasting connections. We're consuming about 4x less memory with service_count=0, it's day and night. There's something somewhat close documented on https://doc.dovecot.org/configuration_manual/service_configuration/#service-limits: "Otherwise when the service_count is beginning to be reached, the total number of available connections will shrink. With very bad luck that could mean that all the processes are simply waiting for the existing connections to die away before the process can die and a new one can be created. " Though not the focus of the discussion, it does say that processes don't die until their connections have died. It could perhaps benefit from mentioning a few more things like: - service_count = 0 has no protection against potential memory leaks. - service_count > 1 + high process_limit coud produce many processes since these don't actually die until their connections have all died, which consumes isgnificantly more memory. One workaround to the lack of memory leaks protection could be to set process_limit close to process_min_avail while keeping service_count > 1. But we'd end up in the risky case described in the docs: "With very bad luck that could mean that all the processes are simply waiting for the existing connections to die away before the process can die and a new one can be created." So for now we don't see any way out of service_count = 0 and it's associated memory leak risk. ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
Re: Tons of imap-login processes despite client_limit very high
After further testing we realized that it was due to service_count = 100. We suspect that when the service count is reached, a new process is spawned, explaining the large number of imap-login processes. With service_count = 0 we stick with only 4 processes (process_min_avail). However, we're concerned with having these processes run indefinitely in case of memory leaks. Any insights on that? ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
Re: Tons of imap-login processes despite client_limit very high
> On 18/07/2023 09:18 EEST Joseph Tam wrote: > > > https://www.mail-archive.com/dovecot%40dovecot.org/msg85850.html > > > From: D D > > > We're seeing a ton of imap-login processes running even when using high > > performance mode > > (https://doc.dovecot.org/admin_manual/login_processes/#high-performance-mode). > > According to the docs: > > > > "process_min_avail should be set to be at least the number of CPU cores in > > the system, so that all of them will be used. Otherwise new processes are > > created only once an existing one’s connection count reaches client_limit" > > > > We have process_min_avail=4, client_limit=0 and > > default_client_limit=20. So we'd expect seeing only 4 imap-login > > processes serving a ton of connections each. Yet, we see thousands of > > imap-login processes (more than half of all the imap processes): > > ... > > > > Is having so many imap-login processes normal with our config? Did we > > misunderstand the docs or is there something wrong here? > > > > > > default_client_limit = 1048576 > > default_process_limit = 20 > > > > service imap-login { > > # client_limit = 0 # default is 0 > > # process_limit = 0 # default is 0 > > service_count = 100 > > This service limit might be your culprit. > > I wrote about the strange interaction between service_count and > process_limit here: > > https://www.mail-archive.com/dovecot%40dovecot.org/msg85850.html > > This gotcha should really be documented. > > Joseph Tam Did you check the https://doc.dovecot.org/configuration_manual/service_configuration/#service-limits to see if it is documented? A pull request would be appreciated if it's still wrong. Aki ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
Re: Tons of imap-login processes despite client_limit very high
https://www.mail-archive.com/dovecot%40dovecot.org/msg85850.html From: D D > We're seeing a ton of imap-login processes running even when using high > performance mode > (https://doc.dovecot.org/admin_manual/login_processes/#high-performance-mode). > According to the docs: > > "process_min_avail should be set to be at least the number of CPU cores in > the system, so that all of them will be used. Otherwise new processes are > created only once an existing one’s connection count reaches client_limit" > > We have process_min_avail=4, client_limit=0 and default_client_limit=20. > So we'd expect seeing only 4 imap-login processes serving a ton of > connections each. Yet, we see thousands of imap-login processes (more than > half of all the imap processes): > ... > > Is having so many imap-login processes normal with our config? Did we > misunderstand the docs or is there something wrong here? > > > default_client_limit = 1048576 > default_process_limit = 20 > > service imap-login { > # client_limit = 0 # default is 0 > # process_limit = 0 # default is 0 > service_count = 100 This service limit might be your culprit. I wrote about the strange interaction between service_count and process_limit here: https://www.mail-archive.com/dovecot%40dovecot.org/msg85850.html This gotcha should really be documented. Joseph Tam ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org