Re: Timeout when opening folder

[John Stoffel via dovecot]

> "Entrepreneur" == Entrepreneur AJ via dovecot  
> writes:

Ok so does it work moving mails to a new folder or not?  The lots
are showing a bunch of useless (to this problem of saving mails to a
new folder) SSL: issues.  

The rest of the log lines look good, since you can look up users in
the userdb.  But have you been able to fix your problem?  

> Applied John's recommendation and getting less in the logs now but
> here is the full log from fresh pod start (confirmed to be a single
> pod deployment only. Personal IP MODIFIED:



> [eaj@lpt1 ~]$ k logs -f pod/dovecot-86c75498c8-hdqtr
> Jun 26 09:32:10 master: Info: Dovecot v2.3.21 (47349e2482) starting up for 
> imap, lmtp
> Jun 26 09:32:38 imap-login: Debug: SSL: where=0x10, ret=1: before SSL 
> initialization
> Jun 26 09:32:38 imap-login: Debug: SSL: where=0x2001, ret=1: before SSL 
> initialization
> Jun 26 09:32:38 imap-login: Debug: SSL: where=0x2002, ret=-1: before SSL 
> initialization
> Jun 26 09:32:38 auth: Debug: Loading modules from directory: 
> /usr/lib/dovecot/auth
> Jun 26 09:32:38 imap-login: Debug: SSL: where=0x2001, ret=1: before SSL 
> initialization
> Jun 26 09:32:38 auth: Debug: Module loaded: 
> /usr/lib/dovecot/auth/lib20_auth_var_expand_crypt.so
> Jun 26 09:32:38 auth: Debug: Module loaded: 
> /usr/lib/dovecot/auth/libdriver_pgsql.so
> Jun 26 09:32:38 imap-login: Debug: SSL: where=0x2001, ret=1: SSLv3/TLS read 
> client hello
> Jun 26 09:32:38 auth: Debug: sqlpool(pgsql): Creating new connection
> Jun 26 09:32:38 imap-login: Debug: SSL: where=0x2001, ret=1: SSLv3/TLS write 
> server hello
> Jun 26 09:32:38 imap-login: Debug: SSL: where=0x2001, ret=1: SSLv3/TLS write 
> change cipher spec
> Jun 26 09:32:38 imap-login: Debug: SSL: where=0x2001, ret=1: TLSv1.3 write 
> encrypted extensions
> Jun 26 09:32:38 imap-login: Debug: SSL: where=0x2001, ret=1: SSLv3/TLS write 
> certificate
> Jun 26 09:32:38 auth: Debug: Wrote new auth token secret to 
> /var/run/dovecot//auth-token-secret.dat
> Jun 26 09:32:38 imap-login: Debug: SSL: where=0x2001, ret=1: TLSv1.3 write 
> server certificate verify
> Jun 26 09:32:38 imap-login: Debug: SSL: where=0x2001, ret=1: SSLv3/TLS write 
> finished
> Jun 26 09:32:38 imap-login: Debug: SSL: where=0x2001, ret=1: TLSv1.3 early 
> data
> Jun 26 09:32:38 imap-login: Debug: SSL: where=0x2002, ret=-1: TLSv1.3 early 
> data
> Jun 26 09:32:38 imap-login: Debug: SSL: where=0x2002, ret=-1: TLSv1.3 early 
> data
> Jun 26 09:32:38 auth: Debug: sqlpool(pgsql): Creating new connection
> Jun 26 09:32:38 auth: Debug: auth client connected (pid=10)
> Jun 26 09:32:38 imap-login: Debug: SSL: where=0x2002, ret=-1: TLSv1.3 early 
> data
> Jun 26 09:32:38 imap-login: Debug: SSL: where=0x2002, ret=-1: TLSv1.3 early 
> data
> Jun 26 09:32:38 imap-login: Debug: SSL: where=0x2001, ret=1: TLSv1.3 early 
> data
> Jun 26 09:32:38 imap-login: Debug: SSL: where=0x2001, ret=1: SSLv3/TLS read 
> finished
> Jun 26 09:32:38 imap-login: Debug: SSL: where=0x20, ret=1: SSLv3/TLS write 
> session ticket
> Jun 26 09:32:38 imap-login: Debug: SSL: where=0x2001, ret=1: SSLv3/TLS write 
> session ticket
> Jun 26 09:32:38 imap-login: Debug: SSL: where=0x2001, ret=1: SSLv3/TLS write 
> session ticket
> Jun 26 09:32:38 imap-login: Debug: SSL: where=0x2002, ret=1: SSL negotiation 
> finished successfully
> Jun 26 09:32:38 auth: Debug: client in: AUTH  1   PLAIN   service=imap
> secured=tls session=qx70sccb5tfCz3qglip=10.244.24.239   
> rip=194.207.0.0 lport=993   rport=55270 local_name=imap.eajglobal.net
> Jun 26 09:32:38 auth: Debug: client passdb out: CONT  1   
> Jun 26 09:32:38 auth: Debug: client in: CONT
> Jun 26 09:32:38 auth: Debug: 
> sql(e...@eajglobal.com,194.207.0.0,): Performing passdb 
> lookup
> Jun 26 09:32:38 auth: Debug: 
> sql(e...@eajglobal.com,194.207.0.0,): query: SELECT userid 
> as user, password, '/srv/vmail/eaj' as userdb_home, 'maildir:/srv/vmail/eaj' 
> as userdb_mail, 1000 as  userdb_uid, 1000 as userdb_gid FROM mailboxes WHERE 
> userid = 'eaj' AND deleted_at IS NULL
> Jun 26 09:32:38 auth: Debug: pgsql(postgres-primary.postgres.svc): Finished 
> query 'SELECT userid as user, password, '/srv/vmail/eaj' as userdb_home, 
> 'maildir:/srv/vmail/eaj' as userdb_mail, 1000 as  userdb_uid, 1000 as 
> userdb_gid FROM mailboxes WHERE userid = 'eaj' AND deleted_at IS NULL' in 2 
> msecs
> Jun 26 09:32:38 auth: Debug: 
> sql(e...@eajglobal.com,194.207.0.0,): username changed 
> e...@eajglobal.com -> eaj
> Jun 26 09:32:38 auth: Debug: sql(eaj,194.207.0.0,): 
> Finished passdb lookup
> Jun 26 09:32:38 auth: Debug: auth(eaj,194.207.0.0,): Auth 
> request finished
> Jun 26 09:32:38 auth: Debug: client passdb out: OK1   user=eaj
> original_user=e...@eajglobal.com
> Jun 26 09:32:38 auth: Debug: master in: REQUEST   2651455489  10  
> 1   3f75659e5b7188588f19d7ed4874cb8asession_pid=13  
> request_auth_token
> Jun 26 09:32:38 auth: Debug: 

Re: Timeout when opening folder

[John Stoffel via dovecot]

> "Entrepreneur" == Entrepreneur AJ via dovecot  
> writes:

> Migrating everything to a k0s kubernetes cluster, trying to migrate
> dovecot and all so far seems to be well with PVC for mail storage,
> ssl working great, authentication vis PostgreSQL working great.

So how do you have your local storage defined in your kubernetes
cluster?  That strikes me (as a total Kubernetes noob) as the possible
issue you're running into.  Your /src/vmail/sr
> Issue is when trying to look at another folder say Deleted or Spam etc 
> folder I am getting a timeout error with no logs.

> I disabled apparmour on the nodes the try again and the following error 
> message appears;

> Jun 24 21:01:55 imap-login: Info: Login: user=, method=PLAIN, 
> rip=194.207.0.0, lip=10.244.24.235, mpid=15, TLS, session=
> Jun 24 21:01:55 imap(eaj)<15>: Debug: Added userdb 
> setting: mail=maildir:/srv/vmail/eaj
> Jun 24 21:01:55 imap(eaj)<15>: Debug: Effective 
> uid=1000, gid=1000, home=/srv/vmail/eaj
> Jun 24 21:01:55 imap(eaj)<15>: Debug: 
> open(/proc/self/io) failed: Permission denied
> Jun 24 21:01:55 imap(eaj)<15>: Debug: Namespace inbox: 
> type=private, prefix=, sep=, inbox=yes, hidden=no, list=yes, 
> subscriptions=yes location=maildir:/srv/vmail/eaj
> Jun 24 21:01:55 imap(eaj)<15>: Debug: maildir++: 
> root=/srv/vmail/eaj, index=, indexpvt=, control=, inbox=/srv/vmail/eaj, alt=
> Jun 24 21:01:55 imap(eaj)<14><3PRZF6kb0JHCz3qg>: Debug: Mailbox INBOX: 
> Mailbox opened
> Jun 24 21:02:24 imap(eaj)<15>: Debug: Mailbox Sent: 
> Couldn't open mailbox in list index: Storage size changed 160 != 396
> Jun 24 21:02:24 imap(eaj)<15>: Debug: Mailbox Sent: 
> Mailbox opened
> Jun 24 21:02:24 imap(eaj)<15>: Debug: Mailbox Drafts: 
> Couldn't open mailbox in list index: Storage size changed 160 != 396
> Jun 24 21:02:24 imap(eaj)<15>: Debug: Mailbox Drafts: 
> Mailbox opened

> Thunderbird keeps showing a timeout error after a few minutes.

> Dovecot Version: 2.3.21 (47349e2482)

> dovecot -n config:
> # 2.3.21 (47349e2482): /etc/dovecot/dovecot.conf
> # OS: Linux 5.15.0-112-generic x86_64
> # Hostname: dovecot-549bdc98ff-tzwcf
> auth_debug = yes
> auth_debug_passwords = yes
> auth_mechanisms = plain login
> auth_verbose = yes
> auth_verbose_passwords = yes
> base_dir = /var/run/dovecot/
> first_valid_uid = 1000
> import_environment = TZ
> last_valid_uid = 1000
> log_path = /dev/stdout
> login_greeting = IMAP ready.
> mail_debug = yes
> mail_home = /srv/vmail/%n
> mail_location = maildir:~

I think this is wrong, you really want something like:

  mail_home = 
  mail_location = maildir:/srv/vmail/%d/%n/Maildir

Because you don't have local directories for your users.  You're doing
completely virtual, so I think this is what you want instead.  Also,
for future growth, you might want to add %d (domain) in your path just
in case.  

> namespace inbox {
>    hidden = no
>    inbox = yes
>    list = yes
>    location =
>    mailbox Drafts {
>      auto = subscribe
>      special_use = \Drafts
>    }
>    mailbox Junk {
>      special_use = \Junk
>    }
>    mailbox Sent {
>      auto = subscribe
>      special_use = \Sent
>    }
>    mailbox "Sent Messages" {
>      special_use = \Sent
>    }
>    mailbox Spam {
>      auto = subscribe
>      special_use = \Junk
>    }
>    mailbox Trash {
>      auto = subscribe
>      special_use = \Trash
>    }
>    prefix =
>    subscriptions = yes
>    type = private
> }
> passdb {
>    args = /etc/dovecot/dovecot-sql.conf.ext
>    driver = sql
> }
> plugin {
>    acl = vfile:/etc/dovecot/global-acls:cache_secs=300
>    acl_shared_dict = file:/var/lib/dovecot/shared-mailboxes
>    imap_compress_deflate_level = 9
>    mail_crypt_global_private_key = # hidden, use -P to show it
>    mail_crypt_global_public_key = # hidden, use -P to show it
>    mail_crypt_save_version = 2
>    zlib_save = zstd
>    zlib_save_level = 3
> }
> service auth-worker {
>    user = vmail
> }
> service auth {
>    unix_listener auth-userdb {
>      group = vmail
>      mode = 0600
>      user = vmail
>    }
> }
> service dict {
>    unix_listener dict {
>      group = vmail
>      mode = 0660
>      user = vmail
>    }
> }
> service imap-login {
>    inet_listener imap {
>      port = 143
>    }
>    inet_listener imaps {
>      port = 993
>      ssl = yes
>    }
> }
> service lmtp {
>    inet_listener lmtp {
>      port = 24
>    }
>    user = vmail
> }
> service pop3-login {
>    inet_listener pop3 {
>      port = 0
>    }
>    inet_listener pop3s {
>      port = 0
>      ssl = yes
>    }
> }
> service submission-login {
>    inet_listener submission {
>      port = 0
>    }
>    inet_listener submissions {
>      port = 0
>    }
> }
> ssl = required
> ssl_cert =  ssl_cipher_list = 
> 

Re: Sieve generate a lot of hard link copies of mails in mailboxes

[John Stoffel via dovecot]

> "George" == George Asenov via dovecot  writes:

I don't have a real suggestion, but I do think you can clarify your problem.

> No one have idea what is wrong here?
> On 07-Jun-24 4:10 PM, George Asenov via dovecot wrote:
>> Hello,
>> 
>> I have very strange issue. Sieve generate copies of users messages i.e. 
>> not real copies but hardlinks for the same message. It happens to many 
>> messages but not every message and not every time, it is not a single 
>> user issue I have couple users with that issue.

Are you expecting sieve to generate copies?  And are the copies in the
same folder or across folders?  I.e. do you find an email in the
INBOX, and a hardlink in the SPAM folder?

What are the sizes of these emails?  Are they all large?  Or have
attachements?  Is there anything that's common amongst those emails?

One idea might be to setup a test account and to just send it a bunch
of emails to try and make the problem occur.  And to also look closely
at the rspamd logs as well.  

What is the size of the system memory on your dovecot server?  And
what is the size of the dovecot.index.cache file when you see this
error?  You should be able to delete the index and recreate it using
doveadm.  

But from the sound of it, you have users with many thousands of emails
in a folder or folders.  Can you check to see if there's any
relationship between users with larger numbers of hardlinks and those
with large numbers of emails?  

And maybe instead of having sieve call rspam, maybe you can put it
into a milter and just have the 


>> It happens during auto reporting for spam/ham with sieve.
>> But I'm unable to reproduce it.
>> 
>> At some point the hardlinks copies become so many that the mailbox index 
>> files become so bug that dovecot start throwing error:
>> 
>> dovecot[3385911]: imap(redac...@domain.tld)<1992901>: 
>> Error: Mailbox Junk: mmap(size=520636784) failed with file 
>> /var/lib/dovecot-virtualmin/index/redac...@domain.tld/.Junk/dovecot.index.cache:
>>  Cannot allocate memory
>> 
>> other relevant logs are:
>> 
>> dovecot: imap-login: Login: user=, method=PLAIN, 
>> rip=YYY.YYY.YYY.YYY, lip=XXX.XXX.XXX.XXX, mpid=3393763, TLS, 
>> session=
>> dovecot: imap(redacted.user)<3393763>: sieve: DEBUG: 
>> learn-spam.sieve was triggered on imap.cause=COPY: 
>> msgid=<87584056G78841203D85243127W62181551P@idomziqnd>
>> dovecot: imap(redacted.user)<3393763>: sieve: DEBUG: 
>> learn-spam on imap.cause=COPY: from=redacted.mail, to=redacted2.mail, 
>> subject=Asseyez-vous confortablement, n'importe où..., 
>> msgid=<87584056G78841203D85243127W62181551P@idomziqnd>, 
>> X-Spamd-Result=default: False [4.49 / 15.00]; 
>> FORGED_RECIPIENTS(2.00)[m:redacted2.mail,s:redacted.user.fr]; 
>> BAYES_SPAM(1.89)[88.30%]; MID_RHS_NOT_FQDN(0.50)[]; 
>> BAD_REP_POLICIES(0.10)[]; RCVD_NO_TLS_LAST(0.10)[]; 
>> MIME_GOOD(-0.10)[multipart/related,multipart/alternative,text/plain]; 
>> ASN(0.00)[asn:34300, ipnet:62.173.128.0/19, country:RU]; 
>> RCVD_COUNT_ONE(0.00)[1]; MIME_TRACE(0.00)[0:+,1:+,2:+,3:~,4:~,5:+]; 
>> RCPT_COUNT_ONE(0.00)[1]; MISSING_XM_UA(0.00)[]; ARC_NA(0.00)[]; 
>> RCVD_VIA_SMTP_AUTH(0.00)[]; GREYLIST(0.00)[pass,body]; 
>> R_DKIM_NA(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; FROM_HAS_DN(0.00)[]; 
>> R_SPF_ALLOW(0.00)[+mx]; TO_DN_NONE(0.00)[]; DMARC_NA(0.00)[or.mg]; 
>> NEURAL_SPAM(0.00)[0.000]
>> dovecot: imap(redacted.user)<3393763>: sieve: DEBUG: 
>> learn-spam send to rspamd spam
>> dovecot: imap(redacted.user)<3393763>: program 
>> exec:/var/lib/dovecot/sieve/rspamd-learn-spam.sh (3397238): Terminated 
>> with non-zero exit code 1
>> dovecot: imap(redacted.user)<3393763>: Error: sieve: 
>> failed to execute to program `rspamd-learn-spam.sh': refer to server log 
>> for more information. [2024-06-03 07:36:40]
>> dovecot: imap(redacted.user)<3393763>: Disconnected: 
>> Connection closed (UID FETCH finished 32.173 secs ago) in=2914 out=39237 
>> deleted=1 expunged=1 trashed=0 hdr_count=14 hdr_bytes=10705 body_count=1 
>> body_bytes=1606
>> 
>> I know that this is because the mail which is reported is too big for 
>> curl but documentation say that

Wait, how large is this email you're trying to process?  So once you
have rspamd-learn-spam.sh crash on you, then you are really having an
rspam problem.  Do you really need to scan large attachements?  

What is your rspam configuration?  And have you talked to people on
the rspamd mailing list on how to configure things?  


>> $
>> pipe :copy :try "rspamd-learn-spam.sh";
>> $
>> this should ignore the error.
>> I have tested also to change it like that:
>> $
>> pipe :copy  "rspamd-learn-spam.sh";
>> $
>> but the issue still persists

So why are you doing a :copy here?  If you're trying to say this email
is spam, why not just move it to your spam folder, and then have
rspamd go through your junk folder once a day instead?  

Have you looked 

Re: Uppercase username emails are rejected

[John Stoffel via dovecot]

>>>>> "Peter" == Peter via dovecot  writes:

> On 14/04/24 12:09, John Stoffel via dovecot wrote:
>> I think you need to update both places, so that your username and
>> password checks are done with lowercase usernames.

> Generally speaking you want auth to be case-sensitive, but go ahead and 
> try it to see if it fixes the issue.

Umm... not for emails you don't.  Since the j...@stoffel.org and
j...@stoffel.org and j...@stoffel.org are all the same email
address... should they be different logins?  Not for email... 

In general, usernames should NOT be case sensitive, that way leads
madness.  Passwords on the other hand...
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: Uppercase username emails are rejected

[John Stoffel via dovecot]

> "lua8ds---" == lua8ds--- via dovecot  writes:

> Updating:
>   args = username_format=%Ln

> under userdb did not fix my issue.

> My server still rejects incoming emails with uppercase username e.g. 
> USERNAME@tld.

> In /etc/dovecot/conf.d/auth-passwd-file.conf.ext, before I updated with your 
> suggestion i.e. %Ln, I had:

>   passdb {
> driver = passwd-file
> args = scheme=CRYPT username_format=%u /etc/dovecot/users
>   }

>   userdb {
> driver = passwd-file
> args = username_format=%u /etc/dovecot/users

> I updated to 

>   passdb {
> driver = passwd-file
> args = scheme=CRYPT username_format=%u /etc/dovecot/users
>   }

I think you need to update both places, so that your username and
password checks are done with lowercase usernames.

>   userdb {
> driver = passwd-file
> args = username_format=%Ln /etc/dovecot/users

> FYI after I updated and tested sending an email with uppercase username 
> again, I ran

> # systemctl reload dovecot postfix

> also 

> # dovecot reload

> What else can I try?

> I tried  args = username_format=%Lu too, in vain.

> PS thanks for pointing that I should look at  username_format rather than 
> auth_ username_format

> # dovecot -n
> # 2.3.7.2 (3c910f64b): /etc/dovecot/dovecot.conf
> # Pigeonhole version 0.5.7.2 ()
> # OS: Linux 5.4.0-156-generic x86_64 Trisquel GNU/Linux 10.0.1, Nabia 
> # Hostname: mail.redacted_tld
> auth_mechanisms = plain login
> auth_username_format = %n
> mail_location = maildir:~/Maildir
> mail_privileged_group = mail
> namespace inbox {
>   inbox = yes
>   location = 
>   mailbox Drafts {
> auto = create
> special_use = \Drafts
>   }
>   mailbox Junk {
> auto = create
> autoexpunge = 30 days
> special_use = \Junk
>   }
>   mailbox Sent {
> special_use = \Sent
>   }
>   mailbox Trash {
> auto = create
> special_use = \Trash
>   }
>   prefix = 
> }
> passdb {
>   driver = pam
> }
> protocols = imap lmtp imap lmtp
> service auth {
>   unix_listener /var/spool/postfix/private/auth {
> group = postfix
> mode = 0660
> user = postfix
>   }
> }
> service lmtp {
>   unix_listener /var/spool/postfix/private/dovecot-lmtp {
> group = postfix
> mode = 0600
> user = postfix
>   }
> }
> ssl = required
> ssl_cert =  ssl_client_ca_dir = /etc/ssl/certs
> ssl_dh = # hidden, use -P to show it
> ssl_key = # hidden, use -P to show it
> ssl_min_protocol = TLSv1.2
> ssl_prefer_server_ciphers = yes
> userdb {
>   driver = passwd
> }
> ___
> dovecot mailing list -- dovecot@dovecot.org
> To unsubscribe send an email to dovecot-le...@dovecot.org
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: doveadm user '*' not working, virtual users only with sqlite

[John Stoffel]

>>>>> "Aki" == Aki Tuomi via dovecot  writes:

That did the trick.  Thanks!  Maybe the docs can be updated to make
this crystal clear?  Especially the section which talks about using
the static driver with userdb and how it doesn't allow the use of
interate_query.  

> iterate_query only works with
> userdb {
>   driver = sql
>   args = /path/to/auth-sql.conf.ext
> }

> Aki

>> On 02/12/2023 00:02 EET John Stoffel  wrote:
>> 
>> 
>> >>>>> "John" == John Stoffel  writes:
>> 
>> Do I think I'm on the right track here, since I removed the following
>> from /etc/dovecot/conf.d/auth-sql.conf.ext
>> 
>> #userdb {
>> #  driver = static
>> #  args = uid=mail gid=mail home=/var/mail/%d/%n
>> #}
>> 
>> So now my error is as follows:
>> 
>> # doveadm user -u '*'
>> Error: auth-master: userdb list: User listing returned failure
>> Fatal: user listing failed
>> 
>> Because now when I restart dovecot, I see the following in the log:
>> 
>> Dec 01 16:55:14 master: Info: Dovecot v2.3.21 (47349e2482) starting up
>> for imap, lmtp, sieve (core dumps disabled)
>> Dec 01 16:55:14 auth: Warning: sql: Ignoring changed iterate_query in
>> /etc/dovecot/dovecot-sql.conf.ext, because userdb sql not used. (If
>> this is intentional, set userdb_warning_disable=yes)
>> Dec 01 16:55:14 auth: Error: auth-master client: Trying to iterate
>> users, but userdbs don't support it (created 0 msecs ago, handshake 0
>> msecs ago)
>> 
>> So I commented out my 'iterate_query = ...' (see below) from
>> /etc/postfix/dovecot-sql.conf.ext and now I get the error on startup
>> which says:
>> 
>> Dec 01 16:57:42 master: Info: Dovecot v2.3.21 (47349e2482) starting up
>> for imap, lmtp, sieve (core dumps disabled)
>> Dec 01 16:57:42 auth: Error: auth-master client: Trying to iterate
>> users, but userdbs don't support it (created 0 msecs ago, handshake 0
>> msecs ago)
>> Dec 01 16:57:42 replicator: Error: auth-master: userdb list: User
>> listing returned failure
>> Dec 01 16:57:42 replicator: Error: listing users failed, can't
>> replicate existing data
>> 
>> Which tells me I need the iteracte_users setting, but I've got a bogus
>> query in there.  So I think I should be using something like this:
>> 
>> iterate_query = SELECT email AS user from virtual_users;
>> 
>> where 'virtual_users' is the one and only table in my sqlite db file.
>> And I'm just returning the 'email' column as 'user', since that's what
>> it seems to expect.  
>> 
>> Hmmm...
>> 
>> 
>> > I've been pounding my head against the sand for a while here trying to
>> > figure out why I can't get:
>> 
>> >doveadm user '*' 
>> 
>> > working properly.  I've got a Debian 11 VPS runnig dovecot version
>> > 2.3.21-1+debian10 and it works great.  But not I'm trying to add in
>> > simple replication to a home dovecot instance over a wireguard tunnel
>> > so I can do backups and have a little better resiliency.  Maybe.
>> 
>> > In any case, my sqlite schema looks like this:
>> 
sqlite> .schema virtual_users
>> > CREATE TABLE `virtual_users` (
>> >   `id` integer NOT NULL PRIMARY KEY AUTOINCREMENT
>> > ,  `domain_id` integer NOT NULL
>> > ,  `password` varchar(106) NOT NULL
>> > ,  `email` varchar(100) NOT NULL
>> > ,  UNIQUE (`email`)
>> > ,  CONSTRAINT `virtual_users_ibfk_1` FOREIGN KEY (`domain_id`) 
>> > REFERENCES `virtual_domains` (`id`) E
>> > );
>> > CREATE INDEX "idx_virtual_users_domain_id" ON "virtual_users" 
>> > (`domain_id`);
>> 
>> 
>> > and I don't have any other tables.  The 'domain_id' was/is a leftover
>> > from my thinking I needed it for extra testing of other domains and
>> > such.  
>> 
>> > I can do 'doveadm user j...@stoffel.org' and it works just fine.  When
>> > I do "doveadm user '*'" it fails and I get:
>> 
>> > doveadm user '*'
>> > Error: auth-master: userdb list: User listing returned failure
>> > Fatal: user listing failed
>> 
>> 
>> > So my config looks like this:
>> 
>> >root@mail:/etc/dovecot/conf.d# cat auth-sql.conf.ext
>> ># Authentication for SQL users. Included from 10-auth.conf.
>> >#
>> ># 
>> 
>> >passdb {
>> >  driver = sql
>> 
>> >  # Path for SQ

Re: doveadm user '*' not working, virtual users only with sqlite

[John Stoffel]

>>>>> "Aki" == Aki Tuomi  writes:

> iterate_query only works with
> userdb {
>   driver = sql
>   args = /path/to/auth-sql.conf.ext
> }

Thanks, that was the key part I was missing!  Can you maybe think to
update the code to give a more useful error message, or even a warning
on startup which says something like:  'static driver does not support
iterating users'?   I tried looking at the source code, but it's going
to take me quite a while to wrap my brain around how it's structured
and how error messages propogate.  

Even just listing which userdb block failed would be a help, since you
can have multiple ones defined.  

In any case, I've got it working now once I updated both areas in my
configuration which referred to the 'static' driver.  

Thanks, really appreciate your help and all the work you guys do on
this software!

John


>> On 02/12/2023 00:02 EET John Stoffel  wrote:
>> 
>> 
>> >>>>> "John" == John Stoffel  writes:
>> 
>> Do I think I'm on the right track here, since I removed the following
>> from /etc/dovecot/conf.d/auth-sql.conf.ext
>> 
>> #userdb {
>> #  driver = static
>> #  args = uid=mail gid=mail home=/var/mail/%d/%n
>> #}
>> 
>> So now my error is as follows:
>> 
>> # doveadm user -u '*'
>> Error: auth-master: userdb list: User listing returned failure
>> Fatal: user listing failed
>> 
>> Because now when I restart dovecot, I see the following in the log:
>> 
>> Dec 01 16:55:14 master: Info: Dovecot v2.3.21 (47349e2482) starting up
>> for imap, lmtp, sieve (core dumps disabled)
>> Dec 01 16:55:14 auth: Warning: sql: Ignoring changed iterate_query in
>> /etc/dovecot/dovecot-sql.conf.ext, because userdb sql not used. (If
>> this is intentional, set userdb_warning_disable=yes)
>> Dec 01 16:55:14 auth: Error: auth-master client: Trying to iterate
>> users, but userdbs don't support it (created 0 msecs ago, handshake 0
>> msecs ago)
>> 
>> So I commented out my 'iterate_query = ...' (see below) from
>> /etc/postfix/dovecot-sql.conf.ext and now I get the error on startup
>> which says:
>> 
>> Dec 01 16:57:42 master: Info: Dovecot v2.3.21 (47349e2482) starting up
>> for imap, lmtp, sieve (core dumps disabled)
>> Dec 01 16:57:42 auth: Error: auth-master client: Trying to iterate
>> users, but userdbs don't support it (created 0 msecs ago, handshake 0
>> msecs ago)
>> Dec 01 16:57:42 replicator: Error: auth-master: userdb list: User
>> listing returned failure
>> Dec 01 16:57:42 replicator: Error: listing users failed, can't
>> replicate existing data
>> 
>> Which tells me I need the iteracte_users setting, but I've got a bogus
>> query in there.  So I think I should be using something like this:
>> 
>> iterate_query = SELECT email AS user from virtual_users;
>> 
>> where 'virtual_users' is the one and only table in my sqlite db file.
>> And I'm just returning the 'email' column as 'user', since that's what
>> it seems to expect.  
>> 
>> Hmmm...
>> 
>> 
>> > I've been pounding my head against the sand for a while here trying to
>> > figure out why I can't get:
>> 
>> >doveadm user '*' 
>> 
>> > working properly.  I've got a Debian 11 VPS runnig dovecot version
>> > 2.3.21-1+debian10 and it works great.  But not I'm trying to add in
>> > simple replication to a home dovecot instance over a wireguard tunnel
>> > so I can do backups and have a little better resiliency.  Maybe.
>> 
>> > In any case, my sqlite schema looks like this:
>> 
sqlite> .schema virtual_users
>> > CREATE TABLE `virtual_users` (
>> >   `id` integer NOT NULL PRIMARY KEY AUTOINCREMENT
>> > ,  `domain_id` integer NOT NULL
>> > ,  `password` varchar(106) NOT NULL
>> > ,  `email` varchar(100) NOT NULL
>> > ,  UNIQUE (`email`)
>> > ,  CONSTRAINT `virtual_users_ibfk_1` FOREIGN KEY (`domain_id`) 
>> > REFERENCES `virtual_domains` (`id`) E
>> > );
>> > CREATE INDEX "idx_virtual_users_domain_id" ON "virtual_users" 
>> > (`domain_id`);
>> 
>> 
>> > and I don't have any other tables.  The 'domain_id' was/is a leftover
>> > from my thinking I needed it for extra testing of other domains and
>> > such.  
>> 
>> > I can do 'doveadm user j...@stoffel.org' and it works just fine.  When
>> > I do "doveadm user '*'" it fails and I get:
>> 
>> > doveadm user '*'
>> > Error: auth-ma

Re: doveadm user '*' not working, virtual users only with sqlite

[John Stoffel]

>>>>> "John" == John Stoffel  writes:

Do I think I'm on the right track here, since I removed the following
from /etc/dovecot/conf.d/auth-sql.conf.ext

#userdb {
#  driver = static
#  args = uid=mail gid=mail home=/var/mail/%d/%n
#}

So now my error is as follows:

# doveadm user -u '*'
Error: auth-master: userdb list: User listing returned failure
Fatal: user listing failed

Because now when I restart dovecot, I see the following in the log:

Dec 01 16:55:14 master: Info: Dovecot v2.3.21 (47349e2482) starting up
  for imap, lmtp, sieve (core dumps disabled)
Dec 01 16:55:14 auth: Warning: sql: Ignoring changed iterate_query in
  /etc/dovecot/dovecot-sql.conf.ext, because userdb sql not used. (If
  this is intentional, set userdb_warning_disable=yes)
Dec 01 16:55:14 auth: Error: auth-master client: Trying to iterate
  users, but userdbs don't support it (created 0 msecs ago, handshake 0
  msecs ago)

So I commented out my 'iterate_query = ...' (see below) from
/etc/postfix/dovecot-sql.conf.ext and now I get the error on startup
which says:

   Dec 01 16:57:42 master: Info: Dovecot v2.3.21 (47349e2482) starting up
 for imap, lmtp, sieve (core dumps disabled)
   Dec 01 16:57:42 auth: Error: auth-master client: Trying to iterate
 users, but userdbs don't support it (created 0 msecs ago, handshake 0
 msecs ago)
   Dec 01 16:57:42 replicator: Error: auth-master: userdb list: User
 listing returned failure
   Dec 01 16:57:42 replicator: Error: listing users failed, can't
 replicate existing data

Which tells me I need the iteracte_users setting, but I've got a bogus
query in there.  So I think I should be using something like this:

  iterate_query = SELECT email AS user from virtual_users;

where 'virtual_users' is the one and only table in my sqlite db file.
And I'm just returning the 'email' column as 'user', since that's what
it seems to expect.  

Hmmm...


> I've been pounding my head against the sand for a while here trying to
> figure out why I can't get:

>doveadm user '*' 

> working properly.  I've got a Debian 11 VPS runnig dovecot version
> 2.3.21-1+debian10 and it works great.  But not I'm trying to add in
> simple replication to a home dovecot instance over a wireguard tunnel
> so I can do backups and have a little better resiliency.  Maybe.

> In any case, my sqlite schema looks like this:

sqlite> .schema virtual_users
> CREATE TABLE `virtual_users` (
>   `id` integer NOT NULL PRIMARY KEY AUTOINCREMENT
> ,  `domain_id` integer NOT NULL
> ,  `password` varchar(106) NOT NULL
> ,  `email` varchar(100) NOT NULL
> ,  UNIQUE (`email`)
> ,  CONSTRAINT `virtual_users_ibfk_1` FOREIGN KEY (`domain_id`) REFERENCES 
> `virtual_domains` (`id`) E
> );
> CREATE INDEX "idx_virtual_users_domain_id" ON "virtual_users" 
> (`domain_id`);


> and I don't have any other tables.  The 'domain_id' was/is a leftover
> from my thinking I needed it for extra testing of other domains and
> such.  

> I can do 'doveadm user j...@stoffel.org' and it works just fine.  When
> I do "doveadm user '*'" it fails and I get:

> doveadm user '*'
> Error: auth-master: userdb list: User listing returned failure
> Fatal: user listing failed


> So my config looks like this:

>root@mail:/etc/dovecot/conf.d# cat auth-sql.conf.ext
># Authentication for SQL users. Included from 10-auth.conf.
>#
># 

>passdb {
>  driver = sql

>  # Path for SQL configuration file, see
>example-config/dovecot-sql.conf.ext
>  args = /etc/dovecot/dovecot-sql.conf.ext
>}

>userdb {
>  driver = static
>  args = uid=mail gid=mail home=/var/mail/%d/%n
>}

> My /etc/dovecot/dovecot-sql.conf.ext has the following:

>driver = sqlite
>connect = /etc/dovecot/private/virtual_users.sqlite3

>default_pass_scheme = SHA512-CRYPT

>password_query = SELECT '/var/mail/%d/%u' AS userdb_home, 'mail' AS 
> userdb_uid, 'mail' AS userdb_gid, email as user, password FROM virtual_users 
> WHERE email='%u';

>iterate_query = SELECT email AS user from virtual_users;

> And my general doveadm config output is this, slightly edited down to
> remove stuff I don't think I need to show is at the end.  Any hints on
> what I've done wrong here?  Do I need a more complete sqlite3 schema?
> I wish I could get more debugging info on what query it's trying to
> run and the error(s) it's getting.  

> Thanks,
> John



> # 2.3.21 (47349e2482): /etc/dovecot/dovecot.conf
> # Pigeonhole version 0.5.21 (f6cd4b8e)
> # OS: Linux 5.10.0-26-amd64 x86_64 Debian 11.8 ext4
> # Hostname: localhost
> # NOTE: Send doveconf -n output instead when asking for h

doveadm user '*' not working, virtual users only with sqlite

[John Stoffel]

Hi all,
I've been pounding my head against the sand for a while here trying to
figure out why I can't get:

   doveadm user '*' 

working properly.  I've got a Debian 11 VPS runnig dovecot version
2.3.21-1+debian10 and it works great.  But not I'm trying to add in
simple replication to a home dovecot instance over a wireguard tunnel
so I can do backups and have a little better resiliency.  Maybe.

In any case, my sqlite schema looks like this:

sqlite> .schema virtual_users
CREATE TABLE `virtual_users` (
  `id` integer NOT NULL PRIMARY KEY AUTOINCREMENT
,  `domain_id` integer NOT NULL
,  `password` varchar(106) NOT NULL
,  `email` varchar(100) NOT NULL
,  UNIQUE (`email`)
,  CONSTRAINT `virtual_users_ibfk_1` FOREIGN KEY (`domain_id`) REFERENCES 
`virtual_domains` (`id`) E
);
CREATE INDEX "idx_virtual_users_domain_id" ON "virtual_users" (`domain_id`);


and I don't have any other tables.  The 'domain_id' was/is a leftover
from my thinking I needed it for extra testing of other domains and
such.  

I can do 'doveadm user j...@stoffel.org' and it works just fine.  When
I do "doveadm user '*'" it fails and I get:

doveadm user '*'
Error: auth-master: userdb list: User listing returned failure
Fatal: user listing failed


So my config looks like this:

   root@mail:/etc/dovecot/conf.d# cat auth-sql.conf.ext
   # Authentication for SQL users. Included from 10-auth.conf.
   #
   # 

   passdb {
 driver = sql

 # Path for SQL configuration file, see
   example-config/dovecot-sql.conf.ext
 args = /etc/dovecot/dovecot-sql.conf.ext
   }

   userdb {
 driver = static
 args = uid=mail gid=mail home=/var/mail/%d/%n
   }

My /etc/dovecot/dovecot-sql.conf.ext has the following:

   driver = sqlite
   connect = /etc/dovecot/private/virtual_users.sqlite3

   default_pass_scheme = SHA512-CRYPT

   password_query = SELECT '/var/mail/%d/%u' AS userdb_home, 'mail' AS 
userdb_uid, 'mail' AS userdb_gid, email as user, password FROM virtual_users 
WHERE email='%u';

   iterate_query = SELECT email AS user from virtual_users;

And my general doveadm config output is this, slightly edited down to
remove stuff I don't think I need to show is at the end.  Any hints on
what I've done wrong here?  Do I need a more complete sqlite3 schema?
I wish I could get more debugging info on what query it's trying to
run and the error(s) it's getting.  

Thanks,
John



# 2.3.21 (47349e2482): /etc/dovecot/dovecot.conf
# Pigeonhole version 0.5.21 (f6cd4b8e)
# OS: Linux 5.10.0-26-amd64 x86_64 Debian 11.8 ext4
# Hostname: localhost
# NOTE: Send doveconf -n output instead when asking for help.
auth_anonymous_username = anonymous
auth_cache_negative_ttl = 1 hours
auth_cache_size = 0
auth_cache_ttl = 1 hours
auth_cache_verify_password_with_worker = no
auth_debug = no
auth_debug_passwords = no
auth_failure_delay = 2 secs
auth_gssapi_hostname = 
auth_krb5_keytab = 
auth_master_user_separator = 
auth_mechanisms = plain login
auth_policy_check_after_auth = yes
auth_policy_check_before_auth = yes
auth_policy_hash_mech = sha256
auth_policy_hash_nonce = 
auth_policy_hash_truncate = 12
auth_policy_log_only = no
auth_policy_reject_on_fail = no
auth_policy_report_after_auth = yes
auth_policy_request_attributes = login=%{requested_username} 
pwhash=%{hashed_password} remote=%{rip} device_id=%{client_id} protocol=%s 
session_id=%{session}
auth_policy_server_api_header = 
auth_policy_server_timeout_msecs = 2000
auth_policy_server_url = 
auth_proxy_self = 
auth_realms = 
auth_socket_path = auth-userdb
auth_ssl_require_client_cert = no
auth_ssl_username_from_cert = no
auth_stats = no
auth_use_winbind = no
auth_username_chars = 
abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ01234567890.-_@
auth_username_format = %Lu
auth_username_translation = 
auth_verbose = no
auth_verbose_passwords = no
auth_winbind_helper_path = /usr/bin/ntlm_auth
auth_worker_max_count = 30
base_dir = /run/dovecot
config_cache_size = 1 M
debug_log_path = 
default_client_limit = 1000
default_idle_kill = 1 mins
default_internal_group = dovecot
default_internal_user = dovecot
default_login_user = dovenull
default_process_limit = 100
default_vsz_limit = 256 M
deliver_log_format = msgid=%m: %$
dict_db_config = 
disable_plaintext_auth = yes
dotlock_use_excl = yes
doveadm_allowed_commands = 
doveadm_api_key = 
doveadm_http_rawlog_dir = 
doveadm_password = 
doveadm_port = 0
doveadm_socket_path = doveadm-server
doveadm_ssl = no
doveadm_username = doveadm
doveadm_worker_count = 0
first_valid_gid = 1
first_valid_uid = 0
import_environment = TZ CORE_OUTOFMEM CORE_ERROR LISTEN_PID LISTEN_FDS 
NOTIFY_SOCKET
info_log_path = 
libexec_dir = /usr/lib/dovecot
listen = *
log_core_filter = 
log_debug = 
log_path = /var/log/dovecot.log
log_timestamp = "%b %d %H:%M:%S "
mail_access_groups = 
mail_always_cache_fields = 
mail_attachment_detection_options = 
mail_attachment_dir = 
mail_attachment_fs = sis posix
mail_attachment_hash = 

Re: [auth] epoll_ctl(add, 13) failed: Operation not permitted (fd doesn't support epoll)

[John Stoffel]

Hi Alex,

I don't know anything about SELinux, beyond that it's a pain to work
with and causes all kinds of funky issues.  Make sure you turn on
verbose logging with SELinux so that you can see all that it's doing,
but honestly, I cannot help you much more.  

John



> just for completeness, here are the additional policies to SELinux that
> I had enabled (prior to semanage permissive -a dovecot_auth_t): 

> #= dovecot_auth_t ==

> # This avc is allowed in the current policy
> allow dovecot_auth_t dovecot_t:tcp_socket { accept getattr };

> # This avc is allowed in the current policy
> allow dovecot_auth_t postgresql_port_t:tcp_socket name_connect;

> With these, I do not see any avc in audit.log, but see the core dump. 

> Best regards
> Alex

> On Mon, 2023-11-20 at 08:47 +0100, Alexander Vogt wrote:
>> Hi John, 
>> 
>> thanks - yes, this is a new setup (I am migrating to CentOS 9). SELinux
>> is enabled, but audit.log does not show an AVC. However, I ran 
>> 
>> semanage permissive -a dovecot_t
>> 
>> and I am now able to dump the core. It is attached. With
>> 
>> semanage permissive -a dovecot_auth_t
>> 
>> auth seems to work. Now that it is established that the issue is due to
>> SELinux, I need to figure out how to solve it. SELinux was one of the
>> key motivations for the migration :)
>> Could you see what is going on from the dump? 
>> 
>> Best regards
>> Alex
>> 
>> 
>> On Sun, 2023-11-19 at 20:39 -0500, John Stoffel wrote:
>> > > > > > > "Alexander" == Alexander Vogt via dovecot  
>> > > > > > > writes:
>> > 
>> > Is this a new setup?  Do you have SELinux enabled?  Or are you doing
>> > chroot'd setup?  If so, back it all off one by one and see what's
>> > going on.  The fact that you can't dump core because you can't write
>> > somewhere tells me that your systems is locked down really hard in
>> > some manner.  
>> > 
>> > The fd not supporting epoll() is also suspect to me.  Can you give
>> > more details on your system setup?  Do you have apparmor turned on?
>> > Have you looked in your system logs as well?
>> > 
>> > John
>> > 
>> > 
>> > > dovecot auth service is failing when using an inet_service. The
>> > > configuration is essentially: 
>> > 
>> > > service auth {
>> > >   inet_listener {
>> > > address = *
>> > > port = 12345
>> > >   }
>> > >   unix_listener auth-userdb {
>> > > group = vmail
>> > > mode = 0666
>> > > user = vmail
>> > >   }
>> > > }
>> > 
>> > > When I connect to port 12345 (real IMAP client or telnet doesn't make a
>> > > difference), the auth service crashes. 
>> > 
>> > > Nov 19 22:21:54 imap.linexus.de dovecot[7195]: auth: Panic:
>> > > epoll_ctl(add, 13) failed: Operation not permitted (fd doesn't support
>> > > epoll)
>> > > Nov 19 22:21:54 imap.linexus.de dovecot[7195]: auth: Error: Raw
>> > > backtrace: /usr/lib64/dovecot/libdovecot.so.0(backtrace_append+0x46)
>> > > [0x7f9319f89486] ->
>> > > /usr/lib64/dovecot/libdovecot.so.0(backtrace_get+0x22) [0x7f9319f895a2]
>> > -> /usr/lib64/dovecot/libdovecot.so.0(+0x10a41b) [0x7f9319f9841b] ->
>> > > /usr/lib64/dovecot/libdovecot.so.0(+0x10a4b7) [0x7f9319f984b7] ->
>> > > /usr/lib64/dovecot/libdovecot.so.0(+0x5d11a) [0x7f9319eeb11a] ->
>> > > /usr/lib64/dovecot/libdovecot.so.0(+0x609b0) [0x7f9319eee9b0] ->
>> > > /usr/lib64/dovecot/libdovecot.so.0(+0x1215ba) [0x7f9319faf5ba] ->
>> > > /usr/lib64/dovecot/libdovecot.so.0(io_add_to+0x1d) [0x7f9319faf62d] ->
>> > > /usr/lib64/dovecot/libdovecot.so.0(io_add+0x28) [0x7f9319faf668] ->
>> > > /usr/lib64/dovecot/libdovecot.so.0(master_service_io_listeners_add+0x8a
>> > > ) [0x7f9319f1d16a] ->
>> > > /usr/lib64/dovecot/libdovecot.so.0(master_service_init_finish+0xff)
>> > > [0x7f9319f24bdf] -> dovecot/auth(main+0x389) [0x55745603a4f9] ->
>> > > /lib64/libc.so.6(+0x3feb0) [0x7f931963feb0] ->
>> > > /lib64/libc.so.6(__libc_start_main+0x80) [0x7f931963ff60] ->
>> > > dovecot/auth(_start+0x25) [0x55745603a715]
>> > 
>> > > System info (sysreport attached): 
>> > > # 2.3.16 (7e2e900c1a): /etc/dovecot/dovecot.conf
>> > > # Pigeonhole version 0.5.16 (09c2

Re: [auth] epoll_ctl(add, 13) failed: Operation not permitted (fd doesn't support epoll)

[John Stoffel]

> "Alexander" == Alexander Vogt via dovecot  writes:

Is this a new setup?  Do you have SELinux enabled?  Or are you doing
chroot'd setup?  If so, back it all off one by one and see what's
going on.  The fact that you can't dump core because you can't write
somewhere tells me that your systems is locked down really hard in
some manner.  

The fd not supporting epoll() is also suspect to me.  Can you give
more details on your system setup?  Do you have apparmor turned on?
Have you looked in your system logs as well?

John


> dovecot auth service is failing when using an inet_service. The
> configuration is essentially: 

> service auth {
>   inet_listener {
> address = *
> port = 12345
>   }
>   unix_listener auth-userdb {
> group = vmail
> mode = 0666
> user = vmail
>   }
> }

> When I connect to port 12345 (real IMAP client or telnet doesn't make a
> difference), the auth service crashes. 

> Nov 19 22:21:54 imap.linexus.de dovecot[7195]: auth: Panic:
> epoll_ctl(add, 13) failed: Operation not permitted (fd doesn't support
> epoll)
> Nov 19 22:21:54 imap.linexus.de dovecot[7195]: auth: Error: Raw
> backtrace: /usr/lib64/dovecot/libdovecot.so.0(backtrace_append+0x46)
> [0x7f9319f89486] ->
> /usr/lib64/dovecot/libdovecot.so.0(backtrace_get+0x22) [0x7f9319f895a2]
-> /usr/lib64/dovecot/libdovecot.so.0(+0x10a41b) [0x7f9319f9841b] ->
> /usr/lib64/dovecot/libdovecot.so.0(+0x10a4b7) [0x7f9319f984b7] ->
> /usr/lib64/dovecot/libdovecot.so.0(+0x5d11a) [0x7f9319eeb11a] ->
> /usr/lib64/dovecot/libdovecot.so.0(+0x609b0) [0x7f9319eee9b0] ->
> /usr/lib64/dovecot/libdovecot.so.0(+0x1215ba) [0x7f9319faf5ba] ->
> /usr/lib64/dovecot/libdovecot.so.0(io_add_to+0x1d) [0x7f9319faf62d] ->
> /usr/lib64/dovecot/libdovecot.so.0(io_add+0x28) [0x7f9319faf668] ->
> /usr/lib64/dovecot/libdovecot.so.0(master_service_io_listeners_add+0x8a
> ) [0x7f9319f1d16a] ->
> /usr/lib64/dovecot/libdovecot.so.0(master_service_init_finish+0xff)
> [0x7f9319f24bdf] -> dovecot/auth(main+0x389) [0x55745603a4f9] ->
> /lib64/libc.so.6(+0x3feb0) [0x7f931963feb0] ->
> /lib64/libc.so.6(__libc_start_main+0x80) [0x7f931963ff60] ->
> dovecot/auth(_start+0x25) [0x55745603a715]

> System info (sysreport attached): 
> # 2.3.16 (7e2e900c1a): /etc/dovecot/dovecot.conf
> # Pigeonhole version 0.5.16 (09c29328)
> # OS: Linux 5.14.0-383.el9.x86_64 x86_64 CentOS Stream release 9 

> This exact configuration is known to work on this system: 
> # 2.2.33.2 (d6601f4ec): /etc/dovecot/dovecot.conf
> # Pigeonhole version 0.4.21 (92477967)

> I tried for almost two hours to get a core dump for this, but finally
> gave up. I followed https://www.dovecot.org/bugreport-mail/#coredumps
> and other sources but the best I could get was

> Nov 19 22:21:54 imap.linexus.de dovecot[7195]: auth: Fatal: master:
> service(auth): child 7198 killed with signal 6 (core not dumped -
> https://dovecot.org/bugreport.html#coredumps - core wasn't writable?)

> for 

> cat /proc/sys/kernel/core_pattern
> /tmp/core.%e.%p

> (which is 1777). 

> Any help to get this resolved would be much appreciated! 
> Thanks and best regards
> Alex
> [DELETED ATTACHMENT dovecot-sysreport-imap.linexus.de-1700427979.tar.gz, 
> application/x-compressed-tar]
> ___
> dovecot mailing list -- dovecot@dovecot.org
> To unsubscribe send an email to dovecot-le...@dovecot.org
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: Remove attachments

[John Stoffel]

>>>>> "Oliver" == Oliver Glas  writes:

https://mimedefang.org/ is where I'd go to look for details on how to
implement it. I haven't done it myself.

> Is it possible to get a documentation for that ?
> Or a description how to implement that ?

> Am 03.06.23 um 23:28 schrieb John Stoffel:

> "Oliver" == Oliver Glas  writes:

> I am looking for a way to remove attachments, based on a condition.
> Like attachments starting with "TimeReport" shall be removed, and
> then the mail should be delivered, with all other attachments.

> This is not a solution that dovecat can work with.  You need to use a
> milter in your mailserver (like postfix) which can strip out mime
> attachements or do other modifications to an email before it's
> delivered to the mailbox.  

> I did so far not find a solution to remove attachments.  Do I need a
> plugin / extension  ? If so, how to implement and use that ?

> milters are what you want.  MIMEdefang is one.  

> John

___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: Remove attachments

[John Stoffel]

> "Oliver" == Oliver Glas  writes:

> I am looking for a way to remove attachments, based on a condition.
> Like attachments starting with "TimeReport" shall be removed, and
> then the mail should be delivered, with all other attachments.

This is not a solution that dovecat can work with.  You need to use a
milter in your mailserver (like postfix) which can strip out mime
attachements or do other modifications to an email before it's
delivered to the mailbox.  

> I did so far not find a solution to remove attachments.  Do I need a
> plugin / extension  ? If so, how to implement and use that ?

milters are what you want.  MIMEdefang is one.  

John
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: Help - Permissions issue with new mail

[John Stoffel]

> "michele" == michele clark--- via dovecot  writes:


> We have an install of Dovecot running with Postfix and Roundcube
> using virtual mail to allow us to connect to our AD domain which got
> messed up after a power issue.

Was this working before the AD problems, or is this a new install?

> I can log into our Roundcube and see all my old emails, however new
> emails keep getting stuck in the CUR files with permissions of
> -rw---

> If I chmod the email it will deliver to the inbox in Roundcube.

Where are you changing the permissions?  The destination directory?

> Can anyone advise on how I can adjust this so all new emails will
> come thru with the correct permissions? - Thanks in advance

It all depends on how your email are delivered, and you need to post
some of your configuration details, such as the postfix config and the
dovecot config.  

For postfix, just post the changed from defaults info, which you can
get with the following bash sheel command:

  comm -23 <(postconf -n) <(postconf -d)

See the postconf man page for details.

Also post your dovecot info as well.

___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: Postfix : root and system user authentication

[John Stoffel]

> "dovecot" == dovecot   writes:


> Me personally, this is why i prefer to use virtual users stored in a
> database for email and never use linux users. I have ultimate
> control over what users can be authenticated or receive email. I can
> add flags to the DB query to fail an otherwise valid user. Why would
> i want a root@ email address? Why would i want my system to accept
> email for httpd from some stranger on the internet? Why would i want
> to have to create a linux user at the OS level just to add a
> mailbox?

This is 110% agree.  It's just so simple to use purely virtual users,
even if you are pulling the login info from LDAP/AD for real users.
But you don't need to allow *any* logins to the dovecot or postfix
server using local logins at all.  It's just better security.

John

Re: Winbind auhentication

[John Stoffel]

> "Luciano" == Luciano Mannucci  writes:

> I'm trying to set up a dovecot server so that it authenticates local
> user via /etc/passwd (I'm on a Freebsd 13.1) and via winbindd for
> those that it cannot find localy. The samba suite is alive and well,
> postfix gets happily mail from domain users and saves it with
> correct name and permissions from the windows domain. If I try to
> authenticate a domain user via wbinfo it works, with dovecot it
> doesn't.  I guess I'v forgot something in the dovecot config... :)

I can't help you with your config, but I would *strongly* recommend
that you just make all your users virtual ones, and all using the same
backend.  Now you don't say if your local user account works or not,
but I'd work on getting just the AD part (really, you're using
windind?) first.

Also, have you compared your postfix and dovecot setups?  There are
good docs out there on how you combine them to use the same
authentication backend.


And the info you posted really don't help much, since you don't post
any log messages from when the authentication fails.  That will tell
you more I'm sure.

John

Re: Hide local IP from non delivery notifications

[John Stoffel]

> "Claudio" == Claudio Corvino  writes:


> I have an external MTA configured with Postfix that delivers email to an 
> internal IMAP/LMTP
> Dovecot server configured to bind an LDAP to check if users exist.

You should have postfix do the checking for whether or not users exist
and then have postfix reject and deny the message.  Then you don't
care because the IP of the postfix server is almost certainly your MX server.

> If the user does not exist, my MX sends a non delivery notification,
> and into the e-mail there's the local IP address of the Dovecot
> server.

> I can't find a way to remove or hide that IP, here is an example of line that 
> contains the IP
> (with X) that I would like to hide/remove:

> "The mail system

> : host hostname.domain.com[XXX.XXX.XXX.XXX] said: 550"

> Thanks in advance!

> Cheers

> --
> Claudio

Re: mail filters

[John Stoffel]

> "mick" == mick crane  writes:

> I'm using what ever is in Debian Bookworm, I'm pretty sure.
> These filters?

sieve of some sort.

> I collect from different email addresses like gmail and that.  If
> filter for the "Sent to" email address, move to this directory If
> "From" contains "blah", move to this other directory.  If something
> is Sent to and also From contains "blah" then I get 2 copies of the
> message in both filter directories.  How can I fix this?  regards

By posting an example of the non-working filter code?  Otherwise how
do we know what's giong wrong?

Maybe you're filtering the wrong way?  

I use something like this and it's pretty good:

   require ["fileinto", "envelope"];
   require "imap4flags";
   require "regex";

   if header :contains "Sender" "linux-kernel-ow...@vger.kernel.org" {
 fileinto "lkml";
   }
   elsif header :contains "X-Mailing-List" "linux-ker...@vger.kernel.org"
   {
 fileinto "lkml";
   }
   else {
 # The rest goes into INBOX
 # default is "implicit keep", we do it explicitly here
 keep;
   }


There's an implicit exit in each block as I recall and as my notes in
there show... I've got a whole bunch of other matches.  

Now where *my* rules fail at times is handling duplicates which were
went to a mailing list AND sent to me directly.  I need to fix it, but
haven't bothered to spend the time yet.

Re: Dovecot mail-crypt webmail can't read encrypted messages

[John Stoffel]

> "Serveria" == Serveria Support  writes:

> Yes, there is a tiny problem letting the attacker change this value back 
> to yes and instantly get access to users' passwords in plain text. Apart 
> from that - no problems at all. :)

Honestly, if the attacker has penetrated you to such an extent, then
you're toast anyway, because they can just attach to the dovecot
process with 'gdb' and dump the data directly as well. 

Encryption is not a magic solution here, and there's no real way to
secure the system so well that once an attacker can modify files and
restart processes they are blocked.  Because they honestly looks like
an Admin doing work on the system.  



> On 2022-10-11 12:15, Benny Pedersen wrote:
>> Serveria Support skrev den 2022-10-11 10:37:
>>> Thanks, but I suspect you've missed a part of this discussion
>> 
>> if you set all to no, is there any problem to solve ?
>> 
>> i am only human, not perfect
>> 
>>> 
>>> On 2022-10-11 01:25, Benny Pedersen wrote:
 Serveria Support skrev den 2022-10-10 23:18:
> Hi Benny,
> 
> Sorry I must have missed your email. Here's the output of doveconf 
> -P
> | grep auth:
> 
> doveconf: Warning: NOTE: You can get a new clean config file with:
> doveconf -Pn > dovecot-new.conf
> doveconf: Warning: Obsolete setting in /etc/dovecot/dovecot.conf:25:
> 'imaps' protocol is no longer necessary, remove it
 
 remove imaps in protocol as it says
 
> auth_debug = yes
> auth_debug_passwords = yes
> auth_verbose = yes
> auth_verbose_passwords = yes
 
 change yes to no
 
 problem solved imho :)

Re: Pigeonhole redirect is adding a message-id header when it already exists

[John Stoffel]

> "Michael" == Michael Peddemors  writes:

> This should almost be an RFC discussion, rather than a dovecot 
> discussion, for clarity on what to do with a malformed Message-Id.

Seems simple, just rename it to "Message-Id-Orig: ..." and insert your
own.  Fixes the problem, still let's you trace the message, etc. 



> For the record, if you start modifying it by deleting the bad message 
> id, and adding your own, you can start breaking other things, such as 
> DKIM signing etc..

> IMHO, Dovecot should simply refuse to accept or deliver a message with a 
> 'bad' message id, so that the sending system can identify and correct 
> the problem.

> That way Dovecot doesn't need to address/modify the email message.

>   -- Michael --

> On 2022-10-01 21:35, Sébastien Riccio wrote:
>> Hi,
>> 
>> After reading a bit the code and trying to understand it, here is what I 
>> think happens here:
>> 
>> Given a bogus Message-ID, for example (notice it's missing angle 
>> brackets < >:
>> 
>> Message-ID: 
>> 1883biz_pay_after_purchase:0:0_572392900$ae7ed6e4d53b424c84aaf83b30c507e7
>> 
>> Dovecot is parsing Message-ID headers and is looking for the angle 
>> bracket as the begining of the Message-ID:
>> https://github.com/dovecot/core/blob/d2ff32792ac052610cea7d65f30de1ee139cb55c/src/lib-mail/message-id.c#L75
>>  
>> 
>> 
>> As none is found it will act as if there was no Message-ID header in the 
>> mail (even that the header is present).
>> 
>> Then, pigeonhole's redirect function  is told to generate a new 
>> Message-ID if none was previously detected:
>> https://github.com/dovecot/pigeonhole/blob/5a3f4bd672cc2fb9e755a4b09c4753ac86e15f99/src/lib-sieve/cmd-redirect.c#L569
>>  
>> 
>> 
>> The result is the mail being forwarded, in this case, is now having dual 
>> Message-ID and is not RFC 5322 compliant anymore and can be rejected for 
>> this reason (hi, gmail?)
>> https://www.spamresource.com/2022/08/gmail-weird-rfc-5322-bounces-and-what.html
>>  
>> 
>> 
>> Some thoughts:
>> 
>> - First, to be honest, I'm not sure gmail would accept the original mail 
>> with the bogus Message-ID sent directly to their servers, but if it was 
>> refused, I would assume that these senders would have fixed the issue on 
>> their side so their message are delivered (unless there is some 
>> whitelisting going on?)
>> 
>> - What options could we have to resolve this?
>> 
>> a) Having dovecot core to remove the Message-ID header line from the 
>> mail if it is not going to consider it valid ? (So there is no dupe 
>> headers when pigeonhole adds one?)
>> b) Having pigeonhole check, when adding a new valid Message-ID, if there 
>> is already one existing, and remove the bogus one ?
>> For now, to workaround this, I'm trying to find a way in the mail flow 
>> on our servers to keep only the top most Message-ID when more than one 
>> exists.
>> Maybe using: https://www.postfix.org/postconf.5.html#smtp_header_checks 
>>  but I'm not 
>> sure how to achieve it yet or even if it's possible.
>> 
>> Kind regards
>> 
>> *Sébastien RICCIO*
>> 
>> *SYSTEM ADMINISTRATOR*
>> 
>> *P*  +41 840 888 888
>> 
>> *F***+41 840 888 000
>> 
>> *Msric...@swisscenter.com *
>> 
>> *
>> *
>> 
>> 
>> 
>> -- Message d'origine --
>> De "michael.z...@feierfighter.de " 
>> mailto:michael.z...@feierfighter.de>>
>> À "dovecot@dovecot.org " 
>> mailto:dovecot@dovecot.org>>
>> Date 01.10.2022 14:49:13
>> Objet Re: Re[6]: Pigeonhole redirect is adding a message-id header when 
>> it already exists
>> 
>>> Hi there,
>>> I can confirm this behavior. A few months ago I introduced a milter 
>>> which is checking for multiple headers when the RFC says that there 
>>> just should be one of them For example "Message-Id".
>>> I found the described problem in an email coming from Alibaba, which 
>>> had an invalid "Message-Id" header. It didn't contain an "@" sign or 
>>> similar. It was RFC-invalid.
>>> This email was sent from Alibaba to a German email provider. There was 
>>> a redirect at that email provider, pointing to my mailserver.
>>> My server rejected the email because there were 2 "Message-Id" 
>>> headers: The original invalid "Message-Id" header from Alibaba, and a 
>>> new "Message-Id" header from the German provider, which seems to have 
>>> been added during the redirect. There were "Dovecot-sieve" headers in 
>>> that mail, so my guess was that it happened because of 
>>> Dovecot-sieve/pigeonhole implementation.
>>> I contacted the email provider, asking for help. Asking if it really 

Re: Outlook 365 MUA produces dovecot changing filename on some mailbox

[John Stoffel]

>>>>> "Emilio" == Emilio Augusto Lazo Zaia  writes:

> Hello John! Thanks for your answer.

You're welcome, but I'm not sure I'm going to be able to help here.  I
suspect this is a client issue, just like I hate how the Apple Mail
client marks messages SEEN for no reason at all because it does
"SELECT INBOX" vs "EXAMINE INBOX". 


> On 12/9/22 12:30, John Stoffel wrote:

> I have a setup on which dovecot is the server in a domain on which 
> mailboxes are >20GB, some of
> them are 150GB or more.

> It looks like you're using Maildir format?  In any case, please post
> more details of your configuration. 

> Sure. It is a simple setup, with Maildir.

> # doveconf -N

> auth_debug = no
> auth_mechanisms = plain login
> auth_verbose = no
> debug_log_path = /var/log/syslog/mail.debug
> disable_plaintext_auth = no
> login_greeting = Hermes.
> mail_debug = yes
> mail_index_rewrite_max_log_bytes = 1 M
> mail_location = maildir:~/Maildir
> namespace inbox {
>  inbox = yes
>  location =  
>  mailbox Drafts {
>    special_use = \Drafts
>  }
>  mailbox Junk {
>    special_use = \Junk
>  }
>  mailbox Sent {
>    special_use = \Sent
>  }
>  mailbox "Sent Messages" {
>    special_use = \Sent
>  }
>  mailbox Trash {
>    special_use = \Trash
>  }
>  prefix =  
> }
> passdb {
>  driver = pam
>  name =  
> }
> protocols = " imap"
> service auth {
>  unix_listener auth-client {
>    group = Debian-exim
>    mode = 0660
>    user = Debian-exim
>  }
> }
> service imap {
>  vsz_limit = 2 G
> }
> ssl = yes
> ssl_cert =  ssl_client_ca_dir = /etc/ssl/certs
> ssl_dh = # hidden, use -P to show it
> ssl_key = # hidden, use -P to show it
> userdb {
>  driver = passwd
>  name =  
> }
> verbose_proctitle = yes
> protocol imap {
>  mail_max_userip_connections = 20
> }

> When reconfiguring _some_ mailboxes with Outlook 365 (configuring a 
> previously used mailbox on a
> new PC), the process takes too long even if mail visibility on 
> Outlook is 3 months.

> So the outlook client can't download and parse the number of messages
> on there without breaking?  How many messages are there?  It's
> probably not the size of the mailbox, but the number of messages.

> 50K mails on some folders... total 5.5M on the whole server. I did discover 
> which was the problem
> but I don't have a solution.

Yeah, that's alot.  You might need to push your clients to start
putting more emails into sub-folders, maybe split up by year say, so
they don't have quite so many emails sitting around.  

> Inspecting at INBOX cur directory I did note that the 'ls' command 
> shows the last file listed is a
> file from year 2020, 2019 or whatever not being the last received 
> mail; that means the
> alphabetical ordering of 'ls' is showing past years after the last 
> received mail, i.e. dovecot
> changed the epoch in filename. This only happens with Outlook. 
> Following is an excerpt of ls:

> Sure, 'ls -l' doesn't do any sorting, it just reads the directory
> information as returned from the disk and show you the results.  If
> you want it by time, you need to do:

> ls -ltr

> to have the newest files be at the end.  But how 'ls' sees the
> directry entries doen't matter to dovecot.  

> Newest files are at the end; 'ls' without '-U' or '--sort=none' lists files 
> ordered
> alphabetically, so the last entry would be the last mail placed on the folder 
> because the epoch
> time is part of the filename in Maildir structure...

> -rw-r--r-- 1 coord.ventas ventas   544765 sep  9 10:57 
> 1662769412.M329925P1259441.xyz.com,S=
> 544765,W=552045:2,
> -rw-r--r-- 1 coord.ventas ventas   163491 sep  5 12:14 
> 1662769412.M74257P1259441.xyz.com,S=
> 163491,W=165846:2,S
> -rw-r--r-- 1 coord.ventas ventas  3043536 sep  9 12:02 
> 1662769412.M777084P1259441.xyz.com,S=
> 3043536,W=3083246:2,
> -rw-r--r-- 1 coord.ventas ventas   161002 feb 27  2020 
> 1662874173.M608062P128.xyz.com,S=
> 161002,W=163373:2,S
> -rw-r--r-- 1 coord.ventas ventas  2230491 feb 27  2020 
> 1662874176.M281294P128.xyz.com,S=
> 2230491,W=2259506:2,S
> -rw-r--r-- 1 coord.ventas ventas   167925 feb 27  2020 
> 1662874176.M741229P128.xyz.com,S=
> 167925,W=170373:2,S

> The process of renaming files is continuous until all folder was
> reread and renamed every file on it. If I issue 'lsof' (with grep) I
> see

Re: Outlook 365 MUA produces dovecot changing filename on some mailbox

[John Stoffel]

>>>>> "Jaroslaw" == Jaroslaw Rafa  writes:

> Dnia 12.09.2022 o godz. 12:30:29 John Stoffel pisze:
>> Sure, 'ls -l' doesn't do any sorting, it just reads the directory
>> information as returned from the disk and show you the results.  If
>> you want it by time, you need to do:
>> 
>> ls -ltr
>> 
>> to have the newest files be at the end.

> 'ls' (and 'ls -l' as well) by default sorts files alphabetically. It has
> always been so.

Ooops, you're right!  I was blanking on that since I was still
thinking of the opendir() and readdir() calls which don't sort
directory entries.

> For unsorted list, one needs to use 'ls -lU' (that applies to GNU version of
> 'ls'; other versions may not recognize the '-U' switch).

Yup!

Re: Thousands of SSL certificates stalls new logins during reload - problem with Dovecot config process

[John Stoffel]

> "Bartosz" == Bartosz Kwitniewski  writes:

> Out of other services on that machine that are able to handle such 
> number of certificates during reloads:
> - proftpd loads configs dynamically based on SNI domain
> - exim loads certificates dynamically based on SNI domain
> - LiteSpeed switches to a new process after loading whole configuration

Are you running all these services on one machine?  Maybe you could
get an SSL termination device which terminates the SSL connections and
then forwards them into the proper backend application?  This way only
one system needs to be managed for certs, and only one (or two since I
assume you have an HA pair :-) needs to then reload when new certs are
inserted.

If you could hack the proftpd cert code into dovecot, that might also
be a way around it.  I haven't a clue how this works since I haven't
looked at either code base.  It won't be simple, but I'm sure others
would apprecaite it.  

If it's critical, paying for the feature to be added is another
option.  


> Best regards,
> --
> Bartosz Kwitniewski

> On 02/09/2022 14:52, Felipe Gasper wrote:
>> For hosting environments--where TLS certs can change hundreds of times in a 
>> matter of minutes--it would be a boon for Dovecot to load those certificates 
>> dynamically rather than all at once.
>> 
>> Pure-FTPd implements a nice solution to this: a standalone service that 
>> fetches TLS certificates & keys. Documented here:
>> 
>> https://github.com/jedisct1/pure-ftpd/blob/9d25440e5b5283fbeca94dd0595aa6672c3f8428/README.TLS#L161
>> 
>> -FG
>> 
>> 
>>> On Sep 2, 2022, at 08:44, Bartosz Kwitniewski  wrote:
>>> 
>>> Hello,
>>> 
>>> I'm running a dovecot 2.3.19.1 server that has around 6000 SSL certificates 
>>> in separate config files, each containing:
>>> local_name "domain" {
>>> ssl_cert = ...
>>> ssl_key = ...
>>> }
>>> When new certificate is added, dovecot is reloaded (around 20 times a day). 
>>> When dovecot is being reloaded, users are unable to log in for around 30 
>>> seconds.
>>> 
>>> The main problem here seems to be that during reload, new config process is 
>>> immediately designated as the one serving config requests and then it 
>>> starts parsing config files, which takes around 20-30 seconds. If it would 
>>> parse config files first, and only then would become a new process for 
>>> serving config requests, then it would probably solve the problem. Or 
>>> perhaps there is a better way to load new certificates or a way to optimize?
>>> 
>>> There is another problem with config process and shutdown_clients=no. We do 
>>> not want to disconnect users during reload, because e.g. Thunderbird 
>>> displays a popup that server is shutting down. When there are long lasting 
>>> IMAP connections from Google and other services that aggregate e-mail, old 
>>> config process is not being killed. Because config process with ~6000 
>>> certificates is using ~1 GB of RAM, it can quickly rise to 20 GB of memory 
>>> used. This is not a big issue however, because we have created a task that 
>>> kills old processes, but there could be a built-in mechanism to solve that 
>>> problem.
>>> 
>>> I have created minimal configuration and scripts to recreate problem. 
>>> Reproduction steps below.
>>> 
>>> (...)

Re: dovecot/config processes open, and consuming all memory

[John Stoffel]

>>>>> "filipe@digirati" == filipe@digirati com br  
>>>>> writes:


filipe@digirati> I tested the suggestion to decrease the service_count and the 
accounts 
filipe@digirati> stopped logging in, then I gradually increased it to 500, and 
now it's 
filipe@digirati> working. But the dovecot/config processes are still being 
created, there 
filipe@digirati> has been no change for the better.

So maybe you can give us more background on this setup.  How many
users are you supporting?  How many mailboxes?  What is your backend
mailbox format?  Maybe you need to share your dovecot config as well.  

Have you tried upgrading to 2.3.19.1 as well?

So when you say "the accounts stopped logging in" does that mean your
users started having problems logging into the system?  Does that mean
IMAP sessions timed out?

filipe@digirati> Now I have an error constantly appearing in the mail.err file

filipe@digirati> Aug 15 14:19:34 box6 dovecot: imap(USER_NAME): Error: Error 
reading 
filipe@digirati> configuration: read(/run/dovecot/config) failed: 
read(size=8192) failed: 
filipe@digirati> Interrupted system call - Also failed to read config by 
executing 
filipe@digirati> doveconf: /run/dovecot/config is a UNIX socket (path is from 
CONFIG_FILE 
filipe@digirati> environment)


Have you stopped and completely restarted your dovecot setup when you
made the config changes?  

filipe@digirati> Aug 15 14:19:34 box6 dovecot: imap(USER_NAME): Error: Error 
reading 
filipe@digirati> configuration: read(/run/dovecot/config) failed: 
read(size=8192) failed: 
filipe@digirati> Interrupted system call - Also failed to read config by 
executing 
filipe@digirati> doveconf: /run/dovecot/config is a UNIX socket (path is from 
CONFIG_FILE 
filipe@digirati> environment)
filipe@digirati> Aug 15 14:19:34 box6 dovecot: imap(USER_NAME): Error: Error 
reading 
filipe@digirati> configuration: read(/run/dovecot/config) failed: 
read(size=8192) failed: 
filipe@digirati> Interrupted system call - Also failed to read config by 
executing 
filipe@digirati> doveconf: /run/dovecot/config is a UNIX socket (path is from 
CONFIG_FILE 
filipe@digirati> environment)

filipe@digirati> On 14/08/2022 22:24, John Stoffel wrote:
>>>>>>> "filipe@digirati" == filipe@digirati com br  
>>>>>>> writes:
>> filipe@digirati> I'm having strange behavior in dovecot 2.3.16.
>> filipe@digirati> It's opening dozens of dovecot/config process and consuming 
>> all server
>> filipe@digirati> memory. Normally each process consumes between 700Mb and 
>> 1Gb of ram.
>> 
>> filipe@digirati> Would anyone have an idea about this?
>> 
>> filipe@digirati> service config {
>> filipe@digirati>    vsz_limit = 2048M
>> filipe@digirati>    idle_kill = 60s
>> filipe@digirati>    service_count = 1024
>> filipe@digirati> }
>> 
>> I wonder why you have the service count so high?  I'd drop it down, or
>> do you really have 1025 connections in 60 seconds?
>> 
>> From the docs:
>> 
>> 
>> service_count
>> 
>> Number of client connections to handle until the process kills
>> itself. 0 means unlimited. 1 means only a single connection is handled
>> until the process is stopped - this is the most secure choice since
>> there’s no way for one connection’s state to leak to the next one. For
>> better performance this can be set higher, but ideally not unlimited
>> since more complex services can have small memory leaks and/or memory
>> fragmentation and the process should get restarted eventually. For
>> example 100..1000 can be good values.
>> 
>> So maybe drop it down to 100 for now and see how that works for you.
>> 
>> 
>> filipe@digirati> pstree
>> filipe@digirati> systemd─┬─ModemManager───2*[{ModemManager}]
>> filipe@digirati>      ├─agetty
>> filipe@digirati>      ├─cron
>> filipe@digirati>      ├─dbus-daemon
>> filipe@digirati>      ├─dovecot─┬─anvil
>> filipe@digirati>      │ ├─6*[auth]
>> filipe@digirati>      │ ├─46*[config]
>> filipe@digirati>      │ ├─1212*[imap]
>> filipe@digirati>      │ ├─155*[imap-login]
>> filipe@digirati>      │ ├─12*[lmtp]
>> filipe@digirati>      │ ├─38*[log]
>> filipe@digirati>      │ ├─10*[managesieve]
>> filipe@digirati>      │ ├─19*[pop3]
>> filipe@digirati>      │ ├─3*[pop3-login]
>> filipe@digirati>      │ └─18*[stats]
>> 
>> 
>> filipe@digirati> root   45831  0.0

Re: dovecot/config processes open, and consuming all memory

[John Stoffel]

> "filipe@digirati" == filipe@digirati com br  
> writes:

filipe@digirati> I'm having strange behavior in dovecot 2.3.16.
filipe@digirati> It's opening dozens of dovecot/config process and consuming 
all server 
filipe@digirati> memory. Normally each process consumes between 700Mb and 1Gb 
of ram.

filipe@digirati> Would anyone have an idea about this?

filipe@digirati> service config {
filipe@digirati>    vsz_limit = 2048M
filipe@digirati>    idle_kill = 60s
filipe@digirati>    service_count = 1024
filipe@digirati> }

I wonder why you have the service count so high?  I'd drop it down, or
do you really have 1025 connections in 60 seconds?

>From the docs:


 service_count

 Number of client connections to handle until the process kills
 itself. 0 means unlimited. 1 means only a single connection is handled
 until the process is stopped - this is the most secure choice since
 there’s no way for one connection’s state to leak to the next one. For
 better performance this can be set higher, but ideally not unlimited
 since more complex services can have small memory leaks and/or memory
 fragmentation and the process should get restarted eventually. For
 example 100..1000 can be good values.

So maybe drop it down to 100 for now and see how that works for you.  


filipe@digirati> pstree
filipe@digirati> systemd─┬─ModemManager───2*[{ModemManager}]
filipe@digirati>      ├─agetty
filipe@digirati>      ├─cron
filipe@digirati>      ├─dbus-daemon
filipe@digirati>      ├─dovecot─┬─anvil
filipe@digirati>      │ ├─6*[auth]
filipe@digirati>      │ ├─46*[config]
filipe@digirati>      │ ├─1212*[imap]
filipe@digirati>      │ ├─155*[imap-login]
filipe@digirati>      │ ├─12*[lmtp]
filipe@digirati>      │ ├─38*[log]
filipe@digirati>      │ ├─10*[managesieve]
filipe@digirati>      │ ├─19*[pop3]
filipe@digirati>      │ ├─3*[pop3-login]
filipe@digirati>      │ └─18*[stats]


filipe@digirati> root   45831  0.0  1.1 774688 752732 ?   S    09:31 
0:31 
filipe@digirati> dovecot/config
filipe@digirati> root  388792  0.0  1.1 775060 753276 ?   S    14:00   
0:15 
filipe@digirati> dovecot/config
filipe@digirati> root  510685  0.0  1.1 775384 753604 ?   S    15:06   
0:20 
filipe@digirati> dovecot/config
filipe@digirati> root  675638  0.0  1.1 775348 753620 ?   S    16:56   
0:15 
filipe@digirati> dovecot/config
filipe@digirati> root  795375  0.0  1.1 775460 753516 ?   S    18:03   
0:07 
filipe@digirati> dovecot/config
filipe@digirati> root  798754  0.2  1.1 775592 753712 ?   S    18:05   
0:30 
filipe@digirati> dovecot/config
filipe@digirati> root 1082696  0.2  1.1 774892 753216 ?   S    21:10   
0:07 
filipe@digirati> dovecot/config
filipe@digirati> root 1098433  0.4  1.1 774924 753244 ?   S    21:33   
0:07 
filipe@digirati> dovecot/config
filipe@digirati> root 1109255  0.9  1.1 774924 753344 ?   S    21:50   
0:07 
filipe@digirati> dovecot/config
filipe@digirati> root 1112976  2.0  1.1 774956 753528 ?   S    21:57   
0:07 
filipe@digirati> dovecot/config
filipe@digirati> root 1114137  3.0  1.1 775028 753308 ?   S    21:59   
0:07 
filipe@digirati> dovecot/config
filipe@digirati> root 1115382  5.4  1.1 774924 753496 ?   S    22:01   
0:06 
filipe@digirati> dovecot/config
filipe@digirati> root 1883627  0.0  1.1 759120 728832 ?   S    Aug11   
0:07 
filipe@digirati> dovecot/config
filipe@digirati> root 1889705  0.0  1.8 1251460 1221872 ? S    Aug11   
0:11 
filipe@digirati> dovecot/config
filipe@digirati> root 1895022  0.0  1.8 1253280 1224284 ? S    Aug11   
0:11 
filipe@digirati> dovecot/config
filipe@digirati> root 1900690  0.0  1.8 1255684 1227528 ? S    Aug11   
0:12 
filipe@digirati> dovecot/config
filipe@digirati> root 1905648  0.0  1.8 1257880 1229912 ? S    Aug11   
0:12 
filipe@digirati> dovecot/config
filipe@digirati> root 1910857  0.0  1.8 1259156 1231552 ? S    Aug11   
0:12 
filipe@digirati> dovecot/config
filipe@digirati> root 1914332  0.0  1.1 764328 736552 ?   S    Aug11   
0:20 
filipe@digirati> dovecot/config
filipe@digirati> root 2343896  0.0  1.8 1259472 1231516 ? S    Aug11   
0:12 
filipe@digirati> dovecot/config
filipe@digirati> root 2346351  0.0  1.8 1259472 1231836 ? S    Aug11   
0:13 
filipe@digirati> dovecot/config
filipe@digirati> root 2348559  0.0  1.1 764704 736440 ?   S    Aug11   
0:14 
filipe@digirati> dovecot/config
filipe@digirati> root 2445701  0.0  1.1 764276 736540 ?   S    Aug11   
0:19 
filipe@digirati> dovecot/config
filipe@digirati> root 2572525  0.0  1.1 764640 736880 ?   S    Aug11   
0:18 
filipe@digirati> dovecot/config
filipe@digirati> root 2734251  0.0  1.1 764776 737696 ?   S    Aug11   
0:08 

Re: Tools to get a report of which folders have new mail?

[John Stoffel]

> "Peter" == Peter   writes:

Peter> On 19/07/22 3:18 pm, Steve Litt wrote:
>> Is there any way I could use
>> doveadm or other tools to create a report that shows all my folders in a
>> hierarchy?

Peter> See doveadm(1) and doveadm-mailbox(1), specifically the `doveadm mailbox 
Peter> list` command.

>> Also, is there a way to show only those with new mail?

Peter> Look at doveadm-search(1) and doveadm-search-query(7) for this.

Peter> You can loop through the list of mailboxes from doveadm mailbox
Peter> list and pass them one at a time to `doveadm search NEW MAILBOX
Peter> mailboxname` to see if any messages are returned from the
Peter> search.

You will run into problems if you use an iPhone to also access your
mail via IMAP as well, for some reason it marks all new email as SEEN,
even if they haven't been opened.  It's  annoying to say the
least.  It's an iPhone issue from what I see.  The archives will have
more on this from three or four years ago when I last beat my head
against this.

Maybe it's time to fine a new iPhone mail client now that you can get
rid of the default iOS mail client if you like.

Re: Multidomain ssl config ?

[John Stoffel]

> "Maurizio" == Maurizio Caloro  writes:

Maurizio> on postfix now this seems to run, and with dovecot i need
Maurizio> also handle this two domains, but appairing this error
Maurizio> messages. like:

Why aren't you just using a single domain as the MX record for all the
domains?  Then you only need one SSL cert pair for all of this, and if
you publish the right SPF records, each domain can send from the same
MX host as well.




Maurizio> Jun 29 20:49:28 Dovecot/imap-login: Info: Disconnected (no auth 
attempts in 0 secs): user=<>,
Maurizio> rip=a.b.c.d, lip=37.120.190.188, TLS handshaking: SSL_accept() 
failed: error:14094416:SSL routines:
Maurizio> ssl3_read_bytes:sslv3 alert certificate unknown: SSL alert number 46, 
session=

Maurizio> Running with Debian Buster

Maurizio> # dovecot --version
Maurizio> 2.3.4.1 (f79e8e7e4)

Maurizio> # nmail.caloro.ch
Maurizio> local_name nmail.caloro.ch {
Maurizio> ssl_cert =  ssl_key =  }
Maurizio> # nmail.calm-ness.ch
Maurizio> local_name nmail.calm-ness.ch {
Maurizio> ssl_cert =  ssl_key =  }

Maurizio> thanks for possible help

Re: doveadm-deduplicate deletes non-duplicates

[John Stoffel]

Aki> We have released 2.3.19.1 instead, and should be fixed now.

Thanks!

Re: doveadm-deduplicate deletes non-duplicates

[John Stoffel]

> "Aki" == Aki Tuomi  writes:

Will 2.3.20 be released ASAP with this fix?  

Aki> This has now been fixed in main with
Aki> 
https://github.com/dovecot/core/commit/2780f106e3b185981dd7aaf5cbf2e88daa2f7c64.patch

Aki> Aki

>> On 13/06/2022 10:43 gravitini  wrote:
>> 
>> 
>> Please consider as critical (data loss) and recommend a warning is 
>> issued for 2.3.19 users.
>> 
>> 
>> On 13/06/22 5:25 pm, Aki Tuomi wrote:
>> >> On 13/06/2022 02:09 gravitini  wrote:
>> >>
>> >>   
>> >> Replying to: https://dovecot.org/pipermail/dovecot/2022-May/124816.html
>> >>
>> >>
>> >> Hi,
>> >>
>> >> Looking at the code (and tested via local build from source) it looks
>> >> like doveadm deduplicate in 2.3.19 can cause significant data loss.
>> >>
>> >> A 2022-02-11 commit removed key duplication resulting in undefined
>> >> behaviour which is often truncation of a mailbox to 67 entries.
>> >> (HASH_TABLE_MIN_SIZE)
>> >>
>> >> https://github.com/dovecot/core/commit/320844f50cd669b602d30210e2e5216f65d2050f?diff=split#diff-5842cf9d4248dc515d80ebb45575341b7d76832f979a8ac5f602784cb5b03f2cL121
>> >>
>> >> diff --git a/src/doveadm/doveadm-mail-deduplicate.c
>> >> b/src/doveadm/doveadm-mail-deduplicate.c
>> >>
>> >> index caec758112..2152482876 100644
>> >> --- a/src/doveadm/doveadm-mail-deduplicate.c
>> >> +++ b/src/doveadm/doveadm-mail-deduplicate.c
>> >> @@ -63,8 +63,10 @@ cmd_deduplicate_box(struct doveadm_mail_cmd_context
>> >> *_ctx,
>> >>       if (key != NULL && *key != '\0') {
>> >>       if (hash_table_lookup(hash, key) != NULL)
>> >>       mail_expunge(mail);
>> >> -   else
>> >> +   else {
>> >> +   key = p_strdup(pool, key);
>> >>       hash_table_insert(hash, key,
>> >> POINTER_CAST(1));
>> >> +   }
>> >>       }
>> >>       }
>> > Thank you both for the report, we'll look into this!
>> >
>> > Aki

Re: Issue with one user only, exceeding connections

[John Stoffel]

> "Jeremy" == Jeremy Schaeffer  writes:

Jeremy> Thanks for the command, that is very useful.  That user is
Jeremy> actually me, I know why where are so many open. I have my
Jeremy> computer, and two tablets, and since I am using server side
Jeremy> filtering (procmail) I have to set watch on all the folders
Jeremy> that are filtered to or I miss a email. But I am doing the
Jeremy> same for about 4 other users accounts I also monitor, so I am
Jeremy> not sure why it's just my username that is doing that. I am
Jeremy> going to shut down all the clients one at a time and see what
Jeremy> client is opening all those connections.

I too used to use procmail, but it's old and un-maintained and it's
not a great solution any more.  Instead just setup Sieve and you can
manage it remotely.  I personally do it from a makefile which does all
the steps needed, and I just edit the file in emacs as needed.

I can give you a copy of my makefile if that will help.


Jeremy> Once I close the client, I assume the connection should also close and 
Jeremy> the count go down, correct?

Jeremy> I turned off both tablets and the connection count for my username 
still 
Jeremy> is at 60, since I am writing this email with my computer client I will 
Jeremy> send it and close my client and see what happens. Thanks! - Jeremy

Jeremy> On 6/9/2022 11:29, Richard wrote:
>> 
>>> Date: Thursday, June 09, 2022 11:07:38 -0500
>>> From: Jeremy Schaeffer 
>>> 
>>> On 6/9/2022 10:59, Richard wrote:
> Date: Thursday, June 09, 2022 10:46:25 -0500
> From: Jeremy Schaeffer 
> 
> That was the first thing I tried, I lowered the cache connections
> in Thunderbird. Actually the max connections was 50, not 500, but
> I could see why as I do have a lot of folders, but what is odd is
> I have other mailboxes that have even more folders, but it's only
> one mailbox that is trowing the error.
> 
> "# ps -axww | grep imap" does not give me the same results -
> 
> .
> 
> 19897 ?    S  0:00 dovecot/imap
> 19900 ?    S  0:02 dovecot/imap
> 19901 ?    S  0:00 dovecot/imap
> 19902 ?    S  0:00 dovecot/imap
> .
> 
> I wish it did give me the mailbox, is there a option to get it to
> give me that information?
 Try "auxw" on your "ps". I.e., add in the "u" which will get you
 the user detail in the first column, otherwise you just get the
 process id.
 
 
>>> Thank you! That worked, I piped the output to a file, grep the
>>> username and sure enough there are 60 lines. So I guess going over
>>> 50 was a possibility.
>>> 
>>> Learn something new every day. I set the maximum to 100 so I should
>>> not have any errors on that anymore.
>>> 
>> 
>> Rather than simply upping the limit I think a reasonable question to
>> ask is why/how they are managing to do that. That's a lot of open
>> folders.
>> 
>> By the way, the single command:
>> 
>> ps auxw | grep imap | cut -d" " -f1 | sort | uniq -c
>> 
>> will get you a nice list with the users and their connection counts.
>> 
>> 
>> 
>> 

Re: Extremely long line

[John Stoffel]

> "Dmitriy" == Dmitriy Fitisov  writes:

Dmitriy> Hello everyone,
Dmitriy> I’m using https://github.com/karastojko/mailio
Dmitriy> to get messages from dovecot-2.3.7.2 (3c910f64b)

So post your code that you're using to get the message from dovecot.
I assume you're using IMAP?  

Dmitriy> and 1 of the messages which is 2.5 mbytes in size and has an
Dmitriy> image attached, has a half megabyte of <80> (0x80) characters
Dmitriy> at the end.

Do you have a test case message that works?  What version of dovecot
is the server running?  

Dmitriy> And that is with default encoding of 7 bit (meaning there is
Dmitriy> no part headers at the end), right after the message those
Dmitriy> characters.  I got a feeling that max length was 998
Dmitriy> characters.  On other hand, looks like dovecot is replacing
Dmitriy> NULL chars with 0x80.



Dmitriy> Is that is correct behavior, should I expect that half meg of bytes in 
1 line?

It sounds more like a problem with the tool encoding the mail
message.  How did you create the message that you are trying to
download?

Start with simple test cases.

1. create a plain ASCII message and send it to dovecot.
2. download the message and compare the message bodies.

Repeat steps 1 and 2 with more complex messages, such as messages with
long long lines, etc.

But you really need to give us more details on how these messages are
created, and how you're trying to read them.

What happens if you look at the message with another mail client?
Does it fail?  Can the message be read?

John

Re: Random behavior with sieve extprograms

[John Stoffel]

>>>>> "Thomas" == Thomas Sommer  writes:

Thomas> Hi John
Thomas> On 2022-06-01 02:50, John Stoffel wrote:
>>>>>>> "Thomas" == Thomas Sommer  writes:
>> 
Thomas> I have a random behavior with dovecot and sieve extprograms.
>> 
Thomas> Here is my sieve file:
Thomas> require ["fileinto", "vnd.dovecot.pipe", "copy", "imap4flags"];
Thomas> # rule:[DABS]
Thomas> if header :contains "X-Original-To" "d...@mydomain.ch"
Thomas> {
Thomas> pipe "sieve-dabs-execute.sh";
Thomas> setflag "\\Seen";
Thomas> fileinto "acme.DABS";
Thomas> stop;
Thomas> }
>> 
>> Can you post the code of this script?  Are you trapping all exceptions
>> in that script and making sure you only return errors when there
>> really is an error?
>> 
Thomas> Emails matching the condition are processed by a laravel (php) 
>> artisan
Thomas> command. See service sieve-pipe-script below.
Thomas> The exit code of this php command is 0.
>> 
>> You are calling the php command from a shell script, so there's
>> multiple places things could go wrong.  Why not just pipe directly to
>> the php script (which wasn't included unless I'm totally blind and
>> dumb tonight... :-) instead?

Thomas> "sieve-dabs-execute.sh" is just the socket name. It was a
Thomas> shell script previously and I never updated the socket
Thomas> name. See service sieve-pipe-script in the dovecot -n output.
Thomas> It calls the php script directly: executable = script
Thomas> /usr/bin/php /srv/www/mydomain/status/artisan
Thomas> dabs:processEmail

Thanks for the clarification, I missed that part before.  

Thomas> When testing directly on the cli, it works flawlessly, return
Thomas> code is 0.  bash: php artisan dabs:processEmail < email.file

How about if you run multiple copies of the script at the same time on
the console?  You might be running into contention there.  

Thomas> Here is the handle method of the php script:

Thomas> public function handle()
Thomas>  {
Thomas>  $fd = \fopen('php://stdin', 'rb');

Thomas>  $parser = new MailMimeParser();
Thomas>  $message = $parser->parse($fd, true);

Thomas>  $subject = $message->getHeader(HeaderConsts::SUBJECT);
Thomas>  $dabsDate = \substr(\trim($subject), -11, 8);
Thomas>  $date = \Carbon\Carbon::parse($dabsDate);
Thomas>  $version = \substr($message->getHeader(HeaderConsts::SUBJECT), 
Thomas> -2);

Thomas>  $attachment = $message->getAttachmentPart(0);
Thomas>  $filename = $attachment->getFilename();

Thomas>  if (Storage::exists('/dabs/' . $filename)) {
Thomas>  Log::info('Processing DABS duplicate version: ' . $version 
. 
Thomas> ' of: ' . $date->format('Y-m-d'));
Thomas>  // increment number to filename
Thomas>  $a = 1;
Thomas>  do {
Thomas>  $filename_new = \basename($filename, '.pdf') . '_' . 
$a 
Thomas> . '.pdf';
Thomas>  $a++;
Thomas>  if ($a > 9) {
Thomas>  Log::error('DABS duplicate processing > 9. 
Thomas> Exiting.');
Thomas>  $this->error('DABS duplicate processing > 9. 
Thomas> Exiting.');
Thomas>  exit(1);
Thomas>  }
Thomas>  $filename = $filename_new;
Thomas>  } while ($this->dabsFileExists($filename_new));
Thomas>  }

Thomas>  Storage::put('/dabs/' . $filename, $attachment->getContent());
Thomas>  $dabs = Dabs::create(
Thomas>  [
Thomas>  'date' => $date,
Thomas>  'version' => $version,
Thomas>  'file' => 'dabs/' . $filename,
Thomas>  ]
Thomas>  );


This part might break because you assume that you're the only
instance of the script running.  You really want to do some locking,
and one way to do that is to try and create a new file in a loop,
since that is an atomic operation.  So in the while loop, once the
Storage::exists call fails, you need to make the file with the
Storage::put, but you need to double check the return value and
continue looping if that file already exists, otherwise you need to
exit and retun the error message (out of disk space, no more inodes,
can't write to file, etc).

With possibly multiple copies of the script running at the same time,
you could be stomping on each other and if you don't handle it
properly, it will do wierd things.  


Thomas>  if ($date->eq(today()) || $date->eq(today()->addDay())) {
Thomas>  

Re: Random behavior with sieve extprograms

[John Stoffel]

> "Thomas" == Thomas Sommer  writes:

Thomas> I have a random behavior with dovecot and sieve extprograms.

Thomas> Here is my sieve file:
Thomas> require ["fileinto", "vnd.dovecot.pipe", "copy", "imap4flags"];
Thomas> # rule:[DABS]
Thomas> if header :contains "X-Original-To" "d...@mydomain.ch"
Thomas> {
Thomas> pipe "sieve-dabs-execute.sh";
Thomas> setflag "\\Seen";
Thomas> fileinto "acme.DABS";
Thomas> stop;
Thomas> }

Can you post the code of this script?  Are you trapping all exceptions
in that script and making sure you only return errors when there
really is an error?  

Thomas> Emails matching the condition are processed by a laravel (php) artisan 
Thomas> command. See service sieve-pipe-script below.
Thomas> The exit code of this php command is 0.

You are calling the php command from a shell script, so there's
multiple places things could go wrong.  Why not just pipe directly to
the php script (which wasn't included unless I'm totally blind and
dumb tonight... :-) instead?

It honestly sounds like a timing issue, maybe just putting a sleep
into your shell script at the end would be good?  Or maybe run with
the -vx switches so you log all the commands and their results?  


Thomas> I randomly get the following in my postfix logs:
Thomas> Sieve thinks that the command failed, but the email was always 
processed 
Thomas> correctly. In that case I get a copy in my Inbox.
Thomas> I'm wondering what could be the cause for this random behavior.
Thomas> My guess is that approximately 70% are processed correctly, 30% is as 
Thomas> below.

Thomas> May 31 13:50:38 star dovecot[99425]: 
Thomas> lda(user)<99425>: sieve: 
Thomas> msgid=<62961d1c.5y4hr0vqi97jfnyb%dabs.zsmsv...@example.com>: fileinto 
Thomas> action: stored mail into mailbox 'acme.DABS'
Thomas> May 31 13:50:39 star dovecot[99425]: 
Thomas> lda(user)<99425>: sieve: 
Thomas> msgid=<62961d1c.5y4hr0vqi97jfnyb%dabs.zsmsv...@example.com>: stored 
mail 
Thomas> into mailbox 'INBOX'
Thomas> May 31 13:50:39 star dovecot[99425]: 
Thomas> lda(user)<99425>: sieve: Execution of script 
Thomas> /home/user/sieve/.dovecot.sieve failed, but implicit keep was 
successful 
Thomas> (user logfile /home/user/sieve/.dovecot.sieve.log may reveal additional 
Thomas> details)

Thomas> .dovecot.sieve.log:
Thomas> sieve: info: started log at May 31 13:50:39.
Thomas> error: failed to pipe message to program `sieve-dabs-execute.sh': refer 
Thomas> to server log for more information. [2022-05-31 13:50:39].

Thomas> It's weird. "failed to pipe message to program" is simply not true. The 
Thomas> command was processed correctly.

Thomas> Any ideas where to look for clues or how to debug this?

Thomas> Regards
Thomas> Thomas

Thomas> config:

Thomas> # 2.3.14 (cee3cbc0d): /etc/dovecot/dovecot.conf
Thomas> # Pigeonhole version 0.5.14 (1b5c82b2)
Thomas> # OS: Linux 5.17.5-x86_64-linode154 x86_64 Ubuntu 20.04.4 LTS
Thomas> auth_mechanisms = plain login
Thomas> auth_username_format = %n
Thomas> auth_verbose = yes
Thomas> mail_location = maildir:~/Maildir
Thomas> mail_plugins = " quota"
Thomas> managesieve_notify_capability = mailto
Thomas> managesieve_sieve_capability = fileinto reject envelope 
Thomas> encoded-character vacation subaddress comparator-i;ascii-numeric 
Thomas> relational regex imap4flags copy include variables body enotify 
Thomas> environment mailbox date index ihave duplicate mime foreverypart 
Thomas> extracttext vnd.dovecot.pipe vnd.dovecot.execute
Thomas> namespace inbox {
Thomas>inbox = yes
Thomas>location =
Thomas>mailbox Drafts {
Thomas>  special_use = \Drafts
Thomas>}
Thomas>mailbox Junk {
Thomas>  special_use = \Junk
Thomas>}
Thomas>mailbox Sent {
Thomas>  special_use = \Sent
Thomas>}
Thomas>mailbox "Sent Messages" {
Thomas>  special_use = \Sent
Thomas>}
Thomas>mailbox Trash {
Thomas>  special_use = \Trash
Thomas>}
Thomas>prefix =
Thomas> }
Thomas> passdb {
Thomas>driver = pam
Thomas> }
Thomas> plugin {
Thomas>quota = fs:User quota
Thomas>quota_grace = 1%%
Thomas>quota_status_nouser = DUNNO
Thomas>quota_status_overquota = 552 5.2.2 Mailbox is full
Thomas>quota_status_success = DUNNO
Thomas>sieve = file:~/sieve;active=~/sieve/.dovecot.sieve
Thomas>sieve_execute_socket_dir =
Thomas>sieve_extensions = +vnd.dovecot.pipe +vnd.dovecot.execute
Thomas>sieve_pipe_exec_timeout = 30s
Thomas>sieve_pipe_socket_dir =
Thomas>sieve_plugins = sieve_extprograms
Thomas>sieve_redirect_envelope_from = recipient
Thomas>sieve_trace_debug = no
Thomas>sieve_trace_dir = ~/sieve/trace
Thomas>sieve_trace_level = matching
Thomas> }
Thomas> protocols = imap sieve
Thomas> service auth {
Thomas>unix_listener /var/spool/postfix/private/dovecot-auth {
Thomas>  group = postfix
Thomas>  mode = 0660
Thomas>  user = postfix
Thomas>}
Thomas> }
Thomas> service quota-status {
Thomas>client_limit = 1

Re: Dovecot v2.3.19 released

[John Stoffel]

> "A" == A Schulze  writes:

A> Am 10.05.22 um 08:33 schrieb Aki Tuomi:
>> Hi all!
>> 
>> We are pleased to release v2.3.19 of Dovecot.
>> 
>> The docker images have been upgraded to use bullseye as base image.
>> 
>> https://dovecot.org/releases/2.3/dovecot-2.3.19.tar.gz
>> https://dovecot.org/releases/2.3/dovecot-2.3.19.tar.gz.sig

A> Hello,

A> "make check" fail here:

A> test-crypto.c:827: Assert failed: ret == TRUE
A> Panic: file dcrypt-openssl.c: line 2639 
(dcrypt_openssl_private_to_public_key): assertion failed: (priv_key != NULL && 
pub_key_r != NULL)
A> Error: Raw backtrace: ./test-crypto(backtrace_append+0x42) [0x556260e86cb2] 
-> ./test-crypto(backtrace_get+0x1e) [0x556260e86dce] -> 
./test-crypto(+0x25bcb) [0x556260e65bcb] -> ./test-crypto(+0x25c01) 
[0x556260e65c01] -> ./test-crypto(+0x13dab) [0x556260e53dab] -> 
.libs/libdcrypt_openssl.so(+0x5f13) [0x7f6133c60f13] -> ./test-crypto(+0x1e436) 
[0x556260e5e436] -> ./test-crypto(+0x21aef) [0x556260e61aef] -> 
./test-crypto(test_run+0x47) [0x556260e626c7] -> ./test-crypto(main+0x50) 
[0x556260e582a0] -> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xea) 
[0x7f6133c99d0a] -> ./test-crypto(_start+0x2a) [0x556260e583ba]
A> /bin/bash: line 1:90 Aborted ./$bin

A> any advise is welcome...

What linux distro are you using?  Which compiler?  Can you post the
config.log output to see what you're settings are?

Re: Doveadm backup...

[John Stoffel]

>>>>> "Sami" == Sami Ketola  writes:

>> On 27. Mar 2022, at 23.09, John Stoffel  wrote:
>> 
>>>>>>> "Stephane" == Stephane Magnier  writes:
>> 
>> Sorry, I deleted your most recent email post before I could reply.
>> But why don't you just do 'imapsync' instead from your production
>> dovecot box to some other backup system?  Otherwise I'd probably work
>> to setup dovecot's own replication but only have it go one way.
>> 
>> For example, I've got a VPS out in the cloud for my email, and I
>> should probably back it up to my home system using replication, but it
>> would be strictly primary->secondary.  I wouldn't be trying to run two
>> primaries replicating between each other.
>> 
>> Imapsync would be an improvement over rsync because it works within
>> dovecot, so you'd get a more consistent view, but maybe not quite as
>> upto date.  But how important is your email if you worry about losing
>> 20 minutes worth of it?  If it's that critical, then you should be
>> investing in a more robust setup.

Sami> I would not recommend using imapsync to do backups as it loses
Sami> data.  There is no way for imapsync to retain all data as for
Sami> example IMAP protocol does not allow client to set email UID
Sami> numbers. Backup made with imapsync is always partial copy of the
Sami> original and cannotbe restored identically.

That's true, it's a bit brute force and not ideal.  But if you can't
setup properly doveadm replication, I would think that imapsync is
better than rsync, unless you can pause dovecot/postfix, snapshot the
filesystem, unpause dovecot/postfix then rsync the snapshot to your
remote backup server.

But I also think that if you're that terrified about losing email,
getting dovecot's replication working even just one way would be an
improvement.


But the OP doesn't really state his requirements for getting back
online.  If it's a personal site (like mine) then I can take my time,
it's not *that* critical.  If it's a customer setup, then why aren't
you running replication already, with dovecot director and shared
resilient storage behind it?

Re: Doveadm backup...

[John Stoffel]

> "Stephane" == Stephane Magnier  writes:

Sorry, I deleted your most recent email post before I could reply.
But why don't you just do 'imapsync' instead from your production
dovecot box to some other backup system?  Otherwise I'd probably work
to setup dovecot's own replication but only have it go one way.

For example, I've got a VPS out in the cloud for my email, and I
should probably back it up to my home system using replication, but it
would be strictly primary->secondary.  I wouldn't be trying to run two
primaries replicating between each other.

Imapsync would be an improvement over rsync because it works within
dovecot, so you'd get a more consistent view, but maybe not quite as
upto date.  But how important is your email if you worry about losing
20 minutes worth of it?  If it's that critical, then you should be
investing in a more robust setup.

John


Stephane> I've seen in a previous post, that the fact to do an RSYNC
Stephane> might break the  index.. So, I've heard that this is not
Stephane> recommended. that's the reason why I decided to find a way
Stephane> to do a "clean" backup and be able to come back online if
Stephane> needed.

Stephane> So, do you use an RSYNC and in case of restoring the mailbox, do you 
do a  simple

Stephane>  doveadm index -u USERx INBOX

Stephane> And that's it ? works fine ?

Stephane> 2) I will try the Backup , or Sync.. locally.. Effectively, I don't 
know where the problem comes from..I have effectively an NFS Mount
Stephane>  for the mailboxes and a VM for the Email server and that could be 
another another point of failure :-(

Stephane> Thanks for sharing.. I am a bit in a rush... I realized that my 
backup maybe not correct.. and I prefer not to discover it while running into 
trouble..

Stephane> On 2/28/22 19:03, Ben Burk wrote:

Stephane> I'm not sure what you are attempting to do here. It looks like 
you just ran a doveadm backup
Stephane> and the process completed for userx with a warning that the 
remote system (your nfs mount)
Stephane> lost a particular mailbox (possibly your indexes changed or a 
mailbox was deleted). From the
Stephane> logs you pasted it appears the process completed normally.
   
Stephane> I personally do not use dovecot's backup or replication 
processes. If I needed to I would use
Stephane> its replication process to sync active data between multiple 
systems, but I have no need for
Stephane> this as of yet. Personally I chose to create offsite backups 
using rsync a long time ago, as
Stephane> rebuilding a mailbox (reindexing) is very simple.
   
Stephane> Try running    doveadm mailbox status -u userx guid '*'   
 as the mailbox
Stephane> administrator and see if you can find that GUID, 
7e05c335174bf1608f0a02004eac7fb4. Also, see
Stephane> if the backup you've written to nfs has the GUID.

Stephane> On 2/27/22 23:33, Stephane Magnier wrote:

Stephane> Well no ..I thought that dsync was for synchro " realtime for 
2 different places ?
   
Stephane> Having no 2 machines in parallel ( Just a single machine ) , 
I thought that a backup at
Stephane> regular interval  would be enough ?
   
Stephane> So, a simple backup should be done by dsync finally ?
   
Stephane> Do you recommend finally NOT to do a backup ( Doveadm backup 
) but a replication process ?
Stephane> ( https://wiki.dovecot.org/Replication  ) ?
   
Stephane> On 2/28/22 06:24, Ben Burk wrote:

Stephane> Did you try running dsync?
   
Stephane> On 2/27/22 23:15, Stephane Magnier wrote:

Stephane> HI,
   
Stephane> Any idea ? Any clue ?
   
Stephane> On 2/25/22 21:50, Stephane Magnier wrote:

Stephane> Hi
   
Stephane> I've recently tried to use the Dovecadm backup to 
backup the emails..  with
Stephane> the following syntax
   
Stephane> doveadm -Dv backup -u userx 
maildir:/mnt/nfs-backup/userx
   
Stephane> Sounds to be OK with few emails... Some of them 
got a lot of emails and one f
Stephane> them got an error and stop !
   
Stephane> dsync(userx): Debug: brain S: Import Trash: 
Import change type=expunge
Stephane> GUID=1725fa475d774ee19cb98dfb6737b4f1 
UID=24891 hdr_hash= result=GUIDs
Stephane> match
Stephane> dsync(userx): Debug: brain S: Import Trash: 
Import change type=expunge
Stephane> GUID=916ed110b4b1522868be6194f1ae36ff 
UID=24892 hdr_hash= result=GUIDs
Stephane> match
Stephane> dsync(userx): Debug: brain S: Import Trash: 
Import change type=expunge
Stephane>

Re: General Locking & Replication Issues

[John Stoffel]

> "Paul" == Paul Kudla (Scom ca Internet Services Inc )  
> writes:

Paul> Ok I am running two dovecot servers and feel pretty close to
Paul> getting things resolved config wise there are however 2 or 3
Paul> issues that all seem to be related

Paul> i have enclosed FULL config files for both mail18.scom.ca & mail19.scom.ca

Paul> I have a private network that is used for the replication
Paul> (10.221.0.0/24) - port 12345 is blocked by pf firewall from
Paul> outside.

Paul> I am running freebsd 12.1 and built one server and then raid
Paul> copied it to the second (mail19) so all the ports / builds etc
Paul> are identical

Paul> I did start with dovecot-2.3.14 but pigeonhole would not build
Paul> on freebsd (thanks for that advice / reply )until upgrade to
Paul> dovecot-2.3.18

Paul> dovecot-2.3.18 was built and installed on both servers along
Paul> with a good build (freebsd build error was fixed after
Paul> upgrading) dovecot-2.3-pigeonhole-0.5.18

Paul> please note that the second mailserver (mail19) is on an nfs
Paul> share dedicated to the server (i am awaiting sdram drives to
Paul> replace this issue)

Paul> but since i am getting errors on both servers i feel nfs share
Paul> is not the issue (and yes i enabled dlock)

Paul> mail18 is on normal sdram drives (more of an fyi as i dont think this is 
the issue)

Paul> neither mail server is connected to the other in any kind of a
Paul> shared way - other then replication

Paul> postfix should be delivering via :

Paul> pipe -n dovecot -t unix flags=DRhu user=vmail:vmail
Paul> argv=/usr/local/libexec/dovecot/deliver -f $ {sender} -d
Paul> ${recipient}

Why are you doing it this way?  I'm just curious because in my small
mail server, I just have a definition in my master.cf which looks like
this:

  # Updated to support + addressing, 20210402
  spamass-dovecot unix - n   n   -   -   pipe
flags=DRhu user=mail:mail argv=/usr/bin/spamc -u debian-spamd -e
/usr/lib/dovecot/deliver -a ${rec
ipient} -d ${user}@${domain}
  
And in my main.cf I have:

  virtual_transport = spamass-dovecot

The whole idea being that postfix uses the dovecot deliver tool to
insert mail into dovecot, which gets the right notifications workins.
It's not clear to me what mail tool you're using. 

How is your 'dovecot' transport defined in the postfix master.cf file?
I'm honestly not going any replication, but I'm thinking that the
'pipe -n ...' stuff you're doing is the root cause here.  

John



Paul> Replication only works properly at startup (ie the mail boxes do
Paul> sync), afterwhich nothing seems to trigger the differences
Paul> between the servers to update (notify i believe)

Paul> manually running :

Paul> [05:04:46] mail18.scom.ca [root:0] /usr/local/etc/dovecot
Paul> # doveadm sync -u p...@scom.ca remote:10.221.0.19

Paul> &

Paul> [05:21:18] mail19.scom.ca [root:0] /usr/local/etc/dovecot
Paul> # doveadm sync -u p...@scom.ca remote:10.221.0.18 

Paul> does seem to work but with mixed results.

Paul> I also included postfix main.cf & master.cf as well as some
Paul> googling has indicated the file locking may be on the postfix
Paul> side ??

Paul> Also more then happy to donate just can not figure out how ???

Paul> so that being said :

Paul> General logging errors are as follows ?

Paul> [04:46:43] peer1.scom.ca [paul:0] /home/paul
Paul> ## log dovecot rror

Paul> Filtering by : dovecot & rror

Paul> mail18  02-28 04:46:58 {dovecot}    [30374] (859753649)
Paul>     
doveadm(e...@scom.ca)<30486>
Paul> :
Paul>     Error: write() 
failed: Timed out after
Paul> 60 seconds
Paul> mail19  02-28 04:47:00 {dovecot}    [22194] (859753669)
Paul>     
doveadm(p...@scom.ca)<22219>
Paul> :
Paul>     Error: write() 
failed: Timed out after
Paul> 60 seconds
Paul> mail18  02-28 04:47:05 {dovecot}    [30374] (859753724) 
doveadm(ed.ha...@dssmgmt.com)
Paul> <30484><0dYbI8G
Paul>     ZHGIUdwAAz1jc/w>: 
Error: write()
Paul> failed: Timed
Paul>     out after 60 seconds
Paul> mail18  02-28 04:47:14 {dovecot}    [30374] (859753799)
Paul>     
doveadm(p...@scom.ca)<30485> KsI8GZHGIVdwAAz1jc/w>:
Paul>     Error: write() 
failed: Timed out after
Paul> 60 seconds
Paul> mail18  02-28 04:47:57 {dovecot}    [30374] (859754250) 
doveadm(p...@paulkudla.net)<30676>
Paul>      GLUdwAAz1jc/w>: 
Error: write() failed:
Paul> Timed out
Paul>     after 60 seconds
Paul> mail18  02-28 04:49:05 {dovecot}    [30374] (859754939) 

Re: Build with MySQL -> libmysqlclient not found

[John Stoffel]

> "Dimitri" == Dimitri   writes:

Dimitri> John:
Dimitri> ./configure --help
   
Dimitri> might also be a good idea, I think it might have some options like:
   
Dimitri> --mysql-libs=/path/to/lib
Dimitri> --mysql-includes=/path/to/include
   
Dimitri> or something similiar to use.

Dimitri> Unfortunately not, just the "--with-mysql" Option.

Dimitri> Oscar:
Dimitri> You might need CPPFLAGS as well. (e.g. 
CPPFLAGS="-I/test/core/mariadb/include")
Dimitri> Check also "config.log" for errors.

Dimitri> I had already done both, just forgot to mention it.
Dimitri> But i should have looked further.

Dimitri> Near the end of config.log is saw:
Dimitri> 

Dimitri> | /* Override any GCC internal prototype to avoid an error.
Dimitri> |Use char because int might match the return type of a GCC
Dimitri> |builtin and then its argument prototype would still apply.  */
Dimitri> | #ifdef __cplusplus
Dimitri> | extern "C"
Dimitri> | #endif
Dimitri> | char mysql_init ();
Dimitri> | int
Dimitri> | main ()
Dimitri> | {
Dimitri> | return mysql_init ();
Dimitri> |   ;
Dimitri> |   return 0;
Dimitri> | }
Dimitri> configure:23386: result: no
Dimitri> configure:23520: error: Can't build with MySQL support: 
libmysqlclient not found

Dimitri> 
Dimitri> and thought that wouldn't help me.

Dimitri> Now i looked again but further up an saw:
Dimitri> 

Dimitri> configure:23377: gcc -o conftest -std=gnu99 -g -O2 
-fstack-protector-strong -U_FORTIFY_SOURCE
Dimitri> -D_FORTIFY_SOURCE=2 -mfunction-return=keep -mindirect-branch=keep 
-Wall -W
Dimitri> -Wmissing-prototypes -Wmissing-declarations> ...
Dimitri> /usr/bin/ld: cannot find -lz
Dimitri> collect2: error: ld returned 1 exit status

Dimitri> 

Dimitri> Zlib i have (of course ;) ) also compiled from source.
Dimitri> After adding "/test/dep/zlib/include" to CPPFLAGS and 
"/test/dep/zlib/lib" to LDFLAGS the
Dimitri> configuration runs without errors.

Dimitri> However i don't understand why the configuration-script takes
Dimitri> the non-standard path for openssl automatically and correctly
Dimitri> from the PKG_CONFIG_PATH, but the paths to my zlib and
Dimitri> mariadb installations, which i also set in PKG_CONFIG_PATH
Dimitri> not...

I suspect that either the configure setup scripts doesn't support
using pkg-config for mysql/mariadb, or that there's a problem with how
it works.

In any case, it's good to see you got this working for yourself.  It
might also be that the configure.ac needs to be setup so that mariadb
depends on libz being found and complaining/failing if not.

Cheers,
John

Re: Build with MySQL -> libmysqlclient not found

[John Stoffel]

> "Oscar" == Oscar del Rio  writes:

Oscar> On 2022-02-26 10:29 a.m., Dimitri wrote:
Oscar> After that i've tried to configure with following:

Oscar> LDFLAGS="-L/test/core/mariadb/lib" ./configure 
--prefix=/test/core/dovecot --with-ssldir=/
Oscar> test/core/dovecot/tls --with-mysql

Oscar> You might need CPPFLAGS as
Oscar> well. (e.g. CPPFLAGS="-I/test/core/mariadb/include") Check also
Oscar> "config.log" for errors.

As Oscar says, looking at the config.log after your configure run is a
good step.  


./configure --help

might also be a good idea, I think it might have some options like:

 --mysql-libs=/path/to/lib
 --mysql-includes=/path/to/include

or something similiar to use.  And I can understand wanting to compile
from the source and having your own setup to work from, god knows I've
done it for enough years to compile various pieces of software over
the years.

I haven't checked the dovecot source myself for the actual flags, but
this should get you going.

John

Re: Build with MySQL -> libmysqlclient not found

[John Stoffel]

> "Dimitri" == Dimitri   writes:

Dimitri> Hi folks,
Dimitri> just step into another problem and don't know why?

Dimitri> If i try to configure dovecot with the following:

Dimitri> ./configure --prefix=/test/core/dovecot
Dimitri> --with-ssldir=/test/core/dovecot/tls --with-mysql

Dimitri> i get

Dimitri> ...
Dimitri> checking for auth_userokay... no
Dimitri> checking for mysql_config... NO
Dimitri> checking for mysql_init in -lmysqlclient... no
Dimitri> configure: error: Can't build with MySQL support: libmysqlclient 
not found

Dimitri> After that i've tried to configure with following:

You probably need to tell 



Dimitri> LDFLAGS="-L/test/core/mariadb/lib" ./configure 
--prefix=/test/core/dovecot --with-ssldir=/test
Dimitri> /core/dovecot/tls --with-mysql

Dimitri> but with same result.

Dimitri> The Content of "/test/core/mariadb/lib" is:

Dimitri> libmariadbclient.a
Dimitri> libmariadb.so -> libmariadb.so.3
Dimitri> libmariadb.so.3
Dimitri> libmysqlclient.a -> libmariadbclient.a
Dimitri> libmysqlclient_r.a -> libmariadbclient.a
Dimitri> libmysqlclient_r.so -> libmariadb.so.3
Dimitri> libmysqlclient.so -> libmariadb.so.3
Dimitri> libmysqlservices.a
Dimitri> pkgconfig
Dimitri> plugin

Dimitri> Also the Path "/test/core/mariadb/lib/pkgconfig" is in the
Dimitri> PKG_CONFIG_PATH variable and the Library-Path
Dimitri> "/test/core/mariadb/lib" is known by the dynamic linker
Dimitri> (/etc/ld.so.conf.d/ mylibs.conf).

Dimitri> So what am i doing wrong?

Dimitri> My Dovecot version: 2.3.18
Dimitri> My Mariadb version: 10.6.5
Dimitri> My OS: Ubuntu 20.04

Why aren't you just using the Ubuntu 20.04 packaged version instead?
Also, did you install the headers for libmysqlclient properly as
well?

What does /test/core/mariadb/includes/ or
/test/core/includes/... show?

John

Re: mail-crypt and mbox format

[John Stoffel]

Unfortunately, this document doesn't really address the OP's need,
which is to migrate mailbox formats on the same server.  Now migrating
to a new server would work, where the new server was setup to use
maildir as the default.

Maybe a new section could be added talking about this situation in
more explicit terms, with some real examples of conversions?

John

Aki> https://doc.dovecot.org/admin_manual/migrating_mailboxes/

Aki> Aki

>> On 11/02/2022 21:29 cincodemayo...@yahoo.com  
>> wrote:
>> 
>> 
>> Thank you for confirming. That was the conclusion I came to, particularly 
>> after seeing the structure of Maildir mailboxes and how the individual 
>> messages were encrypted. Clearly it would be difficult to do the same with 
>> an unlimited number of messages stored in a single file. 
>> 
>> A followup question if I may. I probably should just start another thread, 
>> but, how difficult is it to convert to Maildir? Any gotchas? Any differences 
>> in how to manage the server? How effective is the mb2md? Not looking for a 
>> cookbook, just an opinion on whether it is worth converting.
>> 
>> We've used mbox format going back before CentOS 5, so change is hard.
>> 
>> Environment is CentOS 7, Dovecot, Sendmail, Pigeonhole, MailScanner, 
>> Mailwatch SQL, Thunderbird clients. 
>> 
>> Thanks,
>> Doug
>> On Friday, February 11, 2022, 09:31:09 AM EST, Aki Tuomi 
>>  wrote:
>> 
>> 
>> 
>> 
>> > On 11/02/2022 16:26 cincodemayo...@yahoo.com  
>> > wrote:
>> > 
>> > 
>> > 
>> > 
>> > Hi,
>> > 
>> > 
>> > My Dovecot server of many years has been set up to use mbox email folders. 
>> > I want to implement mail-crypt and after banging my head against a wall 
>> > for a few days trying to get mail-crypt to work I decided to try it 
>> > against a test instance of my server that I reset to use Maildir format 
>> > and mail-crypt worked instantly.
>> > 
>> > 
>> > Does mail-crypt work with mbox format mail folders, or am I wasting my 
>> > time unless I switch over to Maildir? The documentation at 
>> > https://doc.dovecot.org/configuration_manual/mail_crypt_plugin/ doesn't 
>> > explicitly say Maildir is required.
>> > 
>> > 
>> > Thanks in advance,
>> > 
>> > 
>> > Doug 
>> > 
>> > 
>> > 
>> > 
>> > 
>> > 
>> > 
>> > 
>> > 
>> > 
>> > 
>> Mail crypt will not work with it. Mbox format has limited support of 
>> features.
>> ---
>> Aki Tuomi

Re: Too many wait in auth process

[John Stoffel]

> "ismael" == ismael tanguy@univ-brest fr  
> writes:


ismael> I'm currently benchmarking new hardware aimed to serve around
ismael> 70k users For now, our IMAP server have 13k users.

This doesn't help us help you.  Is this a new rasperry Pi 4?  Is it a
Dual CPU AMD Rzyzen with 128gb of memory and fast NVMe disks?  What is
your system setup? 

ismael> To run imaptest, I've spwan some bench clients.

Are these tests run from remote hosts?  What kind of network are you
using?  

ismael> Each bench client can run imaptest with 1000 clients.
ismael> More than 1000 clients will load CPU of this bench client

ismael> imaptest command (command are chosen from usage stat on our other IMAP 
servers):

ismael> imaptest host=x port=xxx userfile=userfile mbox=/root/dovecot-crlf
ismael> pass=s seed=123 clients=1000 select=194 uidfetch=94 noop=70
ismael> status=82 append=49 fetch=276 list=12 store=19 expunge=22
ismael> msubs=4 search=4 logout=1 delete=81 no_pipelining

ismael> With one bench client, everything runs smoothly.

ismael> # ps aux | grep dovecot | awk '{print $11,$12,$13,$14,$15,$16,$17,$18}' 
| sort | uniq -c
ismael>      1 anvil: [221 connections] (anvil)
ismael>    1 auth: [13 wait, 0 passdb, 0 userdb] (auth)
ismael>    1 dovecot/config
ismael>    1 dovecot/imap
ismael>   84 dovecot/imap-login
ismael>    1 dovecot/log
ismael>   20 dovecot/pop3-login
ismael>    1 grep dovecot
ismael>    1 stats: [1307 connections] (stats)

ismael> When a second instance bench instance start imaptest, clients
ismael> of first and second instance begin to stall :

ismael>  1400 stalled for 20 secs in command: 1 LOGIN 
"fakeuser644@mailbench" "password"

So how is your dovecot authentication setup?  Are you using a mysql
backend?  LDAP?  Where is the server you're querying against?  Are you
running mysql on the same server you're running dovecot on?

Are you running multiple dovecot servers with dovecot director in
front of them to help spread the load and to offer resilience if/when
a backend server fails?  

ismael> And :

ismael> # ps aux | grep dovecot | awk '{print $11,$12,$13,$14,$15,$16,$17,$18}' 
| sort | uniq -c
ismael>    1 anvil: [221 connections] (anvil)
ismael>    1 auth: [1227 wait, 0 passdb, 0 userdb] (auth)
ismael>    1 dovecot/config
ismael>    1 dovecot/imap
ismael>   37 dovecot/imap-login
ismael>    1 dovecot/log
ismael>   20 dovecot/pop3-login
ismael>    1 grep dovecot
ismael>    1 stats: [680 connections] (stats)

ismael> Every auth go in wait, number of connection decreases.

ismael> Using mysql or a password file give same results.

Where is mysql located?  

ismael> I have used different values for service_count with also no success.

Post your configuration details.

ismael> I think my use of imaptest could be false.

It could be.  Are you thinking that 2000 users will all be logging
into the system at the same time?  

ismael> My understanding of service auth is limited for now because
ismael> I'm quite new to Dovecot (I have previously worked with
ismael> Cyrus).

Can't really give you any hints until you tell us more about your
setup.

John

Re: Set mail crypt private password with OAUTH?

[John Stoffel]

> "Aki" == Aki Tuomi  writes:

Max,
It would be awesome if you could post a summary of what your setup is,
what you were trying to accomplish, and the configuration you came up
with after all this work with Aki and the rest of the team.

Digging through the entire chain would be a chore and while I've
enjoyed reading it all, I've already forgotten what the original setup
was you're were working on!

As a bonus, this writeup will help you in the future if you run into
problems again.  :-)

John


Aki> Ah. This is because you have a mistake in your userdb query:
Aki>SHA2(CONCAT(username, random_key), 256) AS
Aki> userdb_mail_crypt_private_password \

Aki> should be
Aki>SHA2(CONCAT(username, random_key), 256) AS
Aki>mail_crypt_private_password \

Aki> userdb_ prefix should only be used in passdb **or** passwd-file.

Aki> Aki

>> On 31/01/2022 13:00 Max Kostikov  wrote:
>> 
>> 
>> With removed userdb_mail_crypt_private_password part in the 
>> password_query it doesn't work at all even with standard password 
>> authentication.
>> 
>> 
>> Aki Tuomi писал(а) 2022-01-31 12:52:
>> > Using oauth2 or not should make no difference if the key is loaded in
>> > userdb. Can you check with mail_debug=yes to see that it gets loaded
>> > even if you remove it from passdb sql?
>> > 
>> > Aki
>> > 
>> >> On 31/01/2022 12:41 Max Kostikov  wrote:
>> >> 
>> >> 
>> >> Correction. Mail crypt works fine when I'm logged with the regular
>> >> password authentication
>> >> but doesn't when OAUTH2 is used.
>> >> 
>> >> Max Kostikov писал(а) 2022-01-31 12:30:
>> >> > Yes, that's right.
>> >> > I tried to get key with userdb before I wrote in the Dovecot list but
>> >> > this doesn't work for me.
>> >> > Yes, the decryption key is correct but for some reason it doesn't
>> >> > applied when key decryption.
>> >> >
>> >> >
>> >> > Aki Tuomi писал(а) 2022-01-31 12:09:
>> >> >> In fact now that I looked through your configs one more, this is
>> >> >> already what you are doing, except you are exporting the private key
>> >> >> password in three different places.
>> >> >>
>> >> >> So basically, if you do `doveadm user foobar` it should already give
>> >> >> you a correct key.
>> >> >>
>> >> >> You can see if the key is correct with `doveadm mailbox cryptokey
>> >> >> export -u user -U`
>> >> >>
>> >> >> Aki
>> >> >>
>> >> >>> On 31/01/2022 12:03 Aki Tuomi  wrote:
>> >> >>>
>> >> >>>
>> >> >>> Hgm. You have userdb lookups enabled, why not just move the entire
>> >> >>> mail_crypt_private_password handling there instead of passdb? This
>> >> >>> way it'll work with LMTP/LDA as well.
>> >> >>>
>> >> >>> So move all user related fields to the userdb lookup, and keep only
>> >> >>> the authentication handling in passdb.
>> >> >>>
>> >> >>> In your configuration, passdb lookups are not done for LMTP/LDA etc.
>> >> >>>
>> >> >>> Aki
>> >> >>>
>> >> >>> > On 31/01/2022 12:00 Max Kostikov  wrote:
>> >> >>> >
>> >> >>> >
>> >> >>> > Unfortunately there are no "master out" entries in the log, but I 
>> >> >>> > have
>> >> >>> > "userdb out"
>> >> >>> >
>> >> >>> > Jan 31 09:56:40 example.com dovecot: auth: Debug: master userdb out:
>> >> >>> > USER#0111609564161#011max.kosti...@gmail.com#011home=/var/vmail/gmail.com/max.kostikov/#011mail=maildir:/var/vmail/gmail.com/max.kostikov/#011uid=150#011gid=8#011quota=dirsize:storage=0#011userdb_mail_crypt_private_password=#011auth_mech=XOAUTH2#011auth_token=a8a38b3119780448ae96debd5687df75f5043378
>> >> >>> >
>> >> >>> >
>> >> >>> > Aki Tuomi писал(а) 2022-01-31 11:47:
>> >> >>> > > Was the field present in auth debug logs, it should be shown in 
>> >> >>> > > the
>> >> >>> > > "master out" log line and also it should be visible on 
>> >> >>> > > mail_debug=yes
>> >> >>> > > logs as `plugin/mail_crypt_private_key_password`.
>> >> >>> > >
>> >> >>> > > Aki
>> >> >>> > >
>> >> >>> > >> On 31/01/2022 11:40 Max Kostikov  wrote:
>> >> >>> > >>
>> >> >>> > >>
>> >> >>> > >> Unfortunatelly I still get decryption error with "Password not
>> >> >>> > >> available"
>> >> >>> > >>
>> >> >>> > >> ...
>> >> >>> > >> Jan 31 09:39:03 dev-message-portal-08.healthycareservice.com 
>> >> >>> > >> dovecot:
>> >> >>> > >> imap(max.kosti...@gmail.com)<22267><59cRjt3Wbtx/AAAB>: Error: 
>> >> >>> > >> Mailbox
>> >> >>> > >> INBOX: UID=1: read() failed:
>> >> >>> > >> read(/var/vmail/gmail.com/max.kostikov/cur/1643481212.M353350P24555.dev-message-portal-08.healthycareservice.com,S=2140,W=2193:2,S)
>> >> >>> > >> failed: Private key not available: Cannot decrypt key
>> >> >>> > >> 34255c3a029dc86ba4f07fa9bae2e87e4254de1d582f220a99b46f20bd382870:
>> >> >>> > >> Cannot
>> >> >>> > >> decrypt key
>> >> >>> > >> 98ae0f998f9139ebe20a97de77f162dcdeed496e38c9b5910186f999f3ef66c8:
>> >> >>> > >> Password not available
>> >> >>> > >> Jan 31 09:39:03 dev-message-portal-08.healthycareservice.com 
>> >> >>> > >> dovecot:
>> >> >>> > >> imap(max.kosti...@gmail.com)<22267><59cRjt3Wbtx/AAAB>: 
>> >> >>> > >> Disconnected:
>> >> >>> 

RE: silly quesiton

[John Stoffel]

> "Marc" == Marc   writes:

Marc> Why? Just disallow login, and that is from the perspective that
Marc> a mail user should be limited mail resources.
>> 
>> If the user does NOT need to login to the dovecot/mail servers, then
>> not having these users at all is more secure.

Marc> No, because there is a difference between a need to login and
Marc> the presence of a uid. Lots of daemons run under accounts that
Marc> cannot login.


You're missing my point.  Yes, the daemons running the services are
locked down.  But the users using those services have no need to for
logins or access to the system.  They only need access to the
application.

That's why virtual users are good.  Also, UIDs used to be limited to
under 65,000 seperate logins, but early on large FTP and ISP sites
disovered that they wanted to have more user than that, so moving to
virtual users was the solution.  


Marc> I argue exactly the opposite. Keep as much as possible linux
Marc> users. As linux has been engineered for allowing multiple user
Marc> accounts, and most other virtual user providers that are used
Marc> here, have not.
>> 
>> I'm having a hard time to parse what you are saying here.
>> 
>> I'm saying that if the mail/dovecot server is only providing mail
>> services, then putting all the users (across multiple domains even)
>> into a virtual user database is more secure

Marc> No it is not more secure, eg. 

Marc> 1. if a user does not exist on the os, how can processes be
Marc> spawned as these uid's. Everything is running under the same
Marc> uid.

Yes, the daemons/applications running the service being provided runs
under a single UID.  Which is more secure becuase now you have just
one UID to lock down tight, using apparmour, selinux or other OS level
tools.  

Marc> 2. if you do not use separate users, everything is written under
Marc> the same uid.

So?  

Marc> 3. most amateurs use a crappy mysql as backend for virtual
Marc> users. The likelihood of that being compromised compared to the
Marc> linux os is much and much higher.

How would it be compromised?  What makes you think that the backend
database is even exposed to the internet at all?  In a smart setup,
it's configured so that only local access works, or only access from a
restricted set of IPs with restricted logins is allowed access.

Marc> 4. Say you are more professional and setup an ldap server (with
Marc> correct acls (which is not trivial at all)) If you would have
Marc> dovecot use it as a backend for virtual users. Does dovecot
Marc> relay that user auth information or does it need some static
Marc> bind. The static bind is already an increased attack
Marc> surface. Better is have the os use the ldap backend and have
Marc> dovecot use the os.

The static bind is fine, because you do not bind to AD as a root user,
but only as a user with the minimum needed access to do the queries.  

Marc> 5. I would even argue that having dovecot 'outsource' the user
Marc> management to the linux os is more secure. Because dovecot
Marc> developers are more experienced in programming the email
Marc> application and have far less experience with authorization,
Marc> authentication than the linux developers. There is much more
Marc> scrutiny on the linux os than the dovecot user system.

You really don't know how authentication and access to IMAP mailboxes
works, do you?  And how postfix submission port works.  Regular port
25 SMTP traffic doesn't have access controls, but it's also not where
you accept email that gets sent to other domains, you only accept
email for your destination domains.

Submission port 587, for accepting outgoing email to be sent outside
your your domain, needs and requires authentication.  It's part of the
specs that mail clients need to implement properly. 

>> and more scalable.

Marc> Not relevant, that is different discussion.
 
>> General users don't need accounts on the mail server, and security in
>> depth argues that keeping them off the server entirely is a good
>> thing.
>> 

Marc> You constantly apply incorrect logic. You think that "keeping
Marc> them off the server entirely" equals virtual user. "keeping them
Marc> off the server entirely" also includes /sbin/nologin.  According
Marc> to your incorrect logic’s, you support my statement because in
Marc> my case users are kept off.

Again, you're not being clear here. 

Marc> If your logic’s is incorrect, how can your conclusion be
Marc> correct? Repeating this does not make it true, the alternative
Marc> is far worse.

You're telling me my logic is broken, but I keep giving you reasons
why I stand by my assertion that having virtual users is more secure,
because it lowers the attack surface.  

Marc> Linux always does a better job on permissions, users,
Marc> authentication than whatever 3rd party software. And if you
Marc> outsource this to linux you have even more possibilities by
Marc> using selinux rules.

You need to think of security happening in layers.  Keeping users

RE: silly quesiton

[John Stoffel]

> "Marc" == Marc   writes:

>> So just to be clear, each user has a login on your mail server in
>> /etc/passwd?  If so, I would strongly urge you to move to using only
>> virtual users on your mail infrastructure.
>> 

Marc> Why? Just disallow login, and that is from the perspective that
Marc> a mail user should be limited mail resources.

If the user does NOT need to login to the dovecot/mail servers, then
not having these users at all is more secure. 

Marc> I argue exactly the opposite. Keep as much as possible linux
Marc> users. As linux has been engineered for allowing multiple user
Marc> accounts, and most other virtual user providers that are used
Marc> here, have not.

I'm having a hard time to parse what you are saying here.

I'm saying that if the mail/dovecot server is only providing mail
services, then putting all the users (across multiple domains even)
into a virtual user database is more secure and more scalable.

General users don't need accounts on the mail server, and security in
depth argues that keeping them off the server entirely is a good
thing.

John

Re: silly quesiton

[John Stoffel]

steph> Up to now, I used PAM of each user in order to send and receive
steph> email. ( BTW, sending email, a use authentication was required
steph> and we used the login and passwd of the user on the system

So just to be clear, each user has a login on your mail server in
/etc/passwd?  If so, I would strongly urge you to move to using only
virtual users on your mail infrastructure.

steph> Now, for dovecot, I start to use MD5 passwrd.. and that sounds to be OK

steph> auth_mechanisms = plain login cram-md5
steph> passdb {
steph>   driver = passwd-file
steph>   # Path for passwd-file. Also set the default password scheme.
steph>   args = scheme=cram-md5 /etc/cram-md5.pwd
steph> }


steph> But changing the passwrd for the user1..  he can retrieve
steph> emails from dovecot, but cannot send anymore, because sending
steph> emails kept the old passwrd. ( using the PAM)

What is your mail software?  I assume you are having your users
connect to port 587 to submit emails to be sent out, correct?  If so,
are you using postfix, exim, sendmail or some other mailer to access
email submissions and then send them out?  If so, you should be able
to configure your mail server to use the same password file as your
new md5 password file. 

steph> 1) How can I says sendmail to use the same passwd file ( with MD5) than 
dovecot ?

Ah... just saw this.  And I don't know how to configure sendmail for
this.  I would suggest you look on the sendmail.org site for help.  

steph> 2) Ideally, I would like to create virtual users for the same
steph> mailbox  Is that possible ?

steph> like 2 files Users and PAsswrds pointing out the mailbox :
steph> maildir :/home/mailbox/user1 ex : us...@foo.com  passwrd1 
steph> /home/mailbox/generic_mails and user2 passwrd2 
steph> home/mailbox/generic_mails

I do this myself using postfix and dovecot and it works well.  I have
my users defined in an sqlite3 DB, though for a small number of users
I think a flat file is simpler.

The trick is to have the dovecot and postfix/sendmail using the same
files for the virtual users and their passwords.  There are a number
of tutorials out there for doing this.

John

Re: noob maildir question

[John Stoffel]

mikfum> thanks John for the reply

No problem! I'm not an expert by any stretch, but I've been using
dovecot for years and doing It for way too many years... LOL!

mikfum> what I would like to do is implement an autoarchive function
mikfum> at server level that, in the night while dovecot is down,
mikfum> moves messages older than n days from the user inbox to a
mikfum> subfolder of the same user (cur to cur)

Why do you bring dovecot down?  What maintenance are you running them?
I'm curious because I never reboot my dovecot instance unless there's
a problem.  And these days, if you are running a business providing
email service, it seems better to run a cluster of dovecot servers
behind dovecot director to load balance things.

I also feel that using the doveadm commands to do this work is the
better way, since it will properly handle locking and consistency of
the folder(s).

Why do you think that doing this with dovecot is down is the best way
to do this?

John

Re: noob maildir question

[John Stoffel]

> "mikfum" == mikfum   writes:

mikfum> I would like to ask if it is an acceptable practice to manage
mikfum> messages in the maildir as a file (move them from one folder
mikfum> to another) while dovecot is in stop state thinking that it
mikfum> will be rebuild to the next imap user login

No, it's not a good idea, bad things might happen.

what are you trying to accomplish?  Maybe we can give a suggestion if
we know what you are trying to accomplish.  Don't assume that a
certain method is the only way, just talk about the problem and what
you wnat to achieve, not HOW you want to achieve it.

Cheers,
John

Re: source code doesn't compile

[John Stoffel]

> "Ruben" == Ruben Safir  writes:

Ruben> On Wed, Jan 12, 2022 at 09:37:12AM +0200, Aki Tuomi wrote:
>> 
>> > On 12/01/2022 08:20 Ruben Safir  wrote:
>> > 
>> >  
>> > On 1/12/22 01:06, Aki Tuomi wrote:
>> > > I tried to reproduce this issue on debian stretch, but it worked just 
>> > > fine. I suspect your distro is just too old for 2.3. Can you see if 
>> > > 2.2.36 works better?
>> > 
>> > 
>> > something in the autoconf config caused it to try to put auth and the
>> > auth directory in the same local.. that should narrow the issue to a
>> > couple of lines of config code.  I am not an expert in autoconf
>> > 
>> > -- 
>> 
>> There is limited amount of interest in trying to fix old operating systems, 
>> unfortunately. Especially as there is no such thing as "couple of lines of 
>> autoconf code".
>> 
>> Aki

Ruben> That is perfectly understandable. 

Ruben> However, there should be interest that ./configure doesn't contruct a 
make file
Ruben> which steps on itself which is a condition that should never happen.
Ruben> Autoconf tools are supposed to handle these problems.  It should compile
Ruben> from Slackware to Gentoo to Red Hat Enterprise.  At the end of the day,
Ruben> they are all posix compliant systems.

Ruben> make install tried to first make a file
Ruben> called auth and then tries to use the same 
Ruben> location on the file system to
Ruben> make a directory.  That is a fixable bug.

I think people would be happy to help out more if you would provide
more details on this appliance and the exact version of OpenSUSE (or
not) it is based off of.  Giving a list of the packages installed and
their versions will also help.

The config.log output from the configure script would also help.

Assume people are trying to be helpful when they reply to "use the
packaged versions if possible" instead of being mean or snarky to
you.

Compiling from source isn't always easy, I've been doing it for 30+
years and I remember all the hell I went through getting stuff to
compile on DEC Ultrix, DEC OSF/1, SunOS 4, Solaris 1.x, Irix 4 (?
forget the version), HP-UX and AIX all at the same time.  It was
painful at times and exposed all kinds of issues.

These days most linux distros are much better, but as newer tools and
security and encryption gets deployed, sometimes it's hard to keep old
appliances with old base OS images supported.

So just send alot of details and people will see what they can do.
Don't try to hide versions or details if possible.  Assume we know
nothing of your setup.

Cheers,
John

Re: 2.3.17 broken on CentOS8 / bug

[John Stoffel]

>>>>> "Aki" == Aki Tuomi  writes:

Aki> You are correct that the problem is not fully fixed yet. It,
Aki> however, only affects practically cases where you do doveadm -c
Aki> /path 

Thanks for the update.

Aki> We will fix it properly in a future release, now it has been
Aki> fixed to work as it used to before, so no new regression is
Aki> introduced.

As long as no one trips over this issue with too long certs some other way.

>> On 03/11/2021 14:54 John Stoffel  wrote:
>> 
>> 
>> >>>>> "Aki" == Aki Tuomi  writes:
>> 
Aki> This issue is now fixed for Dovecot on master with
Aki> https://github.com/dovecot/core/compare/ca2237e%5E..6fff8d5.patch
>> 
>> Looking at the patch, I've got a couple of comments.
>> 
>> 1. Even your added comment says this issue could still happen is
>> doveadm reads the config setting through doveconf, instead of the
>> config socket.  To me that smells like the problem isn't really where
>> you patched it, but more in the parsing of options in doveadm.
>> 
>> 2. This is much more bike-shedding, but you have the following:
>> 
>> -if (input->module != NULL || input->extra_modules != NULL) {
>> +if ((service->flags & MASTER_SERVICE_FLAG_DISABLE_SSL_SET) ==
>> 0 &&
>> + (input->module != NULL || input->extra_modules != NULL)) {
>> 
>> And I would think that the last line would be more readable with:
>> 
>> (input->module || input->extra_modules)) {
>> 
>> The != NULL test just seems really redundant.  I haven't looked at the
>> rest of the main.c to see if this pattern is repeated all over the
>> place or not.
>> 
>> John
>> 
>> 
Aki> and for pigeonhole master with
>> 
Aki> 
https://github.com/dovecot/pigeonhole/commit/29750ba54c20eea0afd4ca436ddc1325723ce93f.patch
>> 
Aki> Regards,
Aki> Aki
>> 
>> >> On 01/11/2021 08:38 Aki Tuomi  wrote:
>> >> 
>> >> 
>> >> Hi all!
>> >> 
>> >> We are looking into this issue.
>> >> 
>> >> Aki
>> >> 
>> >> > On 30/10/2021 19:36 TG Servers  wrote:
>> >> > 
>> >> > 
>> >> > Thanks Robert, I read that. I will also wait for a patch and stay
>> >> >  
>> >> >  Cheers
>> >> > 
>> >> > 
>> >> > On 30/10/2021 12:59, Robert Nowotny wrote:
>> >> > 
>> >> > > the reason is : 
>> >> > >  
>> >> > > ssl_ca = > >> > >  
>> >> > >  if "ca-bundle.crt"is too big, You will get that error.
>> >> > >  this should be fixed, but as a workaround You might pull out the 
>> >> > > certificates You need.
>> >> > >  I personally wait for the patch and stay at 2.3.16 for the time 
>> >> > > beeing.
>> >> > >  
>> >> > >  yours sincerely
>> >> > >  Robert
>> >> > > 
>> >> > >  
>> >> > > 
>> >> > > Am 30.10.2021 um 10:34 schrieb TG Servers:
>> >> > > 
>> >> > > > Hello,
>> >> > > >  
>> >> > > >  tonight my dovecot upgraded to 2.3.17 and completely broke on 
>> >> > > > recent CentOS 8 installation.
>> >> > > >  
>> >> > > >  I found the service in status 
>> >> > > >  
>> >> > > >  [root@riot ~]# systemctl status dovecot
>> >> > > >  ● dovecot.service - Dovecot IMAP/POP3 email server
>> >> > > >  Loaded: loaded (/usr/lib/systemd/system/dovecot.service; enabled; 
>> >> > > > vendor preset: disabled)
>> >> > > >  Active: failed (Result: exit-code) since Sat 2021-10-30 09:59:11 
>> >> > > > CEST; 58s ago
>> >> > > >  Docs: man:dovecot(1)
>> >> > > >  https://doc.dovecot.org/
>> >> > > >  Process: 1515 ExecStart=/usr/sbin/dovecot -F (code=exited, 
>> >> > > > status=89)
>> >> > > >  Process: 1429 ExecStartPre=/usr/libexec/dovecot/prestartscript 
>> >> > > > (code=exited, status=0/SUCCESS)
>> >> > > >  Main PID: 1515 (code=exited, status=89)
>> >> > > >  
>> >> > > >  Oct 30 09:59:10 riot..com systemd[1]: Starting Dovecot 
>> >

Re: 2.3.17 broken on CentOS8 / bug

[John Stoffel]

> "Aki" == Aki Tuomi  writes:

Aki> This issue is now fixed for Dovecot on master with
Aki> https://github.com/dovecot/core/compare/ca2237e%5E..6fff8d5.patch

Looking at the patch, I've got a couple of comments.

1. Even your added comment says this issue could still happen is
   doveadm reads the config setting through doveconf, instead of the
   config socket.  To me that smells like the problem isn't really where
   you patched it, but more in the parsing of options in doveadm.

2. This is much more bike-shedding, but you have the following:

-   if (input->module != NULL || input->extra_modules != NULL) {
+   if ((service->flags & MASTER_SERVICE_FLAG_DISABLE_SSL_SET) ==
0 &&
+ (input->module != NULL || input->extra_modules != NULL)) {

And I would think that the last line would be more readable with:

(input->module || input->extra_modules)) {

The != NULL test just seems really redundant.  I haven't looked at the
rest of the main.c to see if this pattern is repeated all over the
place or not.

John


Aki> and for pigeonhole master with

Aki> 
https://github.com/dovecot/pigeonhole/commit/29750ba54c20eea0afd4ca436ddc1325723ce93f.patch

Aki> Regards,
Aki> Aki

>> On 01/11/2021 08:38 Aki Tuomi  wrote:
>> 
>> 
>> Hi all!
>> 
>> We are looking into this issue.
>> 
>> Aki
>> 
>> > On 30/10/2021 19:36 TG Servers  wrote:
>> > 
>> > 
>> > Thanks Robert, I read that. I will also wait for a patch and stay
>> >  
>> >  Cheers
>> > 
>> > 
>> > On 30/10/2021 12:59, Robert Nowotny wrote:
>> > 
>> > > the reason is : 
>> > >  
>> > > ssl_ca = > > >  
>> > >  if "ca-bundle.crt"is too big, You will get that error.
>> > >  this should be fixed, but as a workaround You might pull out the 
>> > > certificates You need.
>> > >  I personally wait for the patch and stay at 2.3.16 for the time beeing.
>> > >  
>> > >  yours sincerely
>> > >  Robert
>> > > 
>> > >  
>> > > 
>> > > Am 30.10.2021 um 10:34 schrieb TG Servers:
>> > > 
>> > > > Hello,
>> > > >  
>> > > >  tonight my dovecot upgraded to 2.3.17 and completely broke on recent 
>> > > > CentOS 8 installation.
>> > > >  
>> > > >  I found the service in status 
>> > > >  
>> > > >  [root@riot ~]# systemctl status dovecot
>> > > >  ● dovecot.service - Dovecot IMAP/POP3 email server
>> > > >  Loaded: loaded (/usr/lib/systemd/system/dovecot.service; enabled; 
>> > > > vendor preset: disabled)
>> > > >  Active: failed (Result: exit-code) since Sat 2021-10-30 09:59:11 
>> > > > CEST; 58s ago
>> > > >  Docs: man:dovecot(1)
>> > > >  https://doc.dovecot.org/
>> > > >  Process: 1515 ExecStart=/usr/sbin/dovecot -F (code=exited, status=89)
>> > > >  Process: 1429 ExecStartPre=/usr/libexec/dovecot/prestartscript 
>> > > > (code=exited, status=0/SUCCESS)
>> > > >  Main PID: 1515 (code=exited, status=89)
>> > > >  
>> > > >  Oct 30 09:59:10 riot..com systemd[1]: Starting Dovecot 
>> > > > IMAP/POP3 email server...
>> > > >  Oct 30 09:59:11 riot..com dovecot[1515]: doveconf: Fatal: 
>> > > > execvp(/usr/libexec/dovecot/managesieve) failed: Argument list too long
>> > > >  Oct 30 09:59:11 riot..com dovecot[1515]: doveconf: Error: 
>> > > > managesieve-login: dump-capability process returned 89
>> > > >  Oct 30 09:59:11 riot..com dovecot[1515]: doveconf: Fatal: 
>> > > > execvp(/usr/sbin/dovecot) failed: Argument list too long
>> > > >  Oct 30 09:59:11 riot..com systemd[1]: dovecot.service: Main 
>> > > > process exited, code=exited, status=89/n/a
>> > > >  Oct 30 09:59:11 riot..com systemd[1]: dovecot.service: Failed 
>> > > > with result 'exit-code'.
>> > > >  Oct 30 09:59:11 riot..com systemd[1]: Failed to start Dovecot 
>> > > > IMAP/POP3 email server.
>> > > >  
>> > > >  This seems to be like a bug as no configuration was changed by me in 
>> > > > the middle of the night.
>> > > >  I recall there were similar errors/bug reports in the past were it 
>> > > > seemed it was managesieve but wasn't, people had some 
>> > > > misconfigurations in the dovecot.conf. I did not change my 
>> > > > dovecot.conf since April.
>> > > >  But maybe here it is a pigeonhole issue.
>> > > >  
>> > > >  As I did not find any reason for it I changed the repo and downgraded 
>> > > > to 2.3.16-2 now and it runs without any flaws, like all the time 
>> > > > before. I had no time to investigate this any longer thand 2 hours 
>> > > > with 2.3.17 installed as this is a production server and I need the 
>> > > > email access. I also did not find anything adressable in the logs.
>> > > >  
>> > > >  [root@riot dovecot]# systemctl status dovecot
>> > > >  ● dovecot.service - Dovecot IMAP/POP3 email server
>> > > >  Loaded: loaded (/usr/lib/systemd/system/dovecot.service; enabled; 
>> > > > vendor preset: disabled)
>> > > >  Active: active (running) since Sat 2021-10-30 10:18:11 CEST; 2s ago
>> > > >  Docs: man:dovecot(1)
>> > > >  https://doc.dovecot.org/
>> > > >  Process: 32398 ExecStartPre=/usr/libexec/dovecot/prestartscript 
>> > > > (code=exited, status=0/SUCCESS)
>> > > > 

Re: How can I always send a vacation response with sieve?

[John Stoffel]

> "Steve" == Steve Dondley  writes:

>> So share your solution!  Just because you found a solution, doesn't
>> mean others won't run into the same problem...  *hint* *hint*

Steve> My solution had nothing to do with dovecot. The solution
Steve> involved hacking the php code of an ancient cms so that your
Steve> could reply directly to the person who filled out an email
Steve> form.

Thanks for that info, since it clarified that the solution you found
wasn't dovecot related, even though you posted on the mailing list.

https://imgs.xkcd.com/comics/wisdom_of_the_ancients.png

*grin*

Re: How can I always send a vacation response with sieve?

[John Stoffel]

> "Steve" == Steve Dondley  writes:

Steve> On 2021-09-04 05:50 PM, Marc wrote:
>> You do not want to do that because that can create loops.

Steve> Yeah, right after I posted this I did some more googling and someone 
Steve> else was saying the same thing.

Steve> I found another way around the problem I was trying to solve,
Steve> though. So I'm good. Thanks for your response.

So share your solution!  Just because you found a solution, doesn't
mean others won't run into the same problem...  *hint* *hint*

Re: Snarf plugin retirement

[John Stoffel]

> "Brent" == Brent Busby  writes:

Brent> I wondered if it's possible to get the Snarf plugin back, if not in the
Brent> official Dovecot distribution, then at least to somehow build the plugin
Brent> source on a modern release...because:

Brent> Though the documented purpose for Snarf was to ease migration
Brent> off of UW-IMAP, that's not the only thing it was useful for.
Brent> Unfortunately, every mailreader that I've seen that can run
Brent> within Emacs -- and I've tried Gnus, VM, and RMAIL -- all
Brent> _require_ the use of a ~/mbox file when pulling from a local
Brent> mail spool on the same machine.  (This does not affect IMAP
Brent> usage of those mailreaders, only local file access.)

I'm confused... since I'm using VM to read and write these emails, I'm
trying to understand your issue.  Basically, when I moved to IMAP for
my personal domain, I made the concious choice to only allow IMAP
access, since I wanted proper locking from both Phones, mutt, and
other IMAP clients.  So I had to stop using local email spools to get
email.

So if you're offering IMAP access to mail, just turn off the local
mail spool completely. 

Brent> Non-Emacs mailreaders like Alpine and Mutt have sufficiently
Brent> robust file-locking safety that they can work on the local mail
Brent> spool in /var/spool/mail or /var/mail or what have you directly
Brent> without copying your mail somewhere else first, so for Alpine
Brent> and Mutt, use of ~/mbox is possible but not necessary.  But it
Brent> seems that for every mailreader that does run within Emacs, if
Brent> the mail source is local rather than IMAP, there is no other
Brent> way to operate the program than to have it copy your mail from
Brent> the spool to ~/mbox.  On a system where there are users who
Brent> sometimes work that way, doing local mail access in Emacs, and
Brent> other times use IMAP via Dovecot, you have a mess without the
Brent> Snarf plugin.

You have a mess period.  Don't do this.  It's a really bad idea and
trying to make it work is just too painful.  Now I *do* use a mix of
IMAP and local mbox format folders, where once I save it locally via
mbox, I can't acces those folders using my IMAP only clients.  But
that's an accepted and known limitation.

Brent> I realize this is a peculiar situation, so I don't expect Snarf
Brent> to necessarily be put back into Dovecot.  I was wondering how
Brent> it might be possible to build it today though...for any out
Brent> there who may use Emacs for mail most of the time...but
Brent> occasionally also want Dovecot IMAP on the same machine where
Brent> they're normally using an ~/mbox file for Emacs' sake.  UW-IMAP
Brent> may be dead...but long live GNU Emacs!

Just don't use VM without IMAP as your main mailbox any more.  And
also accept that once you get over a thousand or so emails, or lots of
large emails, then VM inside emacs really starts to suck.

I'd move to Mutt, but I haven't spent the time to re-create the VM
keybindings in mutt yet, which is what's holding me back.

And if you have users who want both types of access, just tell them
no.  It's *not* worth the hassle, and things *will* break.

John

Re: dovecot + Procmail -> junk emails

[John Stoffel]

>>>>> "Ted" == Ted Hatfield  writes:

Ted> On Mon, 2 Aug 2021, John Stoffel wrote:
>>>>>>> "Stephane" == Stephane Magnier  writes:
>> 
Stephane> Having moved from 'mbox' to 'Maildir', I'm trying to use a
Stephane> way to move a spam email to the Junk folder
>> 
>> Have you looked at 'sieve' and 'pigeonhole' setup?  procmail is old
>> old old and completely unsupported.
>> 
Stephane> I declared the junk folder as;
>> 
Stephane> namespace inbox {
Stephane> type = private
Stephane> disabled = no
Stephane> ignore_on_failure = no
Stephane>   list = yes
Stephane>     subscriptions = yes
Stephane> #hidden = no
Stephane> #inbox = yes
Stephane> #prefix = INBOX/
>> 
Stephane>   mailbox "Drafts" {
Stephane>       auto = subscribe
Stephane>     special_use = \Drafts
Stephane>   }
Stephane>   mailbox "Junk" {
Stephane>       auto = subscribe
Stephane>     autoexpunge = 12 weeks
Stephane>     special_use = \Junk
Stephane>   }
>> 
Stephane>   mailbox "Trash" {
Stephane>       auto = no
Stephane>     autoexpunge = 12 weeks
Stephane>     special_use = \Trash
Stephane>   }
>> 
Stephane>   mailbox "Sent" {
Stephane>       auto = subscribe
Stephane>     special_use = \Sent
Stephane>   }
>> 
Stephane> and spamassassin, very basic
>> 
Stephane> loadplugin Mail::SpamAssassin::Conf
>> 
Stephane> required_hits 5
Stephane> required_score 5
Stephane> rewrite_header subject [** SPAM ** ]
Stephane> add_header all Status _YESNO_, score=_SCORE_ required=_REQD_ 
tests=_TESTS_ autolearn=
Stephane> _AUTOLEARN_ version=_VERSION_
Stephane> spf_timeout 60
>> 
Stephane> rewrite_header to [Email could be SPAM ]
Stephane> fold_headers 0
Stephane> #defang_mime 0
Stephane> report_safe 0
>> 
Stephane> and Procmail.. basic and I used what I found  on the webpage :
Stephane> https://wiki.dovecot.org/procmail
>> 
>> This is probably old and should be removed or replaced.  I have
>> /etc/dovecot/conf.d/90-sieve.conf which looks like this:
>> 
>> plugin {
>> sieve = ~/.dovecot.sieve
>> sieve_dir = ~/sieve
>> sieve_global_dir =
>> sieve_before = /var/lib/dovecot/sieve/before.d
>> sieve_after = /etc/dovecot/sieve-after
>> sieve_extensions = +spamtest +spamtestplus
>> sieve_spamtest_status_type = score
>> sieve_spamtest_status_header = X-Spamd-Result: default: [[:alnum:]]+ 
>> [(-?[[:digit:]]+\.[[:digit:]]+) / -?[[:digit:]]+\.[[:digit:]]+\]
>> sieve_spamtest_max_header= X-Spamd-Result: default: [[:alnum:]]+ 
>> [-?[[:digit:]]+\.[[:digit:]]+ / (-?[[:digit:]]+\.[[:digit:]]+)\]
>> }
>> 
>> And I have spamassissin scoring my mails, this script to move them
>> into 'Junk' as needed:
>> 
>> # more /etc/dovecot/sieve-after/spam-to-folder.sieve
>> require ["fileinto","mailbox"];
>> 
>> if header :contains "X-Spam-Flag" "YES" {
>> fileinto :create "Junk";
>> stop;
>> }
>> 

Ted> There are still a number of people using procmail.  Including me
Ted> for instance.  Rather than removing the page perhaps updating the
Ted> article with a reference to a replacement instead but leaving the
Ted> procmail documentation in place.

Certainly that's doable, but I don't have any access to that web page
for updates, but... I honestly haven't checked either!  LOL!  Just
replying from what I've seen over the years.

I too used to use procmail and it did a great job, but with Dovecot
and IMAP, having the integrated sieve protocol setup makes alot of
sense to keep down complexity.  Even thoguh you now have to learn a
new language to filter emails, but it's not that hard.

John

Re: dovecot + Procmail -> junk emails

[John Stoffel]

> "Stephane" == Stephane Magnier  writes:

Stephane> Having moved from 'mbox' to 'Maildir', I'm trying to use a
Stephane> way to move a spam email to the Junk folder

Have you looked at 'sieve' and 'pigeonhole' setup?  procmail is old
old old and completely unsupported.  

Stephane> I declared the junk folder as;

Stephane> namespace inbox {
Stephane> type = private
Stephane> disabled = no
Stephane> ignore_on_failure = no
Stephane>   list = yes
Stephane>     subscriptions = yes
Stephane> #hidden = no
Stephane> #inbox = yes
Stephane> #prefix = INBOX/
   
Stephane>   mailbox "Drafts" {
Stephane>       auto = subscribe
Stephane>     special_use = \Drafts
Stephane>   }
Stephane>   mailbox "Junk" {
Stephane>       auto = subscribe
Stephane>     autoexpunge = 12 weeks
Stephane>     special_use = \Junk
Stephane>   }
   
Stephane>   mailbox "Trash" {
Stephane>       auto = no
Stephane>     autoexpunge = 12 weeks
Stephane>     special_use = \Trash
Stephane>   }
   
Stephane>   mailbox "Sent" {
Stephane>       auto = subscribe
Stephane>     special_use = \Sent
Stephane>   }

Stephane> and spamassassin, very basic
   
Stephane> loadplugin Mail::SpamAssassin::Conf
   
Stephane> required_hits 5
Stephane> required_score 5
Stephane> rewrite_header subject [** SPAM ** ]
Stephane> add_header all Status _YESNO_, score=_SCORE_ 
required=_REQD_ tests=_TESTS_ autolearn=
Stephane> _AUTOLEARN_ version=_VERSION_
Stephane> spf_timeout 60
   
Stephane> rewrite_header to [Email could be SPAM ]
Stephane> fold_headers 0
Stephane> #defang_mime 0
Stephane> report_safe 0

Stephane> and Procmail.. basic and I used what I found  on the webpage : 
Stephane> https://wiki.dovecot.org/procmail
   
This is probably old and should be removed or replaced.  I have
/etc/dovecot/conf.d/90-sieve.conf which looks like this:

plugin {
  sieve = ~/.dovecot.sieve
  sieve_dir = ~/sieve
  sieve_global_dir =
  sieve_before = /var/lib/dovecot/sieve/before.d
  sieve_after = /etc/dovecot/sieve-after
  sieve_extensions = +spamtest +spamtestplus
  sieve_spamtest_status_type = score
  sieve_spamtest_status_header = X-Spamd-Result: default: [[:alnum:]]+ 
[(-?[[:digit:]]+\.[[:digit:]]+) / -?[[:digit:]]+\.[[:digit:]]+\]
  sieve_spamtest_max_header= X-Spamd-Result: default: [[:alnum:]]+ 
[-?[[:digit:]]+\.[[:digit:]]+ / (-?[[:digit:]]+\.[[:digit:]]+)\]
}

And I have spamassissin scoring my mails, this script to move them
into 'Junk' as needed:

   # more /etc/dovecot/sieve-after/spam-to-folder.sieve
   require ["fileinto","mailbox"];

   if header :contains "X-Spam-Flag" "YES" {
 fileinto :create "Junk";
 stop;
   }


Cheers,
John

Re: Disable authentication for submission service

[John Stoffel]

> "Dan" == Dan Conway  writes:

Are you sure?  I know that postfix can use the same backend database
for authentication as dovecot, and dovecot can be the master, but dovecot
does NOT listen on port 25 or 587 at all, those are all just used by
Postfix.


Dan> Yes Dovecot will proxy the connection to the real MTA. My
Dan> question is why authentication is always required on Dovecot when
Dan> submission is used, as MTAs usually have an option to allow
Dan> non-authenticated relaying.

Dan> On 7/28/21 10:19 AM, justina colmena ~biz wrote:

Dan> I am quite curious about the circumstances of this question. I was not 
aware that Dovecot
Dan> actually offered mail submission service. If Dovecot does offer such a 
service, then it will
Dan> have to relay the submitted mail to the real MTA, which is very likely 
not Dovecot. At the
Dan> moment I have Postfix set up as MTA for that purpose —
   
Dan> Relaying on port 25 is usually quick and easy to whitelist for certain 
permitted hosts, but
Dan> otherwise port 587, optionally with STARTTLS, and/or port 465 with 
SSL/TLS is generally set up
Dan> for user authenticated mail submissions.
   
Dan> See also:
Dan> 
https://www.mailgun.com/blog/which-smtp-port-understanding-ports-25-465-587/

Dan> On July 28, 2021 6:10:28 AM AKDT, Dan Conway  
wrote:
   
Dan> Hello,

Dan> Is it possible to disable the requirement for authentication on 
the 
Dan> submission service? I'm trying to require authentication for all, 
except 
Dan> for a handful of IP addresses.

Dan> Thank you.

Dan> ehlo test.com
Dan> 250-aaa
Dan> 250-AUTH PLAIN LOGIN
Dan> 250-BURL imap
Dan> 250-CHUNKING
Dan> 250-DSN
Dan> 250-ENHANCEDSTATUSCODES
Dan> 250-SIZE
Dan> 250 PIPELINING
Dan> MAIL FROM:
Dan> 530 5.7.0 Authentication required.

Dan> --
Dan> Sent from my Android device with K-9 Mail. Please excuse my brevity.

Re: good options for Multiple users on a common email account

[John Stoffel]

> "Pat" == Pat G  writes:


Pat> i ve a mail server to manage with some email accounts but with
Pat> multiples users (+50) using a common email.  it indicates
Pat> sometimes that it can't connect cause too many connections.

Can you post some log messages from around when this happens?  It
should be easy enough to generate loadby running abunch of fetchmail
processes or other imap CLI tools against those shared accounts. 

Pat> what are the good options to allow a lot of users for a specific
Pat> account ?

I'm honestly not sure, but it shouldn't care whether it's one account
logged in multiple time, or a whole bunch of accounts all accessing
their own mailboxes.  

Pat> i modifyied these options :
Pat> auth_worker_max_count = 60
Pat> mail_max_userip_connections = 60

Pat> is it sufficient ?

Pat> dovecot -n :

Pat> auth_worker_max_count = 60
Pat> mail_location = maildir:/home/%u/Maildir
Pat> namespace inbox {
Pat>   inbox = yes
Pat>   location = 
Pat>   mailbox Drafts {
Pat> special_use = \Drafts
Pat>   }
Pat>   mailbox Junk {
Pat> special_use = \Junk
Pat>   }
Pat>   mailbox Sent {
Pat> special_use = \Sent
Pat>   }
Pat>   mailbox "Sent Messages" {
Pat> special_use = \Sent
Pat>   }
Pat>   mailbox Trash {
Pat> special_use = \Trash
Pat>   }
Pat>   prefix = 
Pat> }
Pat> passdb {
Pat>   driver = pam
Pat> }
Pat> protocols = " imap"
Pat> service imap-login {
Pat>   inet_listener imaps {
Pat> port = 993
Pat> ssl = yes
Pat>   }
Pat> }
Pat> ssl_cert =  ssl_key =  # hidden, use -P to show it
Pat> userdb {
Pat>   driver = passwd
Pat> }
Pat> protocol imap {
Pat>   mail_max_userip_connections = 60
Pat> }

I would just bump those two numbers up and load test with an IMAP
client you can script out.  Shouldn't be too hard to do.

John

Re: Mapping usernames used for authentication to UNIX usernames

[John Stoffel]

Frank> I'm looking for some advice or pointers how to best solve a
Frank> small problem that I have. I have no doubt that this can be
Frank> done in dovecot, but I'm struggling to find the easiest way to
Frank> implement it.

Frank> First of all, what I have:
Frank> I have a relative small dovecot setup for a dozen domains, and about 50 
Frank> users in total. All users use IMAP to retrieve mail, and SMTP submission 
Frank> protocol to submit email.

Frank> Because of the small size, every user has its own UNIX account,
Frank> authentication is done using PAM and mail is stored in a
Frank> Maildir folder in their home directory.  Works perfectly!

Do these users ever login and use their Unix account?  Or do they only
access the system via IMAP to read email?  If this, then I would
completely move away from local accounts and unix home dirs and just
use virtual users instead.  Then you login with your email address and
password to get mail.  Much simpler!

Frank> There is one minor inconvenience. When a new mail client is
Frank> configured, users (often guided by the auto config generator of
Frank> the mail client) tend to use their email address as the
Frank> username to authenticate instead of their UNIX account name,
Frank> which fails of course.

Frank> Would it be possible to configure something that will map an
Frank> email address to the UNIX account name and use the account name
Frank> for authentication and obtaining the related information (uid,
Frank> gid, home dir)?

Frank> I do have two concerns:

Frank> 1) I do not want to break existing mail configurations, so 
Frank> authentication with the UNIX username should still be possible.

I think you can have multiple usernames pointing to the same backend
account, so moving to virtual users would be even simpler.

Frank> 2)  I cannot do a simple reg. exp for the translation because
Frank> every email domain has e.g. an i...@domain.com mailbox, and I
Frank> do not want them all to go to UNIX user "info".

Even if you do offer Unix logins, I would still seperate the user
email logins from the Unix logins.  Just having all email access
happen via IMAP makes things simpler.  And if they want to read email
from their unix acocunt, a text based IMAP tool like mutt should be
good enough.

John

Re: What imap ssl/auth settings work best with MS Outlook?

[John Stoffel]

> "@lbutlr" == @lbutlr   writes:

@lbutlr> On 30 Apr 2021, at 01:20, Arjen de Korte  
wrote:
>> Citeren "@lbutlr" :
>> 
>>> When you enter your email address, it would be TRIVIAL to check the MX 
>>> records for the domain and fill those in for the SMTP and IMAP servers, 
>>> allowing users to more easily add (if needed) the domain prefix.
>>> 
>>> No one does this.
>> 
>> Rightfully so. There is absolutely no guarantee that the server on the 
>> inbound (MX) record also handles outbound and/or IMAP. In many cases, these 
>> will be different systems.

lbutlr> It is very very common. It's been at least a decade since I
lbutlr> saw a configuration in which the SMTP/IMAP servers were on a
lbutlr> different domain than the MX domain.

My current $WORK used to have different incoming MX servers vs the
outgoing, since we used an external spam filtering service.  

John

Re: Emails to multiple recipients on same server not getting delivered

[John Stoffel]

> "Steve" == Steve Dondley  writes:

>> So where are you calling Spamassassin for each email?  Hmm... maybe
>> you need to have -d ${recipient} in your spamassassin call?
>> Or better yet, call the 'deliver' program from dovecot like I showed
>> isntead.
>> 
>> spamass-dovecot_destination_recipient_limit = 1
>> virtual_alias_maps = hash:/etc/postfix/virtual-alias-maps
>> virtual_mailbox_maps = sqlite:/etc/postfix/virtual_users.cf
>> virtual_transport = spamass-dovecot
>> 

Steve> I've simplified the configuration by turning off spamassassin and 
Steve> removing any mention of it from master.cf. Things are working now. But 
Steve> I'm still baffled why basically the same master.cf config is working on 
Steve> one server but not another. I wonder if different SA configs might be 
Steve> the problem.

Have you setup "spamassassin-recipient_limit = 1" in your postfix.cf
file, assuming you're still delivering with spamassassin in your
master.cf?

Also, what do the postfix logs show?  And the spamassassin logs?  And
syslog in general?

I suspect you're only delivering to the first recipient, in the email,
and not calling the delivery for each and every local recipient
individually.

John

Re: Emails to multiple recipients on same server not getting delivered

[John Stoffel]

> "Steve" == Steve Dondley  writes:

Steve> When I send an email to a single user on a server, it is received by the 
Steve> user without a problem. But when sending to multiple users, the emails 
Steve> disappear into a black hole. The logs contain no errors and indicate the 
Steve> emails were sent:

How is postfix delivering email to dovecot?  You give your postconf
output, but we need to see oyur master.cf file as well.  I use the
spamass-milter in my postfix config, and my master.cf has something
like this:

# Updated to support + addressing, 20210402
spamass-dovecot unix - n   n   -   -   pipe
  flags=DRhu user=mail:mail argv=/usr/bin/spamc -u debian-spamd -e
  /usr/lib/dovecot/deliver -a ${recipient} -d ${user}@${domain}
  
As you can guess, my mail setup runs on Debian Buster.  So the details
can matter.  But all my emails are delivered through spamassisin, then
filtered with Sieve on the header into the INBOX or Junk folders.


Steve> Apr  5 13:10:29 email postfix/pipe[31703]: F3A912027D: 
Steve> to=, relay=spamassassin, delay=1.6, 
Steve> delays=0.12/0/0/1.5, dsn=2.0.0, status=sent (delivered via spamassassin 
Steve> service (X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on 
Steve> email.example.org X-Spam-Level:  X-Spam-Stat))

Steve> Apr  5 13:10:29 email postfix/pipe[31703]: F3A912027D: 
Steve> to=, relay=spamassassin, delay=1.6, 
Steve> delays=0.12/0/0/1.5, dsn=2.0.0, status=sent (delivered via spamassassin 
Steve> service (X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on 
Steve> email.example.org X-Spam-Level:  X-Spam-Stat))

Steve> However, when I check the inboxes for the recipients, the email is 
Steve> nowhere to be found. I tried lifting the receiving/concurrent limits but 
Steve> to no effect. I have other servers with very similar configurations to 
Steve> this one but I'm not having issues with them. lmtp is the local delivery 
Steve> agent.

Steve> My postconf:

Steve> alias_maps = hash:/etc/aliases
Steve> biff = no
Steve> broken_sasl_auth_clients = yes
Steve> command_directory = /usr/sbin
Steve> compatibility_level = 2
Steve> daemon_directory = /usr/lib/postfix/sbin
Steve> data_directory = /var/lib/postfix
Steve> debugger_command = PATH=/bin:/usr/bin:/usr/local/bin:/usr/X11R6/bin ddd 
Steve> $daemon_directory/$process_name $process_id & sleep 5
Steve> default_destination_concurrency_limit = 5
Steve> home_mailbox = Maildir/
Steve> inet_interfaces = all
Steve> lmtp_destination_concurrency_limit = 5
Steve> lmtp_destination_recipient_limit = 5
Steve> local_destination_concurrency_limit = 5
Steve> local_destination_recipient_limit = 5
Steve> mail_owner = postfix
Steve> mailbox_size_limit = 3145728000
Steve> mailbox_transport = lmtp:unix:private/dovecot-lmtp
Steve> mailq_path = /usr/bin/mailq
Steve> message_size_limit = 26214400
Steve> milter_default_action = accept
Steve> milter_protocol = 6
Steve> mydestination = $myhostname localhost.$mydomain localhost $mydomain
Steve> mydomain = example.org
Steve> myhostname = email.example.org
Steve> mynetworks_style = subnet
Steve> myorigin = example.org
Steve> non_smtpd_milters = $smtpd_milters
Steve> policyd-spf_time_limit = 3600
Steve> recipient_bcc_maps = pcre:/etc/postfix/recipient_bcc
Steve> recipient_delimiter = +
Steve> sendmail_path = /usr/sbin/sendmail
Steve> setgid_group = postdrop
Steve> smtp_tls_note_starttls_offer = yes
Steve> smtp_tls_session_cache_database = btree:${data_directory}/smtp_scache
Steve> smtp_use_tls = yes
Steve> smtpd_banner = $myhostname ESMTP
Steve> smtpd_milters = unix:/opendkim/opendkim.sock
Steve> smtpd_recipient_restrictions = permit_mynetworks, 
Steve> permit_sasl_authenticated, reject_unauth_destination, 
Steve> check_policy_service unix:private/policyd-spf
Steve> smtpd_relay_restrictions = permit_mynetworks permit_sasl_authenticated 
Steve> defer_unauth_destination
Steve> smtpd_sasl_auth_enable = yes
Steve> smtpd_sasl_local_domain = $myhostname
Steve> smtpd_sasl_path = private/auth
Steve> smtpd_sasl_security_options = noanonymous
Steve> smtpd_sasl_type = dovecot
Steve> smtpd_sender_login_maps = pcre:/etc/postfix/login_maps.pcre
Steve> smtpd_sender_restrictions = reject_authenticated_sender_login_mismatch
Steve> smtpd_tls_cert_file = 
Steve> /etc/letsencrypt/live/email.example.org/fullchain.pem
Steve> smtpd_tls_key_file = /etc/letsencrypt/live/email.example.org/privkey.pem
Steve> smtpd_tls_session_cache_database = btree:${data_directory}/smtpd_scache
Steve> smtpd_use_tls = yes
Steve> unknown_local_recipient_reject_code = 550
Steve> virtual_alias_maps = hash:/etc/postfix/virtual
Steve> virtual_mailbox_limit = 26214400
Steve> virtual_transport = lmtp:unix:private/dovecot-lmtp

Steve> And doveconf:

Steve> # 2.3.4.1 (f79e8e7e4): /etc/dovecot/dovecot.conf
Steve> # Pigeonhole version 0.5.4 ()
Steve> # OS: Linux 4.19.0-14-cloud-amd64 x86_64 Debian 10.9
Steve> # Hostname: email.example.org
Steve> auth_mechanisms = plain login
Steve> auth_username_format = %Ln
Steve> 

Re: Authentication segfault with Dovecot 2.3.13

[John Stoffel]

> "Harald" == Harald Leithner  writes:

Harald> we have a problem upgrading Dovecot from 2.2 to 2.3.13 on one
Harald> server it seems one user triggers a segfault while trying to
Harald> authenticate.

Can you get the user to change their password to not have quite some
special characters maybe?  Or maybe ask them if they have any special
characters?  There was a bug recently with passwords as I recall.  

In this case, it's probably a dovecot bug, but it's fixable for now if
the user changes their password.  I think.  

Harald> The user works fine with 2.2.36.4, our last update try mid-late 2020 
was 
Harald> aborted because of this reason.

Harald> While debugging the problem we noticed that the "auth_debug_passwords = 
Harald> no" option doesn't work at least in our logfiles are the passwords 
Harald> logged. We replaced the password in the log with 
Harald> "THIS_SHOULD_NOT_GET_LOGGED" and the user part with "user@redacted".

Harald> The user uses APOP for authentication, but other users login 
Harald> successfully with APOP.

Harald> Here is a stacktrace and a log dump:

Harald> Jan  6 16:29:44 mail kernel: auth[2208397]: segfault at ec ip 
Harald> 7f67fc147174 sp 7ffeed993150 error 4 in 
Harald> libdovecot.so.0.0.0[7f67fc06e000+fc000]
Harald> Jan  6 16:29:44 mail kernel: Code: 1f 80 00 00 00 00 41 54 e8 79 fd ff 
Harald> ff 31 f6 49 89 c4 48 89 c7 31 c0 e8 ca f8 ff ff 4c 89 e0 41 5c c3 0f 1f 
Harald> 40 00 53 48 89 fb  87 ec 00 00 00 04 75 43 48 83 3d 7b aa 0a 00 00 
Harald> 0f 85 50 15 f4
Harald> Jan  6 16:29:44 mail systemd[1]: Started Process Core Dump (PID 
Harald> 2208677/UID 0).
Harald> Jan  6 16:29:44 mail systemd-coredump[2208678]: Process 2208397 (auth) 
Harald> of user 489 dumped core.#012#012Stack trace of thread 2208397:#012#0 
Harald> 0x7f67fc147174 event_create_passthrough (libdovecot.so.0 + 
Harald> 0x116174)#012#1  0x555678812d6e auth_request_finished_event (auth + 
Harald> 0x1bd6e)#012#2  0x5556788159ae auth_request_log_finished (auth + 
Harald> 0x1e9ae)#012#3  0x555678816ee0 n/a (auth + 0x1fee0)#012#4 
Harald> 0x555678826dc1 passdb_handle_credentials (auth + 0x2fdc1)#012#5 
Harald> 0x555678816c7e n/a (auth + 0x1fc7e)#012#6  0x555678824f27 n/a 
Harald> (auth + 0x2df27)#012#7  0x55567881b02d 
Harald> auth_request_handler_auth_begin (auth + 0x2402d)#012#8 
Harald> 0x55567880dfaf n/a (auth + 0x16faf)#012#9  0x7f67fc143a79 
Harald> io_loop_call_io (libdovecot.so.0 + 0x112a79)#012#10 0x7f67fc144ae2 
Harald> io_loop_handler_run_internal (libdovecot.so.0 + 0x113ae2)#012#11 
Harald> 0x7f67fc143b21 io_loop_handler_run (libdovecot.so.0 + 
Harald> 0x112b21)#012#12 0x7f67fc143ce0 io_loop_run (libdovecot.so.0 + 
Harald> 0x112ce0)#012#13 0x7f67fc0b96f3 master_service_run (libdovecot.so.0 
Harald> + 0x886f3)#012#14 0x55567880c2db main (auth + 0x152db)#012#15 
Harald> 0x7f67fbc9d042 __libc_start_main (libc.so.6 + 0x27042)#012#16 
Harald> 0x55567880c48e _start (auth + 0x1548e)

Harald> Jan  6 16:29:44 mail dovecot[2208071]: auth: Debug: client in: 
Harald> 
AUTH#011134#011PLAIN#011service=imap#011session=tgog/jy4erm5j7ZO#011lip=lan-ip#011rip=client-ip#011lport=143#011rport=47482
Harald> Jan  6 16:29:44 mail dovecot[2208071]: auth: Debug: client passdb out: 
Harald> CONT#011134
Harald> Jan  6 16:29:44 mail dovecot[2208071]: auth: Debug: client in: 
CONT
Harald> Jan  6 16:29:44 mail dovecot[2208071]: auth: Debug: 
Harald> sql(user@redacted,client-ip,): Performing passdb 
lookup
Harald> Jan  6 16:29:44 mail dovecot[2208071]: auth: Debug: 
Harald> sql(user@redacted,client-ip,): cache expired
Harald> Jan  6 16:29:44 mail dovecot[2208071]: auth-worker(2208404): Debug: 
conn 
Harald> unix:auth-worker (pid=2208397,uid=489): auth-worker<229>: Handling 
PASSV 
Harald> request
Harald> Jan  6 16:29:44 mail dovecot[2208071]: auth-worker(2208404): Debug: 
conn 
Harald> unix:auth-worker (pid=2208397,uid=489): auth-worker<229>: 
Harald> sql(user@redacted,client-ip,): Performing passdb 
lookup
Harald> Jan  6 16:29:44 mail dovecot[2208071]: auth-worker(2208404): Debug: 
conn 
Harald> unix:auth-worker (pid=2208397,uid=489): auth-worker<229>: 
Harald> sql(user@redacted,client-ip,): query: SELECT passwd 
as 
Harald> password, '127.0.0.1' as host, userid as destuser, passwd AS pass, 'Y' 
Harald> AS nologin, 'Y' AS nodelay, 'Y' AS proxy FROM dbmail_users WHERE 
Harald> userid='user@redacted' UNION (SELECT password as password, '127.0.0.1' 
Harald> as host, username as destuser, password AS pass, 'Y' AS nologin, 'Y' AS 
Harald> nodelay, 'Y' AS proxy FROM sasl_fake_auth WHERE 
Harald> username='user@redacted') limit 1;
Harald> Jan  6 16:29:44 mail dovecot[2208071]: auth-worker(2208404): conn 
Harald> unix:auth-worker (pid=2208397,uid=489): auth-worker<229>: 
Harald> sql(user@redacted,client-ip,): Password mismatch
Harald> Jan  6 16:29:44 mail dovecot[2208071]: auth-worker(2208404): Debug: 
conn 
Harald> unix:auth-worker (pid=2208397,uid=489): 

Re: doveadm sync usage ; root INBOX unsynced

[John Stoffel]

> "François" == François Poulain  writes:

François> Le Sat, 31 Oct 2020 13:36:32 +0100,
François> François Poulain  a écrit :

>> I am trying to import a mail IMAP account using doveadm.

François> Am I wrong trying to do so?

I think so.  Have you tried 'imapsysnc' or other imap downloading tools?

Re: Delivering locally through the Submission Server

[John Stoffel]

R> I am learning Dovecot step by step. I have enabled the Submission
R> Server, in the hope that I would not need to learn other MTAs like
R> Postfix.

It's not that hard to setup postfix to accept incoming email from the
internet and your local users, and to then pass it to dovecot as
needed.  Dovecot is for IMAP/POP (shudder) access of your stored
emails.  It's not for sending email, or even receiving it from the
outside world.  

R> The Submission Server is very comfortable: it picks up the existing
R> Dovecot configuration, so that you do not need to configure any
R> user authentication separately. It is working fine on my test
R> setup.

It's not hard to setup postfix/dovecot to use the same
authentication.  My system used plain files.  Trivial. 

R> My first thought was that, if the recipient is a local mailbox, the
R> Submission Server would not need to relay the message to any
R> external SMTP server, as it could just deliver it locally. After
R> all, it is running on the same Dovecot.

Really, you're trying to optimize the wrong thing.  Just setup a
linode (or anything else except digital ocean since charter.net blocks
them completely for email delivery) at $5/month and install
postfix/dovecot together.  Works great.

John

Re: strange file .temp.1592374672.P11164Q21M692534.hostname - doveadm altmove

[John Stoffel]

>>>>> "Gionatan" == Gionatan Danti  writes:

Gionatan> Il 2020-11-01 13:23 John Stoffel ha scritto:
>> How is your "mail_location" defined in your configuration?  And what
>> is your search term?

Gionatan> Hi, I just discovered having such strange .temp files on an
Gionatan> old server of mine.

I honestly haven't a clue... I don't have these on my server at all.  

Gionatan> In some folders I have hundred of such files, but they
Gionatan> really are hardlinks to the same .temp file (for each
Gionatan> folder), so they do not consume much space. These files
Gionatan> seems much more common in the Trash folder, but they appear
Gionatan> even in other folder.

I would probably just nuke these files, but maybe after confirming
what their contents really is.  

Gionatan> An example of a folder with 6 .temp files:
Gionatan> [root@mail dbox-Mails]# ls -al 
Gionatan> /var/vmail/domain/user/dbox/mailboxes/INBOX/Trash/dbox-Mails
Gionatan> drwxr-x---. 2 vmail vmail 4096 Oct 31 12:42 .
Gionatan> drwxr-x---. 3 vmail vmail 4096 Mar 17  2020 ..
Gionatan> -rw-r-. 1 vmail vmail  512 Oct 29 09:45 dovecot.index
Gionatan> -rw-r-. 1 vmail vmail  392 Oct 28 13:59 dovecot.index.backup
Gionatan> -rw-r-. 1 vmail vmail55296 Oct 31 12:42 dovecot.index.cache
Gionatan> -rw-r-. 1 vmail vmail23536 Oct 31 12:42 dovecot.index.log
Gionatan> -rw-r-. 6 vmail vmail 22215889 Oct 12 10:42 
Gionatan> .temp.1602492703.P47358Q0M194142.hostname
Gionatan> -rw-r-. 6 vmail vmail 22215889 Oct 12 10:42 
Gionatan> .temp.1602494002.P50402Q0M898232.hostname
Gionatan> -rw-r-. 6 vmail vmail 22215889 Oct 12 10:42 
Gionatan> .temp.1602494013.P50446Q0M771524.hostname
Gionatan> -rw-r-. 6 vmail vmail 22215889 Oct 12 10:42 
Gionatan> .temp.1602494621.P51761Q0M119496.hostname
Gionatan> -rw-r-. 6 vmail vmail 22215889 Oct 12 10:42 
Gionatan> .temp.1602494745.P51954Q0M387071.hostname
Gionatan> -rw-r-. 6 vmail vmail 22215889 Oct 12 10:42 
Gionatan> .temp.1602495419.P53185Q0M671512.hostname
Gionatan> -rw-r-. 1 vmail vmail   848909 Oct 29 21:12 u.1623

Gionatan> 6x 22M files = 128M...
Gionatan> [root@mail dbox-Mails]# du -hscl .temp.160249*
Gionatan> 22M .temp.1602492703.P47358Q0M194142.hostname
Gionatan> 22M .temp.1602494002.P50402Q0M898232.hostname
Gionatan> 22M .temp.1602494013.P50446Q0M771524.hostname
Gionatan> 22M .temp.1602494621.P51761Q0M119496.hostname
Gionatan> 22M .temp.1602494745.P51954Q0M387071.hostname
Gionatan> 22M .temp.1602495419.P53185Q0M671512.hostname
Gionatan> 128Mtotal

Gionatan> ...but only 22M really consumed (1 file, 6 hardlinks)
Gionatan> [root@mail dbox-Mails]# du -hsc .temp.160249*
Gionatan> 22M .temp.1602492703.P47358Q0M194142.hostname
Gionatan> 22M total

Gionatan> I never run doveadm altmove; moreover, these .tempo files
Gionatan> are both old (>1 year) and new (~3 weeks ago). Below you can
Gionatan> find the output from dovecot -n. Any ideas?

I really don't know, sorry!  I'd have to got poke at the source to see
how they're created and where. I don't use dbox, I use Maildir for my
setup.

So I suspect (but I'm not sure!) that these are just leftovers from
mailbox accesses which got left behind.  Try googling for 'dovecot
dbox temp files' and maybe that will help.  Sorry!

John

Re: strange file .temp.1592374672.P11164Q21M692534.hostname - doveadm altmove

[John Stoffel]

> "Maciek" == Maciek Jackowski  writes:

Maciek> dovecot 2.2.33.2 Ubuntu 18.04
Maciek> I got strange file
Maciek>  .temp.1592374672.P11164Q21M692534.hostname
Maciek> after doveadm altmove command
Maciek> in INBOX on alternate storage server

Maciek> What can I do with this file? Remove?
Maciek> I am using dbox

You need to give us much more information, such as any log messages,
your configuration with 'doveconf -n'.  Can you give us the output of:

  doveadm -v altmove ''

where  is the query you're using?  Basically, you are just saying
to us:  "Broke!  Fix!"  and we can't help much without details.

How is your "mail_location" defined in your configuration?  And what
is your search term?

John

Re: Sieve filter script EXECUTION FAILED

[John Stoffel]

> "@lbutlr" == @lbutlr   writes:

@lbutlr> On 30 Oct 2020, at 12:34, @lbutlr  wrote:
>> I am not sure about the $1. I think filter just pipes the message (or part 
>> of the message.
>> 
>> I will see what happens without the echo I suppose.
>> 
>> Nope, still the same.
>> 
>> 32:   starting `:contains' match with `i;ascii-casemap' comparator:
>> 32:   matching value `> lang="en">29-Oct-2020 "" ?? ...'
>> 32: with key `' => 1
>> 32:   finishing match with result: matched
>> 32: jump if result is false
>> 32:   not jumping
>> 34: filter action
>> 34:   execute program `darkmode.sh'
>> 34:   [[EXECUTION ABORTED]]

@lbutlr> Here is the relevant part of doveconf -n

@lbutlr> plugin {
@lbutlr>   imapsieve_mailbox1_before = 
file:/usr/lib/dovecot/sieve/report-spam.sieve
@lbutlr>   imapsieve_mailbox1_causes = COPY
@lbutlr>   imapsieve_mailbox1_name = Junk
@lbutlr>   imapsieve_mailbox2_before = 
file:/usr/lib/dovecot/sieve/report-ham.sieve
@lbutlr>   imapsieve_mailbox2_causes = COPY
@lbutlr>   imapsieve_mailbox2_from = Junk
@lbutlr>   imapsieve_mailbox2_name = *
@lbutlr>   imapsieve_mailbox3_before = 
file:/usr/lib/dovecot/sieve/mark-read.sieve
@lbutlr>   imapsieve_mailbox3_causes = COPY
@lbutlr>   imapsieve_mailbox3_name = Archive
@lbutlr>   quota_rule2 = .EXPUNGED:ignore
@lbutlr>   sieve = file:~/.sieve;active=~/.active_sieve
@lbutlr>   sieve_before = file:/usr/lib/dovecot/sieve/bcc.sieve
@lbutlr>   sieve_before3 = file:/usr/lib/dovecot/sieve/filespam.sieve
@lbutlr>   sieve_default_name = spamassassin
@lbutlr>   sieve_duplicate_default_period = 1h
@lbutlr>   sieve_duplicate_max_period = 12d
@lbutlr>   sieve_extensions = +vnd.dovecot.pipe +vnd.dovecot.environment 
+vnd.dovecot.filter +editheader
@lbutlr>   sieve_filter_bin_dir = /usr/lib/dovecot/sieve /usr/local/virtual/
@lbutlr>   sieve_pipe_bin_dir = /usr/lib/dovecot/sieve
@lbutlr>   sieve_plugins = sieve_imapsieve sieve_extprograms
@lbutlr>   sieve_trace_dir = ~/.trace
@lbutlr>   sieve_trace_level = matching
@lbutlr>   sieve_user_log = ~/sieve.log
@lbutlr> }

@lbutlr> Current shell script is

@lbutlr> #!/bin/sh
@lbutlr> sed -e 's||