Re: Replication issue 2.3.7

[Stuart Henderson via dovecot]

On 2019/07/16 19:46, Aki Tuomi wrote:
> 
> > On 16/07/2019 18:40 Stuart Henderson via dovecot  
> > wrote:
> > 
> >  
> > On 2019-07-13, Reio Remma via dovecot  wrote:
> > > Hello!
> > >
> > > I noticed these in the logs since upgrading from 2.3.6. to 2.3.7:
> > >
> > > Jul 13 11:52:10 turin dovecot: doveadm: Error: 
> > > dsync-remote(r...@mrstuudio.ee): Error: 
> > > Exporting mailbox INBOX failed: Mailbox attribute 
> > > vendor/vendor.dovecot/pvt/server/sieve/files/MR lookup failed: Mailbox 
> > > attributes not enabled
> > > Jul 13 11:52:11 turin dovecot: doveadm: Error: 
> > > dsync-remote(r...@mrstuudio.ee): Error: 
> > > Exporting mailbox INBOX failed: Mailbox attribute 
> > > vendor/vendor.dovecot/pvt/server/sieve/files/MR lookup failed: Mailbox 
> > > attributes not enabled
> > 
> > Same here (in my case: on OpenBSD -current, mdbox, pigeonhole 0.5.7).
> > I have backed out to 2.3.6 + pigeonhole 0.5.6 and it's happy again.
> 
> Instead of downgrading, you could've attempted
> 
> mail_attribute_dict = file:%h/dovecot-attributes
> 
> to enable mailbox attributes. This should fix sieve script replication too.
> 
> Aki

Perhaps, but when dealing with mailboxes out of sync between the two
replicated machines the first thought is to backout the change that
triggered the problem, rather than guess things that might help,
especially when they aren't mentioned in the release notes (actually
the only thing I find in either NEWS or ChangeLog relating to
attributes/metadata at all since the last release is adding methods
to access this via lua API?).

Re: Auto-duplicate mailstore?

[@lbutlr via dovecot]

On 12 Jul 2019, at 16:15, @lbutlr via dovecot  wrote:
> I am looking for something that is similar to replication, but without a 
> second server. I currently have a system setup using postfix bcc to write out 
> copies of all email, but what I would rather do is just have all the mail 
> written to /usr/local/virtual/%u/Maildir and also to 
> /backup/usr/local/virtual/%u/Maildir (for example) as this will make 
> restoring messages much simpler.

Any ideas on how to do this?

A sort of copy on save function.



-- 
From deep inside the tears that I'm forced to cry From deep inside the
pain I--I chose to hide

Re: Replication issue 2.3.7

[Aki Tuomi via dovecot]

> On 16/07/2019 18:40 Stuart Henderson via dovecot  wrote:
> 
>  
> On 2019-07-13, Reio Remma via dovecot  wrote:
> > Hello!
> >
> > I noticed these in the logs since upgrading from 2.3.6. to 2.3.7:
> >
> > Jul 13 11:52:10 turin dovecot: doveadm: Error: 
> > dsync-remote(r...@mrstuudio.ee): Error: 
> > Exporting mailbox INBOX failed: Mailbox attribute 
> > vendor/vendor.dovecot/pvt/server/sieve/files/MR lookup failed: Mailbox 
> > attributes not enabled
> > Jul 13 11:52:11 turin dovecot: doveadm: Error: 
> > dsync-remote(r...@mrstuudio.ee): Error: 
> > Exporting mailbox INBOX failed: Mailbox attribute 
> > vendor/vendor.dovecot/pvt/server/sieve/files/MR lookup failed: Mailbox 
> > attributes not enabled
> 
> Same here (in my case: on OpenBSD -current, mdbox, pigeonhole 0.5.7).
> I have backed out to 2.3.6 + pigeonhole 0.5.6 and it's happy again.

Instead of downgrading, you could've attempted

mail_attribute_dict = file:%h/dovecot-attributes

to enable mailbox attributes. This should fix sieve script replication too.

Aki

Re: Replication issue 2.3.7

[Stuart Henderson via dovecot]

On 2019-07-13, Reio Remma via dovecot  wrote:
> Hello!
>
> I noticed these in the logs since upgrading from 2.3.6. to 2.3.7:
>
> Jul 13 11:52:10 turin dovecot: doveadm: Error: 
> dsync-remote(r...@mrstuudio.ee): Error: 
> Exporting mailbox INBOX failed: Mailbox attribute 
> vendor/vendor.dovecot/pvt/server/sieve/files/MR lookup failed: Mailbox 
> attributes not enabled
> Jul 13 11:52:11 turin dovecot: doveadm: Error: 
> dsync-remote(r...@mrstuudio.ee): Error: 
> Exporting mailbox INBOX failed: Mailbox attribute 
> vendor/vendor.dovecot/pvt/server/sieve/files/MR lookup failed: Mailbox 
> attributes not enabled

Same here (in my case: on OpenBSD -current, mdbox, pigeonhole 0.5.7).
I have backed out to 2.3.6 + pigeonhole 0.5.6 and it's happy again.

Re: Unexpected result from LIST EXTENDED command

[Kostya Vasilyev via dovecot]

As a workaround (if this turns out to be a Dovecot bug) - 

- you could use STATUS instead.

It would be several commands, one per folder, so you'd want to send them 
pipelined for performance.

I've done this before in an email app I worked on and haven't seen any issues 
with Dovecot or any other server (e.g. Gmail, Cyrus, Fastmail ...)

-- K

On Tue, Jul 16, 2019, at 11:41 AM, Emil Kalchev via dovecot wrote:
> There is no error in the server logs. I checked those particular folders on 
> the server and they don’t seems to have anything special about them, like 
> permission or etc.

> 

> Yes, calling STATUS on those particular folders returns the status. The 
> folders can be opened and they have emails in them so nothing special about 
> those folders.

> 

> # 2.3.5 (513208660): /etc/dovecot/dovecot.conf

> # OS: Linux 3.10.0-957.21.3.el7.x86_64 x86_64 CentOS Linux release 7.6.1810 
> (Core)

> # Hostname: mail.HIDDEN

> auth_cache_size = 1 M

> auth_mechanisms = plain login

> auth_username_chars = 
> "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789!#$-=?^_{}~./@+%"

> disable_plaintext_auth = no

> first_valid_uid = 201

> lda_mailbox_autocreate = yes

> lmtp_rcpt_check_quota = yes

> lmtp_save_to_detail_mailbox = yes

> lmtp_user_concurrency_limit = 4

> mail_access_groups = dovecot

> mail_plugins = quota quota_clone zlib fts fts_solr

> mail_prefetch_count = 20

> maildir_very_dirty_syncs = yes

> namespace inbox {

> inbox = yes

> location =

> mailbox Archive {

> auto = create

> special_use = \Archive

> }

> mailbox Archives {

> auto = no

> special_use = \Archive

> }

> mailbox Drafts {

> auto = subscribe

> special_use = \Drafts

> }

> mailbox Sent {

> auto = subscribe

> special_use = \Sent

> }

> mailbox "Sent Messages" {

> auto = no

> special_use = \Sent

> }

> mailbox Trash {

> auto = subscribe

> special_use = \Trash

> }

> mailbox spam {

> auto = subscribe

> special_use = \Junk

> }

> prefix = INBOX.

> separator = .

> type = private

> }

> passdb {

> args = /usr/local/cpanel/etc/dovecot/cpauthd-dict.conf

> driver = dict

> result_failure = return-fail

> }

> plugin {

> acl = vfile:cache_secs=86400

> fts = solr

> fts_solr = url=http://#hidden_use-P_to_show#@127.0.0.1:8984/solr/dovecot/

> quota_exceeded_message = Mailbox is full / Blocks limit exceeded / Inode 
> limit exceeded

> }

> protocols = lmtp imap pop3

> service auth {

> unix_listener auth-client {

> mode = 0666

> }

> }

> service config {

> vsz_limit = 2 G

> }

> service dict {

> unix_listener dict {

> group = dovecot

> mode = 0660

> }

> }

> service imap-login {

> client_limit = 500

> inet_listener imap {

> address = *,::

> }

> inet_listener imaps {

> address = *,::

> }

> process_limit = 50

> process_min_avail = 2

> service_count = 0

> vsz_limit = 128 M

> }

> service imap {

> process_limit = 512

> vsz_limit = 512 M

> }

> service lmtp {

> client_limit = 1

> process_limit = 500

> unix_listener lmtp {

> group = mail

> mode = 0660

> user = mailnull

> }

> vsz_limit = 512 M

> }

> service managesieve-login {

> client_limit = 500

> process_limit = 50

> process_min_avail = 2

> service_count = 0

> vsz_limit = 128 M

> }

> service managesieve {

> process_limit = 512

> vsz_limit = 512 M

> }

> service pop3-login {

> client_limit = 500

> inet_listener pop3 {

> address = *,::

> }

> inet_listener pop3s {

> address = *,::

> }

> process_limit = 50

> process_min_avail = 2

> service_count = 0

> vsz_limit = 128 M

> }

> service pop3 {

> process_limit = 512

> vsz_limit = 512 M

> }

> service quota-status {

> executable = quota-status -p postfix

> unix_listener quota-status {

> mode = 0666

> }

> }

> service stats {

> client_limit = 2000

> unix_listener stats-writer {

> mode = 0666

> }

> }

> ssl_cert =  ssl_cipher_list = 
> ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:!DSS

> ssl_dh = # hidden, use -P to show it

> ssl_key = # hidden, use -P to show it

> userdb {

> driver = prefetch

> }

> userdb {

> args = /usr/local/cpanel/etc/dovecot/cpauthd-dict.conf

> driver = dict

> }

> protocol imap {

> imap_capability = +NAMESPACE

> imap_idle_notify_interval = 24 mins

> imap_logout_format = in=%i, out=%o, bytes=%i/%o

> mail_max_userip_connections = 20

> mail_plugins = acl quota imap_quota zlib imap_zlib quota_clone virtual fts 
> fts_solr

> namespace sent {

> hidden = yes

> list = no

> location = 

Re: Dovecot release v2.3.7

[Eirik Rye via dovecot]

Please disregard this. My apologies.

The issue appears to have been caused by an unrelated network issue with 
one of the directors, that coincidentally occurred at almost the exact 
same time.


A potential improvement would be if the "Ring SYNC seq=XX appears to 
have got lost" error message would indicate which director/server it 
pertains to.


Best regards,
Eirik Rye

On 16/07/2019 11:17, Eirik Rye via dovecot wrote:
Version 2.3.7 appears to break director ring sync. After upgrading, our 
logs are full of these kinds of messages:


Jul 14 04:09:20 yyy dovecot: director: Error: director: User xxx host 
lookup failed: Timeout because ring not synced - queued for 30 secs 
(Ring not synced for 1410 secs, hash=271838221)
Jul 14 04:09:20 yyy dovecot: director: Error: director: User xxx host 
lookup failed: Timeout because ring not synced - queued for 30 secs 
(Ring not synced for 1410 secs, hash=3482757093)


As well as:

Jul 14 04:09:21 yyy dovecot: director: Error: Ring SYNC seq=53 appears 
to have got lost, resending


These are absent from the logs prior to July 12.

I have downgraded to 2.3.6 in hopes of resolving the issue.

Regards,
Eirik Rye

On 12/07/2019 14:29, Aki Tuomi via dovecot wrote:

Hi!

We are pleased to release Dovecot release v2.3.7.

Tarball is available at

https://dovecot.org/releases/2.3/dovecot-2.3.7.tar.gz
https://dovecot.org/releases/2.3/dovecot-2.3.7.tar.gz.sig

Binary packages are available at https://repo.dovecot.org/

Changes
---

* fts-solr: Removed break-imap-search parameter
+ Added more events for the new statistics, see
   https://doc.dovecot.org/admin_manual/list_of_events/
+ mail-lua: Add IMAP metadata accessors, see
   https://doc.dovecot.org/admin_manual/lua/
+ Add event exporters that allow exporting raw events to log files and
   external systems, see
   https://doc.dovecot.org/configuration_manual/event_export/
+ SNIPPET is now PREVIEW and size has been increased to 200 characters.
+ Add body option to fts_enforced. This triggers building FTS index only
   on body search, and an error using FTS index fails the search rather
   than reads through all the mails.
- Submission/LMTP: Fixed crash when domain argument is invalid in a
   second EHLO/LHLO command.
- Copying/moving mails using Maildir format loses IMAP keywords in the
   destination if the mail also has no system flags.
- mail_attachment_detection_options=add-flags-on-save caused email body
   to be unnecessarily opened when FETCHing mail headers that were
   already cached.
- mail attachment detection keywords not saved with maildir.
- dovecot.index.cache may have grown excessively large in some
   situations. This happened especially when using autoexpunging with
   lazy_expunge folders. Also with mdbox format in general the cache file
   wasn't recreated as often as it should have.
- Autoexpunged mails weren't immediately deleted from the disk. Instead,
   the deletion from disk happened the next time the folder was opened.
   This could have caused unnecessary delays if the opening was done by
   an interactive IMAP session.
- Dovecot's TCP connections sometimes add extra 40ms latency due to not
   enabling TCP_NODELAY. HTTP and SMTP/LMTP connections weren't
   affected, but everything else was. This delay wasn't always visible -
   only in some situations with some message/packet sizes.
- imapc: Fix various crash conditions
- Dovecot builds were not always reproducible.
- login-proxy: With shutdown_clients=no after config reload the
   existing connections could no longer be listed or kicked with doveadm.
- "doveadm proxy kick" with -f parameter caused a crash in some
   situations.
- Auth policy can cause segmentation fault crash during auth process
   shutdown if all auth requests have not been finished.
- Fix various minor bugs leading into incorrect behaviour in mailbox
   list index handling. These rarely caused noticeable problems.
- LDAP auth: Iteration accesses freed memory, possibly crashing
   auth-worker
- local_name { .. } filter in dovecot.conf does not correctly support
   multiple names and wildcards were matched incorrectly.
- replicator: dsync assert-crashes if it can't connect to remote TCP
   server.
- config: Memory leak in config process when ssl_dh setting wasn't
   set and there was no ssl-parameters.dat file.
   This caused config process to die once in a while
   with "out of memory".

---
Aki Tuomi
Open-Xchange oy

Re: Dovecot release v2.3.7

[Eirik Rye via dovecot]

Downgrading to 2.3.6 did indeed resolve the issue.

Our Directors are running Ubuntu 18.04.

Regards,
Eirik Rye

On 16/07/2019 11:17, Eirik Rye via dovecot wrote:
Version 2.3.7 appears to break director ring sync. After upgrading, our 
logs are full of these kinds of messages:


Jul 14 04:09:20 yyy dovecot: director: Error: director: User xxx host 
lookup failed: Timeout because ring not synced - queued for 30 secs 
(Ring not synced for 1410 secs, hash=271838221)
Jul 14 04:09:20 yyy dovecot: director: Error: director: User xxx host 
lookup failed: Timeout because ring not synced - queued for 30 secs 
(Ring not synced for 1410 secs, hash=3482757093)


As well as:

Jul 14 04:09:21 yyy dovecot: director: Error: Ring SYNC seq=53 appears 
to have got lost, resending


These are absent from the logs prior to July 12.

I have downgraded to 2.3.6 in hopes of resolving the issue.

Regards,
Eirik Rye

On 12/07/2019 14:29, Aki Tuomi via dovecot wrote:

Hi!

We are pleased to release Dovecot release v2.3.7.

Tarball is available at

https://dovecot.org/releases/2.3/dovecot-2.3.7.tar.gz
https://dovecot.org/releases/2.3/dovecot-2.3.7.tar.gz.sig

Binary packages are available at https://repo.dovecot.org/

Changes
---

* fts-solr: Removed break-imap-search parameter
+ Added more events for the new statistics, see
   https://doc.dovecot.org/admin_manual/list_of_events/
+ mail-lua: Add IMAP metadata accessors, see
   https://doc.dovecot.org/admin_manual/lua/
+ Add event exporters that allow exporting raw events to log files and
   external systems, see
   https://doc.dovecot.org/configuration_manual/event_export/
+ SNIPPET is now PREVIEW and size has been increased to 200 characters.
+ Add body option to fts_enforced. This triggers building FTS index only
   on body search, and an error using FTS index fails the search rather
   than reads through all the mails.
- Submission/LMTP: Fixed crash when domain argument is invalid in a
   second EHLO/LHLO command.
- Copying/moving mails using Maildir format loses IMAP keywords in the
   destination if the mail also has no system flags.
- mail_attachment_detection_options=add-flags-on-save caused email body
   to be unnecessarily opened when FETCHing mail headers that were
   already cached.
- mail attachment detection keywords not saved with maildir.
- dovecot.index.cache may have grown excessively large in some
   situations. This happened especially when using autoexpunging with
   lazy_expunge folders. Also with mdbox format in general the cache file
   wasn't recreated as often as it should have.
- Autoexpunged mails weren't immediately deleted from the disk. Instead,
   the deletion from disk happened the next time the folder was opened.
   This could have caused unnecessary delays if the opening was done by
   an interactive IMAP session.
- Dovecot's TCP connections sometimes add extra 40ms latency due to not
   enabling TCP_NODELAY. HTTP and SMTP/LMTP connections weren't
   affected, but everything else was. This delay wasn't always visible -
   only in some situations with some message/packet sizes.
- imapc: Fix various crash conditions
- Dovecot builds were not always reproducible.
- login-proxy: With shutdown_clients=no after config reload the
   existing connections could no longer be listed or kicked with doveadm.
- "doveadm proxy kick" with -f parameter caused a crash in some
   situations.
- Auth policy can cause segmentation fault crash during auth process
   shutdown if all auth requests have not been finished.
- Fix various minor bugs leading into incorrect behaviour in mailbox
   list index handling. These rarely caused noticeable problems.
- LDAP auth: Iteration accesses freed memory, possibly crashing
   auth-worker
- local_name { .. } filter in dovecot.conf does not correctly support
   multiple names and wildcards were matched incorrectly.
- replicator: dsync assert-crashes if it can't connect to remote TCP
   server.
- config: Memory leak in config process when ssl_dh setting wasn't
   set and there was no ssl-parameters.dat file.
   This caused config process to die once in a while
   with "out of memory".

---
Aki Tuomi
Open-Xchange oy

Re: Dovecot release v2.3.7

[Eirik Rye via dovecot]

Version 2.3.7 appears to break director ring sync. After upgrading, our 
logs are full of these kinds of messages:


Jul 14 04:09:20 yyy dovecot: director: Error: director: User xxx host 
lookup failed: Timeout because ring not synced - queued for 30 secs 
(Ring not synced for 1410 secs, hash=271838221)
Jul 14 04:09:20 yyy dovecot: director: Error: director: User xxx host 
lookup failed: Timeout because ring not synced - queued for 30 secs 
(Ring not synced for 1410 secs, hash=3482757093)


As well as:

Jul 14 04:09:21 yyy dovecot: director: Error: Ring SYNC seq=53 appears 
to have got lost, resending


These are absent from the logs prior to July 12.

I have downgraded to 2.3.6 in hopes of resolving the issue.

Regards,
Eirik Rye

On 12/07/2019 14:29, Aki Tuomi via dovecot wrote:

Hi!

We are pleased to release Dovecot release v2.3.7.

Tarball is available at

https://dovecot.org/releases/2.3/dovecot-2.3.7.tar.gz
https://dovecot.org/releases/2.3/dovecot-2.3.7.tar.gz.sig

Binary packages are available at https://repo.dovecot.org/

Changes
---

* fts-solr: Removed break-imap-search parameter
+ Added more events for the new statistics, see
   https://doc.dovecot.org/admin_manual/list_of_events/
+ mail-lua: Add IMAP metadata accessors, see
   https://doc.dovecot.org/admin_manual/lua/
+ Add event exporters that allow exporting raw events to log files and
   external systems, see
   https://doc.dovecot.org/configuration_manual/event_export/
+ SNIPPET is now PREVIEW and size has been increased to 200 characters.
+ Add body option to fts_enforced. This triggers building FTS index only
   on body search, and an error using FTS index fails the search rather
   than reads through all the mails.
- Submission/LMTP: Fixed crash when domain argument is invalid in a
   second EHLO/LHLO command.
- Copying/moving mails using Maildir format loses IMAP keywords in the
   destination if the mail also has no system flags.
- mail_attachment_detection_options=add-flags-on-save caused email body
   to be unnecessarily opened when FETCHing mail headers that were
   already cached.
- mail attachment detection keywords not saved with maildir.
- dovecot.index.cache may have grown excessively large in some
   situations. This happened especially when using autoexpunging with
   lazy_expunge folders. Also with mdbox format in general the cache file
   wasn't recreated as often as it should have.
- Autoexpunged mails weren't immediately deleted from the disk. Instead,
   the deletion from disk happened the next time the folder was opened.
   This could have caused unnecessary delays if the opening was done by
   an interactive IMAP session.
- Dovecot's TCP connections sometimes add extra 40ms latency due to not
   enabling TCP_NODELAY. HTTP and SMTP/LMTP connections weren't
   affected, but everything else was. This delay wasn't always visible -
   only in some situations with some message/packet sizes.
- imapc: Fix various crash conditions
- Dovecot builds were not always reproducible.
- login-proxy: With shutdown_clients=no after config reload the
   existing connections could no longer be listed or kicked with doveadm.
- "doveadm proxy kick" with -f parameter caused a crash in some
   situations.
- Auth policy can cause segmentation fault crash during auth process
   shutdown if all auth requests have not been finished.
- Fix various minor bugs leading into incorrect behaviour in mailbox
   list index handling. These rarely caused noticeable problems.
- LDAP auth: Iteration accesses freed memory, possibly crashing
   auth-worker
- local_name { .. } filter in dovecot.conf does not correctly support
   multiple names and wildcards were matched incorrectly.
- replicator: dsync assert-crashes if it can't connect to remote TCP
   server.
- config: Memory leak in config process when ssl_dh setting wasn't
   set and there was no ssl-parameters.dat file.
   This caused config process to die once in a while
   with "out of memory".

---
Aki Tuomi
Open-Xchange oy

Re: Unexpected result from LIST EXTENDED command

[Timo Sirainen via dovecot]

On 16 Jul 2019, at 11.41, Emil Kalchev  wrote:
> 
> There is no error in the server logs. I checked those particular folders on 
> the server and they don’t seems to have anything special about them, like 
> permission or etc.
>  
> Yes, calling STATUS on those particular folders returns the status. The 
> folders can be opened and they have emails in them so nothing special about 
> those folders.

https://github.com/dovecot/core/blob/master/src/imap/cmd-list.c#L195 
 seems to 
have a bug. If LIST is requesting SUBSCRIBED results, and it finds there is a 
folder that is not subscribed but has a child that is subscribed, then the 
parent isn't requested for STATUS. That matches:

S: * LIST (\HasChildren \UnMarked) "." INBOX.spam.&-BD0EOQQ9BDkEPQ-
S: * LIST (\Subscribed \HasChildren \UnMarked) "." 
INBOX.spam.&-BD0EOQQ9BDkEPQ-.jhfhg

But in your LIST output INBOX.Archive didn't have any children, so I'm not sure 
if that's the same issue or not.

RE: Unexpected result from LIST EXTENDED command

[Emil Kalchev via dovecot]

There is no error in the server logs. I checked those particular folders on the 
server and they don’t seems to have anything special about them, like 
permission or etc.

Yes, calling STATUS on those particular folders returns the status. The folders 
can be opened and they have emails in them so nothing special about those 
folders.

# 2.3.5 (513208660): /etc/dovecot/dovecot.conf
# OS: Linux 3.10.0-957.21.3.el7.x86_64 x86_64 CentOS Linux release 7.6.1810 
(Core)
# Hostname: mail.HIDDEN
auth_cache_size = 1 M
auth_mechanisms = plain login
auth_username_chars = 
"abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789!#$-=?^_{}~./@+%"
disable_plaintext_auth = no
first_valid_uid = 201
lda_mailbox_autocreate = yes
lmtp_rcpt_check_quota = yes
lmtp_save_to_detail_mailbox = yes
lmtp_user_concurrency_limit = 4
mail_access_groups = dovecot
mail_plugins = quota quota_clone zlib fts fts_solr
mail_prefetch_count = 20
maildir_very_dirty_syncs = yes
namespace inbox {
inbox = yes
location =
mailbox Archive {
auto = create
special_use = \Archive
}
mailbox Archives {
auto = no
special_use = \Archive
}
mailbox Drafts {
auto = subscribe
special_use = \Drafts
}
mailbox Sent {
auto = subscribe
special_use = \Sent
}
mailbox "Sent Messages" {
auto = no
special_use = \Sent
}
mailbox Trash {
auto = subscribe
special_use = \Trash
}
mailbox spam {
auto = subscribe
special_use = \Junk
}
prefix = INBOX.
separator = .
type = private
}
passdb {
args = /usr/local/cpanel/etc/dovecot/cpauthd-dict.conf
driver = dict
result_failure = return-fail
}
plugin {
acl = vfile:cache_secs=86400
fts = solr
fts_solr = url=http://#hidden_use-P_to_show#@127.0.0.1:8984/solr/dovecot/
quota_exceeded_message = Mailbox is full / Blocks limit exceeded / Inode limit 
exceeded
}
protocols = lmtp imap pop3
service auth {
unix_listener auth-client {
mode = 0666
}
}
service config {
vsz_limit = 2 G
}
service dict {
unix_listener dict {
group = dovecot
mode = 0660
}
}
service imap-login {
client_limit = 500
inet_listener imap {
address = *,::
}
inet_listener imaps {
address = *,::
}
process_limit = 50
process_min_avail = 2
service_count = 0
vsz_limit = 128 M
}
service imap {
process_limit = 512
vsz_limit = 512 M
}
service lmtp {
client_limit = 1
process_limit = 500
unix_listener lmtp {
group = mail
mode = 0660
user = mailnull
}
vsz_limit = 512 M
}
service managesieve-login {
client_limit = 500
process_limit = 50
process_min_avail = 2
service_count = 0
vsz_limit = 128 M
}
service managesieve {
process_limit = 512
vsz_limit = 512 M
}
service pop3-login {
client_limit = 500
inet_listener pop3 {
address = *,::
}
inet_listener pop3s {
address = *,::
}
process_limit = 50
process_min_avail = 2
service_count = 0
vsz_limit = 128 M
}
service pop3 {
process_limit = 512
vsz_limit = 512 M
}
service quota-status {
executable = quota-status -p postfix
unix_listener quota-status {
mode = 0666
}
}
service stats {
client_limit = 2000
unix_listener stats-writer {
mode = 0666
}
}
ssl_cert = 
Sent: Tuesday, July 16, 2019 10:59 AM
To: Emil Kalchev 
Cc: Dovecot Mailing List 
Subject: Re: Unexpected result from LIST EXTENDED command

On 16 Jul 2019, at 9.51, Emil Kalchev via dovecot 
mailto:dovecot@dovecot.org>> wrote:

I am executing this command below to dovecot-2.3.5-6.cp1178.x86_64 server

Notice that some status responses are missing (For folders INBOX.Archive, 
INBOX.spam.&-BD0EOQQ9BDkEPQ-). I wonder If this is a bug or working as expected

In rfc5819 there is this:

If the server runs into unexpected problems while trying to look up
the STATUS information, it MAY drop the corresponding STATUS reply.
In such a situation, the LIST command would still return a tagged OK
reply.

May be that is the reason for this response? Is it possible to find more 
details in server logs why STATUS is missing?

Do you see any errors logged? Does it work if you ask with STATUS command 
directly those folders? What's your doveconf -n?

Re: [ext] 2.3.7 slower than 2.3.6?

[Ralf Hildebrandt via dovecot]

* Timo Sirainen via dovecot :

> > And alas, I had a look at the monitoring and found that disk IO has
> > increase A LOT after the update: 
> > https://www.arschkrebs.de/images/dovecot-disk-io-increase.png
> > 
> > I zoomed in and found that the sudden increace in IO coincides with
> > the update to 2.3.7 (14:35): 
> > https://www.arschkrebs.de/images/dovecot-io-2.png 
> > 
> 

> Do the different disks have different kinds of data? 

All of the data is an /dev/sda1

> Like you seem to be using external attachments and fts? 

Jup, using that, but on one disk

> Also your doveconf -n
> doesn't show what fts backend you're using. Or is it just
> half-configured and not really used?

I used to use lucene, but then I had issues building it
and commented it out. 

Looking now, it turns out that your packages come with lucene support...
Reenabling:

mail_plugins = zlib fts fts_lucene

...

plugin {
  fts = lucene
  fts_autoindex = yes
  fts_languages = de,en
  fts_lucene = whitespace_chars=@.
  zlib_save = gz
  zlib_save_level = 5
}



-- 
Ralf Hildebrandt
  Geschäftsbereich IT | Abteilung Netzwerk
  Charité - Universitätsmedizin Berlin
  Campus Benjamin Franklin
  Hindenburgdamm 30 | D-12203 Berlin
  Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962
  ralf.hildebra...@charite.de | https://www.charite.de

Re: Unexpected result from LIST EXTENDED command

[Timo Sirainen via dovecot]

On 16 Jul 2019, at 9.51, Emil Kalchev via dovecot  wrote:
> 
> I am executing this command below to dovecot-2.3.5-6.cp1178.x86_64 server
>  
> Notice that some status responses are missing (For folders INBOX.Archive, 
> INBOX.spam.&-BD0EOQQ9BDkEPQ-). I wonder If this is a bug or working as 
> expected
>  
> In rfc5819 there is this:
>  
> If the server runs into unexpected problems while trying to look up
> the STATUS information, it MAY drop the corresponding STATUS reply.
> In such a situation, the LIST command would still return a tagged OK
> reply.
>  
> May be that is the reason for this response? Is it possible to find more 
> details in server logs why STATUS is missing?

Do you see any errors logged? Does it work if you ask with STATUS command 
directly those folders? What's your doveconf -n?

Re: Dovecot release v2.3.7

[Timo Sirainen via dovecot]

On 13 Jul 2019, at 14.44, Tom Sommer via dovecot  wrote:
> 
> LMTP is broken on director:
> 
> Jul 13 13:42:41 lmtp(34824): Panic: file smtp-params.c: line 685 
> (smtp_params_mail_add_body_to_event): assertion failed: ((caps & 
> SMTP_CAPABILITY_8BITMIME) != 0)

Thanks, fixed: 
https://github.com/dovecot/core/commit/c4de81077c11d09eddf6a5c93676ee82350343a6 

auth: Warning: Event 0x1280fe20 leaked

[John Doe Vecot via dovecot]

Lately I get some errors in my logfile, saying:

dovecot[72337]: auth: Warning: Event 0x1280fe20 leaked (parent=0x0): 
auth-client-connection.c:338: 1 Time(s)

Can you tell me what is going wrong here? Thanks.

Re: Dovecot release v2.3.7

[Timo Sirainen via dovecot]

On 13 Jul 2019, at 18.39, Michael Grimm via dovecot  wrote:
> 
> Now replication is working from my point of view, besides occational error 
> messages like:
> 
> | imap-login: Error: file_ostream.net_set_tcp_nodelay(, TRUE) failed: 
> Connection reset by peer

Here's the final fix for it:

https://github.com/dovecot/core/commit/25028730cd1b76e373ff989625132d526eea2504 

Re: [ext] 2.3.7 slower than 2.3.6?

[Timo Sirainen via dovecot]

On 15 Jul 2019, at 23.17, Ralf Hildebrandt via dovecot  
wrote:
> 
> * Ralf Hildebrandt via dovecot :
>> We're using dovecot (via LMTP) as a backup for all incoming mail.
>> 
>> I upgraded from 2.3.6 to 2.3.7 on the 12th:
>> 2019-07-12 14:35:44 upgrade dovecot-imapd:amd64 2:2.3.6-2~bionic 
>> 2:2.3.7-8~bionic
>> 
>> and it seems that 2.3.7 is slower than 2.3.6 -- mail to the backup
>> IMAP box is suddenly taking quite some time to get delivered and is piling up
>> in the queue.
> 
> And alas, I had a look at the monitoring and found that disk IO has
> increase A LOT after the update: 
> https://www.arschkrebs.de/images/dovecot-disk-io-increase.png
> 
> I zoomed in and found that the sudden increace in IO coincides with
> the update to 2.3.7 (14:35): 
> https://www.arschkrebs.de/images/dovecot-io-2.png 
> 

Do the different disks have different kinds of data? Like you seem to be using 
external attachments and fts? Also your doveconf -n doesn't show what fts 
backend you're using. Or is it just half-configured and not really used?

Unexpected result from LIST EXTENDED command

[Emil Kalchev via dovecot]

I am executing this command below to dovecot-2.3.5-6.cp1178.x86_64 server

Notice that some status responses are missing (For folders INBOX.Archive, 
INBOX.spam.&-BD0EOQQ9BDkEPQ-). I wonder If this is a bug or working as expected

In rfc5819 there is this:

If the server runs into unexpected problems while trying to look up
the STATUS information, it MAY drop the corresponding STATUS reply.
In such a situation, the LIST command would still return a tagged OK
reply.

May be that is the reason for this response? Is it possible to find more 
details in server logs why STATUS is missing?

C: B0007 LIST "" "INBOX.*" RETURN (SUBSCRIBED CHILDREN STATUS 
(HIGHESTMODSEQ))
S: * LIST (\Subscribed \HasNoChildren \UnMarked) "." INBOX.gfgfee888
S: * STATUS INBOX.gfgfee888 (HIGHESTMODSEQ 1)
S: * LIST (\Subscribed \HasNoChildren \UnMarked) "." INBOX.
S: * STATUS INBOX. (HIGHESTMODSEQ 3)
S: * LIST (\Subscribed \HasNoChildren \UnMarked) "." INBOX.y
S: * STATUS INBOX.y (HIGHESTMODSEQ 1)
S: * LIST (\Subscribed \HasNoChildren \UnMarked) "." INBOX.y
S: * STATUS INBOX.y (HIGHESTMODSEQ 1)
S: * LIST (\Subscribed \HasNoChildren \UnMarked) "." INBOX.fgfgfg
S: * STATUS INBOX.fgfgfg (HIGHESTMODSEQ 61)
S: * LIST (\Subscribed \HasChildren \UnMarked) "." INBOX.Mailspring
S: * STATUS INBOX.Mailspring (HIGHESTMODSEQ 1)
S: * LIST (\Subscribed \HasNoChildren \UnMarked) "." INBOX.Mailspring.Snoozed
S: * STATUS INBOX.Mailspring.Snoozed (HIGHESTMODSEQ 1)
S: * LIST (\Subscribed \HasNoChildren \UnMarked) "." INBOX.fgfgf
S: * STATUS INBOX.fgfgf (HIGHESTMODSEQ 3)
S: * LIST (\Subscribed \HasNoChildren \UnMarked) "." INBOX.34
S: * STATUS INBOX.34 (HIGHESTMODSEQ 5)
S: * LIST (\Subscribed \HasChildren \UnMarked) "." INBOX.
S: * STATUS INBOX. (HIGHESTMODSEQ 2)
S: * LIST (\Subscribed \HasNoChildren \UnMarked) "." INBOX..
S: * STATUS INBOX.. (HIGHESTMODSEQ 1)
S: * LIST (\Subscribed \HasNoChildren \UnMarked) "." INBOX.mkmkm
S: * STATUS INBOX.mkmkm (HIGHESTMODSEQ 1)
S: * LIST (\Subscribed \HasChildren \UnMarked) "." INBOX.sdfsf
S: * STATUS INBOX.sdfsf (HIGHESTMODSEQ 1)
S: * LIST (\HasChildren \UnMarked) "." INBOX.sdfsf.sdfsdfsdfsd
S: * STATUS INBOX.sdfsf.sdfsdfsdfsd (HIGHESTMODSEQ 1)
S: * LIST (\HasNoChildren \UnMarked) "." INBOX.sdfsf.sdfsdfsdfsd.
S: * STATUS INBOX.sdfsf.sdfsdfsdfsd. (HIGHESTMODSEQ 2263)
S: * LIST (\Subscribed \HasChildren \UnMarked) "." INBOX.uuyy
S: * STATUS INBOX.uuyy (HIGHESTMODSEQ 1)
S: * LIST (\Subscribed \HasChildren \UnMarked) "." INBOX.uuyy.sdfsdf
S: * STATUS INBOX.uuyy.sdfsdf (HIGHESTMODSEQ 1)
S: * LIST (\Subscribed \HasChildren \UnMarked) "." "INBOX.uuyy.sdfsdf.New 
Folder"
S: * STATUS "INBOX.uuyy.sdfsdf.New Folder" (HIGHESTMODSEQ 1)
S: * LIST (\Subscribed \HasChildren \UnMarked) "." "INBOX.uuyy.sdfsdf.New 
Folder.dfgdfg"
S: * STATUS "INBOX.uuyy.sdfsdf.New Folder.dfgdfg" (HIGHESTMODSEQ 1)
S: * LIST (\Subscribed \HasChildren \UnMarked) "." "INBOX.uuyy.sdfsdf.New 
Folder.dfgdfg.sdfsdfsdf"
S: * STATUS "INBOX.uuyy.sdfsdf.New Folder.dfgdfg.sdfsdfsdf" (HIGHESTMODSEQ 1)
S: * LIST (\Subscribed \HasChildren \UnMarked) "." "INBOX.uuyy.sdfsdf.New 
Folder.dfgdfg.sdfsdfsdf.sdfsdfsfsdf"
S: * STATUS "INBOX.uuyy.sdfsdf.New Folder.dfgdfg.sdfsdfsdf.sdfsdfsfsdf" 
(HIGHESTMODSEQ 1)
S: * LIST (\Subscribed \HasNoChildren \UnMarked) "." "INBOX.uuyy.sdfsdf.New 
Folder.dfgdfg.sdfsdfsdf.sdfsdfsfsdf.asdasdasd"
S: * STATUS "INBOX.uuyy.sdfsdf.New 
Folder.dfgdfg.sdfsdfsdf.sdfsdfsfsdf.asdasdasd" (HIGHESTMODSEQ 1)
S: * LIST (\Subscribed \HasNoChildren \UnMarked) "." INBOX.uuyy.fdfgdfg
S: * STATUS INBOX.uuyy.fdfgdfg (HIGHESTMODSEQ 2)
S: * LIST (\Subscribed \HasNoChildren \UnMarked) "." INBOX.vvvcv
S: * STATUS INBOX.vvvcv (HIGHESTMODSEQ 2627)
S: * LIST (\Subscribed \HasNoChildren \UnMarked) "." INBOX.hallo
S: * STATUS INBOX.hallo (HIGHESTMODSEQ 96)
S: * LIST (\Subscribed \HasChildren \UnMarked) "." "INBOX.New Folder"
S: * STATUS "INBOX.New Folder" (HIGHESTMODSEQ 1)
S: * LIST (\Subscribed \HasNoChildren \UnMarked) "." "INBOX.New 
Folder.omgfolder"
S: * STATUS "INBOX.New Folder.omgfolder" (HIGHESTMODSEQ 1)
S: * LIST (\Subscribed \HasNoChildren \UnMarked) "." "INBOX.New 
Folder.testfolder"
S: * STATUS "INBOX.New Folder.testfolder" (HIGHESTMODSEQ 1)
S: * LIST (\HasNoChildren \UnMarked) "." "INBOX.New Folder.hkhk"
S: * STATUS "INBOX.New Folder.hkhk" (HIGHESTMODSEQ 1)
S: * LIST (\Subscribed \HasNoChildren \UnMarked) "." "INBOX.New Folder.zzz"
S: * STATUS "INBOX.New Folder.zzz" (HIGHESTMODSEQ 1)
S: * LIST (\Subscribed \HasNoChildren \UnMarked) "." "INBOX.New Folder.aaa"
S: * STATUS "INBOX.New Folder.aaa" (HIGHESTMODSEQ 1)
S: * LIST (\Subscribed \HasNoChildren \UnMarked) "." "INBOX.New Folder.kgkgkk"
S: * STATUS "INBOX.New Folder.kgkgkk" (HIGHESTMODSEQ 1)
S: * LIST (\Subscribed \HasNoChildren \UnMarked) "." "INBOX.New Folder.gggfff"
S: * STATUS "INBOX.New Folder.gggfff" (HIGHESTMODSEQ 1)
S: * LIST (\Subscribed \HasNoChildren \UnMarked) "." "INBOX.New Folder.New 
Folder"
S: * STATUS