Re: Bug with shared access to mailbox

[John van]

On 03-06-16 08:20, aki.tuomi at dovecot.fi (Aki Tuomi) wrote:

On 30.05.2016 16:41, van der Kamp, John wrote:

Hello,

I'm testing dovecot with some setups, and one of them is with shared
mailboxes. The test I wrote will create and delete mail using multiple
connections to the same user and folder. Each connection makes a couple of
mails, remembers the uid from APPENDUID, and will delete those emails again.
At the end of the test I expect an empty folder.

This is not what happens. At the end I still have several mails in the
folder. I lack insight in the dovecot source to tell exactly what's going
on. I've tested this with different setups:
1) local system user, connecting over localhost -> bug is present
2) local system user, connecting over internet -> bug is present, but is
harder to reproduce
3) dovecot as proxy to another imap server -> bug is present
In step 3, you can even setup a dovecot to be a proxy to another dovecot
server.

  From logging in the other imap server I've seen that a client command to the
proxy like:
TAG UID STORE 1:3 +FLAGS (\Deleted)
TAG UID EXPUNGE 1:3
will be sent to the other imap server in 3 steps, one for each message. When
running the test with multiple threads, that logging shows that some uids
are never sent to the other imap server, and some uids are sent over
different connections than they original were sent to. (Thread 1 deletes
1:3, Thread 2 deletes 4:6, the proxy of Thread 1 might expunge messages from
Thread 2 and vice versa).

Attached is a python script which tests the behavior. The script expects a
file named "testmail.eml" to upload to the imap server. I used an email
which was about 75 kB.
I tested using version: 2.2.22 (fe789d2).
Let me know if I can help in any other way too.

John


Hi!

We tested with 2.2.24, and were unable to reproduce the error. Can you
try again with 2.2.24?

Aki


Hi,

Sorry for the late reply. Didn't notice when this was picked up.
I've tried again with out-of-the-box Ubuntu Xenial 16.04, which ships 
with 2.2.24. Here the problem is still present. I patched the packages 
with the commits Timo mentioned in another reply. That did not fix the 
problem the python script tries to reproduce.


I tried 2.2.25 on a Debian testing installation. This too is just 
out-of-the-box local users with mailbox in homedir configuration. I had 
to run the script a couple of times before it showed up again. So I 
guess things have been made better, but not flawless.


John

Re: Bug with shared access to mailbox

[Timo Sirainen]

On 03 Jun 2016, at 11:26, Dave  wrote:
> 
>> We tested with 2.2.24, and were unable to reproduce the error. Can you
>> try again with 2.2.24?
> 
> Apologies for butting in, but I've been seeing exactly the same issue post 
> upgrade to 2.2.24 (from 2.2.18):
> 
> [2016-06-02T10:38:28+0100] imap(x): Error: Corrupted index cache file 
> /mnt/index/8cc/95
> 2952/.INBOX/dovecot.index.cache: Broken MIME parts for mail UID 13758 in 
> mailbox INBOX: Cached MIME parts don't
> match message during parsing: Cached header size mismatch 
> (parts=4100f7020a030508000

This bug should have been fixed by 
https://github.com/dovecot/core/commit/20faa69d801460e89aa0b1214f3db4b026999b1e 
+ 
https://github.com/dovecot/core/commit/1bc6f1c54b4d77830288b8cf19060bd8a6db7b27

Re: Bug with shared access to mailbox

[Dave]

On 03/06/2016 07:20, Aki Tuomi wrote:


We tested with 2.2.24, and were unable to reproduce the error. Can you
try again with 2.2.24?


Apologies for butting in, but I've been seeing exactly the same issue 
post upgrade to 2.2.24 (from 2.2.18):


[2016-06-02T10:38:28+0100] imap(x): Error: Corrupted index cache 
file /mnt/index/8cc/95
2952/.INBOX/dovecot.index.cache: Broken MIME parts for mail UID 13758 in 
mailbox INBOX: Cached MIME parts don't
match message during parsing: Cached header size mismatch 
(parts=4100f7020a030508000

031080300480018035b005f00f100f50
00400400086042700290064016b01440
033061e002000860497041100010041005106000
085049504)

etc.

At first I assumed it was better at picking up corrupted indexes and 
would reduce in severity over time. However, I've also seen that 
attempting to force-resync or remove and rebuild indexes doesn't help - 
it reoccurs on the same mailboxes.


Unfortunately I'm unable to reveal mails from affected mailboxes, so I'm 
not sure how much help this is beyond a "me too".


doveconf (dovecot behind directors)

# doveconf -n

# 2.2.24 (a82c823): /etc/dovecot/dovecot.conf
# Pigeonhole version 0.4.14 (099a97c)
# OS: Linux 2.6.32-573.26.1.el6.x86_64 x86_64 CentOS release 6.7 (Final)
auth_anonymous_username =
auth_failure_delay = 0
auth_master_user_separator = *
auth_username_chars = 
abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ01234567890.-_

auth_worker_max_count = 64
base_dir = /var/run/dovecot/
default_client_limit = 5120
default_process_limit = 128
default_vsz_limit = 64 M
deliver_log_format = msgid=%m from=<%e> (%f) to=<%{to_envelope}>: %$
disable_plaintext_auth = no
first_valid_gid = 2000
first_valid_uid = 2000
lda_mailbox_autocreate = yes
lda_mailbox_autosubscribe = yes
lda_original_recipient_header = X-Original-To
listen = 192.168.0.214
lmtp_hdr_delivery_address = none
lmtp_rcpt_check_quota = yes
log_path = /var/log/dovecot/dovecot.log
log_timestamp = "[%Y-%m-%dT%H:%M:%S%z] "
login_greeting = Mail Server Ready
login_log_format_elements = pid=%e user=<%u> ip=%r
login_trusted_networks = 192.168.0.223 192.168.0.224 192.168.0.225 
192.168.0.226 192.168.0.227 192.168.0.228

mail_access_groups = doveuser
mail_fsync = always
mail_home = /mnt/mail/%3Mu/%u
mail_location = 
maildir:~:INDEX=/mnt/index/%3Mu/%i:CONTROL=/mnt/control/%3Mu/%u

mail_plugins = " stats"
mailbox_idle_check_interval = 2 mins
mailbox_list_index = yes
maildir_very_dirty_syncs = yes
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope 
encoded-character vacation subaddress comparator-i;ascii-numeric 
relational regex imap4flags copy include variables body enotify mailbox 
date index ihave duplicate mime foreverypart extracttext

mmap_disable = yes
passdb {
   args = /etc/dovecot/passwd.master
   driver = passwd-file
   master = yes
   pass = yes
   result_failure = return-fail
   result_internalfail = return-fail
}
passdb {
   args = /etc/dovecot/sql.d/user.ext
   driver = sql
}
plugin {
   sieve = file:~/sieve;active=~/sieve/.active.sieve
   sieve_extensions = -environment
   sieve_max_actions = 32
   sieve_max_redirects = 4
   sieve_max_script_size = 1M
   sieve_quota_max_scripts = 16
   sieve_quota_max_storage = 2M
   sieve_redirect_envelope_from = sender
   sieve_user_log = /dev/null
   stats_refresh = 30 secs
   stats_track_cmds = no
}
protocols = lmtp imap pop3 sieve
recipient_delimiter =
service auth-worker {
   user = $default_internal_user
   vsz_limit = 128 M
}
service auth {
   client_limit = 7664
   vsz_limit = 128 M
}
service imap-login {
   inet_listener imap {
 port = 10143
   }
   service_count = 0
}
service imap {
   process_limit = 5120
   process_min_avail = 24
   vsz_limit = 512 M
}
service lmtp {
   executable = lmtp -L
   inet_listener lmtp {
 port = 10024
   }
   process_limit = 128
   process_min_avail = 12
   vsz_limit = 512 M
}
service managesieve-login {
   inet_listener sieve {
 port = 14190
   }
   service_count = 0
   vsz_limit = 512 M
}
service managesieve {
   process_limit = 128
}
service pop3-login {
   inet_listener pop3 {
 port = 10110
   }
   service_count = 0
}
service pop3 {
   process_limit = 2048
   process_min_avail = 24
   vsz_limit = 512 M
}
service stats {
   fifo_listener stats-mail {
 group = doveuser
 mode = 0660
   }
   vsz_limit = 256 M
}
ssl = no
stats_command_min_time = 0
stats_ip_min_time = 2 mins
stats_memory_limit = 64 M
stats_session_min_time = 2 mins
stats_user_min_time = 2 mins
userdb {
   args = /etc/dovecot/sql.d/user.ext
   driver = sql
}
verbose_proctitle = yes
protocol lmtp {
   auth_username_chars =
   auth_username_format = %Ln

Re: Bug with shared access to mailbox

[Aki Tuomi]

On 30.05.2016 16:41, van der Kamp, John wrote:

Hello,

I'm testing dovecot with some setups, and one of them is with shared
mailboxes. The test I wrote will create and delete mail using multiple
connections to the same user and folder. Each connection makes a couple of
mails, remembers the uid from APPENDUID, and will delete those emails again.
At the end of the test I expect an empty folder.

This is not what happens. At the end I still have several mails in the
folder. I lack insight in the dovecot source to tell exactly what's going
on. I've tested this with different setups:
1) local system user, connecting over localhost -> bug is present
2) local system user, connecting over internet -> bug is present, but is
harder to reproduce
3) dovecot as proxy to another imap server -> bug is present
In step 3, you can even setup a dovecot to be a proxy to another dovecot
server.

 From logging in the other imap server I've seen that a client command to the
proxy like:
TAG UID STORE 1:3 +FLAGS (\Deleted)
TAG UID EXPUNGE 1:3
will be sent to the other imap server in 3 steps, one for each message. When
running the test with multiple threads, that logging shows that some uids
are never sent to the other imap server, and some uids are sent over
different connections than they original were sent to. (Thread 1 deletes
1:3, Thread 2 deletes 4:6, the proxy of Thread 1 might expunge messages from
Thread 2 and vice versa).

Attached is a python script which tests the behavior. The script expects a
file named "testmail.eml" to upload to the imap server. I used an email
which was about 75 kB.
I tested using version: 2.2.22 (fe789d2).
Let me know if I can help in any other way too.

John


Hi!

We tested with 2.2.24, and were unable to reproduce the error. Can you 
try again with 2.2.24?


Aki