-------- Original Message --------
Subject: Re: Dovecot/K9/Thunderbird IMAP interaction issues: replicated
messages, constant reloading (TB) & missing new messages (K-9)
From: David Gessel <ges...@blackrosetech.com>
To: Alex JOST <jost+li...@dimejo.at>, dovecot@dovecot.org
Date: Tue Jul 15 2014 15:08:31 GMT+0300 (Arabic Standard Time)
> -------- Original Message --------
> Subject: Re: Dovecot/K9/Thunderbird IMAP interaction issues: replicated
> messages, constant reloading (TB) & missing new messages (K-9)
> From: Alex JOST <jost+li...@dimejo.at>
> To: dovecot@dovecot.org
> Date: Tue Jul 15 2014 13:30:31 GMT+0300 (Arabic Standard Time)
>
>> Am 15.07.2014 03:12, schrieb David Gessel:
>>> * In thunderbird, I get constant downloading of messages. Every time I
>>> switch to the inbox from another folder, the messages start
>>> redownloading. This sometimes, but not always, seems to create
>>> duplicate messages, but I'm now fairly sure they are only created in the
>>> client and not on the server. Searching through various forums, it
>>> appears this has been a problem for Thunderbird for some time, but it
>>> wasn't for me on Courier. As posts about similar errors seem to have
>>> petered out over the last year without any clear guidance on how to fix
>>> it that I could find, I expect there's a simple fix I just haven't found
>>> yet, either in TB configuration or in Dovecot. Any hints? I tried
>>> Thunderfix to remove the .msf files and will be redownloading my mail
>>> over the thin pipe out to Iraq for the next 6 months (DOH!). One
>>> artifact is that when I restart thunderbird, my inbox is reported as
>>> having about 130 messages in it, then thunderbird starts redownloading
>>> headers and eventually gets back up to the 1297 currently listed both in
>>> the TB pane and by ls | wc -l in /cur. Is it possible there is a time
>>> out that needs to be extended? In TB, I've set (among other less
>>> apparently relevant options)
>>> mailnews.tcptimeout;1000
>>> mail.db.idle_limit;30000000
>>
>> How old is your Thunderbird profile. Have you tried creating a new one
>> and see if the problem persists?
>
> It is a new profile created with the transition from Cyrus to Dovecot
> about a week ago, but I used IMAPSync to move all my mail from the
> ancient Cyrus server to the shiny new Dovecot server. I then deleted
> the old profile.
>
>>
>> Thunderfix should not be needed by the way. Just right click on the
>> folder => Properties => Repair
>
> I did see that, but first thought there was something magic about
> "thunderfix" then decided to use it anyway because I have annual
> sent/recieved archives from 2000 (and one for pre-2000) so that would
> have been a lotta right-clicking.
>
>>
>>
>>> server # dovecot -n
>>> # 2.2.13: /usr/local/etc/dovecot/dovecot.conf
>>> # OS: FreeBSD 9.2-RELEASE amd64
>>> auth_default_realm = mydomain.com
>>> auth_mechanisms = plain login
>>> auth_realms = mydomain.com mydomain2.org
>>> disable_plaintext_auth = no
>>> first_valid_gid = 5000
>>> first_valid_uid = 5000
>>> hostname = mydomain.com
>>> imap_client_workarounds = tb-extra-mailbox-sep tb-lsub-flags
>>> imap_idle_notify_interval = 29 mins
>>> last_valid_gid = 5000
>>> last_valid_uid = 5000
>>> lda_mailbox_autocreate = yes
>>> listen = *
>>> mail_gid = 5000
>>> mail_location = maildir:~/:CONTROL=/var/no-quota/%u
>>> mail_plugins = quota mail_log notify
>>> mail_temp_dir = /var/tmp
>>> mail_uid = 5000
>>> maildir_broken_filename_sizes = yes
>>> managesieve_notify_capability = mailto
>>> managesieve_sieve_capability = fileinto reject envelope
>>> encoded-character vacation subaddress comparator-i;ascii-numeric
>>> relational regex imap4flags copy include variables body enotify
>>> environment mailbox date ihave duplicate spamtest spamtestplus editheader
>>> mbox_lazy_writes = no
>>> namespace inbox {
>>> inbox = yes
>>> location =
>>> mailbox Drafts {
>>> special_use = \Drafts
>>> }
>>> mailbox Junk {
>>> special_use = \Junk
>>> }
>>> mailbox Sent {
>>> special_use = \Sent
>>> }
>>> mailbox "Sent Messages" {
>>> special_use = \Sent
>>> }
>>> mailbox Trash {
>>> special_use = \Trash
>>> }
>>> prefix =
>>> }
>>> passdb {
>>> args = /usr/local/etc/dovecot/dovecot-sql.conf.ext
>>> driver = sql
>>> }
>>> plugin {
>>> expire = Trash
>>> mail_log_events = delete undelete expunge copy mailbox_delete
>>> mailbox_rename
>>> mail_log_fields = uid box msgid size
>>> quota = maildir:User quota
>>> quota_exceeded_message = Storage quota for this account has been
>>> exceeded, please try again later.
>>> quota_rule = *:storage=50G
>>> quota_rule2 = Trash:storage=+30%%
>>> quota_rule3 = Sent:storage=+30%%
>>> quota_warning = storage=90%% quota-warning 90 %u
>>> quota_warning2 = storage=75%% quota-warning 75 %u
>>> sieve = ~/.dovecot.sieve
>>> sieve_before = /usr/local/etc/dovecot/sieve/
>>> sieve_dir = ~/sieve
>>> sieve_extensions = +spamtest +spamtestplus +relational
>>> +comparator-i;ascii-numeric +editheader
>>> }
>>> postmaster_address = ges...@mydomain.com
>>> protocols = imap lmtp sieve sieve
>>> sendmail_path = /usr/local/sbin/sendmail
>>> service auth {
>>> unix_listener /var/spool/postfix/private/auth {
>>> group = postfix
>>> mode = 0666
>>> user = postfix
>>> }
>>> unix_listener auth-userdb {
>>> group = vmail
>>> mode = 0600
>>> user = vmail
>>> }
>>> }
>>> service managesieve-login {
>>> inet_listener sieve {
>>> port = 4190
>>> }
>>> service_count = 1
>>> vsz_limit = 128 M
>>> }
>>> service quota-warning {
>>> executable = script /usr/local/bin/quota-warning.sh
>>> unix_listener quota-warning {
>>> user = vmail
>>> }
>>> user = vmail
>>> }
>>> ssl = required
>>> ssl_cert = </usr/local/etc/ca/server-cert.pem
>>> ssl_cipher_list =
>>> ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-RSA-RC4-SHA:ECDHE-RSA-AES256-SHA:!SSLv2
>>>
>>> ssl_key = </usr/local/etc/ca/server-unencrypted-key.pem
>>> ssl_prefer_server_ciphers = yes
>>> userdb {
>>> driver = prefetch
>>> }
>>> userdb {
>>> args = /usr/local/etc/dovecot/dovecot-sql.conf.ext
>>> driver = sql
>>> }
>>> protocol lda {
>>> mail_plugins = quota mail_log notify sieve
>>> }
>>> protocol imap {
>>> mail_max_userip_connections = 20
>>> mail_plugins = quota mail_log notify imap_quota imap_zlib
>>> }
>>> protocol lmtp {
>>> mail_plugins = quota mail_log notify sieve
>>> }
>>>
>>
>> I don't know if it makes a difference for you as you are not using
>> "LAYOUT=fs" but try to disable those imap_client_workarounds.
>>
>
> Good catch, thank you. (I used LAYOUT=fs at first, but found behavior a
> little anomalous. I forgot to disable the workarounds).
>
> This may have stopped the problem of redownloading the entire mailbox
> (on restart I got downloading 1 of 8 instead of 1 of 170... 30 of 1330
> in the status pane) . My other mailboxes still need to sync and it
> might take a few days to verify that things are stable. I did get 81
> immediate "Warning: Maildir /mail/mydomain.com/gessel//.Bulkmail:
> Expunged message reappeared, giving a new UID" warnings on connect, but
> if this is the problem, it might stop after all the directories are
> resynced.
>
> I'll update with further data as it comes in.
>
> I am also somewhat suspect of QRSYNC from this post:
> http://dovecot.org/pipermail/dovecot/2011-August/130687.html - though
> this is from a long time ago and it seems a little surprising that bugs
> on this seemingly useful capability would persist. In any event (if
> useful information), my capability enumeration reads:
>
> # telnet localhost 143
>
> * OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE
> IDLE STARTTLS AUTH=PLAIN AUTH=LOGIN] Dovecot ready.
> a login user pass
> a OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE
> IDLE SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS
> THREAD=ORDEREDSUBJECT MULTIAPPEND URL-PARTIAL CATENATE UNSELECT CHILDREN
> NAMESPACE UIDPLUS LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH
> ESORT SEARCHRES WITHIN CONTEXT=SEARCH LIST-STATUS SPECIAL-USE BINARY
> MOVE COMPRESS=DEFLATE QUOTA] Logged in
>
>
> I did try setting mail.server.default.use_condstore;false in
> Thunderbird, but that didn't change the behavior of reloading the entire
> inbox on startup so I've changed that back as that wasn't the desired
> result.
>
Disabling the imap_client_workarounds in TB ended up neither helping nor
harming. The first sync was promising but as soon as that was finished
(it can take a while to download 8 messages to Iraq) it started over
re-downloading the entire mailbox. I don't seem to be generating
duplicate messages, but I am redownloading every message over and over.
My TB activity manager is filled with paused "bringing folder XXX up to
date" messages.
Is there any indication of current problems with CONDSTORE/QRESYNC since
2011 or so?
I've set:
imap_capability = IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE
IDLE SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS
THREAD=ORDEREDSUBJECT MULTIAPPEND URL-PARTIAL CATENATE UNSELECT CHILDREN
NAMESPACE UIDPLUS LIST-EXTENDED I18NLEVEL=1 ESEARCH ESORT SEARCHRES
WITHIN CONTEXT=SEARCH LIST-STATUS SPECIAL-USE BINARY MOVE
COMPRESS=DEFLATE QUOTA
and will report if this yields any changes
