[enhancement] fts-solr low performance

[azurit]

Hi,

we have activated fts-solr about a week ago and immediately started to  
experience really *low* performance with MOVE and EXPUNGE commands.  
After several days of googling, tcpdumping and straceing i was able to  
find and resolve the problem.


We are using Dovecot 2.2.27 from Debian Jessie (jessie-backports),  
which is doing a soft commit in solr after every MOVE or EXPUNGE  
command - this behavior cannot be, currently, changed. The problem is  
that this was causing every MOVE/EXPUNGE to take about 6 seconds to  
complete. The problem appears to be in very old version of Solr -  
3.6.2 (!!). This is the only version which is shipped with current  
(Jessie) and also next (Stretch) version of Debian, don't ask my why,  
i don't understand it either. Solr versions below 4.0 are NOT  
supporting soft commits, so all commits are hard and this was the  
problem. Finally, i decided to patch our Dovecot to not send a commit  
at all and everything started to be super fast. I'm doing hard commits  
every minute via cron so the only consequence of this is that you  
cannot search for messages delivered before less then a minute (which  
you, usually, don't need to do anyway).


While googling i also find out that Solr supports autoCommit function  
(and from version 4.0 also autoSoftCommit), so there's no reason for  
Dovecot to handle this on it's own (and potentially doing hundreds or  
thousands of soft commits every second) - you can just set Solr to,  
for example, do autoSoftCommit every second and autoCommit every minute:

https://cwiki.apache.org/confluence/display/solr/UpdateHandlers+in+SolrConfig#UpdateHandlersinSolrConfig-autoCommit

Also this wiki page should be updated with warning about old versoins  
of Solr not supporting soft commits (you could also mention the  
auto[Soft]Commit function):

http://wiki2.dovecot.org/Plugins/FTS/Solr

I suggest to allow completely disable Solr commits in Dovecot by  
configuration, so people like me can handle this easily. What do you  
think?


azur

Re: fts_solr and connection via https://

[Jan Vonde]

Am 04.03.2017 um 15:32 schrieb Stephan Bosch:
> Op 3/4/2017 om 2:39 PM schreef Jan Vonde:
>> Am 17.02.2017 um 17:27 schrieb Jan Vonde:
>>> Am 17.02.2017 um 11:45 schrieb Stephan Bosch:
 Op 8-2-2017 om 21:07 schreef Jan Vonde:
 We seem to have found another issue there. More on this will follow.

>>> Thanks for the update and have a nice weekend,
>>>
>> I don't want to push, am just interested: any news on this?
>>
>>
>> Thanks, Jan :-)
> 
> Oh, good point. We added a few fixes, but unfortunately the last of
> those was too late for 2.2.28:
> 
> https://git.dovecot.net/dovecot/core/commit/8f251da1b6dfe6dc3d86ae71b377d99afe2d4bd2
> 
> So, currently, may not yet work for you. I will be in 2.2.29. You can
> try the master branch of course if you want to test it early.
> 
> 
> Regards,
> 
> Stephan.
> 
It's working. Awesome! Thanks a lot!


\Jan :-)

Re: fts_solr and connection via https://

[Stephan Bosch]

Op 3/4/2017 om 2:39 PM schreef Jan Vonde:
> Am 17.02.2017 um 17:27 schrieb Jan Vonde:
>> Am 17.02.2017 um 11:45 schrieb Stephan Bosch:
>>> Op 8-2-2017 om 21:07 schreef Jan Vonde:
>>> We seem to have found another issue there. More on this will follow.
>>>
>> Thanks for the update and have a nice weekend,
>>
> I don't want to push, am just interested: any news on this?
>
>
> Thanks, Jan :-)

Oh, good point. We added a few fixes, but unfortunately the last of
those was too late for 2.2.28:

https://git.dovecot.net/dovecot/core/commit/8f251da1b6dfe6dc3d86ae71b377d99afe2d4bd2

So, currently, may not yet work for you. I will be in 2.2.29. You can
try the master branch of course if you want to test it early.


Regards,

Stephan.

Re: fts_solr and connection via https://

[Jan Vonde]

Am 17.02.2017 um 17:27 schrieb Jan Vonde:
> Am 17.02.2017 um 11:45 schrieb Stephan Bosch:
>> Op 8-2-2017 om 21:07 schreef Jan Vonde:
>>> Am 07.02.2017 um 12:29 schrieb Stephan Bosch:
 Op 31-1-2017 om 6:33 schreef Jan Vonde:
> Am 31.01.2017 um 00:04 schrieb Stephan Bosch:
>> Op 1/22/2017 om 12:01 PM schreef Stephan Bosch:
>>> Op 1/22/2017 om 10:01 AM schreef Jan Vonde:
 I tried adding the following settings but that didn't help:
ssl_ca = < /etc/ssl/certs/ca-certificates.crt
ssl_client_ca_dir = /etc/ssl/certs

 Can you give me a hint how I can get the ssl certificate accepted?
>>> That should normally have done the trick. However, the sources
>>> tell me
>>> that no ssl_client settings are propagated to the http_client
>>> used by
>>> fts-solr, so SSL is not currently supported it seems.
>>>
>>> I'll check how easy it is to add that.
>> Just to keep you informed: I created a patch, but it is still being
>> tested.
>>
> Thanks for the update Stephan! Awesome! Looking forward to test it
> myself :-)
 https://github.com/dovecot/core/commit/526631052ca3175357302af8fa7dcbf763b40c53



>>> Thank you. I am using now the following version:
>>>2.3.0.alpha0 (2eeea57) [XI:2:2.3.0~alpha0-1~auto+650]
>>>
>>> The error messages I am getting now are like this:
>>>
>>> doveadm(user@host): Info: Received invalid SSL certificate: unable to
>>> get local issuer certificate: /C=US/O=Let's Encrypt/CN=Let's Encrypt
>>> Authority X3
>>> doveadm(user@host): Error: fts_solr: Lookup failed: 9002 SSL handshaking
>>> with 5.45.106.248:443 failed: read(SSL 5.45.106.248:443) failed:
>>> Received invalid SSL certificate: unable to get local issuer
>>> certificate: /C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
>>>
>>>
>>> You can connect to 5.45.106.248:443 and IMHO everything is correct with
>>> the chain.
>>>
>>>
>>> I am no SSL expert, but I am reading it as "doveadm and its ssl part
>>> cannot verify the Let's Encrypt certificate". It would need the DST Root
>>> CA X3 and this is in the local trust store (ssl_client_ca_dir...)
>>>
>>>
>>> Do you have another hint maybe?
>>
>> We seem to have found another issue there. More on this will follow.
>>
> Thanks for the update and have a nice weekend,
> 
I don't want to push, am just interested: any news on this?


Thanks, Jan :-)

mdbox Inconsistency in map index

[bOnK]

After a (power)crash two accounts have been corrupted.
I tried to rescue things by running force-resync multiple times, but 
that didn't work out.
Searching the archives, I found a recent suggestion (by Timo) to delete 
the index files, but I'm not sure which files to delete and what the 
consequences will be.
When I deleted ``storage/dovecot.map.index'' in an unimportant account 
of my own, things only grew worse and I decided to delete the account 
and start fresh...


Something I obviously don't want to do with the other account as it 
belongs to a client with about 2.5G mails (856 m.* files), which we 
don't want to lose.


I have observed the following behavior in the logs:
When a new mail arrives:
(The second line below is always the same, however there's no mail with 
uid 12819 according to doveadm search uid 12819)
Mar 04 11:43:28 lda(user): Warning: mdbox /var/mail/user/storage: 
Inconsistency in map index (117,1060 != 117,1883660)
Mar 04 11:43:28 lda(user): Error: Log synchronization error at 
seq=117,offset=1546660 for /var/mail/user/storage/dovecot.map.index: 
Extension record inc drops number below zero (uid=12819, diff=-1, orig=0)
Mar 04 11:43:28 lda(user): Warning: fscking index file 
/var/mail/user/storage/dovecot.map.index
Mar 04 11:43:28 lda(user): Info: 
msgid=: saved 
mail to INBOX
Mar 04 11:43:28 lda(user): Warning: mdbox /var/mail/user/storage: 
Inconsistency in map index (117,1060 != 117,1883784)
Mar 04 11:43:28 lda(user): Warning: fscking index file 
/var/mail/user/storage/dovecot.map.index
Mar 04 11:43:28 lda(user): Warning: mdbox /var/mail/user/storage: 
rebuilding indexes
Mar 04 11:43:28 lda(user): Warning: fscking index file 
/var/mail/user/storage/dovecot.map.index
Mar 04 11:43:28 imap(user): Error: 
/var/mail/user/mailboxes/INBOX/dbox-Mails/dovecot.index reset, view is 
now inconsistent
Mar 04 11:43:28 imap(user): Info: IMAP session state is inconsistent, 
please relogin. in=2011 out=108358 deleted=0 expunged=0 trashed=0


When user logs in next (might be from another box/MUA)
Mar 04 11:44:36 imap-login: Info: Login: user=, method=PLAIN, 
rip=82.95.XX.XX, lip=37.97.XX.XX, mpid=14884, TLS, 
session=
Mar 04 11:44:37 imap(user): Warning: mdbox /var/mail/user/storage: 
Inconsistency in map index (117,1060 != 117,1884336)
Mar 04 11:44:37 imap(user): Warning: fscking index file 
/var/mail/user/storage/dovecot.map.index
Mar 04 11:44:37 imap(user): Warning: mdbox /var/mail/user/storage: 
rebuilding indexes
Mar 04 11:44:37 imap(user): Error: 
/var/mail/user/mailboxes/INBOX/dbox-Mails/dovecot.index reset, view is 
now inconsistent
Mar 04 11:44:37 imap(user): Info: IMAP session state is inconsistent, 
please relogin. in=331 out=178391 deleted=0 expunged=0 trashed=0
Mar 04 11:44:37 imap(user): Warning: mdbox /var/mail/user/storage: 
Inconsistency in map index (117,1060 != 117,1884396)
Mar 04 11:44:37 imap(user): Warning: fscking index file 
/var/mail/user/storage/dovecot.map.index
Mar 04 11:44:37 imap(user): Warning: fscking index file 
/var/mail/user/storage/dovecot.map.index
Mar 04 11:44:37 imap-login: Info: Login: user=, method=PLAIN, 
rip=82.95.XX.XX, lip=37.97.XX.XX, mpid=14886, TLS, 
session=


Besides the nuisance of getting kicked of every time a new mail arrives, 
user cannot delete mails (as in: move to Trash).


TL;DR: which files to delete and what will be the consequences?

--
b.