On 2025-08-14 6:22 PM, Jonathan Kamens wrote:
On 8/14/25 2:23 PM, Nels Lindquist wrote:
What configuration directives do you have for the conversations
database (which is required for xapian) and for search?
conversations: 1
conversations_db: twoskip
search_engine: xapian
defaultsearchtier: default
defaultsearchpartition-default: /var/spool/imap-search
sync_log: on
sync_log_channels: squatter
You probably also want to add:
search_fuzzy_always: yes
That won't affect the behaviour you're seeing, but since Xapian can only
use fuzzy searches, it'll translate client requests for standard
searches for use with the Xapian engine which may increase client
performance.
There's a longstanding issue with xapian indexing which can be worked
around by setting search_batchsize to a very large number (like
1000000), blowing away the contents of your search partition and then
regenerating with "squatter -v -i".
Yes, that sort of resolves the issue. I was able to rebuild my indexes,
but squatter -R still appears to be doing a ton of extra work every time
it starts up and causing excessive locking which is blocking clients
from functioning for noticeable delays while it is considering
reindexing every single message in every mailbox every time it starts up.
It's disappointing that this longstanding issue hasn't been fixed and
doesn't seem to have been documented anywhere obvious that I could find.
Because of that I wasted many hours upgrading to 3.12.1 and then digging
into squatter in a debugger to try to figure out what's going on.
Of note: when I said in my last message that the rolling squatter
process was doing the right thing I was mistaken. In fact the rolling
squatter was having the same problem, it was just rotating through the
mailboxes and reindexing the same messages in each one in turn, rather
than sticking with a single mailbox and indexing the same messages in
that mailbox over and over, so I didn't notice right away that it was
also malfunctioning.
Is your "squatter -R" setup in cyrus.conf still in the EVENTS section,
by chance?
If so, it should be moved to the DAEMON section. Once it's up and
running it should be continuously rolling search indexing from the
squatter replication channel you've defined, and shouldn't be doing much
of anything otherwise.
If you set up multiple search tiers for different types of storage, you
might still need some squatter EVENTS for periodically repacking the
search indexes between storage tiers. We haven't bothered with that as
the performance has been great with a single tier with our typical
loads. (See
http://lists.tartarus.org/pipermail/xapian-discuss/2014-October/009112.html
for some examples.)
We haven't noted mailbox locking due to squatter activity (even on
startup) in our environment, but we're still running Cyrus IMAPD 3.8.5
(self-built RPMs) on AlmaLinux 8, so no idea whether the behaviour's
changed in 3.10.x or 3.12.x
What kind of squatter logs are you seeing? This is pretty typical for us:
Aug 15 11:54:35 edm-cmbe01 cyrus/squatter[2716]: indexing mailbox
user/[redacted1]@maei.ca...
Aug 15 11:54:35 edm-cmbe01 cyrus/squatter[2716]: Xapian committed 3
updates in 0.353203 sec
Aug 15 12:10:49 edm-cmbe01 cyrus/squatter[2716]: indexing mailbox
user/[redacted2]/[email protected]...
Aug 15 12:10:53 edm-cmbe01 cyrus/sync_client[2700]: Reprocessing sync
log file /var/lib/imap/sync/squatter/log-run
Aug 15 12:10:54 edm-cmbe01 cyrus/squatter[2716]: Xapian committed 2
updates in 4.963152 sec
Aug 15 12:10:54 edm-cmbe01 cyrus/squatter[2716]: mappedfile: longlock
/var/lib/imap/user/uuid/5/c/5cf27ed7-050e-44dd-924d-eefb6835df76/xapianactive.db
for 5.0 seconds
--
Nels Lindquist
[email protected]
------------------------------------------
Cyrus: Info
Permalink:
https://cyrus.topicbox.com/groups/info/T5a3c00cdf15ad8da-Md013a07964c79d42e9c7c413
Delivery options: https://cyrus.topicbox.com/groups/info/subscription
