Re: Searching 30 GB mailbox

2021-12-03 Thread Peter
Am 03.12.21 um 13:34 schrieb Einar Bjarni Halldórsson: Why are you still going through Dovecot for this ? @ 4.5 million messages you could just reprogram your app to search solr directly, it will return the uid of the message and you can use that directly in imap uid fetch. That is a good id

Re: Searching 30 GB mailbox

2021-12-03 Thread Einar Bjarni Halldórsson
Why are you still going through Dovecot for this ? @ 4.5 million messages you could just reprogram your app to search solr directly, it will return the uid of the message and you can use that directly in imap uid fetch. That is a good idea. If this becomes a showstopper, that's one way we coul

RE: Searching 30 GB mailbox

2021-12-02 Thread Scott
Halldórsson Sent: Thursday, December 2, 2021 7:47 AM To: dovecot@dovecot.org Subject: Re: Searching 30 GB mailbox > > If I search for the UID I get > > -- > 3 fetch 5166713 ENVELOPE > 3 BAD Error in IMAP command FETCH: Invalid messageset (0.001 + 0.000 > secs). > --

Re: Searching 30 GB mailbox

2021-12-02 Thread Einar Bjarni Halldórsson
If I search for the UID I get -- 3 fetch 5166713 ENVELOPE 3 BAD Error in IMAP command FETCH: Invalid messageset (0.001 + 0.000 secs). -- If I run the search in IMAP I get -- 4 SEARCH HEADER Messgage-ID <64jf4cstuxzuvatjbbrvp3ibjnmbeguvxzoa1osm...@min.isnic.is> * SEARCH * 4465180 EXISTS *

Re: Searching 30 GB mailbox

2021-12-02 Thread Einar Bjarni Halldórsson
If you have fts_enforced = yes then that SEARCH probably goes to solr even if message-id is found from cache. So the delay is probably on solr side. Can you try to remove fts_enforced for that one particular user? I only added fts_enforced yesterday. When I sent the original post I wasn't runn

Re: Searching 30 GB mailbox

2021-12-02 Thread Sami Ketola
> On 2. Dec 2021, at 10.57, Einar Bjarni Halldórsson wrote: > root@ht-mailstore01:/data/mail/hostmasterlog/mdbox/mailboxes/INBOX/dbox-Mails > # doveadm dump . | grep -c hdr.MESSAGE-ID > 4464736 > > In dovecot config I have `fts_enforced = yes` and after the search for > Message-ID finishes a

Re: Searching 30 GB mailbox

2021-12-02 Thread Einar Bjarni Halldórsson
You can inspect the index files with doveadm dump to check what is cached. Not sure how it went with mdbox storage driver. According to `man doveadm-dump` it just seems to dump index files. I tried dumping the cache file but it complains that it can't auto detect the file type. Try this: [

Re: Searching 30 GB mailbox

2021-12-01 Thread Sami Ketola
> On 1. Dec 2021, at 13.05, Einar Bjarni Halldórsson wrote: > > >> Adding it to mail_always_cache_fields does not miraculously make cache >> updated. Each mail needs to be accessed for that to happen. >> Ie. With doveadm fetch. > I couldn't figure out the doveadm fetch syntax for searching so

Re: Searching 30 GB mailbox

2021-12-01 Thread Einar Bjarni Halldórsson
Adding it to mail_always_cache_fields does not miraculously make cache updated. Each mail needs to be accessed for that to happen. Ie. With doveadm fetch. doveadm fetch -u username hdr.message-id all Tried that and also tried deleting the dovecot.index.cache and running again, same result.

Re: Searching 30 GB mailbox

2021-12-01 Thread Aki Tuomi
> On 01/12/2021 13:05 Einar Bjarni Halldórsson wrote: > > > > Adding it to mail_always_cache_fields does not miraculously make cache > > updated. Each mail needs to be accessed for that to happen. > > Ie. With doveadm fetch. > I couldn't figure out the doveadm fetch syntax for searching so I

Re: Searching 30 GB mailbox

2021-12-01 Thread Einar Bjarni Halldórsson
Adding it to mail_always_cache_fields does not miraculously make cache updated. Each mail needs to be accessed for that to happen. Ie. With doveadm fetch. I couldn't figure out the doveadm fetch syntax for searching so I just ran the search manually twice. I assume that would update the cache

Re: Searching 30 GB mailbox

2021-12-01 Thread Sami Ketola
> On 1. Dec 2021, at 10.59, Einar Bjarni Halldórsson wrote: > >> Did you have Messsage-ID listed in mail_always_cache_fields when you >> imported the mailbox? >> Do you have it now? >> >> If not, please add it and then use doveadm fetch to populate that in cache. >> >> I think the correct s

Re: Searching 30 GB mailbox

2021-12-01 Thread Einar Bjarni Halldórsson
Did you have Messsage-ID listed in mail_always_cache_fields when you imported the mailbox? Do you have it now? If not, please add it and then use doveadm fetch to populate that in cache. I think the correct setting is mail_always_cache_fields = hdr.message-id But you also might want to add so

Re: Searching 30 GB mailbox

2021-12-01 Thread Sami Ketola
> On 1. Dec 2021, at 0.30, Einar Bjarni Halldórsson wrote: > > Hi, > > We're migrating our mailboxes from Zimbra to dovecot, running on FreeBSD > 12.2. We're using mdbox and solr for fts. dovecot version 2.3.17. > > One of our mailboxes gets a bcc of every outgoing mail from our system and

Re: Searching 30 GB mailbox

2021-12-01 Thread Einar Bjarni Halldórsson
Sorry, resent because it didn't go to the list... Can you provide us output of `doveconf -n` please? Of course! # 2.3.17 (e2aa53df5b): /usr/local/etc/dovecot/dovecot.conf # Pigeonhole version 0.5.17 (054dddfa) # OS: FreeBSD 12.2-RELEASE-p7 amd64  zfs # Hostname: ht-mailstore01.isnic.is auth_us

Searching 30 GB mailbox

2021-11-30 Thread Einar Bjarni Halldórsson
Hi, We're migrating our mailboxes from Zimbra to dovecot, running on FreeBSD 12.2. We're using mdbox and solr for fts. dovecot version 2.3.17. One of our mailboxes gets a bcc of every outgoing mail from our system and it currently holds ~4.5 million messages taking 30 GB. One of the ways we