Re: 2.4 ballpark release date?

[Shawn Heisey via dovecot]

On 4/30/23 22:53, Aki Tuomi via dovecot wrote:

2.4 is planned for sometime after summer this year.
Aki


Considering that it's now almost a year later, this seems to have slipped.

Thanks,
Shawn

___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Logging for doveadm index?

[Shawn Heisey]

I have dovecot using Solr for FTS.  Dovecot version is 
2:2.3.20-3+ubuntu20.04 pulled from the official APT repo.


This is my reindex script:

https://paste.elyograg.org/view/74accd23

The reindex script kicks off the indexing process in the background.

Dovecot logs to syslog, and those logs ultimately end up in mail.log.

I have not seen any logging for the FTS reindexing.  Specifically I 
would like to see how long the process took.  Getting that info from the 
solr logs is a manual process and I can't think of a way to properly 
automate it.


Is this something I can easily have dovecot log?  Or is there a way to 
have 'doveadm index' run in the foreground instead of the background so 
the time can be calculated by my script?


Thanks,
Shawn

___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

[offtopic] Re: DOvecot requires both IPv4 and IPV6 to start

[Shawn Heisey]

On 9/4/23 15:40, Arjen de Korte wrote:

Citeren Shawn Heisey :


An interesting read:

https://cr.yp.to/djbdns/ipv6mess.html


FUD.

This document was written something like two decades ago and pretty much 
all arguments against IPv6 are no longer relevant anymore. In most cases 
nowadays, the user experience for IPv6 will be far superior over CGNAT 
connections which unfortunately become the standard due to the 
increasing cost of IPv4 addresses. Running services over IPv6 may have 
been experimental and prone with not so fun problems to diagnose in 
2013, but today this is no longer the case.


So you're saying that I can change my entire home network to IPv6, 
eliminate IPv4 entirely, and I will have no problem connecting to sites 
like bbc.com or cnn.com, which have no  record?  My ISP is Comcast, 
which does support IPv6, though I have it entirely disabled.


If there is a way for an IPv6 computer to connect to websites with no 
IPv6 addresses, how could it possibly happen transparently?


Thanks,
Shawn

___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: DOvecot requires both IPv4 and IPV6 to start

[Shawn Heisey]

On 9/4/23 13:42, TWHG Technical via dovecot wrote:

I believe it really would be good practice to fallback to IPV4 only listeners 
if IPV6 listener fails. Rather than crash out.


On my previous reply, I had not yet read the other messages.  They were 
part of another thread in Thunderbird that I did not see at first.


After reading that and doing a little investigation, I can see that the 
comment in my main dovecot config is correct ... * means all Ipv4 
interfaces.  The default config uses '*, ::' for the listen option.


Dovecot does at least clearly log the problem when IPv6 is configured 
and no IPv6 is available:


Sep  4 14:26:41 bilbo dovecot: master: Error: socket() failed: Address 
family not supported by protocol
Sep  4 14:26:41 bilbo dovecot: master: Error: 
service(managesieve-login): listen(::, 4190) failed: Address family not 
supported by protocol
Sep  4 14:26:41 bilbo dovecot: master: Error: socket() failed: Address 
family not supported by protocol
Sep  4 14:26:41 bilbo dovecot: master: Error: service(pop3-login): 
listen(::, 110) failed: Address family not supported by protocol
Sep  4 14:26:41 bilbo dovecot: master: Error: socket() failed: Address 
family not supported by protocol
Sep  4 14:26:41 bilbo dovecot: master: Error: service(pop3-login): 
listen(::, 995) failed: Address family not supported by protocol
Sep  4 14:26:41 bilbo dovecot: master: Error: socket() failed: Address 
family not supported by protocol
Sep  4 14:26:41 bilbo dovecot: master: Error: service(imap-login): 
listen(::, 143) failed: Address family not supported by protocol
Sep  4 14:26:41 bilbo dovecot: master: Error: socket() failed: Address 
family not supported by protocol
Sep  4 14:26:41 bilbo dovecot: master: Error: service(imap-login): 
listen(::, 993) failed: Address family not supported by protocol

Sep  4 14:26:41 bilbo dovecot: master: Fatal: Failed to start listeners

The default config does explicitly ask for ipv6, so Dovecot did try (and 
fail) to start with the requested config, and most importantly, it did 
report the problem.


I am not sure that it should have failed to start, though.  I think that 
ipv4/ipv6 might need to be a special case, where a program logs an 
error, but continues to start up and run with support for the other 
address family.


I disable ipv6 on all my servers by adding "ipv6.disable=1" to the 
kernel commandline in /etc/default/grub.  I do it for two reasons.  1) 
Some software can behave strangely when ipv6 is enabled.  2) I am not 
using ipv6 and do not want my programs doing something unexpected 
because an IPv6 interface was contacted.  #1 is probably very rare in 
recent years, but it was common in the past.   I know that a lot of 
other admins routinely disable ipv6 on their machines for similar reasons.


An interesting read:

https://cr.yp.to/djbdns/ipv6mess.html

Thanks,
Shawn

___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: DOvecot requires both IPv4 and IPV6 to start

[Shawn Heisey]

On 9/4/23 12:36, Michael Peddemors wrote:

However, I 'get' this persons' opinion, from a developers perspective.

The system should either run, or provide a clear reason why it didn't 
startup (that reason could be .. You have selected * but IPv6 is not 
available). Doesn't really matter what the dependency is, whether a 
missing package, or a service not responding, there should be sane 
checks, and turning off IPv6 is probably a lot more popular than you 
think, given the increased attack vector and other observed issues.


But of course, the listen directive can easily be modified.  Just harder 
for newbies looking for an 'out of the box' solution.



The following is in my /etc/dovecot/dovecot.conf file.  Ubuntu 22, using 
the official dovecot APT repo:


# A comma separated list of IPs or hosts where to listen in for 
connections.

# "*" listens in all IPv4 interfaces, "::" listens in all IPv6 interfaces.
# If you want to specify non-default ports or anything more complex,
# edit conf.d/master.conf.
listen = *

My dovecot (version 2:2.3.20-3+ubuntu20.04) works just fine.  I have 
IPv6 completely disabled on the server.


The comment in my config file indicates that * should be all available 
IPv4 interfaces, which contradicts what you said above.  Maybe on my 
version the comment is right and for newer versions it isn't?


I have found a bunch of software that will refuse to run if IPv6 is 
disabled unless configured explicitly to use v4 and disable v6.  I find 
this trend disturbing.  In one case (atftpd) it took me a very long time 
to determine that the lack of an IPv6 interface was the cause for the 
program not starting.  Once I figured that out, I just added "--ipv4" to 
the /etc/default/atftpd file and it started working.


Even if * really means "all interfaces" and not "all IPv4 interfaces" as 
the comment indicates ... if no IPv6 interfaces are found, dovecot 
should proceed with the interfaces it finds, not fail when none of those 
interfaces have IPv6 addressing.


To the OP:  Was it obvious in logs that IPv6 was the problem?  In a lot 
of cases programs that refuse to start without IPv6 being available will 
do so silently.  It's the silently part that's the real problem with 
this trend.


Thanks,
Shawn

___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: Motion for OCSP Support in Dovecot

[Shawn Heisey]

On 7/10/23 21:47, Sean Gallagher wrote:
While I am always for security improvements, the utility of this 
unclear. I will ABSTAIN from this poll.


Presently, any system administrator who intends to issue must-staple 
certificates, faces the dilemma to either chose to


a) Refrain from issuing must-staple certificates at all, resulting in 
the loss of a valuable security feature.
b) Issue must-staple certificates without an OCSP response in Dovecot, 
thereby breaking the TLS RFC (and “hope for the best” on the client 
side…).


or c) use must-staple on a host-by-host basis


I am not using must-staple ... but I have haproxy stapling OCSP for any 
tcp/443 or udp/443 connection.  I do not have any other endpoint (like 
submission, smtp, imap, etc) doing stapling, but I would like that to be 
possible.


For those who don't know about it ... OCSP stapling makes the TLS 
handshake faster because the client does not need to make a separate 
outgoing OCSP request (which may be quite slow) to verify that the 
server certificate hasn't been revoked.  The stapled OCSP response is 
signed by the CA and has a very short lifetime, so forging a response is 
difficult.


Question) Do any popular email user agents validate an OCSP response if 
stapled?  (gut feeling is MAYBE/NO)


Question) Do any query an OCSP server if the OCSP response is not 
stapled?  (gut feeling is NO)


Browsers definitely do validate OCSP and make a query if the OCSP 
response isn't stapled.  I have no idea whether that's done in the 
browser or the TLS library.  If it's in the TLS library (openssl being 
the most prevalent example), then it is at least POSSIBLE for dovecot 
and other server software to do it.


Question) Has OCSP really got a future? (gut feeling - a few years at 
least)


OCSP is something I have been hearing about for quite a while.  I think 
it's probably going to stick around.


Thanks,
Shawn
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: Dovecot reposting inactivity as auth failed

[Shawn Heisey]

On 6/19/2023 10:52 PM, Shawn Heisey wrote:
Interesting.  In the 2.3.20 tag, I found the code that prints that 
message in src/login-common/client-common.c ... but that same code is 
not present in the main branch.  It looks to me like that code has been 
completely overhauled and maybe 2.4 won't have this problem when it gets 
released.


Hmm.  Maybe I am wrong.  I grepped my log for inactivity disconnects 
that do not say "auth failed".  All of the things it found were for my 
email address, and had a timeout of 1800 seconds, not 180 seconds like 
the ones for the other user that say "auth failed."


elyograg@bilbo:~$ sudo grep Inactivity /var/log/mail.log | grep -v "auth 
failed"
Jun 18 04:20:18 bilbo dovecot: 
imap(elyog...@elyograg.org)<412591>: Disconnected: 
Inactivity - no input for 1800 secs in=38963 out=457431 deleted=0 
expunged=0 trashed=0 hdr_count=1 hdr_bytes=230 body_count=1 body_bytes=10647
Jun 18 04:24:28 bilbo dovecot: 
imap(elyog...@elyograg.org)<340404>: Disconnected: 
Inactivity - no input for 1800 secs in=69198 out=823917 deleted=0 
expunged=0 trashed=0 hdr_count=3 hdr_bytes=2163 body_count=3 
body_bytes=47071
Jun 18 22:42:04 bilbo dovecot: 
imap(elyog...@elyograg.org)<701301>: Disconnected: 
Inactivity - no input for 1801 secs in=4726 out=13697 deleted=0 
expunged=0 trashed=0 hdr_count=0 hdr_bytes=0 body_count=0 body_bytes=0
Jun 19 03:18:37 bilbo dovecot: 
imap(elyog...@elyograg.org)<579102>: Disconnected: 
Inactivity - no input for 1800 secs in=8330 out=102365 deleted=0 
expunged=0 trashed=0 hdr_count=0 hdr_bytes=0 body_count=0 body_bytes=0
Jun 19 08:58:05 bilbo dovecot: 
imap(elyog...@elyograg.org)<773980>: Disconnected: 
Inactivity - no input for 1800 secs in=4710 out=13725 deleted=0 
expunged=0 trashed=0 hdr_count=0 hdr_bytes=0 body_count=0 body_bytes=0
Jun 19 18:05:50 bilbo dovecot: 
imap(elyog...@elyograg.org)<841537>: Disconnected: 
Inactivity - no input for 1800 secs in=174 out=4799 deleted=0 expunged=0 
trashed=0 hdr_count=0 hdr_bytes=0 body_count=0 body_bytes=0
Jun 19 19:29:21 bilbo dovecot: 
imap(elyog...@elyograg.org)<577941>: Disconnected: 
Inactivity - no input for 1801 secs in=28256 out=331141 deleted=0 
expunged=0 trashed=0 hdr_count=0 hdr_bytes=0 body_count=0 body_bytes=0


So maybe the other user with the auth failed messages was actually 
failing auth.  Today they actually got banned by fail2ban because they 
had a ton of the "auth failed" logs in quick succession.  I re-iterated 
the correct IMAP server settings and there hasn't been a repeat of that 
log message since that time.  Maybe they had actually typed in the 
password wrong.


Apologies for the noise!

Thanks,
Shawn
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: Dovecot reposting inactivity as auth failed

[Shawn Heisey]

On 6/19/2023 2:21 PM, pe...@netsecpt.pt wrote:

I am not using custom rules for fail2ban , in fact my fail2ban is using the 
default settings ,
i do not have that file zzz-custom in that directory however jail.conf is using 
the default jail time for every filter , witch is :


I created the zz-custom.conf file myself.  You won't find it in a 
standard fail2ban install.  It is my way of achieving the config I want 
without changing the main .conf files.  This makes upgrades a lot cleaner.


I believe dovecot does have a bug here.  My user that shows those failed 
auth messages has not been using the wrong password.  My guess is that 
anytime dovecot disconnects a user for inactivity, it claims that auth 
failed, even when that is not the case.


Interesting.  In the 2.3.20 tag, I found the code that prints that 
message in src/login-common/client-common.c ... but that same code is 
not present in the main branch.  It looks to me like that code has been 
completely overhauled and maybe 2.4 won't have this problem when it gets 
released.


It's been decades since I last did any C development, so I'm not really 
good at tracking what's going on in the code, but I see that the 
client_get_extra_disconnect_reason function in that source file uses 
that "auth failed" message as its fall-through option, and there doesn't 
seem to be any test earlier in the function to detect when the 
disconnect happens because of inactivity on a successfully authenticated 
connection.


Fixing the problem is beyond my current skill level.  It will require 
somebody who is intimately familiar with that code.


Thanks,
Shawn
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: Dovecot reposting inactivity as auth failed

[Shawn Heisey]

On 6/16/23 10:53, pe...@netsecpt.pt wrote:

Hi , i am having an issue with dovecot , in log files of imap inactivity lines have the 
word included "auth failed" , witch is not true , what happens next is that 
fail2ban is looking for that word too in log file of dovecot ,and when it finds it it 
bans my public ip address .
Is there any change to change this behavior in dovecot , what i mean is to insert 
"auth failed" when in fact it is an authentication failed , and not use it as 
general for every thing in log file .


Interesting.  It does look like maybe dovecot is reporting "auth failed" 
on inactivity even if the auth actually succeeded.  I am on Dovecot 
2.3.20 from the dovecot APT repo for Ubuntu.


I don't see this for my mail account, because my mail clients are never 
inactive for 3 minutes.  But I do see it for another account on my mail 
server.  I will need to ask that user if it is possible they are 
actually sending an incorrect password, but I think that is unlikely. 
The log entries are not frequent and it looks like there are only ever 
two such failures close enough together for fail2ban to notice, so the 
source address is not being banned because my fail2ban config requires 
five failures within 20 minutes before it takes action.


What is the findtime and maxretry in your fail2ban jail config for 
dovecot?  Maybe the time span is too large or the retry value too low.


I think you should have fail2ban ignore your public IP which would solve 
the problem for your IP address.  Below is the contents of 
/etc/fail2ban/jail.d/zz-custom.conf on my mail server.  I have not 
changed any of the conf files in /etc/fail2ban itself, I have only added 
a config file to jail.d.


The public /24 ranges in the ignoreip setting are in there for qualys 
labs SSL tests.  Without them, their test IP address gets banned because 
their tests generate a ton of failures in the haproxy log.  Also 
included is the AWS private IP address of the mail server, and the AWS 
public IP address.


elyograg@bilbo:/etc/fail2ban/jail.d$ cat zz-custom.conf
[INCLUDES]
before = common.conf

[DEFAULT]
maxretry = 5
bantime = 8h
findtime = 20m
ignoreself = true
ignoreip = 127.0.0.1/8 ::1 192.168.217.0/24 REDACTED1 172.31.8.104 
REDACTED2 REDACTED3 64.41.200.0/24 54.67.1.252 64.77.246.0/24

banaction = %(banaction_allports)s
protocol = all

[sshd]
enabled = true
port= 0:65535

[dovecot]
enabled = true
port= 0:65535

[sieve]
enabled = true
port= 0:65535

[postfix]
enabled = true
findtime = 30m
bantime = 8h
port= 0:65535

[postfix-rbl]
enabled = true
findtime = 30m
bantime = 8h
port= 0:65535

[postfix-sasl]
enabled = true
findtime = 30m
bantime = 8h
port= 0:65535

[haproxy-http-auth]
enabled = true
logpath  = /var/log/debug-haproxy
port= 0:65535

[haproxy-custom]
enabled = true
findtime = 30m
logpath  = /var/log/debug-haproxy
maxretry = 10
port= 0:65535
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: fts crash

[Shawn Heisey]

On 5/20/23 00:13, Przemysław Kwiatkowski via dovecot wrote:
I started full reindex of whole emails base (doveadm index -A '*') and 
solr crashed due to out of memory. I had plenty of such messages:


If you are seeing Java's "OutOfMemoryError" exception, you must figure 
out what resource was depleted.  It actually might not be memory.  If 
it's not, then increasing the Solr heap will not help.


I increased solr memory and restarted it. Do I have to restart dovedam 
index? The "request queued" message suggest that Dovecot is smart and I 
don't need to do anything. Am I right?


I would guess that you DO need to restart the reindex.  I have no idea 
how long dovecot will queue the updates it sends.  I would bet on it 
being something like 30 or 60 seconds.


Here is a script that I wrote to do a full dovecot reindex to Solr:

https://paste.elyograg.org/view/f183f91c

One caveat:   I had to comment lines 50 through 52 because SolrCloud 
refuses to reload an alias.  It has to be an actual collection.  If 
you're not in cloud mode you don't need to worry about it all.  If you 
are in cloud mode but using an actual collection name, then you can 
uncomment those lines and delete line 53.


This script has been tested with 9.x Solr versions.  I am not sure 
whether it will work properly on earlier Solr versions.


Thanks,
Shawn
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: 2.4 ballpark release date?

[Shawn Heisey]

On 1/17/23 14:04, Shawn Heisey wrote:
I'm wondering whether there is a ballpark release date for Dovecot 2.4. 
I know an exact release date is probably not available, but can it at 
least be narrowed to a specific quarter?  Is it likely in 2023?


I have seen on previous messages that an Ubuntu repository for 22.04 
(jammy) will appear with the 2.4 release.


My mail server in AWS is running Ubuntu 20.04 (focal), which I have 
avoided upgrading for various reasons, including websites that don't 
work with PHP 8.1.


Now the last remaining hurdle is the fact that there is no dovecot 
repository for jammy.  I would prefer the official Dovecot repo over 
dovecot packages included with Ubuntu, because it's likely to be a newer 
release, and built from vanilla project code.  At the moment jammy 
includes dovecot 2.3.16, which is even older than the 2.3.20 that I am 
running on focal from the official dovecot repo.


Again, I know that a precise release date won't be possible.  I'm just 
hoping for something like "soon" or "in about six months."


Is the release so far off that nobody wants to even make a guess?  It 
has been three months since I wrote the message I am replying to.


I don't see any indication in the github repo of a branch for 
stabilization work on the new version.


Thanks,
Shawn
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: Error: /proc/self/status is larger than expected

[Shawn Heisey]

On 3/20/23 21:29, Elisamuel Resto Donate wrote:

Hello,


I don't know what the error means or how to get rid of it.

But in the replies, I do see something that might indicate a 
misunderstanding of how /proc works.


Every pid will be getting a different file than another pid when it 
accesses things in /proc/self.  You can't apply anything you learn from 
accessing /proc/self/status in a shell prompt to dovecot, because 
dovecot gets a different file when it accesses /proc/self/status.


Open up two shells on the same system and then do this command in both:

ls -al /proc/self

You'll see that one shell has a different symlink target than the other.

Instead, find the pid for dovecot and look at /proc/N/status.  This 
command might work:


ls -al /proc/`pgrep dovecot`/status

Thanks,
Shawn

Re: ot: how to t/s TBird problems ?

[Shawn Heisey]

On 3/21/23 07:27, Christopher Wensink wrote:

Jim,

can you explain the format of the Date header, what each digit should 
represent?


Chris


A date value that's a big number like that is "seconds since epoch". 
The epoch is 1970-01-01 00:00:00 UTC.  This is a common way of telling 
time on POSIX systems like Linux.  Java uses something similar but not 
identical -- milliseconds since epoch.


The number 1665532450 is equivalent to Tuesday, October 11, 2022 
11:54:10 PM in the UTC timezone.


https://www.epochconverter.com/

The digits in the number have no specific meaning.  It's just a big 
decimal number.


Thanks,
Shawn

Re: iPhone Mail clients generate high Solr I/O

[Shawn Heisey]

On 3/20/23 09:52, Alessio Cecchi wrote:
I'm running a mail server with Dovecot 2.3.20 and Apache Solr as FTS 
backend.


It seems that iPhone/iPad Mail clients are generating some IMAP 
searches, especially on header Message-ID, that are increasing our Solr 
I/O load, especially during the night.


Are these kind of queries normal? What are they?


I don't know whether they are normal or not.  That requires knowledge 
about IMAP that I do not have.  It does seem odd that a mail client 
would be doing header searches unless the user is actually doing a 
search.  I would expect it to look messages up by the ID using other 
imap commands, not with a header search.  But my knowledge about IMAP is 
not complete enough to say for sure.



You can find some logs below.


Those Solr logs show what SHOULD be low-impact queries on a system 
that's appropriately sized.


I suspect what's happening is that the Solr server doesn't have enough 
free memory (memory not allocated by ANY program) to effectively cache 
the index.


If there is enough free memory for the OS disk cache and the cache is 
reasonably warmed up, queries like the ones you showed should produce 
very little disk I/O, because the data that they need will be sitting in 
memory and an actual disk read is unnecessary.


Once the search completes, the record retrievals might result in a cache 
miss, but the amount of data that gets retrieved for a dovecot search 
result is quite small.  The data is compressed, so a few CPU cycles are 
required to decompress it.


Memory-related performance issues with Solr come up frequently enough 
that I wrote a wiki article about it:


https://cwiki.apache.org/confluence/display/solr/solrperformanceproblems

Thanks,
Shawn

2.4 ballpark release date?

[Shawn Heisey]

I'm wondering whether there is a ballpark release date for Dovecot 2.4. 
I know an exact release date is probably not available, but can it at 
least be narrowed to a specific quarter?  Is it likely in 2023?


I have seen on previous messages that an Ubuntu repository for 22.04 
(jammy) will appear with the 2.4 release.


My mail server in AWS is running Ubuntu 20.04 (focal), which I have 
avoided upgrading for various reasons, including websites that don't 
work with PHP 8.1.


Now the last remaining hurdle is the fact that there is no dovecot 
repository for jammy.  I would prefer the official Dovecot repo over 
dovecot packages included with Ubuntu, because it's likely to be a newer 
release, and built from vanilla project code.  At the moment jammy 
includes dovecot 2.3.16, which is even older than the 2.3.20 that I am 
running on focal from the official dovecot repo.


Again, I know that a precise release date won't be possible.  I'm just 
hoping for something like "soon" or "in about six months."


Thanks,
Shawn

Re: mail filters

[Shawn Heisey]

On 12/19/22 11:08, mick.crane wrote:

On 2022-12-19 16:07, Shawn Heisey wrote:

I actually have a filter for exactly that purpose:

https://www.dropbox.com/s/5alqcawio2e1vki/roundcube_managesieve_lkml.png?dl=0


I should experiment further then to see what filters actually do.
"stop evaluating rules" in next to last filter seemed to ignore 
next/last filter.


The "stop evaluating rules" will normally be the last thing in a filter. 
 I use it in all of my filters so that once a filter matches, that will 
be the only filter that runs.


You can do some interesting things with multiple filters, but for my 
purposes, I only ever want one filter to run.


Thanks,
Shawn

Re: mail filters

[Shawn Heisey]

On 12/18/22 20:59, mick.crane wrote:

On 2022-12-13 00:42, John Stoffel wrote:


Maybe you're filtering the wrong way?

I use something like this and it's pretty good:

   require ["fileinto", "envelope"];
   require "imap4flags";
   require "regex";

   if header :contains "Sender" "linux-kernel-ow...@vger.kernel.org" {
 fileinto "lkml";
   }
   elsif header :contains "X-Mailing-List" "linux-ker...@vger.kernel.org"
   {
 fileinto "lkml";
   }
   else {
 # The rest goes into INBOX
 # default is "implicit keep", we do it explicitly here
 keep;
   }


I'm sorry, question should have gone to roundcube list as it is 
roundcube plugin "manage_sieve" am using to make rules.

Don't see "elsif" in rule options which is likely what I want.

I'll search for the sieve config file on Debian bullseye.


This is the sieve config in my dovecot:

root@bilbo:/etc/dovecot/conf.d# grep \/var\/vmail *
10-mail.conf:mail_location = maildir:/var/vmail/%d/%u
90-sieve.conf:  sieve = 
file:/var/vmail/sieve/%d/%u/sieve;active=/var/vmail/sieve/%d/%u/.dovecot.sieve

90-sieve.conf:  sieve_default = file:/var/vmail/global.sieve

That config results in this:

root@bilbo:/etc/dovecot/conf.d# ls -al 
/var/vmail/sieve/elyograg.org/elyog...@elyograg.org/

total 44
drwxr-x--- 3 vmail mail   4096 Sep 30 18:11 .
drwxr-x--- 5 vmail mail   4096 May 18  2022 ..
lrwxrwxrwx 1 vmail mail 21 Jul 23 14:55 .dovecot.sieve -> 
sieve/phpscript.sieve

-rw-r- 1 vmail mail   7856 Aug  5  2021 .dovecot.sieve.log
-rw-r- 1 vmail vmail 10305 Feb 18  2020 .dovecot.sieve.log.0
-rw-r- 1 vmail mail  10054 Sep 30 17:46 .dovecot.svbin
drwxr-x--- 3 vmail mail   4096 Sep 30 17:45 sieve

I actually have a filter for exactly that purpose:

https://www.dropbox.com/s/5alqcawio2e1vki/roundcube_managesieve_lkml.png?dl=0

I cannot remember how I added list-id to the choices for the header, 
though.  I tried looking it up with google and everything that came up 
wasn't done on my install.  I can't find the customization.


Thanks,
Shawn

Re: Backups

[Shawn Heisey]

On 12/11/22 01:42, John Tulp wrote:

sorry, no clue.  i did find this link quickly...

https://doc.dovecot.org/configuration_manual/quota/

looks like the following section near the bottom of page may apply ?

"Maximum Saved Mail Size"


I was looking into this on my server.  My config has the 100MB value, 
which came from dovecot defaults as I did not set it. Looks like none of 
the messages in my install is that big ... "find /var/vmail -size +99M" 
returns nothing.  I have a configured limit in postfix of 50MB, so I am 
probably never going to exceed the dovecot limit.


Looking over that link, I see the "quota_vsizes" option.  I was unable 
to find anything about this other than "Indicates that the quota plugin 
should use virtual sizes rather than physical sizes when calculating 
message sizes."  Can anyone tell me what that actually means in simple 
terms?  There is no info that I could find about how it actually 
calculates the virtual size.


Thanks,
Shawn

Re: sasl service for other app

[Shawn Heisey]

On 12/7/22 21:53, Henry R wrote:

can dovecot run as a general sasl service for other apps? such as webdav.


I am using dovecot to provide authentication for postfix submission.  
This is the config in postfix:


smtpd_sasl_type = dovecot
# Referring to /var/spool/postfix/private/auth
smtpd_sasl_path = private/auth

In /etc/dovecot/conf.d/10-master.conf I have this:

  unix_listener /var/spool/postfix/private/auth {
    mode = 0666
    user = postfix
    group = postfix
  }

If the application supports using a socket for sasl, then I would 
imagine that Dovecot should work.


Postfix is using the same postfixadmin database for email addresses that 
Dovecot is, but for authentication, it's all Dovecot.


I should probably look into Dovecot's submission support so I don't need 
to have postfix using that auth socket, just haven't found the time.


Thanks,
Shawn

Re: Backups and disaster recovery

[Shawn Heisey]

On 12/3/22 20:16, Sam Kuper wrote:

You might want to consider using ZFS.


Bystander reply:

I do like ZFS.  There are two reasons that I chose btrfs for my deployment:

1) License incompatibilities with ZFS and Linux.  These don't affect end 
users, but the situation is a little murky if you try to distribute 
something that uses both.


2) The ARC cache for ZFS is completely separate from the Linux disk 
cache, so that memory cannot be reclaimed under memory pressure as the 
Linux cache can.  Because of the license issues, ZFS cannot be a 
first-class Linux citizen, and I do not know if the module maintainers 
can have ZFS use Linux cache instead of ARC.


There is a little bit of worry that btrfs is not as mature as ZFS.  But 
I know there are Linux distributions using btrfs for the root filesystem 
by default, so that's probably a lot less of a worry than it was a few 
years ago.


Thanks,
Shawn

Re: Backups and disaster recovery

[Shawn Heisey]

On 12/3/22 16:02, GDS wrote:
After reading some of the past threads on backups, I was wondering if 
I could get a sanity check... I run a Maildir configuration for a 
small (10 mailboxes) mail server. Using "/doveadm backup"/, for each 
mailbox I do:


- Weekly full backups and then copy the files to a network-based 
filesystem.
- Daily incremental backups and then copy the files to a network-based 
filesystem.


My recovery assumption is that in case of hardware failure, I would 
re-set up the mail service and for each mailbox I will recover at the 
right directory the latest full mailbox backup and on top of it, each 
incremental backup to the latest day.  Does this sound like a sound 
strategy?


Also, I was thinking of setting up a second dovecot server on another 
server and replicating my primary on an hourly basis to decrease 
recovery time.  But I looked at mbsync and it seems to require mailbox 
login/password for each mailbox (which I don't have). Is there an 
alternative?


At 07:00 every day, my mailserver rsyncs its entire root filesystem to 
another server with a large btrfs filesystem.  All the mail is on that 
filesystem.  I am using Maildir, not sure how to figure out whether that 
is Maildir++ or not:


00 7 * * * rsync -axH --delete --delete-excluded --exclude=.git / 
server.domain.tld:/storage0/bilbofull/


The target server has a snapshot maintenance script I wrote that runs 
every night, as well as weekly and monthly:


45 23 * * * /usr/local/sbin/snapshot-maintenance storage0 daily
50 23 1 * * /usr/local/sbin/snapshot-maintenance storage0 monthly
55 23 * * 7 /usr/local/sbin/snapshot-maintenance storage0 weekly

The script keeps 7 daily snaps, 4 weekly snaps, and 6 monthly snaps.   I 
deleted all the snapshots recently because I was doing significant 
maintenance on the filesystem and wanted to actually recover the disk 
space.  This shows the snaps that currently exist:


elyograg@smeagol:/usr/local/sbin$ sudo btrfs subvolume list /storage0
ID 68160 gen 711353 top level 5 path .snapshot/2022.11.01.storage0.monthly
ID 68163 gen 718255 top level 5 path .snapshot/2022.11.06.storage0.weekly
ID 68170 gen 729261 top level 5 path .snapshot/2022.11.13.storage0.weekly
ID 68177 gen 746501 top level 5 path .snapshot/2022.11.20.storage0.weekly
ID 68181 gen 756805 top level 5 path .snapshot/2022.11.24.storage0.daily
ID 68182 gen 759352 top level 5 path .snapshot/2022.11.25.storage0.daily
ID 68183 gen 761844 top level 5 path .snapshot/2022.11.26.storage0.daily
ID 68184 gen 779474 top level 5 path qemu
ID 68185 gen 764384 top level 5 path .snapshot/2022.11.27.storage0.weekly
ID 68186 gen 766942 top level 5 path .snapshot/2022.11.28.storage0.daily
ID 68187 gen 769388 top level 5 path .snapshot/2022.11.29.storage0.daily
ID 68188 gen 771998 top level 5 path .snapshot/2022.11.30.storage0.daily
ID 68189 gen 774588 top level 5 path .snapshot/2022.12.01.storage0.monthly
ID 68190 gen 777168 top level 5 path .snapshot/2022.12.02.storage0.daily

The end result of this is that I have backups of all the email at many 
different points in time.  I back up a lot of other things into the 
btrfs filesystem as well.


elyograg@smeagol:~$ df -h /storage0
Filesystem  Size  Used Avail Use% Mounted on
/dev/sdb1    22T  3.5T   19T  17% /storage0

elyograg@smeagol:~$ sudo du -hs /storage0/bilbofull/var/vmail
6.5G    /storage0/bilbofull/var/vmail

I also have 10 mailboxes across several domains.  There are more than 10 
defined, but only 10 of them actually have email.  The users are in a 
postfixadmin database.


You should be able to use rsync to copy Maildirs from one server to 
another.  That is IMHO one of the advantages to Maildir ... each change 
to the mailbox occurs with a single message file, so it is unlikely the 
mailbox will be corrupted if it changes during the copy.


Thanks,
Shawn

Re: Dovecot 2.2 to 2.3 migration

[Shawn Heisey]

On 11/17/22 09:28, Oscar del Rio wrote:

I believe there is no Dovecot APT repo for Ubuntu 22 (yet)
https://repo.dovecot.org/


Very true.  I believe I read somewhere that there will not be a Dovecot 
2.3 repo for jammy, but there will be one for 2.4 when it is released.


Thanks,
Shawn

Re: Dovecot 2.2 to 2.3 migration

[Shawn Heisey]

On 11/15/22 03:29, Markus Wienhöfer wrote:
we are currently using Dovecot 2.2 (from the Ubuntu 18.04 repos) and 
would like to migrate to 2.3


I have read the migration documentation regarding the configuration 
changes. Apart from that, is it feasible/possible to change from 
Ubuntu to official Dovecot repositories and upgrade the dovecot 
package this way?


This is what I did.  After upgrading from Ubuntu 18 to Ubuntu 20, I 
added the dovecot APT repo and installed from there.  It all went 
smoothly, and the migration notes were sufficient.


At some point I will upgrade that system to Ubuntu 22, but I will be 
waiting until all the PHP webapps I am using are upgraded to support PHP 
8.1.  If you use a lot of PHP apps, the upgrade from PHP 7.4 in Ubuntu 
20 to PHP 8.1 in Ubuntu 22 is probably going to break at least one of 
them.  App developers are not adjusting to PHP 8.x very quickly.


For the gurus:  Is there an expected ballpark release date for 2.4 
and/or 3.0?


Thanks,
Shawn

Re: Dovecot does not start on MacOS Monterey

[Shawn Heisey]

On 10/30/22 07:24, Doro Rose wrote:

Thanks for the explanation. In the other thread it was mentioned that the only 
setting that works with this problem is setting default_vsz_limit to 0 which 
sounds like the lowest limit possible.


A setting of 0 tells dovecot to not set a limit, at which point whatever 
the OS has set will take effect.  Whenever I look at the setting for an 
OS that I use it says "unlimited."


Thanks,
Shawn

Re: how to configure imapsieve to be used per user

[Shawn Heisey]

On 10/17/22 04:46, Marc wrote:

I only see configurations that are active for all users, how to configure this 
in the user sieve rules. I only need this for specific users.


I have a working setup where every user can have their own sieve rules 
and manage them with managesieve, which I think is provided by 
pigeonhole.  I have a very extensive sieve script on my mailbox that has 
been built using a managesieve plugin for Roundcube webmail.  I tried to 
get an extension for Thunderbird going, but it doesn't work in the 
latest Thunderbird (102 at the time), and now that I am running a 
version 106 beta, that support is even less likely to happen.


This output should cover all my sieve-related configs:

https://paste.elyograg.org/view/c34f48ee

My users are in a mysql postfixadmin database.  The username is the 
fully qualified email address.  It's installed on a AWS instance running 
Ubuntu Server 20.04.  I do want to upgrade that to Ubuntu 22, but a lot 
of my PHP software will not run on PHP 8.1, which is what Ubuntu 22 
includes.  So I am waiting for that.


Thanks,
Shawn

Re: Dovecot does not start on MacOS Monterey

[Shawn Heisey]

On 10/26/22 04:27, Doro Rose wrote:

Thanks for the quick response!
No. If I set default_vsz_limit to 0 it works. My understanding was 
that this shouldn't be necessary in the first place, shouldn't it?


Most systems and software do not limit the virtual memory size. This is 
because virtual memory is just address space until something actually 
uses it, and the OS can ensure that an application that is working 
correctly with regard to virtual memory will NOT allocate too much real 
memory because of virtual memory.


Dovecot is actually the only software I have seen that has a native 
config for limiting the virtual memory size.  I know why it's done ... 
there are certain classes of bug that can result in virtual memory 
leakage.  Because virtual memory is NOT real memory, this class of bug 
might go unnoticed if there is no limit.  Putting a limit on the size 
makes sure that bugs of that nature ARE caught.


My dovecot install is not huge ... only 200K total email messages 
stored.  But I had to increase the default_vsz_limit to get dovecot 
working.  I think it defaults to 256M, I increased it to 1024M.  
Dovecot's log should tell you how much virtual memory Dovecot is 
requesting, if the request fails.


Thanks,
Shawn

Re: Matching Addresses in Sieve

[Shawn Heisey]

On 9/30/22 15:14, Doug Hardie wrote:

I have an email with the following header line:

From: 'Thank you!Kohls' 

I am trying to match that with:
if address :contains "from" "Thank you!Kohls"
{
 addflag "\\Seen";
 fileinto "Junk";
 stop;
}

However, the matching portion of the from address is only the section between < and 
>.  Since there are changing sections that are different for each email, I can't use 
that.  I wanted to match the stuff before <.  I have tried numerous formats for the 
if statement but none of them have worked.  What is the proper way to make that match 
work?  Thanks,


I did what looked like the right thing in a sieve plugin for roundcube:

https://www.dropbox.com/s/abhpc7rf9rokmfl/junk_rule_for_sieve.png?dl=0


And this is what that created in the script.  Only one word of 
difference from yours -- it looks at the entire From header and not an 
address.


# rule:[testing]
if header :contains "from" "Testing"
{
    addflag "\\Seen";
    fileinto "Junk";
    stop;
}

Hope this helps.

Thanks,
Shawn

Re: dovecot-fts-solr Solr9 support

[Shawn Heisey]

On 9/27/22 19:32, Nathanael Anderson wrote:
I was trying a new install of dovecot w/ solr9.   I've manually fixed 
the file linking to the proper directories, however one plugin is no 
longer shipped.   Since the solr files aren't updated yet to 9, can 
anyone tell me if I need the discontinued velocity plugin that was 
default in the dovecot solr 7.7 config file.   It appears it is now a 
third party plugin that hasn't been updated for 3 years.


The velocity stuff that Solr ships with is a templating system that 
allows Solr to host a little website showcasing its capabilities.  It is 
strongly recommended to never use this in production, as it requires 
that end users have direct network access to the Solr install, which is 
never a good idea.


Dovecot accesses the API directly and does not need velocity.

I am running a dev version of Solr 9.1.0 with the config and schema 
stripped down to just what is needed for Dovecot.  I have added the jars 
necessary for the ICU analysis components and I am using two of those 
analysis components in my schema.


I installed Solr on Ubuntu Server using the service installer script 
included in the download.  This extracts the tarball in /opt, and then 
sets up /opt/solr as a symlink to the version-specific directory in 
/opt.  It creates a directory structure under /var/solr and creates 
/etc/default/solr.in.sh.  If you use a service name other than solr, 
that will be named /etc/default/${servicename}.in.sh and I believe the 
data will go to /var/${servicename}.


For ICU, I created /var/solr/data/lib, then copied icu4j-70.1.jar and 
lucene-analysis-icu-9.3.0.jar from /opt/solr/modules/analysis-extras/lib 
to that new lib directory. Solr 9.0.0 would have lucene jars from Lucene 
9.0.0, but the 9.x branch is currently using Lucene 9.3.0.  Do not use 
 config elements in solrconfig.xml to load the jars.  My 
solrconfig.xml and managed-schema.xml files can be found here:


https://paste.elyograg.org/view/97597ed3
https://paste.elyograg.org/view/dca55086

My index is quite small by Solr standards, which is why I have such a 
low maxTime on autoSoftCommit.  Larger indexes may do better with a 
larger interval there.


I use LATEST for luceneMatchVersion, which generates a warning when Solr 
starts.  I am also using 2.0 for the schema version so that it will 
automatically pick up new defaults after the 1.6 version when those 
versions are created in later versions of Solr.


This is the current contents of /etc/default/solr.in.sh with commented 
lines removed:


---
SOLR_PID_DIR="/var/solr"
SOLR_HOME="/var/solr/data"
LOG4J_PROPS="/var/solr/log4j2.xml"
SOLR_LOGS_DIR="/var/solr/logs"
SOLR_PORT="8983"
SOLR_HEAP="1g"
GC_TUNE=" \
  -XX:+UseG1GC \
  -XX:+ParallelRefProcEnabled \
  -XX:MaxGCPauseMillis=100 \
  -XX:+UseLargePages \
  -XX:+AlwaysPreTouch \
  -XX:+ExplicitGCInvokesConcurrent \
  -XX:ParallelGCThreads=2 \
  -XX:+UseStringDeduplication \
  -XX:+UseNUMA \
"
SOLR_JAVA_STACK_SIZE="-Xss1m"
SOLR_ULIMIT_CHECKS=false
SOLR_GZIP_ENABLED=true
SOLR_JETTY_HOST=0.0.0.0
---

Once you have all that in place, start and stop solr using service or 
systemctl.  Don't run the solr script directly except to create the 
index ... and for that you must run it as the solr user. Running it as 
root is prohibited by default, and forcing it will cause problems.


My Solr install is running in cloud mode, but I have removed the things 
that configure that to make this info easier to use.


One final note:  Solr 9 cannot use indexes touched by Solr 7 or 
earlier.  You will need to completely reindex.


Thanks,
Shawn

Re: Replication

[Shawn Heisey]

On 9/5/22 08:18, Silvio Siefke wrote:

On Mon, 5 Sep 2022 14:59:01 +0200
Narcis Garcia  wrote:


I SEE THIS IN LOG COPY:

Sep  5 18:02:18 asia dovecot: replicator: Panic: data stack: Out of
memory when allocating 268435496 bytes

Yes but Memory is enough free. I had follow the link


It is very likely virtual memory (just address space, not actual memory) 
that cannot be allocated.  Dovecot restricts the amount of virtual 
memory it can allocate ... something that most programs do not do.  This 
makes it possible to prevent a certain class of bug from using all the 
memory.  I think it defaults to 256M which would be 268435456 bytes.  
Just a tiny bit less than the amount in the error message.


I think the setting in the link would only affect the replicator, not 
all of dovecot.  Your error does indicate it is the replicator that had 
the problem, but I think setting the limit more globally would be 
desirable.  If I have the wrong idea here, can someone please let me know?


In my config, I set default_vsz_limit and one instance of vsz_limit to 
1024M because I was running into a very similar error message.  I could 
probably remove the explicit vsz_limit setting because I set the 
default, but I haven't tried it, and this config works:


-
elyograg@bilbo:/etc/dovecot$ cat conf.d/10-master.conf
#default_process_limit = 100
#default_client_limit = 1000

# Default VSZ (virtual memory size) limit for service processes. This is 
mainly

# intended to catch and kill processes that leak memory before they eat up
# everything.
default_vsz_limit = 1024M

# Login user is internally used by login processes. This is the most 
untrusted

# user in Dovecot system. It shouldn't have access to anything at all.
#default_login_user = dovenull

# Internal user is used by unprivileged processes. It should be separate 
from

# login user, so that login processes can't disturb other processes.
#default_internal_user = dovecot

service imap-login {
  inet_listener imap {
    #port = 143
  }
  inet_listener imaps {
    #port = 993
    #ssl = yes
  }

  # Number of connections to handle before starting a new process. 
Typically

  # the only useful values are 0 (unlimited) or 1. 1 is more secure, but 0
  # is faster. 
  #service_count = 1

  # Number of processes to always keep waiting for more connections.
  #process_min_avail = 0

  # If you set service_count=0, you probably need to grow this.
  #vsz_limit = $default_vsz_limit
}

#service pop3-login {
#  inet_listener pop3 {
#    #port = 110
#  }
#  inet_listener pop3s {
#    #port = 995
#    #ssl = yes
#  }
#}

service lmtp {
  unix_listener lmtp {
    #mode = 0666
  }

  # Create inet listener only if you can't use the above UNIX socket
  #inet_listener lmtp {
    # Avoid making LMTP visible for the entire internet
    #address =
    #port =
  #}
}

service imap {
  # Most of the memory goes to mmap()ing files. You may need to 
increase this

  # limit if you have huge mailboxes.
  vsz_limit = 1024M

  # Max. number of IMAP processes (connections)
  #process_limit = 1024
}

service pop3 {
  # Max. number of POP3 processes (connections)
  #process_limit = 1024
}

service auth {
  # auth_socket_path points to this userdb socket by default. It's 
typically
  # used by dovecot-lda, doveadm, possibly imap process, etc. Users 
that have
  # full permissions to this socket are able to get a list of all 
usernames and

  # get the results of everyone's userdb lookups.
  #
  # The default 0666 mode allows anyone to connect to the socket, but the
  # userdb lookups will succeed only if the userdb returns an "uid" 
field that
  # matches the caller process's UID. Also if caller's uid or gid 
matches the
  # socket's uid or gid the lookup succeeds. Anything else causes a 
failure.

  #
  # To give the caller full permissions to lookup all users, set the 
mode to

  # something else than 0666 and Dovecot lets the kernel enforce the
  # permissions (e.g. 0777 allows everyone full permissions).
  unix_listener auth-userdb {
    mode = 0666
    user = vmail
    group = mail
  }

  # Postfix smtp-auth
  unix_listener /var/spool/postfix/private/auth {
    mode = 0666
    user = postfix
    group = postfix
  }

  # Auth process is run as this user.
  #user = $default_internal_user
}

service auth-worker {
  # Auth worker process is run as root by default, so that it can access
  # /etc/shadow. If this isn't necessary, the user should be changed to
  # $default_internal_user.
  #user = root
}

service dict {
  # If dict proxy is used, mail processes should have access to its socket.
  # For example: mode=0660, group=vmail and global mail_access_groups=vmail
  unix_listener dict {
    mode = 0660
    user = vmail
    group = postfix
  }
}
elyograg@bilbo:/etc/dovecot$
-

Thanks,
Shawn

Re: Maybe a bit offtopic, Sieve beaviour

[Shawn Heisey]

On 8/23/22 03:44, João Silva wrote:

If a sieve filter contains a rule

if header :regex ["From"] ["@xxx.com","@yyy.pt"] {
    fileinto :create "ac";
}

The mail should be copied to the folder ac and then proceed to other 
rules that a user might have or I understood the fileinto wrong?


I just took a look at my fairly extensive sieve script that has been 
built by the managesieve plugin for the roundcube webmail server.  I 
don't often look at the script itself because that plugin lets me edit 
the script in a GUI.


The fileinto command is used in almost every one of my rules.  It is not 
copying the message, it is moving the message to the destination 
folder.  Here is one of my rules:


# rule:[blog]
if header :contains ["To","cc"] "b...@elyograg.org"
{
    fileinto "admin.blog";
    stop;
}

There is an extension to sieve that adds a :copy option to certain 
commands like fileinto for situations where that is actually what you 
want to do.  Pigeonhole does support it:


https://www.rfc-editor.org/rfc/rfc3894.html

Thanks,
Shawn

Re: solr-jetty package in Debian 11 bullseye

[Shawn Heisey]

On 5/25/22 13:30, John Gateley wrote:

I am in the process of upgrading the OS on my mailserver, currently 
Debian 9, to Debian 11.


The solr-jetty package does not appear to exist in Debian 11.


The Solr package that was included in Debian and derivatives was REALLY 
ancient -- Solr 3.6.  Looks like they chose to remove it rather than 
update it.


Current Solr versions have a service installer script in the binary 
download that works on modern Linux distros.  If you need some 
assistance with it, I can help, but it would be off-list as it's not 
really Dovecot related.


Thanks,
Shawn

Re: dovecot-fts-solr Solr9 support

[Shawn Heisey]

On 5/17/2022 11:32 AM, PGNet Dev wrote:

CaffeineCache  _should_ be a solution for both series.
I'm just starting to do a bit of testing, but so far, indexing/search 
with Caffeine seems quicker, at least.
I do not yet know in which case(s) CaffeineCache is _not_ an 
appropriate solution.


CaffeineCache was added in Solr 8.3.0.  So it is not available in all 
8.x releases.


https://issues.apache.org/jira/browse/SOLR-8241

I know more than a little bit about Solr's caches.  I am the author of 
LFUCache, added way back in Solr 3.6.0.  That is an extremely naive 
"intro to programming" implementation, but it works.


https://issues.apache.org/jira/browse/SOLR-2906

Caffeine is a highly optimized and battle-tested cache, with very good 
performance.  Shortly after it was added to Solr, all the other cache 
implementations were deprecated, and then removed in 9.0.0.


I should probably see if I can write some end to end instructions for 
installing Solr and configuring dovecot to use it.


Having something well-vetted and doc'd will be useful.


If I ever find any time I will definitely work on that.

Thanks,
Shawn

Re: FTS Solr -- no commit on message delete

[Shawn Heisey]

On 5/18/2022 8:24 AM, Shawn Heisey wrote:
Today I discovered a related problem that I think IS in Dovecot.  I 
did a shift-delete on some messages, and then exited Thunderbird.  
Dovecot did delete the messages from Solr, but it did not send a 
commit to make those changes affect subsequent searches.


Followup:  It looks like when message deleting is done normally, where 
messages are moved to trash, then the trash is emptied, Dovecot DOES 
send a commit after each delete request.


Thanks,
Shawn

FTS Solr -- no commit on message delete

[Shawn Heisey]

I've been chasing down oddities in behavior related to FTS Solr for a 
while now.


First the part that isn't a problem with dovecot:  In cases where 
message deletes bypass the Trash folder (such as shift-delete and normal 
workflow with the Drafts and Sent folders), Thunderbird does not inform 
Dovecot about the change in a way that causes Dovecot to delete the 
entries from Solr. When you completely exit Thunderbird, it then sends 
something to dovecot that results in the messages being deleted from Solr.


Today I discovered a related problem that I think IS in Dovecot.  I did 
a shift-delete on some messages, and then exited Thunderbird.  Dovecot 
did delete the messages from Solr, but it did not send a commit to make 
those changes affect subsequent searches.


Is this something worth opening a bug?  Where would I do that?

Thanks,
Shawn

Re: dovecot-fts-solr Solr9 support

[Shawn Heisey]

On 5/17/22 11:32, PGNet Dev wrote:
Something that is missing from the current instructions is how to 
initiate a FULL reindex of all data in the dovecot install.


Beyond looping over each $acct

 doveadm fts rescan -u ${acct}
 doveadm index  -u ${acct} -q '*'


I hadn't seen the fts rescan before.  That was what I was missing to 
make things really clean.  Before I was deleting all the dovecot.*index* 
files, and now I don't need to do that.  This is my new script, with the 
old way commented out:


---
#!/bin/sh
DOVECOT_SOLR_URL_BASE="http://localhost:8983/solr/dovecot;
#MAIL_ROOT=/var/vmail

# DO NOT MODIFY BELOW
# WITHOUT GOOD REASON
#
DEL_ALL_QUERY_XML="*:*"
service dovecot stop
curl \
  "${DOVECOT_SOLR_URL_BASE}/update?commit=true=true" \
  -H "Content-Type: text/xml" \
  --data-binary "${DEL_ALL_QUERY_XML}"
#find ${MAIL_ROOT} -name "dovecot.*index*" -print0 | xargs -0 rm -f
service dovecot start
doveadm force-resync -A '*'
doveadm fts rescan -A
doveadm index -A -q '*'
---

The script stops Dovecot, wipes out the Solr index, then restarts 
Dovecot and does the doveadm commands necessary for a full reindex of 
all mailboxes, without specifying them one by one.  The force-resync 
probably isn't required, but I figure it can't hurt. I've tested this 
new version a few times.  Good thing a full reindex only takes about ten 
minutes for my system.


Thanks,
Shawn

Re: dovecot-fts-solr Solr9 support

[Shawn Heisey]

On 5/17/22 09:02, PGNet Dev wrote:


Modifying the config to use CaffeineCache

cp solr-config-7.7.0.xml solr-config-9.0.0.xml
perl -pi -e '\
 s|solr.FastLRUCache|solr.CaffeineCache|g; \
 s|solr.LRUCache|solr.CaffeineCache|g; \
 s|solr.search.LRUCache|solr.search.CaffeineCache|g;' \
solr-config-9.0.0.xml

appears to do the trick.
Atm, indexing & search -- both manually triggered, and on delivery 
receipt to Dovecot store -- is working with no errors (yet).


Yesterday I upgraded my Solr install for dovecot from 8.11.2-SNAPSHOT to 
10.0.0-SNAPSHOT.  It is still working and the config did not need 
further changes from what I already had.


I have been customizing my solrconfig.xml and managed-schema files ... I 
had made the change to CaffeineCache quite a while back.  I have also 
changed the analyzer chain in the schema to use a contrib component, so 
my config won't be directly usable -- it requires loading some extra 
jars.  When I upgraded, I did need to change the extra jars that were 
loaded.


I can come up with a new config that should work in 8.x, 9.x, and the 
current dev branch by almost anyone.


I should probably see if I can write some end to end instructions for 
installing Solr and configuring dovecot to use it.  Something that is 
missing from the current instructions is how to initiate a FULL reindex 
of all data in the dovecot install.


Thanks,
Shawn

Re: Recovering deleted mailbox

[Shawn Heisey]

On 4/27/2022 11:48 PM, Aki Tuomi wrote:

# drop fts data
doveadm fts rescan -u user
# rebuild index
doveadm index -u user "*"


I do full reindexes a lot more often than a typical user would. When I 
do a reindex, I want it to happen for all users, not one at a time.  Do 
you have a sequence of commands to accomplish that?


I'm part of the Solr project.  I do reindexes for performance testing 
when I think of something to try, or I want to upgrade Solr.  And also 
when I think of a change that I want to make to the index config for 
dovecot.  A full reindex for my install takes less than 10 minutes as I 
only have about 160K messages total. Not something I would recommend 
doing frequently when there are millions of messages.


Thanks,
Shawn

Re: Recovering deleted mailbox

[Shawn Heisey]

On 4/27/2022 11:27 PM, Aki Tuomi wrote:

There is no reason to delete the dovecot files after recovery. You can run 
`doveadm force-resync` to ensure everything is synced. Removing the files just 
cause more problems than benefit usually.


Thanks for that information!  Very helpful for future fiddling.

Does that resync command also maybe force a full FTS reindex?  I'm using 
fts_solr.  The way that I currently manage a full reindex is with the 
following shell script:


---
#!/bin/sh
DOVECOT_SOLR_URL_BASE="http://localhost:8983/solr/dovecot;
MAIL_ROOT=/var/vmail

# DO NOT MODIFY BELOW
# WITHOUT GOOD REASON
#
DEL_ALL_QUERY_XML="*:*"
service dovecot stop
curl \
  "${DOVECOT_SOLR_URL_BASE}/update?commit=true=true" \
  -H "Content-Type: text/xml" \
  --data-binary "${DEL_ALL_QUERY_XML}"
find ${MAIL_ROOT} -name "dovecot.*index*" -print0 | xargs -0 rm -f
service dovecot start
doveadm index -A -q '*'
---

-s

Re: Recovering deleted mailbox

[Shawn Heisey]

On 4/27/22 16:18, Sean McBride wrote:

I have a user (coworker) that accidentally deleted a mailbox and all its 
sub-mailboxes.

I use Maildir format storage.  I have backups.

Is it enough to put the mailbox folder back where it was?  I'm talking about 
the folder that contains 'cur', 'new', 'tmp', 'dovecot-uidlist', etc.   Or 
would this desynchronize or otherwise confuse dovecot?  Or is it preferable to 
use some doveadm command?  Or...?



Disclaimer:  I am not affiliated with the project, and I am definitely 
not an expert.  I've been running dovecot for my personal mail server 
for a long time, thankfully with very few incidents.  I have done some 
manual surgery on my maildir mailbox and seen how it reacts.  Dovecot is 
very resilient.


What you describe should be sufficient.  It's how I would proceed with a 
restore.  In most cases I would copy the backup on top of any existing 
structure, rather than doing a wholesale replace, because any new mail 
received should have different filenames than what is in the backup.


If it were me, I would probably delete all the files that have a 
filename starting with "dovecot" in that user's mailbox, and restart 
dovecot, letting dovecot rebuild those files when the user connects.  I 
don't really have any experience with how things operate over POP3, I've 
always used IMAP with dovecot.


I'm interested to know whether the real experts here have different 
advice than this, in case I ever find myself in that situation.  There 
might be some doveadm commands that accomplish the dovecot* file 
rebuilding in a cleaner way.


Thanks,
Shawn

Re: how to setup IMAPs with letsencrypt

[Shawn Heisey]

On 4/23/2022 6:45 PM, Richard Hector wrote:
_A_ web server has to be there. It doesn't have to serve anything else 
useful. My mail server has a web server that only serves the LE 
challenge. Well, actually it's a proxy server that serves several 
other domains too, but there's nothing else served on that domain (at 
the moment).


I didn't want to mess with creating a web infrastructure for the usual 
web-based validation that is common with LE.  Getting that working for 
my services would be very messy.  So I use DNS validation with 
LetsEncrypt, and I have wildcards in my cert.  You can see the cert at 
this location:


https://http3test.elyograg.org/

Reload the page to see if your browser can do http/3 -- the initial 
connection will usually be http/2.


Certbot has plugins for many common DNS providers that let it 
automatically add the validation records to your DNS.  I use a DNS 
provider which is not covered by the official plugins, but I found a 
third party hook script on github, so I have built scripts that 
accomplish completely automated certificate renewals with DNS 
validation.  I run the renew script with cron every other day, and have 
it waiting until 5 days before expiration before it actually does the 
renewal.  So I get a new cert about every 85 days, and it even installs 
the cert and restarts services on everything that needs it.


Thanks,
Shawn

Re: auth between postfix and dovecot?

[Shawn Heisey]

On 4/23/2022 9:07 AM, Alexander Dalloz wrote:

With this Postfix configuration you do not make use of LMTP delivery.


Interesting.  I thought it was using LMTP but it looks like master.cf 
has it running /usr/lib/dovecot/dovecot-lda to deliver. Would LMTP be a 
better option? It has always worked, so I didn't look at it very closely.


I thought I had something in the postfix/dovecot combination using LMTP, 
but it looks like I was wrong about that.  The communication between 
postfix and mailman3 running in docker containers (which is a very 
recent addition) uses LMTP.


Thanks,
Shawn

Re: auth between postfix and dovecot?

[Shawn Heisey]

On 4/22/2022 10:35 PM, ミユナ (alice) wrote:

my question is:

when postfix talks to dovecot, does it require user's 
username/password for authentication? or this communication just goes 
without authentication?


I asked this, b/c my webmail send mail from localhost has been going 
without authentication to postifx. so i am not sure if postfix talks 
to dovecot without requiring auth too.


My setup is virtual users in a postfixadmin database.  Dovecot does all 
authentication, even with posfix.  I believe the config snippets I have 
included below are the relevant things that make it possible for postfix 
to talk to dovecot for mail delivery and authentication.


Mail sent from localhost on port 25 does not require authentication on 
my system, because 127.0.0.0/8 is in postfix's mynetworks config and 
port 25's access restrictions include permit_mynetworks.  Anything sent 
via submission (port 587) does require auth, even from trusted 
networks.  If you can configure your webmail to use submission instead 
of smtp, maybe that can be authenticated.  You'll need to consult 
support resources for your webmail to see if that is possible.   I can 
say for sure that roundcube can do it ... I have roundcube configured to 
talk to port 587, which as mentioned, ALWAYS requires authentication.


When postfix sends mail to dovecot for delivery, I'm pretty sure that 
happens without authentication.  It's LMTP via unix socket, not 
something an outside client can access directly.


# In 10-master.conf
service lmtp {
  unix_listener lmtp {
    #mode = 0666
  }
}

service auth {
  unix_listener auth-userdb {
    mode = 0666
    user = vmail
    group = mail
  }

  unix_listener /var/spool/postfix/private/auth {
    mode = 0666
    user = postfix
    group = postfix
  }
}


# In 10-auth.conf
disable_plaintext_auth = yes
auth_mechanisms = plain
!include auth-sql.conf.ext


# In postfix master.cf
dovecot  unix   -    n  n   -   -   pipe
  flags=DRhu user=vmail:mail argv=/usr/lib/dovecot/dovecot-lda -d 
$(recipient)



# In postfix main.cf
# Use Dovecot to authenticate.
smtpd_sasl_type = dovecot
# Referring to /var/spool/postfix/private/auth
smtpd_sasl_path = private/auth
smtpd_sasl_auth_enable = yes
#broken_sasl_auth_clients = yes
smtpd_sasl_security_options = noanonymous
smtpd_sasl_local_domain =
smtpd_sasl_authenticated_header = yes

# Tell postfix to hand off mail to the definition for dovecot in master.cf
virtual_transport = dovecot
dovecot_destination_recipient_limit = 1

Re: how to setup IMAPs with letsencrypt

[Shawn Heisey]

On 4/22/22 02:20, Jean-Daniel Dupas wrote:
While it's true for SMTP, my experience is that IMAP clients prefer 
imaps in 993 instead of STARTTLS.


I have a server with only port 993 opened, and almost never had any 
issue with client configuration.


I have noticed the opposite.  Every time I have configured a new mail 
client (which is most often but not always Thunderbird), it defaults to 
143 with STARTTLS.  Port 993 is available too, but my mail clients have 
never used it unless I explicitly configure it.


My dovecot is configured with "disable_plaintext_auth = yes" so only 
source IPs that are local to the machine (so the traffic never goes out 
on any network) are allowed to login without TLS. My webmail uses 
localhost so it is configured to use port 143 without encryption.


I know a lot of people are going to clamor that such traffic should be 
encrypted because it could be sniffed ... but if somebody has enough 
access such that they could sniff my backend services, the security 
battle is already lost, and they would be able to get any in-flight 
passwords even if the connection is encrypted.


Thanks,
Shawn

Re: Bad Signature - Both Roundcube and Squirrelmail webmail cannot search for anything + cannot open many emails because there are more than 200, 000 emails in my Inbox

[Shawn Heisey]

On 4/20/22 07:51, Turritopsis Dohrnii Teo En Ming wrote:

I believe Gmail is using IMAP. The instructions for configuring Gmail
email accounts in Outlook specifically mention IMAP server hostname:

imap.gmail.com

TCP Port 993, SSL


Indeed you CAN use IMAP with gmail.  For work, I mostly use my email 
(which is gmail-based) via Thunderbird, with IMAP.  I am familiar with it.


But if I use the Gmail app for Android or the gmail website, I feel 
absolutely certain that those applications are NOT using IMAP.  Chances 
are that they use protocols developed by Google.


Regarding your next email on this thread, the initial sync of a large 
number of messages via IMAP does take a really long time. But once the 
initial sync is complete, syncing the differences is very fast.  The big 
problem with accessing a large folder via a webmail app like Roundcube 
is that it never sets up a local copy of the mailbox.  I actually don't 
know where the copy of all those message headers resides ... whether it 
is in the PHP interpreter or in the browser.  But it's not persisted to 
disk as it would be with a more traditional IMAP client such as Thunderbird.


Thanks,
Shawn

Re: Both Roundcube and Squirrelmail webmail cannot search for anything + cannot open many emails because there are more than 200, 000 emails in my Inbox

[Shawn Heisey]

On 4/20/22 08:03, Turritopsis Dohrnii Teo En Ming wrote:

Dear Shawn Heisey,

Your instructions for integrating Apache SOLR with Dovecot is too much
for me to understand. Do you have links/URLs to excellent guides for
doing this?


Here's the dovecot manual page on it:

https://doc.dovecot.org/configuration_manual/fts/solr/

This doesn't explain how to tell dovecot to index new incoming messages, 
or how to initiate a full reindex, but I can guide you on those.


Thanks,
Shawn

Re: Bad Signature - Both Roundcube and Squirrelmail webmail cannot search for anything + cannot open many emails because there are more than 200, 000 emails in my Inbox

[Shawn Heisey]

On 4/19/2022 5:12 PM, Turritopsis Dohrnii Teo En Ming wrote:

It is not true that no email system was ever designed for 200,000 or
more emails.

If I did not remember wrongly, more than 10 years ago, I used to have
a Gmail email account, with 4-5 MILLION email messages in the Inbox.
There was no problem searching even though I had 4-5 MILLION email
messages in the Inbox.


If you were using either a Google app or the gmail.com website to access 
that inbox, then it doesn't compare.  I really doubt that either of 
those uses the IMAP protocol.  They would be using something proprietary 
that is highly optimized for the way that Google stores data and 
leverages the enormous amounts of computing power that they maintain.


I would bet that if you accessed a gmail folder with 5 million messages 
in it using IMAP, you would have similar problems with it to those that 
have been described here in this thread.  IMAP is a beautiful protocol, 
but I don't think it was designed for handling that many messages.


Thanks,
Shawn

Re: Both Roundcube and Squirrelmail webmail cannot search for anything + cannot open many emails because there are more than 200, 000 emails in my Inbox

[Shawn Heisey]

On 4/17/2022 7:44 AM, Turritopsis Dohrnii Teo En Ming wrote:

My Dovecot, Postfix, Roundcube and Squirrelmail were automatically
installed, configured and deployed by Virtualmin web hosting control
panel install scripts.

How do I integrate Apache SOLR searching capability with Dovecot? Are
there any excellent guides for this?


I have this in /etc/dovecot/conf.d/10-mail.conf:

mail_plugins = $mail_plugins quota fts fts_solr

And in conf.d/90-plugin.conf, I have this:

plugin {
  fts = solr
  fts_solr = url=http://localhost:8983/solr/dovecot/
  fts_autoindex = yes
  fts_enforced = yes
}

I'm running a version of Solr that I built myself from the branch_8_11 
source code -- 8.11.2-SNAPSHOT.  I installed that with the service 
installer script that it includes.  I have been customizing the index 
config, it started with the config and schema provided by dovecot, which 
did work.  I'm going to have some questions for those who write the 
fts_solr plugin, and I still need to glance at the source code.


I am a committer on the Apache Solr project, so I do know a little bit 
about Solr. :)  I am not as competent in the C code that makes dovecot 
as I am in the Java code that makes Solr.  C was one of my first 
languages, but I have not spent any real time with it in decades.


My install of dovecot has a grand total of 180079 messages in it right 
now, 154K of which are in my mailbox.  (I deleted the test folder I 
created to test your problem out).  If you can upgrade roundcube and 
dovecot, plus incorporate Solr, things might work better for you.


I would suggest that you move most of the messages out of your inbox 
into some kind of archival folder.  Keep between 3 months and one year 
of messages in the actual inbox so working with that folder is fast, and 
you can visit the archive if you need to look at something older.


I have never liked dealing with do-everything packages like webmin or 
virtualmin.  I want to understand each of the pieces that make up the 
whole, and be able to upgrade and reconfigure each piece independently 
from the others.  I find the configuration restraints of such a system 
very limiting.  Very often if you customize something yourself outside 
the GUI, you lose the ability to administer it in the GUI, because it 
doesn't know how to deal with the change.  Or if you do change something 
in the GUI afterwards, your customization might get deleted.


Thanks,
Shawn

Re: Both Roundcube and Squirrelmail webmail cannot search for anything + cannot open many emails because there are more than 200, 000 emails in my Inbox

[Shawn Heisey]

On 4/16/2022 11:32 PM, Turritopsis Dohrnii Teo En Ming wrote:

Both Roundcube and Squirrelmail webmail cannot search for anything
because there are more than 200,000 emails in my Inbox.





Both Roundcube and Squirrelmail webmail cannot open many emails
because there are more than 200,000 emails in my inbox.





My IMAP server is Dovecot version 2.2.36.


I artificially created a situation like this in my mailbox.  I created a 
new folder and copied all my other messages in my mailbox to it in the 
Linux commandline.  Ended up with 154K messages in that folder.  Then I 
had to force a complete reindex. Which I have a script for.


I'm using Solr for dovecot searching.  If you do something similar and 
have dovecot connect to something external for search, that should solve 
the first problem.


The folder did eventually populate in roundcube, though it did take a 
while.  My roundcube version is 1.5.2, and my dovecot version is 
2:2.3.18-4+ubuntu20.04.  Dovecot is installed from the official project 
APT repository on Ubuntu Server 20.04.


I seem to be having a problem with my Solr search engine.  No matter 
what keyword I search for in that folder, it always gets 127168 hits.  
That is just not possible when searching for "dragon" and then "aardvark".


Thanks,
Shawn

Re: Set up per user sieve rules

[Shawn Heisey]

On 4/9/2022 9:36 PM, Shawn Heisey wrote:

sieve_default = file:/var/vmail/global.sieve


And in case it matters, here are the contents of the global.sieve file:


---
require ["envelope", "fileinto", "reject", "vacation", "regex",
 "relational", "comparator-i;ascii-numeric"];

#if header :contains "X-Spam-Flag" "YES" {
#  fileinto "Spam";
#}

# discard all mail with a spam score 25 or higher
#if not header :contains "X-Spam-Score" "-" {
#    if header :value "ge" :comparator "i;ascii-numeric"
#  ["X-Spam-Score"] ["25"] {
#    discard;
#    stop;
#    }
#}
---

I should probably uncomment at least the last block.  For the other one, 
I don't know if that's safe to uncomment.  If a destination folder does 
not exist, will sieve create it?  If so, will it also auto-subscribe 
that folder?  Based on errors I've gotten in my own sieve log, I don't 
think sieve will create it.


Thanks,
Shawn

Re: Set up per user sieve rules

[Shawn Heisey]

On 4/9/2022 9:29 PM, Austin Witmer wrote:
I am curious Shawn, what the contents of your 
/etc/dovecot/conf.d/90-sieve.conf file is?


It's almost all comments.  A LOT of detailed comments.  Stripping those out:


---
root@bilbo:~# cat /etc/dovecot/conf.d/90-sieve.conf | egrep -v "^#|^\s+#|^$"
plugin {
  sieve = 
file:/var/vmail/sieve/%d/%u/sieve;active=/var/vmail/sieve/%d/%u/.dovecot.sieve

  sieve_default = file:/var/vmail/global.sieve




}
---

I stole this from a website a long time ago about setting up a 
phpmyadmin/postfix/dovecot system.


Thanks,
Shawn

Re: Set up per user sieve rules

[Shawn Heisey]

On 4/9/2022 6:00 PM, Austin Witmer wrote:

I am struggling to wrap my mind around how to set up sieve rules for my 
individual users on my mail server.

I set up a global sieve script file at "/var/lib/dovecot/sieve/default.sieve” 
and that works great.

My users maildirs are located at /mnt/volume1/mailserver/maildir/user. Do I 
need to create a sieve script file inside of that folder for sieve scripts to 
work on an individual basis? If so, what permissions will that script file need?

Thanks in advance for your help!


I've got the user database for dovecot in a MySQL database managed by 
phpmyadmin.


This is what can be found as far as files and permissions for my email 
user's sieve setup:


https://paste.elyograg.org/view/8442b4d0

I use the managesieve plugin for roundcube to handle changes to my sieve 
script.  At least I think that's what the plugin is called. I set it up 
a LONG time ago.  Most of the time I access mail with Thunderbird.  This 
is my doveconf -n output:


https://paste.elyograg.org/view/08536e83

If somebody sees something I should change in my config, please tell 
me.  I am not an expert.


Thanks,
Shawn

Re: resend whole inbox to user

[Shawn Heisey]

On 2022-04-06 13:29, Marc wrote:

I was wondering if there is some way to force an imap client to
're-download' all the messages from the inbox. I can remember in the
'old days' that when the connection was dropped during a pop download,
the whole inbox was re-downloaded, resulting in quite a lot of
duplicates. I am looking for such action.


I have my mail in Maildir format.  Here's a directory listing of one of 
my folders:


root@bilbo:/var/vmail/elyograg.org/elyog...@elyograg.org/.Sent# ls -al
total 9428
drwxr-x---  5 vmail mail4096 Apr  6 22:30 .
drwxr-x--- 82 vmail mail4096 Apr  7 00:38 ..
drwxr-x---  2 vmail mail  917504 Apr  6 22:08 cur
-rw-r-  1 vmail mail  131824 Apr  5 17:12 dovecot.index
-rw-r-  1 vmail mail 7851560 Apr  6 22:30 dovecot.index.cache
-rw-r-  1 vmail mail 552 Apr  6 22:30 dovecot.index.log
-rw-r-  1 vmail mail   91520 Apr  5 17:12 dovecot.index.log.2
-rw-r-  1 vmail mail  84 Feb 26 00:39 dovecot-keywords
-rw-r-  1 vmail mail  549725 Apr  6 22:08 dovecot-uidlist
drwxr-x---  2 vmail mail4096 Jan 28  2003 new
drwxr-x---  2 vmail mail   69632 Apr  6 22:08 tmp

Something I discovered mostly by accident:  If I delete all the files in 
a mail folder matching a glob pattern of dovecot* then any IMAP client 
that accesses it will re-download all the messages in the folder.  There 
are nearly 11000 messages in that Sent folder.  Takes a while to 
redownload them.


I don't know if there might be unintended side effects from deleting all 
those files.  My mailbox seems to have survived it.


Thanks,
Shawn

Re: email location - files or sql

[Shawn Heisey]

On 2/22/2022 9:12 AM, Robert Moskowitz wrote:
What I am seeing is that many of the packages seem to roll the messages 
into some SQL database.  My Dovecot setup uses the 
/home/vmail/doman/../{cur,new,etc} tree structure.


I use dovecot and postfix in conjunction with a postfixadmin database in 
mysql.


The emails are stored on the filesystem using Maildir, not in the 
database.  Configuration and authentication information for domains and 
users are in the database.  Quotas are in the database.  I have several 
gigabytes of email between the many accounts and would not want that 
stored in a huge database table file.  Although I am sure something 
exists that stores messages in a database with dovecot, I have yet to 
see it.


While a database would probably be more efficient in terms of disk space 
utilization, it would be much less efficient to make incremental 
rsync-based backups like I do currently.  I use rsync to copy only 
changes rather than the entire archive, and btrfs snapshots for a history.


Thanks,
Shawn

Re: Error: fts_solr: received invalid uid

[Shawn Heisey]

On 1/2/22 3:08 PM, John Fawcett wrote:
unless I'm missing something that doesn't look like the type of query 
dovecot launches - dovecot sends various parameters including the 
request for xml results. The thing that looks odd to me is that the 
results fields should be single values not arrays enclosed in [], ie I 
would have expected

"uid":21,

instead of

"uid":[21],

I'd be interested to see the equivalent xml output produced by running 
a query that dovecot sends.




This Solr instance is not configured properly for dovecot.  Most of the 
fields are configured as multiValued (which is why you see an array 
rather than a single value), and the response is JSON.  A Solr instance 
that is properly configured for dovecot would return XML, not JSON.


You're going to want to provide a proper solrconfig.xml and 
managed-schema file, restart Solr or reload the core for dovecot, and do 
a full reindex.  At the following URL, you can find the shell script 
that I use to do a full reindex:


https://apaste.info/tK1B

At the following URL are links to the solrconfig.xml and schema that are 
appropriate for dovecot.  These instructions say to name the schema file 
"schema.xml" ... this will work, as long as Solr has read-write access 
to the conf directory so it can rename the file to "managed-schema".  If 
it doesn't, then you should name the file "managed-schema" yourself.


https://doc.dovecot.org/configuration_manual/fts/solr/

Thanks,
Shawn

Re: modernizing dovecot.conf

[Shawn Heisey]

On 12/29/2021 9:53 PM, Benny Pedersen wrote:

On 2021-12-30 05:40, Laura Steynes wrote:

Because gmail is dumb. good mail client interfaces see reply-all with
a mailing list as reply to list only


so you are just a gmail victim ? :=)


I have no experience on this with gmail.  I haven't subscribed to any 
mailing lists with my gmail account.


Thunderbird changed a while back so it ignores a Reply-To header sent by 
a mailing list and replies to the sender of the email... so if you just 
press Ctrl-R you will not be sending to the mailing list.  It is 
possible to get the old behavior back by disabling the 
"mail.override_list_reply_to" setting in the Thunderbird config editor. 
 I can't remember which version made this change.



I'll await someone more knowledgeable to answer my  original questions
I think


knowledge ?

would imap protocol work on gmail ?, can roundcube use gmail imap 
protocol ?, would it break leggs or bones ?


Yes, roundcube can access gmail.  This is the config I needed to put in 
roundcube's config file:


$config['default_host'] = 'imaps://imap.gmail.com';
$config['default_port'] = 993;

My mail server cannot connect to imap.gmail.com on port 143, so I had to 
change it to 993.  I tested this by making a copy of my roundcube 
install on another URL and reconfiguring the copy.


Thanks,
Shawn

Re: No source packages in APT repo?

[Shawn Heisey]

On 12/8/21 3:46 AM, Aki Tuomi wrote:

You only need dovecot-dev for that, not the sources.


The dovecot-dev package looks like it's for writing programs that 
integrate with dovecot, not for building dovevot.


If I'm wrong about that, can somebody explain how I can use it to build 
custom dovecot packages?


Thanks,
Shawn

Re: No source packages in APT repo?

[Shawn Heisey]

On 12/3/21 1:28 PM, Aki Tuomi wrote:

Is there a particular reason you need to build source packages?



There are sometimes moments when I want to test out a code change in a 
program I'm running on a server.  If a source package is available, I 
can make the change and build a new package that includes all of the 
Ubuntu customizations for that program plus the change I am testing.  To 
test, I just manually install the package.  To revert, I reinstall the 
original package with apt.  Without a source package, testing might 
require complex steps with the upstream source and customization 
information that I may not have access to.


Thanks,
Shawn

No source packages in APT repo?

[Shawn Heisey]

I added a deb-src entry to my APT config matching the deb entry:

deb https://repo.dovecot.org/ce-2.3-latest/ubuntu/focal focal main
deb-src https://repo.dovecot.org/ce-2.3-latest/ubuntu/focal focal main

When this is there, I see a warning when running "apt update".

W: Skipping acquire of configured file 'main/source/Sources' as 
repository 'https://repo.dovecot.org/ce-2.3-latest/ubuntu/focal focal 
InRelease' does not seem to provide it (sources.list entry misspelt?)


Were source packages excluded deliberately?  The github repo for dovecot 
does not have a "debian" directory, which is the other route that would 
allow me to build packages.


Thanks,
Shawn

Re: FTS config - cannot search substrings

[Shawn Heisey]

On 11/18/2021 4:43 PM, infoomatic wrote:

sometimes ignoring the problem for a few moments and then starting over
again helps ...

the solution was already solved here in a mail over 10 years ago -
https://dovecot.org/list/dovecot/2011-May/059338.html

Thanks Daniel Miller!


There's something to keep in mind here.  This is the critical piece that 
makes the substring search work:




Because the minimum size is 3, that config means that it is impossible 
to search for one or two letter words -- terms shorter than minGramSize 
are dropped.  This could affect the quality of search results.


If you are running at least version 7.4.0 of Solr, you can add this to 
the config for that filter so that isn't a problem:


preserveOriginal="true"

https://javadoc.io/static/org.apache.lucene/lucene-analyzers-common/7.4.0/org/apache/lucene/analysis/ngram/NGramFilterFactory.html

Thanks,
Shawn

Re: Solr FTS - message deletes not working as expected

[Shawn Heisey]

On 11/3/21 11:45 PM, Shawn Heisey wrote:
Manual expunges of existing messages also are not sending a delete 
request to Solr.  I waited several minutes for that too. 


Update on this, since you're all on the edge of your seat waiting. :)

I did a Send test with TypeApp, a mail client for Android.  It behaved 
completely as expected:  Dovecot immediately sent a delete request to 
Solr for the temporary message in Drafts.


So I went to Mozilla forums and started a discussion about what I am 
seeing with Thunderbird.


I did another test run with sending a message with unique text, and 
doing a manual Solr query for that unique text, I saw one result before 
sending and two results after sending.  Watching the Solr log, no delete 
request was sent.


Then something completely unexpected:  I needed to reboot the laptop, so 
I quit Thunderbird, and I hadn't yet closed my ssh session to the mail 
server, which was still tailing the solr log with a grep for delete 
messages.  As soon as I did the quit, Solr got a delete request for the 
Drafts message.  I did another test -- shift-delete a message, see that 
Solr did not see the delete request.  And then after I waited a while, I 
quit Thunderbird again.  Instantly got a delete request in Solr's log.


Definitely a Thunderbird problem, not dovecot.  Thanks for your 
patience, everyone.  I will pursue it with Mozilla now.


Thanks,
Shawn

Re: Solr FTS - message deletes not working as expected

[Shawn Heisey]

On 11/3/21 12:38 PM, Michael Slusarz wrote:

Have you tried another client?



I tried evolution on Linux.

I can't work out how to do an expunge in evolution.  I'm not sure that 
it CAN do it.  Whenever I delete a message, even with shift-delete, it 
is moved to the Trash folder (with delete and add sent to Solr), and 
then emptying the trash does delete from Solr.  This is exactly how 
Thunderbird works with "normal" deletes.


I use shift-delete in Thunderbird a lot.  Because when I delete a 
message, I really want it gone, I do not want another step to remember 
later -- emptying the trash folder.


The FTS behavior on sending a message with Thunderbird seems completely 
wrong.  When I send a message with evolution, I do see a delete in the 
solr log, but there are still two results in a query for the special 
text I included in the message, so something is still not quite right.


I couldn't get Windows Mail to connect to my email account. Even with 
their advanced settings.  I enter all the right information, and it just 
fails to connect.  There are only "Use SSL" checkboxes, and I have no 
idea what port/security settings it's using.  I can't tell it to use 
STARTTLS instead of mandatory TLS, or to use port 143 for imap and 587 
for submission.  I think that I did see a successful imap login in the 
mail.log from the attempt to add the account, but I had other clients 
running so it is difficult to be sure about that. I will have to close 
all my other clients (which includes an Android client) and try the 
connect again.  When the account setup fails to connect, it doesn't have 
an option to add the account anyway (which Thunderbird does have), so I 
can't edit the account settings.


The android client seems to not have an expunge option for deleting 
messages either.  It goes through Trash.


Thanks,
Shawn

Re: Solr FTS - message deletes not working as expected

[Shawn Heisey]

On 11/3/2021 11:10 PM, Shawn Heisey wrote:
Then I downloaded the source archive from the main site, extracted it, 
and the configure script was included in that.  After compiling, I found 
the .so file and moved it into place, and I will be testing it.


The new library is significantly larger than the one it replaced, 332 KB 
versus 55KB.  I'm guessing it has things like debug symbols that get 
stripped when it's packaged.


I see no change in behavior even with that section of code removed.  I 
included some unique text in the body of the message I wrote and sent, 
and after sending I find both the copy in Drafts and the copy in Sent 
when I do a manual Solr query for that unique text.  I waited several 
minutes ... I didn't hit Send and then immediately do the query.


Manual expunges of existing messages also are not sending a delete 
request to Solr.  I waited several minutes for that too.


Thanks,
Shawn

Re: Solr FTS - message deletes not working as expected

[Shawn Heisey]

On 11/3/2021 10:12 PM, Shawn Heisey wrote:
Maybe I can do a custom compile of the source code and replace the 
/usr/lib/dovecot/modules/lib21_fts_solr_plugin.so file with what the 
compile produces.  I'm going to try that, and see if it explodes. :)


Bit of a problem trying to do this.  I pulled the source code from 
github, switched to the release-2.3.17 branch, edited the source code 
file you referenced, and found that the "configure" script that is 
mentioned as the first step in the INSTALL.md file is not present.  It's 
not present in any of the other branches I looked at either.


Then I downloaded the source archive from the main site, extracted it, 
and the configure script was included in that.  After compiling, I found 
the .so file and moved it into place, and I will be testing it.


There should maybe be something in the README that describes what is 
needed to go from 'git clone' to the point where the instructions in 
INSTALL will work.


Thanks,
Shawn

Re: Solr FTS - message deletes not working as expected

[Shawn Heisey]

On 11/3/21 1:09 PM, Michael Slusarz wrote:

For Solr, there's a code path in the FTS expunge code that will silently toss 
expunge requests:

 if (ctx->last_indexed_uid == 0 ||
 uid > ctx->last_indexed_uid + 100) {
 /* don't waste time asking Solr to expunge a message that is
highly unlikely to be indexed at this time. */
 return;
 }

So it's possible you are running into that.


Interesting.  I don't know dovecot code well enough to figure out if 
maybe *every* expunge that happens is classified by that code as 
"unlikely to be indexed at this time."  For the situations I have tried, 
I know that the message HAS been indexed and should be deleted from the 
index.


I was going to try removing that whole code construct and running it to 
see what happens, but it looks like the dovecot apt repo does not have 
source packages.  When I add a deb-src line to the dovecot repo config, 
then do "apt update" (which works without complaint) followed by 
"apt-get source dovecot-solr" it pulls source packages from the Ubuntu 
repo, not the dovecot repo, so the source I end up with is for version 
2.3.7, not the 2.3.17 that I am running.


Maybe I can do a custom compile of the source code and replace the 
/usr/lib/dovecot/modules/lib21_fts_solr_plugin.so file with what the 
compile produces.  I'm going to try that, and see if it explodes. :)


Thanks,
Shawn

Re: Solr FTS - message deletes not working as expected

[Shawn Heisey]

On 11/3/21 12:38 PM, Michael Slusarz wrote:

Have you tried another client?


I only have two clients configured:  Thunderbird 78.13.0 on Linux and 
Thunderbird 91.2.1 on Windows.  Behavior is the same on both.


I will see if I can get another client configured.  Windows Mail is 
included on Windows 10, so I can try that.  I have a client system 
running Ubuntu, and I bet that there are a LOT of mail clients available 
there.



Thunderbird does NOT necessarily process expunges immediately.  Depends on what 
else it is doing in the background.  So you can't click delete in the UI and 
not immediately see anything on the backend and definitively correlate the two.


The message is deleted by dovecot immediately.  I double checked this by 
purging a message on my Linux client and saw the message immediately 
disappear on my Windows client.  It happened even faster than I would 
have expected.  IMAP seems to be a very good protocol!  I can't say it 
definitively without more evidence, but the problem *seems* to be in 
FTS, not imap or core dovecot.



Another option is to ensure debug logging is enabled for Dovecot so you can see what the FTS code 
is doing.  "log_debug = category=fts" and/or "mail_debug = yes" will help in 
that regard.



When I have some real time available, I will look into debug logging.  
I'm going to have to research exactly where to put those config options.


Thanks,
Shawn

Re: Solr FTS - message deletes not working as expected

[Shawn Heisey]

On 10/28/21 8:00 AM, Shawn Heisey wrote:
Also, when I send a message with Thunderbird, which deletes the 
message in Drafts and adds one to Sent, I am not seeing a delete 
request in the Solr log.  I do see the add. So this isn't limited to 
just the shift-delete workflow. 



I have confirmed this with multiple attempts.

I start a new message in Thunderbird.  Then I wait around for that 
message to be auto-saved to Drafts.  When that happens, I see an "add" 
request in the solr log.


Then I send the message.  At that point, I see another add in Solr's 
log.  Based on the message number in the add request, I know that this 
time the add happens in the Sent folder.  But despite the fact that 
Thunderbird deletes the message from Drafts, Solr never sees a delete 
request.  My dovecot version has been updated since the last time I 
indicated what version it is.  Now it is "2:2.3.17-3+ubuntu20.04" from 
the dovecot repo, not the ubuntu repo.  The same thing happened with 2.3.16.


As I mentioned in the first message on this thread, when shift-delete is 
used in Thunderbird to delete messages, that also never sends a delete 
to Solr.


Something somewhere is misbehaving.  Is it Thunderbird accessing IMAP 
incorrectly, or is it Dovecot?


I will do a packet capture to see if maybe dovecot is sending requests 
that are not logged by Solr.  That seems unlikely -- even bad requests 
should result in some kind of entry in the solr log.


Thanks,
Shawn

Re: Solr FTS - message deletes not working as expected

[Shawn Heisey]

On 10/26/21 12:18 PM, Shawn Heisey wrote:
But if I use shift-delete in Thunderbird, which deletes the message 
immediately without going through Trash, things are different. 



Also, when I send a message with Thunderbird, which deletes the message 
in Drafts and adds one to Sent, I am not seeing a delete request in the 
Solr log.  I do see the add.  So this isn't limited to just the 
shift-delete workflow.


Thanks,
Shawn

Solr FTS - message deletes not working as expected

[Shawn Heisey]

I have Solr FTS enabled in dovecot.  It is behaving differently than I 
would expect.


When I delete messages in Thunderbird normally, using IMAP to talk to 
dovecot, the deletion hits Solr right away.  If I then ask Thunderbird 
to empty the trash, that delete also hits Solr immediately.


But if I use shift-delete in Thunderbird, which deletes the message 
immediately without going through Trash, things are different.  One 
delete that I did (6 messages) took at least a minute before Solr saw 
the request.  And on other tests that I ran, deleting one message, I 
never did see the delete request hit Solr.  I waited for several minutes.


I use the shift-delete mechanism a lot more than the method that sends 
to trash.  That way I do not need to remember to empty the trash.


Dovecot version is 2:2.3.16-2+ubuntu20.04 installed from the dovecot 
repo, not from distro packages.  I don't think the Solr version matters, 
but just in case I am wrong, that is 8.10.1.  I don't recall exactly 
what I changed in the solrconfig.xml and managed-schema files, but it 
should not be anything that affects how Solr FTS works.  The problem 
seems to be that Solr FTS isn't sending deletes, not that Solr is 
behaving incorrectly.


I just discovered that trying to get the dovecot config failed:

root@bilbo:~# doveconf -n
# 2.3.16 (7e2e900c1a): /etc/dovecot/dovecot.conf
# Pigeonhole version 0.5.16 (09c29328)
doveconf: Fatal: Error in configuration file 
/etc/dovecot/conf.d/10-master.conf line 64: Invalid size: $default_vsz_limit


After fixing that, this is what I get for the config:

root@bilbo:/etc/dovecot# doveconf -n
# 2.3.16 (7e2e900c1a): /etc/dovecot/dovecot.conf
# Pigeonhole version 0.5.16 (09c29328)
# OS: Linux 5.11.0-1020-aws x86_64 Ubuntu 20.04.3 LTS ext4
# Hostname: REDACTED
default_vsz_limit = 1 G
dict {
  quota = mysql:/etc/dovecot/dovecot-dict-sql.conf.ext
}
first_valid_uid = 150
last_valid_uid = 150
listen = *
mail_gid = mail
mail_location = maildir:/var/vmail/%d/%u
mail_plugins = " quota fts fts_solr"
mail_privileged_group = mail
mail_uid = vmail
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope 
encoded-character vacation subaddress comparator-i;ascii-numeric 
relational regex imap4flags copy include variables body enotify 
environment mailbox date index ihave duplicate mime foreverypart extracttext

namespace inbox {
  inbox = yes
  location =
  mailbox Drafts {
    special_use = \Drafts
  }
  mailbox Junk {
    special_use = \Junk
  }
  mailbox Sent {
    special_use = \Sent
  }
  mailbox "Sent Messages" {
    special_use = \Sent
  }
  mailbox Trash {
    special_use = \Trash
  }
  prefix =
}
passdb {
  args = /etc/dovecot/dovecot-sql.conf.ext
  driver = sql
}
plugin {
  fts = solr
  fts_autoindex = yes
  fts_solr = url=http://localhost:8983/solr/dovecot/
  quota = dict:User quota::proxy::quota
  quota_grace = 10%%
  quota_max_mail_size = 100M
  quota_status_nouser = DUNNO
  quota_status_overquota = 552 5.2.2 Mailbox is full
  quota_status_success = DUNNO
  sieve = 
file:/var/vmail/sieve/%d/%u/sieve;active=/var/vmail/sieve/%d/%u/.dovecot.sieve

  sieve_default = file:/var/vmail/global.sieve
}
postmaster_address = postmaster@REDACTED
protocols = " imap lmtp sieve pop3"
service auth {
  unix_listener /var/spool/postfix/private/auth {
    group = postfix
    mode = 0666
    user = postfix
  }
  unix_listener auth-userdb {
    group = mail
    mode = 0666
    user = vmail
  }
}
service dict {
  unix_listener dict {
    group = postfix
    mode = 0660
    user = vmail
  }
}
service imap {
  vsz_limit = 1 G
}
service stats {
  unix_listener stats-reader {
    group = vmail
    mode = 0660
    user = vmail
  }
  unix_listener stats-writer {
    group = vmail
    mode = 0660
    user = vmail
  }
}
ssl_cert = I repeated the delete test after fixing the config and there was no 
change -- the delete request never made it to Solr.


Reindexing completely for my dovecot install takes about 8 minutes.  I 
have less than 150K messages.  Reindexing for some could take hours or 
days.  This is what I use to initiate a full reindex:


root@bilbo:~# cat /usr/local/sbin/reindexsolr
#!/bin/sh
DOVECOT_SOLR_URL_BASE="http://localhost:8983/solr/dovecot;
DEL_ALL_QUERY_XML="*:*"
MAIL_ROOT=/var/vmail

service dovecot stop
curl \
  "${DOVECOT_SOLR_URL_BASE}/update?commit=true=true" \
  -H "Content-Type: text/xml" \
  --data-binary "${DEL_ALL_QUERY_XML}"
find ${MAIL_ROOT} -name "dovecot.index*" -print0 | xargs -0 rm -f
service dovecot start
doveadm index -A -q '*'

Re: Dovecot - FTS Solr: disk usage & position information?

[Shawn Heisey]

On 9/6/2021 12:58 AM, Vincent Brillault wrote:

Hi Alessio,


this optimization also produce a less RAM requirements on Solr server?


Unfortunately we didn't measure this before/after the change. Since we 
are removing features (position information), I wouldn't expect the 
memory requirement to increase, but I'm no expert.


To be honest, I've not been able to measure in any sensible way the 
memory really required by Solr. The memory directly used by the Solr 
process is rather limited, but a lot of memory is used for file caches, 
which also feels (again not an expert) important for good performances.


There likely would be a decrease in the amount of memory required, 
especially at index time.


The best way to see what the minimum requirements really are for a Java 
program is through GC logs.  Run the program for a really long time, 
exercising it hard.  Gather the GC logs, and have the gceasy.io website 
analyze those logs.  Solr comes configured out of the box to generate GC 
logs.


One of the graphs that gceasy.io has is "Heap After GC" ... the low 
points in that graph (as long as the program was busy during that 
timeframe) will be the minimum requirement.  You would want to add some 
arbitrary value to that number and set your max heap to that.  For 
instance, if I was seeing 8GB as the minimum required, I would probably 
want the heap to be 10GB or 12GB.  Java memory management works best 
when it has a little breathing room.  If the heap is too close to the 
minimum requirement in size, Java will be spending more time doing 
garbage collection than it spends running the application.


At least the solrconfig.xml shouldn't work with 7.7.0 since I increased 
the luceneMatchVersion to match 8.6 and imported a few defaults from the 
default upstream 8.6 configuration. I think these changes could be 
ignored for 7.7.0.


I was curious what a significantly newer luceneMatchVersion would do. 
So I unpacked solr 6.1.0 (which I already had downloaded), started it, 
created a core, and edited the config for that core so the version was 
8.9.0, then restarted Solr.  As expected, it came up with no problem.


Then I did another test, with my Solr install for dovecot.  It's running 
8.9.0, so I set lMV to 9.5.0 and restarted Solr.  Then I sent myself an 
email to trigger an update to the Solr index.  And I did a query in the 
Solr admin UI.  Everything worked.


So having a luceneMatchVersion that's far beyond the actual Solr version 
won't cause any problems, and I doubt that it makes any difference in 
how things work.  What this setting is for is the ability for some 
Lucene analysis components to work as they did in older versions -- so 
users could keep older behavior that they relied on that changed when a 
bug was fixed.  For example, in version 4.8.0 a major bug was fixed in 
the word delimiter filter ... the fix for this bug caused it to work 
very differently than it did before.  For a while after that, it was 
possible to set lMV to 4.7.0 or earlier and regain the old behavior.


For schema.xml, I made quite a few changes, but all seem to be backward 
compatible:

  - Remove unused 'boolean' field type
  - Remove KeywordMarkerFilterFactory: protwords are usually empty anyway
  - User a simper 'text_basic' field type (no StopFilterFactory, 
SynonymGraphFilterFactory or PorterStemFilterFactory) for processing 
non-human fields (all but body and subject)
  - Remplace autoGeneratePhraseQueries & positionIncrementGap by 
omitTermFreqAndPositions="true" & omitPositions="true" on TextField 
fieldtypes (as discussed in this thread)
  - Minor modifications on WordDelimiterGraphFilterFactory when used in 
search to have better match (things like 'covid19' are indexed as 
['covid', '19', 'covid19'] but only searched as 'covid19')


Having an unused fieldType, or even an unused field, does not cause any 
problems.  It will use up an extremely small amount of memory as Solr 
converts the schema into an in-memory structure.  But it makes zero 
difference in the index size, and any overhead at index time and query 
time would probably be nearly impossible to measure, unless there is a 
huge number of them unused.


The keyword marker filter is one of those things that most people will 
have no idea how to use.  And if the config file for it is empty or 
nonexistent, it doesn't do anything.  Good riddance. :)


It's also a good thing to remove the stopword filter.  Unless you 
actually define some stopwords, it doesn't do anything.  Removing 
stopwords was a performance enhancement that was hugely beneficial when 
CPU and memory capacities were a tiny fraction of what they are today. 
Now, it doesn't provide much benefit, and comes with some pretty 
well-known downsides.  Also happy to see that one go.  On an index the 
size of yours, removing really common stopwords could make the index 
noticeably smaller, but I bet a few of your users would notice the 
downsides.


For most people, there's not a lot of 

Re: Solr FTS - when does indexing happen?

[Shawn Heisey]

On 9/4/2021 4:52 PM, Shawn Heisey wrote:
I see something talking about autoindex, but it does not have an example 
so that I can see where it needs to go.  I cannot work it out from what 
is there.


With a little googling, I was able to figure out where it needs to go. 
And now it acts like I was expecting.


Deletes are an interesting thing with autoindex.  If I use the "Del" key 
in Thunderbird (which moves the message to the Trash), I see an 
immediate delete (from the original folder) and add (to the Trash 
folder) in Solr's log.  And if I choose the "Empty Trash" option, I see 
those deletes in Solr's log immediately.


But if I press Shift-Del in Thunderbird (which immediately deletes the 
message, bypassing Trash), then it takes about 15 seconds before the 
Solr log shows the delete request.  Is that expected?  It's not causing 
me any problems, as it's highly unlikely that I'm going to do a query 
matching a message that I deleted ten seconds ago.  I can stand to wait 
15 seconds for the index to be updated.


Dovecot version is 2:2.3.16-2+ubuntu20.04, pulled from the Dovecot 
repository.


I have been doing some fiddling with the solrconfig and schema.  I have 
more fields stored now -- added from, to, and subject.  I couldn't tell 
what the matching messages were when accessing Solr directly.


I also implemented TrimFieldUpdateProcessorFactory which trims leading 
and trailing whitespace from fields before they are indexed.  I happened 
to notice that some of the new stored fields I added had EOL characters 
in them (not sure if it was \n or \r\n).


IMHO, a rather glaring omission from the fields in Solr is a 
timestamp/date field.  Does dovecot's FTS have the ability to send that 
data?  I know that Dovecot might not use it, but it would be a very 
useful thing to have for querying the dovecot index from something other 
than dovecot.  Not something I *NEED*, just nice to have.  I haven't 
looked at the fts or fts_solr code.


Thanks,
Shawn

Re: Solr FTS - when does indexing happen?

[Shawn Heisey]

On 9/4/2021 4:06 PM, Steve Dondley wrote:
As I recall, indexing an email is triggered immediately when an email is 
received if you have you dovecot settings set properly to trigger the 
indexing. The dovecot documentation for FTS, it spells it out.


See 
https://doc.dovecot.org/configuration_manual/fts/solr/?highlight=fts%20user%20plugin 


There is an autoindex setting that neeeds to be set to "yes".


I see something talking about autoindex, but it does not have an example 
so that I can see where it needs to go.  I cannot work it out from what 
is there.


With a little googling, I was able to figure out where it needs to go. 
And now it acts like I was expecting.


Since most people will want fts_autoindex, the wiki page should include 
it in its example configuration that goes into 90-plugin.conf.  Possibly 
better ... maybe it should default to "yes".


Thanks,
Shawn

Solr FTS - when does indexing happen?

[Shawn Heisey]

I have Solr FTS on my dovecot install.  I followed the instructions on 
the dovecot wiki.


How long a delay should I expect to see between new mail being delivered 
with the dovecot LDA and an indexing request sent to Solr?  Because I 
get a LOT of email from various mailing lists, and I do not see any 
activity in Solr's log.  When I did doveadm index -A -q '*' there was a 
lot of indexing activity in Solr's log, as expected.


One time I looked at the Solr index and it had been 23 hours since it's 
last update ... I can guarantee that I received a lot of new messages in 
that time.


What do I need to look at for further troubleshooting?

I can confirm that when I issue a search in the TypeApp app on my phone 
(an IMAP app for android), I see the query in Solr's logfile.


Thanks,
Shawn

Re: [OT] Re: What kind of search response time are you setting with solr full text search?

[Shawn Heisey]

On 8/25/2021 2:10 PM, Steve Dondley wrote:


And it looks like I'm running into a major bug in the slightly dated 
version of dovecot debian uses:


https://www.mail-archive.com/dovecot@dovecot.org/msg78825.html



Recently I did a fairly major upgrade.  I had an older Ubuntu release 
with Dovecot 1.x and an older Postfix version, wanted to upgrade it to 
v20, which had significantly newer versions of both programs.


Before I did the Ubuntu upgrade, I took advice received here and 
installed the dovecot repo, upgrading it first.  I looked at the 
following link and only found a couple of things in my config I needed 
to change.  I didn't even use the conversion command at the top of the 
page.  Everything still worked after I upgraded, which I found a little 
surprising.  Postfix also worked after the upgrade, with no config 
changes required.


https://doc.dovecot.org/installation_guide/upgrading/from-1.2-to-2.0/

I did the fts_solr integration *after* I upgraded dovecot.

You can find the dovecot repos here, there are options for Debian:

https://repo.dovecot.org/

Thanks,
Shawn

Re: [OT] Re: What kind of search response time are you setting with solr full text search?

[Shawn Heisey]

On 8/25/2021 12:05 PM, Steve Dondley wrote:


OK I had to modify the query path slightly to get it to work with my 
core to:


time curl 
http://localhost:8983/solr/dondley/select?q=maynez=edismax=body+to+subjec:t+ccfrom


But it didn't return any results:





I only have emails for this person in a subfolder of my main Inbox 
folder so maybe it's only searching the top level folder? 


That query should search ALL emails that dovecot has indexed to Solr.  
There is no restriction for mailbox or folder.


Try replacing "maynez" with something else that you know will be in the 
index.


Note that you can ask dovecot to completely reindex everybody's email 
with this command run as root, and then you can try searching again a 
few minutes later:


doveadm index -A -q '*'

It took me forever to work out the right options for that command.  I 
think it should probably get added to the wiki page about fts_solr.


Question for experts on the dovecot side of this:  What does it take for 
dovecot to index new messages delivered with dovecot's LDA by postfix?  
Is it a matter of waiting a certain amount of time?  I saw in Solr that 
the last update to the index was 3 hours earlier, but I had received 
email only a few minutes before looking at Solr's stats.


Thanks,
Shawn

Re: [OT] Re: What kind of search response time are you setting with solr full text search?

[Shawn Heisey]

On 8/25/2021 11:33 AM, Steve Dondley wrote:
OK, I take this back. I did an imap search via telnet and solr reports 
the search takes about 3 to 4 seconds. Here's the output:


a search text "maynez" 


Try this in on the commandline of the Solr server:

time curl 
"http://localhost:YYY/solr/dovecot/select?q=maynez=edismax=body+to+subject+cc+from;


Where YYY is the port Solr is listening on.

In the response, QTime is how long it took for Solr to do the search.  
This does not include time for the result set to be gathered or the time 
to transfer the response over the network, which is where the "time" 
part of that comes in.  I see this doing a similar query on my Solr:


real    0m0.026s
user    0m0.011s
sys 0m0.010s

QTime for that request was 1 millisecond -- so it took 25 milliseconds 
for the stuff that is not counted in QTime.


You could do the query remotely by changing "localhost" to the hostname 
or IP address of the Solr server.


Thanks,
Shawn

[OT] Re: What kind of search response time are you setting with solr full text search?

[Shawn Heisey]

On 8/24/2021 7:19 PM, Steve Dondley wrote:

THE PROBLEM: When I do a full text search through all my inbox and all
subfolders on a single word, search results are returned in about 10
to 15 seconds. This is better than the 40 seconds or so I'm getting
when I turn off the fts and fts_solr plugins but still a little
disappointing.


I did some experimenting. I noticed that if the word I'm searching on 
is fairly rare, results will pop up quickly, like in around 3 to 5 
seconds. Words that don't exist at all in any email returns nothing 
almost instantly.


But words that appear in several hundred emails are the ones that are 
take a much longer time. 


This is offtopic for this list, but I will try to help you.  If I am 
unsuccessful, you should raise the issue on the solr-users mailing list.


How much of the total server memory of 4GB did you give to Solr for its 
heap?  Is there other software running on that server besides Solr?


What's the total size of all the Solr indexes on the Solr server?

Can you get the screenshot mentioned at the following URL, put it on a 
file-sharing site, and give me the URL?


https://cwiki.apache.org/confluence/display/SOLR/SolrPerformanceProblems#SolrPerformanceProblems-Askingforhelponamemory/performanceissue

(disclaimer: I wrote that Solr wiki page)

You should read the entire page, the link above is to the section 
describing useful screenshots)


General note:  A Solr search that takes 3 seconds (let alone 15) would 
have me concerned.  If the system is sized appropriately, I would expect 
a search even on a massive index to complete in less than a second.


I happen to be using Solr for dovecot myself.  If I search my index for 
"the" which is very common in English text, the query takes 19 
milliseconds, and that is searching on five fields, as well as doing a 
facet on the user field.  My Solr index has 150048 messages (122K of 
those are in my personal mailbox) and takes up 628 megabytes of disk 
space.  The total size of the email that is indexed is 7 gigabytes.


|+(cc:the | from:the | to:the | 
body:the | subject:the) |


My index is using the stopword filter but the list of stopwords is empty.

The following response may interest you:

https://apaste.info/hm42

This is a search for "a" which I had run several times, so Solr was 
serving it from its cache, and this time it only took 6 milliseconds.  
It also shows what a facet can do.  The longest time I got for the "a" 
search was 15 milliseconds, before the query was in the cache.


Thanks,
Shawn

Re: Dovecot - FTS Solr: disk usage & position information?

[Shawn Heisey]

On 8/5/2021 1:00 AM, Vincent Brillault wrote:

Indeed, I should have. I'm using Solr 8.6, which is clearly not the same
as Solr 6.2.0, but when looking at more recent versions of the
documentation, no information about the use of each file appeared.
That's why I was mentioning it was slightly outdated.


Here's documentation from 8.6 about Lucene file formats:

https://lucene.apache.org/core/8_6_0/core/org/apache/lucene/codecs/lucene86/package-summary.html

Thanks,
Shawn

Re: Dovecot - FTS Solr: disk usage & position information?

[Shawn Heisey]

On 8/4/2021 1:24 AM, Vincent Brillault wrote:

On a local dovecot cluster currently hosting roughly 2.1TB of data,
using Solr as its FTS backend, we now have 256GB of data in Solr, split
in 12 shard (to which replication adds 256GB of data through 12
additional cores).

I'm now trying to see if we can optimize that data. Looking at one core
at random (22G), I see that the data is split mostly between
- .pos files: 12G
- .tim files: 4.2G
- .doc files: 3.8G
- .cfs files: 1.8G

Looking around a bit, I found
https://lucene.apache.org/core/6_2_0/core/org/apache/lucene/codecs/lucene50/Lucene50PostingsFormat.html
(which is unfortunately a bit outdated I think) that explains each file
content:
- .tim: Term Dictionary
- .tip: Term Index
- .doc: Frequencies and Skip Data
- .pos: Positions
- .pay: Payloads and Offsets


This is completely off-topic for the dovecot list.  I am involved with 
the Solr project, so I can discuss it.  My message will also be off 
topic here.


You didn't say what version of Solr you're on.  That document for Lucene 
6.2.0 would be relevant for Solr 6.2.0.  There are versions of that 
document for all Lucene releases, which have been in lock-step with Solr 
releases since one of the early 3.x versions. (Aside: Solr has been 
split into its own top-level Apache project, so there is no longer a 
guarantee moving forward that Solr X.Y.Z will be based on Lucene X.Y.Z)


Not all of the lucene file types will be involved on every install of 
Solr.  It will depend on the configuration.


The .cfs file is a file where all of the other file types for a segment 
are compounded into a single file.  Within that single file, each file 
type will use the same format as it would if it had its own extension.  
I'm not completely clear on when Lucene (under Solr's control) will 
choose the CFS format .. but I think it happens when the segments are 
small, not large.



Looking at Solr documentation on search
(https://solr.apache.org/guide/8_6/the-standard-query-parser.html) it
seems that position aware query are written as `"term1 term2"~[0-9]+`.
Looking at the dovecot code
(https://github.com/dovecot/core/blob/master/src/plugins/fts-solr/fts-backend-solr.c),
I don't see this kind of query being made, `~` only being used for fuzzy
search.


Positions are required for a phrase query -- where the query text is in 
double quotes.  The number after ~ on a phrase query refers to phrase 
slop -- think of it as a fuzziness factor for the phrase, not for each 
term.  Right now you noticed that dovecot's FTS Solr plugin doesn't 
explicitly use phrase queries, but there's no guarantee that this will 
always be the case.  Position data will only be accessed if it is needed 
for a query, so if it is not needed it should not affect query 
performance.  I cannot speak as to whether the FTS Solr plugin relies on 
the autoGenereatePhraseQueries functionality, but if it does, then you 
definitely want position data in the index.  That functionality can do a 
lot to improve relevancy ranking, so I would expect it to be 
instrumental in good full-text searching -- disabling positions will 
probably not help your search results.


If you want an in-depth discussion beyond this email, please subscribe 
to the solr-user mailing list and ask there.


Note that general Solr recommendations are to have enough space 
available that the Solr index can triple in size temporarily -- this is 
to accommodate all possible scenarios for Lucene segment merging.  
Running Solr on systems with limited disk space is not recommended.


Solr does have an "optimize" operation which will combine all the 
segments into one, removing space taken up by deleted documents as it 
works.  Lucene calls that operation "forceMerge".  Running an optimize 
can help performance, but it's extremely resource intensive and can take 
a long time to run -- performance gets worse before it gets better.  
Also, the amount of performance gain is not usually significant.


Thanks,
Shawn

Re: Major upgrade of mail server

[Shawn Heisey]

On 7/8/2021 7:46 AM, Oscar del Rio wrote:
It would also be a good opportunity to switch to using the official 
Dovecot repositories.

https://repo.dovecot.org/


I tried this, and at first couldn't figure out why the packages were in 
a broken state that wouldn't install.  Then I realized I had added the 
repo for focal (20.04) but I'm still on bionic (18.04).  Now the 
packages are good to go for getting dovecot upgraded and problems 
resolved before I do the OS upgrade.  I have gone through the notes on 
the wiki for upgrading from 2.2 to 2.3 and only found a very small 
number of things that needed updating.


I'm going to start doing upgrades in about an hour.

Thanks,
Shawn

Major upgrade of mail server

[Shawn Heisey]

I have a mail server in AWS that is currently running Ubuntu 18.  Every 
time I log in, I am reminded that I can upgrade to Ubuntu 20.


On Ubuntu 18, the dovecot version is 3.3.0-1ubuntu0.3.  On Ubuntu 20, it 
is 2.3.7.2-1ubuntu3.  Many other packages, probably including the mysql 
server, would also be upgraded.


Dovecot and Postfix use a postfixadmin database in mysql for users, and 
postfix is using dovecot-lda to deliver mail.  I am using managesieve 
from dovecot on roundcube webmail.  As far as I know, my own user is the 
only one with sieve scripts actually in use ... and I have a LOT of 
filters/folders for various mailing lists.


I've been a little bit terrified of doing an upgrade, because I do have 
a couple of people using my mail server for real work email and I don't 
want to disrupt them.


I'm writing today to find out what are the likely pain points I might 
encounter when doing this kind of major upgrade, and if there is any 
helpful information that can help me get through those problems.  I'm 
hoping that it will go smoothly and everything just works.


Here's the doveconf -n output:
https://apaste.info/FUgF

If I have been silly enough to include sensitive data from the config, I 
would appreciate a heads up so I know what passwords to change.  I did a 
quick glance and didn't see anything.


Thanks,
Shawn

OT: Re: Bypass Trash Folder Sometimes.

[Shawn Heisey]

On 7/3/2021 1:18 PM, dove...@ptld.com wrote:

this is imho not controlled in dovecot at all


Oops, wrong mailing list.
I meant to send this to the roundcube mailing list.


Roudcube user here, talking to dovecot.

I've just verified something I thought was true.  If you select one or 
more messages in roundcube, then press shift-delete on the keyboard, and 
confirm the deletion, it DOES skip the trash folder.  Exactly what 
you're after.


I've got roundcube 1.4.11, but I am pretty sure that I saw the same 
behavior with 1.3.9.


Thanks,
Shawn

Re: Incorrect value sent to solr

[Shawn Heisey]

On 4/7/2021 4:13 AM, Łukasz Szczepański wrote:

I've prepared pull request with fixed rows parameter:
https://github.com/dovecot/core/pull/160
I've tested this fix on mailbox which caused a problem earlier, and its 
works fine.


I admit to not being very familiar with dovecot source, but I did a 
little digging and that fix looks good to my untrained eye.


If somebody has 2.2 billion or more messages in one place, the query 
will still fail, but it should be pretty rare to ever run into that use 
case in the wild.  And I think it might cause performance problems in 
dovecot too.


Does the solr backend code ever use pagination on the search results by 
sending a fixed rows value and increasing start value on subsequent queries?


I think that I and my fellow Solr committers need to treat this as a bug 
on our end, since it would be perfectly valid to request that many rows 
on a distributed index.  A query like that where there really are that 
many rows would be embarrassingly slow to execute and require a LOT of 
memory, but it's still valid.  I am asking on the solr users list 
whether they think I should file a bug report.


Thanks,
Shawn

Re: Incorrect value sent to solr

[Shawn Heisey]

On 4/3/2021 12:12 PM, Łukasz Szczepański wrote:
Rows isn't part of manage-schema for solr collection its build in common 
query parameters in solr, and type of this field cannot be changed. It 
controls the number of returned results (by default 10).
I still think that an issue on dovecot side, but I was also wrong, this 
parameter is important and have to be sent. Dovecot should send a count 
of messages in mailbox (or imap folder) not the highest uid in folder.


I just tried a query on a stock Solr example with rows set to 30 
and got an error similar to the one you reported.


org.apache.solr.common.SolrException: For input string: "30"

No mention of "int" in this error.  I'm wondering if that part was 
something you added to the email, not text that was actually in the error.


I was using Solr 8.5.1, the newest version I have downloaded right now.

This error also occurs in cloud mode. With distributed indexes, it could 
be perfectly acceptable to use a rows parameter that exceeds what a 
signed int can hold.  Performance would probably suck when using a value 
that high, but it would be acceptable.  Which smells like a bug in Solr.


I can tell you that using a value from uid in the rows parameter is 
almost certainly not the right thing to do.  That field would have no 
connection to rows.


Thanks,
Shawn

Re: Incorrect value sent to solr

[Shawn Heisey]

On 4/2/2021 7:11 AM, Łukasz Szczepański wrote:

On a Solr side I got an error:
java.lang.NumberFormatException: For input string: "2206267083"
NumberFormat expect an int which max value is 2147483647.

Quick test reviles that rows parameter isn't required for correct 
response from solr.
I think that dovecot should check if the value of rows is a correct int, 
and if not, just skip this argument.


That error comes from Solr.  You can fix it on the Solr side by changing 
the type on the field from int to long in the schema.


When I look at the provided example schema for fts_solr, I do not even 
see a definition for an "int" type, so if you have one, it's wrong.


https://raw.githubusercontent.com/dovecot/core/master/doc/solr-schema-7.7.0.xml

Thanks,
Shawn

Re: Sieve by the addressee

[Shawn Heisey]

On 2/20/2021 8:50 AM, Markus Schönhaber wrote:

I consider it a better idea to filter mailing list messages by their
List-ID header.


I agree with Markus.  It's what I do.  This works well:

if header :regex "list-id" "solr-user.lucene.apache.org"
{
fileinto "asf.solr-user";
stop;
}

I do not know if List-ID is common to all mailing list software, but 
even if it's not, there should be something available in the message 
headers that you can use.


Thanks,
Shawn

Re: v2.3.11.3 solr plugin search via MUA fails to match accented ascii characters; cmd line exec of `doveadm fts lookup` PANICs (assertion failed)

[Shawn Heisey]

On 10/18/2020 6:33 PM, PGNet Dev wrote:
is it that you're not using the dovecot plugin, but _DO_ have solr 
search setup?  by what method/mean?


or that you're avoiding solr usage altogether?


I am using dovecot for my mail service.  My host in AWS only has 2GB of 
memory ... I would expect an instance of Solr that can handle all my 
mail to require at least a 2GB heap, which leaves no memory for postfix, 
dovecot, the OS, or any other software.  Memory is one of the most 
expensive resources on AWS, so I don't really want to upgrade that.


I have a lot of experience with Solr in a commercial setting unrelated 
to email, though at the moment I am not using it for work.  I did once 
set up the fts-solr on a server at home that had a copy of the mail I 
have in production, but as already stated, my production system is in 
AWS and can't handle it.


Thanks,
Shawn

Re: v2.3.11.3 solr plugin search via MUA fails to match accented ascii characters; cmd line exec of `doveadm fts lookup` PANICs (assertion failed)

[Shawn Heisey]

On 10/11/2020 4:27 PM, PGNet Dev wrote:

I'm running,

dovecot --version
2.3.11.3 (502c39af9)

solr -version
8.6.3





Attempting to test/debug from from cmd line,

doveadm fts lookup -u myu...@example.com body "tambien"

causes a PANIC


I am a committer on the lucene-solr project.  So I know that product 
very well.  I am less confident about dovecot, but I do use it.  I do 
not use the fts-solr plugin, because my mail host in AWS does not have 
enough memory for that.


If you are using something like the following schema:

https://raw.githubusercontent.com/dovecot/core/master/doc/solr-schema-7.7.0.xml

That schema does not have anything that would fold accented characters. 
I do see "normalizer-icu" in your dovecot config ... if this filters 
messages before they get to Solr during indexing, then maybe the Solr 
config does not need to do the folding.


Solr does have a set of ICU filters, which I would recommend using 
rather than the lowercase filter, because they are aware of all of 
Unicode.  Those filters are not present in the main Solr distribution, 
but they are in the Solr binary package under "contrib".


I do not have a setup where I can test this.  If I did, I would have 
done that testing.


I cannot say much about the panic you're getting when using the doveadm 
command.  The stacktrace says it is happening in dovecot code, not Solr 
code.  And it looks like the panic had nothing to do with FTS or Solr 
... what I see points to mailbox storage code.


Thanks,
Shawn

Re: Trying to use solr

[Shawn Heisey]

On 7/26/2020 3:48 AM, r...@med-lo.eu wrote:

Thanks a lot Shawn. I will send the full error when I get home a week
from now. But this was my first attempt to use solr - is it still
subject to those conflicts?


You'll have to be more specific ... but I believe the answer to your
question is likely to be "yes.

I am the person who wrote the "HowToReindex" wiki page that I linked.
Nothing in that wiki page is addressed at any particular version of
Solr.  The possibility of a schema change requiring a complete reindex
(and sometimes deleting the index directory entirely) applies to ANY
version.

Thanks,
Shawn

Re: Trying to use solr

[Shawn Heisey]

On 7/22/2020 11:13 AM, Francis Augusto Medeiros-Logeay wrote:

On Solr I get this error:

org.apache.solr.common.SolrException: Exception writing document id 
210/9fd7941e8297d25d9160c3fdd3da/fran...@francisaugusto.com 
 
to the index; possible analysis error: cannot change field "box" from 
index options=DOCS_AND_FREQS_AND_POSITIONS to inconsistent index 
options=DOCS


Parallel to this, I got some log messages on Solr before attempting to 
reindex the user (sorry for the garbadged text:

Time (Local)Level   CoreLogger  Message


7/22/2020, 6:43:46 PM 	ERROR false 	x:dovecot 	RequestHandlerBase 
java.lang.IllegalStateException: Type mismatch: uid was indexed as 
SORTED_NUMERIC


These errors are a LOT longer than what we see here.  It appears that 
you are showing us what you can see in the logging tab of the admin UI 
... you will see far more detail if you find the solr.log file and get 
information from there.  Full detail can be seen on a single log entry 
in the admin UI by clicking on the little "i" icon, but it will close 
again *VERY* quickly, so that's not a very reliable option.


The full detail is usually required for deciphering log messages.

As for the source of the problem ... these look like the kind of errors 
that can be encountered if you complete some indexing with one schema, 
then change the schema without deleting the existing index, 
restart/reload, and try to index more data.  Most changes to the schema 
will require full reindexing, and some changes require actually deleting 
the index directory entirely and allowing it to be rebuilt.  In some 
rare cases, upgrading Solr *might* result in a change to the schema that 
requires deleting the index even though the schema config file itself is 
the same.  I do not know if your setup is one that would be affected in 
this way.


https://cwiki.apache.org/confluence/display/solr/HowToReindex

Your question is more appropriate for the solr-user mailing list at 
Apache than this list ... but I did not expect you to know that in 
advance.  I can try to help you.


Thanks,
Shawn

Re: solr and dovecot-2.2.36.4

[Shawn Heisey]

On 7/17/2020 3:23 AM, Maciej Milaszewski wrote:

I try schema.xml and solrconfig.xml from working solr-6.6.5 (dovecot)
"dovecot:
org.apache.solr.common.SolrException:org.apache.solr.common.SolrException:
Error initializing QueryElevationComponent"


That looks like a Solr error.  This list is not really the right place 
to get help on it ... but I have a lot of general experience with Solr, 
so I can try.


The error will be a LOT longer than this, probably dozens of lines that 
show a complete java stacktrace.  I will need that additional detail to 
make any determination about what went wrong.


You can generally find the additional detail in the solr.log file or one 
of its rotated cousins.  Exactly where this log file lives will vary 
depending on how you installed Solr.


If you are looking at the error on the "Logging" tab of the Solr admin 
UI, you can open the additional detail by clicking the little "i" icon 
on the log entry.  Be aware that it will close again VERY quickly -- so 
looking at the log file is usually a better option.


When I look at the solrconfig and schema available on the wiki page you 
linked, I do not find the word "elevation" in either file. 
QueryElevationComponent should not be necessary for a dovecot index.  I 
am guessing that your config contains some reference to it, and for a 
reason that I do not know at this time, initialization is failing.


Based on the limited information available, this would be my suggested 
course of action:  Wipe and rebuild the index using the solrconfig and 
schema found on the wiki page (which contain no mention of the elevation 
component), and then ask dovecot to force a full reindex.


It is *VERY* possible that if you provide additional information that my 
suggested course of action will change.


Thanks,
Shawn

Re: Dovecot - Upgrade Solr 7.7.2 to 8.4.1

[Shawn Heisey]

On 1/22/2020 4:03 PM, Domenico Pastore wrote:

So, with Dovecot is it possible to use Apache Solr 8.4?
High RAM usage is the only problem?


I don't know whether it's possible, but I imagine there is some way to 
make it work.  I don't see anything in the solrconfig or schema provided 
for fts_solr with Solr 7.7 that wouldn't work in 8.x.  The solrconfig 
does contain config that sets the default field to _text_, which is a 
nonexistent field in the schema.  If that were going to be a problem, it 
would affect all Solr versions.


As for the memory ... Solr is a Java program.  Java programs have a 
configurable limit on the amount of heap memory they can allocate, 
enforced by Java itself.  There is no difference between 7.x and 8.x in 
this.  Java is only going to use more memory if you tell it that it is 
allowed to do so.


Thanks,
Shawn

Re: solr fts and removing accounts

[Shawn Heisey]

On 1/19/2020 3:31 AM, Wojciech Puchar wrote:
i use solr fts indexing. It worls very well but it have one database per 
system, not per user.


Lets suppose i delete one or more e-mail users in system.

How to remove them in solr database to reclaim space?


I cannot say whether there is anything that you can do from the Dovecot 
side, but I can explain how to delete them from the Solr side if you 
like.  The end result would likely be the same either way.


I'm not sure that doing so is worthwhile.  Deleting data from a Solr 
index just marks it as deleted.  Until the index segments are merged, 
space will not be reclaimed.  And it is entirely possible that the 
segment in question might never get merged during normal operation.


It is possible to force the merging ... the operation is called 
"optimize" by Solr.  But it is a heavyweight operation, not one that 
should be done frequently, and in general the Solr project doesn't 
recommend doing it at all.


Deleting one user's data, even if that user has a large amount of email 
and the delete is followed by an optimize, is not likely to make much of 
a change in the size of the inverted index, just due to how it works. 
Looking over the schema for fts_solr, the large fields are not stored, 
so there will not be all that much stored data.  Stored data in the 
index is compressed as of Solr 4.1, which makes it even smaller.


Thanks,
Shawn

Re: fts_solr and # char

[Shawn Heisey via dovecot]

On 12/2/2019 12:34 PM, Marcio Merlone via dovecot wrote:
Dovecot wiki says the solr url is something like 'solr/dovecot/' but 
this gives a 404, if I install solr as per the docs I have a 
'solr/#/dovecot/' url:


Solr URLs with # in them are ONLY for use in a browser.  You cannot use 
them for API calls.  Specifically, they are used by the complex 
javascript application that is the Solr admin UI.


The reason is that all characters in a URL starting with # are NOT sent 
to the server, they are ONLY handled by the client.  Typically only a 
browser is capable of using those characters.


If the Solr core/collection is named "dovecot" then the example URL in 
the wiki is completely correct.  If the index has a different name, then 
you can replace the string "dovecot" with the correct name.


Trying the exact URL that you give to dovecot *will* give you a 404 
error.  When a Solr client (like fts_solr) uses that URL, it will add 
the necessary information for it to work correctly.


Thanks,
Shawn

Re: Problem Solr and centos 7

[Shawn Heisey via dovecot]

On 8/7/2019 4:23 AM, HTMLServices.it via dovecot wrote:

Thanks Shawn for your reply
I tried to bring the heap size to 5gb as you would like, but the problem 
was not solved.


That machine only has 4GB of total memory, so setting the heap to 5GB 
will eventually be problematic and lead to major performance issues.  If 
I were doing this with that hardware, I would set the heap to 1GB to 
start and maybe go as high as 2 GB.  With only 28000 documents, this 
*should* be enough.


"/If you have configured fts_solr with a URL that contains a # 
character, it's never going to work./"
I'm not sure how to configure this but in the 90-plugins.conf file I 
configured this:


plugin {
   #setting_name = value
   fts = solr
   fts_solr = url = http://5.39.2.59: 8987/solr/dovecot/
}


That URL looks good.  It doesn't have the # character, and from what I 
can see, should be completely correct.


Note that you should restrict access to the Solr server, not allow the 
Internet to reach it.  With it publicly accessible, anyone can delete 
the entire index or issue queries that cause denial of service.  I was 
able to connect to it, and I can see that your Solr index has no data in it:


https://www.dropbox.com/s/9bsx2vfu3kab19j/dovecot-solr-user-issue-screenshot.png?dl=0

I don't know if it's empty because nothing ever got indexed, or if it's 
empty because somebody found the URL for your server on this mailing 
list and decided to delete your index.


If I attempt a query on your index, it fails, complaining that there is 
no _text_ field:


http://5.39.2.59:8987/solr/dovecot/select?q=hello

The solrconfig.xml file that is in the dovecot source should be setting 
the df parameter to a field that exists, perhaps "body".  The field that 
it is using does not exist in the schema that is also in the dovecot source.


It might be that when fts_solr issues queries, it qualifies everything 
to search specific fields.  If that's what it does, then this 
configuration issue would not break fts_solr in the wild.


Thanks,
Shawn

Re: Problem Solr and centos 7

[Shawn Heisey via dovecot]

On 8/5/2019 12:02 PM, HTMLServices.it via dovecot wrote:
Given that I am not an expert, I am doing tests with Solr, I installed 
following the guide but I have no benefits on the search, the search on 
the body on 28000 mails takes a few minutes and then goes to timeout.


If the problems you're having are with Solr itself and not fts_solr, 
then the Solr mailing list or IRC channel is probably a better place to 
get help.


https://lucene.apache.org/solr/community.html#mailing-lists-irc

The following info, combined with your document count of 28000, will be 
very useful:


https://cwiki.apache.org/confluence/display/solr/SolrPerformanceProblems#SolrPerformanceProblems-Askingforhelponamemory/performanceissue

When gathering the screenshot, be sure that the process listing is 
sorted as described.


With no other info to go on, I suspect that maybe your Solr install is 
still configured with a 512MB heap and that the heap size needs to be 
increased to handle the index you've built.


I did several tests but I can't get it to work, this is the test server 
link: http://5.39.2.59:8987/solr/#/


If you have configured fts_solr with a URL that contains a # character, 
it's never going to work.  URLs containing # are only usable in a 
browser and will not function correctly anywhere else.


Thanks,
Shawn

Re: solr

[Shawn Heisey via dovecot]

On 7/10/2019 2:49 AM, Maciej Milaszewski IQ PL via dovecot wrote:

On the other hand solr replication is quite complicated process and
rollback or master-slave switch in this case is non-trivial task, that
may have result in whole dataset inconsistency.

Do you have any experience in such cases ? Maby load-balance in HAProxy
colud do the thing ? Something like:

.
server search1 192.168.1.1:8983 check port 8983 inter 20s fastinter 2
server search2 192.168.1.2:8983 backup
.


If you are using master-slave replication, you do not want to do this. 
Say the master went down, and it took you a while to get it back up. 
Unless you change the replication config, any data indexed on the slave 
would be gone as soon as the master was started back up, because the 
index would be replicated from the master.  You are quite correct that 
switching replication roles is non-trivial.


Using a load balancer in this manner is a great option if Solr is 
running in SolrCloud mode.  SolrCloud is a true cluster -- no masters, 
no slaves.  You can send indexing or queries to any system in the cloud 
and everything works.  The only real downside is that ZooKeeper (which 
is what turns Solr into SolrCloud) requires three servers for high 
availability, not two.  The third server could have much lower specs 
than the other two, as ZooKeeper's system requirements are typically 
quite modest.


Thanks,
Shawn

Re: What does Solr index do and how to handle its high avaliablity?

[Shawn Heisey via dovecot]

On 6/14/2019 2:11 AM, luckydog xf via dovecot wrote:

   And based on https://dovecot.org/pipermail/dovecot/2019-April/115575.html
I'm going to use an VIP to host 2 mail servers. Currently, it works in 
fail over and fail back test except solr index, so how to resolve this?


Solr has high availability options.  Probably the easiest is SolrCloud, 
because the legacy master-slave replication option basically has the 
master as a single point of failure.  Switching a slave to master is a 
very manual job.  SolrCloud is a true cluster -- no masters and no slaves.


Normally I would suggest a load balancer in front of Solr, but with 
SolrCloud and this usage, you could have a VIP switch between two hosts. 
 You will need at least three servers for a redundant SolrCloud 
install, because ZooKeeper absolutely requires three servers for 
redundancy.  SolrCloud is Solr+ZooKeeper.  Only two of the servers would 
need to run Solr, though.  The third server would only need to run 
ZooKeeper, and normally the system requirements for that are very low.



Is it possible to put dovecot index on shared storage like NFS?


Solr does not do well on network file systems.  They usually don't offer 
the file locking capability that Lucene wants.  Solr is written using 
the Lucene API.


And you can't have two running Solr servers using the same index 
directory even if you disable file locking so that NFS works.  Each 
server needs its own copy of the index.


How to rebuild if I don't put index data on shared storage in case of 
fail over?


Building and rebuilding is accomplished by dovecot.  Here's a wiki page 
for rebuilding Solr indexes.  Admittedly it falls into the "not all that 
helpful" category:


https://wiki.apache.org/solr/HowToReindex

Thanks,
Shawn

Re: IMAPSieve message marked as HAM should go through full sieve script of user

[Shawn Heisey via dovecot]

On 5/10/2019 2:34 AM, Matthias Fechner via dovecot wrote:

The problem I have now is:
A message was by accident marked by rspamd as spam, so a false-positive 
and is moved to the Junk folder.
If I click in my email program now on `Not Spam` it is learn as HAM but 
moved to the INBOX.
I would like that the email is not moved to INBOX but it should be moved 
through the sieve script of the user again, as it is a new email.


Chances are that it is your mail client doing the move -- directly, not 
as a message delivery.  If your connection is IMAP, then the move would 
be done through that protocol.


To get the message to go through your sieve script again would require 
delivering it as a new message to Dovecot's LDA, and that's probably not 
something your mail client can do.


Thanks,
Shawn

Re: SolrCore 'dovecot' is not available due to init failure: fieldType 'text_general' not found in the schema

[Shawn Heisey via dovecot]

On 4/22/2019 9:31 PM, luckydog xf via dovecot wrote:

https://wiki.dovecot.org/Plugins/FTS/Solr
Only offers a solr-7.7.0 solrconfig.xml, does it apply to solr-8.0.0?


If the config was actually designed for 7.7, then it will work in 8.0. 
If it was designed for 5.x or 6.x, there's a chance it won't work.


It sounds like you have a solrconfig.xml file and a schema file that are 
not matched to each other.


The schema file found here looks to me like it will work in 8.0:

https://raw.githubusercontent.com/dovecot/core/master/doc/solr-schema-7.7.0.xml

I think instead of using a solrconfig.xml that comes with Solr, you 
should be using the one made for dovecot:


https://raw.githubusercontent.com/dovecot/core/master/doc/solr-config-7.7.0.xml

I do see a potential problem with that config, though ... there is 
config that sets the df parameter (default field) to a field named 
"_text_" (with underscores) ... this field does not exist in the schema.


Thanks,
Shawn

Re: dovecot Digest, Vol 192, Issue 52

[Shawn Heisey via dovecot]

On 4/14/2019 8:59 AM, Peter Mogensen via dovecot wrote:

I run with

SOLR_JAVA_MEM="-Xmx8g -Xms2g"


Without other details, I cannot even offer a guess as to whether an 8GB 
heap is enough.  How many documents are in all the indexes that Solr 
instance is handling?  Can you share your solrconfig.xml file?



Being able to configure it is great.
But I don't think it solves much. I recompiled with 100 as batch size
and it still ended in timeouts.
Then I recompiled with 10min timeout and now I see all the batches
completing and their processesing time is mostly between 1 and 2 minutes
(so all would have failed).


1 to 2 minutes for 100 documents to index would be an indication that 
something is very wrong with that Solr instance.  It seems to me that 
the batch should take less than a second if there is no commit.


Thanks,
Shawn