Re: Strategy for fts

[Joan Moreau]

I updated fts-xapian to make it compatible with dovecot 2.2

On 2020-02-04 12:37, Peter Chiochetti wrote:

Am 04.02.20 um 11:46 schrieb Francis Augusto Medeiros-Logeay: 


Hi Philon,

Thanks a lot for your thoughts!

Can I ask you if using Solr improved things for you? I have a mailbox with 15 
years of e-mail and searching things take a long time.


Here, SOLR itself searches a quarter million mails in split seconds and returns 
very good results. That is on a low memory average machine.

If you dont mind the standard, you can change the schema, so headers (from, to) 
get indexed in body text. That can help narrowing results.

Only problem is search through e.g. nested folders from IMAP: something like 
ESEARCH would be nice - https://tools.ietf.org/html/rfc6237

Peter

On 04.02.2020 09:39, Philon wrote: Hi Francis,

next to fts-solr there was fts-lucene. But that Lucene there seems
heavily outdated why the Dovecot docs also suggest using Solr.
Elasticsearch probably is similar to Solr but the later is maintained
by Dovecot team.

I started with downloading the Solr binary distribution to Debian with
JRE preinstalled and things were running like after 10 min. Yes it's a
bit more complicated to find the schema and edit things like header
size (in tips section). It's running quite nicely since then and has
zero maintenance. 
I will try again - I kept getting some weird errors, so I don't know if that's why I wasn't seing much of improvement.


As FTS indexes are separate in external Solr instance I'd guess that
it won't interfere with dsync. What I don't know is if dsync'ing would
trigger indexing. This brings me to wonder how one could actually
replicate the Solr instance!? 
Good question. But what I thought about doing was to install FTS on my backup instance, and if things go fine, then I install an FTS instance on my production server - that is, if one doesn't interfere with the other.


I will give Solr another shot - my worries are mostly if Solr is supported on 
ARM (my prod instance is running on ARM) - I know Elasticsearch has an ARM 
build.

Ii thought about the Xapian engine, but since it requires dovecot 2.3, I will 
have to wait.

Best,

Francis

Philon

On 31 Jan 2020, at 17:24, Francis Augusto Medeiros-Logeay  
wrote:

Hi there,

I got successfully to replicate my mail server to another dovecot install using 
dsync, mainly for redundancy, and it works great.

I want to try to install fts, as some of the mailboxes have tens of thousands 
of messages, and it takes minutes to get some results when searching via IMAP 
on a Roundcube interface.

I want to experiment with fts-solr first, and firstly on my redundant server, 
ie., not on my main dovecot install. Is it ok to do this? I ask because I am 
afraid of how this whole reindexing on the redundant install will affect the 
production server.

Also, any tips on something else than fts-solr? I tried it once, but it was so 
hard to get it right, so many configurations, java, etc., that I'd rather try 
something else. I also could try fts-elastic or something like that, but, 
again, having to maintain an elasticsearch install might use more resources 
than I think is worth. Any thoughts on that?

Best,

-- Francis

Re: shared folder name or public folder with mail address

[Armin Schindler]

On 15.02.2020 14:45, Tobias Kirchhofer wrote:
> On 15 Feb 2020, at 12:27, Armin Schindler wrote:
> 
> On 15.02.2020 12:16, Tobias Kirchhofer wrote:
> ...
> 
> the sieve part seems to be very tricky.
> My test is now working with a public folder and I have a
> "sieve_before" script in place which works for rules of the
> private namespace.
> 
> But when the rule shall "fileinto" a public folder, the mail isn't
> filtered (moved) and syslog says
> dovecot: lmtp(.): Error ... sieve...: Mailbox doesn't exist.
> 
> After some tests I found out it always looks in the private (or
> shared)
> namespace for that specfied folder.
> Do I miss a config option to let dovecot sieve "fileinto" a public
> namespace folder?
> I use dovecot version 2.2.13.
> 
> We „fileinto“ in this manner :
> 
> |fileinto „Namespace/folder“; |
> 
> This is what is not working for me.
> 
> Your procedure below is good and maybe I will use parts of it, thank
> you. But this part is working for already. The incoming mail
> is sent to dovecot (postfix virtual alias working) and the
> match in sieve script is correct too.
> The problem is that dovecot (sieve) refuses to
> fileinto "Namespace/folder"
> because it seems it wants to store the mail
> in private namespace with "folder1/folder2" instead.
> The first part of the fileinto path isn't used as
> namespace name.
> Is there a config option for this behaviour?
...

> Are you sure the ACL are correct? Can be that the error message „Mailbox
> doesn't exist“ is misleading and the permissions are not correct.

You are correct. The message is misleading. And even worse, if I used
  fileinto :create ""
the folder was created in private namespace then.

The shared and public folders need of course the
 anyone post
right to let fileinto work.

Now it works. Thank you Tobias.

Armin



smime.p7s
Description: S/MIME Cryptographic Signature

Re: shared folder name or public folder with mail address

[Tobias Kirchhofer]

On 15 Feb 2020, at 12:27, Armin Schindler wrote:


On 15.02.2020 12:16, Tobias Kirchhofer wrote:
...

the sieve part seems to be very tricky.
My test is now working with a public folder and I have a
"sieve_before" script in place which works for rules of the
private namespace.

But when the rule shall "fileinto" a public folder, the mail 
isn't

filtered (moved) and syslog says
dovecot: lmtp(.): Error ... sieve...: Mailbox doesn't exist.

After some tests I found out it always looks in the private (or 
shared)

namespace for that specfied folder.
Do I miss a config option to let dovecot sieve "fileinto" a 
public

namespace folder?
I use dovecot version 2.2.13.

We „fileinto“ in this manner :

|fileinto „Namespace/folder“; |


This is what is not working for me.

Your procedure below is good and maybe I will use parts of it, thank
you. But this part is working for already. The incoming mail
is sent to dovecot (postfix virtual alias working) and the
match in sieve script is correct too.
The problem is that dovecot (sieve) refuses to
 fileinto "Namespace/folder"
because it seems it wants to store the mail
in private namespace with "folder1/folder2" instead.
The first part of the fileinto path isn't used as
namespace name.
Is there a config option for this behaviour?


Maybe you are missing an include?

We have:

```
require "fileinto";
require "mailbox";
require "envelope";
require "subaddress";
require "variables";
require "regex";
require "editheader";
require "include";

```

In dovecot.conf* i found no directly related settings. Just the usual 
sieve settings explained here 
https://wiki.dovecot.org/Pigeonhole/Sieve/Configuration


Can you see the Public namespace and the folder in your MUA or via 
telnet?


Are you sure the ACL are correct? Can be that the error message 
„Mailbox doesn't exist“ is misleading and the permissions are not 
correct.


Here you can see how to debug if not already 
https://wiki.dovecot.org/Pigeonhole/Sieve/Configuration#Trace_Debugging





Armin


The procedure in our setup is like this:

 1. Virtual alias for each public-Folder email address within 
postfix:


Pattern:
fol...@domain.tld 
listadm+Namespace/fol...@domain.tld 
Example:
i...@domain.tld 
listadm+Verteiler/i...@domain.tld 

Verteiler is the namespace, info the mailbox. listadm is a pseudo 
user
in this context which holds the ACL for fileinto the resulting 
mailbox

with proper permissions.

 2. Disassemble the virtual alias address with sieve:

Excerpt of the script. It is actually more code because we sort out 
spam

for public folder. German comments, sorry.

|# Hat Subadresse? if envelope :matches :detail "to" "*" { #
Adressaufbau: listadm+Namespace/liste@ # ${liste} wird Namespace/box 
set

"liste" "${1}"; } […] fileinto "${liste}"; |

I can send you the whole script if you need.



--
collect@shift.agency

Re: .IMAP

[Jos Chrispijn]

On 15-2-20 11:55, Thomas Zajic wrote:


That's probably because the second part of Aki's advise hasn't been followed yet
("... and ensure the mails are under there, ..."). You need to physically move 
all
mail related files and folders to the ~/mail subdir of each user. The "Deleted"
mailbox probably still shows up because your MUA has been configured to use a 
local
folder for it instead of an IMAP folder.


Just found out why it didn't work:

>> mail_location = mbox:/home/%u:INBOX=/var/mail/%u

should read

>> mail_location = mbox:/usr/home/%u:INBOX=/var/mail/%u

Thanks guys for you input; really appreciate your support!

Best, Jos

Re: Dovecot 2.3 repo for CentOS 8.

[Tobias Kirchhofer]

On 14 Feb 2020, at 10:59, Peter wrote:


On 14/02/20 10:10 pm, Tobias Kirchhofer wrote:
would it be useful/advisable to use this repo for productive 
operation?


An official repo is still not available for CentOS 8 
https://repo.dovecot.org/


We would like to set up our new mail server on CentOS 8 and are 
waiting… :)


What is your advice?


As others have mentioned there are missing -devel packages in CentOS 
8, which is because there are missing -devel packages in RHEL 8, most 
notably in this case is quota-devel.  tcp wrappers is also missing but 
that's because they are deprecated in CentOS 8 so building without tcp 
wrapper support is not a big deal, but I would assume that a lot of 
people will want quota support, so I wouldn't want to build a dovecot 
package without it.


Personally I'm trying to get the GhettoForge build system modified so 
it can get missing -devel packages by rebuilding the source rpms for 
them, this is a work in progress.  There is also work on the CentOS 
side to build and offer up the missing -devel packages.  At the end of 
the day nobody will be able to build decent dovecot packages until one 
of these things happens.


For now you can use the stock dovecot 2.2.36 that comes with CentOS or 
you can wait.  CentOS and Red Hat have not made this easy so it is 
going to take time.


Okay, will try 2.2.36. Our protoype is on Debian 10 with the latest 
Dovecot. Don’t know yet if we rely on features from the latest 
version. Will see. Thank you.


Also on a personal note, I think that pushing out any production 
server on CentOS 8 at this time is premature.  CentOS 8 simply is not 
ready yet, imo.


We have CentOS 8.1 VMs productive with either of nginx, PostgreSQL, 
MariaDB, Node.js. No problems so far. Cross your fingers! :-)




Peter



--
collect@shift.agency

Re: shared folder name or public folder with mail address

[Armin Schindler]

On 15.02.2020 12:16, Tobias Kirchhofer wrote:
...
> the sieve part seems to be very tricky.
> My test is now working with a public folder and I have a
> "sieve_before" script in place which works for rules of the
> private namespace.
> 
> But when the rule shall "fileinto" a public folder, the mail isn't
> filtered (moved) and syslog says
> dovecot: lmtp(.): Error ... sieve...: Mailbox doesn't exist.
> 
> After some tests I found out it always looks in the private (or shared)
> namespace for that specfied folder.
> Do I miss a config option to let dovecot sieve "fileinto" a public
> namespace folder?
> I use dovecot version 2.2.13.
> 
> We „fileinto“ in this manner :
> 
> |fileinto „Namespace/folder“; |

This is what is not working for me.

Your procedure below is good and maybe I will use parts of it, thank
you. But this part is working for already. The incoming mail
is sent to dovecot (postfix virtual alias working) and the
match in sieve script is correct too.
The problem is that dovecot (sieve) refuses to
 fileinto "Namespace/folder"
because it seems it wants to store the mail
in private namespace with "folder1/folder2" instead.
The first part of the fileinto path isn't used as
namespace name.
Is there a config option for this behaviour?

Armin

> The procedure in our setup is like this:
> 
>  1. Virtual alias for each public-Folder email address within postfix:
> 
> Pattern:
> fol...@domain.tld 
> listadm+Namespace/fol...@domain.tld 
> Example:
> i...@domain.tld 
> listadm+Verteiler/i...@domain.tld 
> 
> Verteiler is the namespace, info the mailbox. listadm is a pseudo user
> in this context which holds the ACL for fileinto the resulting mailbox
> with proper permissions.
> 
>  2. Disassemble the virtual alias address with sieve:
> 
> Excerpt of the script. It is actually more code because we sort out spam
> for public folder. German comments, sorry.
> 
> |# Hat Subadresse? if envelope :matches :detail "to" "*" { #
> Adressaufbau: listadm+Namespace/liste@ # ${liste} wird Namespace/box set
> "liste" "${1}"; } […] fileinto "${liste}"; |
> 
> I can send you the whole script if you need.



smime.p7s
Description: S/MIME Cryptographic Signature

Re: shared folder name or public folder with mail address

[Tobias Kirchhofer]

On 15 Feb 2020, at 10:43, Armin Schindler wrote:


On 14.02.2020 11:44, Tobias Kirchhofer wrote:

On 14 Feb 2020, at 11:06, Armin Schindler wrote:

...


The sorting in the according public folder happens in a global sieve
script before user scripts are fireing. It is combined with the 
‚+‘
notation in postfix for sub addresses and an special ‚listadm‘ 
user


postfix:
|
listadm+Verteiler/info@$DOMAIN
|

The sieve script is then reading the generated address and delivers 
it
into the according public folder. We learned a lot about sieve 
scripting

with this task :-)

|sieve_before = /var/vmail/sieve/global/global-before.sieve […] |

More details would be to much here.

To be honest, we needed some time to get it done properly. It is a
combination of features and A LOT OF testing and analysing the 
behaviour

of postfix/dovecot/public-folder/sieve/acl.


the sieve part seems to be very tricky.
My test is now working with a public folder and I have a
"sieve_before" script in place which works for rules of the
private namespace.

But when the rule shall "fileinto" a public folder, the mail isn't
filtered (moved) and syslog says
dovecot: lmtp(.): Error ... sieve...: Mailbox doesn't exist.

After some tests I found out it always looks in the private (or 
shared)

namespace for that specfied folder.
Do I miss a config option to let dovecot sieve "fileinto" a public
namespace folder?
I use dovecot version 2.2.13.


We „fileinto“ in this manner :

```
fileinto „Namespace/folder“;
```

The procedure in our setup is like this:

1. Virtual alias for each public-Folder email address within postfix:

Pattern:
fol...@domain.tld listadm+Namespace/fol...@domain.tld
Example:
i...@domain.tld listadm+Verteiler/i...@domain.tld

Verteiler is the namespace, info the mailbox. listadm is a pseudo user 
in this context which holds the ACL for fileinto the resulting mailbox 
with proper permissions.


2. Disassemble the virtual alias address with sieve:

Excerpt of the script. It is actually more code because we sort out spam 
for public folder. German comments, sorry.


```
# Hat Subadresse?
if envelope :matches :detail "to" "*"
{
# Adressaufbau: listadm+Namespace/liste@
# ${liste} wird Namespace/box
set "liste" "${1}";
}
[…]

fileinto "${liste}";

```

I can send you the whole script if you need.

Greetings!

Tobias

--
collect@shift.agency

Re: shared folder name or public folder with mail address

[Tobias Kirchhofer]

On 14 Feb 2020, at 15:02, Armin Schindler wrote:


On 2/14/20 2:23 PM, Tobias Kirchhofer wrote:

On 14 Feb 2020, at 14:05, Armin Schindler wrote:


On 2/14/20 11:44 AM, Tobias Kirchhofer wrote:

On 14 Feb 2020, at 11:06, Armin Schindler wrote:

    Hello!


[…]

For our new setup (planned with CentOS 8, waiting for the official 
repo for the latest dovecot
version…) we have a prototype on Debian base with the latest 
version of Dovecot. In opposite to our
current solution we will implement global acls with user and groups 
from LDAP. This is very nice.


Wow, thanks. This seem to be exactly what I need to go on.
Our base system is a Univention UCS, so LDAP users and groups is 
what we already have working :-)


With your elements I think we really could have the cyrus shared 
folders with dovecot by

using a namespace of
 type=public
but with
 prefix=shared/
(we might use type=shared with prefix=usershared/ later).


There is not something like one way do to this. You could also make 
your life easier and choose to
create a „Share user“ which is only there to share folders. This 
will then be Dovecot „Shared
folders“ shared by this pseudo user to other users. You then could 
use Roundcube or such to maintain
acl with this pseudo user. But we did not follow this path then for 
reasons i cannot remember

anymore :)

For each Public folder we have a defined namespace. „Newsletter“, 
„Client1“, „Client2“ etc. We have
many, around 30 or so. ACL goes on this namespace (namespace/*). I 
guess ACL are first match. Each
namespace has its own prefix. (eq. prefix = Newsletter/, prefix = 
Client1 etc.) User can subscribe

namespace by namespace acl assumed.


What is the benefit using own namespace for each public folder?
If the config is equal for all folders, wouldn't one namespace with 
subfolders

be the same?
One public namespace would look like it is with cyrus in the client: 
one "shared/"

folder to open/close.


Yes, this was a misunderstanding - we have a lot of subfolders per 
namespace. The subfolders have ACLs.



--
collect@shift.agency

Re: .IMAP

[Thomas Zajic]

* Jos Chrispijn, 14.02.20 14:47

> On 14-2-20 13:39, Aki Tuomi wrote:
> 
>> This is why you put mail_location=driver:~/Mail and ensure the mails are 
>> under there, instead of mail_location=driver:~/
> 
> Yes, that is what I thought; when I use that setting, I get this error:
> [...]

Of course you shouldn't put "driver" there literally, but replace it with the 
actual
mailbox type (ie. "mbox", "maildir", "dbox", ...). The error message below 
contains
a hint to the problem, but admittedly it's easy to miss:

> Feb 14 14:32:15  dovecot[8549]: imap(jos)<8739><5ErUO4meZthSsH9H>: 
> Initializing mail storage from mail_location setting failed:
> Unknown mail storage driver driver in=0 out=375 deleted=0 expunged=0 
> trashed=0 hdr_count=0 hdr_bytes=0 body_count=0 body_bytes=0
  ^^
It would probably be easier to find if the actual driver name would be put in 
quotes
in the logging line (Unknown mail storage driver "driver" in=0 ...).

> [...]
> When I change
> 
> mail_location = mbox:/home/%u:INBOX=/var/mail/%u
> 
> into
> 
> mail_location = mbox:/home/%u/mail:INBOX=/var/mail/%u
> 
> I only get inbox (the /var/mail/%u content) and the Deleted mailbox.

That's probably because the second part of Aki's advise hasn't been followed yet
("... and ensure the mails are under there, ..."). You need to physically move 
all
mail related files and folders to the ~/mail subdir of each user. The "Deleted"
mailbox probably still shows up because your MUA has been configured to use a 
local
folder for it instead of an IMAP folder.

HTH,
Thomas

Re: shared folder name or public folder with mail address

[Armin Schindler]

On 14.02.2020 11:44, Tobias Kirchhofer wrote:
> On 14 Feb 2020, at 11:06, Armin Schindler wrote:
...
> 
> The sorting in the according public folder happens in a global sieve
> script before user scripts are fireing. It is combined with the ‚+‘
> notation in postfix for sub addresses and an special ‚listadm‘ user
> 
> postfix:
> |
> listadm+Verteiler/info@$DOMAIN
> |
> 
> The sieve script is then reading the generated address and delivers it
> into the according public folder. We learned a lot about sieve scripting
> with this task :-)
> 
> |sieve_before = /var/vmail/sieve/global/global-before.sieve […] |
> 
> More details would be to much here.
> 
> To be honest, we needed some time to get it done properly. It is a
> combination of features and A LOT OF testing and analysing the behaviour
> of postfix/dovecot/public-folder/sieve/acl.

the sieve part seems to be very tricky.
My test is now working with a public folder and I have a
"sieve_before" script in place which works for rules of the
private namespace.

But when the rule shall "fileinto" a public folder, the mail isn't
filtered (moved) and syslog says
dovecot: lmtp(.): Error ... sieve...: Mailbox doesn't exist.

After some tests I found out it always looks in the private (or shared)
namespace for that specfied folder.
Do I miss a config option to let dovecot sieve "fileinto" a public
namespace folder?
I use dovecot version 2.2.13.

Armin



smime.p7s
Description: S/MIME Cryptographic Signature

Re: shared folder name or public folder with mail address

[Armin Schindler]

Hello Chris,

On 15.02.2020 09:45, chris2...@postbox.xyz wrote:
> 
>> Armin Schindler  hat am 14. Februar 2020 15:02 
>> geschrieben:
> 
>> One public namespace would look like it is with cyrus in the client: one 
>> "shared/"
>> folder to open/close.
> 
> That's right, but there is one important difference: Cyrus sorts those 
> folders alphabetically, Dovecot doesn't. Dovecot guys say it's the client's 
> job and that's also what the RFC says. So if you have many subfolders, e.g. 
> one per order it can be a bit confusing. Thunderbird is sorting those 
> folders, but not always. It seems to depend on when you refreshed the list 
> last time and if it was cached.

thanks for the hint.
I will check this in our environment.

Armin



smime.p7s
Description: S/MIME Cryptographic Signature

Re: shared folder name or public folder with mail address

[chris2014]

> Armin Schindler  hat am 14. Februar 2020 15:02 
> geschrieben:

> One public namespace would look like it is with cyrus in the client: one 
> "shared/"
> folder to open/close.

That's right, but there is one important difference: Cyrus sorts those folders 
alphabetically, Dovecot doesn't. Dovecot guys say it's the client's job and 
that's also what the RFC says. So if you have many subfolders, e.g. one per 
order it can be a bit confusing. Thunderbird is sorting those folders, but not 
always. It seems to depend on when you refreshed the list last time and if it 
was cached.

- Chris