Re: Changing the default delivery mailbox to something other than INBOX

[Aki Tuomi]

> On 18/01/2021 08:56 Ron Garret  wrote:
> 
>  
> On Jan 17, 2021, at 10:48 PM, Aki Tuomi  wrote:
> 
> > Don't touch the INBOX setting (leave it out), use the -m parameter for 
> > dovecot-lda.
> 
> I don’t think that will work.  My MTA is postfix and it’s connected to 
> dovecot via LMTP, so AFAIK postfix is talking to dovecot over a socket, and 
> dovecot-lda isn’t being invoked so there is no place to pass that the -m 
> parameter.  But we are at the limits of my understanding of how all this 
> stuff actually works under the hood so please correct me if I’m wrong here.
> 
> rg

Ah, you are using LMTP... then the easiest is to use Sieve here. 

You can use simple global script, e.g. /etc/dovecot/default-mbox.sieve

```
require ["fileinto", "mailbox"];
fileinto "OtherBox";
```

then sievec the script, and use

plugin {
  sieve_before = /etc/dovecot/default-mbox.sieve
}

this way it will change the default mailbox, but allows per-user scripts to 
change it.

Aki

Re: Changing the default delivery mailbox to something other than INBOX

[Ron Garret]

On Jan 17, 2021, at 10:48 PM, Aki Tuomi  wrote:

> Don't touch the INBOX setting (leave it out), use the -m parameter for 
> dovecot-lda.

I don’t think that will work.  My MTA is postfix and it’s connected to dovecot 
via LMTP, so AFAIK postfix is talking to dovecot over a socket, and dovecot-lda 
isn’t being invoked so there is no place to pass that the -m parameter.  But we 
are at the limits of my understanding of how all this stuff actually works 
under the hood so please correct me if I’m wrong here.

rg

Re: Changing the default delivery mailbox to something other than INBOX

[Aki Tuomi]

> On 18/01/2021 08:46 Ron Garret  wrote:
> 
>  
> I’m looking at the docs here:
> 
> https://doc.dovecot.org/configuration_manual/mail_location/
> 
> There I see:
> 
> > Mailbox autocreation
> > 
> > Dovecot in the 1.x era created mailboxes automatically regardless of 
> > whether mail_location was set. In 2.x autocreation only gets triggered if 
> > mail_location is correctly set. You’ll see something like this if you 
> > enable debug logging:
> > 
> > …
> 
> I’m pretty sure that autocreation is working because the mailbox is in fact 
> being created.  And AFAICT it is being created in the right place with the 
> right permissions.  Despite this, mail delivery is failing when I use INBOX=…
> 
> All of the messages disappearing from my client is also mighty hinky.  The 
> only way I can explain that is that dovecot is telling my client that my 
> inbox is no longer named Inobx.  If that is the case then this whole approach 
> won’t work because that will defeat the purpose.  The whole point of this 
> exercise is to get the dovecot LDA to put mail in something OTHER than what 
> the client thinks is the main inbox.
> 

Don't touch the INBOX setting (leave it out), use the -m parameter for 
dovecot-lda.

Aki

> On Jan 17, 2021, at 10:04 PM, Aki Tuomi  wrote:
> 
> > I don't see how that would the obvious way, and that, as you found out, 
> > does cause your mails to disappear.
> > 
> > Looking at `man dovecot-lda` you'll find
> > 
> > -m mailbox
> > 
> > Destination  mailbox (default is INBOX). If the mailbox doesn't exist, it 
> > will not be created (unless the lda_mailbox_autocreate setting is set to 
> > yes). If a message couldn't be saved to the  mailbox for any reason, it's 
> > delivered to INBOX instead.
> > 
> > Aki
> > 
> >> On 18/01/2021 04:42 Ron Garret  wrote:
> >> 
> >> 
> >> I tried the obvious:
> >> 
> >> mail_location = maildir:/var/mail/vhosts/%d/%n/mail:INBOX=mail/.Incoming
> >> 
> >> and that failed in an even more bizarre way.  The Incoming mailbox was 
> >> created, it showed up in my mail client as it should have, but mail 
> >> delivery still failed.  Not only that, but all the messages that were in 
> >> my Inbox disappeared from my mail client.  I reverted the change and all 
> >> of the messages that had previously been in Inbox reappeared.
> >> 
> >> WTF?
> >> 
> >> On Jan 17, 2021, at 5:00 PM, Ron Garret  wrote:
> >> 
> >>> I groveled around in the docs and discovered the INBOX=… option to the 
> >>> mail_location config parameter.  I tried that, and it didn’t work, but it 
> >>> failed in a very strange way.
> >>> 
> >>> I currently have:
> >>> 
> >>> mail_location = maildir:/var/mail/vhosts/%d/%n/mail
> >>> 
> >>> I tried changing that to:
> >>> 
> >>> mail_location = maildir:/var/mail/vhosts/%d/%n/mail:INBOX=incoming
> >>> 
> >>> That failed with the following log message:
> >>> 
> >>> Jan 17 23:41:17 lmtp(s...@rngh.net)<32476>: 
> >>> Error: Mailbox INBOX: Failed to autocreate mailbox: Permission denied
> >>> 
> >>> But the weird thing is that it DID create a directory called incoming, 
> >>> and that directory has the same permissions as the rest of the mailbox 
> >>> hierarchy.
> >>> 
> >>> There is something else which I was not expecting, and that is that the 
> >>> directory is a peer to /var/mail/vhosts/%d/%n/mail.  What I want is to 
> >>> create a folder *inside* /var/mail/vhosts/%d/%n/mail.
> >>> 
> >>> Advice on how to proceed would be much appreciated.  (BTW, I’d be happy 
> >>> to pay someone a consulting fee for help with this project.  If you’re 
> >>> interested contact me off-list.)
> >>> 
> >>> rg
> >>> 
> >>> On Jan 17, 2021, at 12:08 PM, Ron Garret  wrote:
> >>> 
>  Is there an easy way (i.e. a built-in configuration setting) to change 
>  the name of the mailbox that the dovecot LDA delivers mail into?  The 
>  default is INBOX but I’d like mail to be delivered to some other 
>  mailbox.  The reason for this is that I want all incoming mail to be 
>  invisible to the user by default until it has been screened for viruses 
>  and spam.  I know I could do this with Sieve, but that is a PITA.
>  
>  More details in case anyone is interested:
>  
>  The goal of this filter is to make it work with very little training.  
>  To bootstrap the process, the filter is given access to outgoing mail 
>  (via a milter) which it uses as a reliable training corpus for good 
>  messages.  It then leverages that information to filter incoming 
>  messages.  For example, messages from senders which have been the 
>  recipients of outgoing messages are presumed to be good.  There is also 
>  a spam honeypot to provide a reliable spam corpus.
>  
>  One of the heuristics I want to use is to look for the same subject line 
>  in multiple messages from unknown users received over a period of an 
>  hour or so because those are almost invariably spam.  But that requires 
>  a time delay betwe

Re: Changing the default delivery mailbox to something other than INBOX

[Ron Garret]

I’m looking at the docs here:

https://doc.dovecot.org/configuration_manual/mail_location/

There I see:

> Mailbox autocreation
> 
> Dovecot in the 1.x era created mailboxes automatically regardless of whether 
> mail_location was set. In 2.x autocreation only gets triggered if 
> mail_location is correctly set. You’ll see something like this if you enable 
> debug logging:
> 
> …

I’m pretty sure that autocreation is working because the mailbox is in fact 
being created.  And AFAICT it is being created in the right place with the 
right permissions.  Despite this, mail delivery is failing when I use INBOX=…

All of the messages disappearing from my client is also mighty hinky.  The only 
way I can explain that is that dovecot is telling my client that my inbox is no 
longer named Inobx.  If that is the case then this whole approach won’t work 
because that will defeat the purpose.  The whole point of this exercise is to 
get the dovecot LDA to put mail in something OTHER than what the client thinks 
is the main inbox.

On Jan 17, 2021, at 10:04 PM, Aki Tuomi  wrote:

> I don't see how that would the obvious way, and that, as you found out, does 
> cause your mails to disappear.
> 
> Looking at `man dovecot-lda` you'll find
> 
> -m mailbox
> 
> Destination  mailbox (default is INBOX). If the mailbox doesn't exist, it 
> will not be created (unless the lda_mailbox_autocreate setting is set to 
> yes). If a message couldn't be saved to the  mailbox for any reason, it's 
> delivered to INBOX instead.
> 
> Aki
> 
>> On 18/01/2021 04:42 Ron Garret  wrote:
>> 
>> 
>> I tried the obvious:
>> 
>> mail_location = maildir:/var/mail/vhosts/%d/%n/mail:INBOX=mail/.Incoming
>> 
>> and that failed in an even more bizarre way.  The Incoming mailbox was 
>> created, it showed up in my mail client as it should have, but mail delivery 
>> still failed.  Not only that, but all the messages that were in my Inbox 
>> disappeared from my mail client.  I reverted the change and all of the 
>> messages that had previously been in Inbox reappeared.
>> 
>> WTF?
>> 
>> On Jan 17, 2021, at 5:00 PM, Ron Garret  wrote:
>> 
>>> I groveled around in the docs and discovered the INBOX=… option to the 
>>> mail_location config parameter.  I tried that, and it didn’t work, but it 
>>> failed in a very strange way.
>>> 
>>> I currently have:
>>> 
>>> mail_location = maildir:/var/mail/vhosts/%d/%n/mail
>>> 
>>> I tried changing that to:
>>> 
>>> mail_location = maildir:/var/mail/vhosts/%d/%n/mail:INBOX=incoming
>>> 
>>> That failed with the following log message:
>>> 
>>> Jan 17 23:41:17 lmtp(s...@rngh.net)<32476>: Error: 
>>> Mailbox INBOX: Failed to autocreate mailbox: Permission denied
>>> 
>>> But the weird thing is that it DID create a directory called incoming, and 
>>> that directory has the same permissions as the rest of the mailbox 
>>> hierarchy.
>>> 
>>> There is something else which I was not expecting, and that is that the 
>>> directory is a peer to /var/mail/vhosts/%d/%n/mail.  What I want is to 
>>> create a folder *inside* /var/mail/vhosts/%d/%n/mail.
>>> 
>>> Advice on how to proceed would be much appreciated.  (BTW, I’d be happy to 
>>> pay someone a consulting fee for help with this project.  If you’re 
>>> interested contact me off-list.)
>>> 
>>> rg
>>> 
>>> On Jan 17, 2021, at 12:08 PM, Ron Garret  wrote:
>>> 
 Is there an easy way (i.e. a built-in configuration setting) to change the 
 name of the mailbox that the dovecot LDA delivers mail into?  The default 
 is INBOX but I’d like mail to be delivered to some other mailbox.  The 
 reason for this is that I want all incoming mail to be invisible to the 
 user by default until it has been screened for viruses and spam.  I know I 
 could do this with Sieve, but that is a PITA.
 
 More details in case anyone is interested:
 
 The goal of this filter is to make it work with very little training.  To 
 bootstrap the process, the filter is given access to outgoing mail (via a 
 milter) which it uses as a reliable training corpus for good messages.  It 
 then leverages that information to filter incoming messages.  For example, 
 messages from senders which have been the recipients of outgoing messages 
 are presumed to be good.  There is also a spam honeypot to provide a 
 reliable spam corpus.
 
 One of the heuristics I want to use is to look for the same subject line 
 in multiple messages from unknown users received over a period of an hour 
 or so because those are almost invariably spam.  But that requires a time 
 delay between when a message is received and when it is filtered, and that 
 in turn requires a place to store messages for a while before they are 
 processed.  While they are in that temporary storage, I don’t want them in 
 the user’s face, but I do want them to be accessible if the user wants to 
 see them (e.g. if they know that an important message is com

Re: Changing the default delivery mailbox to something other than INBOX

[Ron Garret]

Um, yeah.  mail_location is in dovecot.conf.  (I am using dovecot for my LDA, 
which is why I’m asking this on the dovecot list and not the postfix list.)

On Jan 17, 2021, at 9:30 PM, Noel  wrote:

> I expect the delivery location used by the Dovecot LDA is set in the Dovecot 
> LDA and not in postfix.
> 
> 
> 
>   -- Noel Jones
> 
> 
> On 1/17/2021 8:42 PM, Ron Garret wrote:
>> I tried the obvious:
>> 
>> mail_location = maildir:/var/mail/vhosts/%d/%n/mail:INBOX=mail/.Incoming
>> 
>> and that failed in an even more bizarre way.  The Incoming mailbox was 
>> created, it showed up in my mail client as it should have, but mail delivery 
>> still failed.  Not only that, but all the messages that were in my Inbox 
>> disappeared from my mail client.  I reverted the change and all of the 
>> messages that had previously been in Inbox reappeared.
>> 
>> WTF?
>> 
>> On Jan 17, 2021, at 5:00 PM, Ron Garret  wrote:
>> 
>>> I groveled around in the docs and discovered the INBOX=… option to the 
>>> mail_location config parameter.  I tried that, and it didn’t work, but it 
>>> failed in a very strange way.
>>> 
>>> I currently have:
>>> 
>>> mail_location = maildir:/var/mail/vhosts/%d/%n/mail
>>> 
>>> I tried changing that to:
>>> 
>>> mail_location = maildir:/var/mail/vhosts/%d/%n/mail:INBOX=incoming
>>> 
>>> That failed with the following log message:
>>> 
>>> Jan 17 23:41:17 lmtp(s...@rngh.net)<32476>: Error: 
>>> Mailbox INBOX: Failed to autocreate mailbox: Permission denied
>>> 
>>> But the weird thing is that it DID create a directory called incoming, and 
>>> that directory has the same permissions as the rest of the mailbox 
>>> hierarchy.
>>> 
>>> There is something else which I was not expecting, and that is that the 
>>> directory is a peer to /var/mail/vhosts/%d/%n/mail.  What I want is to 
>>> create a folder *inside* /var/mail/vhosts/%d/%n/mail.
>>> 
>>> Advice on how to proceed would be much appreciated.  (BTW, I’d be happy to 
>>> pay someone a consulting fee for help with this project.  If you’re 
>>> interested contact me off-list.)
>>> 
>>> rg
>>> 
>>> On Jan 17, 2021, at 12:08 PM, Ron Garret  wrote:
>>> 
 Is there an easy way (i.e. a built-in configuration setting) to change the 
 name of the mailbox that the dovecot LDA delivers mail into?  The default 
 is INBOX but I’d like mail to be delivered to some other mailbox.  The 
 reason for this is that I want all incoming mail to be invisible to the 
 user by default until it has been screened for viruses and spam.  I know I 
 could do this with Sieve, but that is a PITA.
 
 More details in case anyone is interested:
 
 The goal of this filter is to make it work with very little training.  To 
 bootstrap the process, the filter is given access to outgoing mail (via a 
 milter) which it uses as a reliable training corpus for good messages.  It 
 then leverages that information to filter incoming messages.  For example, 
 messages from senders which have been the recipients of outgoing messages 
 are presumed to be good.  There is also a spam honeypot to provide a 
 reliable spam corpus.
 
 One of the heuristics I want to use is to look for the same subject line 
 in multiple messages from unknown users received over a period of an hour 
 or so because those are almost invariably spam.  But that requires a time 
 delay between when a message is received and when it is filtered, and that 
 in turn requires a place to store messages for a while before they are 
 processed.  While they are in that temporary storage, I don’t want them in 
 the user’s face, but I do want them to be accessible if the user wants to 
 see them (e.g. if they know that an important message is coming in which 
 may be stuck in the temp storage).  So I’d like to make something like an 
 INCOMING mailbox where all mail is delivered.  The messages in INCOMING 
 are scanned by the filter and moved to Junk or INBOX as appropriate.
 
 Thanks,
 rg
 
> 

Re: fts_solr: Indexing failed (fails on particular messages/mailboxes by timeout)

[Alexey Panov]

Dear John,

Thank you very much for this reply! You saved me really a lot of hours of work! 

I have managed to index those mailboxes finally after applying your patch. I 
don’t know why this patch never got any attention and didn’t find its way to 
the upstream...

But it became clear that in current configuration dovecot sends as message body 
tons of useless binary information that not just hard to index but also affects 
search performance badly. Moreover due to this on some mailboxes up to 1% of 
all messages are now ignored completely (it’s of course better than stuck index 
and overloaded IO of the whole system - thanks again!)

I have to wait for non-business hours to run some more tests, but it seems it 
happens really often when dovecot sends binary attachments of any kind for 
indexing. Once done I’ll try to submit more detailed separate bug report.

At the moment I am thinking about developing a small proxy that I gonna put 
between dovecot and solr and strip unnecessary body parts on that stage… 

> On 16. Jan 2021, at 19:16, John Fawcett  wrote:
> 
> On 15/01/2021 11:39, Alexey Panov wrote:
>> I guess I was indeed correct. As message body is the part of mail index 
>> dovecot indeed sends all those binary messages as a part of message body:
>> 
>>> doveadm fetch -u test.u...@interpont.com  
>>> "body" mailbox KS2 uid 2 | wc --bytes
>>> 21549696
>> 
>> This affects FTS indexing dramatically I guess… 
>> 
>>> On 15. Jan 2021, at 17:10, Alexey Panov >> > wrote:
>>> 
>>> I could find messages causing problems as solr log indeed shows the IMAP 
>>> UID. 
>>> 
>>> Reducing batch_size to a single message I could identify one of those 
>>> messages causing the problem. 
>>> 
>>> Now I can confirm the following behaviour: there are some SINGLE messages 
>>> where 60 seconds is NOT ENOUGH to index. The message I am looking at now is 
>>> 20MB big (compressed 17MB) and has a lot of inline pasted screenshots. I 
>>> guess that dovecot might send those pasted images as a part of the message 
>>> body to solr?
>>> 
>>> Here are some logs for better understanding:
>>> 
>>> Folder contents (single message):
>>> 
 doveadm search -u u@d mailbox KS2
 e836ff1d20640160340b532cee39 2
>>> 
>>> 
>>> Force indexing:
>>> 
 doveadm index -u u@d KS2
 doveadm(u@d): Error: fts_solr: Indexing failed: Request timed out (Request 
 queued 61.105 secs ago, 1 send attempts in 61.089 secs, 60.071 in http 
 ioloop, 0.000 in other ioloops, connected 61.089 secs ago)
 doveadm(u@d): Error: Mailbox KS2: Transaction commit failed: FTS 
 transaction commit failed: backend deinit
>>> 
>>> 
>>> Corresponding solr logs:
>>> 
 solr-mailcow_1   | 2021-01-15 10:02:34.901 INFO  (qtp524223214-2127) [ 
   x:dovecot-fts] o.a.s.u.p.LogUpdateProcessorFactory [dovecot-fts]  
 webapp=/solr path=/update 
 params={}{add=[2/e836ff1d20640160340b532cee39/u@d 
 (1688946475272241152)]} 0 80683
>>> 
>>> I would greatly appreciate any help. My users using pasted images heavily 
>>> (designers exchanging mockups etc) and if this truly is a problem then I am 
>>> screwed :) 
>>> 
>>> 
 On 14. Jan 2021, at 23:46, Alexey Panov >>> > wrote:
 
 Hello, 
 
 I am running recent dovecot 2.3.13 (89f716dc2) within dockerized mailcow 
 setup. 
 
 During just another update routine I decided to reindex all mailboxes and 
 some of them fail (timeout) on exactly SAME messages in SAME mailboxes:
 
> doveadm(u@d): Error: fts_solr: Indexing failed: Request timed out 
> (Request queued 73.213 secs ago, 1 send attempts in 73.098 secs, 60.597 
> in http ioloop, 0.000 in other ioloops, connected 127.875 secs ago)
> doveadm(u@d): Error: Mailbox Sent: Mail search failed: Internal error 
> occurred. Refer to server log for more information. [2021-01-14 11:23:46]
> doveadm(u@d): Error: Mailbox Sent: Transaction commit failed: FTS 
> transaction commit failed: backend deinit
 
 This error is persistent and affects only some, but always same folders of 
 same mailboxes (i mean solr can't finish indexing them at all). 
 
 I/O is in perfect condition and has no corruptions. I also tried to do fts 
 rescan which made no change. RAM is fine, I haven't seen OOM conditions.
 
 During my research I found this discussion 
  which 
 lead me idea of reducing the batch_size to as low as 10 (for testing 
 purposes). Observing SOLR logs during reindexing on the mailbox/folder 
 causing the error I noticed the following strange behaviour. 
 
 Firstly, it goes very fast but then always kinda freezes on these messages:
 
> solr-mailcow_1   | 2021-01-14 16:05:09.615 INFO  (qtp524223214-1151) 
> [   x:dovecot-fts] o.a.s.u.p.LogUpdateProcessorFac

Re: Changing the default delivery mailbox to something other than INBOX

[Aki Tuomi]

I don't see how that would the obvious way, and that, as you found out, does 
cause your mails to disappear.

Looking at `man dovecot-lda` you'll find

-m mailbox

Destination  mailbox (default is INBOX). If the mailbox doesn't exist, it will 
not be created (unless the lda_mailbox_autocreate setting is set to yes). If a 
message couldn't be saved to the  mailbox for any reason, it's delivered to 
INBOX instead.

Aki

> On 18/01/2021 04:42 Ron Garret  wrote:
> 
>  
> I tried the obvious:
> 
> mail_location = maildir:/var/mail/vhosts/%d/%n/mail:INBOX=mail/.Incoming
> 
> and that failed in an even more bizarre way.  The Incoming mailbox was 
> created, it showed up in my mail client as it should have, but mail delivery 
> still failed.  Not only that, but all the messages that were in my Inbox 
> disappeared from my mail client.  I reverted the change and all of the 
> messages that had previously been in Inbox reappeared.
> 
> WTF?
> 
> On Jan 17, 2021, at 5:00 PM, Ron Garret  wrote:
> 
> > I groveled around in the docs and discovered the INBOX=… option to the 
> > mail_location config parameter.  I tried that, and it didn’t work, but it 
> > failed in a very strange way.
> > 
> > I currently have:
> > 
> > mail_location = maildir:/var/mail/vhosts/%d/%n/mail
> > 
> > I tried changing that to:
> > 
> > mail_location = maildir:/var/mail/vhosts/%d/%n/mail:INBOX=incoming
> > 
> > That failed with the following log message:
> > 
> > Jan 17 23:41:17 lmtp(s...@rngh.net)<32476>: Error: 
> > Mailbox INBOX: Failed to autocreate mailbox: Permission denied
> > 
> > But the weird thing is that it DID create a directory called incoming, and 
> > that directory has the same permissions as the rest of the mailbox 
> > hierarchy.
> > 
> > There is something else which I was not expecting, and that is that the 
> > directory is a peer to /var/mail/vhosts/%d/%n/mail.  What I want is to 
> > create a folder *inside* /var/mail/vhosts/%d/%n/mail.
> > 
> > Advice on how to proceed would be much appreciated.  (BTW, I’d be happy to 
> > pay someone a consulting fee for help with this project.  If you’re 
> > interested contact me off-list.)
> > 
> > rg
> > 
> > On Jan 17, 2021, at 12:08 PM, Ron Garret  wrote:
> > 
> >> Is there an easy way (i.e. a built-in configuration setting) to change the 
> >> name of the mailbox that the dovecot LDA delivers mail into?  The default 
> >> is INBOX but I’d like mail to be delivered to some other mailbox.  The 
> >> reason for this is that I want all incoming mail to be invisible to the 
> >> user by default until it has been screened for viruses and spam.  I know I 
> >> could do this with Sieve, but that is a PITA.
> >> 
> >> More details in case anyone is interested:
> >> 
> >> The goal of this filter is to make it work with very little training.  To 
> >> bootstrap the process, the filter is given access to outgoing mail (via a 
> >> milter) which it uses as a reliable training corpus for good messages.  It 
> >> then leverages that information to filter incoming messages.  For example, 
> >> messages from senders which have been the recipients of outgoing messages 
> >> are presumed to be good.  There is also a spam honeypot to provide a 
> >> reliable spam corpus.
> >> 
> >> One of the heuristics I want to use is to look for the same subject line 
> >> in multiple messages from unknown users received over a period of an hour 
> >> or so because those are almost invariably spam.  But that requires a time 
> >> delay between when a message is received and when it is filtered, and that 
> >> in turn requires a place to store messages for a while before they are 
> >> processed.  While they are in that temporary storage, I don’t want them in 
> >> the user’s face, but I do want them to be accessible if the user wants to 
> >> see them (e.g. if they know that an important message is coming in which 
> >> may be stuck in the temp storage).  So I’d like to make something like an 
> >> INCOMING mailbox where all mail is delivered.  The messages in INCOMING 
> >> are scanned by the filter and moved to Junk or INBOX as appropriate.
> >> 
> >> Thanks,
> >> rg
> >> 
> >

Re: Changing the default delivery mailbox to something other than INBOX

[Noel]

I expect the delivery location used by the Dovecot LDA is set in the 
Dovecot LDA and not in postfix.




  -- Noel Jones


On 1/17/2021 8:42 PM, Ron Garret wrote:

I tried the obvious:

mail_location = maildir:/var/mail/vhosts/%d/%n/mail:INBOX=mail/.Incoming

and that failed in an even more bizarre way.  The Incoming mailbox was created, 
it showed up in my mail client as it should have, but mail delivery still 
failed.  Not only that, but all the messages that were in my Inbox disappeared 
from my mail client.  I reverted the change and all of the messages that had 
previously been in Inbox reappeared.

WTF?

On Jan 17, 2021, at 5:00 PM, Ron Garret  wrote:


I groveled around in the docs and discovered the INBOX=… option to the 
mail_location config parameter.  I tried that, and it didn’t work, but it 
failed in a very strange way.

I currently have:

mail_location = maildir:/var/mail/vhosts/%d/%n/mail

I tried changing that to:

mail_location = maildir:/var/mail/vhosts/%d/%n/mail:INBOX=incoming

That failed with the following log message:

Jan 17 23:41:17 lmtp(s...@rngh.net)<32476>: Error: 
Mailbox INBOX: Failed to autocreate mailbox: Permission denied

But the weird thing is that it DID create a directory called incoming, and that 
directory has the same permissions as the rest of the mailbox hierarchy.

There is something else which I was not expecting, and that is that the 
directory is a peer to /var/mail/vhosts/%d/%n/mail.  What I want is to create a 
folder *inside* /var/mail/vhosts/%d/%n/mail.

Advice on how to proceed would be much appreciated.  (BTW, I’d be happy to pay 
someone a consulting fee for help with this project.  If you’re interested 
contact me off-list.)

rg

On Jan 17, 2021, at 12:08 PM, Ron Garret  wrote:


Is there an easy way (i.e. a built-in configuration setting) to change the name 
of the mailbox that the dovecot LDA delivers mail into?  The default is INBOX 
but I’d like mail to be delivered to some other mailbox.  The reason for this 
is that I want all incoming mail to be invisible to the user by default until 
it has been screened for viruses and spam.  I know I could do this with Sieve, 
but that is a PITA.

More details in case anyone is interested:

The goal of this filter is to make it work with very little training.  To 
bootstrap the process, the filter is given access to outgoing mail (via a 
milter) which it uses as a reliable training corpus for good messages.  It then 
leverages that information to filter incoming messages.  For example, messages 
from senders which have been the recipients of outgoing messages are presumed 
to be good.  There is also a spam honeypot to provide a reliable spam corpus.

One of the heuristics I want to use is to look for the same subject line in 
multiple messages from unknown users received over a period of an hour or so 
because those are almost invariably spam.  But that requires a time delay 
between when a message is received and when it is filtered, and that in turn 
requires a place to store messages for a while before they are processed.  
While they are in that temporary storage, I don’t want them in the user’s face, 
but I do want them to be accessible if the user wants to see them (e.g. if they 
know that an important message is coming in which may be stuck in the temp 
storage).  So I’d like to make something like an INCOMING mailbox where all 
mail is delivered.  The messages in INCOMING are scanned by the filter and 
moved to Junk or INBOX as appropriate.

Thanks,
rg

Re: dovecot 2.3.13 : make check FAILURE : Assert failed: buffer_append_full_file

[Aki Tuomi]

> On 18/01/2021 00:35 J Lumby  wrote:
> 
>  
> Thanks Aki.
> 
> 
> On 1/16/21 10:18 AM, Aki Tuomi wrote:
> >> On 16/01/2021 16:36 J Lumby  wrote:
> >>
> >>
> >>
> >> make check had one failure as follows :
> >>
> >> test-buffer-istream.c:54: Assert failed: buffer_append_full_file(result,
> >> TEST_FILENAME, SIZE_MAX, &error) == BUFFER_APPEND_READ_ERROR
> >> test-buffer-istream.c:56: Assert failed: error != NULL && *error != '\0'
> >> buffer_append_full_file .. :
> >> FAILED
> >>
> > Hi!
> >
> > You are running make check as root. Try running it as non-root user.
> 
> You were absolutely correct with your diagnosis -  yes I was running 
> make check as root.
> 
> I did so because I run dovecot itself as root  -  as instructed by the 
> wiki :
> 
> 
>   Running Dovecot  : Starting  :
> 
> "Dovecot can simply be started by running dovecot as root"
> 
> And (I assume) it is best to run the make check under same userid,  i.e. 
> root, otherwise it is not testing what will actually be running.
> 
> But anyway I tried running make check under non-root and the result was 
> much worse -   some indeterminate FAIL much earlier :
> 
> Making check in lib-ssl-iostream
> make[2]: Entering directory 
> '/mnt/julywext/wextmisc/fed30GBroot/ahcombld/dovecot-2.3.13/src/lib-ssl-iostream'
> make  check-local
> make[3]: Entering directory 
> '/mnt/julywext/wextmisc/fed30GBroot/ahcombld/dovecot-2.3.13/src/lib-ssl-iostream'
> for bin in test-iostream-ssl; do \
>    if ! /bin/sh ../../run-test.sh ../.. ./$bin; then exit 1; fi; \
> done
> collect2: error: ld returned 213 exit status
> Failed to run: ./test-iostream-ssl
> 

This is because you have also compiled the source as root and fails to write to 
those directories. You need to build as non-root, too.

> I ran it with sh -x to see if it would tell me any more but all it 
> indicated is that,   somewhere during execution of valgrind,   it hits 
> this error from ld.
> 
> My valgrind is the latest,   3.16.1,
> 
> 
> May we re-visit the FAIL I reported originally ?  Why does it FAIL 
> when run under userid root?
> 

Because it's testing that it cannot read a file it has no permissions to read. 
root can read all files. You can see it does a chmod few lines before.

> I am guessing that somehow root gets a higher limit for the count 
> parameter on read(fd , *buf , count) than non-root?    Although the man 
> page for read() does not indicate any such distinction.
> 
> Or is dovecot implementing its own user-id-specific limit?
> 
> And,    what code could I add to test-buffer-istream.c to make it print 
> out the offending errno?
> 
> Or,  perhaps easier -  is it safe to ignore this one FAIL?
> 
> >
> > Aki
> > .
> Cheers,    John Lumby

Aki

Re: Changing the default delivery mailbox to something other than INBOX

[Ron Garret]

I tried the obvious:

mail_location = maildir:/var/mail/vhosts/%d/%n/mail:INBOX=mail/.Incoming

and that failed in an even more bizarre way.  The Incoming mailbox was created, 
it showed up in my mail client as it should have, but mail delivery still 
failed.  Not only that, but all the messages that were in my Inbox disappeared 
from my mail client.  I reverted the change and all of the messages that had 
previously been in Inbox reappeared.

WTF?

On Jan 17, 2021, at 5:00 PM, Ron Garret  wrote:

> I groveled around in the docs and discovered the INBOX=… option to the 
> mail_location config parameter.  I tried that, and it didn’t work, but it 
> failed in a very strange way.
> 
> I currently have:
> 
> mail_location = maildir:/var/mail/vhosts/%d/%n/mail
> 
> I tried changing that to:
> 
> mail_location = maildir:/var/mail/vhosts/%d/%n/mail:INBOX=incoming
> 
> That failed with the following log message:
> 
> Jan 17 23:41:17 lmtp(s...@rngh.net)<32476>: Error: 
> Mailbox INBOX: Failed to autocreate mailbox: Permission denied
> 
> But the weird thing is that it DID create a directory called incoming, and 
> that directory has the same permissions as the rest of the mailbox hierarchy.
> 
> There is something else which I was not expecting, and that is that the 
> directory is a peer to /var/mail/vhosts/%d/%n/mail.  What I want is to create 
> a folder *inside* /var/mail/vhosts/%d/%n/mail.
> 
> Advice on how to proceed would be much appreciated.  (BTW, I’d be happy to 
> pay someone a consulting fee for help with this project.  If you’re 
> interested contact me off-list.)
> 
> rg
> 
> On Jan 17, 2021, at 12:08 PM, Ron Garret  wrote:
> 
>> Is there an easy way (i.e. a built-in configuration setting) to change the 
>> name of the mailbox that the dovecot LDA delivers mail into?  The default is 
>> INBOX but I’d like mail to be delivered to some other mailbox.  The reason 
>> for this is that I want all incoming mail to be invisible to the user by 
>> default until it has been screened for viruses and spam.  I know I could do 
>> this with Sieve, but that is a PITA.
>> 
>> More details in case anyone is interested:
>> 
>> The goal of this filter is to make it work with very little training.  To 
>> bootstrap the process, the filter is given access to outgoing mail (via a 
>> milter) which it uses as a reliable training corpus for good messages.  It 
>> then leverages that information to filter incoming messages.  For example, 
>> messages from senders which have been the recipients of outgoing messages 
>> are presumed to be good.  There is also a spam honeypot to provide a 
>> reliable spam corpus.
>> 
>> One of the heuristics I want to use is to look for the same subject line in 
>> multiple messages from unknown users received over a period of an hour or so 
>> because those are almost invariably spam.  But that requires a time delay 
>> between when a message is received and when it is filtered, and that in turn 
>> requires a place to store messages for a while before they are processed.  
>> While they are in that temporary storage, I don’t want them in the user’s 
>> face, but I do want them to be accessible if the user wants to see them 
>> (e.g. if they know that an important message is coming in which may be stuck 
>> in the temp storage).  So I’d like to make something like an INCOMING 
>> mailbox where all mail is delivered.  The messages in INCOMING are scanned 
>> by the filter and moved to Junk or INBOX as appropriate.
>> 
>> Thanks,
>> rg
>> 
> 

Re: Changing the default delivery mailbox to something other than INBOX

[Ron Garret]

I groveled around in the docs and discovered the INBOX=… option to the 
mail_location config parameter.  I tried that, and it didn’t work, but it 
failed in a very strange way.

I currently have:

mail_location = maildir:/var/mail/vhosts/%d/%n/mail

I tried changing that to:

mail_location = maildir:/var/mail/vhosts/%d/%n/mail:INBOX=incoming

That failed with the following log message:

Jan 17 23:41:17 lmtp(s...@rngh.net)<32476>: Error: 
Mailbox INBOX: Failed to autocreate mailbox: Permission denied

But the weird thing is that it DID create a directory called incoming, and that 
directory has the same permissions as the rest of the mailbox hierarchy.

There is something else which I was not expecting, and that is that the 
directory is a peer to /var/mail/vhosts/%d/%n/mail.  What I want is to create a 
folder *inside* /var/mail/vhosts/%d/%n/mail.

Advice on how to proceed would be much appreciated.  (BTW, I’d be happy to pay 
someone a consulting fee for help with this project.  If you’re interested 
contact me off-list.)

rg

On Jan 17, 2021, at 12:08 PM, Ron Garret  wrote:

> Is there an easy way (i.e. a built-in configuration setting) to change the 
> name of the mailbox that the dovecot LDA delivers mail into?  The default is 
> INBOX but I’d like mail to be delivered to some other mailbox.  The reason 
> for this is that I want all incoming mail to be invisible to the user by 
> default until it has been screened for viruses and spam.  I know I could do 
> this with Sieve, but that is a PITA.
> 
> More details in case anyone is interested:
> 
> The goal of this filter is to make it work with very little training.  To 
> bootstrap the process, the filter is given access to outgoing mail (via a 
> milter) which it uses as a reliable training corpus for good messages.  It 
> then leverages that information to filter incoming messages.  For example, 
> messages from senders which have been the recipients of outgoing messages are 
> presumed to be good.  There is also a spam honeypot to provide a reliable 
> spam corpus.
> 
> One of the heuristics I want to use is to look for the same subject line in 
> multiple messages from unknown users received over a period of an hour or so 
> because those are almost invariably spam.  But that requires a time delay 
> between when a message is received and when it is filtered, and that in turn 
> requires a place to store messages for a while before they are processed.  
> While they are in that temporary storage, I don’t want them in the user’s 
> face, but I do want them to be accessible if the user wants to see them (e.g. 
> if they know that an important message is coming in which may be stuck in the 
> temp storage).  So I’d like to make something like an INCOMING mailbox where 
> all mail is delivered.  The messages in INCOMING are scanned by the filter 
> and moved to Junk or INBOX as appropriate.
> 
> Thanks,
> rg
> 

Re: dovecot 2.3.13 : make check FAILURE : Assert failed: buffer_append_full_file

[J Lumby]

Thanks Aki.


On 1/16/21 10:18 AM, Aki Tuomi wrote:

On 16/01/2021 16:36 J Lumby  wrote:



make check had one failure as follows :

test-buffer-istream.c:54: Assert failed: buffer_append_full_file(result,
TEST_FILENAME, SIZE_MAX, &error) == BUFFER_APPEND_READ_ERROR
test-buffer-istream.c:56: Assert failed: error != NULL && *error != '\0'
buffer_append_full_file .. :
FAILED


Hi!

You are running make check as root. Try running it as non-root user.


You were absolutely correct with your diagnosis -  yes I was running 
make check as root.


I did so because I run dovecot itself as root  -  as instructed by the 
wiki :



 Running Dovecot  : Starting  :

"Dovecot can simply be started by running dovecot as root"

And (I assume) it is best to run the make check under same userid,  i.e. 
root, otherwise it is not testing what will actually be running.


But anyway I tried running make check under non-root and the result was 
much worse -   some indeterminate FAIL much earlier :


Making check in lib-ssl-iostream
make[2]: Entering directory 
'/mnt/julywext/wextmisc/fed30GBroot/ahcombld/dovecot-2.3.13/src/lib-ssl-iostream'

make  check-local
make[3]: Entering directory 
'/mnt/julywext/wextmisc/fed30GBroot/ahcombld/dovecot-2.3.13/src/lib-ssl-iostream'

for bin in test-iostream-ssl; do \
  if ! /bin/sh ../../run-test.sh ../.. ./$bin; then exit 1; fi; \
done
collect2: error: ld returned 213 exit status
Failed to run: ./test-iostream-ssl

I ran it with sh -x to see if it would tell me any more but all it 
indicated is that,   somewhere during execution of valgrind,   it hits 
this error from ld.


My valgrind is the latest,   3.16.1,


May we re-visit the FAIL I reported originally ?  Why does it FAIL 
when run under userid root?


I am guessing that somehow root gets a higher limit for the count 
parameter on read(fd , *buf , count) than non-root?    Although the man 
page for read() does not indicate any such distinction.


Or is dovecot implementing its own user-id-specific limit?

And,    what code could I add to test-buffer-istream.c to make it print 
out the offending errno?


Or,  perhaps easier -  is it safe to ignore this one FAIL?



Aki
.

Cheers,    John Lumby

Changing the default delivery mailbox to something other than INBOX

[Ron Garret]

Is there an easy way (i.e. a built-in configuration setting) to change the name 
of the mailbox that the dovecot LDA delivers mail into?  The default is INBOX 
but I’d like mail to be delivered to some other mailbox.  The reason for this 
is that I want all incoming mail to be invisible to the user by default until 
it has been screened for viruses and spam.  I know I could do this with Sieve, 
but that is a PITA.

More details in case anyone is interested:

The goal of this filter is to make it work with very little training.  To 
bootstrap the process, the filter is given access to outgoing mail (via a 
milter) which it uses as a reliable training corpus for good messages.  It then 
leverages that information to filter incoming messages.  For example, messages 
from senders which have been the recipients of outgoing messages are presumed 
to be good.  There is also a spam honeypot to provide a reliable spam corpus.

One of the heuristics I want to use is to look for the same subject line in 
multiple messages from unknown users received over a period of an hour or so 
because those are almost invariably spam.  But that requires a time delay 
between when a message is received and when it is filtered, and that in turn 
requires a place to store messages for a while before they are processed.  
While they are in that temporary storage, I don’t want them in the user’s face, 
but I do want them to be accessible if the user wants to see them (e.g. if they 
know that an important message is coming in which may be stuck in the temp 
storage).  So I’d like to make something like an INCOMING mailbox where all 
mail is delivered.  The messages in INCOMING are scanned by the filter and 
moved to Junk or INBOX as appropriate.

Thanks,
rg

Re: mbox to pst advice

[Odhiambo Washington]

On Sun, 17 Jan 2021 at 22:42, Benny Pedersen  wrote:

> On 2021-01-17 18:43, Odhiambo Washington wrote:
>
> > Personally, I would not bother looking for a script or even asking
> > anyone if they knew how to convert
> > mbox2pst.
>
> there is always alternatives :-)
>
> aid4mail
>

Yes :-)

Spend some bucks!


-- 
Best regards,
Odhiambo WASHINGTON,
Nairobi,KE
+254 7 3200 0004/+254 7 2274 3223
"Oh, the cruft.", grep ^[^#] :-)

Re: mbox to pst advice

[Benny Pedersen]

On 2021-01-17 18:43, Odhiambo Washington wrote:


Personally, I would not bother looking for a script or even asking
anyone if they knew how to convert
mbox2pst.


there is always alternatives :-)

aid4mail

Re: Antispam plugin

[Håkon Alstadheim]

Den 17.01.2021 02:15, skrev Toni Mueller:

Hi,

On Tue, Sep 22, 2020 at 02:01:21PM +0300, Aki Tuomi wrote:

You don't need antispam plugin:

https://doc.dovecot.org/configuration_manual/howto/antispam_with_sieve/

I also don't see any docs about the plugin on the wiki.
Does it mean that this plugin is deprecated?

The link describes how the functionality of the antispam plugin is no 
longer needed, you can set up antispam with built-in functionality of 
dovecot.

Re: Obtaining the IMAP GUID from a sieve script

[Ron Garret]

Just for the record, it turns out that doveadm is the key to happiness here.  
It lets me find and manipulate messages by message-id (or any other header) 
which is what I needed.

rg

Additional custom fields in map dict with static values.

[Mimiko]

Hello.

Dovector version 2.2.27.

How can I add custom fields in the map for the last_login. For example:

map {
  pattern = shared/last_login/imap/$user/$client_ip
  table = users_last_login
  value_field = last_login_imap_int
  value_type = string
  fields {
userid = $user
last_login_client_ip_imap = $client_ip
custom_field_name = some_static_value
  }
}

Re: mbox to pst advice

[Odhiambo Washington]

On Sun, 17 Jan 2021 at 20:27, Marc Roos  wrote:

>
>
> >
> >   Anyone an idea how to convert mbox ot pst on linux?
> >
> >
> >
> >
> >
> > Just run let Outlook fetch the mail and it will automatically create a
> > PST.
> >
> > There is no need for wasting CPU cycles and time converting :-)
>
> Scripting this on linux would be much faster for me than doing it manually
> with outlook.
>

I disagree.
Outlook will take shorter than you take to write and run a script.
In fact, if you gave me the login credential to the server, I'd fetch all
the mail with Outlook in
a short time and upload for you the PST back into the server :-)
Personally, I would not bother looking for a script or even asking anyone
if they knew how to convert
mbox2pst.


-- 
Best regards,
Odhiambo WASHINGTON,
Nairobi,KE
+254 7 3200 0004/+254 7 2274 3223
"Oh, the cruft.", grep ^[^#] :-)

RE: mbox to pst advice

[Marc Roos]

> 
>   Anyone an idea how to convert mbox ot pst on linux?
> 
> 
> 
> 
> 
> Just run let Outlook fetch the mail and it will automatically create a
> PST.
> 
> There is no need for wasting CPU cycles and time converting :-)

Scripting this on linux would be much faster for me than doing it manually with 
outlook.

Re: mbox to pst advice

[Odhiambo Washington]

On Sun, 17 Jan 2021 at 15:41, Marc Roos  wrote:

>
>
> Anyone an idea how to convert mbox ot pst on linux?
>
>
>
Just run let Outlook fetch the mail and it will automatically create a PST.

There is no need for wasting CPU cycles and time converting :-)


-- 
Best regards,
Odhiambo WASHINGTON,
Nairobi,KE
+254 7 3200 0004/+254 7 2274 3223
"Oh, the cruft.", grep ^[^#] :-)

Re: Where is dovemon

[Steven Varco]

Poolmon already exists as a repalcement for this matter.

However, it should be made more clearly in the documentation, that „Dovemon“ is 
not part of the community package and therefore not available in the free 
version, IMHO.

-- 
https://steven.varco.ch/ 

> Am 17.01.2021 um 15:37 schrieb Scott Q. :
> 
> You can write something similar and I think others did already if you Google 
> for it. Shouldn't take you more than 1-2h to write something in Perl:
> connect to backend, no login, remove from director
> 
> 
> On Saturday, 16/01/2021 at 22:37 Steven Varco wrote:
> Hi Christian
> 
> This confused me as well.
> I later found out, that „Dovemon“ is an exlcusive part of the dovecot „pro" 
> Package (the paid docecot).
> This is why you cannot find it.
> 
> best regards,
> Steven
> 
> -- 
> https://steven.varco.ch/ 
> 
> > Am 13.01.2021 um 12:30 schrieb li...@mlserv.org:
> > 
> > Hello,
> > 
> > I found this link in the documentation:
> > 
> > https://doc.dovecot.org/configuration_manual/dovemon/
> > 
> > But where can I find the program "dovemon"? I searched all over whithout 
> > luck. In the source code, Google, nothing. It seems as only the web site 
> > would exist.
> > 
> > Can somebody help me please
> > 
> > Christian Rößner
> > -- 
> > Rößner-Network-Solutions
> > Zertifizierter ITSiBe / CISO
> > Karl-Bröger-Str. 10, 36304 Alsfeld
> > Fax: +49 6631 78823409, Mobil: +49 171 9905345
> > USt-IdNr.: DE225643613, https://roessner.website
> > PGP fingerprint: 658D 1342 B762 F484 2DDF 1E88 38A5 4346 D727 94E5 
> >

Re: Where is dovemon

[Scott Q.]

You can write something similar and I think others did already if
you Google for it. Shouldn't take you more than 1-2h to write
something in Perl:connect to backend, no login, remove from director



On Saturday, 16/01/2021 at 22:37 Steven Varco wrote:


Hi Christian

This confused me as well.
I later found out, that „Dovemon“ is an exlcusive part of the
dovecot „pro" Package (the paid docecot).
This is why you cannot find it.

best regards,
Steven

-- 
https://steven.varco.ch/ 

> Am 13.01.2021 um 12:30 schrieb li...@mlserv.org:
> 
> Hello,
> 
> I found this link in the documentation:
> 
> https://doc.dovecot.org/configuration_manual/dovemon/
> 
> But where can I find the program "dovemon"? I searched all over
whithout luck. In the source code, Google, nothing. It seems as only
the web site would exist.
> 
> Can somebody help me please
> 
> Christian Rößner
> -- 
> Rößner-Network-Solutions
> Zertifizierter ITSiBe / CISO
> Karl-Bröger-Str. 10, 36304 Alsfeld
> Fax: +49 6631 78823409, Mobil: +49 171 9905345
> USt-IdNr.: DE225643613, https://roessner.website
> PGP fingerprint: 658D 1342 B762 F484 2DDF 1E88 38A5 4346 D727 94E5 
>

mbox to pst advice

[Marc Roos]

Anyone an idea how to convert mbox ot pst on linux?