Lazy Expunge - why oh why deprecating namespace storage for this

[Matt Bryant]

Hi Dovecot,

Noticed on your web site that you have marked lazy_expunge storage 
location to namespace as deprecated since v2.3.0 and suggest a single 
mailbox with acl is the way to go. This seems to be a backward step for 
those that make use of the alt storage facility.


Is there anyway we can persuade you NOT to deprecate this - it still 
works atm


Why ? Well scenario time 

Installations make use of alt-storage feature so that new messages can 
be stored along with indexes etc on faster storage, whilst offloading 
older less viewed mail to a different storage tier.


With storage to namespace - that namespace can have a location set that 
separates it from the mailbox (maybe even on a 3rd tier of storage) BUT 
with the mailbox solution then any deleted mail will in fact return from 
the alternate storage to the primary storage, and then will need to have 
a job run to alt-move it back off the faster storage to sit with the 
older messages and also can no longer be a potential 3rd tier of storage.


This seems crazy and in fact some of reason mdbox is avoided since 
purging an mdbox means that any messages on alt-storage gets pulled back 
to primary storage.


Alternately are there plans to allow a mailbox to have a 'location' in 
which case the location of the expunge mailbox could be set differently ?


rgds

Matt

Re: syslog logging + 2.3.10 + centos 7

[Matt Bryant]

Managed to capture and strace a snippet

|sendto(40, "<22>Aug 19 05:38:37 dovecot: imap-login: 
Login: user=, method=PLAIN, rip=127.0.0.1, 
lip=127.0.0.1, mpid=30681, secured, session=<2b5XajStxtp/AAAB>", 183, 
MSG_NOSIGNAL, NULL, 0) = -1 ECONNREFUSED (Connection refused) > No DWARF 
information found close(40) = 0 > No DWARF 
information found socket(AF_UNIX, SOCK_DGRAM|SOCK_CLOEXEC, 0) = 
40 > No DWARF information found 
connect(40, {sa_family=AF_UNIX, sun_path="/dev/log"}, 
110) = -1 ECONNREFUSED (Connection refused) > No DWARF information found 
close(40) = 0|


and the usual connection

|u_dgr UNCONN 0 0 * 11469151 * 11469130 users:(("log",pid=28694,fd=38))|
|seems to just drop out of existance, the above connection on 2.2 though 
does reconnect ... |


Has anyone else experienced this issue ?? Not sure if dovecot or centos 
issue tbh  ? Anyone have any ideas ???


rgds

Matt

On 19/8/20 5:23 pm, Matt Bryant wrote:

All,

Running dovecot 2.3.10 on a CentOS 7.x and logging via syslog seems to 
stop when i restart rsyslog process. Reload doesnt get it working 
again only a dovecot restart. Tested this with dovecot 2.2.36 and this 
did not exhibit the same behaviour.


systemctl restart dovecot rsyslog - this gets a working state

systemctl restart rsyslog - this restarts syslog other 
processes like exim continue to log but dovecot stops


systemctl reload dovecot -- this has no impact logging 
still does not work


systemctl restart dovecot - this fixes the issue.

Are there any changes to the log process between 2.2 and 2.3 that 
might explain this ?


rgds

Matt

syslog logging + 2.3.10 + centos 7

[Matt Bryant]

All,

Running dovecot 2.3.10 on a CentOS 7.x and logging via syslog seems to 
stop when i restart rsyslog process. Reload doesnt get it working again 
only a dovecot restart. Tested this with dovecot 2.2.36 and this did not 
exhibit the same behaviour.


systemctl restart dovecot rsyslog - this gets a working state

systemctl restart rsyslog - this restarts syslog other 
processes like exim continue to log but dovecot stops


systemctl reload dovecot -- this has no impact logging still 
does not work


systemctl restart dovecot - this fixes the issue.

Are there any changes to the log process between 2.2 and 2.3 that might 
explain this ?


rgds

Matt

2.3.10 + lazy expunge

[Matt Bryant]

Hi,

so currently run dovecot 2.2.x on CentOS 7 box and was looking to 
upgrade to 2.3.10. I use lazy expunge atm into a different namespace but 
saw on the new doco pages that is is depracted in 2.3.0.


Have tested and still seems to work so question is more around why ? Why 
would this be deprecated given using a different namespace allows 2 things


a) the folder where message is located is taken accross so if I want to 
restore and know where it went


b) namespace allows me to specify an alternate location for storing the 
messages.


Also given it does still work is there any indication as to what version 
the deprecation basically equates to it not working.


rgds

Matt

-- 8< -


   namespace

Deprecated since version v2.3.0.

2.3.7 + stats

[Matt Bryant via dovecot]

Is there any additional documentation/information around the new stats 
module.


Have added some metrics just to see what they produce

##
## Metrics
###

metric imap {
    event_name = imap_command_finished
    #source_location = example.c:123

    #categories =

    fields = name args running_usecs bytes_in bytes_out

    #filter {
    #    field_key = wildcard
    #}
}

metric sql {
    event_name = sql_query_finished
}

metric auth {
    event_name = auth_request_finished
    fields = user transport error successful
}

and get the following


[root@stargate dovecot]# doveadm stats dump
metric_name    field    count    sum    min    max    avg median    
stddev    %95
imap    duration    370    2007904499    130    62955249 5426768.92    
2068    16436817.37    60026465

imap    name    370    0    0    0    0.00    0    0.00    0
imap    args    0    0    0    0    0.00    0    0.00    0
imap    running_usecs    370    2007865330    81 62955127    
5426663.05    1991    16436816.76    60026329

imap    bytes_in    370    5366    2    173    14.50    8 19.04    35
imap    bytes_out    370    2119971    0    941517 5729.65    41    
53760.89    2082
sql    duration    18    28991    991    2305    1610.61 1660    
377.36    2305
auth    duration    12    26046980    8146    7079879 2170581.67    
847730    2457811.23    7079879

auth    user    12    0    0    0    0.00    0    0.00    0
auth    transport    12    0    0    0    0.00    0    0.00 0
auth    error    0    0    0    0    0.00    0    0.00    0
auth    successful    0    0    0    0    0.00    0    0.00 0

the main wiki page on stats/events doesnt really hold much detail whats 
stores for each event the above fields dont make much sense


and top no longer works out of the box

[root@stargate dovecot]# doveadm stats top

usage: doveadm [-Dv] [-f ] stats  []
  dump [-s ] [-r] [-f ]


has is been removed ? do you need to specify something additional now ???


rgds


Matt

iterate query and dsync replication 2.2.33.2

[Matt Bryant]

All,

According to doco replication gets its list of users to replicate from
the iterate query but this doesn't seem to be the case or its just being
totally ignored ..

Have ldap passbd and sql userdb, with the iterate query in the userdb
sql file. -The query itself returns the handful of users I want to
replicate BUT if I run the command  dovecot replicator status '*' it
shows pretty much all users in the system  ..  is there a bug here or is
documentation just completely wrong when it comes to what is/is not
replicated.

rgds

Matt

dsync replication

[Matt Bryant]

We have a setup with ldap and sql passdbs and sql userdb, from my
understanding of the documentation dsync repliction picks up the users
to replicate via the iterate query. This query returns 646 users in sql
(not everyone gets replicated) but when i run a doveadm replicator status

Total number of known users   92236

seriously that kinda lot more than 646

Does iterate_query belong in user_db or passdb? Is it the fact I have
ldap and sql passdb and its picking up dome default ldap query as well
the specified sql iterate query ?

rgds

Matt

Re: dovecot + centos 7 + internal ca + hostname change

[Matt Bryant]

Turns out this was an openldap config issue .. connecting to ldap via
self signed cert and had

/etc/openldap/ldap.conf as


TLS_CACERT /etc/dovecot/ldap_ca
TLS_REQCERT allow
TLS_CACERTDIR/etc/openldap/certs

SASL_NOCANONon

Seems what ever gets generated in TLS_CACERTDIR is problem .. commentng
that out seems to have resolved issue ..


> Matt Bryant <mailto:m...@the-bryants.net>
> 13 September 2018 at 12:52 pm
> Not sure if this is dovecot or not but can find very little ie no info
> around on this ... and added the pem file into
> /etc/pki/ca-trust/source/anchors and run udpate-ca-trust .. all works ok
> .. (this is on centos 7 btw)
>
> So wanted to change the hostname away from ip-x-x-x-x to something a
> little bit more descriptive .. but then kaboom .. doesnt work any more
> and the following errors are seen.
>
> Have created and internal CA for domain and added it to
> Sep 13 10:42:04 ip-10-0-40-230 dovecot: master: Dovecot v2.2.33.2
> (d6601f4ec) starting up for imap, pop3, lmtp, sieve (core dumps disabled)
> Sep 13 10:42:04 ip-10-0-40-230 dovecot: auth: Error: p11-kit:
> 'attr->pValue != NULL' not true at attrs_build
> Sep 13 10:42:04 ip-10-0-40-230 dovecot: auth: Error: p11-kit:
> 'lexer->tok.field.name && lexer->tok.field.value' not true at
> p11_lexer_next
> Sep 13 10:42:04 ip-10-0-40-230 dovecot: auth: Error: p11-kit: 'attrs !=
> NULL' not true at attrs_build
> Sep 13 10:42:04 ip-10-0-40-230 dovecot: message repeated 16 times: [
> auth: Error: p11-kit: 'attrs != NULL' not true at attrs_build]
> Sep 13 10:42:04 ip-10-0-40-230 dovecot: auth: Error: p11-kit:
> 'new_memory != NULL' not true at maybe_expand_array
> Sep 13 10:42:04 ip-10-0-40-230 dovecot: auth: Error: p11-kit: shouldn't
> be reached at p11_array_push
> Sep 13 10:42:04 ip-10-0-40-230 dovecot: auth: Error: p11-kit: shouldn't
> be reached at sink_object
> Sep 13 10:42:04 ip-10-0-40-230 dovecot: auth: Error: p11-kit: 'attrs !=
> NULL' not true at attrs_build
> Sep 13 10:42:04 ip-10-0-40-230 dovecot: auth: Error: p11-kit:
> 'new_memory != NULL' not true at maybe_expand_array
> Sep 13 10:42:04 ip-10-0-40-230 dovecot: auth: Error: p11-kit: shouldn't
> be reached at p11_array_push
> Sep 13 10:42:04 ip-10-0-40-230 dovecot: auth: Error: p11-kit: shouldn't
> be reached at sink_object
> ...
> ...
>
> Sep 13 10:42:04 ip-10-0-40-230 dovecot: auth: Error: p11-kit:
> 'new_memory != NULL' not true at maybe_expand_array
> Sep 13 10:42:04 ip-10-0-40-230 dovecot: auth: Error: p11-kit: shouldn't
> be reached at p11_array_push
> Sep 13 10:42:04 ip-10-0-40-230 dovecot: auth: Error: p11-kit: shouldn't
> be reached at sink_object
> Sep 13 10:42:04 ip-10-0-40-230 dovecot: auth: Error: p11-kit:
> 'attr->pValue != NULL' not true at attrs_build
> Sep 13 10:42:04 ip-10-0-40-230 dovecot: auth: Error: p11-kit:
> 'new_memory != NULL' not true at maybe_expand_array
> Sep 13 10:42:04 ip-10-0-40-230 dovecot: auth: Error: p11-kit: shouldn't
> be reached at p11_array_push
> Sep 13 10:42:04 ip-10-0-40-230 dovecot: auth: Error: p11-kit: shouldn't
> be reached at sink_object
> Sep 13 10:42:04 ip-10-0-40-230 dovecot: auth: Error: p11-kit: no
> CKA_CLASS attribute found
> Sep 13 10:42:04 ip-10-0-40-230 dovecot: auth: Error: p11-kit: couldn't
> load file into objects:
> /usr/share/pki/ca-trust-source/ca-bundle.trust.p11-kit
> Sep 13 10:42:05 ip-10-0-40-230 dovecot: auth-worker: Error: p11-kit:
> 'attrs != NULL' not true at attrs_build
> Sep 13 10:42:05 ip-10-0-40-230 dovecot: auth-worker: Fatal: master:
> service(auth-worker): child 14389 killed with signal 11 (core dumps
> disabled)
> Sep 13 10:42:05 ip-10-0-40-230 dovecot: auth-worker: Error: p11-kit:
> 'attrs != NULL' not true at attrs_build
> Sep 13 10:42:05 ip-10-0-40-230 dovecot: auth-worker: Fatal: master:
> service(auth-worker): child 14391 killed with signal 11 (core dumps
> disabled)
> Sep 13 10:42:05 ip-10-0-40-230 dovecot: auth-worker: Error: p11-kit:
> 'attrs != NULL' not true at attrs_build
> Sep 13 10:42:05 ip-10-0-40-230 dovecot: auth-worker: Fatal: master:
> service(auth-worker): child 14393 killed with signal 11 (core dumps
> disabled)
>
> why would a hostname change make any difference here .. the certs
> specified in dovecot config are all complete in their chain so not sure
> what its trying to do ... set hostname back to original works find .. so
> something is obviously tied or keyed to hostname though cant find
> anything specific
>
> anyone seen anything like this at all ??
>
> rgds
>
> Matt

dovecot + centos 7 + internal ca + hostname change

[Matt Bryant]

Not sure if this is dovecot or not but can find very little ie no info
around on this ... and added the pem file into
/etc/pki/ca-trust/source/anchors and run udpate-ca-trust .. all works ok
.. (this is on centos 7 btw)

So wanted to change the hostname away from ip-x-x-x-x to something a
little bit more descriptive .. but then kaboom .. doesnt work any more
and the following errors are seen.

Have created and internal CA for domain and added it to
Sep 13 10:42:04 ip-10-0-40-230 dovecot: master: Dovecot v2.2.33.2
(d6601f4ec) starting up for imap, pop3, lmtp, sieve (core dumps disabled)
Sep 13 10:42:04 ip-10-0-40-230 dovecot: auth: Error: p11-kit:
'attr->pValue != NULL' not true at attrs_build
Sep 13 10:42:04 ip-10-0-40-230 dovecot: auth: Error: p11-kit:
'lexer->tok.field.name && lexer->tok.field.value' not true at p11_lexer_next
Sep 13 10:42:04 ip-10-0-40-230 dovecot: auth: Error: p11-kit: 'attrs !=
NULL' not true at attrs_build
Sep 13 10:42:04 ip-10-0-40-230 dovecot: message repeated 16 times: [
auth: Error: p11-kit: 'attrs != NULL' not true at attrs_build]
Sep 13 10:42:04 ip-10-0-40-230 dovecot: auth: Error: p11-kit:
'new_memory != NULL' not true at maybe_expand_array
Sep 13 10:42:04 ip-10-0-40-230 dovecot: auth: Error: p11-kit: shouldn't
be reached at p11_array_push
Sep 13 10:42:04 ip-10-0-40-230 dovecot: auth: Error: p11-kit: shouldn't
be reached at sink_object
Sep 13 10:42:04 ip-10-0-40-230 dovecot: auth: Error: p11-kit: 'attrs !=
NULL' not true at attrs_build
Sep 13 10:42:04 ip-10-0-40-230 dovecot: auth: Error: p11-kit:
'new_memory != NULL' not true at maybe_expand_array
Sep 13 10:42:04 ip-10-0-40-230 dovecot: auth: Error: p11-kit: shouldn't
be reached at p11_array_push
Sep 13 10:42:04 ip-10-0-40-230 dovecot: auth: Error: p11-kit: shouldn't
be reached at sink_object
...
...

Sep 13 10:42:04 ip-10-0-40-230 dovecot: auth: Error: p11-kit:
'new_memory != NULL' not true at maybe_expand_array
Sep 13 10:42:04 ip-10-0-40-230 dovecot: auth: Error: p11-kit: shouldn't
be reached at p11_array_push
Sep 13 10:42:04 ip-10-0-40-230 dovecot: auth: Error: p11-kit: shouldn't
be reached at sink_object
Sep 13 10:42:04 ip-10-0-40-230 dovecot: auth: Error: p11-kit:
'attr->pValue != NULL' not true at attrs_build
Sep 13 10:42:04 ip-10-0-40-230 dovecot: auth: Error: p11-kit:
'new_memory != NULL' not true at maybe_expand_array
Sep 13 10:42:04 ip-10-0-40-230 dovecot: auth: Error: p11-kit: shouldn't
be reached at p11_array_push
Sep 13 10:42:04 ip-10-0-40-230 dovecot: auth: Error: p11-kit: shouldn't
be reached at sink_object
Sep 13 10:42:04 ip-10-0-40-230 dovecot: auth: Error: p11-kit: no
CKA_CLASS attribute found
Sep 13 10:42:04 ip-10-0-40-230 dovecot: auth: Error: p11-kit: couldn't
load file into objects:
/usr/share/pki/ca-trust-source/ca-bundle.trust.p11-kit
Sep 13 10:42:05 ip-10-0-40-230 dovecot: auth-worker: Error: p11-kit:
'attrs != NULL' not true at attrs_build
Sep 13 10:42:05 ip-10-0-40-230 dovecot: auth-worker: Fatal: master:
service(auth-worker): child 14389 killed with signal 11 (core dumps
disabled)
Sep 13 10:42:05 ip-10-0-40-230 dovecot: auth-worker: Error: p11-kit:
'attrs != NULL' not true at attrs_build
Sep 13 10:42:05 ip-10-0-40-230 dovecot: auth-worker: Fatal: master:
service(auth-worker): child 14391 killed with signal 11 (core dumps
disabled)
Sep 13 10:42:05 ip-10-0-40-230 dovecot: auth-worker: Error: p11-kit:
'attrs != NULL' not true at attrs_build
Sep 13 10:42:05 ip-10-0-40-230 dovecot: auth-worker: Fatal: master:
service(auth-worker): child 14393 killed with signal 11 (core dumps
disabled)

why would a hostname change make any difference here .. the certs
specified in dovecot config are all complete in their chain so not sure
what its trying to do ... set hostname back to original works find .. so
something is obviously tied or keyed to hostname though cant find
anything specific

anyone seen anything like this at all ??

rgds

Matt

Re: mdbox+ ALT storage + lazyexpunge

[Matt Bryant]

dumb question ... ignore ... its a different namespace so can have
different location for storage anyhow ..



> Matt Bryant <mailto:m...@the-bryants.net>
> 14 July 2018 at 12:01 pm
> All,
>
> Anyone know that if you setup lazyexpounge with mdbox that as ALT
> storage defined, whether the mail location for namespace used by
> lazyexpunge also need the ALT= specified ??
>
> rgds
>
> Matt

mdbox + alt storage + purge = wtf ...

[Matt Bryant]

All,

Is there a bug in doveadm purge ? Configured alt storage and move
messages in inbox to the alternate path all well and good. However,
after deleting some messages and running a purge on the mailbox found
that all messages got recopied over to primary location. I guess this
maybe because its re-writing the file and ditching the purged mail ..
but shouldnt it keep a track that they were located on alt storage path ??

rgds

Matt

mdbox+ ALT storage + lazyexpunge

[Matt Bryant]

All,

Anyone know that if you setup lazyexpounge with mdbox that as ALT
storage defined, whether the mail location for namespace used by
lazyexpunge also need the ALT= specified ??

rgds

Matt

Re: lazy_expuge + mdbox

[Matt Bryant]

To a degree I agree .. given mdbox doesnt delete until purge .. but then
would be nice if purge took a dovecot search rather than just ditch
everything with refcount=0 ... in that way could say purge everything
thats been marked as delete according to a specified criteria.

rgds

Matt

completly unrelated moved mailserver back home and listed on cyren so
atm getting bounced by dovecot.fi .. have requested de-listing
> Aki Tuomi <mailto:aki.tu...@dovecot.fi>
> 24 May 2018 at 3:35 pm
>
> I am not sure if lazy expunge makes much sense with mdbox, at least if
> you use the same mdbox storage with it. It might make sense with
> separate mdbox/sdbox/something for .EXPUNGED.
>
> Aki
> Matt Bryant <mailto:m...@the-bryants.net>
> 24 May 2018 at 11:53 am
> So moving on from earlier issue now looking at converting mail inbox to
> mdbox and keeping lazy_expunge as mdbox also .. but again seeing some
> wierdness so questions 
>
> 1) When i purge trash folder I seem to get 2 copies of the message in
> .EXPUNGED/Trash folder .. however if I set
> lazy_expunge_only_last_instance = yes then dont seem to get ANY copy ..
> just nothing.
>
> 2) If I go into Trash and delete a message ... then it does not get
> moved .. I assume that a delete in Trash is not actually an expunge on
> TB and hence does get caught by the plugin.
>
> 3) Since mdbox doesnt delte anything anyway until purged would it be
> better not to use lazy-expunge but just have a job that moves any
> messages in mdbox_deleted: storage to the hidden .EXPUNGED namespace ..
> does doveadm move work with mdbox_deleted ???
>
> any help appreciated
>
> rgds
>
> Matt

lazy_expuge + mdbox

[Matt Bryant]

So moving on from earlier issue now looking at converting mail inbox to
mdbox and keeping lazy_expunge as mdbox also .. but again seeing some
wierdness  so questions 

1) When i purge trash folder I seem to get 2 copies of the message in
.EXPUNGED/Trash folder .. however if I set
lazy_expunge_only_last_instance = yes then dont seem to get ANY copy ..
just nothing.

2) If I go into Trash and delete a message ... then it does not get
moved .. I assume that a delete in Trash is not actually an expunge on
TB and hence does get caught by the plugin.

3) Since mdbox doesnt delte anything anyway until purged would it be
better not to use lazy-expunge but just have a job that moves any
messages in mdbox_deleted: storage to the hidden .EXPUNGED namespace ..
does doveadm move work with mdbox_deleted ???

any help appreciated

rgds

Matt

Re: lazy_expunge ...

[Matt Bryant]

Hmm so changed that but whatever I add still get the following pop up ...

doveadm(m...@the-bryants.net): Error: fts: Failed to sync mailbox
.EXPUNGED: Mailbox doesn't exist: .EXPUNGED

should

fts_autoindex_exclude3 = .EXPUNGED

cover the whole namespace or do I need the

fts_autoindex_exclude4 = .EXPUNGED/*


rgds

Matt
> Aki Tuomi <mailto:aki.tu...@dovecot.fi>
> 23 May 2018 at 8:54 pm
>
>
>
> On 23.05.2018 13:46, Matt Bryant wrote:
>> deeper still looks like culprit was
>>
>> fts_enforced = "no"
>>
>> which i removed given default anyway ..
>>
>> still get error
>>
>> [root@stargate dovecot.rawlog]#  doveadm search -u
>> m...@the-bryants.net mailbox .EXPUNGED/* from nore...@apress.com
>> doveadm(m...@the-bryants.net): Error: fts: Failed to sync mailbox
>> .EXPUNGED: Mailbox doesn't exist: .EXPUNGED
>> 6156a416748c035b5c0bef794dd1 1
>>
>> even though in theory have disabled .EXPUNGED from being indexed ..
>>
>> fts_autoindex_exclude = \Junk
>> fts_autoindex_exclude = \Trash
>> fts_autoindex_exclude = .EXPUNGED
>> fts_autoindex_exclude = .EXPUNGED/*
>>
>> but at least dont get internal server error ..
>>
>>
>> rgds
>>
>> Matt
>>
>
> Uh. This is wrong, you only exclude .EXPUNGED/* now
>
> You should use:
>
> fts_autoindex_exclude = \Junk
> fts_autoindex_exclude2 = \Trash
> fts_autoindex_exclude3 = .EXPUNGED
> fts_autoindex_exclude4 = .EXPUNGED/*
>
> Aki
>
>>> Matt Bryant <mailto:m...@the-bryants.net>
>>> 23 May 2018 at 8:29 pm
>>> So digging a little deeper it seems this maybe an issue between
>>> fts/fts_lucene and lazy_expunge .. if I disable the fts plugin
>>>
>>> [root@stargate dovecot.rawlog]#  doveadm search -u
>>> m...@the-bryants.net mailbox .EXPUNGED/Trash from nore...@apress.com
>>> 6156a416748c035b5c0bef794dd1 1
>>>
>>> search works ..
>>>
>>> so begs the question whats fts_lucene doing ...
>>>
>>> rgds
>>>
>>> Matt
>>>
>>> Matt Bryant <mailto:m...@the-bryants.net>
>>> 23 May 2018 at 3:37 pm
>>> So I decided to configure this plugin and take it for a test drive ..
>>> configured using mdbox (main namespace is
>>> Maildir) .. So as per Wiki have the following namespaces defined ...
>>> location is held within db ... but basically Maildir:~/Maildir for inbox
>>> namespace
>>>
>>>  8< ===
>>>
>>> namespace inbox {
>>>
>>> list = yes
>>> type = private
>>> inbox = yes
>>> prefix = INBOX/
>>> separator = /
>>>
>>> mailbox Drafts {
>>> auto = create
>>> special_use = \Drafts
>>> }
>>>
>>> mailbox Spam {
>>> auto = create
>>> special_use = \Junk
>>> }
>>>
>>> mailbox Trash {
>>> auto = create
>>> special_use = \Trash
>>> }
>>>
>>> mailbox Sent {
>>> auto = create
>>> special_use = \Sent
>>> }
>>>
>>> mailbox Archive {
>>> auto = create
>>> special_use = \Archive
>>> }
>>>
>>> }
>>>
>>> namespace lazyexpunge {
>>> prefix = .EXPUNGED/
>>> hidden = yes
>>> list = children
>>> subscriptions = no
>>> separator = /
>>> location =
>>> mdbox:~/mdbox:INDEX=~/mdbox:MAILBOXDIR=expunged:LISTINDEX=expunged.list.index
>>> }
>>>
>>> 
>>>
>>> Can see its working since can see files in the mdbox storage along with
>>> indexs etc ... but there seems little to no documentation on how to
>>> delete from the expunged namespace and atm trying to even search doesnt
>>> seem to work ..
>>>
>>> doveadm -Dvvv search -u m...@the-bryants.net mailbox .EXPUNGED/*
>>> subject space
>>>
>>> gets errors below .. and although says refer to server log .. nothing in
>>> there either .. mail_debug is on ...
>>>
>>> doveadm(m...@the-bryants.net): Debug: fts: Indexes disabled for
>>> namespace ''
>>> doveadm(m...@the-bryants.net): Debug: .EXPUNGED/Trash: Mailbox opened
>>> because: search
>>> doveadm(m...@the-bryants.net): Error: fts: Failed to sync mailbox
>>> .EXPUNGED: Mailbox doesn't exist: .EXPUNGED
>>> doveadm(m...@the-bryants.net): Error: Searching mailbox .EXPUNGED/Trash
>>> fai

Re: lazy_expunge ...

[Matt Bryant]

good point forgot about quota ..

  quota_rule3 = .EXPUNGED:ignore

added .. thanks for that ..

rgds

Matt

> Aki Tuomi <mailto:aki.tu...@dovecot.fi>
> 23 May 2018 at 8:44 pm
>
> You should probably exclude the .DUMPSTER folder from quota and FTS
>
> plugin {
> fts_autoindex_exclude = .DUMPSTER
> quota_ruleN = .DUMPSTER:ignore
> }
> Aki
>
> On 23.05.2018 13:29, Matt Bryant wrote:
>
> Matt Bryant <mailto:m...@the-bryants.net>
> 23 May 2018 at 8:29 pm
> So digging a little deeper it seems this maybe an issue between
> fts/fts_lucene and lazy_expunge .. if I disable the fts plugin
>
> [root@stargate dovecot.rawlog]#  doveadm search -u
> m...@the-bryants.net mailbox .EXPUNGED/Trash from nore...@apress.com
> 6156a416748c035b5c0bef794dd1 1
>
> search works ..
>
> so begs the question whats fts_lucene doing ...
>
> rgds
>
> Matt
>
> Matt Bryant <mailto:m...@the-bryants.net>
> 23 May 2018 at 3:37 pm
> So I decided to configure this plugin and take it for a test drive ..
> configured using mdbox (main namespace is
> Maildir) .. So as per Wiki have the following namespaces defined ...
> location is held within db ... but basically Maildir:~/Maildir for inbox
> namespace
>
>  8< ===
>
> namespace inbox {
>
> list = yes
> type = private
> inbox = yes
> prefix = INBOX/
> separator = /
>
> mailbox Drafts {
> auto = create
> special_use = \Drafts
> }
>
> mailbox Spam {
> auto = create
> special_use = \Junk
> }
>
> mailbox Trash {
> auto = create
> special_use = \Trash
> }
>
> mailbox Sent {
> auto = create
> special_use = \Sent
> }
>
> mailbox Archive {
> auto = create
> special_use = \Archive
> }
>
> }
>
> namespace lazyexpunge {
> prefix = .EXPUNGED/
> hidden = yes
> list = children
> subscriptions = no
> separator = /
> location =
> mdbox:~/mdbox:INDEX=~/mdbox:MAILBOXDIR=expunged:LISTINDEX=expunged.list.index
> }
>
> 
>
> Can see its working since can see files in the mdbox storage along with
> indexs etc ... but there seems little to no documentation on how to
> delete from the expunged namespace and atm trying to even search doesnt
> seem to work ..
>
> doveadm -Dvvv search -u m...@the-bryants.net mailbox .EXPUNGED/*
> subject space
>
> gets errors below .. and although says refer to server log .. nothing in
> there either .. mail_debug is on ...
>
> doveadm(m...@the-bryants.net): Debug: fts: Indexes disabled for
> namespace ''
> doveadm(m...@the-bryants.net): Debug: .EXPUNGED/Trash: Mailbox opened
> because: search
> doveadm(m...@the-bryants.net): Error: fts: Failed to sync mailbox
> .EXPUNGED: Mailbox doesn't exist: .EXPUNGED
> doveadm(m...@the-bryants.net): Error: Searching mailbox .EXPUNGED/Trash
> failed: Internal error occurred. Refer to server log for more
> information. [2018-05-23 15:07:21]
> doveadm(m...@the-bryants.net): Debug: .EXPUNGED/Trash/cobbler: Mailbox
> opened because: search
> doveadm(m...@the-bryants.net): Error: fts: Failed to sync mailbox
> .EXPUNGED: Mailbox doesn't exist: .EXPUNGED
> doveadm(m...@the-bryants.net): Error: Searching mailbox
> .EXPUNGED/Trash/cobbler failed: Internal error occurred. Refer to server
> log for more information. [2018-05-23 15:07:21]
>
>
> Any ideas is there any better doco around this plugin and use etc ...
>
> thanks & rgds
>
> Matt

Re: lazy_expunge ...

[Matt Bryant]

deeper still looks like culprit was

fts_enforced = "no"

which i removed given default anyway ..

still get error

[root@stargate dovecot.rawlog]#  doveadm search -u m...@the-bryants.net
mailbox .EXPUNGED/* from nore...@apress.com
doveadm(m...@the-bryants.net): Error: fts: Failed to sync mailbox
.EXPUNGED: Mailbox doesn't exist: .EXPUNGED
6156a416748c035b5c0bef794dd1 1

even though in theory have disabled .EXPUNGED from being indexed ..

fts_autoindex_exclude = \Junk
fts_autoindex_exclude = \Trash
fts_autoindex_exclude = .EXPUNGED
fts_autoindex_exclude = .EXPUNGED/*

but at least dont get internal server error ..


rgds

Matt

> Matt Bryant <mailto:m...@the-bryants.net>
> 23 May 2018 at 8:29 pm
> So digging a little deeper it seems this maybe an issue between
> fts/fts_lucene and lazy_expunge .. if I disable the fts plugin
>
> [root@stargate dovecot.rawlog]#  doveadm search -u
> m...@the-bryants.net mailbox .EXPUNGED/Trash from nore...@apress.com
> 6156a416748c035b5c0bef794dd1 1
>
> search works ..
>
> so begs the question whats fts_lucene doing ...
>
> rgds
>
> Matt
>
> Matt Bryant <mailto:m...@the-bryants.net>
> 23 May 2018 at 3:37 pm
> So I decided to configure this plugin and take it for a test drive ..
> configured using mdbox (main namespace is
> Maildir) .. So as per Wiki have the following namespaces defined ...
> location is held within db ... but basically Maildir:~/Maildir for inbox
> namespace
>
>  8< ===
>
> namespace inbox {
>
> list = yes
> type = private
> inbox = yes
> prefix = INBOX/
> separator = /
>
> mailbox Drafts {
> auto = create
> special_use = \Drafts
> }
>
> mailbox Spam {
> auto = create
> special_use = \Junk
> }
>
> mailbox Trash {
> auto = create
> special_use = \Trash
> }
>
> mailbox Sent {
> auto = create
> special_use = \Sent
> }
>
> mailbox Archive {
> auto = create
> special_use = \Archive
> }
>
> }
>
> namespace lazyexpunge {
> prefix = .EXPUNGED/
> hidden = yes
> list = children
> subscriptions = no
> separator = /
> location =
> mdbox:~/mdbox:INDEX=~/mdbox:MAILBOXDIR=expunged:LISTINDEX=expunged.list.index
> }
>
> 
>
> Can see its working since can see files in the mdbox storage along with
> indexs etc ... but there seems little to no documentation on how to
> delete from the expunged namespace and atm trying to even search doesnt
> seem to work ..
>
> doveadm -Dvvv search -u m...@the-bryants.net mailbox .EXPUNGED/*
> subject space
>
> gets errors below .. and although says refer to server log .. nothing in
> there either .. mail_debug is on ...
>
> doveadm(m...@the-bryants.net): Debug: fts: Indexes disabled for
> namespace ''
> doveadm(m...@the-bryants.net): Debug: .EXPUNGED/Trash: Mailbox opened
> because: search
> doveadm(m...@the-bryants.net): Error: fts: Failed to sync mailbox
> .EXPUNGED: Mailbox doesn't exist: .EXPUNGED
> doveadm(m...@the-bryants.net): Error: Searching mailbox .EXPUNGED/Trash
> failed: Internal error occurred. Refer to server log for more
> information. [2018-05-23 15:07:21]
> doveadm(m...@the-bryants.net): Debug: .EXPUNGED/Trash/cobbler: Mailbox
> opened because: search
> doveadm(m...@the-bryants.net): Error: fts: Failed to sync mailbox
> .EXPUNGED: Mailbox doesn't exist: .EXPUNGED
> doveadm(m...@the-bryants.net): Error: Searching mailbox
> .EXPUNGED/Trash/cobbler failed: Internal error occurred. Refer to server
> log for more information. [2018-05-23 15:07:21]
>
>
> Any ideas is there any better doco around this plugin and use etc ...
>
> thanks & rgds
>
> Matt

Re: lazy_expunge ...

[Matt Bryant]

So digging a little deeper it seems this maybe an issue between
fts/fts_lucene and lazy_expunge .. if I disable the fts plugin

[root@stargate dovecot.rawlog]#  doveadm search -u m...@the-bryants.net
mailbox .EXPUNGED/Trash from nore...@apress.com
6156a416748c035b5c0bef794dd1 1

search works ..

so begs the question whats fts_lucene doing ...

rgds

Matt
> Matt Bryant <mailto:m...@the-bryants.net>
> 23 May 2018 at 3:37 pm
> So I decided to configure this plugin and take it for a test drive ..
> configured using mdbox (main namespace is
> Maildir) .. So as per Wiki have the following namespaces defined ...
> location is held within db ... but basically Maildir:~/Maildir for inbox
> namespace
>
>  8< ===
>
> namespace inbox {
>
> list = yes
> type = private
> inbox = yes
> prefix = INBOX/
> separator = /
>
> mailbox Drafts {
> auto = create
> special_use = \Drafts
> }
>
> mailbox Spam {
> auto = create
> special_use = \Junk
> }
>
> mailbox Trash {
> auto = create
> special_use = \Trash
> }
>
> mailbox Sent {
> auto = create
> special_use = \Sent
> }
>
> mailbox Archive {
> auto = create
> special_use = \Archive
> }
>
> }
>
> namespace lazyexpunge {
> prefix = .EXPUNGED/
> hidden = yes
> list = children
> subscriptions = no
> separator = /
> location =
> mdbox:~/mdbox:INDEX=~/mdbox:MAILBOXDIR=expunged:LISTINDEX=expunged.list.index
> }
>
> 
>
> Can see its working since can see files in the mdbox storage along with
> indexs etc ... but there seems little to no documentation on how to
> delete from the expunged namespace and atm trying to even search doesnt
> seem to work ..
>
> doveadm -Dvvv search -u m...@the-bryants.net mailbox .EXPUNGED/*
> subject space
>
> gets errors below .. and although says refer to server log .. nothing in
> there either .. mail_debug is on ...
>
> doveadm(m...@the-bryants.net): Debug: fts: Indexes disabled for
> namespace ''
> doveadm(m...@the-bryants.net): Debug: .EXPUNGED/Trash: Mailbox opened
> because: search
> doveadm(m...@the-bryants.net): Error: fts: Failed to sync mailbox
> .EXPUNGED: Mailbox doesn't exist: .EXPUNGED
> doveadm(m...@the-bryants.net): Error: Searching mailbox .EXPUNGED/Trash
> failed: Internal error occurred. Refer to server log for more
> information. [2018-05-23 15:07:21]
> doveadm(m...@the-bryants.net): Debug: .EXPUNGED/Trash/cobbler: Mailbox
> opened because: search
> doveadm(m...@the-bryants.net): Error: fts: Failed to sync mailbox
> .EXPUNGED: Mailbox doesn't exist: .EXPUNGED
> doveadm(m...@the-bryants.net): Error: Searching mailbox
> .EXPUNGED/Trash/cobbler failed: Internal error occurred. Refer to server
> log for more information. [2018-05-23 15:07:21]
>
>
> Any ideas is there any better doco around this plugin and use etc ...
>
> thanks & rgds
>
> Matt

lazy_expunge ...

[Matt Bryant]

So I decided to configure this plugin and take it for a test drive ..
configured using mdbox (main namespace is
Maildir) .. So as per Wiki  have the following namespaces defined ...
location is held within db ... but basically Maildir:~/Maildir for inbox
namespace

 8< ===

namespace inbox {

list = yes
type = private
inbox = yes
prefix = INBOX/
separator = /

  mailbox Drafts {
auto = create
special_use = \Drafts
  }

  mailbox Spam {
auto = create
special_use = \Junk
  }

  mailbox Trash {
auto = create
special_use = \Trash
  }

  mailbox Sent {
auto = create
special_use = \Sent
  }

  mailbox Archive {
auto = create
special_use = \Archive
  }

}

namespace lazyexpunge {
prefix = .EXPUNGED/
hidden = yes
list = children
subscriptions = no
separator = /
location =
mdbox:~/mdbox:INDEX=~/mdbox:MAILBOXDIR=expunged:LISTINDEX=expunged.list.index
}



Can see its working since can see files in the mdbox storage along with
indexs etc ... but there seems little to no documentation on how to
delete from the expunged namespace and atm trying to even search doesnt
seem to work ..

doveadm -Dvvv search -u m...@the-bryants.net mailbox  .EXPUNGED/*
subject space

gets errors below .. and although says refer to server log .. nothing in
there either .. mail_debug is on ...

doveadm(m...@the-bryants.net): Debug: fts: Indexes disabled for namespace ''
doveadm(m...@the-bryants.net): Debug: .EXPUNGED/Trash: Mailbox opened
because: search
doveadm(m...@the-bryants.net): Error: fts: Failed to sync mailbox
.EXPUNGED: Mailbox doesn't exist: .EXPUNGED
doveadm(m...@the-bryants.net): Error: Searching mailbox .EXPUNGED/Trash
failed: Internal error occurred. Refer to server log for more
information. [2018-05-23 15:07:21]
doveadm(m...@the-bryants.net): Debug: .EXPUNGED/Trash/cobbler: Mailbox
opened because: search
doveadm(m...@the-bryants.net): Error: fts: Failed to sync mailbox
.EXPUNGED: Mailbox doesn't exist: .EXPUNGED
doveadm(m...@the-bryants.net): Error: Searching mailbox
.EXPUNGED/Trash/cobbler failed: Internal error occurred. Refer to server
log for more information. [2018-05-23 15:07:21]


Any ideas is there any better doco around this plugin and use etc ...

thanks & rgds

Matt

Re: dsync migration from maildir layout=fs to maildir seperator / - folders or inbox but not both

[Matt Bryant]

imapc_prefix doesnt appear to be a valid option any more ... just gets a
config error .. on the otherhand updated the src dovecot to have an
explicitly defined named space which does seem to be working ..


the namespace i created was pretty much default ie ..

namespace inbox {
  type = private
  separator = .
  prefix = INBOX.
  location = maildir:/var/vmail/%d/%n/Maildir:LAYOUT=fs
  inbox = yes
  hidden = no
  list = yes
  subscriptions = yes
}

but by doing this folders became INBOX.folder.subfolder instead of
/Folder/subfolder so wondering if something in imapc does not like
default namespace thats created and cant decide whats inbox/folders ...

rgds

Matt
> Sami Ketola <mailto:sami.ket...@dovecot.fi>
> 13 May 2018 at 9:11 pm
>
> Try setting imapc_prefix=INBOX instead of imapc_list_prefix.
>
> Sami
>
> Matt Bryant <mailto:m...@the-bryants.net>
> 13 May 2018 at 9:07 pm
> Sigh ... ok so had if statement in SQL land that appended LAYOUT=fs if
> Maildir which explains the non-converting folder layout but NOT why
> the process is not picking up the actual contents of INBOX ...
>
> Matt
>
>
> Matt Bryant <mailto:m...@the-bryants.net>
> 13 May 2018 at 8:01 pm
> Am attempting to bring my mailserver back to home server post house
> move and whilst I could use rsync have been trying to get to grips
> with dsync.
>
> However, I seem to have an issue - my current server specifies
> mail_location as
>
>  maildir:/var/vmail/%d/%n/Maildir:LAYOUT=fs
>
> and my imapc config on new server is as follows ...
>
> imapc_host = old.mailserver.host
> imapc_user = %u
> imapc_master_user = master
> imapc_password = **
> imapc_features = $imapc_features rfc822.size fetch-headers
> imapc_list_prefix = INBOX
> imapc_port = 993
> imapc_ssl = imaps
> imapc_ssl_verify = no
>
> the trouble is if I leave
>
> imapc_list_prefix = INBOX
>
> I get the inbox only - no folders ... If I comment this out delete
> maildir on target and  re-run .. I get the folders but no inbox but
> folders are in fs layout format rather than being converted to
> INBOX.folder.subfolder
>
> bug perhaps  or am I doing something wrong ???
>
> src dovecot version 2.0.9
> tgt dovecot version 2.2.33.2
>
> Any ideas would be appreciated ...
>
> rgds
>
> Matt

-- 

<https://www.postbox-inc.com/?utm_source=email&utm_medium=siglink&utm_campaign=reach>

Re: dsync migration from maildir layout=fs to maildir seperator / - folders or inbox but not both

[Matt Bryant]

Sigh ... ok so had if statement in SQL land that appended LAYOUT=fs if
Maildir which explains the non-converting folder layout but NOT why the
process is not picking up the actual contents of INBOX ...

Matt

> Matt Bryant <mailto:m...@the-bryants.net>
> 13 May 2018 at 8:01 pm
> Am attempting to bring my mailserver back to home server post house
> move and whilst I could use rsync have been trying to get to grips
> with dsync.
>
> However, I seem to have an issue - my current server specifies
> mail_location as
>
>  maildir:/var/vmail/%d/%n/Maildir:LAYOUT=fs
>
> and my imapc config on new server is as follows ...
>
> imapc_host = old.mailserver.host
> imapc_user = %u
> imapc_master_user = master
> imapc_password = **
> imapc_features = $imapc_features rfc822.size fetch-headers
> imapc_list_prefix = INBOX
> imapc_port = 993
> imapc_ssl = imaps
> imapc_ssl_verify = no
>
> the trouble is if I leave
>
> imapc_list_prefix = INBOX
>
> I get the inbox only - no folders ... If I comment this out delete
> maildir on target and  re-run .. I get the folders but no inbox but
> folders are in fs layout format rather than being converted to
> INBOX.folder.subfolder
>
> bug perhaps  or am I doing something wrong ???
>
> src dovecot version 2.0.9
> tgt dovecot version 2.2.33.2
>
> Any ideas would be appreciated ...
>
> rgds
>
> Matt

-- 

<https://www.postbox-inc.com/?utm_source=email&utm_medium=siglink&utm_campaign=reach>

dsync migration from maildir layout=fs to maildir seperator / - folders or inbox but not both

[Matt Bryant]

Am attempting to bring my mailserver back to home server post house move
and whilst I could use rsync have been trying to get to grips with dsync.

However, I seem to have an issue - my current server specifies
mail_location as

 maildir:/var/vmail/%d/%n/Maildir:LAYOUT=fs

and my imapc config on new server is as follows ...

imapc_host = old.mailserver.host
imapc_user = %u
imapc_master_user = master
imapc_password = **
imapc_features = $imapc_features rfc822.size fetch-headers
imapc_list_prefix = INBOX
imapc_port = 993
imapc_ssl = imaps
imapc_ssl_verify = no

the trouble is if I leave

imapc_list_prefix = INBOX

I get the inbox only - no folders ... If I comment this out delete
maildir on target and  re-run .. I get the folders but no inbox but
folders are in fs layout format rather than being converted to
INBOX.folder.subfolder

bug perhaps  or am I doing something wrong ???

src dovecot version 2.0.9
tgt dovecot version 2.2.33.2

Any ideas would be appreciated ...

rgds

Matt
-- 

Alt Storage and dsync

[Matt Bryant]

Stupid question but if dovecot mdbox is configured using alt path
storage on a bi directional dysnc replicated pair of servers ... will
the doveadm altmove need to be run on both halves of the cluster ???

rgds

Matt
-- 

Fwd: Re: 2.2.33.2 - Virtual Mailbox Issue bug maybe ?

[Matt Bryant]

Thought I would get it another send and see if anyone out there has come
up with any ideas or thoughts on this .. certainly I couldnt seem to
replicate in 2.2.31 so am wondering looking at the change log if one of
the virtual 'fixes' in 2.2.32 .. maybe broke something else.

Either way can anyone confirm that

'inthread refs x-mailbox INBOX

should filter all message according to thread - seems to indicate that
in the 'gmail' example given on the dovecot wiki ...

rgds

Matt


> *From:* Matt Bryant 
> *Date:* 19 February 2018 at 9:21 am
> *To:* dovecot@dovecot.org
> *Subject:* 2.2.33.2 - Virtual Mailbox Issue bug maybe ?
> hmm down graded to 2.2.19 and dont see the issue .. other issues
> particularly a segfault in libstorage to do with virtual .. but at
> least mails seem to be appearing correctly and cant seem to replicate
> what I have seen in 2.2.33.2
>
> as an aside .. outlook on mac doesnt appear to like 2.2.33.2 either ..
> seeing same kind of issue where some messages not appearing in inbox
> .. but could be some outlook/mac bug not catered for perhaps given no
> issues in thunderbird and some other 3rd party clients tested.
>
> rgds
>
> Matt
>
>
> *From:* Matt Bryant 
> *Date:* 18 February 2018 at 9:14 am
> *To:* dovecot@dovecot.org
> *Subject:* 2.2.33.2 - Virtual Mailbox Issue bug maybe ?
> Been experimenting with virtual mailbox and have a question :-
>
> I have defined 2 virtual mailboxes 'all' and 'combined' as follows ...
>
>
> virtual/all/dovecot-virtual
>
> virtual/combined
>   inthread refs x-mailbox INBOX
>
> virtual/combined/dovecot-virtual
>
> !INBOX
> INBOX/Sent
> INBOX/Drafts
>   all
>
> When new mail comes I would assume that both virtual mailboxes should
> actually see the message but whilst an imap search command gets a
> result in the virtual/combined for some reason not every message is
> seen in virtual all - so was wondering what exactly does
>
> 'inthread refs x-mailbox INBOX'
>
> do ? I would have thought new mail to inbox would just have a thread
> of 1 and be seen. If i remove the indexes from virtual then the
> messages appear so am thinking something not quite triggering an
> action. Any ideas ? What can I do to try and track down where problem
> lies or am i misunderstanding what 'inthread refs x-mailbox INBOX'
> should do 
>
>
> == 8< ==
>
> # doveconf -n
> # 2.2.33.2 (d6601f4ec): /etc/dovecot/dovecot.conf
> # Pigeonhole version 0.4.21 (92477967)
> doveconf: Warning: service auth { client_limit=5000 } is lower than
> required under max. load (26096)
> doveconf: Warning: service anvil { client_limit=5000 } is lower than
> required under max. load (9715)
> # OS: Linux 3.10.0-693.11.6.el7.x86_64 x86_64 CentOS Linux release
> 7.3.1611 (Core)
> auth_cache_negative_ttl = 2 mins
> auth_cache_ttl = 3 mins
> auth_master_user_separator = *
> auth_mechanisms = plain login
> auth_verbose = yes
> auth_worker_max_count = 100
> default_client_limit = 8192
> default_process_limit = 8192
> default_vsz_limit = 512 M
> disable_plaintext_auth = no
> doveadm_password =  # hidden, use -P to show it
> imap_client_workarounds = delay-newmail tb-extra-mailbox-sep tb-lsub-flags
> imap_idle_notify_interval = 1 mins
> login_greeting = IMAP/POP3 ready - us11-001dm
> mail_attachment_dir = /var/mail/sis/%Ld
> mail_cache_min_mail_count = 5
> mail_debug = yes
> mail_plugins = " quota virtual"
> mailbox_list_index = yes
> managesieve_notify_capability = mailto
> managesieve_sieve_capability = fileinto reject envelope
> encoded-character vacation subaddress comparator-i;ascii-numeric
> relational regex imap4flags copy include variables body enotify
> environment mailbox date index ihave duplicate mime foreverypart
> extracttext vacation-seconds spamtest spamtestplus editheader
> mbox_write_locks = fcntl
> mmap_disable = yes
> namespace inbox {
>   inbox = yes
>   list = yes
>   location =
>   mailbox Archive {
> auto = create
> special_use = \Archive
>   }
>   mailbox Drafts {
> auto = create
> special_use = \Drafts
>   }
>   mailbox Sent {
> auto = create
> special_use = \Sent
>   }
>   mailbox Spam {
> auto = create
> special_use = \Junk
>   }
>   mailbox Trash {
> auto = create
> special_use = \Trash
>   }
>   prefix = INBOX/
>   separator = /
>   type = private
> }
> namespace virtual {
>   list = no
>   location = virtual:/var/lib/mail/dovecot/virtual:INDEX=~/virtual
>   prefix = virtual/
>   separator = /
>   type = private
> }
> passdb {
>   args = /etc/dovecot/sql_users.conf
&g

Re: 2.2.33.2 - Virtual Mailbox Issue bug maybe ?

[Matt Bryant]

hmm down graded to 2.2.19 and dont see the issue .. other issues
particularly a segfault in libstorage to do with virtual .. but at least
mails seem to be appearing correctly and cant seem to replicate what I
have seen in 2.2.33.2

as an aside .. outlook on mac doesnt appear to like 2.2.33.2 either ..
seeing same kind of issue where some messages not appearing in inbox ..
but could be some outlook/mac bug not catered for perhaps given no
issues in thunderbird and some other 3rd party clients tested.

rgds

Matt

> Matt Bryant <mailto:m...@the-bryants.net>
> 18 February 2018 at 9:14 am
> Been experimenting with virtual mailbox and have a question :-
>
> I have defined 2 virtual mailboxes 'all' and 'combined' as follows ...
>
>
> virtual/all/dovecot-virtual
>
> virtual/combined
>   inthread refs x-mailbox INBOX
>
> virtual/combined/dovecot-virtual
>
> !INBOX
> INBOX/Sent
> INBOX/Drafts
>   all

-- 

<https://www.postbox-inc.com/?utm_source=email&utm_medium=siglink&utm_campaign=reach>

Re: 2.2.33.2 - Virtual Mailbox Issue bug maybe ?

[Matt Bryant]

So removed the ! since pretty much uneeded but get the same behaviour
... send a mail to the underlying inbox .. gets found in
virtual/combined but not virtual/all  remove the indexes from
virtual/all and re-login .. they get recreated and bingo the new message
is now found (shown below) ...

even tried running a doveadm index on virtual/all ... but that didnt do
anything either 


a2 select virtual/all
* FLAGS (\Answered \Flagged \Deleted \Seen \Draft unknown-0 $Forwarded
home $MDNSent personal)
* OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft unknown-0
$Forwarded home $MDNSent personal \*)] Flags permitted.
* 30579 EXISTS
* 30579 RECENT
* OK [UNSEEN 139] First unseen.
* OK [UIDVALIDITY 1518990022] UIDs valid
* OK [UIDNEXT 30580] Predicted next UID
* OK [HIGHESTMODSEQ 8] Highest
a2 OK [READ-WRITE] Select completed (0.031 + 0.000 + 0.030 secs).
? search header Message-ID <5a89f365.1050...@the-bryants.net>
* SEARCH
? OK Search completed (0.254 + 0.000 + 0.253 secs).
a2 select virtual/combined
* OK [CLOSED] Previous mailbox closed.
* FLAGS (\Answered \Flagged \Deleted \Seen \Draft unknown-0 $Forwarded
home $MDNSent personal)
* OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft unknown-0
$Forwarded home $MDNSent personal \*)] Flags permitted.
* 30999 EXISTS
* 30999 RECENT
* OK [UNSEEN 141] First unseen.
* OK [UIDVALIDITY 1518990022] UIDs valid
* OK [UIDNEXT 31000] Predicted next UID
* OK [HIGHESTMODSEQ 10] Highest
a2 OK [READ-WRITE] Select completed (0.015 + 0.000 + 0.015 secs).
? search header Message-ID <5a89f365.1050...@the-bryants.net
* SEARCH 30999
? OK Search completed (0.217 + 0.000 + 0.216 secs).


if i delete the index files for virtual/all and then re-login and search ...

a2 select virtual/all
* FLAGS (\Answered \Flagged \Deleted \Seen \Draft unknown-0 $Forwarded
home $MDNSent personal)
* OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft unknown-0
$Forwarded home $MDNSent personal \*)] Flags permitted.
* 30581 EXISTS
* 30581 RECENT
* OK [UNSEEN 139] First unseen.
* OK [UIDVALIDITY 1518990876] UIDs valid
* OK [UIDNEXT 30582] Predicted next UID
* OK [HIGHESTMODSEQ 7] Highest
a2 OK [READ-WRITE] Select completed (0.032 + 0.000 + 0.031 secs).
? search header Message-ID <5a89f365.1050...@the-bryants.net>
* SEARCH 30581
? OK Search completed (0.274 + 0.000 + 0.273 secs).

message is found ...



> Matt Bryant <mailto:m...@the-bryants.net>
> 19 February 2018 at 7:24 am
> So virtual/combined was meant to combine INBOX/Sent/Drafts and
> virtual/all display them in a threaded manner .. at least that was the
> intention ...
>
> rgds
>
> Matt
>
> Aki Tuomi <mailto:aki.tu...@dovecot.fi>
> 18 February 2018 at 11:10 pm
>
> Uhm.
>
> combined folder seems to use INBOX for saving mails and contains mails
> from Sent and Drafts.
>
> Not sure what your all folder is supposed to do.
>
> Aki
>
>
> On 2018-02-18 01:14, Matt Bryant wrote:
>
> Matt Bryant <mailto:m...@the-bryants.net>
> 18 February 2018 at 9:14 am
> Been experimenting with virtual mailbox and have a question :-
>
> I have defined 2 virtual mailboxes 'all' and 'combined' as follows ...
>
>
> virtual/all/dovecot-virtual
>
> virtual/combined
>   inthread refs x-mailbox INBOX
>
> virtual/combined/dovecot-virtual
>
> !INBOX
> INBOX/Sent
> INBOX/Drafts
>   all

-- 

<https://www.postbox-inc.com/?utm_source=email&utm_medium=siglink&utm_campaign=reach>

Re: 2.2.33.2 - Virtual Mailbox Issue bug maybe ?

[Matt Bryant]

actually probably could ditch the ! out of combined 'cos i dont think
need to save anything .. just view

> Matt Bryant <mailto:m...@the-bryants.net>
> 19 February 2018 at 7:24 am
> So virtual/combined was meant to combine INBOX/Sent/Drafts and
> virtual/all display them in a threaded manner .. at least that was the
> intention ...
>
> rgds
>
> Matt
>
> Aki Tuomi <mailto:aki.tu...@dovecot.fi>
> 18 February 2018 at 11:10 pm
>
> Uhm.
>
> combined folder seems to use INBOX for saving mails and contains mails
> from Sent and Drafts.
>
> Not sure what your all folder is supposed to do.
>
> Aki
>
>
> On 2018-02-18 01:14, Matt Bryant wrote:
>
> Matt Bryant <mailto:m...@the-bryants.net>
> 18 February 2018 at 9:14 am
> Been experimenting with virtual mailbox and have a question :-
>
> I have defined 2 virtual mailboxes 'all' and 'combined' as follows ...
>
>
> virtual/all/dovecot-virtual
>
> virtual/combined
>   inthread refs x-mailbox INBOX
>
> virtual/combined/dovecot-virtual
>
> !INBOX
> INBOX/Sent
> INBOX/Drafts
>   all

-- 

<https://www.postbox-inc.com/?utm_source=email&utm_medium=siglink&utm_campaign=reach>

Re: 2.2.33.2 - Virtual Mailbox Issue bug maybe ?

[Matt Bryant]

So virtual/combined was meant to combine INBOX/Sent/Drafts and
virtual/all display them in a threaded manner .. at least that was the
intention ...

rgds

Matt
> Aki Tuomi <mailto:aki.tu...@dovecot.fi>
> 18 February 2018 at 11:10 pm
>
> Uhm.
>
> combined folder seems to use INBOX for saving mails and contains mails
> from Sent and Drafts.
>
> Not sure what your all folder is supposed to do.
>
> Aki
>
>
> On 2018-02-18 01:14, Matt Bryant wrote:
>
> Matt Bryant <mailto:m...@the-bryants.net>
> 18 February 2018 at 9:14 am
> Been experimenting with virtual mailbox and have a question :-
>
> I have defined 2 virtual mailboxes 'all' and 'combined' as follows ...
>
>
> virtual/all/dovecot-virtual
>
> virtual/combined
>   inthread refs x-mailbox INBOX
>
> virtual/combined/dovecot-virtual
>
> !INBOX
> INBOX/Sent
> INBOX/Drafts
>   all

-- 

<https://www.postbox-inc.com/?utm_source=email&utm_medium=siglink&utm_campaign=reach>

Re: Dovecot says the message was stored into a Spam mailbox but it's not there

[Matt Bryant]

In theory delivery (for Maildir) should be to 'new' directory

ie: /var/vmail/domain/user/Maildir/.Spam/new/

only when a message is read/downloaded should it migrate to

/var/vmail/domain/user/Maildir/.Spam/cur/

upping the mail_debug setting to true should give you more information
around what sieve is doing and what path mail stored in.

rgds

Matt

> Kris 
> 18 February 2018 at 4:49 pm
>
> Hi,
>
> I have a problem with storing messages into a Spam folder by dovecot.
> Log says that message was stored into Spam mailbox but when I check
> there is no message in users's Spam mailbox.
>
> cat /var/log/mail.log | grep sieve | grep Spam :
>
> *Feb 17 05:09:11 mailserver dovecot: lda(user@domain): sieve:
> msgid=: stored mail
> into mailbox 'Spam'*
>
>
> Where are these messages if not in the Spam folder where they should
> be?  /var/vmail/domain/user/Maildir/.Spam/cur/
>
>
>
>
> * dovecot -n*
> # 2.2.22 (fe789d2): /etc/dovecot/dovecot.conf
> # Pigeonhole version 0.4.13 (7b14904)
> # OS: Linux 4.4.0-104-generic x86_64 Ubuntu 16.04.3 LTS ext4
> auth_gssapi_hostname = mailserver.belpol.local
> auth_krb5_keytab = /etc/dovecot/dovecot.keytab
> auth_mechanisms = gssapi plain
> first_valid_gid = 117
> first_valid_uid = 111
> listen = *
> log_timestamp = "%Y-%m-%d %H:%M:%S "
> mail_gid = ebox
> mail_home = /var/vmail/%d/%u
> mail_location = maildir:/var/vmail/%d/%u/Maildir
> mail_uid = ebox
> managesieve_notify_capability = mailto
> managesieve_sieve_capability = fileinto reject envelope
> encoded-character vacation subaddress comparator-i;ascii-numeric
> relational regex imap4flags copy include variables body enotify
> environment mailbox date index ihave duplicate mime foreverypart
> extracttext
> namespace inbox {
>   inbox = yes
>   location =
>   mailbox Drafts {
> auto = subscribe
> special_use = \Drafts
>   }
>   mailbox Sent {
> auto = subscribe
> special_use = \Sent
>   }
>   mailbox "Sent Messages" {
> auto = no
> special_use = \Sent
>   }
>   mailbox Spam {
> auto = create
> special_use = \Junk
>   }
>   mailbox Trash {
> auto = subscribe
> special_use = \Trash
>   }
>   prefix =
>   separator = /
>   subscriptions = yes
> }
> passdb {
>   args = /etc/dovecot/dovecot-ldap.conf
>   driver = ldap
> }
> plugin {
>   quota = maildir:User quota
>   quota_rule = *:storage=0
>   sieve = /var/vmail/%Ld/%Ln/sieve-script
>   sieve_before = /var/vmail/sieve/default.sieve
>   sieve_default = /var/vmail/sieve/default.sieve
>   sieve_dir = /var/vmail/%Ld/%Ln
>   sieve_global_path = /var/vmail/sieve/default.sieve
>   sieve_storage = /var/vmail/%Ld/%Ln
> }
> protocols = pop3 imap sieve
> service auth {
>   executable = /usr/lib/dovecot/auth
>   unix_listener /var/spool/postfix/private/auth {
> group = postfix
> mode = 0660
> user = postfix
>   }
>   unix_listener auth-master {
> group = ebox
> mode = 0600
> user = ebox
>   }
> }
> service imap-login {
>   inet_listener imap {
> address = *
> port = 143
>   }
>   inet_listener imaps {
> address = *
> port = 993
>   }
> }
> service lmtp {
>   unix_listener /var/spool/postfix/private/dovecot-lmtp {
> group = postfix
> mode = 0666
> user = postfix
>   }
> }
> service managesieve-login {
>   inet_listener sieve {
> port = 4190
>   }
> }
> service pop3-login {
>   inet_listener pop3 {
> port = 110
>   }
>   inet_listener pop3s {
> port = 995
>   }
> }
> ssl_cert =  ssl_key =  userdb {
>   args = /etc/dovecot/dovecot-ldap.conf
>   default_fields = uid=111 gid=117
>   driver = ldap
> }
> protocol imap {
>   mail_max_userip_connections = 20
>   mail_plugins = quota imap_quota
> }
> protocol pop3 {
>   mail_plugins = quota
> }
> protocol lda {
>   auth_socket_path = /var/run/dovecot/auth-master
>   mail_plugins = sieve quota
>   postmaster_address = postmaster
> }
>
>
>
> *default.sieve:*
>
> require ["fileinto"];
> # rule:[SPAM]
> if header :contains "X-Spam-Flag" "YES" {
>  fileinto "Spam";
> }
> if header :contains "subject" ["***UNCHECKED***", "SPAM"] {
>  fileinto "Spam";
> }
>
>
>
>
>
>

-- 

2.2.33.2 - Virtual Mailbox Issue bug maybe ?

[Matt Bryant]

Been experimenting with virtual mailbox and have a question :-

I have defined 2 virtual mailboxes 'all' and 'combined' as follows ...


virtual/all/dovecot-virtual

virtual/combined
  inthread refs x-mailbox INBOX

virtual/combined/dovecot-virtual

!INBOX
INBOX/Sent
INBOX/Drafts
  all

When new mail comes I would assume that both virtual mailboxes should
actually see the message but whilst an imap search command gets a result
in the virtual/combined for some reason not every message is seen in
virtual all - so was wondering what exactly does

'inthread refs x-mailbox INBOX'

do ? I would have thought new mail to inbox would just have a thread of
1 and be seen. If i remove the indexes from virtual then the messages
appear so am thinking something not quite triggering an action. Any
ideas ? What can I do to try and track down where problem lies or am i
misunderstanding what 'inthread refs x-mailbox INBOX' should do 


== 8< ==

# doveconf -n
# 2.2.33.2 (d6601f4ec): /etc/dovecot/dovecot.conf
# Pigeonhole version 0.4.21 (92477967)
doveconf: Warning: service auth { client_limit=5000 } is lower than
required under max. load (26096)
doveconf: Warning: service anvil { client_limit=5000 } is lower than
required under max. load (9715)
# OS: Linux 3.10.0-693.11.6.el7.x86_64 x86_64 CentOS Linux release
7.3.1611 (Core)
auth_cache_negative_ttl = 2 mins
auth_cache_ttl = 3 mins
auth_master_user_separator = *
auth_mechanisms = plain login
auth_verbose = yes
auth_worker_max_count = 100
default_client_limit = 8192
default_process_limit = 8192
default_vsz_limit = 512 M
disable_plaintext_auth = no
doveadm_password =  # hidden, use -P to show it
imap_client_workarounds = delay-newmail tb-extra-mailbox-sep tb-lsub-flags
imap_idle_notify_interval = 1 mins
login_greeting = IMAP/POP3 ready - us11-001dm
mail_attachment_dir = /var/mail/sis/%Ld
mail_cache_min_mail_count = 5
mail_debug = yes
mail_plugins = " quota virtual"
mailbox_list_index = yes
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope
encoded-character vacation subaddress comparator-i;ascii-numeric
relational regex imap4flags copy include variables body enotify
environment mailbox date index ihave duplicate mime foreverypart
extracttext vacation-seconds spamtest spamtestplus editheader
mbox_write_locks = fcntl
mmap_disable = yes
namespace inbox {
  inbox = yes
  list = yes
  location =
  mailbox Archive {
auto = create
special_use = \Archive
  }
  mailbox Drafts {
auto = create
special_use = \Drafts
  }
  mailbox Sent {
auto = create
special_use = \Sent
  }
  mailbox Spam {
auto = create
special_use = \Junk
  }
  mailbox Trash {
auto = create
special_use = \Trash
  }
  prefix = INBOX/
  separator = /
  type = private
}
namespace virtual {
  list = no
  location = virtual:/var/lib/mail/dovecot/virtual:INDEX=~/virtual
  prefix = virtual/
  separator = /
  type = private
}
passdb {
  args = /etc/dovecot/sql_users.conf
  driver = sql
}
passdb {
  args = /etc/dovecot/sql_admin.conf
  driver = sql
}
plugin {
  mail_log_events = delete expunge
  mail_log_fields = uid box msgid size
  quota = dict:UserQuota::file:%h/dovecot-quota
  quota_rule2 = INBOX/Trash:storage=+10%%
  sieve = file:~/sieve/user;active=~/.dovecot.sieve
  sieve_default = file:/var/lib/mail/dovecot/sieve/default.sieve
  sieve_default_name = default
  sieve_editheader_max_header_size = 1k
  sieve_extensions = +spamtest +spamtestplus +editheader +vacation-seconds
  sieve_global = file:/var/lib/mail/dovecot/sieve
  sieve_quota_max_scripts = 5
  sieve_spamtest_max_value = 200
  sieve_spamtest_status_header = X-spam-score-int: -?([[:digit:]]+)
  sieve_spamtest_status_type = score
  sieve_vacation_default_period = 10d
  sieve_vacation_max_period = 30d
  sieve_vacation_min_period = 1s
}
pop3_client_workarounds = outlook-no-nuls oe-ns-eoh
pop3_fast_size_lookups = yes
protocols = imap pop3 lmtp sieve
service anvil {
  client_limit = 5000
}
service auth-worker {
  user = $default_internal_user
}
service auth {
  client_limit = 5000
  unix_listener auth_client {
mode = 0660
user = exim
  }
}
service doveadm {
  client_limit = 10
  inet_listener {
address = *
port = 4000
  }
  process_limit = 100
  process_min_avail = 10
  service_count = 1
}
service imap-login {
  inet_listener imap {
port = 143
  }
  inet_listener imaps {
port = 993
ssl = yes
  }
  process_limit = 500
  process_min_avail = 10
  service_count = 1
}
service imap {
  executable = imap postlogin
  process_limit = 500
}
service managesieve-login {
  inet_listener sieve {
port = 4190
  }
  process_limit = 20
  process_min_avail = 10
  service_count = 1
}
service managesieve {
  process_limit = 1024
}
service pop3-login {
  inet_listener pop3 {
port = 110
  }
  inet_listener pop3s {
port = 995
ssl = yes
  }
  process_limit = 1000
  process_min_avail = 5
  service_count = 1
}
service pop3 {
  process_limit = 500
}
service pos

Re: Dovecot LMTP Proxy + STARTTLS?

[Matt Bryant]

Not according to dovecot doco ... from
https://wiki2.dovecot.org/PasswordDatabase/ExtraFields/Proxy

*NOTE: LMTP/doveadm proxying doesn't support SSL/TLS currently - any
ssl/starttls extra field is ignored

*rgds

Matt


> Tobi 
> 23 November 2017 at 6:31 pm
> Hi
>
> I got dovecot 2.2.26 on a Centos7 with latest updates. Dovecot is
> configured to act as director and delivers to my two backend servers.
> I enabled lmtp proxy on director to listen on port 24.
>
> Now I see in msg headers that the connection to the lmtp proxy uses
> STARTTLS but the connection from proxy to backend seems to be
> unencrypted. Is it possible to enforce the use of STARTTLS in the
> connection from the director to the backend as well?
>
> Regards
>
> tobi

mdbox + director + nfs

[Matt Bryant]

Anyone had any experience they can share using a director setup with NFS
and using the mdbox format instead of maildir. Is locking an issue on
index files ??

rgds

Matt
-- 

Re: Failback mailboxes?

[Matt Bryant]

hmm if message cannot be written to disk surely it remains on mda queue
as not delviered and does not just disappear ? or am i reading this
wrong ?!


> Dag Nygren 
> 16 August 2017 at 7:14 am
> Thanks for all the advice on how to configure systemd
> not to loose my emails after every update. Much appreciated.
>
> But there could be other reasons for the mailboxes not being
> writable and what I am really asking for is for
> dovecot-lda not to loose the incoming emails into thin air
> in these cases.
>
> Could we have some kind of collective place/places where they would
> be saved in this case and then reintroduced into the system
> after the problem is fixed? One file for example?
>
> Best
> Dag
> Dag Nygren 
> 14 August 2017 at 4:24 pm
> Hi!
>
> Have been using Fedora as my dovecot server for
> some time and am struggling with systemd
> at every update.
> Fedora insists on setting
> ProtectSystem=full in both dovecot.service and postfix.service
> at every update of the packages.
>
> This makes my mailstore which is in /usr/local/var/mail
> Read-only.
>
> And this makes the incoming emails delivered through
> dovecot-lda disappear into /dev/null until I notice
> the problem and we lose incoming emails.
>
> My question is:
> Is there any way to set up a failback mailstore
> for these occasions?
>
> PS! I really hate systemd - Destroys the UNIX way of
> doing things with a heavy axe
>
>
> Best
> Dag

Re: rawlog

[Matt Bryant]

Sigh ... one issue was misconfig kinda forgot to add the script into
imap service  .. the other is I thought I had updated the dovecot
package but that was on another instance to rawlog_dir wouldnt have been
in that version. Rats .. its a shame there is not auto create though
mis-read that one.

rgds

Matt
> Aki Tuomi <mailto:aki.tu...@dovecot.fi>
> 4 August 2017 at 4:05 am
>
> Most common mistake with rawlogs is to assume that the target
> directory gets created. It doesn't.
>
> You need to make sure the target directory exists fully expanded, e.g.
> if you have target directory /tmp/rawlogs/%u, you need to create
> /tmp/rawlogs/victim and chmod it to 0777.
>
> Aki
> Alexander Dalloz <mailto:ad+li...@uni-x.org>
> 4 August 2017 at 3:57 am
> Am 03.08.2017 um 01:04 schrieb Matt Bryant:
>> Hi,
>>
>> Trying to get rawlog working on dovecot 2.2.31 configured as per
>>
>> https://wiki2.dovecot.org/Debugging/Rawlog
>>
>> but
>>
>> a) it doesnt appear to be loggin anything
>> b) rawlog_dir which is supposed to be v2.2.26+ seems no where in site ..
>> in fact complains about unknown variable
>>
>> So does rawlog still do anything  Or am I missing something ...
>> config is below 
>>
>>
>> # 2.2.19: /etc/dovecot/dovecot.conf
>
> You run dovecot 2.2.19, not v2.2.26+.
>
> From where did you take that specific version? CentOS 7 ships dovecot
> 2.2.10. I can recommend the usage of the dovecot packages from the
> ghettoforge.org repository. Then you are current (2.2.31 actually).
>
>> # Pigeonhole version 0.4.9 (357ac0a0e68b+)
>> doveconf: Warning: service auth { client_limit=3 } is lower than
>> required under max. load (150032)
>> doveconf: Warning: service anvil { client_limit=22000 } is lower than
>> required under max. load (50027)
>> # OS: Linux 3.10.0-327.4.4.el7.x86_64 x86_64 CentOS Linux release
>> 7.2.1511 (Core)
>
> Please, keep your systems up to date! At least your kernel is terribly
> out of date.
>
> Regards
>
> Alexander
> Matt Bryant <mailto:dev...@atmail.com>
> 3 August 2017 at 9:04 am
> Hi,
>
> Trying to get rawlog working on dovecot 2.2.31 configured as per
>
> https://wiki2.dovecot.org/Debugging/Rawlog
>
> but
>
> a) it doesnt appear to be loggin anything
> b) rawlog_dir which is supposed to be v2.2.26+ seems no where in site ..
> in fact complains about unknown variable
>
> So does rawlog still do anything  Or am I missing something ...
> config is below 
>
>
> # 2.2.19: /etc/dovecot/dovecot.conf
> # Pigeonhole version 0.4.9 (357ac0a0e68b+)
> doveconf: Warning: service auth { client_limit=3 } is lower than
> required under max. load (150032)
> doveconf: Warning: service anvil { client_limit=22000 } is lower than
> required under max. load (50027)
> # OS: Linux 3.10.0-327.4.4.el7.x86_64 x86_64 CentOS Linux release
> 7.2.1511 (Core)
> auth_cache_negative_ttl = 2 mins
> auth_cache_size = 10 M
> auth_cache_ttl = 10 mins
> auth_master_user_separator = *
> auth_mechanisms = plain login
> auth_worker_max_count = 1
> default_client_limit = 5
> default_process_limit = 5
> disable_plaintext_auth = no
> doveadm_password = # hidden, use -P to show it
> imap_client_workarounds = delay-newmail tb-extra-mailbox-sep tb-lsub-flags
> imap_hibernate_timeout = 1 mins
> imap_idle_notify_interval = 1 mins
> login_greeting = IMAP/POP3 ready - dev-dh-ro-ms-001-b
> mail_attachment_dir = /var/lib/dovecot/attachments/%Ld
> mail_cache_min_mail_count = 5
> mail_plugins = " notify replication quota virtual"
> mailbox_list_index = yes
> managesieve_notify_capability = mailto
> managesieve_sieve_capability = fileinto reject envelope
> encoded-character vacation subaddress comparator-i;ascii-numeric
> relational regex imap4flags copy include variables body enotify
> environment mailbox date index ihave duplicate vacation-seconds spamtest
> spamtestplus editheader
> mbox_write_locks = fcntl
> mmap_disable = yes
> namespace {
> inbox = yes
> list = yes
> location =
> mailbox Archive {
> auto = create
> special_use = \Archive
> }
> mailbox Drafts {
> auto = create
> special_use = \Drafts
> }
> mailbox Sent {
> auto = create
> special_use = \Sent
> }
> mailbox Spam {
> auto = create
> special_use = \Junk
> }
> mailbox Trash {
> auto = create
> special_use = \Trash
> }
> prefix = INBOX/
> separator = /
> type = private
> }
> namespace {
> list = no
> location = virtual:/var/lib/dovecot/virtual:INDEXPVT=~/virtual
> prefix = virtual/
> separator = /
> type = private
> }
> passdb {
&g

rawlog

[Matt Bryant]

Hi,

Trying to get rawlog working on dovecot 2.2.31 configured as per

https://wiki2.dovecot.org/Debugging/Rawlog

but

a) it doesnt appear to be loggin anything
b) rawlog_dir which is supposed to be v2.2.26+ seems no where in site ..
in fact complains about unknown variable

So does rawlog still do anything  Or am I missing something ...
config is below 


# 2.2.19: /etc/dovecot/dovecot.conf
# Pigeonhole version 0.4.9 (357ac0a0e68b+)
doveconf: Warning: service auth { client_limit=3 } is lower than
required under max. load (150032)
doveconf: Warning: service anvil { client_limit=22000 } is lower than
required under max. load (50027)
# OS: Linux 3.10.0-327.4.4.el7.x86_64 x86_64 CentOS Linux release
7.2.1511 (Core)
auth_cache_negative_ttl = 2 mins
auth_cache_size = 10 M
auth_cache_ttl = 10 mins
auth_master_user_separator = *
auth_mechanisms = plain login
auth_worker_max_count = 1
default_client_limit = 5
default_process_limit = 5
disable_plaintext_auth = no
doveadm_password =  # hidden, use -P to show it
imap_client_workarounds = delay-newmail tb-extra-mailbox-sep tb-lsub-flags
imap_hibernate_timeout = 1 mins
imap_idle_notify_interval = 1 mins
login_greeting = IMAP/POP3 ready - dev-dh-ro-ms-001-b
mail_attachment_dir = /var/lib/dovecot/attachments/%Ld
mail_cache_min_mail_count = 5
mail_plugins = " notify replication quota virtual"
mailbox_list_index = yes
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope
encoded-character vacation subaddress comparator-i;ascii-numeric
relational regex imap4flags copy include variables body enotify
environment mailbox date index ihave duplicate vacation-seconds spamtest
spamtestplus editheader
mbox_write_locks = fcntl
mmap_disable = yes
namespace {
  inbox = yes
  list = yes
  location =
  mailbox Archive {
auto = create
special_use = \Archive
  }
  mailbox Drafts {
auto = create
special_use = \Drafts
  }
  mailbox Sent {
auto = create
special_use = \Sent
  }
  mailbox Spam {
auto = create
special_use = \Junk
  }
  mailbox Trash {
auto = create
special_use = \Trash
  }
  prefix = INBOX/
  separator = /
  type = private
}
namespace {
  list = no
  location = virtual:/var/lib/dovecot/virtual:INDEXPVT=~/virtual
  prefix = virtual/
  separator = /
  type = private
}
passdb {
  args = /etc/dovecot/sql_users.conf
  driver = sql
}
plugin {
  mail_log_events = delete expunge
  mail_log_fields = uid box msgid size
  mail_replica = tcp:dev-ms-001-a:4000
  quota = dict:UserQuota::file:%h/dovecot-quota
  quota_rule2 = INBOX/Trash:storage=+10%%
  sieve = file:~/sieve/user;active=~/.dovecot.sieve
  sieve_default = file:/var/lib/dovecot/sieve/default.sieve
  sieve_default_name = default
  sieve_editheader_max_header_size = 1k
  sieve_extensions = +spamtest +spamtestplus +editheader +vacation-seconds
  sieve_global = file:/var/lib/dovecot/sieve
  sieve_quota_max_scripts = 5
  sieve_spamtest_max_value = 200
  sieve_spamtest_status_header = X-Spam-score-int: -?([[:digit:]]+)
  sieve_spamtest_status_type = score
  sieve_vacation_default_period = 10d
  sieve_vacation_max_period = 30d
  sieve_vacation_min_period = 1s
}
pop3_client_workarounds = outlook-no-nuls oe-ns-eoh
pop3_fast_size_lookups = yes
protocols = imap pop3 lmtp sieve
replication_dsync_parameters = -d -n INBOX -l 30 -U
service aggregator {
  fifo_listener replication-notify-fifo {
user = vmail
  }
  unix_listener replication-notify {
user = vmail
  }
}
service anvil {
  client_limit = 22000
}
service auth-worker {
  user = $default_internal_user
}
service auth {
  client_limit = 3
  unix_listener auth_client {
mode = 0660
user = exim
  }
}
service doveadm {
  client_limit = 1
  inet_listener {
address = *
port = 4000
  }
  process_limit = 80
  process_min_avail = 8
  service_count = 10
}
service imap-hibernate {
  client_limit = 8000
  process_limit = 8
  process_min_avail = 8
  service_count = 0
  unix_listener imap-hibernate {
group = vmail
mode = 0660
  }
}
service imap-login {
  client_limit = 8000
  inet_listener imap {
port = 143
  }
  process_limit = 8
  process_min_avail = 8
  service_count = 0
}
service imap {
  client_limit = 1
  process_limit = 5
  service_count = 100
}
service managesieve-login {
  client_limit = 1000
  inet_listener sieve {
port = 4190
  }
  process_limit = 8
  process_min_avail = 8
  service_count = 0
}
service managesieve {
  process_limit = 1024
}
service pop3-login {
  client_limit = 1000
  inet_listener pop3 {
port = 110
  }
  process_limit = 8
  process_min_avail = 8
  service_count = 0
}
service pop3 {
  process_limit = 1
}
service postlogin {
  executable = script-login -d rawlog
  unix_listener postlogin {
group = atmail
mode = 0660
  }
}
service replicator {
  process_min_avail = 1
  unix_listener replicator-doveadm {
group = atmail
mode = 0660
  }
}
shutdown_clients = no
ssl_cert = 

Large Scale Cluster using dsync and tagged director

[Matt Bryant]

Just been reading up on some of dovecot features and was wondering if
anyone had played around with large scale systems made up from mutliple
small dsync clusters and using tagged director to split users between
nodes of a sync'd cluster (rather than nfs).

I know traditionally its been shared storage and multiple nodes and
dovecots own doco suggest dsync for several thousands rather than
millions but given the cloud seems to be all about replication what
about many smaller nodes using dsync and scaling horizontally ?

The only issue is how does director scale when there are serveral(or
rather lots of) 'tags' ... in fact what are the limitations with director ?
 

Matt
-- 

tagged director -

[Matt Bryant]

Hey,

Does anyone know how the new(ish) director tags work in respect to which
one is select for a user ?? Is there a query or field returned on userdb
lookup to specify the director tag to use ???

rgds

Matt

Any documentation around timeout states ?

[Matt Bryant]

Been seeing a number of proxy timouts recently with errors such as ...

timed out in state=4
timed out in state=2

Is there any doco around what state 2/4 actually means ... atm got
auth_debug on to see if we can track further information down

centos 7 x86_64
dovecot 2.2.19

rgds

Matt

Re: 2.2.30.2 fails to compile on centos 7

[Matt Bryant]

thx

my actual RPM failed at the same place .. the below was actually not the
result of an rpmbuild command .. cleared out the BUILD directory, 
untar'd, configure and ran make manually - failed at the same spot as
the RPM package .. given 2.2.30.1 built ok  saw no reason why 2.2.30.2
should suddenly stop ..

rgds

Matt

> Peter <mailto:pe...@pajamian.dhs.org>
> 7 June 2017 at 10:39 am
> On 07/06/17 10:52, Matt Bryant wrote:
>> Just downloaded the latest release 2.2.30.2, ran ./configure with all
>> defaults and make ...
>>
>> and it fails to build .. error somewhere whilst running Makefile in lib-http
>
> Just built fine for me.  I'll give you a link once I get it pushed out
> to GhettoForge.
>
>> libtool: link: gcc -std=gnu99 -g -O2 -Wall -W -Wmissing-prototypes
>> -Wmissing-declarations -Wpointer-arith -Wchar-subscripts -Wformat=2
>> -Wbad-function-cast -fno-builtin-strftime -Wstrict-aliasing=2
>> -Wl,--as-needed -o test-http-server-errors test-http-server-errors.o
>> -Wl,--export-dynamic  ./.libs/libhttp.a ../lib-dns/.libs/libdns.a
>> ../lib-ssl-iostream/.libs/libssl_iostream.a
>> ../lib-master/.libs/libmaster.a ../lib-settings/.libs/libsettings.a
>> ../lib-test/.libs/libtest.a ../lib/.libs/liblib.a -ldl
>> collect2: error: ld returned 1 exit status
>
> libtool: link: gcc -std=gnu99 -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2
> -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4
> -grecord-gcc-switches -m64 -mtune=generic -fno-strict
> -aliasing -Wall -W -Wmissing-prototypes -Wmissing-declarations
> -Wpointer-arith -Wchar-subscripts -Wformat=2 -Wbad-function-cast
> -fno-builtin-strftime -Wstrict-aliasing=2 -Wl,--as-needed -Wl,
> -z -Wl,relro -o test-http-server-errors test-http-server-errors.o
> -Wl,--export-dynamic  ./.libs/libhttp.a ../lib-dns/.libs/libdns.a
> ../lib-ssl-iostream/.libs/libssl_iostream.a ../lib-master/
> .libs/libmaster.a ../lib-settings/.libs/libsettings.a
> ../lib-test/.libs/libtest.a ../lib/.libs/liblib.a -ldl
> make[3]: Leaving directory
> `/builddir/build/BUILD/dovecot-2.2.30.2/src/lib-http'
>
>> make[3]: Leaving directory
>> `/home/rpmbuild/rpmbuild/BUILD/dovecot-2.2.30.2/src/lib-http'
>
> I suggest that you don't use rpmbuild directly, but use mock to build it
> instead.  That ensures you have a clean build environment every time.
>
>
> Peter
> Matt Bryant <mailto:dev...@atmail.com>
> 7 June 2017 at 8:52 am
> Just downloaded the latest release 2.2.30.2, ran ./configure with all
> defaults and make ...
>
> and it fails to build .. error somewhere whilst running Makefile in
> lib-http
>
> libtool: link: gcc -std=gnu99 -g -O2 -Wall -W -Wmissing-prototypes
> -Wmissing-declarations -Wpointer-arith -Wchar-subscripts -Wformat=2
> -Wbad-function-cast -fno-builtin-strftime -Wstrict-aliasing=2
> -Wl,--as-needed -o test-http-server-errors test-http-server-errors.o
> -Wl,--export-dynamic ./.libs/libhttp.a ../lib-dns/.libs/libdns.a
> ../lib-ssl-iostream/.libs/libssl_iostream.a
> ../lib-master/.libs/libmaster.a ../lib-settings/.libs/libsettings.a
> ../lib-test/.libs/libtest.a ../lib/.libs/liblib.a -ldl
> collect2: error: ld returned 1 exit status
> make[3]: *** [test-http-server-errors] Error 1
> make[3]: Leaving directory
> `/home/rpmbuild/rpmbuild/BUILD/dovecot-2.2.30.2/src/lib-http'
> make[2]: *** [all-recursive] Error 1
> make[2]: Leaving directory
> `/home/rpmbuild/rpmbuild/BUILD/dovecot-2.2.30.2/src'
> make[1]: *** [all-recursive] Error 1
> make[1]: Leaving directory
> `/home/rpmbuild/rpmbuild/BUILD/dovecot-2.2.30.2'
> make: *** [all] Error 2
>
> any ideas ?? 2.2.30.1 built ok ...
>
> rgds
>
> Matt

2.2.30.2 fails to compile on centos 7

[Matt Bryant]

Just downloaded the latest release 2.2.30.2, ran ./configure with all
defaults and make ...

and it fails to build .. error somewhere whilst running Makefile in lib-http

libtool: link: gcc -std=gnu99 -g -O2 -Wall -W -Wmissing-prototypes
-Wmissing-declarations -Wpointer-arith -Wchar-subscripts -Wformat=2
-Wbad-function-cast -fno-builtin-strftime -Wstrict-aliasing=2
-Wl,--as-needed -o test-http-server-errors test-http-server-errors.o
-Wl,--export-dynamic  ./.libs/libhttp.a ../lib-dns/.libs/libdns.a
../lib-ssl-iostream/.libs/libssl_iostream.a
../lib-master/.libs/libmaster.a ../lib-settings/.libs/libsettings.a
../lib-test/.libs/libtest.a ../lib/.libs/liblib.a -ldl
collect2: error: ld returned 1 exit status
make[3]: *** [test-http-server-errors] Error 1
make[3]: Leaving directory
`/home/rpmbuild/rpmbuild/BUILD/dovecot-2.2.30.2/src/lib-http'
make[2]: *** [all-recursive] Error 1
make[2]: Leaving directory
`/home/rpmbuild/rpmbuild/BUILD/dovecot-2.2.30.2/src'
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory `/home/rpmbuild/rpmbuild/BUILD/dovecot-2.2.30.2'
make: *** [all] Error 2

any ideas ?? 2.2.30.1 built ok ...

rgds

Matt