Fwd: Re: 2.2.33.2 - Virtual Mailbox Issue bug maybe ?

2018-02-25 Thread Matt Bryant 
Thought I would get it another send and see if anyone out there has come
up with any ideas or thoughts on this .. certainly I couldnt seem to
replicate in 2.2.31 so am wondering looking at the change log if one of
the virtual 'fixes' in 2.2.32 .. maybe broke something else.

Either way can anyone confirm that

'inthread refs x-mailbox INBOX

should filter all message according to thread - seems to indicate that
in the 'gmail' example given on the dovecot wiki ...

rgds

Matt


> *From:* Matt Bryant 
> *Date:* 19 February 2018 at 9:21 am
> *To:* dovecot@dovecot.org
> *Subject:* 2.2.33.2 - Virtual Mailbox Issue bug maybe ?
> hmm down graded to 2.2.19 and dont see the issue .. other issues
> particularly a segfault in libstorage to do with virtual .. but at
> least mails seem to be appearing correctly and cant seem to replicate
> what I have seen in 2.2.33.2
>
> as an aside .. outlook on mac doesnt appear to like 2.2.33.2 either ..
> seeing same kind of issue where some messages not appearing in inbox
> .. but could be some outlook/mac bug not catered for perhaps given no
> issues in thunderbird and some other 3rd party clients tested.
>
> rgds
>
> Matt
>
>
> *From:* Matt Bryant 
> *Date:* 18 February 2018 at 9:14 am
> *To:* dovecot@dovecot.org
> *Subject:* 2.2.33.2 - Virtual Mailbox Issue bug maybe ?
> Been experimenting with virtual mailbox and have a question :-
>
> I have defined 2 virtual mailboxes 'all' and 'combined' as follows ...
>
>
> virtual/all/dovecot-virtual
>
> virtual/combined
>   inthread refs x-mailbox INBOX
>
> virtual/combined/dovecot-virtual
>
> !INBOX
> INBOX/Sent
> INBOX/Drafts
>   all
>
> When new mail comes I would assume that both virtual mailboxes should
> actually see the message but whilst an imap search command gets a
> result in the virtual/combined for some reason not every message is
> seen in virtual all - so was wondering what exactly does
>
> 'inthread refs x-mailbox INBOX'
>
> do ? I would have thought new mail to inbox would just have a thread
> of 1 and be seen. If i remove the indexes from virtual then the
> messages appear so am thinking something not quite triggering an
> action. Any ideas ? What can I do to try and track down where problem
> lies or am i misunderstanding what 'inthread refs x-mailbox INBOX'
> should do 
>
>
> == 8< ==
>
> # doveconf -n
> # 2.2.33.2 (d6601f4ec): /etc/dovecot/dovecot.conf
> # Pigeonhole version 0.4.21 (92477967)
> doveconf: Warning: service auth { client_limit=5000 } is lower than
> required under max. load (26096)
> doveconf: Warning: service anvil { client_limit=5000 } is lower than
> required under max. load (9715)
> # OS: Linux 3.10.0-693.11.6.el7.x86_64 x86_64 CentOS Linux release
> 7.3.1611 (Core)
> auth_cache_negative_ttl = 2 mins
> auth_cache_ttl = 3 mins
> auth_master_user_separator = *
> auth_mechanisms = plain login
> auth_verbose = yes
> auth_worker_max_count = 100
> default_client_limit = 8192
> default_process_limit = 8192
> default_vsz_limit = 512 M
> disable_plaintext_auth = no
> doveadm_password =  # hidden, use -P to show it
> imap_client_workarounds = delay-newmail tb-extra-mailbox-sep tb-lsub-flags
> imap_idle_notify_interval = 1 mins
> login_greeting = IMAP/POP3 ready - us11-001dm
> mail_attachment_dir = /var/mail/sis/%Ld
> mail_cache_min_mail_count = 5
> mail_debug = yes
> mail_plugins = " quota virtual"
> mailbox_list_index = yes
> managesieve_notify_capability = mailto
> managesieve_sieve_capability = fileinto reject envelope
> encoded-character vacation subaddress comparator-i;ascii-numeric
> relational regex imap4flags copy include variables body enotify
> environment mailbox date index ihave duplicate mime foreverypart
> extracttext vacation-seconds spamtest spamtestplus editheader
> mbox_write_locks = fcntl
> mmap_disable = yes
> namespace inbox {
>   inbox = yes
>   list = yes
>   location =
>   mailbox Archive {
> auto = create
> special_use = \Archive
>   }
>   mailbox Drafts {
> auto = create
> special_use = \Drafts
>   }
>   mailbox Sent {
> auto = create
> special_use = \Sent
>   }
>   mailbox Spam {
> auto = create
> special_use = \Junk
>   }
>   mailbox Trash {
> auto = create
> special_use = \Trash
>   }
>   prefix = INBOX/
>   separator = /
>   type = private
> }
> namespace virtual {
>   list = no
>   location = virtual:/var/lib/mail/dovecot/virtual:INDEX=~/virtual
>   prefix = virtual/
>   separator = /
>   type = private
> }
> passdb {
>   args = /etc/dovecot/sql_users.conf
&g

Re: 2.2.33.2 - Virtual Mailbox Issue bug maybe ?

2018-02-18 Thread Matt Bryant 
hmm down graded to 2.2.19 and dont see the issue .. other issues
particularly a segfault in libstorage to do with virtual .. but at least
mails seem to be appearing correctly and cant seem to replicate what I
have seen in 2.2.33.2

as an aside .. outlook on mac doesnt appear to like 2.2.33.2 either ..
seeing same kind of issue where some messages not appearing in inbox ..
but could be some outlook/mac bug not catered for perhaps given no
issues in thunderbird and some other 3rd party clients tested.

rgds

Matt

> Matt Bryant 
> 18 February 2018 at 9:14 am
> Been experimenting with virtual mailbox and have a question :-
>
> I have defined 2 virtual mailboxes 'all' and 'combined' as follows ...
>
>
> virtual/all/dovecot-virtual
>
> virtual/combined
>   inthread refs x-mailbox INBOX
>
> virtual/combined/dovecot-virtual
>
> !INBOX
> INBOX/Sent
> INBOX/Drafts
>   all

--

Re: 2.2.33.2 - Virtual Mailbox Issue bug maybe ?

2018-02-18 Thread Matt Bryant 

So removed the ! since pretty much uneeded but get the same behaviour
... send a mail to the underlying inbox .. gets found in
virtual/combined but not virtual/all  remove the indexes from
virtual/all and re-login .. they get recreated and bingo the new message
is now found (shown below) ...

even tried running a doveadm index on virtual/all ... but that didnt do
anything either 


a2 select virtual/all
* FLAGS (\Answered \Flagged \Deleted \Seen \Draft unknown-0 $Forwarded
home $MDNSent personal)
* OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft unknown-0
$Forwarded home $MDNSent personal \*)] Flags permitted.
* 30579 EXISTS
* 30579 RECENT
* OK [UNSEEN 139] First unseen.
* OK [UIDVALIDITY 1518990022] UIDs valid
* OK [UIDNEXT 30580] Predicted next UID
* OK [HIGHESTMODSEQ 8] Highest
a2 OK [READ-WRITE] Select completed (0.031 + 0.000 + 0.030 secs).
? search header Message-ID <5a89f365.1050...@the-bryants.net>
* SEARCH
? OK Search completed (0.254 + 0.000 + 0.253 secs).
a2 select virtual/combined
* OK [CLOSED] Previous mailbox closed.
* FLAGS (\Answered \Flagged \Deleted \Seen \Draft unknown-0 $Forwarded
home $MDNSent personal)
* OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft unknown-0
$Forwarded home $MDNSent personal \*)] Flags permitted.
* 30999 EXISTS
* 30999 RECENT
* OK [UNSEEN 141] First unseen.
* OK [UIDVALIDITY 1518990022] UIDs valid
* OK [UIDNEXT 31000] Predicted next UID
* OK [HIGHESTMODSEQ 10] Highest
a2 OK [READ-WRITE] Select completed (0.015 + 0.000 + 0.015 secs).
? search header Message-ID <5a89f365.1050...@the-bryants.net
* SEARCH 30999
? OK Search completed (0.217 + 0.000 + 0.216 secs).


if i delete the index files for virtual/all and then re-login and search ...

a2 select virtual/all
* FLAGS (\Answered \Flagged \Deleted \Seen \Draft unknown-0 $Forwarded
home $MDNSent personal)
* OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft unknown-0
$Forwarded home $MDNSent personal \*)] Flags permitted.
* 30581 EXISTS
* 30581 RECENT
* OK [UNSEEN 139] First unseen.
* OK [UIDVALIDITY 1518990876] UIDs valid
* OK [UIDNEXT 30582] Predicted next UID
* OK [HIGHESTMODSEQ 7] Highest
a2 OK [READ-WRITE] Select completed (0.032 + 0.000 + 0.031 secs).
? search header Message-ID <5a89f365.1050...@the-bryants.net>
* SEARCH 30581
? OK Search completed (0.274 + 0.000 + 0.273 secs).

message is found ...



> Matt Bryant 
> 19 February 2018 at 7:24 am
> So virtual/combined was meant to combine INBOX/Sent/Drafts and
> virtual/all display them in a threaded manner .. at least that was the
> intention ...
>
> rgds
>
> Matt
>
> Aki Tuomi 
> 18 February 2018 at 11:10 pm
>
> Uhm.
>
> combined folder seems to use INBOX for saving mails and contains mails
> from Sent and Drafts.
>
> Not sure what your all folder is supposed to do.
>
> Aki
>
>
> On 2018-02-18 01:14, Matt Bryant wrote:
>
> Matt Bryant 
> 18 February 2018 at 9:14 am
> Been experimenting with virtual mailbox and have a question :-
>
> I have defined 2 virtual mailboxes 'all' and 'combined' as follows ...
>
>
> virtual/all/dovecot-virtual
>
> virtual/combined
>   inthread refs x-mailbox INBOX
>
> virtual/combined/dovecot-virtual
>
> !INBOX
> INBOX/Sent
> INBOX/Drafts
>   all

--

Re: 2.2.33.2 - Virtual Mailbox Issue bug maybe ?

2018-02-18 Thread Matt Bryant 
actually probably could ditch the ! out of combined 'cos i dont think
need to save anything .. just view

> Matt Bryant 
> 19 February 2018 at 7:24 am
> So virtual/combined was meant to combine INBOX/Sent/Drafts and
> virtual/all display them in a threaded manner .. at least that was the
> intention ...
>
> rgds
>
> Matt
>
> Aki Tuomi 
> 18 February 2018 at 11:10 pm
>
> Uhm.
>
> combined folder seems to use INBOX for saving mails and contains mails
> from Sent and Drafts.
>
> Not sure what your all folder is supposed to do.
>
> Aki
>
>
> On 2018-02-18 01:14, Matt Bryant wrote:
>
> Matt Bryant 
> 18 February 2018 at 9:14 am
> Been experimenting with virtual mailbox and have a question :-
>
> I have defined 2 virtual mailboxes 'all' and 'combined' as follows ...
>
>
> virtual/all/dovecot-virtual
>
> virtual/combined
>   inthread refs x-mailbox INBOX
>
> virtual/combined/dovecot-virtual
>
> !INBOX
> INBOX/Sent
> INBOX/Drafts
>   all

--

Re: 2.2.33.2 - Virtual Mailbox Issue bug maybe ?

2018-02-18 Thread Matt Bryant 
So virtual/combined was meant to combine INBOX/Sent/Drafts and
virtual/all display them in a threaded manner .. at least that was the
intention ...

rgds

Matt
> Aki Tuomi 
> 18 February 2018 at 11:10 pm
>
> Uhm.
>
> combined folder seems to use INBOX for saving mails and contains mails
> from Sent and Drafts.
>
> Not sure what your all folder is supposed to do.
>
> Aki
>
>
> On 2018-02-18 01:14, Matt Bryant wrote:
>
> Matt Bryant 
> 18 February 2018 at 9:14 am
> Been experimenting with virtual mailbox and have a question :-
>
> I have defined 2 virtual mailboxes 'all' and 'combined' as follows ...
>
>
> virtual/all/dovecot-virtual
>
> virtual/combined
>   inthread refs x-mailbox INBOX
>
> virtual/combined/dovecot-virtual
>
> !INBOX
> INBOX/Sent
> INBOX/Drafts
>   all

--

Re: 2.2.33.2 - Virtual Mailbox Issue bug maybe ?

2018-02-18 Thread Aki Tuomi 

Uhm.

combined folder seems to use INBOX for saving mails and contains mails 
from Sent and Drafts.


Not sure what your all folder is supposed to do.

Aki


On 2018-02-18 01:14, Matt Bryant wrote:

Been experimenting with virtual mailbox and have a question :-

I have defined 2 virtual mailboxes 'all' and 'combined' as follows ...


virtual/all/dovecot-virtual

virtual/combined
  inthread refs x-mailbox INBOX

virtual/combined/dovecot-virtual

!INBOX
INBOX/Sent
INBOX/Drafts
  all

When new mail comes I would assume that both virtual mailboxes should 
actually see the message but whilst an imap search command gets a 
result in the virtual/combined for some reason not every message is 
seen in virtual all - so was wondering what exactly does


'inthread refs x-mailbox INBOX'

do ? I would have thought new mail to inbox would just have a thread 
of 1 and be seen. If i remove the indexes from virtual then the 
messages appear so am thinking something not quite triggering an 
action. Any ideas ? What can I do to try and track down where problem 
lies or am i misunderstanding what 'inthread refs x-mailbox INBOX' 
should do 



== 8< ==

# doveconf -n
# 2.2.33.2 (d6601f4ec): /etc/dovecot/dovecot.conf
# Pigeonhole version 0.4.21 (92477967)
doveconf: Warning: service auth { client_limit=5000 } is lower than 
required under max. load (26096)
doveconf: Warning: service anvil { client_limit=5000 } is lower than 
required under max. load (9715)
# OS: Linux 3.10.0-693.11.6.el7.x86_64 x86_64 CentOS Linux release 
7.3.1611 (Core)

auth_cache_negative_ttl = 2 mins
auth_cache_ttl = 3 mins
auth_master_user_separator = *
auth_mechanisms = plain login
auth_verbose = yes
auth_worker_max_count = 100
default_client_limit = 8192
default_process_limit = 8192
default_vsz_limit = 512 M
disable_plaintext_auth = no
doveadm_password =  # hidden, use -P to show it
imap_client_workarounds = delay-newmail tb-extra-mailbox-sep tb-lsub-flags
imap_idle_notify_interval = 1 mins
login_greeting = IMAP/POP3 ready - us11-001dm
mail_attachment_dir = /var/mail/sis/%Ld
mail_cache_min_mail_count = 5
mail_debug = yes
mail_plugins = " quota virtual"
mailbox_list_index = yes
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope 
encoded-character vacation subaddress comparator-i;ascii-numeric 
relational regex imap4flags copy include variables body enotify 
environment mailbox date index ihave duplicate mime foreverypart 
extracttext vacation-seconds spamtest spamtestplus editheader

mbox_write_locks = fcntl
mmap_disable = yes
namespace inbox {
  inbox = yes
  list = yes
  location =
  mailbox Archive {
    auto = create
    special_use = \Archive
  }
  mailbox Drafts {
    auto = create
    special_use = \Drafts
  }
  mailbox Sent {
    auto = create
    special_use = \Sent
  }
  mailbox Spam {
    auto = create
    special_use = \Junk
  }
  mailbox Trash {
    auto = create
    special_use = \Trash
  }
  prefix = INBOX/
  separator = /
  type = private
}
namespace virtual {
  list = no
  location = virtual:/var/lib/mail/dovecot/virtual:INDEX=~/virtual
  prefix = virtual/
  separator = /
  type = private
}
passdb {
  args = /etc/dovecot/sql_users.conf
  driver = sql
}
passdb {
  args = /etc/dovecot/sql_admin.conf
  driver = sql
}
plugin {
  mail_log_events = delete expunge
  mail_log_fields = uid box msgid size
  quota = dict:UserQuota::file:%h/dovecot-quota
  quota_rule2 = INBOX/Trash:storage=+10%%
  sieve = file:~/sieve/user;active=~/.dovecot.sieve
  sieve_default = file:/var/lib/mail/dovecot/sieve/default.sieve
  sieve_default_name = default
  sieve_editheader_max_header_size = 1k
  sieve_extensions = +spamtest +spamtestplus +editheader +vacation-seconds
  sieve_global = file:/var/lib/mail/dovecot/sieve
  sieve_quota_max_scripts = 5
  sieve_spamtest_max_value = 200
  sieve_spamtest_status_header = X-spam-score-int: -?([[:digit:]]+)
  sieve_spamtest_status_type = score
  sieve_vacation_default_period = 10d
  sieve_vacation_max_period = 30d
  sieve_vacation_min_period = 1s
}
pop3_client_workarounds = outlook-no-nuls oe-ns-eoh
pop3_fast_size_lookups = yes
protocols = imap pop3 lmtp sieve
service anvil {
  client_limit = 5000
}
service auth-worker {
  user = $default_internal_user
}
service auth {
  client_limit = 5000
  unix_listener auth_client {
    mode = 0660
    user = exim
  }
}
service doveadm {
  client_limit = 10
  inet_listener {
    address = *
    port = 4000
  }
  process_limit = 100
  process_min_avail = 10
  service_count = 1
}
service imap-login {
  inet_listener imap {
    port = 143
  }
  inet_listener imaps {
    port = 993
    ssl = yes
  }
  process_limit = 500
  process_min_avail = 10
  service_count = 1
}
service imap {
  executable = imap postlogin
  process_limit = 500
}
service managesieve-login {
  inet_listener sieve {
    port = 4190
  }
  process_limit = 20
  process_min_avail = 10
  service_count = 1
}
service managesieve {
  process_limit = 1024
}
service pop3-login {

2.2.33.2 - Virtual Mailbox Issue bug maybe ?

2018-02-17 Thread Matt Bryant 
Been experimenting with virtual mailbox and have a question :-

I have defined 2 virtual mailboxes 'all' and 'combined' as follows ...


virtual/all/dovecot-virtual

virtual/combined
  inthread refs x-mailbox INBOX

virtual/combined/dovecot-virtual

!INBOX
INBOX/Sent
INBOX/Drafts
  all

When new mail comes I would assume that both virtual mailboxes should
actually see the message but whilst an imap search command gets a result
in the virtual/combined for some reason not every message is seen in
virtual all - so was wondering what exactly does

'inthread refs x-mailbox INBOX'

do ? I would have thought new mail to inbox would just have a thread of
1 and be seen. If i remove the indexes from virtual then the messages
appear so am thinking something not quite triggering an action. Any
ideas ? What can I do to try and track down where problem lies or am i
misunderstanding what 'inthread refs x-mailbox INBOX' should do 


== 8< ==

# doveconf -n
# 2.2.33.2 (d6601f4ec): /etc/dovecot/dovecot.conf
# Pigeonhole version 0.4.21 (92477967)
doveconf: Warning: service auth { client_limit=5000 } is lower than
required under max. load (26096)
doveconf: Warning: service anvil { client_limit=5000 } is lower than
required under max. load (9715)
# OS: Linux 3.10.0-693.11.6.el7.x86_64 x86_64 CentOS Linux release
7.3.1611 (Core)
auth_cache_negative_ttl = 2 mins
auth_cache_ttl = 3 mins
auth_master_user_separator = *
auth_mechanisms = plain login
auth_verbose = yes
auth_worker_max_count = 100
default_client_limit = 8192
default_process_limit = 8192
default_vsz_limit = 512 M
disable_plaintext_auth = no
doveadm_password =  # hidden, use -P to show it
imap_client_workarounds = delay-newmail tb-extra-mailbox-sep tb-lsub-flags
imap_idle_notify_interval = 1 mins
login_greeting = IMAP/POP3 ready - us11-001dm
mail_attachment_dir = /var/mail/sis/%Ld
mail_cache_min_mail_count = 5
mail_debug = yes
mail_plugins = " quota virtual"
mailbox_list_index = yes
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope
encoded-character vacation subaddress comparator-i;ascii-numeric
relational regex imap4flags copy include variables body enotify
environment mailbox date index ihave duplicate mime foreverypart
extracttext vacation-seconds spamtest spamtestplus editheader
mbox_write_locks = fcntl
mmap_disable = yes
namespace inbox {
  inbox = yes
  list = yes
  location =
  mailbox Archive {
auto = create
special_use = \Archive
  }
  mailbox Drafts {
auto = create
special_use = \Drafts
  }
  mailbox Sent {
auto = create
special_use = \Sent
  }
  mailbox Spam {
auto = create
special_use = \Junk
  }
  mailbox Trash {
auto = create
special_use = \Trash
  }
  prefix = INBOX/
  separator = /
  type = private
}
namespace virtual {
  list = no
  location = virtual:/var/lib/mail/dovecot/virtual:INDEX=~/virtual
  prefix = virtual/
  separator = /
  type = private
}
passdb {
  args = /etc/dovecot/sql_users.conf
  driver = sql
}
passdb {
  args = /etc/dovecot/sql_admin.conf
  driver = sql
}
plugin {
  mail_log_events = delete expunge
  mail_log_fields = uid box msgid size
  quota = dict:UserQuota::file:%h/dovecot-quota
  quota_rule2 = INBOX/Trash:storage=+10%%
  sieve = file:~/sieve/user;active=~/.dovecot.sieve
  sieve_default = file:/var/lib/mail/dovecot/sieve/default.sieve
  sieve_default_name = default
  sieve_editheader_max_header_size = 1k
  sieve_extensions = +spamtest +spamtestplus +editheader +vacation-seconds
  sieve_global = file:/var/lib/mail/dovecot/sieve
  sieve_quota_max_scripts = 5
  sieve_spamtest_max_value = 200
  sieve_spamtest_status_header = X-spam-score-int: -?([[:digit:]]+)
  sieve_spamtest_status_type = score
  sieve_vacation_default_period = 10d
  sieve_vacation_max_period = 30d
  sieve_vacation_min_period = 1s
}
pop3_client_workarounds = outlook-no-nuls oe-ns-eoh
pop3_fast_size_lookups = yes
protocols = imap pop3 lmtp sieve
service anvil {
  client_limit = 5000
}
service auth-worker {
  user = $default_internal_user
}
service auth {
  client_limit = 5000
  unix_listener auth_client {
mode = 0660
user = exim
  }
}
service doveadm {
  client_limit = 10
  inet_listener {
address = *
port = 4000
  }
  process_limit = 100
  process_min_avail = 10
  service_count = 1
}
service imap-login {
  inet_listener imap {
port = 143
  }
  inet_listener imaps {
port = 993
ssl = yes
  }
  process_limit = 500
  process_min_avail = 10
  service_count = 1
}
service imap {
  executable = imap postlogin
  process_limit = 500
}
service managesieve-login {
  inet_listener sieve {
port = 4190
  }
  process_limit = 20
  process_min_avail = 10
  service_count = 1
}
service managesieve {
  process_limit = 1024
}
service pop3-login {
  inet_listener pop3 {
port = 110
  }
  inet_listener pop3s {
port = 995
ssl = yes
  }
  process_limit = 1000
  process_min_avail = 5
  service_count = 1
}
service pop3 {
  process_limit = 500
}
service pos