Re: Autoexpunge not working for Junk?

[Amir Caspi via dovecot]

On Aug 14, 2019, at 1:45 PM, Timo Sirainen  wrote:
> 
> It's not very helpful to look at the indexes after the problem already 
> happened.

Good point!  I guess I'll empty out the mailboxes completely, touch a 
completely empty file, and see if the problem recurs.  I'll shorten the 
autoexpunge time to 7 days so we don't have to wait a month to see if it recurs.

Thanks.

--- Amir

Re: Autoexpunge not working for Junk?

[Timo Sirainen via dovecot]

On 14 Aug 2019, at 22.35, Amir Caspi via dovecot  wrote:
> 
> On Aug 14, 2019, at 1:26 PM, Timo Sirainen via dovecot  > wrote:
>> 
>> It probably has something to do with using mbox format. Are the IMAP UIDs 
>> changing unexpectedly? Errors/warnings logged related to it? Unfortunately 
>> it's a rather troublesome mailbox format. There are likely some bugs in 
>> Dovecot mbox code, but it's difficult and time consuming to try to reproduce 
>> any of the bugs so I've mostly given up trying.
> 
> I'm not getting any errors or warnings as far as I can tell, and I don't 
> think the UIDs are changing unexpectedly -- messages are not getting 
> re-downloaded randomly.  That is, everything SEEMS to be working fine, as far 
> as I can tell.
> 
> So many people still use mbox that I hope we can fix this issue.
> 
> I'm happy to help test or provide further debug output... this problem is 
> certainly reproducible here, and it seems like lbutlr has a similar problem, 
> so hopefully we can address at least this one...
> 
> (I'm also happy to give you the Junk mailbox and index files... there's 
> nothing sensitive in my spam!)

It's not very helpful to look at the indexes after the problem already 
happened. But if you can find a reliably reproducible way to make this happen 
starting from an empty mailbox, I could look into it further. Ideally it would 
be a standalone script that reproduces the problem every time. Possibly 
something like:

 * Deliver mails with procmail
 * Read the mails with doveadm fetch
 * Maybe expunge the mails with doveadm expunge
 * Keep checking the uid and date.saved with doveadm fetch to see if they 
unexpectedly change at some point

Re: Autoexpunge not working for Junk?

[Amir Caspi via dovecot]

On Aug 14, 2019, at 1:26 PM, Timo Sirainen via dovecot  
wrote:
> 
> It probably has something to do with using mbox format. Are the IMAP UIDs 
> changing unexpectedly? Errors/warnings logged related to it? Unfortunately 
> it's a rather troublesome mailbox format. There are likely some bugs in 
> Dovecot mbox code, but it's difficult and time consuming to try to reproduce 
> any of the bugs so I've mostly given up trying.

I'm not getting any errors or warnings as far as I can tell, and I don't think 
the UIDs are changing unexpectedly -- messages are not getting re-downloaded 
randomly.  That is, everything SEEMS to be working fine, as far as I can tell.

So many people still use mbox that I hope we can fix this issue.

I'm happy to help test or provide further debug output... this problem is 
certainly reproducible here, and it seems like lbutlr has a similar problem, so 
hopefully we can address at least this one...

(I'm also happy to give you the Junk mailbox and index files... there's nothing 
sensitive in my spam!)

Cheers.

--- Amir

Re: Autoexpunge not working for Junk?

[Timo Sirainen via dovecot]

On 13 Aug 2019, at 5.57, Amir Caspi via dovecot  wrote:
> 
> On Aug 12, 2019, at 8:54 PM, Thomas Zajic via dovecot  
> wrote:
>> 
>> * Amir Caspi via dovecot, 12.08.19 22:01
>> 
>>> [~]# doveadm mailbox status -u cepheid firstsaved Junk
>>> Junk firstsaved=1563154976
>>> 
>>> I can't tell how that timestamp corresponds to a human-readable date, 
>>> however.
>> 
>> [zlatko@disclosure:~]$ date -d @1563154976
>> Mon Jul 15 03:42:56 CEST 2019
> 
> So this is the same timestamp as date.saved on message 1... as it should be, 
> I guess.  Except that, as I showed, the timestamps are definitely messed up 
> somehow.  The timestamps in my MUA (whether webmail or local mail app) show 
> just fine... so something seems to be corrupted with the timestamps in the 
> dovecot index file, I think.  But the weird thing is that this is affecting 
> all users, not just me.

It probably has something to do with using mbox format. Are the IMAP UIDs 
changing unexpectedly? Errors/warnings logged related to it? Unfortunately it's 
a rather troublesome mailbox format. There are likely some bugs in Dovecot mbox 
code, but it's difficult and time consuming to try to reproduce any of the bugs 
so I've mostly given up trying.

Re: Autoexpunge not working for Junk?

[Amir Caspi via dovecot]

On Aug 12, 2019, at 8:54 PM, Thomas Zajic via dovecot  
wrote:
> 
> * Amir Caspi via dovecot, 12.08.19 22:01
> 
>> [~]# doveadm mailbox status -u cepheid firstsaved Junk
>> Junk firstsaved=1563154976
>> 
>> I can't tell how that timestamp corresponds to a human-readable date, 
>> however.
> 
> [zlatko@disclosure:~]$ date -d @1563154976
> Mon Jul 15 03:42:56 CEST 2019

So this is the same timestamp as date.saved on message 1... as it should be, I 
guess.  Except that, as I showed, the timestamps are definitely messed up 
somehow.  The timestamps in my MUA (whether webmail or local mail app) show 
just fine... so something seems to be corrupted with the timestamps in the 
dovecot index file, I think.  But the weird thing is that this is affecting all 
users, not just me.

Hopefully we can figure this out.

--- Amir

Re: Autoexpunge not working for Junk?

[Thomas Zajic via dovecot]

* Amir Caspi via dovecot, 12.08.19 22:01

> [~]# doveadm mailbox status -u cepheid firstsaved Junk
> Junk firstsaved=1563154976
> 
> I can't tell how that timestamp corresponds to a human-readable date, however.

[zlatko@disclosure:~]$ date -d @1563154976
Mon Jul 15 03:42:56 CEST 2019

HTH,
Thomas

Re: Autoexpunge not working for Junk?

[Amir Caspi via dovecot]

On Aug 12, 2019, at 12:18 AM, Sami Ketola via dovecot  
wrote:
> 
> "1. What's the first mail's saved-timestamp?
> doveadm fetch -u user date.saved mailbox Junk 1

Well... this is giving me strange results.  For myself, when I run this 
command, I get:

[~]# doveadm fetch -u cepheid date.saved mailbox Junk 1
date.saved: 2019-07-15 01:42:56

However, my actual Junk folder shows email from as far back as May 15.  I 
checked this with webmail to ensure it wasn't some problem on my local MUA... 
this is definitely on the server.  And yes, I checked the Received headers to 
make sure the dates were correctly represented (I didn't trust the Date header 
alone).  

HOWEVER... it looks like something is either corrupted or otherwise messed up, 
because messages #2 and 299 have identical timestamps:
[~]# doveadm fetch -u cepheid date.saved mailbox Junk 2
date.saved: 2019-07-25 03:54:14
[~]# doveadm fetch -u cepheid date.saved mailbox Junk 299
date.saved: 2019-07-25 03:54:14

In fact, MANY messages have this same timestamp.

I can't find the "true" first message timestamp, but the first one not in July 
is 107:
[~]# doveadm fetch -u cepheid date.saved mailbox Junk 107
date.saved: 2019-05-29 05:57:55

This seems... not right.

The results are similarly strange for my other users.

> 2. That timestamp should also be the same in dovecot.list.index:
> doveadm mailbox status -u user firstsaved Junk"

For this I get:

[~]# doveadm mailbox status -u cepheid firstsaved Junk
Junk firstsaved=1563154976

I can't tell how that timestamp corresponds to a human-readable date, however.

Cheers and thanks.

--- Amir

Re: Autoexpunge not working for Junk?

[@lbutlr via dovecot]

On 12 Aug 2019, at 00:18, Sami Ketola  wrote:
> "1. What's the first mail's saved-timestamp?
> doveadm fetch -u user date.saved mailbox Junk 1
> 
> 2. That timestamp should also be the same in dovecot.list.index:
> doveadm mailbox status -u user firstsaved Junk"
> 
> can you check that information?

Not for awhile since I manually cleaned out my junk a few days ago.

However, messages are moved to Junk almost entirely automatically. One or two 
outliers, sure, but a bunch of messages over 2 weeks old that have been in junk 
less than two weeks seems… unlikely.

Re: Autoexpunge not working for Junk?

[Sami Ketola via dovecot]

> On 10 Aug 2019, at 3.15, @lbutlr via dovecot  wrote:
> 
> On 8 Aug 2019, at 13:03, Amir Caspi  wrote:
>> IMHO the setting should apply regardless of protocol, but is that actually 
>> the case in practice?
> 
> It seems to be broken.
> 
> I have
> 
> namespace inbox {
> inbox = yes
> location =
> mailbox Drafts {
> special_use = \Drafts
> }
> mailbox Junk {
> autoexpunge = 14 days
> auto = subscribe
> special_use = \Junk
> }
> … 
> }
> 
> and I have messages in my Junk Mail from 16 July 2019


Quoting questions from Timo:

"1. What's the first mail's saved-timestamp?
doveadm fetch -u user date.saved mailbox Junk 1

2. That timestamp should also be the same in dovecot.list.index:
doveadm mailbox status -u user firstsaved Junk"

can you check that information?

Sami

Re: Autoexpunge not working for Junk?

[@lbutlr via dovecot]

On 8 Aug 2019, at 13:03, Amir Caspi  wrote:
> IMHO the setting should apply regardless of protocol, but is that actually 
> the case in practice?

It seems to be broken.

I have

namespace inbox {
inbox = yes
location =
mailbox Drafts {
special_use = \Drafts
}
mailbox Junk {
autoexpunge = 14 days
auto = subscribe
special_use = \Junk
}
… 
}

and I have messages in my Junk Mail from 16 July 2019



-- 
I AM ZOMBOR! (kelly) ZOMBOR!

Re: Autoexpunge not working for Junk?

[Amir Caspi via dovecot]

On Aug 8, 2019, at 12:52 PM, Reio Remma via dovecot  wrote:
> 
> I have the autoexpunge settings defined inside protocol imap thus (and it 
> works):

It looks like my definitions are outside the protocol, as this is the default 
in /etc/dovecot/conf.d/15-mailboxes.conf...

MUST autoexpunge be declared within the imap protocol, even if the mailboxes 
themselves are declared outside of the protocol?  Should the mailboxes be 
declared only within the protocol?

Can I declare autoexpunge for the mailboxes within the imap protocol in 
20-imap.conf while leaving the main mailbox declarations in 15-mailboxes.conf?  
That is, will the settings "stack" or can mailbox definitions occur only in one 
place?

If autoexpunge can only be declared within the imap protocol -- folks who 
access email only through POP (and who never see Junk/Trash) will not have 
things auto-expunged.  The Junk mailbox, in particular, is auto-populated by 
spamassassin via procmail so it will have messages even if the user never sees 
them... so it needs to be autoexpunged even if the user never logs in via imap.

IMHO the setting should apply regardless of protocol, but is that actually the 
case in practice?

Thanks!

--- Amir

Re: Autoexpunge not working for Junk?

[Reio Remma via dovecot]

Hello!

I have the autoexpunge settings defined inside protocol imap thus (and 
it works):


protocol imap {
  mail_plugins = quota notify replication imap_quota imap_sieve
  namespace inbox {
    location =
    mailbox Ham {
  autoexpunge = 365 days
    }
    mailbox Spam {
  autoexpunge = 365 days
    }
    mailbox Trash {
  autoexpunge = 180 days
    }
    prefix =
  }
}

Good luck,
Reio

On 08.08.2019 21:34, Amir Caspi via dovecot wrote:

Hi all,

Might anyone have any idea about this issue?  I can run a cron job if 
needed but it seems like autoexpunge SHOULD be doing this automatically...


Thanks!

--- Amir

On Jul 24, 2019, at 10:18 PM, Amir Caspi > wrote:


Hi all,

I set up dovecot a couple of months ago and am having trouble getting 
autoexpunge=30d to work on my Trash and Junk mailboxes.  Not sure why 
not because I'm not getting error messages in my log.
Running "doveadm search -u  mailbox Junk savedbefore 30d" shows 
me many messages (I've got messages back to mid-May, and a couple of 
other users have them back to early April, although if this setting 
were working, there should be nothing earlier than June 24).  Running 
a manual doveadm expunge works fine... it's just autoexpunge that 
seems to not be running at all.


I'm using sendmail as the MTA and procmail as the LDA, so dovecot is 
running purely for IMAP/POP service.


Any help is much appreciated.

Thanks!

doveconf -n:
# 2.2.36 (1f10bfa63): /etc/dovecot/dovecot.conf
# OS: Linux 3.10.0-957.21.3.el7.x86_64 x86_64 CentOS Linux release 
7.6.1810 (Core)

# Hostname: REDACTED
auth_username_format = %Ln
first_valid_uid = 1000
mail_location = mbox:~/mail:INBOX=/var/spool/mail/%u
mailbox_list_index = yes
mbox_write_locks = fcntl
namespace compat1 {
 alias_for =
 hidden = yes
 list = no
 location =
 prefix = mail/
 separator = /
}
namespace compat2 {
 alias_for =
 hidden = yes
 list = no
 location =
 prefix = ~/mail/
 separator = /
}
namespace compat3 {
 alias_for =
 hidden = yes
 list = no
 location =
 prefix = ~%u/mail/
 separator = /
}
namespace inbox {
 inbox = yes
 location =
 mailbox Archive {
   special_use = \Archive
 }
 mailbox "Deleted Messages" {
   autoexpunge = 30 days
   special_use = \Trash
 }
 mailbox Drafts {
   special_use = \Drafts
 }
 mailbox Junk {
   autoexpunge = 30 days
   special_use = \Junk
 }
 mailbox "Junk E-mail" {
   autoexpunge = 30 days
   special_use = \Junk
 }
 mailbox Sent {
   special_use = \Sent
 }
 mailbox "Sent Messages" {
   special_use = \Sent
 }
 mailbox Spam {
   autoexpunge = 30 days
   special_use = \Junk
 }
 mailbox Trash {
   autoexpunge = 30 days
   special_use = \Trash
 }
 prefix =
 separator = /
}
passdb {
 driver = pam
}
pop3_uidl_format = %08Xv%08Xu
ssl_cert = # REDACTED
ssl_cipher_list = # REDACTED
ssl_dh_parameters_length = # REDACTED
ssl_key =  # hidden, use -P to show it
ssl_prefer_server_ciphers = yes
userdb {
 driver = passwd
}

Re: Autoexpunge not working for Junk?

[Amir Caspi via dovecot]

Hi all,

Might anyone have any idea about this issue?  I can run a cron job if needed 
but it seems like autoexpunge SHOULD be doing this automatically...

Thanks!

--- Amir

> On Jul 24, 2019, at 10:18 PM, Amir Caspi  wrote:
> 
> Hi all,
> 
>   I set up dovecot a couple of months ago and am having trouble getting 
> autoexpunge=30d to work on my Trash and Junk mailboxes.  Not sure why not 
> because I'm not getting error messages in my log.
>   Running "doveadm search -u  mailbox Junk savedbefore 30d" shows 
> me many messages (I've got messages back to mid-May, and a couple of other 
> users have them back to early April, although if this setting were working, 
> there should be nothing earlier than June 24).  Running a manual doveadm 
> expunge works fine... it's just autoexpunge that seems to not be running at 
> all.
> 
> I'm using sendmail as the MTA and procmail as the LDA, so dovecot is running 
> purely for IMAP/POP service.
> 
> Any help is much appreciated.
> 
> Thanks!
> 
> doveconf -n:
> # 2.2.36 (1f10bfa63): /etc/dovecot/dovecot.conf
> # OS: Linux 3.10.0-957.21.3.el7.x86_64 x86_64 CentOS Linux release 7.6.1810 
> (Core)  
> # Hostname: REDACTED
> auth_username_format = %Ln
> first_valid_uid = 1000
> mail_location = mbox:~/mail:INBOX=/var/spool/mail/%u
> mailbox_list_index = yes
> mbox_write_locks = fcntl
> namespace compat1 {
>  alias_for = 
>  hidden = yes
>  list = no
>  location = 
>  prefix = mail/
>  separator = /
> }
> namespace compat2 {
>  alias_for = 
>  hidden = yes
>  list = no
>  location = 
>  prefix = ~/mail/
>  separator = /
> }
> namespace compat3 {
>  alias_for = 
>  hidden = yes
>  list = no
>  location = 
>  prefix = ~%u/mail/
>  separator = /
> }
> namespace inbox {
>  inbox = yes
>  location = 
>  mailbox Archive {
>special_use = \Archive
>  }
>  mailbox "Deleted Messages" {
>autoexpunge = 30 days
>special_use = \Trash
>  }
>  mailbox Drafts {
>special_use = \Drafts
>  }
>  mailbox Junk {
>autoexpunge = 30 days
>special_use = \Junk
>  }
>  mailbox "Junk E-mail" {
>autoexpunge = 30 days
>special_use = \Junk
>  }
>  mailbox Sent {
>special_use = \Sent
>  }
>  mailbox "Sent Messages" {
>special_use = \Sent
>  }
>  mailbox Spam {
>autoexpunge = 30 days
>special_use = \Junk
>  }
>  mailbox Trash {
>autoexpunge = 30 days
>special_use = \Trash
>  }
>  prefix = 
>  separator = /
> }
> passdb {
>  driver = pam
> }
> pop3_uidl_format = %08Xv%08Xu
> ssl_cert = # REDACTED
> ssl_cipher_list = # REDACTED
> ssl_dh_parameters_length = # REDACTED
> ssl_key =  # hidden, use -P to show it
> ssl_prefer_server_ciphers = yes
> userdb {
>  driver = passwd
> }
> 
> 
> --- Amir
> 

Re: Autoexpunge not working for Junk?

[Timo Sirainen via dovecot]

On 25 Jul 2019, at 7.18, Amir Caspi via dovecot  wrote:
> 
> Hi all,
> 
>   I set up dovecot a couple of months ago and am having trouble getting 
> autoexpunge=30d to work on my Trash and Junk mailboxes.  Not sure why not 
> because I'm not getting error messages in my log.
>   Running "doveadm search -u  mailbox Junk savedbefore 30d" shows 
> me many messages (I've got messages back to mid-May, and a couple of other 
> users have them back to early April, although if this setting were working, 
> there should be nothing earlier than June 24).  Running a manual doveadm 
> expunge works fine... it's just autoexpunge that seems to not be running at 
> all.

Autoexpunging tries to be efficient, so it looks only at the first email's 
saved-timestamp. It's also cached in dovecot.list.index. So you should check:

1. What's the first mail's saved-timestamp?
doveadm fetch -u user date.saved mailbox Junk 1

2. That timestamp should also be the same in dovecot.list.index:
doveadm mailbox status -u user firstsaved Junk

Autoexpunge not working for Junk?

[Amir Caspi via dovecot]

Hi all,

I set up dovecot a couple of months ago and am having trouble getting 
autoexpunge=30d to work on my Trash and Junk mailboxes.  Not sure why not 
because I'm not getting error messages in my log.
Running "doveadm search -u  mailbox Junk savedbefore 30d" shows 
me many messages (I've got messages back to mid-May, and a couple of other 
users have them back to early April, although if this setting were working, 
there should be nothing earlier than June 24).  Running a manual doveadm 
expunge works fine... it's just autoexpunge that seems to not be running at all.

I'm using sendmail as the MTA and procmail as the LDA, so dovecot is running 
purely for IMAP/POP service.

Any help is much appreciated.

Thanks!

doveconf -n:
# 2.2.36 (1f10bfa63): /etc/dovecot/dovecot.conf
# OS: Linux 3.10.0-957.21.3.el7.x86_64 x86_64 CentOS Linux release 7.6.1810 
(Core)  
# Hostname: REDACTED
auth_username_format = %Ln
first_valid_uid = 1000
mail_location = mbox:~/mail:INBOX=/var/spool/mail/%u
mailbox_list_index = yes
mbox_write_locks = fcntl
namespace compat1 {
  alias_for = 
  hidden = yes
  list = no
  location = 
  prefix = mail/
  separator = /
}
namespace compat2 {
  alias_for = 
  hidden = yes
  list = no
  location = 
  prefix = ~/mail/
  separator = /
}
namespace compat3 {
  alias_for = 
  hidden = yes
  list = no
  location = 
  prefix = ~%u/mail/
  separator = /
}
namespace inbox {
  inbox = yes
  location = 
  mailbox Archive {
special_use = \Archive
  }
  mailbox "Deleted Messages" {
autoexpunge = 30 days
special_use = \Trash
  }
  mailbox Drafts {
special_use = \Drafts
  }
  mailbox Junk {
autoexpunge = 30 days
special_use = \Junk
  }
  mailbox "Junk E-mail" {
autoexpunge = 30 days
special_use = \Junk
  }
  mailbox Sent {
special_use = \Sent
  }
  mailbox "Sent Messages" {
special_use = \Sent
  }
  mailbox Spam {
autoexpunge = 30 days
special_use = \Junk
  }
  mailbox Trash {
autoexpunge = 30 days
special_use = \Trash
  }
  prefix = 
  separator = /
}
passdb {
  driver = pam
}
pop3_uidl_format = %08Xv%08Xu
ssl_cert = # REDACTED
ssl_cipher_list = # REDACTED
ssl_dh_parameters_length = # REDACTED
ssl_key =  # hidden, use -P to show it
ssl_prefer_server_ciphers = yes
userdb {
  driver = passwd
}


--- Amir