Re: doveadm expunge didn't clear Trash mailbox
Hello Joseph, On Jul 24, 2018 um 13:12:05, Joseph Tam wrote: > On Mon, 23 Jul 2018, Michael Wagner wrote: > > > here works a dovecot 2.2.27 on a raspberrypi and the behaviour is as > > expected. > > > > doveadm -f tab fetch -u "uid date.saved" mailbox Trash > > uid date.saved > > 314 2018-06-23 00:35:59 > > 315 2018-06-23 12:39:10 > > 316 2018-06-24 10:32:43 > > ... > > > > And I have a cron script that expunges the mails older than 30 days. > > /usr/bin/doveadm expunge -u mailbox Trash savedbefore 30d > > Thanks for your observation. I think you probably use maildir format, > yes? > > I think my problem stems from the fact I use mbox, so one file contains > many messages, whereas maildir uses one file for each message. Dovecot, > from my understanding, will use the message file's mtime for date.saved > if it doesn't have it in the cached. This is probably why I am seeing > multiple messages with the same value. yes, I use here the maildir format and I think you are right that dovecot uses the mtime and this is set every time when you write to the mbox file. If you use linux, you can use mb2md for the converting to maildir. apt show mb2md Description: Convert Mbox mailboxes to Maildir format mb2md (mbox to maildir) takes one or more Mbox format mailbox files in a directory and convert them to Maildir format mailboxes. As the Mbox format has some drawbacks, D. J. Bernstein created the Maildir format when he wrote Qmail. With the Mbox format all mail of a specific folder is stored as one large text file. The Maildir format stores each mail as a separate file. It is a faster and more efficient way to store mail. It works particularly well over NFS, which has a long history of locking-related woes. The Mbox format is used by many POP3/IMAP servers, most mail servers (MTAs) and mail readers (MUAs). The Maildir format is used by Qmail, Courier-MTA and can be also used as a alternative mail storage format by Postfix and Exim or any MTA which can use procmail as MDA. POP3/IMAP servers which support Maildirs are e.g. Courier IMAP and Dovecot. mb2md does not only convert Mbox mailbox files into a Maildir but can also convert the /var/spool/mail/$USER mailspool file. It is smart enough to not transfer a dummy message such as the UW IMAPD puts at the start of Mbox mailboxes - and you could add your own search terms into the script to make it ignore other forms of dummy first message. I never repented the converting to maildir. Michael PS: Sorry for the mail I sent private to you, it was a misconfiguration of mutt. -- BOFH excuse #446: Mailer-daemon is busy burning your message in hell. signature.asc Description: PGP signature
Re: doveadm expunge didn't clear Trash mailbox
On Mon, 23 Jul 2018, Michael Wagner wrote: here works a dovecot 2.2.27 on a raspberrypi and the behaviour is as expected. doveadm -f tab fetch -u "uid date.saved" mailbox Trash uid date.saved 314 2018-06-23 00:35:59 315 2018-06-23 12:39:10 316 2018-06-24 10:32:43 ... And I have a cron script that expunges the mails older than 30 days. /usr/bin/doveadm expunge -u mailbox Trash savedbefore 30d Hi Michael, Thanks for your observation. I think you probably use maildir format, yes? I think my problem stems from the fact I use mbox, so one file contains many messages, whereas maildir uses one file for each message. Dovecot, from my understanding, will use the message file's mtime for date.saved if it doesn't have it in the cached. This is probably why I am seeing multiple messages with the same value. What's confusing is that if I query a message's date.saved, that value -- whether from the cache or the mailbox's mtime -- remains in the cache, so the next query will give me that value. So running "doveadm fetch ... date.saved", will get the mailbox's mtime at the time I fetched those values. In order to get near accurate date.saved values, I would have to run "doveadm fetch ... date.saved" frequently. (I think this behaviour must have changed recently because this no longer works consistently.) However, if each message is stored in a separate file (like maildir does), then each message can have an accurate date.saved(=mtime). Once you have accurate date.saved, then the "expunge savedbefore" works correctly. Another reason I really ought to switch to maildir. Joseph Tam
Re: doveadm expunge didn't clear Trash mailbox
On Jul 18, 2018 um 15:05:46, Joseph Tam wrote:
> On Wed, 18 Jul 2018, Sophie Loewenthal wrote:
>
> > Why did my 'doveadm expunge' command not clear the Trash out of a mailbox?
> >
> > The oldest messages was put into the Trash on the 26th June 2018 as shown
> > in the file timestamp.
> >
> > -rw--- 1 vmail mail 33056 Jun 26 16:46
> > 1530031582.M768773P18242.mx10,S=33056,W=33510:2,RS
> >
> > # doveadm -D expunge -u testu...@example.org mailbox Trash savedbefore 2d
>
> I was going to suggest that you dump the savedbefore dates on all your
> messages to see what Dovecot thinks your message's timestamps are.
> I tried it out myself and was surprised to find *all* messages in *any*
> mailbox I looked at gave the same timestamp -- probably that of the
> latest message.
>
> For example,
>
> # doveadm -f tab fetch -u {user} "uid date.saved" mailbox Trash
> uid date.saved
> 23624 2018-07-18 14:50:03
> 23625 2018-07-18 14:50:03
> [... 3657 lines removed ...]
> 27295 2018-07-18 14:50:03
> 27296 2018-07-18 14:50:03
> 27297 2018-07-18 14:50:03
>
> # doveadm -f tab fetch -u {user} "uid date.saved" mailbox INBOX
> uid date.saved
> 94878 2018-07-18 14:53:46
> 100366 2018-07-18 14:53:46
> [... 140 lines removed ...]
> 102313 2018-07-18 14:53:46
> 102314 2018-07-18 14:53:46
> 102315 2018-07-18 14:53:46
>
> This would explain why the cron script I use to blow away old deleted mail
> has not fired in ages.
>
> Is this a bug, or am I not understanding what "savedbefore" means now? It
> certainly has changed behaviour since <2.2, since my script used to
> work before.
Hello Joseph,
here works a dovecot 2.2.27 on a raspberrypi and the behaviour is as
expected.
doveadm -f tab fetch -u "uid date.saved" mailbox Trash
uid date.saved
314 2018-06-23 00:35:59
315 2018-06-23 12:39:10
316 2018-06-24 10:32:43
317 2018-06-24 10:32:43
318 2018-06-24 10:32:47
319 2018-06-24 14:26:24
320 2018-06-24 14:26:24
321 2018-06-24 14:28:45
322 2018-06-24 14:28:45
-- lines skipped
574 2018-07-20 20:58:21
575 2018-07-21 17:22:17
576 2018-07-22 15:15:18
577 2018-07-22 15:28:28
578 2018-07-22 15:28:28
579 2018-07-22 15:28:28
580 2018-07-22 16:12:47
581 2018-07-23 13:27:12
582 2018-07-23 12:55:26
583 2018-07-23 13:19:01
584 2018-07-23 13:19:01
585 2018-07-23 13:49:01
586 2018-07-23 13:52:46
And I have a cron script that expunges the mails older than 30 days.
/usr/bin/doveadm expunge -u mailbox Trash savedbefore 30d
Michael
signature.asc
Description: PGP signature
Re: doveadm expunge didn't clear Trash mailbox
On Wed, 18 Jul 2018, Joseph Tam wrote:
I wrote about a related issue several years back, starting with
https://dovecot.org/list/dovecot/2012-March/134706.html
I'm not sure how I got over this and got usable "date.saved" cache values,
and why it stopped working. Maybe I need to set something like
mail_always_cache_fields = date.saved
mail_cache_fields = date.saved
to force date.saved to be cached.
This doesn't work -- it's not a cachable value.
To get around the wonky date.saved values, I used date.received
instead, which seems to have more reliable values
doveadm -f tab fetch -u {user} "uid date.received" mailbox {box}
and the corresponding expunge command would be
doveadm expunge -u {user} mailbox {box} BEFORE {time-limit}
It not quite the same as it's not the time it was copied into
{box} but rather, the internal date, which may be earlier (i.e when
it was received).
Joseph Tam
Re: doveadm expunge didn't clear Trash mailbox
I wrote:
I was going to suggest that you dump the savedbefore dates on all your
messages to see what Dovecot thinks your message's timestamps are.
I tried it out myself and was surprised to find *all* messages in *any*
mailbox I looked at gave the same timestamp -- probably that of the
latest message.
For example,
# doveadm -f tab fetch -u {user} "uid date.saved" mailbox Trash
uid date.saved
23624 2018-07-18 14:50:03
23625 2018-07-18 14:50:03
[... 3657 lines removed ...]
27295 2018-07-18 14:50:03
27296 2018-07-18 14:50:03
27297 2018-07-18 14:50:03
I wrote about a related issue several years back, starting with
https://dovecot.org/list/dovecot/2012-March/134706.html
I'm not sure how I got over this and got usable "date.saved" cache values,
and why it stopped working. Maybe I need to set something like
mail_always_cache_fields = date.saved
mail_cache_fields = date.saved
to force date.saved to be cached.
On Thu, 19 Jul 2018, Sophie Loewenthal wrote:
And here is mine. All dates are todays!
# doveadm -f tab fetch -u t...@example.org "uid date.saved" mailbox Trash
uid date.saved
68292018-07-18 06:04:50
68302018-07-18 11:10:13
68312018-07-18 16:16:15
SNIP a lot of lines...
68732018-07-18 18:54:07
68742018-07-18 18:54:07
68752018-07-18 18:54:07
68762018-07-18 18:54:07
68772018-07-18 21:23:08
You may or may not have the same problem I do -- your timestamps actually
differ, whereas mine are all the same.
Joseph Tam
Re: doveadm expunge didn't clear Trash mailbox
Sorry, mistake:
I meant all dates in Trash are dated today, but the times differ.
Inbox is different, but the last file in user@domain/cur differs from dovecot
fetch date.saved.
-rw--- 1 vmail mail3566 2011-10-08 13:06:16.0 +
1371065422.M299803P23725.logout,S=3566,W=3659:2,Shj
-rw--- 1 vmail mail 14621 2011-09-16 11:49:45.0 +
1507747187.M109389P30107.logout,W=14988,S=14621:2,Shj
> On 19 Jul 2018, at 00:17, Sophie Loewenthal wrote:
>
> And here is mine. All dates are todays!
>
> # doveadm -f tab fetch -u t...@example.org "uid date.saved" mailbox Trash
> uid date.saved
> 6829 2018-07-18 06:04:50
> 6830 2018-07-18 11:10:13
> 6831 2018-07-18 16:16:15
> 6832 2018-07-18 16:16:15
> 6833 2018-07-18 08:22:55
> 6834 2018-07-18 16:16:15
> 6835 2018-07-18 08:47:10
> 6836 2018-07-18 09:12:11
> 6837 2018-07-18 09:53:31
> 6838 2018-07-18 11:03:31
> SNIP a lot of lines...
> 6873 2018-07-18 18:54:07
> 6874 2018-07-18 18:54:07
> 6875 2018-07-18 18:54:07
> 6876 2018-07-18 18:54:07
> 6877 2018-07-18 21:23:08
>
>
> # doveadm -f tab fetch -u sop...@klunky.org "uid date.saved" mailbox
> Inbox|head
> uid date.saved
> 572018-01-23 21:05:06
> 602018-01-23 21:05:06
> 612018-01-23 21:05:06
> 622018-01-23 21:05:06
> 642018-01-23 21:05:06
> SNIP a lot of lines
> 7734 2018-07-02 20:48:37
> 7780 2018-07-06 18:50:37
> 7794 2018-07-09 08:03:45
> 7833 2018-07-11 15:27:06
>
> This is on Debian 9.5 with dovecot 1:2.2.27-3+deb9u2
>
>
>
>
>> On 19 Jul 2018, at 00:05, Joseph Tam wrote:
>>
>> On Wed, 18 Jul 2018, Sophie Loewenthal wrote:
>>
>>> Why did my 'doveadm expunge' command not clear the Trash out of a mailbox?
>>>
>>> The oldest messages was put into the Trash on the 26th June 2018 as shown
>>> in the file timestamp.
>>>
>>> -rw--- 1 vmail mail 33056 Jun 26 16:46
>>> 1530031582.M768773P18242.mx10,S=33056,W=33510:2,RS
>>>
>>> # doveadm -D expunge -u testu...@example.org mailbox Trash savedbefore 2d
>>
>> I was going to suggest that you dump the savedbefore dates on all your
>> messages to see what Dovecot thinks your message's timestamps are.
>> I tried it out myself and was surprised to find *all* messages in *any*
>> mailbox I looked at gave the same timestamp -- probably that of the
>> latest message.
>>
>> For example,
>>
>> # doveadm -f tab fetch -u {user} "uid date.saved" mailbox Trash
>> uid date.saved
>> 23624 2018-07-18 14:50:03
>> 23625 2018-07-18 14:50:03
>> [... 3657 lines removed ...]
>> 27295 2018-07-18 14:50:03
>> 27296 2018-07-18 14:50:03
>> 27297 2018-07-18 14:50:03
>>
>> # doveadm -f tab fetch -u {user} "uid date.saved" mailbox INBOX
>> uid date.saved
>> 94878 2018-07-18 14:53:46
>> 100366 2018-07-18 14:53:46
>> [... 140 lines removed ...]
>> 102313 2018-07-18 14:53:46
>> 102314 2018-07-18 14:53:46
>> 102315 2018-07-18 14:53:46
>>
>> This would explain why the cron script I use to blow away old deleted mail
>> has not fired in ages.
>>
>> Is this a bug, or am I not understanding what "savedbefore" means now? It
>> certainly has changed behaviour since <2.2, since my script used to
>> work before.
>>
>> Joseph Tam
>
Re: doveadm expunge didn't clear Trash mailbox
And here is mine. All dates are todays!
# doveadm -f tab fetch -u t...@example.org "uid date.saved" mailbox Trash
uid date.saved
68292018-07-18 06:04:50
68302018-07-18 11:10:13
68312018-07-18 16:16:15
68322018-07-18 16:16:15
68332018-07-18 08:22:55
68342018-07-18 16:16:15
68352018-07-18 08:47:10
68362018-07-18 09:12:11
68372018-07-18 09:53:31
68382018-07-18 11:03:31
SNIP a lot of lines...
68732018-07-18 18:54:07
68742018-07-18 18:54:07
68752018-07-18 18:54:07
68762018-07-18 18:54:07
68772018-07-18 21:23:08
# doveadm -f tab fetch -u sop...@klunky.org "uid date.saved" mailbox Inbox|head
uid date.saved
57 2018-01-23 21:05:06
60 2018-01-23 21:05:06
61 2018-01-23 21:05:06
62 2018-01-23 21:05:06
64 2018-01-23 21:05:06
SNIP a lot of lines
77342018-07-02 20:48:37
77802018-07-06 18:50:37
77942018-07-09 08:03:45
78332018-07-11 15:27:06
This is on Debian 9.5 with dovecot 1:2.2.27-3+deb9u2
> On 19 Jul 2018, at 00:05, Joseph Tam wrote:
>
> On Wed, 18 Jul 2018, Sophie Loewenthal wrote:
>
>> Why did my 'doveadm expunge' command not clear the Trash out of a mailbox?
>>
>> The oldest messages was put into the Trash on the 26th June 2018 as shown in
>> the file timestamp.
>>
>> -rw--- 1 vmail mail 33056 Jun 26 16:46
>> 1530031582.M768773P18242.mx10,S=33056,W=33510:2,RS
>>
>> # doveadm -D expunge -u testu...@example.org mailbox Trash savedbefore 2d
>
> I was going to suggest that you dump the savedbefore dates on all your
> messages to see what Dovecot thinks your message's timestamps are.
> I tried it out myself and was surprised to find *all* messages in *any*
> mailbox I looked at gave the same timestamp -- probably that of the
> latest message.
>
> For example,
>
> # doveadm -f tab fetch -u {user} "uid date.saved" mailbox Trash
> uid date.saved
> 23624 2018-07-18 14:50:03
> 23625 2018-07-18 14:50:03
> [... 3657 lines removed ...]
> 27295 2018-07-18 14:50:03
> 27296 2018-07-18 14:50:03
> 27297 2018-07-18 14:50:03
>
> # doveadm -f tab fetch -u {user} "uid date.saved" mailbox INBOX
> uid date.saved
> 94878 2018-07-18 14:53:46
> 100366 2018-07-18 14:53:46
> [... 140 lines removed ...]
> 102313 2018-07-18 14:53:46
> 102314 2018-07-18 14:53:46
> 102315 2018-07-18 14:53:46
>
> This would explain why the cron script I use to blow away old deleted mail
> has not fired in ages.
>
> Is this a bug, or am I not understanding what "savedbefore" means now? It
> certainly has changed behaviour since <2.2, since my script used to
> work before.
>
> Joseph Tam
Re: doveadm expunge didn't clear Trash mailbox
On Wed, 18 Jul 2018, Sophie Loewenthal wrote:
Why did my 'doveadm expunge' command not clear the Trash out of a mailbox?
The oldest messages was put into the Trash on the 26th June 2018 as shown in
the file timestamp.
-rw--- 1 vmail mail 33056 Jun 26 16:46
1530031582.M768773P18242.mx10,S=33056,W=33510:2,RS
# doveadm -D expunge -u testu...@example.org mailbox Trash savedbefore 2d
I was going to suggest that you dump the savedbefore dates on all your
messages to see what Dovecot thinks your message's timestamps are.
I tried it out myself and was surprised to find *all* messages in *any*
mailbox I looked at gave the same timestamp -- probably that of the
latest message.
For example,
# doveadm -f tab fetch -u {user} "uid date.saved" mailbox Trash
uid date.saved
23624 2018-07-18 14:50:03
23625 2018-07-18 14:50:03
[... 3657 lines removed ...]
27295 2018-07-18 14:50:03
27296 2018-07-18 14:50:03
27297 2018-07-18 14:50:03
# doveadm -f tab fetch -u {user} "uid date.saved" mailbox INBOX
uid date.saved
94878 2018-07-18 14:53:46
100366 2018-07-18 14:53:46
[... 140 lines removed ...]
102313 2018-07-18 14:53:46
102314 2018-07-18 14:53:46
102315 2018-07-18 14:53:46
This would explain why the cron script I use to blow away old deleted mail
has not fired in ages.
Is this a bug, or am I not understanding what "savedbefore" means now? It
certainly has changed behaviour since <2.2, since my script used to
work before.
Joseph Tam
doveadm expunge didn't clear Trash mailbox
Hi, Why did my 'doveadm expunge' command not clear the Trash out of a mailbox? The oldest messages was put into the Trash on the 26th June 2018 as shown in the file timestamp. -rw--- 1 vmail mail 33056 Jun 26 16:46 1530031582.M768773P18242.mx10,S=33056,W=33510:2,RS # doveadm -D expunge -u testu...@example.org mailbox Trash savedbefore 2d Debug: Loading modules from directory: /usr/lib/dovecot/modules/doveadm Debug: Skipping module doveadm_acl_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/lib10_doveadm_acl_plugin.so: undefined symbol: acl_lookup_dict_iterate_visible_next (this is usually intentional, so just ignore this message) Debug: Skipping module doveadm_expire_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/lib10_doveadm_expire_plugin.so: undefined symbol: expire_set_deinit (this is usually intentional, so just ignore this message) Debug: Skipping module doveadm_quota_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/lib10_doveadm_quota_plugin.so: undefined symbol: quota_user_module (this is usually intentional, so just ignore this message) Debug: Module loaded: /usr/lib/dovecot/modules/doveadm/lib10_doveadm_sieve_plugin.so Debug: Skipping module doveadm_fts_lucene_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/lib20_doveadm_fts_lucene_plugin.so: undefined symbol: lucene_index_iter_deinit (this is usually intentional, so just ignore this message) Debug: Skipping module doveadm_fts_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/lib20_doveadm_fts_plugin.so: undefined symbol: fts_user_get_language_list (this is usually intentional, so just ignore this message) Debug: Skipping module doveadm_mail_crypt_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/libdoveadm_mail_crypt_plugin.so: undefined symbol: mail_crypt_box_get_pvt_digests (this is usually intentional, so just ignore this message) doveadm(testu...@example.org): Debug: Added userdb setting: mail=maildir:/var/vmail/example.org/testu...@example.org doveadm(testu...@example.org): Debug: Added userdb setting: plugin/quota=dirsize:storage= doveadm(testu...@example.org): Debug: Effective uid=150, gid=8, home=/var/vmail/example.org/testu...@example.org doveadm(testu...@example.org): Debug: Namespace inbox: type=private, prefix=, sep=, inbox=yes, hidden=no, list=yes, subscriptions=yes location=maildir:/var/vmail/example.org/testu...@example.org doveadm(testu...@example.org): Debug: maildir++: root=/var/vmail/example.org/testu...@example.org, index=, indexpvt=, control=, inbox=/var/vmail/example.org/testu...@example.org, alt= Sophie
