Re: Deleting messages marked for deletion older than X days
On Tue, 2008-08-19 at 10:05 +1000, Bron Gondwana wrote: On Mon, Aug 18, 2008 at 05:09:53PM -0500, Kenneth Marshall wrote: In the manual page, the definition of the '-X' option seems to do what you want: -X expunge-days Expunge previously deleted messages older than expunge-days (when using the delayed expunge mode). The default is 0 (zero) days, which will expunge all previously deleted messages. Now, what the -X option actually does is turns this into: but the file is still on disk, just the index record has been moved from the file cyrus.index to a new file cyrus.expunge. A week later: Cleaned up (no file) - cyr_expire -X 7 The cyrus.expunge record and the actual spool file itself get deleted at this point. Until then you can un-delete the record using the unexpunge command in cyrus 2.3.X. I'm working on data-retention for my employer and I was wondering about this. The wording is a bit vague in the manual page (we are now using deleted to mean multiple things...). I have assumed that -X 7 means purge-the-message-seven-days-AFTER-IT-WAS-EXPUNGED. Is this correct? I believe the manual page could also be read to mean purge-the-message-AFTER-IT-WAS-EXPUNGED-AND-IS-OLDER-THAN-SEVEN-DAYS. OLDER-THAN-SEVEN-DAYS meaning it was received more than seven days ago [equivalent to ipurge -d 7 -X]. I think what the original requestor was actually looking for is a tool that can run the EXPUNGE phase on a regular basis. As far as I'm aware there's nothing that ships with Cyrus that can do it. If I was writing something for the job I would make an admin IMAP connection to Cyrus and just cycle through the folders calling 'EXPUNGE' on them. Cheap and nasty, but it would do the trick. You can do this in any language with a TCP library, though something with an IMAP interface library would be nicer. I'd use Perl and Mail::IMAPTalk, but that's just because that's what I already use! I'd do it in python. :) I've done some similar things for walking the entire mailstore and building statistics, etc... but it is REALLY slow! A server side API for making tools like cyr_expire would be really nice. Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Deleting messages marked for deletion older than X days
Wesley, I looked at ipurge but I find it also happily deletes messages that weren't marked for deletion. It appears to delete everything older than X days, regardless of that \Deleted flag mentioned in another reply. Gerard Beekmans IT Manager Achilles Media Tel: 403-678-1216 ext. 102 Email: [EMAIL PROTECTED] Wesley Craig wrote: cyr_expire -D and -X deal with delayed delete expunge, respectively. Neither will address message that the MUA hasn't already deleted (in the case of folders) or expunged (in the case of messages). The tool that is closest to the desired functionality is ipurge. In order for ipurge to do what you're asking, Gerard, it would have to examine whatever the deleted flag is for Outlook -- it varies by MUA, some set a flag, some move messages to a trash folder, etc. :wes On 18 Aug 2008, at 18:09, Kenneth Marshall wrote: In the manual page, the definition of the '-X' option seems to do what you want: -X expunge-days Expunge previously deleted messages older than expunge-days (when using the delayed expunge mode). The default is 0 (zero) days, which will expunge all previously deleted messages. On Mon, Aug 18, 2008 at 03:35:20PM -0600, Gerard Beekmans wrote: I am looking for a way to run a daily script on the Cyrus IMAP server that processes every user's mail folders and deletes every message marked for deletion older than X days. I realize the proper way to do this is client-side and have the client program purge/expunge the mailboxes periodically in some fashion. Unfortunately in the corporate world there are many clients who still use old versions of Outlook that only marks a message for deletion and will not automatically purge the folder. To force users to manually purge every single folder they have, one folder at a time as some versions of Outlook can't do this recursively, is not a feasible option. Users just stopped doing this because it becomes too time consuming a task. So I'm looking for a server-side solution that analyzes every mail folder on the system, finds marked messages older than X days and purges/deletes them on the client's behalf. Some pointers are appreciated. I can write the scripts (shell, perl, whatever) myself if need be, just need to know how to start going about obtaining such a specific list of messages. Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Deleting messages marked for deletion older than X days
Hi Bron, Yes, it sounds like that's what I'm looking for: running expunge periodically. Now there's one slight problem here: I can't just run expunge as it is. I have users who periodically need to undelete messages. I'd like expunge to only act on messages older than X days, giving people the chance to undelete messages within that time period. I'll check out some IMAP libraries. Perhaps one offers to obtain a list of all messages with the \Deleted flag, then check its received time stamp and do the math on that. If older than X, delete/expunge/whatever it, else skip it. Gerard Beekmans IT Manager Achilles Media Tel: 403-678-1216 ext. 102 Email: [EMAIL PROTECTED] Bron Gondwana wrote: On Mon, Aug 18, 2008 at 05:09:53PM -0500, Kenneth Marshall wrote: In the manual page, the definition of the '-X' option seems to do what you want: -X expunge-days Expunge previously deleted messages older than expunge-days (when using the delayed expunge mode). The default is 0 (zero) days, which will expunge all previously deleted messages. Messages go through the following life cycle in traditional IMAP, example: Created (\Recent)- LMTP DELIVER (UID = 9) No flags () - A001 SELECT INBOX (clears the \Recent) Viewed (\Seen) - A002 UID FETCH 9 RFC822 Deleted (\Deleted \Seen) - A003 UID STORE 9 +FLAGS (\Deleted) Purged (no message) - A004 EXPUNGE Now, what the -X option actually does is turns this into: Created (\Recent)- LMTP DELIVER (UID = 9) No flags () - A001 SELECT INBOX (clears the \Recent) Viewed (\Seen) - A002 UID FETCH 9 RFC822 Deleted (\Deleted \Seen) - A003 UID STORE 9 +FLAGS (\Deleted) Purged (no index record)- A004 EXPUNGE but the file is still on disk, just the index record has been moved from the file cyrus.index to a new file cyrus.expunge. A week later: Cleaned up (no file) - cyr_expire -X 7 The cyrus.expunge record and the actual spool file itself get deleted at this point. Until then you can un-delete the record using the unexpunge command in cyrus 2.3.X. --- I think what the original requestor was actually looking for is a tool that can run the EXPUNGE phase on a regular basis. As far as I'm aware there's nothing that ships with Cyrus that can do it. If I was writing something for the job I would make an admin IMAP connection to Cyrus and just cycle through the folders calling 'EXPUNGE' on them. Cheap and nasty, but it would do the trick. You can do this in any language with a TCP library, though something with an IMAP interface library would be nicer. I'd use Perl and Mail::IMAPTalk, but that's just because that's what I already use! Bron. Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Deleting messages marked for deletion older than X days
Hi, I am looking for a way to run a daily script on the Cyrus IMAP server that processes every user's mail folders and deletes every message marked for deletion older than X days. I realize the proper way to do this is client-side and have the client program purge/expunge the mailboxes periodically in some fashion. Unfortunately in the corporate world there are many clients who still use old versions of Outlook that only marks a message for deletion and will not automatically purge the folder. To force users to manually purge every single folder they have, one folder at a time as some versions of Outlook can't do this recursively, is not a feasible option. Users just stopped doing this because it becomes too time consuming a task. So I'm looking for a server-side solution that analyzes every mail folder on the system, finds marked messages older than X days and purges/deletes them on the client's behalf. Some pointers are appreciated. I can write the scripts (shell, perl, whatever) myself if need be, just need to know how to start going about obtaining such a specific list of messages. Thanks, -- Gerard Beekmans IT Manager Achilles Media Tel: 403-678-1216 ext. 102 Email: [EMAIL PROTECTED] Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Deleting messages marked for deletion older than X days
I think that the cyr_expire command may be what you are searching for. Ken On Mon, Aug 18, 2008 at 03:35:20PM -0600, Gerard Beekmans wrote: Hi, I am looking for a way to run a daily script on the Cyrus IMAP server that processes every user's mail folders and deletes every message marked for deletion older than X days. I realize the proper way to do this is client-side and have the client program purge/expunge the mailboxes periodically in some fashion. Unfortunately in the corporate world there are many clients who still use old versions of Outlook that only marks a message for deletion and will not automatically purge the folder. To force users to manually purge every single folder they have, one folder at a time as some versions of Outlook can't do this recursively, is not a feasible option. Users just stopped doing this because it becomes too time consuming a task. So I'm looking for a server-side solution that analyzes every mail folder on the system, finds marked messages older than X days and purges/deletes them on the client's behalf. Some pointers are appreciated. I can write the scripts (shell, perl, whatever) myself if need be, just need to know how to start going about obtaining such a specific list of messages. Thanks, -- Gerard Beekmans IT Manager Achilles Media Tel: 403-678-1216 ext. 102 Email: [EMAIL PROTECTED] Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Deleting messages marked for deletion older than X days
Hi Ken, From the description it seems like cyr_expire will apply to every email in a mailbox. I don't want to delete any regular emails other than the ones already marked for deletion by an email client who is lacking a decent purge feature. Gerard Beekmans IT Manager Achilles Media Tel: 403-678-1216 ext. 102 Email: [EMAIL PROTECTED] Kenneth Marshall wrote: I think that the cyr_expire command may be what you are searching for. Ken On Mon, Aug 18, 2008 at 03:35:20PM -0600, Gerard Beekmans wrote: Hi, I am looking for a way to run a daily script on the Cyrus IMAP server that processes every user's mail folders and deletes every message marked for deletion older than X days. I realize the proper way to do this is client-side and have the client program purge/expunge the mailboxes periodically in some fashion. Unfortunately in the corporate world there are many clients who still use old versions of Outlook that only marks a message for deletion and will not automatically purge the folder. To force users to manually purge every single folder they have, one folder at a time as some versions of Outlook can't do this recursively, is not a feasible option. Users just stopped doing this because it becomes too time consuming a task. So I'm looking for a server-side solution that analyzes every mail folder on the system, finds marked messages older than X days and purges/deletes them on the client's behalf. Some pointers are appreciated. I can write the scripts (shell, perl, whatever) myself if need be, just need to know how to start going about obtaining such a specific list of messages. Thanks, -- Gerard Beekmans IT Manager Achilles Media Tel: 403-678-1216 ext. 102 Email: [EMAIL PROTECTED] Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Deleting messages marked for deletion older than X days
In the manual page, the definition of the '-X' option seems to do what you want: -X expunge-days Expunge previously deleted messages older than expunge-days (when using the delayed expunge mode). The default is 0 (zero) days, which will expunge all previously deleted messages. Ken On Mon, Aug 18, 2008 at 03:53:31PM -0600, Gerard Beekmans wrote: Hi Ken, From the description it seems like cyr_expire will apply to every email in a mailbox. I don't want to delete any regular emails other than the ones already marked for deletion by an email client who is lacking a decent purge feature. Gerard Beekmans IT Manager Achilles Media Tel: 403-678-1216 ext. 102 Email: [EMAIL PROTECTED] Kenneth Marshall wrote: I think that the cyr_expire command may be what you are searching for. Ken On Mon, Aug 18, 2008 at 03:35:20PM -0600, Gerard Beekmans wrote: Hi, I am looking for a way to run a daily script on the Cyrus IMAP server that processes every user's mail folders and deletes every message marked for deletion older than X days. I realize the proper way to do this is client-side and have the client program purge/expunge the mailboxes periodically in some fashion. Unfortunately in the corporate world there are many clients who still use old versions of Outlook that only marks a message for deletion and will not automatically purge the folder. To force users to manually purge every single folder they have, one folder at a time as some versions of Outlook can't do this recursively, is not a feasible option. Users just stopped doing this because it becomes too time consuming a task. So I'm looking for a server-side solution that analyzes every mail folder on the system, finds marked messages older than X days and purges/deletes them on the client's behalf. Some pointers are appreciated. I can write the scripts (shell, perl, whatever) myself if need be, just need to know how to start going about obtaining such a specific list of messages. Thanks, -- Gerard Beekmans IT Manager Achilles Media Tel: 403-678-1216 ext. 102 Email: [EMAIL PROTECTED] Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Deleting messages marked for deletion older than X days
I'll do some more checking then. My cyr_expire version doesn't even have the -X option (I'm testing this on a system that has cyrus imap 2.2.13 as its recent version). Gerard Beekmans IT Manager Achilles Media Tel: 403-678-1216 ext. 102 Email: [EMAIL PROTECTED] Kenneth Marshall wrote: In the manual page, the definition of the '-X' option seems to do what you want: -X expunge-days Expunge previously deleted messages older than expunge-days (when using the delayed expunge mode). The default is 0 (zero) days, which will expunge all previously deleted messages. Ken On Mon, Aug 18, 2008 at 03:53:31PM -0600, Gerard Beekmans wrote: Hi Ken, From the description it seems like cyr_expire will apply to every email in a mailbox. I don't want to delete any regular emails other than the ones already marked for deletion by an email client who is lacking a decent purge feature. Gerard Beekmans IT Manager Achilles Media Tel: 403-678-1216 ext. 102 Email: [EMAIL PROTECTED] Kenneth Marshall wrote: I think that the cyr_expire command may be what you are searching for. Ken On Mon, Aug 18, 2008 at 03:35:20PM -0600, Gerard Beekmans wrote: Hi, I am looking for a way to run a daily script on the Cyrus IMAP server that processes every user's mail folders and deletes every message marked for deletion older than X days. I realize the proper way to do this is client-side and have the client program purge/expunge the mailboxes periodically in some fashion. Unfortunately in the corporate world there are many clients who still use old versions of Outlook that only marks a message for deletion and will not automatically purge the folder. To force users to manually purge every single folder they have, one folder at a time as some versions of Outlook can't do this recursively, is not a feasible option. Users just stopped doing this because it becomes too time consuming a task. So I'm looking for a server-side solution that analyzes every mail folder on the system, finds marked messages older than X days and purges/deletes them on the client's behalf. Some pointers are appreciated. I can write the scripts (shell, perl, whatever) myself if need be, just need to know how to start going about obtaining such a specific list of messages. Thanks, -- Gerard Beekmans IT Manager Achilles Media Tel: 403-678-1216 ext. 102 Email: [EMAIL PROTECTED] Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Deleting messages marked for deletion older than X days
It must be a 2.3 option, we are running 2.3.10+ here. Ken On Mon, Aug 18, 2008 at 04:21:29PM -0600, Gerard Beekmans wrote: I'll do some more checking then. My cyr_expire version doesn't even have the -X option (I'm testing this on a system that has cyrus imap 2.2.13 as its recent version). Gerard Beekmans IT Manager Achilles Media Tel: 403-678-1216 ext. 102 Email: [EMAIL PROTECTED] Kenneth Marshall wrote: In the manual page, the definition of the '-X' option seems to do what you want: -X expunge-days Expunge previously deleted messages older than expunge-days (when using the delayed expunge mode). The default is 0 (zero) days, which will expunge all previously deleted messages. Ken On Mon, Aug 18, 2008 at 03:53:31PM -0600, Gerard Beekmans wrote: Hi Ken, From the description it seems like cyr_expire will apply to every email in a mailbox. I don't want to delete any regular emails other than the ones already marked for deletion by an email client who is lacking a decent purge feature. Gerard Beekmans IT Manager Achilles Media Tel: 403-678-1216 ext. 102 Email: [EMAIL PROTECTED] Kenneth Marshall wrote: I think that the cyr_expire command may be what you are searching for. Ken On Mon, Aug 18, 2008 at 03:35:20PM -0600, Gerard Beekmans wrote: Hi, I am looking for a way to run a daily script on the Cyrus IMAP server that processes every user's mail folders and deletes every message marked for deletion older than X days. I realize the proper way to do this is client-side and have the client program purge/expunge the mailboxes periodically in some fashion. Unfortunately in the corporate world there are many clients who still use old versions of Outlook that only marks a message for deletion and will not automatically purge the folder. To force users to manually purge every single folder they have, one folder at a time as some versions of Outlook can't do this recursively, is not a feasible option. Users just stopped doing this because it becomes too time consuming a task. So I'm looking for a server-side solution that analyzes every mail folder on the system, finds marked messages older than X days and purges/deletes them on the client's behalf. Some pointers are appreciated. I can write the scripts (shell, perl, whatever) myself if need be, just need to know how to start going about obtaining such a specific list of messages. Thanks, -- Gerard Beekmans IT Manager Achilles Media Tel: 403-678-1216 ext. 102 Email: [EMAIL PROTECTED] Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Deleting messages marked for deletion older than X days
cyr_expire -D and -X deal with delayed delete expunge, respectively. Neither will address message that the MUA hasn't already deleted (in the case of folders) or expunged (in the case of messages). The tool that is closest to the desired functionality is ipurge. In order for ipurge to do what you're asking, Gerard, it would have to examine whatever the deleted flag is for Outlook -- it varies by MUA, some set a flag, some move messages to a trash folder, etc. :wes On 18 Aug 2008, at 18:09, Kenneth Marshall wrote: In the manual page, the definition of the '-X' option seems to do what you want: -X expunge-days Expunge previously deleted messages older than expunge-days (when using the delayed expunge mode). The default is 0 (zero) days, which will expunge all previously deleted messages. On Mon, Aug 18, 2008 at 03:35:20PM -0600, Gerard Beekmans wrote: I am looking for a way to run a daily script on the Cyrus IMAP server that processes every user's mail folders and deletes every message marked for deletion older than X days. I realize the proper way to do this is client-side and have the client program purge/expunge the mailboxes periodically in some fashion. Unfortunately in the corporate world there are many clients who still use old versions of Outlook that only marks a message for deletion and will not automatically purge the folder. To force users to manually purge every single folder they have, one folder at a time as some versions of Outlook can't do this recursively, is not a feasible option. Users just stopped doing this because it becomes too time consuming a task. So I'm looking for a server-side solution that analyzes every mail folder on the system, finds marked messages older than X days and purges/deletes them on the client's behalf. Some pointers are appreciated. I can write the scripts (shell, perl, whatever) myself if need be, just need to know how to start going about obtaining such a specific list of messages. Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Deleting messages marked for deletion older than X days
On Mon, Aug 18, 2008 at 05:09:53PM -0500, Kenneth Marshall wrote: In the manual page, the definition of the '-X' option seems to do what you want: -X expunge-days Expunge previously deleted messages older than expunge-days (when using the delayed expunge mode). The default is 0 (zero) days, which will expunge all previously deleted messages. Messages go through the following life cycle in traditional IMAP, example: Created (\Recent)- LMTP DELIVER (UID = 9) No flags () - A001 SELECT INBOX (clears the \Recent) Viewed (\Seen) - A002 UID FETCH 9 RFC822 Deleted (\Deleted \Seen) - A003 UID STORE 9 +FLAGS (\Deleted) Purged (no message) - A004 EXPUNGE Now, what the -X option actually does is turns this into: Created (\Recent)- LMTP DELIVER (UID = 9) No flags () - A001 SELECT INBOX (clears the \Recent) Viewed (\Seen) - A002 UID FETCH 9 RFC822 Deleted (\Deleted \Seen) - A003 UID STORE 9 +FLAGS (\Deleted) Purged (no index record)- A004 EXPUNGE but the file is still on disk, just the index record has been moved from the file cyrus.index to a new file cyrus.expunge. A week later: Cleaned up (no file) - cyr_expire -X 7 The cyrus.expunge record and the actual spool file itself get deleted at this point. Until then you can un-delete the record using the unexpunge command in cyrus 2.3.X. --- I think what the original requestor was actually looking for is a tool that can run the EXPUNGE phase on a regular basis. As far as I'm aware there's nothing that ships with Cyrus that can do it. If I was writing something for the job I would make an admin IMAP connection to Cyrus and just cycle through the folders calling 'EXPUNGE' on them. Cheap and nasty, but it would do the trick. You can do this in any language with a TCP library, though something with an IMAP interface library would be nicer. I'd use Perl and Mail::IMAPTalk, but that's just because that's what I already use! Bron. Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
