Re: Modify stored mail contents?

[Evan Martin]

On 23/07/2017 3:30 PM, Sami Ketola wrote:

On 23 Jul 2017, at 16.14, Tom Hendrikx  wrote:

In general, you should not do this. When a message is stored using IMAP,
it is immutable. The IMAP server also remembers things like size and
assigns messages a unique ID, so mail readers that have already
downloaded the message with that ID, don't have to download the whole
message again to verify whether it's contents have magically changed.
What you're suggesting is not simply compatible with IMAP standards.

The normal way of applying changes to messages is just like a mail
client connecting to IMAP: create a new message and save it to the
store, then delete the old one.

Just like that. But instead of using IMAP interface to do it, you can also do
it with doveadm:

1. doveadm fetch mail
2. doveadm delete mail
3. modify fetched mail
4. doveadm import modified mail back

Simple as that. There is no other supported way to do it. Editing mail objects
on storage will break things.

Sami


Thank you, that's an interesting idea. Exporting the mail and deleting 
it is easy enough, but I'm not sure where I would import the message 
from. doveadm import expects a mailbox store as source, so I'm back to 
the problem of safely writing emails in a mailbox store format that 
Dovecot understands!


I can export to Maildir format [doveadm backup -u USER 
"maildir:/mymaildir:LAYOUT=fs"], where each file contains one message 
and nothing else, but even that has extra files like dovecot-uidlist, 
dovecot.index.cache, etc. I'm not sure if it's safe to import from a 
Maildir where the message contents have been modified, but the other 
files haven't.


Also, would the import create new UIDs? I probably want UIDs to change, 
so that the IMAP client re-downloads the messages. I'd want to preserve 
IMAP flags like "Seen", though and, ideally, the sequence of messages 
inside a mailbox.

Re: Modify stored mail contents?

[Sami Ketola]

> On 23 Jul 2017, at 16.14, Tom Hendrikx  wrote:
> 
> In general, you should not do this. When a message is stored using IMAP,
> it is immutable. The IMAP server also remembers things like size and
> assigns messages a unique ID, so mail readers that have already
> downloaded the message with that ID, don't have to download the whole
> message again to verify whether it's contents have magically changed.
> What you're suggesting is not simply compatible with IMAP standards.
> 
> The normal way of applying changes to messages is just like a mail
> client connecting to IMAP: create a new message and save it to the
> store, then delete the old one.

Just like that. But instead of using IMAP interface to do it, you can also do 
it with doveadm:

1. doveadm fetch mail
2. doveadm delete mail
3. modify fetched mail
4. doveadm import modified mail back

Simple as that. There is no other supported way to do it. Editing mail objects
on storage will break things.

Sami

Re: Modify stored mail contents?

[Tom Hendrikx]

On 23-07-17 13:07, Evan Martin wrote:
> It looks like the mail filter plugin
> [https://wiki2.dovecot.org/Plugins/MailFilter] is almost exactly what I
> want, except for this:
> 
>> Currently the filtering must not modify the message in any way: mail
> -> write filter -> read filter -> must produce exactly the original mail
> back.
>> (TODO: Modifying the mail during writing would be possible with some
> code changes.)
> 
> Is there any prospect of those code changes being made, so that the
> filter can modify mail contents?
> 
> There's no indication in the docs or the code of what would break if the
> contents were modified, but I'm guessing indexes and caches would be out
> of date and would need to be rebuilt? Is it possible to just disable
> those? I don't need high performance.
> 
> On 22/07/2017 12:51 PM, Evan Martin wrote:
>> Is there a safe way to modify the contents of emails stored by
>> Dovecot? I'll probably only want to change the message bodies, not the
>> headers, if that matters. Looking for ways to do this both for
>> existing emails and new emails as they are received (though anything
>> that works for existing emails can probably just be run again for new
>> emails.) My mail storage is currently mdbox, but I could migrate to
>> another format if that helps.

In general, you should not do this. When a message is stored using IMAP,
it is immutable. The IMAP server also remembers things like size and
assigns messages a unique ID, so mail readers that have already
downloaded the message with that ID, don't have to download the whole
message again to verify whether it's contents have magically changed.
What you're suggesting is not simply compatible with IMAP standards.

The normal way of applying changes to messages is just like a mail
client connecting to IMAP: create a new message and save it to the
store, then delete the old one.

Two people already asked you in this thread what the actual problem is
you're trying to solve, but you failed to answer the question. I'm
afraid that we can't help you any further without more detail.

Kind regards,
Tom



signature.asc
Description: OpenPGP digital signature

Re: Modify stored mail contents?

[Evan Martin]

It looks like the mail filter plugin 
[https://wiki2.dovecot.org/Plugins/MailFilter] is almost exactly what I 
want, except for this:


> Currently the filtering must not modify the message in any way: mail 
-> write filter -> read filter -> must produce exactly the original mail 
back.
> (TODO: Modifying the mail during writing would be possible with some 
code changes.)


Is there any prospect of those code changes being made, so that the 
filter can modify mail contents?


There's no indication in the docs or the code of what would break if the 
contents were modified, but I'm guessing indexes and caches would be out 
of date and would need to be rebuilt? Is it possible to just disable 
those? I don't need high performance.


On 22/07/2017 12:51 PM, Evan Martin wrote:
Is there a safe way to modify the contents of emails stored by 
Dovecot? I'll probably only want to change the message bodies, not the 
headers, if that matters. Looking for ways to do this both for 
existing emails and new emails as they are received (though anything 
that works for existing emails can probably just be run again for new 
emails.) My mail storage is currently mdbox, but I could migrate to 
another format if that helps.

Re: Modify stored mail contents?

[James]

On 22/07/2017 11:51, Evan Martin wrote:

Is there a safe way to modify the contents of emails stored by Dovecot?
I'll probably only want to change the message bodies, not the headers,
if that matters.


I use javamail[1] for this.  The javamail API can be used to get and put 
emails with IMAP so dovecot is kept in sync with the changes you make 
between fetching and storing.



1. https://javaee.github.io/javamail/

Re: Modify stored mail contents?

[Jerry]

>> On Sat, 22 Jul 2017 12:51:15 +0200, Evan Martin stated:
>>  
>>> Is there a safe way to modify the contents of emails stored by
>>> Dovecot? I'll probably only want to change the message bodies, not
>>> the headers, if that matters. Looking for ways to do this both for
>>> existing emails and new emails as they are received (though anything
>>> that works for existing emails can probably just be run again for
>>> new emails.) My mail storage is currently mdbox, but I could
>>> migrate to another format if that helps.  

> On 22/07/2017 3:56 PM, Jerry wrote:

>> You could just view the message in your MUA and then save it to
>> another drive, or whatever. Then, using a text editor, you could
>> modify it to your hearts content.
>>
>> Exactly, what problem are you trying to address?

On Sat, 22 Jul 2017 16:05:19 +0200, Evan Martin stated:

> Yes, obviously clients can save messages. I meant: to modify messages
> in bulk, on the server, replacing the existing message bodies stored
> by Dovecot.

I am not understanding what problem you are trying to alleviate. Are
these your messages or those of your clients?

-- 
Jerry

Re: Modify stored mail contents?

[Evan Martin]

Yes, obviously clients can save messages. I meant: to modify messages in 
bulk, on the server, replacing the existing message bodies stored by 
Dovecot.


On 22/07/2017 3:56 PM, Jerry wrote:

On Sat, 22 Jul 2017 12:51:15 +0200, Evan Martin stated:


Is there a safe way to modify the contents of emails stored by
Dovecot? I'll probably only want to change the message bodies, not the
headers, if that matters. Looking for ways to do this both for
existing emails and new emails as they are received (though anything
that works for existing emails can probably just be run again for new
emails.) My mail storage is currently mdbox, but I could migrate to
another format if that helps.

You could just view the message in your MUA and then save it to another
drive, or whatever. Then, using a text editor, you could modify it to
your hearts content.

Exactly, what problem are you trying to address?

Re: Modify stored mail contents?

[Jerry]

On Sat, 22 Jul 2017 12:51:15 +0200, Evan Martin stated:

>Is there a safe way to modify the contents of emails stored by
>Dovecot? I'll probably only want to change the message bodies, not the
>headers, if that matters. Looking for ways to do this both for
>existing emails and new emails as they are received (though anything
>that works for existing emails can probably just be run again for new
>emails.) My mail storage is currently mdbox, but I could migrate to
>another format if that helps.

You could just view the message in your MUA and then save it to another
drive, or whatever. Then, using a text editor, you could modify it to
your hearts content.

Exactly, what problem are you trying to address?

-- 
Jerry