bug(?): trailing tab/whitespace in output

[Marc]

doveadm -v -f table search -u testtest mailbox Archive/2011 guid 
1de6d8632073c49f272bed211b71a8b6

returns this (echo ".$guid.")

.1de6d8632073c49f272bed211b71a8b6.

Re: Mass Stripping Attachments by Directory, Age, Size

[Joseph Tam]

On Thu, 18 Mar 2021, Plutocrat wrote:

I've been looking around for a solution to this problem. I want to prune down 
the attachments on a server before a migration. Some of the emails are 7 
years old and have 40Mb attachments, so this seems like a good opportunity to 
rationalize things. So perhaps I'd like to "Remove all attachments from 
emails older than 2 years, in the .Sent directory", or "Attachments over 10Mb 
anywhere in the mail tree"


I've found the strip_attachments.pl script here 
 which 
works fine on mbox (as tested on my local Thunderbird mboxes), but not on 
maildir which is on the dovecot server. My Perl isn't strong enough to 
re-purpose it.


It you have anything that works on mbox, it will probably work on Maildir
as each file can be considered a single message mbox.  You can combine
the script with

find ~user/MailDir -type f ... -exec /path/to/mbox-strip {} \;

The ... can be replaced with more file tests (like minimum size or age
or only within */cur/) to cut down on processing.

I wrote a gawk script to slim down a multi-Gb Outlook mbox
for a user, but it wasn't really complicated, just matching for
/^Content-Transfer-Encoding:.*base64/i header (virtually all bulky data
will be encoded this way), buffering the base64 data part, then outputting
it if it was small, or deleting/replacing/extracting it otherwise.

It was a one-off discarded tool but I can hunt for it if you're hard up.

I've looked at ripmime and mpack/munpack, and although they seem like useful 
tools to do the job of deconstructing the mail into its constituent parts, it 
doesn't seem to help in re-building the email. I think they could be used 
with a bit of study into mail MIME structure, and used with a helper script.


So before I take a deep dive into scripting my own solution, I just wanted to 
check if anyone else on the list has been through this and has some resources 
or pointers they can share, or maybe even someone to tell me "Duh, you can do 
it with doveadm of course".


MIMEDefang may help.

Joseph Tam 

Re: bug: some table header(?) output goes to stderr instead of stdout

[Joseph Tam]

On 2021-03-18, Marc  wrote:


[@ sbin]# doveadm -f table -o 
mail_location=mdbox_deleted:/home/popusers/testtest/mdbox:INDEX=/home/popindex/testtest/index
 fetch -u testtest 'guid' mailbox INBOX 2> /dev/null
3c967f33b8aea671f3551db1ea8e33e9
6fa01ccc103a7009c7b940657dbcd72c
ba955a6d6218950f42e5b0ee0a33a916


Strange -- my version (2.3.10) dumps headers to stdout, not stderr

# doveadm -f table fetch -u $user guid mailbox INBOX 2>/dev/null | cat 
-n | head -3
1  guid
2  8104226179c70d7cc248c9924cabdb8c
3  0813554a7ed4cf1e113f42a4cc8bc477


From a strictly design point of view, this seems more correct as the

header is part of the data, not "out of band" output like errors.  If headers
are pumped to stderr, you can always employ shell hacks (as a followup poster
did) to unify them to stdout

# doveadm -f table fetch ... 2>&1

Maybe it's better to add another formatter to avoid tricky parsing
or shell hacks e.g.

# doveadm -f tab-nohdr ...

Joseph Tam 

feature request: doveadm purge taking guid argument

[Marc]

Hi Timo,

What about giving doveadm purge an argument to delete a specific message guid?

Is that a lot of work?

RE: bug: some table header(?) output goes to stderr instead of stdout

[Marc]

> On 2021-03-18, Marc  wrote:
> >
> > [@ sbin]# doveadm -f table -o
> mail_location=mdbox_deleted:/home/popusers/testtest/mdbox:INDEX=/home/po
> pindex/testtest/index fetch -u testtest 'guid' mailbox INBOX 2>
> /dev/null
> > 3c967f33b8aea671f3551db1ea8e33e9
> > 6fa01ccc103a7009c7b940657dbcd72c
> > ba955a6d6218950f42e5b0ee0a33a916
> >
> > [@ sbin]# doveadm -f table -o
> mail_location=mdbox_deleted:/home/popusers/testtest/mdbox:INDEX=/home/po
> pindex/testtest/index fetch -u testtest 'guid' mailbox INBOX
> > guid
> > 3c967f33b8aea671f3551db1ea8e33e9
> > 6fa01ccc103a7009c7b940657dbcd72c
> > ba955a6d6218950f42e5b0ee0a33a916
> >
> >
> >
> 
> That's a useful feature, saves messing about to exclude the headers when
> you
> want to use the results to fetch messages as shown in doveadm-search(1).
> 

Using stderr/stdout like this is not according to standards. But I do get that, 
I was tryig to do that with tail -n +1, and thought wtf is it not working!!! ;)
There could be a nasty side effect, if someone starts redirecting 2>/dev/null 
because of this table header. He will not get a real error.

Some questions about code in lib-storage/index/maildir

[songliny]

Hi, Recently, I have read some code in dovecot 2.3.13. And I have 2 questions 
about some lines of code in lib-storage/index/maildir Firstly, in the function, 
maildir_scan_dir, from lib-storage/index/maildir/maildir-sync.c around line 575 
(https://github.com/dovecot/core/blob/6986a435e31dfe5bcd14985534b76b3c735f7831/src/lib-storage/index/maildir/maildir-sync.c#L575)
 I would like to know that after successfully stat the cur directory, does it 
assign the max value between st.st_mtime and start_time intensionally to 
new_check_time? Or maybe should it assign the value to the cur_check_time. if 
(stat(ctx->cur_dir, ) == 0) { ctx->mbox->maildir_hdr.new_check_time = 
I_MAX(st.st_mtime, start_time); ctx->mbox->maildir_hdr.cur_mtime = st.st_mtime; 
ctx->mbox->maildir_hdr.cur_mtime_nsecs = ST_MTIME_NSEC(st); } Secondly, in the 
function, maildir_uidlist_sync_next_uid, from 
lib-storage/index/maildir/maildir-uidlist.c 
(https://github.com/dovecot/core/blob/6986a435e31dfe5bcd14985534b76b3c735f7831/s

rc/lib-storage/index/maildir/maildir-uidlist.c#L1760) Is it ok to remove 
MAILDIR_UIDLIST_REC_FLAG_NEW_DIR flag from the record for the record that is 
not found in ctx->files just like the behavior 
(https://github.com/dovecot/core/blob/6986a435e31dfe5bcd14985534b76b3c735f7831/src/lib-storage/index/maildir/maildir-uidlist.c#L1723)in
 the function, maildir_uidlist_sync_next_partial 
(https://github.com/dovecot/core/blob/6986a435e31dfe5bcd14985534b76b3c735f7831/src/lib-storage/index/maildir/maildir-uidlist.c#L1723),
 from the same file? I think when it gets to the function, 
maildir_uidlist_sync_next_uid, the mail file have already been moved to cur 
directory. And maybe it is ok to remove MAILDIR_UIDLIST_REC_FLAG_NEW_DIR flag 
from the record. That is, is it ok to add this line rec->flags &= 
ENUM_NEGATE(MAILDIR_UIDLIST_REC_FLAG_NEW_DIR); just before rec->flags = 
(rec->flags | flags) & ENUM_NEGATE(MAILDIR_UIDLIST_REC_FLAG_NONSYNCED); in 
maildir_uidlist_sync_next_uid? Thank you in advance to he

lp me get more understanding to these questions, Sony

Re: bug: some table header(?) output goes to stderr instead of stdout

[Stuart Henderson]

On 2021-03-18, Marc  wrote:
>
> [@ sbin]# doveadm -f table -o 
> mail_location=mdbox_deleted:/home/popusers/testtest/mdbox:INDEX=/home/popindex/testtest/index
>  fetch -u testtest 'guid' mailbox INBOX 2> /dev/null
> 3c967f33b8aea671f3551db1ea8e33e9
> 6fa01ccc103a7009c7b940657dbcd72c
> ba955a6d6218950f42e5b0ee0a33a916
>
> [@ sbin]# doveadm -f table -o 
> mail_location=mdbox_deleted:/home/popusers/testtest/mdbox:INDEX=/home/popindex/testtest/index
>  fetch -u testtest 'guid' mailbox INBOX
> guid
> 3c967f33b8aea671f3551db1ea8e33e9
> 6fa01ccc103a7009c7b940657dbcd72c
> ba955a6d6218950f42e5b0ee0a33a916
>
>
>

That's a useful feature, saves messing about to exclude the headers when you
want to use the results to fetch messages as shown in doveadm-search(1).

bug: some table header(?) output goes to stderr instead of stdout

[Marc]

[@ sbin]# doveadm -f table -o 
mail_location=mdbox_deleted:/home/popusers/testtest/mdbox:INDEX=/home/popindex/testtest/index
 fetch -u testtest 'guid' mailbox INBOX 2> /dev/null
3c967f33b8aea671f3551db1ea8e33e9
6fa01ccc103a7009c7b940657dbcd72c
ba955a6d6218950f42e5b0ee0a33a916

[@ sbin]# doveadm -f table -o 
mail_location=mdbox_deleted:/home/popusers/testtest/mdbox:INDEX=/home/popindex/testtest/index
 fetch -u testtest 'guid' mailbox INBOX
guid
3c967f33b8aea671f3551db1ea8e33e9
6fa01ccc103a7009c7b940657dbcd72c
ba955a6d6218950f42e5b0ee0a33a916

Re: dovecot director and keepalived

[Steven Varco]

Hi Sebastiaan

Many thanks for that hint, it does work just perfectly, after I configured an 
address on the service director inet_listener! :)
Now each instance only binds to its own IP, which is exactly what I was looking 
for.

I think it would be helpfull, if this could be added as a note to the docs in 
the director configuration settings.

And to answer the questsions from others regarding my mentioned „haproxy" on 
the server:
haproxy is not directly in connection with doevecot at all, instead I use it 
for proxying the SMTP submission port to postifix until I can upgrade dovecot 
to 2.3 (which I believe should habe the smtp sumbmission service included). And 
it is also used for proxing to other services, like http for webmail, etc.

Regarding the dovecot setup, I just use keepalived -> dovecot director -> 
dovecot IMAP

Sorry for the confusion, might should have mentioned that in the beginning… ;)

thanks,
Steven

-- 
https://steven.varco.ch/ 


> Am 16.03.2021 um 13:39 schrieb Sebastiaan Hoogeveen 
> :
> 
> Hi Steven,
> 
> On 14/03/2021, 17:53, "dovecot on behalf of Steven Varco" 
>  wrote:
> 
>> Now I’m hitting the issue with the way director determines his „Self IP“ by 
>> trying to 
>> bind to all configured director_servers IPs, taking the first one possible.
>> 
>> However this approach only works, when the sysctl setting is: 
>> net.ipv4.ip_nonlocal_bind=0
>> On the other side keepalived needs net.ipv4.ip_nonlocal_bind=1 in order to 
>> bind the VIP.
> 
> This can be fixed by specifying the IP address of the director in the 
> inet_listener section of the director service, like this:
> 
>  service director {
>### other configuration here ###
>inet_listener {
>  address = 172.20.1.4
>  port = 9090
>} 
>  }
> 
> The listener address will be used as the 'self IP' of the director. This also 
> means that each director will have a slightly different configuration file, 
> but that should usually not be a problem. 
> 
> I got this from skimming the source, afaict it is not documented anywhere so 
> I'm not sure if this behavior can always be relied on in future releases (it 
> does seem logical to me though).
> 
> Kind regards,
> 
> --  
> Sebastiaan Hoogeveen
> 
> NederHost
> KvK: 34099781
> 
> 

Re: Mass Stripping Attachments by Directory, Age, Size

[Steven Varco]

I would like such a feature too, but instead of deleting the atatchment files, 
I would like to „detach“ the files and save them into a sperate directory, 
which could be on a different storage like a share in the users home directory 
or even S3 and then replace the attachment in the Mail with a LINK to that file.
Thunderbird does this quite well with its „Detach Attachment“ feature; the MIME 
part looks like this after that:


Content-Type: image/png;
name="funny-picture.png"
Content-Disposition: attachment; filename="funny-picture.png"
X-Mozilla-External-Attachment-URL: 
file:/fileserver/home/svarco/mail/attachments/funny-picture.png
X-Mozilla-Altered: AttachmentDetached; date="Thu Mar 18 09:44:37 2021"

You deleted an attachment from this message. The original MIME headers for the 
attachment were:
Content-Transfer-Encoding: base64
Content-Disposition: inline;
filename=funny-picture.png
Content-Type: image/png;
name="funny-picture.png"


I know that for MS Exchange / Outlook some external archiving solutions as 
components do exist and looking for something similar to offload attachments 
with dovecot. :)

Steven

-- 
https://steven.varco.ch/ 

> Am 18.03.2021 um 08:31 schrieb Plutocrat :
> 
> Hi,
> 
> I've been looking around for a solution to this problem. I want to prune down 
> the attachments on a server before a migration. Some of the emails are 7 
> years old and have 40Mb attachments, so this seems like a good opportunity to 
> rationalize things. So perhaps I'd like to "Remove all attachments from 
> emails older than 2 years, in the .Sent directory", or "Attachments over 10Mb 
> anywhere in the mail tree"
> 
> I've found the strip_attachments.pl script here 
>  which 
> works fine on mbox (as tested on my local Thunderbird mboxes), but not on 
> maildir which is on the dovecot server. My Perl isn't strong enough to 
> re-purpose it.
> 
> I've looked at ripmime and mpack/munpack, and although they seem like useful 
> tools to do the job of deconstructing the mail into its constituent parts, it 
> doesn't seem to help in re-building the email. I think they could be used 
> with a bit of study into mail MIME structure, and used with a helper script.
> 
> So before I take a deep dive into scripting my own solution, I just wanted to 
> check if anyone else on the list has been through this and has some resources 
> or pointers they can share, or maybe even someone to tell me "Duh, you can do 
> it with doveadm of course".
> 
> P.

Mass Stripping Attachments by Directory, Age, Size

[Plutocrat]

Hi,

I've been looking around for a solution to this problem. I want to prune down the attachments on a 
server before a migration. Some of the emails are 7 years old and have 40Mb attachments, so this 
seems like a good opportunity to rationalize things. So perhaps I'd like to "Remove all 
attachments from emails older than 2 years, in the .Sent directory", or "Attachments over 
10Mb anywhere in the mail tree"

I've found the strip_attachments.pl script here 
 which works 
fine on mbox (as tested on my local Thunderbird mboxes), but not on maildir which is 
on the dovecot server. My Perl isn't strong enough to re-purpose it.

I've looked at ripmime and mpack/munpack, and although they seem like useful 
tools to do the job of deconstructing the mail into its constituent parts, it 
doesn't seem to help in re-building the email. I think they could be used with 
a bit of study into mail MIME structure, and used with a helper script.

So before I take a deep dive into scripting my own solution, I just wanted to check if 
anyone else on the list has been through this and has some resources or pointers they can 
share, or maybe even someone to tell me "Duh, you can do it with doveadm of 
course".

P.