Bug#833190: Please provide an option to not add X-getmail-retrieved-from-mailbox header

[Daniel Kahn Gillmor]

On Mon 2017-12-18 16:00:34 -0600, Charles Cazabon wrote:
> Daniel Kahn Gillmor  wrote:
>>  b) sometimes, i find myself forwarding messages to other people.
>> Depending on the account i'm mailing from, I generally prefer to
>> minimize the amount of automatically-published information about my
>> system setup.
>
> Ok.  I'm a bit of a privacy nut myself, so this seems like a legitimate
> problem that you need a solution for (well, that you created a solution
> for...).  Thanks very much for the patch; that definitely reduced the time I
> needed to find to do this.
>
> I've released getmail v.5.5 with this addion, except I shortened the parameter
> name to record_mailbox.

great, thank you very much!

> One thing from your report, however:
>
>> entirely instead, except for the effect it has on %(mailbox) in the
>> external MDA case.  If you have a solution for that problem, then a fix
>> to just remove the X-getmail-retrieved-from-mailbox header entirely
>> would probably be simpler and involve less maintenance and support going
>> forward.
>
> The added header and the %(mailbox) substitution in external command arguments
> are unrelated; other than using the same source for their data, there's no
> connection between them.  Turning the header off does not break %(mailbox)
> substitution in MDA_external aguments in my testing.
>
> If you can show me how to reproduce this problem, I'll take a look at it.

hm, i found this a while ago, and i don't think i have a reproducer for
it.  I'm not sure what i was thinking.  I'm happy to hear that it
doesn't seem to be a problem!  If i find anything similar, i'll be sure
to mention it on-list here.

Regards,

   --dkg

PS is the getmail development published under any form of revision
   control?  I can work with downloaded tarballs, but if you're using
   git or svn or hg, i might be better able to send you patches that
   align with your current development in the odd case where i come up
   with any clever ideas :)


signature.asc
Description: PGP signature

Bug#833190: Please provide an option to not add X-getmail-retrieved-from-mailbox header

[Charles Cazabon]

Daniel Kahn Gillmor  wrote:
>  b) sometimes, i find myself forwarding messages to other people.
> Depending on the account i'm mailing from, I generally prefer to
> minimize the amount of automatically-published information about my
> system setup.

Ok.  I'm a bit of a privacy nut myself, so this seems like a legitimate
problem that you need a solution for (well, that you created a solution
for...).  Thanks very much for the patch; that definitely reduced the time I
needed to find to do this.

I've released getmail v.5.5 with this addion, except I shortened the parameter
name to record_mailbox.  One thing from your report, however:

> entirely instead, except for the effect it has on %(mailbox) in the
> external MDA case.  If you have a solution for that problem, then a fix
> to just remove the X-getmail-retrieved-from-mailbox header entirely
> would probably be simpler and involve less maintenance and support going
> forward.

The added header and the %(mailbox) substitution in external command arguments
are unrelated; other than using the same source for their data, there's no
connection between them.  Turning the header off does not break %(mailbox)
substitution in MDA_external aguments in my testing.

If you can show me how to reproduce this problem, I'll take a look at it.

Charles
-- 
---
Charles Cazabon
GPL'ed software available at:   http://pyropus.ca/software/
---

Bug#833190: Please provide an option to not add X-getmail-retrieved-from-mailbox header

[Daniel Kahn Gillmor]

Hi Charles--

On Mon 2017-12-18 10:23:03 -0600, Charles Cazabon wrote:
> i.e. what is the reason/justification for the feature?  What problems is the
> presence of this header field causing?

I have two reasons i care about it:

 a) i prefer to have the local copy of my message have the exact same
bytestream as the copy on the server.  Depending on the filesystems
on the server, this allows me to make comparisons with tools like
sha256sum or rsync when evaluating who has what file.  The
X-getmail-retrieved-from-mailbox changes the bytestream, which makes
these techniques fail.

 b) sometimes, i find myself forwarding messages to other people.
Depending on the account i'm mailing from, I generally prefer to
minimize the amount of automatically-published information about my
system setup.  In particular, i think we should generally discourage
mail user agents from publishing User-Agent: or equivalent headers,
and we should discourage tools like GnuPG from injecting presence or
version information in message parts that they're involved in
generating.

if X-getmail-retrieved-from-mailbox is present in the forwarded
message, it leaks some information about the toolchain in use that
doesn't need to be leaked.

> I ask because every added feature has added costs for me in terms of
> maintenance and support, and for users, in terms of initial setup and
> configuration.

I sympathize with this concern.  sorry the original report wasn't
clearer about it.

I would have submitted a simplifying patch to just remove the header
entirely instead, except for the effect it has on %(mailbox) in the
external MDA case.  If you have a solution for that problem, then a fix
to just remove the X-getmail-retrieved-from-mailbox header entirely
would probably be simpler and involve less maintenance and support going
forward.

let me know if that makes sense, or if you have any other questions.

 --dkg


signature.asc
Description: PGP signature

Bug#833190: Please provide an option to not add X-getmail-retrieved-from-mailbox header

[Charles Cazabon]

Daniel Kahn Gillmor  wrote:
> On Sun 2017-12-17 22:52:29 +0900, Osamu Aoki wrote:
> > Let me forward this Debian bug report with patch to the ML since I am not
> > sure if my previous mail in Feb 2017 worked or not.
> > https://bugs.debian.org/833190
> 
> I'm the author of the patch in question, and i've been using it regularly
> for over a year.  It would be great to get it upstreamed, and i'd be happy
> to modify it in any way that makes it more likely to be accepted.

I've read the Debian bug, but it's not clear to me what problem this patch
is trying to solve?

i.e. what is the reason/justification for the feature?  What problems is the
presence of this header field causing?

I ask because every added feature has added costs for me in terms of
maintenance and support, and for users, in terms of initial setup and
configuration.  Without knowing what problem this is trying to solve, it's
difficult to estimate the cost/benefit ratio of adding such a feature.

Charles
-- 
---
Charles Cazabon
GPL'ed software available at:   http://pyropus.ca/software/
---

Bug#833190: Please provide an option to not add X-getmail-retrieved-from-mailbox header

[Daniel Kahn Gillmor]

Hi all--

On Sun 2017-12-17 22:52:29 +0900, Osamu Aoki wrote:
> Let me forward this Debian bug report with patch to the ML since I am
> not sure if my previous mail in Feb 2017 worked or not.
>
> I just resubscribed to the ML.  (Looks like I was on on the ML.  Maybe
> too many bounces ...)
>
> Let me know if this is good way or not.  I see these bug reporters are
> good ones ;-)  You can see the full story at:
>   https://bugs.debian.org/833190
>
> Osamu (Just uploaded 5.4 version to Debian)

I'm the author of the patch in question, and i've been using it
regularly for over a year.  It would be great to get it upstreamed, and
i'd be happy to modify it in any way that makes it more likely to be
accepted.

I'm on the mailing list.  Please let me know if you have any feedback or
suggestions about it!

Regards,

--dkg


signature.asc
Description: PGP signature

Bug#833190: Please provide an option to not add X-getmail-retrieved-from-mailbox header

[Osamu Aoki]

Hi,

Let me forward this Debian bug report with patch to the ML since I am
not sure if my previous mail in Feb 2017 worked or not.

I just resubscribed to the ML.  (Looks like I was on on the ML.  Maybe
too many bounces ...)

Let me know if this is good way or not.  I see these bug reporters are
good ones ;-)  You can see the full story at:
  https://bugs.debian.org/833190

Osamu (Just uploaded 5.4 version to Debian)

PS: Excuse us for noise from https://bugs.debian.org/871228

- Forwarded message from Daniel Kahn Gillmor <d...@debian.org> -

Date: Thu, 23 Feb 2017 14:07:09 -0500
From: Daniel Kahn Gillmor <d...@debian.org>
To: Josh Triplett <j...@joshtriplett.org>, 833...@bugs.debian.org
Subject: Bug#833190: Please provide an option to not add
X-getmail-retrieved-from-mailbox header

Control: tags 833190 + patch

On Mon 2016-08-01 19:25:38 -0400, Josh Triplett wrote:

> When getmail retrieves a mail, it unconditionally adds a header
> X-getmail-retrieved-from-mailbox with the name of the mailbox it retrieved the
> mail from.  As far as I can tell, getmail doesn't have any option to avoid
> adding that header.  I'd like to leave the mail completely unmodified.  Would
> you consider adding an option to not add that header?

I went to report this issue on the BTS, but saw that Josh has beaten me
to it, as usual. :)

Attached is a patch that seems to work for me.

To use it, i added:

 record_retrieved_from_mailbox = False

to the [Retriever] section of ~/.getmail/config/mymailprovider

Feel free to forward it upstream :)

hth,

 --dkg


From: Daniel Kahn Gillmor <d...@fifthhorseman.net>
Date: Thu, 23 Feb 2017 13:56:54 -0500
Subject: Make X-getmail-retrieved-from-mailbox optional (Closes: #833190)

In https://bugs.debian/833190, Josh Triplett asked for getmail to
avoid modifying messages as they are fetched via IMAP.

This should provide such an option.
---
 docs/configuration.html| 7 +++
 docs/configuration.txt | 5 +
 getmailcore/_retrieverbases.py | 7 ---
 getmailcore/retrievers.py  | 4 
 4 files changed, 20 insertions(+), 3 deletions(-)

diff --git a/docs/configuration.html b/docs/configuration.html
index 0508ddf..c79610a 100644
--- a/docs/configuration.html
+++ b/docs/configuration.html
@@ -635,6 +635,13 @@ mailboxes = ALL
 http://honk.sigxcpu.org/projects/pykerberos/;>http://honk.sigxcpu.org/projects/pykerberos/;
 for details.
 
+
+record_retrieved_from_mailbox
+(boolean)
+ whether to add a X-getmail-retrieved-from-mailbox 
header
+to each message fetched; the default is True.  Note that setting this 
to False
+will break the use of %(mailbox) as a substitution in MDA_external 
arguments.
+
 
 
 SSL Client Parameters
diff --git a/docs/configuration.txt b/docs/configuration.txt
index a3cbe0d..ed4fdc0 100644
--- a/docs/configuration.txt
+++ b/docs/configuration.txt
@@ -401,6 +401,11 @@ mailboxes = ALL
pykerberos with GSS support is installed; check your OS
distribution or see http://honk.sigxcpu.org/projects/pykerberos/;
for details.
+ * record_retrieved_from_mailbox (boolean) -- whether to add a
+   X-getmail-retrieved-from-mailbox header to each message
+   fetched; the default is True.  Note that setting this to False
+   will break the use of %(mailbox) as a substitution in
+   MDA_external arguments.
 
 SSL Client Parameters
 
diff --git a/getmailcore/_retrieverbases.py b/getmailcore/_retrieverbases.py
index e4592e0..31e7170 100755
--- a/getmailcore/_retrieverbases.py
+++ b/getmailcore/_retrieverbases.py
@@ -1556,9 +1556,10 @@ class IMAPRetrieverBase(RetrieverSkeleton):
 raise getmailRetrievalError('failed to retrieve msgid %s' 
 % msgid)
 
-# record mailbox retrieved from in a header
-msg.add_header('X-getmail-retrieved-from-mailbox', 
-   self.mailbox_selected)
+if self.conf['record_retrieved_from_mailbox']:
+# record mailbox retrieved from in a header
+msg.add_header('X-getmail-retrieved-from-mailbox', 
+   self.mailbox_selected)
 
 # google extensions: apply labels, etc
 if 'X-GM-EXT-1' in self.conn.capabilities:
diff --git a/getmailcore/retrievers.py b/getmailcore/retrievers.py
index 6e82df7..a2b36bb 100755
--- a/getmailcore/retrievers.py
+++ b/getmailcore/retrievers.py
@@ -376,6 +376,7 @@ class SimpleIMAPRetriever(IMAPRetrieverBase, IMAPinitMixIn):
 # .authenticate(), so we can't do this yet (?).
 ConfBool(name='use_cram_md5', required=False, default=False),
 ConfBool(name='use_kerberos', required=False, default=False),
+ConfBool(name='record_retrieved_from_mailbox', required=False, 
default=True),
 )
 received_from = None
 receive

Bug#833190: [d...@debian.org: Bug#833190: Please provide an option to not add X-getmail-retrieved-from-mailbox header]

[Osamu Aoki]

Hi,

This is an interesting FEATURE request/proposal with patch.

I think this should be accepted by the upstream first.

Hi upstream,

This is a forwarded bug report from Debian.  I got kicked out of mailimg
list sometime ago.  I hope this reaches you. (I just resubscribed)


- Forwarded message from Daniel Kahn Gillmor <d...@debian.org> -

Date: Thu, 23 Feb 2017 14:07:09 -0500
From: Daniel Kahn Gillmor <d...@debian.org>
To: Josh Triplett <j...@joshtriplett.org>, 833...@bugs.debian.org
Subject: Bug#833190: Please provide an option to not add
X-getmail-retrieved-from-mailbox header

Control: tags 833190 + patch

On Mon 2016-08-01 19:25:38 -0400, Josh Triplett wrote:

> When getmail retrieves a mail, it unconditionally adds a header
> X-getmail-retrieved-from-mailbox with the name of the mailbox it retrieved the
> mail from.  As far as I can tell, getmail doesn't have any option to avoid
> adding that header.  I'd like to leave the mail completely unmodified.  Would
> you consider adding an option to not add that header?

I went to report this issue on the BTS, but saw that Josh has beaten me
to it, as usual. :)

Attached is a patch that seems to work for me.

To use it, i added:

 record_retrieved_from_mailbox = False

to the [Retriever] section of ~/.getmail/config/mymailprovider

Feel free to forward it upstream :)

hth,

 --dkg


From: Daniel Kahn Gillmor <d...@fifthhorseman.net>
Date: Thu, 23 Feb 2017 13:56:54 -0500
Subject: Make X-getmail-retrieved-from-mailbox optional (Closes: #833190)

In https://bugs.debian/833190, Josh Triplett asked for getmail to
avoid modifying messages as they are fetched via IMAP.

This should provide such an option.
---
 docs/configuration.html| 7 +++
 docs/configuration.txt | 5 +
 getmailcore/_retrieverbases.py | 7 ---
 getmailcore/retrievers.py  | 4 
 4 files changed, 20 insertions(+), 3 deletions(-)

diff --git a/docs/configuration.html b/docs/configuration.html
index 0508ddf..c79610a 100644
--- a/docs/configuration.html
+++ b/docs/configuration.html
@@ -635,6 +635,13 @@ mailboxes = ALL
 http://honk.sigxcpu.org/projects/pykerberos/;>http://honk.sigxcpu.org/projects/pykerberos/;
 for details.
 
+
+record_retrieved_from_mailbox
+(boolean)
+ whether to add a X-getmail-retrieved-from-mailbox 
header
+to each message fetched; the default is True.  Note that setting this 
to False
+will break the use of %(mailbox) as a substitution in MDA_external 
arguments.
+
 
 
 SSL Client Parameters
diff --git a/docs/configuration.txt b/docs/configuration.txt
index a3cbe0d..ed4fdc0 100644
--- a/docs/configuration.txt
+++ b/docs/configuration.txt
@@ -401,6 +401,11 @@ mailboxes = ALL
pykerberos with GSS support is installed; check your OS
distribution or see http://honk.sigxcpu.org/projects/pykerberos/;
for details.
+ * record_retrieved_from_mailbox (boolean) -- whether to add a
+   X-getmail-retrieved-from-mailbox header to each message
+   fetched; the default is True.  Note that setting this to False
+   will break the use of %(mailbox) as a substitution in
+   MDA_external arguments.
 
 SSL Client Parameters
 
diff --git a/getmailcore/_retrieverbases.py b/getmailcore/_retrieverbases.py
index e4592e0..31e7170 100755
--- a/getmailcore/_retrieverbases.py
+++ b/getmailcore/_retrieverbases.py
@@ -1556,9 +1556,10 @@ class IMAPRetrieverBase(RetrieverSkeleton):
 raise getmailRetrievalError('failed to retrieve msgid %s' 
 % msgid)
 
-# record mailbox retrieved from in a header
-msg.add_header('X-getmail-retrieved-from-mailbox', 
-   self.mailbox_selected)
+if self.conf['record_retrieved_from_mailbox']:
+# record mailbox retrieved from in a header
+msg.add_header('X-getmail-retrieved-from-mailbox', 
+   self.mailbox_selected)
 
 # google extensions: apply labels, etc
 if 'X-GM-EXT-1' in self.conn.capabilities:
diff --git a/getmailcore/retrievers.py b/getmailcore/retrievers.py
index 6e82df7..a2b36bb 100755
--- a/getmailcore/retrievers.py
+++ b/getmailcore/retrievers.py
@@ -376,6 +376,7 @@ class SimpleIMAPRetriever(IMAPRetrieverBase, IMAPinitMixIn):
 # .authenticate(), so we can't do this yet (?).
 ConfBool(name='use_cram_md5', required=False, default=False),
 ConfBool(name='use_kerberos', required=False, default=False),
+ConfBool(name='record_retrieved_from_mailbox', required=False, 
default=True),
 )
 received_from = None
 received_with = 'IMAP4'
@@ -423,6 +424,7 @@ class SimpleIMAPSSLRetriever(IMAPRetrieverBase, 
IMAPSSLinitMixIn):
 # .authenticate(), so we can't do this yet (?).
 ConfBool(name='use_cram_md5', required=Fal

Bug#833190: Please provide an option to not add X-getmail-retrieved-from-mailbox header

[Daniel Kahn Gillmor]

Control: tags 833190 + patch

On Mon 2016-08-01 19:25:38 -0400, Josh Triplett wrote:

> When getmail retrieves a mail, it unconditionally adds a header
> X-getmail-retrieved-from-mailbox with the name of the mailbox it retrieved the
> mail from.  As far as I can tell, getmail doesn't have any option to avoid
> adding that header.  I'd like to leave the mail completely unmodified.  Would
> you consider adding an option to not add that header?

I went to report this issue on the BTS, but saw that Josh has beaten me
to it, as usual. :)

Attached is a patch that seems to work for me.

To use it, i added:

 record_retrieved_from_mailbox = False

to the [Retriever] section of ~/.getmail/config/mymailprovider

Feel free to forward it upstream :)

hth,

 --dkg

From: Daniel Kahn Gillmor 
Date: Thu, 23 Feb 2017 13:56:54 -0500
Subject: Make X-getmail-retrieved-from-mailbox optional (Closes: #833190)

In https://bugs.debian/833190, Josh Triplett asked for getmail to
avoid modifying messages as they are fetched via IMAP.

This should provide such an option.
---
 docs/configuration.html| 7 +++
 docs/configuration.txt | 5 +
 getmailcore/_retrieverbases.py | 7 ---
 getmailcore/retrievers.py  | 4 
 4 files changed, 20 insertions(+), 3 deletions(-)

diff --git a/docs/configuration.html b/docs/configuration.html
index 0508ddf..c79610a 100644
--- a/docs/configuration.html
+++ b/docs/configuration.html
@@ -635,6 +635,13 @@ mailboxes = ALL
 http://honk.sigxcpu.org/projects/pykerberos/;>http://honk.sigxcpu.org/projects/pykerberos/;
 for details.
 
+
+record_retrieved_from_mailbox
+(boolean)
+ whether to add a X-getmail-retrieved-from-mailbox header
+to each message fetched; the default is True.  Note that setting this to False
+will break the use of %(mailbox) as a substitution in MDA_external arguments.
+
 
 
 SSL Client Parameters
diff --git a/docs/configuration.txt b/docs/configuration.txt
index a3cbe0d..ed4fdc0 100644
--- a/docs/configuration.txt
+++ b/docs/configuration.txt
@@ -401,6 +401,11 @@ mailboxes = ALL
pykerberos with GSS support is installed; check your OS
distribution or see http://honk.sigxcpu.org/projects/pykerberos/;
for details.
+ * record_retrieved_from_mailbox (boolean) -- whether to add a
+   X-getmail-retrieved-from-mailbox header to each message
+   fetched; the default is True.  Note that setting this to False
+   will break the use of %(mailbox) as a substitution in
+   MDA_external arguments.
 
 SSL Client Parameters
 
diff --git a/getmailcore/_retrieverbases.py b/getmailcore/_retrieverbases.py
index e4592e0..31e7170 100755
--- a/getmailcore/_retrieverbases.py
+++ b/getmailcore/_retrieverbases.py
@@ -1556,9 +1556,10 @@ class IMAPRetrieverBase(RetrieverSkeleton):
 raise getmailRetrievalError('failed to retrieve msgid %s' 
 % msgid)
 
-# record mailbox retrieved from in a header
-msg.add_header('X-getmail-retrieved-from-mailbox', 
-   self.mailbox_selected)
+if self.conf['record_retrieved_from_mailbox']:
+# record mailbox retrieved from in a header
+msg.add_header('X-getmail-retrieved-from-mailbox', 
+   self.mailbox_selected)
 
 # google extensions: apply labels, etc
 if 'X-GM-EXT-1' in self.conn.capabilities:
diff --git a/getmailcore/retrievers.py b/getmailcore/retrievers.py
index 6e82df7..a2b36bb 100755
--- a/getmailcore/retrievers.py
+++ b/getmailcore/retrievers.py
@@ -376,6 +376,7 @@ class SimpleIMAPRetriever(IMAPRetrieverBase, IMAPinitMixIn):
 # .authenticate(), so we can't do this yet (?).
 ConfBool(name='use_cram_md5', required=False, default=False),
 ConfBool(name='use_kerberos', required=False, default=False),
+ConfBool(name='record_retrieved_from_mailbox', required=False, default=True),
 )
 received_from = None
 received_with = 'IMAP4'
@@ -423,6 +424,7 @@ class SimpleIMAPSSLRetriever(IMAPRetrieverBase, IMAPSSLinitMixIn):
 # .authenticate(), so we can't do this yet (?).
 ConfBool(name='use_cram_md5', required=False, default=False),
 ConfBool(name='use_kerberos', required=False, default=False),
+ConfBool(name='record_retrieved_from_mailbox', required=False, default=True),
 )
 received_from = None
 received_with = 'IMAP4-SSL'
@@ -463,6 +465,7 @@ class MultidropIMAPRetriever(MultidropIMAPRetrieverBase, IMAPinitMixIn):
 ConfBool(name='use_cram_md5', required=False, default=False),
 ConfBool(name='use_kerberos', required=False, default=False),
 ConfString(name='envelope_recipient'),
+ConfBool(name='record_retrieved_from_mailbox', required=False, default=True),
 )
 received_from = None
 

Bug#833190: Please provide an option to not add X-getmail-retrieved-from-mailbox header

[Josh Triplett]

Package: getmail4
Severity: wishlist

When getmail retrieves a mail, it unconditionally adds a header
X-getmail-retrieved-from-mailbox with the name of the mailbox it retrieved the
mail from.  As far as I can tell, getmail doesn't have any option to avoid
adding that header.  I'd like to leave the mail completely unmodified.  Would
you consider adding an option to not add that header?

-- System Information:
Debian Release: stretch/sid
  APT prefers unstable
  APT policy: (500, 'unstable'), (1, 'experimental')
Architecture: amd64 (x86_64)

Kernel: Linux 4.7.0-rc7-amd64 (SMP w/4 CPU cores)
Locale: LANG=C.UTF-8, LC_CTYPE=C.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)