Re: Fwd: Mail delivery failed: returning message to sender

[Russ Allbery]

Russ Allbery  writes:

> I will file a bug against reportbug on your behalf.

I think this bug may have been introduced in the Python 2 to Python 3
conversion, if I am understanding it correctly.  Bug report is here:

https://bugs.debian.org/849586

-- 
Russ Allbery (r...@debian.org)   

Re: Fwd: Mail delivery failed: returning message to sender

[Russ Allbery]

TOMAS MARTIŠIUS  writes:

> Thanks for solving this problem, bus I am not expert of all these
> encodings too, so I simply need working reportbug package and I really
> don't know in which point the problem is: in packege or in mailing
> system, which must accept bug report. But if both encodings can be used
> (as says some different RCS's) I prefer to fix mailing system, if it is
> possible, to accept both types of encodings. Especially because bug
> report can be send using some common mail system.

If reportbug generated:

From: =?utf-8?b?VG9tYXMgTWFydGnFoWl1cyA8dG9tYXNAcHVnYS52ZHUubHQ+?=

then this is definitely a bug in reportbug itself.  Looking at the
reportbug source, it is not giving email.message sufficient hints about
how to encode the From header correctly when the name includes non-ASCII
characters, which causes exactly this problem:

| lothlorien:~$ python3
| Python 3.5.2+ (default, Nov 22 2016, 01:00:20) 
| [GCC 6.2.1 20161119] on linux
| Type "help", "copyright", "credits" or "license" for more information.
| >>> from email.mime.text import MIMEText
| >>> message = MIMEText('Some test')
| >>> message['From'] = 'TOMAS MARTIŠIUS '
| >>> message['To'] = 'Russ Allbery '
| >>> print(message)
| Content-Type: text/plain; charset="us-ascii"
| MIME-Version: 1.0
| Content-Transfer-Encoding: 7bit
| From: =?utf-8?b?VE9NQVMgTUFSVEnFoElVUyA8dG9tYXNAcHVnYS52ZHUubHQ+?=
| To: Russ Allbery 
| 
| Some test

There are various ways to fix this.  The easiest is probably to use
email.utils.formataddr:

| >>> message = MIMEText('Some test')
| >>> message['From'] = formataddr(('TOMAS MARTIŠIUS', 'to...@puga.vdu.lt'))
| >>> message['To'] = formataddr(('Russ Allbery', 'ea...@eyrie.org'))
| >>> print(message)
| Content-Type: text/plain; charset="us-ascii"
| MIME-Version: 1.0
| Content-Transfer-Encoding: 7bit
| From: =?utf-8?q?TOMAS_MARTI=C5=A0IUS?= 
| To: Russ Allbery 
| 
| Some test

I will file a bug against reportbug on your behalf.

-- 
Russ Allbery (r...@debian.org)   

Re: Fwd: Mail delivery failed: returning message to sender

[Andrew Shadura]

Tomas,

On 28 December 2016 at 18:52, TOMAS MARTIŠIUS  wrote:
> Thanks for solving this problem, bus I am not  expert of all these encodings
> too,
> so I simply need working reportbug package and I really don't know in which
> point the problem is: in packege or in mailing system, which must accept bug
> report. But if both encodings can be used (as says some different RCS's) I
> prefer to fix mailing system, if it is possible, to accept both types of
> encodings. Especially because bug report can be send using some common mail
> system.

This isn't a common issue to deal with; I understand your reportbug is
misconfigured, hence there's no reason BTS should accept what it
produces.

However, reportbug could detect an incorrect configuration and produce
a warning.

It would be helpful if you posted some more information about your
system and in particular reportbug-related configuration here.

Thanks.

-- 
Cheers,
  Andrew

Re: Fwd: Mail delivery failed: returning message to sender

[TOMAS MARTIŠIUS]

Thanks for solving this problem, bus I am not  expert of all these 
encodings too,
so I simply need working reportbug package and I really don't know in 
which point the problem is: in packege or in mailing system, which must 
accept bug report. But if both encodings can be used (as says some 
different RCS's) I prefer to fix mailing system, if it is possible, to 
accept both types of encodings. Especially because bug report can be 
send using some common mail system.


Best regards,

Pagarbiai,

Tomas Martišius
Vytauto Didžiojo universitetas
vyr. tinklo administratorius

2016.12.28 18:53, Andreas Metzler rašė:

Niels Thykier  wrote:

Andreas Metzler:

[...]

No, the encoding was not correct. Compare how you (your MUA) just did it in
this message with the rejected one.
   From: =?UTF-8?Q?TOMAS_MARTI=c5=a0IUS?= 
   From: =?utf-8?b?VG9tYXMgTWFydGnFoWl1cyA8dG9tYXNAcHVnYS52ZHUubHQ+?=

[...]


Hi,
The latter is "simply" a base64 encoding:
$ echo 'VG9tYXMgTWFydGnFoWl1cyA8dG9tYXNAcHVnYS52ZHUubHQ+' | base64 -d
Tomas Martišius 

[...]

I am not an expert on permitted ways of quoting UTF-8 in mail
headers, but the base64 method does not seem entirely inconceivable
for some RFC to support that (especially not considering you can
encode the body with base64).

Whether VG9tYXMgTWFydGnFoWl1cyA8dG9tYXNAcHVnYS52ZHUubHQ+ can be
/decoded/ to something that contains a mail-address is not relevant. The
header

   From: =?utf-8?b?VG9tYXMgTWFydGnFoWl1cyA8dG9tYXNAcHVnYS52ZHUubHQ+?=

does not fullfill the syntax required by RFC 2822. These syntax
requiremts apply to the (possibly rfc2047-) *encoded* field contents,
not to the decoded values.

cu Andreas

Re: Fwd: Mail delivery failed: returning message to sender

[Andreas Metzler]

Niels Thykier  wrote:
> Andreas Metzler:
[...]
>> No, the encoding was not correct. Compare how you (your MUA) just did it in
>> this message with the rejected one.

>>   From: =?UTF-8?Q?TOMAS_MARTI=c5=a0IUS?= 

>>   From: =?utf-8?b?VG9tYXMgTWFydGnFoWl1cyA8dG9tYXNAcHVnYS52ZHUubHQ+?=
[...]

> Hi,

> The latter is "simply" a base64 encoding:

> $ echo 'VG9tYXMgTWFydGnFoWl1cyA8dG9tYXNAcHVnYS52ZHUubHQ+' | base64 -d
> Tomas Martišius 
[...]
> I am not an expert on permitted ways of quoting UTF-8 in mail
> headers, but the base64 method does not seem entirely inconceivable
> for some RFC to support that (especially not considering you can
> encode the body with base64).

Whether VG9tYXMgTWFydGnFoWl1cyA8dG9tYXNAcHVnYS52ZHUubHQ+ can be
/decoded/ to something that contains a mail-address is not relevant. The
header 

  From: =?utf-8?b?VG9tYXMgTWFydGnFoWl1cyA8dG9tYXNAcHVnYS52ZHUubHQ+?=

does not fullfill the syntax required by RFC 2822. These syntax
requiremts apply to the (possibly rfc2047-) *encoded* field contents,
not to the decoded values.

cu Andreas
-- 
`What a good friend you are to him, Dr. Maturin. His other friends are
so grateful to you.'
`I sew his ears on from time to time, sure'

Re: Fwd: Mail delivery failed: returning message to sender

[Andrew Shadura]

On 28 December 2016 at 17:14, Niels Thykier  wrote:
> Niels Thykier:
>> Andreas Metzler:
>>> [...]
>>>
>>> Hello,
>>>
>>> No, the encoding was not correct. Compare how you (your MUA) just did it in
>>> this message with the rejected one.
>>>
>>>   From: =?UTF-8?Q?TOMAS_MARTI=c5=a0IUS?= 
>>>
>>>   From: =?utf-8?b?VG9tYXMgTWFydGnFoWl1cyA8dG9tYXNAcHVnYS52ZHUubHQ+?=
>>>
>>
>>
>>[...]
>
> >From a quick glance, it looks to be specified in RFC1342, which also
> specified the ?Q? variant that was used in the accepted mail.
>
>
> RFC1342:
> """
> [...]
>
>An "encoded-word" is more precisely defined by the following EBNF
>grammar, using the notation of RFC 822:
>
>encoded-word = "=" "?" charset "?" encoding "?" encoded-text "?" "="
>
>charset = token; legal charsets defined by RFC 1341
>
>encoding = token   ; Either "B" or "Q"
>
>token = 1*
>
>tspecials = "(" / ")" / "<" / ">" / "@" / "," / ";" / ":" / "\" /
><"> / "/" / "[" / "]" / "?" / "." / "="
>
>encoded-text = 1*   ; SPACE> (but see "Use of encoded-words in message
>   ; headers", below)
> [...]
>
>
> The "B" encoding
>
>The "B" encoding is identical to the "BASE64" encoding defined by RFC
>1341.
> """
>
> This is not a comment on whether RFC1342 encoded "From:" headers is
> something we is supporting/should support.

Clint is being slightly unhelpful. BTS does accept base64, see #849535 with:

From: =?utf-8?b?w4HFiMSPxZnDqMW1IMWgxKfDpMSPxa3FlcOl?= 
Subject: =?utf-8?b?dG5hdDY0OiDFpWXFocWlIGLDvMSdIMWZw6nhuZXDs8WVxaU=?=

What Clint is saying, and what Andreas has pointed out before, is that
display name and address have to be separately encoded, i.e. they
can't be a single encoded string, otherwise they decode into "Name
" not "Name" 

-- 
Cheers,
  Andrew

Re: Fwd: Mail delivery failed: returning message to sender

[Alec Leamas]

On 28/12/16 17:02, Niels Thykier wrote:


Note the difference between "=?UTF-8?Q?" vs. "=?UTF-8?b?" (Q vs. b).  I
presume the "b" is for "binary" or/and "Base64 encoded" vs. Q which
would be "quoted printed" or something like that.

I am not an expert on permitted ways of quoting UTF-8 in mail headers,
but the base64 method does not seem entirely inconceivable for some RFC
to support that (especially not considering you can encode the body with
base64).


Nor am I, the experts wrote the MIME RFC [1]. Which basically says that 
both encodings are allowed, but the Q (Quoted printable) is actually 
recommended in a case like this.


Cheers!

--alec


[1] https://www.ietf.org/rfc/rfc2047.txt, section 4,

Re: Fwd: Mail delivery failed: returning message to sender

[Niels Thykier]

Niels Thykier:
> Andreas Metzler:
>> [...]
>>
>> Hello,
>>
>> No, the encoding was not correct. Compare how you (your MUA) just did it in
>> this message with the rejected one.
>>
>>   From: =?UTF-8?Q?TOMAS_MARTI=c5=a0IUS?= 
>>
>>   From: =?utf-8?b?VG9tYXMgTWFydGnFoWl1cyA8dG9tYXNAcHVnYS52ZHUubHQ+?=
>>
> 
> 
>[...]

>From a quick glance, it looks to be specified in RFC1342, which also
specified the ?Q? variant that was used in the accepted mail.


RFC1342:
"""
[...]

   An "encoded-word" is more precisely defined by the following EBNF
   grammar, using the notation of RFC 822:

   encoded-word = "=" "?" charset "?" encoding "?" encoded-text "?" "="

   charset = token; legal charsets defined by RFC 1341

   encoding = token   ; Either "B" or "Q"

   token = 1*

   tspecials = "(" / ")" / "<" / ">" / "@" / "," / ";" / ":" / "\" /
   <"> / "/" / "[" / "]" / "?" / "." / "="

   encoded-text = 1* (but see "Use of encoded-words in message
  ; headers", below)
[...]


The "B" encoding

   The "B" encoding is identical to the "BASE64" encoding defined by RFC
   1341.
"""

This is not a comment on whether RFC1342 encoded "From:" headers is
something we is supporting/should support.

Thanks,
~Niels

Re: Fwd: Mail delivery failed: returning message to sender

[Clint Adams]

On Wed, Dec 28, 2016 at 04:02:00PM +, Niels Thykier wrote:
> I am not an expert on permitted ways of quoting UTF-8 in mail headers,
> but the base64 method does not seem entirely inconceivable for some RFC
> to support that (especially not considering you can encode the body with
> base64).

You might want to look at RFC 2047.

Re: Fwd: Mail delivery failed: returning message to sender

[Niels Thykier]

Andreas Metzler:
> [...]
> 
> Hello,
> 
> No, the encoding was not correct. Compare how you (your MUA) just did it in
> this message with the rejected one.
> 
>   From: =?UTF-8?Q?TOMAS_MARTI=c5=a0IUS?= 
> 
>   From: =?utf-8?b?VG9tYXMgTWFydGnFoWl1cyA8dG9tYXNAcHVnYS52ZHUubHQ+?=
> 


Hi,

The latter is "simply" a base64 encoding:

$ echo 'VG9tYXMgTWFydGnFoWl1cyA8dG9tYXNAcHVnYS52ZHUubHQ+' | base64 -d
Tomas Martišius 

Note the difference between "=?UTF-8?Q?" vs. "=?UTF-8?b?" (Q vs. b).  I
presume the "b" is for "binary" or/and "Base64 encoded" vs. Q which
would be "quoted printed" or something like that.

I am not an expert on permitted ways of quoting UTF-8 in mail headers,
but the base64 method does not seem entirely inconceivable for some RFC
to support that (especially not considering you can encode the body with
base64).

Thanks,
~Niels

Re: Fwd: Mail delivery failed: returning message to sender

[Andreas Metzler]

On 2016-12-28 TOMAS MARTIŠIUS  wrote:
> 2016.12.28 09:09, Andreas Metzler rašė:
>> On 2016-12-27 TOMAS MARTIŠIUS  wrote:
>>> Why I can't report bug using reportbug command? After reporting I get back
>>> e-mail with this message:
[...]
>> The From header does not look too good:

>>> From: =?utf-8?b?VG9tYXMgTWFydGnFoWl1cyA8dG9tYXNAcHVnYS52ZHUubHQ+?=

> But I am not only english speaking man, so there are UTF-8 chars in my
> last name. Thats why subject is encoded. And it is encoded in standard
> way, using reportbug command.

> It seams that if someone using UTF-8 in his name and wants report bug - it
> has a problems.

> I think this is totally bad for debian community - we are losing important
> bug reports.
[...]

Hello,

No, the encoding was not correct. Compare how you (your MUA) just did it in
this message with the rejected one.

  From: =?UTF-8?Q?TOMAS_MARTI=c5=a0IUS?= 

This matches

from=   "From:" mailbox-list CRLF
mailbox-list=   (mailbox *("," mailbox)) / obs-mbox-list
mailbox =   name-addr / addr-spec
name-addr   =   [display-name] angle-addr
angle-addr  =   [CFWS] "<" addr-spec ">" [CFWS] / obs-angle-addr
display-name=   phrase
obs-angle-addr  =   [CFWS] "<" [obs-route] addr-spec ">" [CFWS]
addr-spec   =   local-part "@" domain

in RFC 2822. There is an addr-spec (to...@puga.vdu.lt) and a
display-name ("=?UTF-8?Q?TOMAS_MARTI=c5=a0IUS?=). The rejected message
OTOH has no valid e-mail adress.
 From: =?utf-8?b?VG9tYXMgTWFydGnFoWl1cyA8dG9tYXNAcHVnYS52ZHUubHQ+?=

FWIW I just used reportbug to generate a test-bugreport (with 8-bit
characters in the submitter name), and reportbug generated a correctly
encoded.

Please submit a bug-report against reportbug if there is an easy way to
reproduce the issue.

cu Andreas
-- 
`What a good friend you are to him, Dr. Maturin. His other friends are
so grateful to you.'
`I sew his ears on from time to time, sure'

Re: Fwd: Mail delivery failed: returning message to sender

[Andrew Shadura]

On 28 December 2016 at 10:08, TOMAS MARTIŠIUS  wrote:
> Hello,
>
> But I am not only english speaking man, so there are UTF-8 chars in my last
> name. Thats why subject
> is encoded. And it is encoded in standard way, using reportbug command.
>
> It seams that if someone using UTF-8 in his name and wants report bug - it
> has a problems.
>
> I think this is totally bad for debian community - we are losing important
> bug reports.

I think that's not the reason, I succeeded in creating
http://bugs.debian.org/849535

-- 
Cheers,
  Andrew

Re: Fwd: Mail delivery failed: returning message to sender

[TOMAS MARTIŠIUS]

Hello,

But I am not only english speaking man, so there are UTF-8 chars in my 
last name. Thats why subject

is encoded. And it is encoded in standard way, using reportbug command.

It seams that if someone using UTF-8 in his name and wants report bug - 
it has a problems.


I think this is totally bad for debian community - we are losing 
important bug reports.


Best regards,

Pagarbiai

Tomas Martišius
Vytauto Didžiojo universitetas
vyr. tinklo administratorius

2016.12.28 09:09, Andreas Metzler rašė:

On 2016-12-27 TOMAS MARTIŠIUS  wrote:

Hello,
Why I can't report bug using reportbug command? After reporting I get back
e-mail with this message:

[...]

Remote-MTA: dns; buxtehude.debian.org
Diagnostic-Code: smtp; 550 Invalid header syntax:

The From header does not look too good:


From: =?utf-8?b?VG9tYXMgTWFydGnFoWl1cyA8dG9tYXNAcHVnYS52ZHUubHQ+?=

cu Andreas

Re: Fwd: Mail delivery failed: returning message to sender

[Andreas Metzler]

On 2016-12-27 TOMAS MARTIŠIUS  wrote:
> Hello,

> Why I can't report bug using reportbug command? After reporting I get back
> e-mail with this message:

[...]
> Remote-MTA: dns; buxtehude.debian.org
> Diagnostic-Code: smtp; 550 Invalid header syntax:

The From header does not look too good:

> From: =?utf-8?b?VG9tYXMgTWFydGnFoWl1cyA8dG9tYXNAcHVnYS52ZHUubHQ+?=

cu Andreas
-- 
`What a good friend you are to him, Dr. Maturin. His other friends are
so grateful to you.'
`I sew his ears on from time to time, sure'