Re: [Mailman-Users] Base64 encoding - HTML Entity Code 'encodings'

[Mark Sapiro]

Drew Tenenholz wrote:
>
>I'm running a list where posts are made in Russian (Cyrillic) and 
>have questions about encodings as well.
>Right now, posts are sent in Cyrillic, and I receive in Cyrillic 
>(although I haven't chased down exactly which encoding was 
>sent/received).  What puzzles me is the way the archives are 
>recorded/encoded.
>
>When I look at the Mailman web archives and 'View Source', the page 
>is ALL HTML entity codes (e.g. 
>Сибирск
>...
>which is true for the message subject, titles displayed, etc.etc.
>
>Why are the messages being saved as HTML entity codes and not 
>UNICODE?


Because the list language is English and the character set for English
in Mailman is us-ascii.

If you set the list's preferred_language to Russian, the archives will
be in koi8-r, Mailman's character set for Russian. (This will only
affect messages added after the change unless you rebuild the archive.)

-- 
Mark Sapiro <[EMAIL PROTECTED]>The highway is for gamblers,
San Francisco Bay Area, Californiabetter use your sense - B. Dylan

--
Mailman-Users mailing list
Mailman-Users@python.org
http://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://wiki.list.org/x/AgA3
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/
Unsubscribe: 
http://mail.python.org/mailman/options/mailman-users/archive%40jab.org

Security Policy: http://wiki.list.org/x/QIA9

Re: [Mailman-Users] Base64 encoding?

[Mark Sapiro]

Jim Garrison wrote:

>Mark Sapiro wrote:
>> Jim Garrison wrote:
>> 
>>> Does Mailman ever take a message that it received in plain
>>> text form and re-encode it in base64 (leaving the mime type
>>> as text/plain)?
>> 
>> 
>> Yes. (It's actually the underlying Python email library that does it.)
>> On English language lists, it happens most commonly with messages
>> whose character set is utf-8.
>
>Is there a way to disable this?


Don't add msg_header or msg_footer to the message. Then Mailman won't
modify the text/plain body part and the Python email library won't
recode it.


>What's the reason for base64
>encoding UTF8? Is it trying to avoid using the high bit of
>each message byte?


Yes, it is avoiding creating messages containg characters with the high
order bit set because not all MTAs support the 8BITMIME SMTP service
extension.

The Python email library has a defined message body encoding for every
character set it understands. For utf-8 this is base64.

-- 
Mark Sapiro <[EMAIL PROTECTED]>The highway is for gamblers,
San Francisco Bay Area, Californiabetter use your sense - B. Dylan

--
Mailman-Users mailing list
Mailman-Users@python.org
http://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://wiki.list.org/x/AgA3
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/
Unsubscribe: 
http://mail.python.org/mailman/options/mailman-users/archive%40jab.org

Security Policy: http://wiki.list.org/x/QIA9

Re: [Mailman-Users] Base64 encoding?

[Jim Garrison]

Mark Sapiro wrote:

Jim Garrison wrote:


Does Mailman ever take a message that it received in plain
text form and re-encode it in base64 (leaving the mime type
as text/plain)?



Yes. (It's actually the underlying Python email library that does it.)
On English language lists, it happens most commonly with messages
whose character set is utf-8.


Is there a way to disable this? What's the reason for base64
encoding UTF8? Is it trying to avoid using the high bit of
each message byte?


--
Jim Garrison ([EMAIL PROTECTED])
PGP Keys at http://www.jhmg.net RSA 0x04B73B7F DH 0x70738D88 


--
Mailman-Users mailing list
Mailman-Users@python.org
http://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://wiki.list.org/x/AgA3
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/
Unsubscribe: 
http://mail.python.org/mailman/options/mailman-users/archive%40jab.org

Security Policy: http://wiki.list.org/x/QIA9

Re: [Mailman-Users] Base64 encoding?

[Mark Sapiro]

Jim Garrison wrote:

>Does Mailman ever take a message that it received in plain
>text form and re-encode it in base64 (leaving the mime type
>as text/plain)?


Yes. (It's actually the underlying Python email library that does it.)
On English language lists, it happens most commonly with messages
whose character set is utf-8.

-- 
Mark Sapiro <[EMAIL PROTECTED]>The highway is for gamblers,
San Francisco Bay Area, Californiabetter use your sense - B. Dylan

--
Mailman-Users mailing list
Mailman-Users@python.org
http://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://wiki.list.org/x/AgA3
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/
Unsubscribe: 
http://mail.python.org/mailman/options/mailman-users/archive%40jab.org

Security Policy: http://wiki.list.org/x/QIA9

Re: [Mailman-Users] Base64 encoding?

[Jim Garrison]

Jim Garrison wrote:

Does Mailman ever take a message that it received in plain
text form and re-encode it in base64 (leaving the mime type
as text/plain)?

This just happened to a message I sent to a private list for
which I am the admin, and I'm trying to pin it down between
Mailman and Postfix.  I composed the message in Thunderbird,
and the source of the saved outgoing copy (in the Sent folder)
is plain ASCII (not base64-encoded).  I am subscribed to the
list so I get a copy of whatever is sent, and the copy
received from the server IS base64-encoded.


Additional info.  I re-sent the message to myself through the
MTA directly and it came back in ASCII, not base64.  Since the
Mailman instance and MTA both run on the same machine, the
only difference in the second case is not going through
Mailman.

--
Jim Garrison ([EMAIL PROTECTED])
PGP Keys at http://www.jhmg.net RSA 0x04B73B7F DH 0x70738D88 


--
Mailman-Users mailing list
Mailman-Users@python.org
http://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://wiki.list.org/x/AgA3
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/
Unsubscribe: 
http://mail.python.org/mailman/options/mailman-users/archive%40jab.org

Security Policy: http://wiki.list.org/x/QIA9

Re: [Mailman-Users] Base64 Encoding -Greek characters- Part oftheproblem solved

[Mark Sapiro]

[EMAIL PROTECTED] wrote:
>
>I sent an email to "mylist" containing greek characters.
>Then, i opened mailman's web interface as the list's moderator to decide
>about the fate of the incoming email.
>In the email's body there were unreadable characters such as =E1=E2=E3.

These are quoted-printable encoded characters (each
non-us-ascii-printable 8-bit byte is encoded as an '=' followed by two
hex digits). In the iso-8859-7 character set they correspond to the
upper case greek letters alpha, beta and gamma.


>Then, i chose "Additionally, forward this message to:
>[EMAIL PROTECTED]" and then "Approve".
>
>The members of the list received unreadable base64 encoded characters
>while the [EMAIL PROTECTED] received proper greek characters.
>
>Is this not strange?


No, it is not strange at all. The forwarded message was sent as is to
the address you forwarded it to. That address received the
quoted-printable encoded message and the MUA used to view it
understood and properly decoded the quoted-printable encoding.

The list message probably had a msg_header or msg_footer added or some
other Mailman manipulation done to it that caused Mailman to reset the
contents of the message body which in turn cause the Python email
library to encode the body with its encoding for iso-8859-7 (or
windows-1253) which is base64.

I doubt very much that Mailman's encoding or not encoding the message
body as base64 is really caused by the presence/absence of the
mailscanner headers. I think you probably also changed some list
configuration (such as removing msg_footer) at the same time and this
is what changed Mailman's behavior.

-- 
Mark Sapiro <[EMAIL PROTECTED]>   The highway is for gamblers,
San Francisco Bay Area, Californiabetter use your sense - B. Dylan

--
Mailman-Users mailing list
Mailman-Users@python.org
http://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://www.python.org/cgi-bin/faqw-mm.py
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/
Unsubscribe: 
http://mail.python.org/mailman/options/mailman-users/archive%40jab.org

Security Policy: 
http://www.python.org/cgi-bin/faqw-mm.py?req=show&file=faq01.027.htp

Re: [Mailman-Users] Base64 Encoding -Greek characters- Part of theproblem solved

[test]

Before changing the configuration of mailscanner, i performed a test that
gave surprising results (for me).

I sent an email to "mylist" containing greek characters.
Then, i opened mailman's web interface as the list's moderator to decide
about the fate of the incoming email.
In the email's body there were unreadable characters such as =E1=E2=E3. 
Then, i chose "Additionally, forward this message to:
[EMAIL PROTECTED]" and then "Approve".

The members of the list received unreadable base64 encoded characters
while the [EMAIL PROTECTED] received proper greek characters.

Is this not strange?

Alex



> [EMAIL PROTECTED] wrote:
>>
>>I've managed to solve the biggest part of the problem. The problem was
>>initiated by Mailscanner. For every email in my mail server, the header
>> of
>>every email was modified and lines like the following were added:
>>
>>"X-Faculty of Social Sciences-MailScanner-Information: Please contact the
>>ISP for more information X-Faculty of Social Sciences-MailScanner: Found
>>to be clean X-Faculty of Social Sciences-MailScanner-From:
>>[EMAIL PROTECTED]
>>X-Spam-Status: No"
>>
>>For some reason this caused mailman to encode emails with base64, so
>> being
>>unreadable by the recipients.
>>I reconfigured Mailscanner so as the headers are not modified and my
>>(happy) recipients can now read the emails.
>
>
> This will only work if both msg_header and msg_footer are empty and
> scrub_nondigest is No. Even in this case, There will be an issue with
> the plain format digest if the character set of the list's
> preferred_language does not support Greek characters.
>
> --
> Mark Sapiro <[EMAIL PROTECTED]>   The highway is for gamblers,
> San Francisco Bay Area, Californiabetter use your sense - B. Dylan
>
>
> --
> This message has been scanned for viruses and
> dangerous content by MailScanner, and is
> believed to be clean.
>
>



-- 
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.

--
Mailman-Users mailing list
Mailman-Users@python.org
http://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://www.python.org/cgi-bin/faqw-mm.py
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/
Unsubscribe: 
http://mail.python.org/mailman/options/mailman-users/archive%40jab.org

Security Policy: 
http://www.python.org/cgi-bin/faqw-mm.py?req=show&file=faq01.027.htp

Re: [Mailman-Users] Base64 Encoding -Greek characters- Part of theproblem solved

[Mark Sapiro]

[EMAIL PROTECTED] wrote:
>
>I've managed to solve the biggest part of the problem. The problem was
>initiated by Mailscanner. For every email in my mail server, the header of
>every email was modified and lines like the following were added:
>
>"X-Faculty of Social Sciences-MailScanner-Information: Please contact the
>ISP for more information X-Faculty of Social Sciences-MailScanner: Found
>to be clean X-Faculty of Social Sciences-MailScanner-From:
>[EMAIL PROTECTED]
>X-Spam-Status: No"
>
>For some reason this caused mailman to encode emails with base64, so being
>unreadable by the recipients.
>I reconfigured Mailscanner so as the headers are not modified and my
>(happy) recipients can now read the emails.


This will only work if both msg_header and msg_footer are empty and
scrub_nondigest is No. Even in this case, There will be an issue with
the plain format digest if the character set of the list's
preferred_language does not support Greek characters.

-- 
Mark Sapiro <[EMAIL PROTECTED]>   The highway is for gamblers,
San Francisco Bay Area, Californiabetter use your sense - B. Dylan

--
Mailman-Users mailing list
Mailman-Users@python.org
http://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://www.python.org/cgi-bin/faqw-mm.py
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/
Unsubscribe: 
http://mail.python.org/mailman/options/mailman-users/archive%40jab.org

Security Policy: 
http://www.python.org/cgi-bin/faqw-mm.py?req=show&file=faq01.027.htp

Re: [Mailman-Users] Base64 Encoding -Greek characters- Part of the problem solved

[Brad Knowles]

On 4/28/07, [EMAIL PROTECTED] wrote:

>  I've managed to solve the biggest part of the problem. The problem was
>  initiated by Mailscanner. For every email in my mail server, the header of
>  every email was modified and lines like the following were added:

If you check your previous message at 
, 
you will note that these are not being inserted as actual headers in 
the message, although it probably appears that way to you.  There was 
a blank line between the last "real" header and these lines, which 
made them part of the message body.

If you look at the original message as you sent it, it had the MIME headers:

Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: 7bit

So, SquirrelMail was sending things just fine, without a 
content-transfer-encoding of base64.  And it doesn't look like 
MailScanner changed anything here.  Of course, the original message 
was probably 7-bit clean in US-ASCII (as opposed to using eight-bit 
accented Greek characters), so without more information we don't know 
for sure where the conversion to base64 was happening.

>  For some reason this caused mailman to encode emails with base64, so being
>  unreadable by the recipients.
>  I reconfigured Mailscanner so as the headers are not modified and my
>  (happy) recipients can now read the emails.

The clients should still have been able to deal with base64 in a 
transparent manner, so this is clearly also a client problem.

-- 
Brad Knowles <[EMAIL PROTECTED]>, Consultant & Author
LinkedIn Profile: 
Slides from Invited Talks: 
--
Mailman-Users mailing list
Mailman-Users@python.org
http://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://www.python.org/cgi-bin/faqw-mm.py
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/
Unsubscribe: 
http://mail.python.org/mailman/options/mailman-users/archive%40jab.org

Security Policy: 
http://www.python.org/cgi-bin/faqw-mm.py?req=show&file=faq01.027.htp