On Wednesday 11 September 2002 09:29 pm, Earl Hood wrote:
> On September 11, 2002 at 03:50, Mooffie wrote:
> > Currently, charset conversion routines are not applied to HTML messages.
. . .
> > My suggestion is to apply the charset conversion routines to HTML
> > messages as well.
. . .
> A thing to consider is to have a pre-filtering step for any
> text/* type that allows for a pre-conversion processing before a text
> entity is passed to a filter.
Yes, that's a better idea.
> This relates to the TODO item of having
> chained filters.
Chained? You mean that they'll run one after the other?
If you examine a filter in HSMA, you'll see that its structure is:
sub my_new_HTML_filter {
# 1. do some processing...
# 2. call MHA's HTML filter
# 3. do some more processing...
}
And this looks more usefull than a simple chain.
> > 2.
> >
> > There are two instances where we (and MHA) don't know the charset of the
> > data:
. . .
> > Since MHA doesn't know the charset of the data, a UTF-8 conversion can't
> > be carried out.
. . .
> > My suggestion is to create a new resource, <DefaultCharset>, that allows
> > one to specify a default charset.
. . .
> This is a nice shortcut to something like the following that has
> the same effect:
>
> <CharsetConverters>
> plain; MyMHADefault::str2html; MyMHADefault.pm
> </CharsetConverters>
>
> In MyMHADefault.pm:
>
> package MyMHADefault;
> require 'readmail.pl';
>
> my $default_charset = 'iso-8859-8';
> sub str2html {
> my $charcnv = readmail::load_charset($default_charset);
> $charcnv($_[0], $default_charset);
> }
> 1;
That's right, but you can't expect the user to be a Perl programmer... so this
isn't merely a "shortcut".
> > 3.
> >
> > Misconfigured MUAs, including some web-mails, may declare an incorrect
> > charset.
. . .
> > My suggestion is to create a new resource, <CharsetAliases>, to have MHA
> > treat some charsets as others. Then, for example, if I have a Hebrew
> > mailing list, I'd write:
> >
> > <CharsetAliases>
> > iso-8859-8; us-ascii iso-8859-1 iso-8859-8-i x-unknown x-user-defined
> > </CharsetAlises>
. . .
> You can do the following to get a similiar effect:
>
> <CharsetConverters>
> us-ascii; MyHebrewConverter::str2html; MyHebrewConverter.pm
> iso-8859-1; MyHebrewConverter::str2html; MyHebrewConverter.pm
> iso-8859-8; MyHebrewConverter::str2html; MyHebrewConverter.pm
> iso-8859-8-i; MyHebrewConverter::str2html; MyHebrewConverter.pm
> x-unknown; MyHebrewConverter::str2html; MyHebrewConverter.pm
> x-user-defined; MyHebrewConverter::str2html; MyHebrewConverter.pm
> </CharsetConverters>
Your suggestion may fail. Imagine the following situation:
Let's say MyHebrewConverter::str2html() converts its input to UTF-8.
Now, I have iso-8859-8 data which is incorrectly declared (by the MUA) as
"us-ascii". MyHebrewConverter::str2html() will see "us-ascii" in the
"charset" argument and thus the conversion will fail -- because it's not
"us-ascii".
However, if "us-ascii" is an alias for "iso-8859-8" (using <CharsetAliases>),
MyHebrewConverter::str2html() will think that the data is "iso-8859-8", which
it really is.
> I guess the main sematic difference is that if CharsetAliases
> were used, if mhonarc sees "us-ascii", when it calls
> MyHebrewConverter::str2html, it will actually pass in "iso-8859-8"
> as $charset instead of "us-ascii"
Exactly!
In HSMA I indeed used the above two ways to solve the problem:
1. I provided a resource file with the elaborate <CharsetConverters> resource
you just gave, for use when the user wants the archive encoding to be
windows-1255 (which is a 8-bit encoding),
2. Because the user may want the archive encoding to be UTF-8, I had to
include an aliases table in the code that does the UTF-8 conversion.
A <CharsetAliases> resource can solve this problem. You may have got the
impression that it's only usefull for buggy MUAs, but that's not so. For
example, Hebrew messages may be marked as either "iso-8859-8" or
"iso-8859-8-i" (both are standardized, and both stand for the same encoding,
but the first, which is deprecated nowadays, stands for "Visual Hebrew", and
the later for "Logical Hebrew" (the meaning is not important for our
discussion)). What I want is to let MHA know that "iso-8859-8-i" is actually
the standard "iso-8859-8" encoding, and a <CharsetAliases> mechanism looks
like an elegant solution.
Note that my suggestions are intended to help the ordinary user that manages
MHA (the "administrator"). If all users were programmers, they wouldn't need
them (albeit they'd have to spend a lot of time coding).
> I can see the usefulness of it, especially when used with the existing
> converters.
Excellent.
> > 4.
> >
> > I see that UTF8.pm includes a few hard-coded aliases (e.g.
> > "windows-1250" --> "cp1250"). It might be possible to extend
> > <CharsetAliases> to have this function too; for example:
> >
> > <CharsetAliases>
> > cp1250; windows-1250
> > . . .
> > cp1255; windows-1255
> > . . .
> > apple-hebrew; x-mac-hebrew
> > </CharsetAlises>
>
> True.
Actually, it would be silly for <CharsetAliases> not to have this function,
because the intent is to spare the user editing the source.
> > 5.
> >
> > Although UTF-8 has its advantages, some administrators might prefer
> > their national 8-bit encoding (because it requires less disk space,
> > because they already have 3rd party tools that work with it (e.g. search
> > tools), etc). It seems that it won't be difficult to create a new
> > conversion routine (one can start from MHonArc::UTF8::str2sgml) that
> > converts everything to a common arbitrary encoding, which can be a 8-bit
> > based one. A new resource, e.g. <TargetEncoding> or <ArchiveEncoding>,
> > could determine this target encoding (which could also be "utf-8"(!), so
> > this routine could eventually obsolete MHonArc::UTF8::str2sgml).
>
> Can't someone achieve something similiar with:
>
> <DecodeHeads>
> <CharsetConverters override>
> plain; mhonarc::htmlize;
> default; -decode-
> </CharsetConverters>
No, the intent is to convert all messages to a common encoding the user
specifies. You already have a function, MHonArc::UTF8::str2sgml, that
converts all messages to UTF-8. My suggestion is to extend this function so
that any target encoding is possible, not just UTF-8.
> Of course, it does not allow one to explicitly specify a target
> encoding to allow for "smart" conversion from one charset to the
> final one since whatever is registered for the "plain" set is not
> provided any information on what the source format really is.
Yes, but this can be solved with the <DefaultCharset> resource.
> Ah, my SGML background is showing through. The named entities are
> standard in SGML, and unfortunately, never adopted by HTML. I knew
> this would eventually be a problem.
>
> The named entities have the advantage of being usable across character
> sets while numeric are tied to the current character set in use.
No, numeric character references are _independent_ of the encoding of the
page, because they specify the unicode number of the character.
That is, "þ" is always the the letter Thorn, no matter what the encoding
is.
A good explanation can be found in the HTML spec:
http://www.w3.org/TR/REC-html40/charset.html
And while you're at it, please check the folowing list:
http://www.w3.org/TR/REC-html40/sgml/entities.html
> I welcome numeric character entity reference mappings using the
> &#x...; notation. This should work for most modern browsers and
> avoid have to have dependencies on other modules (at least in the
> default configuration).
Since the named entities don't work anyway (well... the bulk of them), using
numeric character references can't be worse :-)
> A major goal, and I think one reason for it usefulness, is that it
> should be easy to install and get going.
I agree with you.
I urge you to read the HSMA source and *.mrc files in order to understand the
rationales behind my suggestions. In the code I dealt with the above issues,
issues that are not at all specific to Hebrew, and which, in my opinion,
should be handled by MHA itself.
---------------------------------------------------------------------
To sign-off this list, send email to [EMAIL PROTECTED] with the
message text UNSUBSCRIBE MHONARC-DEV
