On June 25, 1999 at 04:06, Jeff Breidenbach wrote:
> I was thinking about automatic language detection. If mailing
> list traffic was predominantly Icelandic, I would like to automaticly
> ask MHonArc switch over to a resource file localized for Icelandic.
This is probably best done before invoking mhonarc. Trying to
change resources on the fly has problems. If we take your language-type
changing, what should happen if messages are encountered with
multiple languages?
Making resource choice is probably best done at the archive level.
> Being completely naive, I pulled up a few non-English emails and
> looked for some line in the headers that identified the language. How
> incredibly depressing. The only relevant headers I found were the
> character set, which appears common for dozens of langages. The only
RFC 2184 provides methods to encode language information in message
header text. From my reading it is a little unclear if it can
be used for specifying language in message bodies. For example:
Content-Type: text/plain; charset="iso-8859-1'en"
You can also have the case where multiple languages exist in the
same message.
> other header clue was the domain of the list server, which is hardly a
> sure thing, given the pervasiveness of both the English language and
> the .com domain name. What do people do for automatic language
> detection for email? Are they stuck with scanning the body for common
> dictionary words? Bleah!!
>
> So the question is:
>
> a) Am I missing something obvious
No. Multi-language/charset spport is never trivial from my observations.
> b) Are there any languages that are easily detected
> (perhaps by a unqiue character set?) If so, are
> those languages supported by MHonArc? Oh, and what
> are they? <grin>
Many charsets support multiple languages. There are some exceptions
where charsets were specifically developed for a given language.
Russian and Japanese are good examples where there are charsets specific
to these languages.
> I guess I'll have to scuttle the whole thing; if so that's too bad,
> since I really think it would be great to automatically customize
> to a particular language.
An approach is to have a mapping of mailing list to language. Whatever
you have that takes the mail and invokes mhonarc, it will check to
see if there is an entry in the mapping for mailing list, and invoke
mhonarc with the proper rcfile.
Or more simply, having mapping that can map a mailing list name to
a specific rcfile. If no map is given for the list, use a default
rcfile. This way, you only have to defined mappings for the exceptions.
--ewh
