Re: [Nmh-workers] I need help reading the mhstore man page

2014-02-28 Thread David Levine
> >The man page for mhstore recommends that, for the sake of security, > >I not put the -auto switch in .mh_profile. Whatever the security > >risk is, would it not also be present if I invoke mhstore with that > >switch? But the man page does not seem to recommend against that. Yes, they're equiva

Re: [Nmh-workers] General question - unsupported charset conversion

2014-02-28 Thread Ken Hornstein
>That is right. On the other hand, you never prevent malformed MIME >parameters. Remember that we're not talking about malformed MIME parameters; we're talking about entirely valid ones. >It is not a problem in case of one or two missing or substituted >symbols in long text. We can guess what is

Re: [Nmh-workers] General question - unsupported charset conversion

2014-02-28 Thread Aleksander Matuszak
Ken Hornstein writes: > >Unfortunately, I have a lot of experience and troubles with character > >set conversion. > > Well, if you just bit the bullet and switched to UTF-8, you wouldn't have > all of these problems! :-) It is not that simple. Utf-8 solves couple of problems but creates some ne

Re: [Nmh-workers] General question - unsupported charset conversion

2014-02-28 Thread Ken Hornstein
>Look, software cannot read minds. People would like it to, but I don't >work for the NSA, so I don't buy into that concept. We have standards. >For a reason. To eliminate ambiguity. MIME has been around for how >many years now? There is no excuse in this day and age for any software >to gener

Re: [Nmh-workers] General question - unsupported charset conversion

2014-02-28 Thread Lyndon Nerenberg
On Feb 28, 2014, at 1:01 PM, Ken Hornstein wrote: > Based on _what you want to happen_, what, exactly, should be > done from a programming perspective? Bail? Yes! Bail! Don't be a vector for someone to do nasties! If people want to see invalid content, they have cat(1) at hand. One of the j

Re: [Nmh-workers] General question - unsupported charset conversion

2014-02-28 Thread Lyndon Nerenberg
On Feb 28, 2014, at 12:55 PM, Lyndon Nerenberg wrote: >> Sigh, IT'S THE SAME THING. iconv() returns EILSEQ at a particular point >> in your conversion buffer. What do you do next? There was argument along the way about iconv versions not groking character sets. Look, software cannot read mi

Re: [Nmh-workers] General question - unsupported charset conversion

2014-02-28 Thread Ken Hornstein
>> Sigh, IT'S THE SAME THING. iconv() returns EILSEQ at a particular point >> in your conversion buffer. What do you do next? > >In your example, emit a Pile Of Poo. I know you're being flippant ... but it's a serious question. Right now, iconv() returns EILSEQ if you cannot convert an input ch

Re: [Nmh-workers] General question - unsupported charset conversion

2014-02-28 Thread Lyndon Nerenberg
On Feb 28, 2014, at 12:42 PM, Ken Hornstein wrote: > Sigh, IT'S THE SAME THING. iconv() returns EILSEQ at a particular point > in your conversion buffer. What do you do next? In your example, emit a Pile Of Poo. signature.asc Description: Message signed with OpenPGP using GPGMail __

Re: [Nmh-workers] General question - unsupported charset conversion

2014-02-28 Thread Ken Hornstein
>> We'd still have to deal with what happens when you want to convert >> U+1F4A9 to ISO-8859-1. > >That's not an illegal parse of the input, it's a composting problem. >Not the same thing at all. Sigh, IT'S THE SAME THING. iconv() returns EILSEQ at a particular point in your conversion buffer. W

Re: [Nmh-workers] General question - unsupported charset conversion

2014-02-28 Thread Lyndon Nerenberg
On Feb 28, 2014, at 12:24 PM, Ken Hornstein wrote: > Fair enough ... but iconv() is part of POSIX, so assuming that it's available > is reasonable (if you don't have iconv(), we basically give up in terms of > handling different character sets). Sadly, iconv() in practice is a nightmare. The v

Re: [Nmh-workers] General question - unsupported charset conversion

2014-02-28 Thread Lyndon Nerenberg
On Feb 28, 2014, at 12:01 PM, Ken Hornstein wrote: > We'd still have to deal with what happens when > you want to convert U+1F4A9 to ISO-8859-1. That's not an illegal parse of the input, it's a composting problem. Not the same thing at all. signature.asc Description: Message signed with O

Re: [Nmh-workers] General question - unsupported charset conversion

2014-02-28 Thread Ken Hornstein
>Recode need not be required, it could just be an option. iconv currently >isn't afterall, although they seem to complement each other. Recode is >part of the core distrib of my older Ubuntu 10.02. Fair enough ... but iconv() is part of POSIX, so assuming that it's available is reasonable (if you

Re: [Nmh-workers] General question - unsupported charset conversion

2014-02-28 Thread Lyndon Nerenberg
On Feb 28, 2014, at 12:01 PM, Ken Hornstein wrote: > If we make sure we're converting all non-printable characters into something > else, I'm unclear as to how that could happen. But if it can happen, please > educate me! It's a case of fooling the GB* and multibyte converters into aborting at

Re: [Nmh-workers] General question - unsupported charset conversion

2014-02-28 Thread Jerrad Pierce
Recode need not be required, it could just be an option. iconv currently isn't afterall, although they seem to complement each other. Recode is part of the core distrib of my older Ubuntu 10.02. Selective recoding would probably require calls for the substrings of interest. As an aside, recode's

Re: [Nmh-workers] General question - unsupported charset conversion

2014-02-28 Thread Ken Hornstein
>My feeling is that if you don't recognize the source character set, you >cannot possibly convert it to a display format in any secure manner. By >default I think we should not display the content, but instead spit out >a diagnostic, with the option to re-run the show (or whatever) with a >command

Re: [Nmh-workers] General question - unsupported charset conversion

2014-02-28 Thread Lyndon Nerenberg
This gets very icky, very quickly :-P My feeling is that if you don't recognize the source character set, you cannot possibly convert it to a display format in any secure manner. By default I think we should not display the content, but instead spit out a diagnostic, with the option to re-run

Re: [Nmh-workers] General question - unsupported charset conversion

2014-02-28 Thread Ken Hornstein
>kr>Um, ouch. Unless there's a common library that already implements >kr>that behavior, that's not on the table at all. > >Supposedly Recode does: http://recode.progiciels-bpi.ca/index.html A super-quick scan of our systems does not show that as something that comes out of the box installed on o

Re: [Nmh-workers] General question - unsupported charset conversion

2014-02-28 Thread Jerrad Pierce
am>>In my personal opinion a very good choice is conversion into am>>html-entities, like ą or ł . It remains quite readable and am>>is still unique enough to convert it back in case of need. kr>Um, ouch. Unless there's a common library that already implements kr>that behavior, that's not on the t

Re: [Nmh-workers] General question - unsupported charset conversion

2014-02-28 Thread Ken Hornstein
>Unfortunately, I have a lot of experience and troubles with character >set conversion. Well, if you just bit the bullet and switched to UTF-8, you wouldn't have all of these problems! :-) >> Should we return the original bytes? > >It is not the best idea. Some sequences of bytes are control s

Re: [Nmh-workers] General question - unsupported charset conversion

2014-02-28 Thread Aleksander Matuszak
Ken Hornstein writes: > I've been grappling with to do when we have issues with character set > conversion. Unfortunately, I have a lot of experience and troubles with character set conversion. > Specifically, I have two issues: > > - What to do if the character set is unsupported. > Should

[Nmh-workers] General question - unsupported charset conversion

2014-02-28 Thread Ken Hornstein
Greetings all, With the greater MIME-ification of nmh (long overdue!) I've been grappling with to do when we have issues with character set conversion. Specifically, I have two issues: - What to do if the character set is unsupported. Like if we don't have iconv, or we get a character set tha

Re: [Nmh-workers] I need help reading the mhstore man page

2014-02-28 Thread Ken Hornstein
>The man page for mhstore recommends that, for the sake of security, I not put >the -auto switch in .mh_profile. Whatever the security risk is, would it not >also be present if I invoke mhstore with that switch? But the man page does >not seem to recommend against that. -auto uses the filename tha

[Nmh-workers] I need help reading the mhstore man page

2014-02-28 Thread norm
The man page for mhstore recommends that, for the sake of security, I not put the -auto switch in .mh_profile. Whatever the security risk is, would it not also be present if I invoke mhstore with that switch? But the man page does not seem to recommend against that. The '|' facility is an obvious