Re: mhfixmsg character set conversion
>Ah, OK, maybe it wasn't lynx. I don't know enough about your >environment to say exactly what heppened. Quick question: would it help if I were to run mhfixmsg under control of strace and send you the output? That's the most definitive way I can think of to show you exactly what happens in my environment. (Yes, the problem occurred when procmail invoked my shell script which then invoked mhfixmsg, but I see the same behaviour when I run mhfixmsg directly on the command line with the same input file, so I believe that strace under those circumstances should capture everything relevant.) - Steven -- ___ Steven Winikoff | "The man who has ceased to learn ought Montreal, QC, Canada | not to be allowed to wander around s...@smwonline.ca | loose in these dangerous days." http://smwonline.ca | | - M. M. Coady
Re: mhfixmsg character set conversion
Steven wrote: > Thank you! I'll check it out tomorrow and see what happens. Do you > think the patch will apply to 1.7.1, or will I need to install the > latest version from the repository? It's been almost 4 years since 1.7.1 so I wouldn't try patching it. This is an easy way to download and build the latest, assuming that you have the prerequisites listed in the MACHINES file (and respond to the build_nmh questions): wget http://git.savannah.gnu.org/cgit/nmh.git/plain/build_nmh sh build_nmh -v >- While testing on Friday, I emptied that file out completely and still > observed the same behaviour. Ah, OK, maybe it wasn't lynx. I don't know enough about your environment to say exactly what heppened. And so I'm not certain that we've solved the problem yet. The mismatch between what external programs did and what mhfixmsg expected was definitely a problem. > While I'm replying to you anyway, I realize I forgot to reply to your > question from yesterday morning. You asked: > >Have you tried the -decodeheaderfieldbodies switch to mhfixmsg? > > I haven't, mainly because I didn't know that switch existed. I don't know > what it does (other than what I can infer from the name, of course), and I > can't find any mention of it in the man page for mhfixmsg, or anywhere in > the source code for version 1.7.1. Was this switch added after 1.7.1 was > released? Yes, it was added in January 2018 but not included in 1.7.1. > >I'm not completely sure that this will fix your problem because > >it's aimed at added text/plain parts. But with -noreplacetextplain > >I think that's the path to your issue. > > Please advise on the easiest way to try it (between applying 41ce4490ac5d to > 1.7.1, or just downloading and building the current version of the master > branch), and I'll do so tomorrow (I'm unable to do it before then due to > a prior commitment). I would try the latest nmh. And/or send me the message off-line and I'll take a look. David
Re: mhfixmsg character set conversion
On Sun, 06 Feb 2022 09:26:14 -0800, David Levine said: > Ok, commit 41ce4490ac5d might fix the problem for you. The cause > was mismatch between the character set of content generated by the > external program (lynx, in this case) and mhfixmsg -textcharset > UTF-8. mhfixmsg wasn't capturing the charset of the generated > output, assuming it was unchanged. It then converted it again. So I did a 'git pull', and 'git show HEAD' reports: commit 51f9b175b18f4f37ebc7a5135346c7f3433cf100 (HEAD -> master, origin/master, origin/HEAD) Author: David Levine Date: Sun Feb 6 11:37:22 2022 -0500 Fixed charset determination code. Fix to commit 41ce4490ac5d5c6a25240c4101cacefa16d885eb. And I'm seeing 2 failures in 'make check', am attaching the output from 'make check |& tee check.output' as a gzip to avoid any possible charset wonkiness... I also hit a third error, but I *think* it's just 'par' not being installed. I wonder where it went, I'm pretty sure I installed it somewhere along the line. In any case, test-convert's message isn't correct, as the directory /home/valdis/src/nmh/test/testdir/Mail/ doesn't exist, and the file 'draft' is obviously not in the non-existent directory. check.output.gz Description: check.output.gz
Re: mhfixmsg character set conversion
>Steven wrote:
>
>> That's probably a helpful thing to do, but the question I was wondering
>> about wasn't why the UTF-to-UTF conversion was reported, but rather why
>> the iso-8859-1-to-UTF conversion wasn't reported.
>
>Ok, commit 41ce4490ac5d might fix the problem for you.
Thank you! I'll check it out tomorrow and see what happens. Do you
think the patch will apply to 1.7.1, or will I need to install the
latest version from the repository?
>The cause was mismatch between the character set of content generated by
>the external program (lynx, in this case) and mhfixmsg -textcharset UTF-8.
>mhfixmsg wasn't capturing the charset of the generated output, assuming it
>was unchanged. It then converted it again.
I'm afraid I have to admit I'm not entirely clear on how lynx is even
involved. I know I have these entry in my system-wide mhn.defaults file:
$ grep lynx /local/pkg/nmh/root-nmh-1.7.1/etc/mhn.defaults
mhbuild-convert-text/html: charset="%{charset}"; /sbin/lynx -child -dump
-force_html ${charset:+--assume_charset} ${charset:+"$charset"} %F | sed
's/^\(.\)/> \1/; s/^$/>/;' | par 64
mhfixmsg-format-text/html: charset="%{charset}"; /sbin/lynx -child -dump
-force_html ${charset:+--assume_charset} ${charset:+"$charset"} %F | expand |
sed -e 's/^ //' -e 's/ *$//'
mhshow-show-text/html: charset="%{charset}"; %l/sbin/lynx -child -dump
-force-html ${charset:+--assume_charset} ${charset:+"$charset"} %F
...and the second of these certainly looks relevant, but:
- While testing on Friday, I emptied that file out completely and still
observed the same behaviour.
- In your message from 12:35 today in the "crufty mhn.default.sh stuff"
thread, you wrote:
> There is a way. etcpath() looks for mhn.defaults in this order:
> * 3) Next, check in nmh Mail directory.
> * 4) Next, check in nmh `etc' directory.
>
> So if the user puts an mhn.defaults in their Mail directory, then
> only it will be read. They'd have to copy any entries that they
> do want from /etc/nmh/mhn.defaults to their own.
I do have an mhn.defaults file in my Mail directory, with (only) these
entries in it:
mhshow-show-application/pdf: %pmime_helper %F %s "%{name}"
mhshow-show-application: %pmime_helper %F %s "%{name}"
mhshow-show-audio: %pmime_helper %F %s "%{name}"
mhshow-show-video: %pmime_helper %F %s "%{name}"
mhshow-show-image: %pmime_helper %F %s "%{name}"
mhshow-show-text/richtext: %pmime_helper %F %s "%{name}"
...so while I believe that lynx is involved, I don't know where that
involvement is coming from.
While I'm replying to you anyway, I realize I forgot to reply to your
question from yesterday morning. You asked:
Have you tried the -decodeheaderfieldbodies switch to mhfixmsg?
I haven't, mainly because I didn't know that switch existed. I don't know
what it does (other than what I can infer from the name, of course), and I
can't find any mention of it in the man page for mhfixmsg, or anywhere in
the source code for version 1.7.1. Was this switch added after 1.7.1 was
released?
>The fix is for mhfixmsg to detect the charset of the content,
>using file --brief --mime-encoding if it can. If it can't, it
>falls back to the -textcharset value. If that wasn't used, it
>gets it from the locale and advises the user.
That sounds reasonable to me.
>I'm not completely sure that this will fix your problem because
>it's aimed at added text/plain parts. But with -noreplacetextplain
>I think that's the path to your issue.
Please advise on the easiest way to try it (between applying 41ce4490ac5d to
1.7.1, or just downloading and building the current version of the master
branch), and I'll do so tomorrow (I'm unable to do it before then due to
a prior commitment).
- Steven
--
___
Steven Winikoff | "The thing is, I mean, there's times when
Montreal, QC, Canada | you look at the universe and you think,
s...@smwonline.ca | 'What about me?' and you can just hear
http://smwonline.ca | the universe replying, 'Well, what about
| you?'"
| - Terry Pratchett (Thief of Time)
Re: automatic decode mime in repl
Philipp wrote: > I don't see a reason for an option[1], because I consider the > current behavior as a bug. There shouldn't be an option to enable/disable > a bug. That's a fair point, but I'd be more reluctant to change default behavior in case users depend on it. Have you tried nmh/share/doc/nmh/contrib/replaliases? (Or whereever it's installed on your system.) It think it could be configured to do what your patch does. In any case, it works as-is with recent nmh. Of course, it has it's own drawbacks, primarily with the user editing quoted-printable. That should be solvable though I haven't been motivatee enough to try. David
Re: crufty mhn.default.sh stuff
I wrote: > Ralph wrote: > > > Can a user's profile ‘delete’ a /etc/nmh/mhn.defaults entry as if it > > never existed without defining something in its place? So that the code > > runs as if the entry didn't exist in either of them? > > Good question. I don't see a way to do that. There is a way. etcpath() looks for mhn.defaults in this order: * 3) Next, check in nmh Mail directory. * 4) Next, check in nmh `etc' directory. So if the user puts an mhn.defaults in their Mail directory, then only it will be read. They'd have to copy any entries that they do want from /etc/nmh/mhn.defaults to their own. David
Re: automatic decode mime in repl
>I can understand that a changed behavior is a problem. But in this case >I would argue there is no other way. Every patch witch tries to fix this >will change the existing behavior, because the existing behavior is the >problem. Right, I get that. But as Paul Fox points out, plenty of people have come up with workarounds for this behavior and I'm reluctant to break that. The eternal problem, sigh. >So I would prefer no option. But if you want one, I can add an option. >I would prefer the new behavior be the default. Also I would prefer a >build time option over a runtime option. I'm bad at naming, has someone >an idea for the name of such an option. Oof, well, if we've learned ONE thing after all of these years, it's that build time options are terrible, TERRIBLE ideas. The big reason is that most users use a nmh install from a distribution, so whatever configuration they end up is based on the person who created the distribution build. I think build time options are fine for things that require external libraries (e.g., TLS or Cyrus-SASL support) but to change user behavior I think this is definitely sub-optimal. So in general, I think the options are: 1) Make the new behavior the default, and require that anyone who wants the old behavior add a switch (that will will allow existing setups to work but it will require that people who have those setups to change something) 2) Make new behavior optional with a switch 3) Make new behavior the default but try to auto-detect if a user has a customized reply configuration. I understand where you're coming from in that the current behavior is a bug and the default should just work; I agree! So to me that completely eliminates #2 as an option. Is #3 feasable? Not sure; I'll have to think about it. Also, just thinking out loud; should the default maybe be "mhshow | sed -e 's/^/>/'"? --Ken
Re: mhfixmsg character set conversion
Steven wrote: > That's probably a helpful thing to do, but the question I was wondering > about wasn't why the UTF-to-UTF conversion was reported, but rather why > the iso-8859-1-to-UTF conversion wasn't reported. Ok, commit 41ce4490ac5d might fix the problem for you. The cause was mismatch between the character set of content generated by the external program (lynx, in this case) and mhfixmsg -textcharset UTF-8. mhfixmsg wasn't capturing the charset of the generated output, assuming it was unchanged. It then converted it again. The fix is for mhfixmsg to detect the charset of the content, using file --brief --mime-encoding if it can. If it can't, it falls back to the -textcharset value. If that wasn't used, it gets it from the locale and advises the user. I'm not completely sure that this will fix your problem because it's aimed at added text/plain parts. But with -noreplacetextplain I think that's the path to your issue. I also added -display_charset UTF-8 to the mhn.defaults config for lynx, based on the assumption that that's what most users want. Maybe it would be better to leave the charset unchanged at that point and have iconv convert it later. But that would lead to requiring a -textcharset to determine the final charset. David
Re: automatic decode mime in repl
[2022-02-06 08:26] Paul Fox > philipp wrote: > > [2022-02-05 22:19] Ken Hornstein > > > > > > My only concern with this patch is that it unilaterally overrides the > > > existing behavior, and one thing I've seen over the years is that there > > > are a LOT of users wedded to existing behavior. So I'd rather make it > > > selectable (I am neutral about it being default or not). I'd welcome > > > discussion on this issue. > > > > I can understand that a changed behavior is a problem. But in this case > > I would argue there is no other way. Every patch witch tries to fix this > > will change the existing behavior, because the existing behavior is the > > problem. > > > > So I would prefer no option. But if you want one, I can add an option. > > I would prefer the new behavior be the default. Also I would prefer a > > build time option over a runtime option. I'm bad at naming, has someone > > an idea for the name of such an option. > > I haven't tried the patch, but I think any change to basic repl > behavior should a) be optional, b) be controlled by a runtime option. > And it should probably have a lot of testing by a lot of users before > being made the default. > > I can't be alone in having customized scripts for doing replies which > already run mhshow in crafting the reply. Perhaps everything would > continue to just work, or perhaps not. I understand your reasaning. I just fear then this will get just an other stopgap solution. I would assume most users with a workaround for the problem will try this once. If it don't work with there workaround, they will just disable it. If someone brings up the idea of change the default they would vote against it. I don't want to blame such behavior. This makes perfect sense from the standpoint of a user with a workaround. It just don't make nmh better. I belive reply to a mime message should just work. The best solution would be implementing mime support in mhl. But this will not happen for a long time. So the goal should be add a solution[0] and make it the default. I don't see a reason for an option[1], because I consider the current behavior as a bug. There shouldn't be an option to enable/disable a bug. As said in my previous mail, I'm not complete against an option. I would just prefere not having one. Philipp [0] I don't care if this is my solution or any other. I just don't see another solution. [1] You still can change the showmimeproc to disable this.
Re: automatic decode mime in repl
philipp wrote: > [2022-02-05 22:19] Ken Hornstein > > > > My only concern with this patch is that it unilaterally overrides the > > existing behavior, and one thing I've seen over the years is that there > > are a LOT of users wedded to existing behavior. So I'd rather make it > > selectable (I am neutral about it being default or not). I'd welcome > > discussion on this issue. > > I can understand that a changed behavior is a problem. But in this case > I would argue there is no other way. Every patch witch tries to fix this > will change the existing behavior, because the existing behavior is the > problem. > > So I would prefer no option. But if you want one, I can add an option. > I would prefer the new behavior be the default. Also I would prefer a > build time option over a runtime option. I'm bad at naming, has someone > an idea for the name of such an option. I haven't tried the patch, but I think any change to basic repl behavior should a) be optional, b) be controlled by a runtime option. And it should probably have a lot of testing by a lot of users before being made the default. I can't be alone in having customized scripts for doing replies which already run mhshow in crafting the reply. Perhaps everything would continue to just work, or perhaps not. paul =-- paul fox, p...@foxharp.boston.ma.us (arlington, ma)
Re: automatic decode mime in repl
[2022-02-05 22:19] Ken Hornstein > >Reply to mime messages is an old problem. I have an idea to workaround > >or fix this. Instand of adding mime to mhl, repl could also first decode > >mime before pipe the mail to mhl. This can be done by mhshow. > > > >I have attached a patch for this. > > From a purely architectural perspective, I think that having mhl be > MIME-aware is the correct solution. But I must acknowledge the reality > on the ground: > > - This has been a problem for a very long time > > - Stopgap solutions (like replyfilter) have not made much headway, because > they require end-user configuration. > > - Getting a MIME-aware mhl (really, all of nmh) is a huge job. > > - I believe "perfect is the enemy of the good". > > My only concern with this patch is that it unilaterally overrides the > existing behavior, and one thing I've seen over the years is that there > are a LOT of users wedded to existing behavior. So I'd rather make it > selectable (I am neutral about it being default or not). I'd welcome > discussion on this issue. I can understand that a changed behavior is a problem. But in this case I would argue there is no other way. Every patch witch tries to fix this will change the existing behavior, because the existing behavior is the problem. So I would prefer no option. But if you want one, I can add an option. I would prefer the new behavior be the default. Also I would prefer a build time option over a runtime option. I'm bad at naming, has someone an idea for the name of such an option. An alternativ would be to depend on fmtproc. When fmtproc is set don't pipe through mhshow. When it is not set pipe through mhshow. Philipp > --Ken > > >This approatch has currently two problems: > > > >* mhshow can be configured to display parts in a graphical programm. This > > can lead to unexpected behavior on repl (i.e. start plaing musik). > > > >I belive there are two options to fixed this problem. First the -textonly > >swtich can be used. In some probably uncommon setups (i.e. text to speech) > >this still can lead to problems. > > > >The secound option is a bit more complex. Instand of only depending on > >the programm name the config can be extended to also depend on the > >calling programm. So you can config diffrent options depending on which > >nmh programm called another nmh programm. This could be done by (ab)using > >argv[0]. > > > >* mhshow display string can contain '%l' which leads to a listing prior > > to displaying content. This listing is contained in the draft. > > > >To fix this I would suggest to remove the '%l' from the display string > >escapes and create a switch for that. Then you can't enable/disable this > >per part. > > > >What do you think about this patch? Did I missed some problems? > > > >Philipp > > > >>> text/x-diff content
