Re: mysterious c0 80
Hi David, > > > > nmh shouldn't comp(1) a new email today with a NUL in the body, > > > > but it should be able to read and show(1) one. > > > > > > I'm thinking of removing the support in post(8) for sending NULs. > > > Any disagreement? It's not a lot of code so could be easily > > > restored in the future if conditions change. > > > > > > > Now, how about dist(1) of that old email? I'd have thought it > > > > should send the old email verbatim, NUL and all. If that causes > > > > a bounce then the sender can MIME-forward instead with a single > > > > message/rfc822 part. > > > > > > Agreed. > > > > But doesn't dist → send → post so if you remove post's support for > > sending NULs then dist won't be able to send the old email verbatim. > > Yes, but isn't that required by RFC 5322? Yes, RFC 5322 says do not send NULs. > I don't object to violating it in this case, so I'm fine with whatever > we can agree on. I was confused by your ‘Agreed’ above which agreed dist should be able to send and post NULs. -- Cheers, Ralph.
Re: mysterious c0 80
Michael wrote: > David Levine wrote: > > > I'm thinking of removing the support in post(8) for sending > > NULs. Any disagreement? It's not a lot of code so could be > > easily restored in the future if conditions change. > > Does that mean an error, or does that mean just skipping it? The old code relied on fprintf(3) and fputs(3), so it truncated at the first NUL. > I'm fine with skipping the NUL, but I'll live with the error; I'll > just have to fix my end :-) As Ken noted, it would be nice to understand the root cause. Ken wrote: + It is not clear to me that any of the OTHER nmh programs could + actually even receive a message with NUL in it, and plenty of other + programs fail if a message contains a NUL. Here's some messages + when I brought this up last year: In one of those messages, I noted that m_getfld and the MIME parser do handle NULs. I haven't tried inc(1). Ralph wrote: # But doesn't dist → send → post so if you remove post's support for # sending NULs then dist won't be able to send the old email verbatim. Yes, but isn't that required by RFC 5322? I don't object to violating it in this case, so I'm fine with whatever we can agree on. David
Re: mysterious c0 80
David Levine wrote: >> nmh shouldn't comp(1) a new email today with a NUL in the body, but it >> should be able to read and show(1) one. > I'm thinking of removing the support in post(8) for sending NULs. Any > disagreement? It's not a lot of code so could be easily restored in the > future if conditions change. Does that mean an error, or does that mean just skipping it? I'm fine with skipping the NUL, but I'll live with the error; I'll just have to fix my end :-) signature.asc Description: PGP signature
Re: mysterious c0 80
>> So while I agree post would fail with this hypothetical dist(1) of a >> message containing a NUL, it's not clear you could inc(1) such a >> message in the first place. > >Today or historically? Historically is a long time. I can't say that I know every single line of nmh and MH code since the dawn of time, but I've been in and out of it a lot. The assumption that you can represent email lines as C strings has been embedded in nmh and MH for as long as I can tell. Such emails don't work with nmh, and they probably don't work well with other MUAs either. Okay, fine, "don't work" is a bit of an oversimplification; it seems like those lines will be silently truncated at the NUL character, so they probably "work" fine and you just don't notice the NULs. --Ken
Re: mysterious c0 80
Hi Ken, > It would be nice to understand the root cause of the bug, though. Yes. > So while I agree post would fail with this hypothetical dist(1) of a > message containing a NUL, it's not clear you could inc(1) such a > message in the first place. Today or historically? Historically is a long time. -- Cheers, Ralph.
Re: mysterious c0 80
>> I'm thinking of removing the support in post(8) for sending NULs. Any >> disagreement? It's not a lot of code so could be easily restored in >> the future if conditions change. I think this makes sense, and it does seem to cause some kind of problem as reported in Cy's message. It would be nice to understand the root cause of the bug, though. >> > Now, how about dist(1) of that old email? I'd have thought it >> > should send the old email verbatim, NUL and all. If that causes a >> > bounce then the sender can MIME-forward instead with a single >> > message/rfc822 part. >> >> Agreed. > >But doesn't dist → send → post so if you remove post's support for >sending NULs then dist won't be able to send the old email verbatim. It is not clear to me that any of the OTHER nmh programs could actually even receive a message with NUL in it, and plenty of other programs fail if a message contains a NUL. Here's some messages when I brought this up last year: https://lists.nongnu.org/archive/html/nmh-workers/2023-02/msg00029.html https://lists.nongnu.org/archive/html/nmh-workers/2023-04/msg00031.html So while I agree post would fail with this hypothetical dist(1) of a message containing a NUL, it's not clear you could inc(1) such a message in the first place. --Ken
Re: mysterious c0 80
Hi David, > > nmh shouldn't comp(1) a new email today with a NUL in the body, but > > it should be able to read and show(1) one. > > I'm thinking of removing the support in post(8) for sending NULs. Any > disagreement? It's not a lot of code so could be easily restored in > the future if conditions change. > > > Now, how about dist(1) of that old email? I'd have thought it > > should send the old email verbatim, NUL and all. If that causes a > > bounce then the sender can MIME-forward instead with a single > > message/rfc822 part. > > Agreed. But doesn't dist → send → post so if you remove post's support for sending NULs then dist won't be able to send the old email verbatim. -- Cheers, Ralph.
Re: mysterious c0 80
Ralph wrote: > nmh shouldn't comp(1) a new email today with a NUL in the body, but it > should be able to read and show(1) one. I'm thinking of removing the support in post(8) for sending NULs. Any disagreement? It's not a lot of code so could be easily restored in the future if conditions change. > Now, how about dist(1) of that old email? I'd have thought it should > send the old email verbatim, NUL and all. If that causes a bounce > then the sender can MIME-forward instead with a single message/rfc822 > part. Agreed. David
Re: mysterious c0 80
Hi Ken, > According to RFC 5322, a NUL in a message body is not permitted. > From §3.5: > >body= (*(*998text CRLF) *998text) / obs-body > >text= %d1-9 /; Characters excluding CR >%d11 / ; and LF >%d12 / >%d14-127 obs-body is what allows the NUL. obs-body= *((*LF *CR *((%d0 / text) *LF *CR)) / CRLF) §4: Though these syntactic forms MUST NOT be generated according to the grammar in section 3, they MUST be accepted and parsed by a conformant receiver. nmh shouldn't comp(1) a new email today with a NUL in the body, but it should be able to read and show(1) one. Now, how about dist(1) of that old email? I'd have thought it should send the old email verbatim, NUL and all. If that causes a bounce then the sender can MIME-forward instead with a single message/rfc822 part. -- Cheers, Ralph.
Re: mysterious c0 80
Ken Hornstein wrote: > However, it seems like Mike's problem is NOT that; the last two bytes of > his draft file are 00 a0. Cy's bug report said that can happen anywhere, > though. ! > I know this change was to handle NUL bytes in outgoing messages, but I am > wondering if maybe we should reject such drafts? Seems like any message > actually sent with a NUL in it would be rather unfriendly. That might > break things for people using MH-E, though, and as we've seen before that > has a very long release cycle. yeah, it's not so much that it does not get releases, as people do not install them often. This break, if it has a good/clear error would get fixed, I think. signature.asc Description: PGP signature
Re: mysterious c0 80
>> ...draft file does NOT contain a '\n' as the last character? My >> memory is that for some strange reason Emacs like to default to doing >> that. I suspect we do not test for that. > >A POSIX text file is zero or more lines where a line is zero or more >bytes terminated by '\n'. I don't make the news, I just report it. My vague memory is that this is centered around Emacs having it's roots in a pre-UNIX operating system. However, it seems like Mike's problem is NOT that; the last two bytes of his draft file are 00 a0. Cy's bug report said that can happen anywhere, though. I know this change was to handle NUL bytes in outgoing messages, but I am wondering if maybe we should reject such drafts? Seems like any message actually sent with a NUL in it would be rather unfriendly. That might break things for people using MH-E, though, and as we've seen before that has a very long release cycle. According to RFC 5322, a NUL in a message body is not permitted. From §3.5: body= (*(*998text CRLF) *998text) / obs-body text= %d1-9 /; Characters excluding CR %d11 / ; and LF %d12 / %d14-127 Obviously if you are sending binary, you can (RFC 2045 explicitly disallows NUL when sending 8bit). I realize that I started this whole mess last time I asked about this. Sigh. --Ken
Re: mysterious c0 80
Hi Michael, > hexdump of this draft folder before I hit send: > > 520 2074 6f66 2072 6874 7461 0a2e 430a 756f > 530 646c 6220 2e65 0a0a 000a > > Note 00 0a. If one wrongly encodes NUL as two-byte UTF-8, intended for U+0080 to U+7fff, one gets c0 80. This is the bastard https://en.wikipedia.org/wiki/UTF-8#Modified_UTF-8 -- Cheers, Ralph.
Re: mysterious c0 80
Ralph Corderoy wrote: > Hi, > Ken wrote: >> ...draft file does NOT contain a '\n' as the last character? My >> memory is that for some strange reason Emacs like to default to doing >> that. I suspect we do not test for that. > A POSIX text file is zero or more lines where a line is zero or more > bytes terminated by '\n'. Sure. Why mh-e/emacs would wind up writing a nul in there is a mystery, but at least it's a mystery that I can track down to the right place. > We shouldn't pander to alien ‘text’ editors and their non-text files. :-) This email does not have a nul in it. Maybe it will after I sign it. signature.asc Description: PGP signature
Re: mysterious c0 80
Ken Hornstein wrote: > Stupid question time: is it possible this bug is only triggered if the > draft file does NOT contain a '\n' as the last character? My memory is > that for some strange reason Emacs like to default to doing that. I suspect > we do not test for that. Could be. hexdump of this draft folder before I hit send: 520 2074 6f66 2072 6874 7461 0a2e 430a 756f 530 646c 6220 2e65 0a0a 000a Note 00 0a. signature.asc Description: PGP signature
Re: mysterious c0 80
Hi, Ken wrote: > ...draft file does NOT contain a '\n' as the last character? My > memory is that for some strange reason Emacs like to default to doing > that. I suspect we do not test for that. A POSIX text file is zero or more lines where a line is zero or more bytes terminated by '\n'. We shouldn't pander to alien ‘text’ editors and their non-text files. MH-E would presumably ensure or warn if the configuration isn't suitable for Unix? -- Cheers, Ralph.
Re: mysterious c0 80
Ken Hornstein wrote: >> It certainly includes that commit above. >> I updated just last week before starting this thread actually. >> Looking at my outbox, I think it did start when I upgraded. >> >> I tried "git revert 8f897f65", but it results in a bunch of conflicts, which >> I decided not to investigate. > Could you just do a 'git checkout 8f897f65^' and try that? Done. Using that in this email. obiwan-[/sandel/src/nmh](2.6.6) mcr 10007 %mhparam -version mhparam -- nmh-1.8+dev 1.8-RC1-29-g82b04d31 built 2024-01-02 13:37:27 + on obiwan This starts to seem like it's a buffer overread and c0 80 just happens to be consistent garbage. signature.asc Description: PGP signature
Re: mysterious c0 80
Stupid question time: is it possible this bug is only triggered if the draft file does NOT contain a '\n' as the last character? My memory is that for some strange reason Emacs like to default to doing that. I suspect we do not test for that. --Ken
Re: mysterious c0 80
>It certainly includes that commit above. >I updated just last week before starting this thread actually. >Looking at my outbox, I think it did start when I upgraded. > >I tried "git revert 8f897f65", but it results in a bunch of conflicts, which >I decided not to investigate. Could you just do a 'git checkout 8f897f65^' and try that? --Ken
Re: mysterious c0 80
David Levine wrote: >> === >> All 101 tests passed >> (19 tests were not run) >> === > That's good. Though I'm surprised that 19 tests weren't run. That happens > when nmh isn't configured with some options. 19 seems high to me. could be. It would be some library test, as I basically just --prefix=/sandel >> I tried running "test/post/test-post-basic" manually, and it seemed to just >> install stuff to test/testdir/inst, but I couldn't tell if it actually ran a >> test. Do I need some arguments to the test? > No, the tests don't take arguments. Some require helper programs. To make > sure those are built, it's best to run a single test this way: > make check TESTS=test/post/test-post-basic make[1]: Entering directory '/sandel/src/nmh' PASS: test/post/test-post-basic = 1 test passed = make[1]: Leaving directory '/sandel/src/nmh' In pretty green. signature.asc Description: PGP signature
Re: mysterious c0 80
Michael wrote: > === > All 101 tests passed > (19 tests were not run) > === That's good. Though I'm surprised that 19 tests weren't run. That happens when nmh isn't configured with some options. 19 seems high to me. > I tried running "test/post/test-post-basic" manually, and it seemed to just > install stuff to test/testdir/inst, but I couldn't tell if it actually ran a > test. Do I need some arguments to the test? No, the tests don't take arguments. Some require helper programs. To make sure those are built, it's best to run a single test this way: make check TESTS=test/post/test-post-basic If the test succeeds, it exits with 0 status. And that test shouldn't produce any output if it succeeds. David
Re: mysterious c0 80
David Levine wrote: >> >> What version of nmh are you using? There was a recent bug reported >> >> by Cy Schubert that may be the cause of this: >> >> >> >> http://savannah.nongnu.org/bugs/?65002 >> > >> >Interesting. Can anyone replicate this bug? Michael, are you running >> >on FreeBSD or something else? >> >> I haven't tried yet; it was on my list to look at. > test/post/test-post-basic should catch it. I tried make check. === All 101 tests passed (19 tests were not run) === I tried running "test/post/test-post-basic" manually, and it seemed to just install stuff to test/testdir/inst, but I couldn't tell if it actually ran a test. Do I need some arguments to the test? signature.asc Description: PGP signature
Re: mysterious c0 80
David Levine wrote: > Interesting. Can anyone replicate this bug? Michael, are you running > on FreeBSD or something else? Linux "debian 12" (actually systemd-free devuan 4).
Re: mysterious c0 80
David Levine wrote: > Ken wrote: >> What version of nmh are you using? There was a recent bug reported >> by Cy Schubert that may be the cause of this: >> >> http://savannah.nongnu.org/bugs/?65002 savannah keeps saying that I didn't pick a group, even after logging and going to the nmh bug list. I can't seem to even copy the stupid summary with that commit id... 8f897f65. > Interesting. Can anyone replicate this bug? Michael, are you running > on FreeBSD or something else? I am presently running 1.8+dev, commit fefdea3fd1b29686418df45063854ed2e3880488 which is Dec.3 origin/master. (Your commit) It certainly includes that commit above. I updated just last week before starting this thread actually. Looking at my outbox, I think it did start when I upgraded. I tried "git revert 8f897f65", but it results in a bunch of conflicts, which I decided not to investigate. signature.asc Description: PGP signature
Re: mysterious c0 80
Ken wrote: > >> What version of nmh are you using? There was a recent bug reported > >> by Cy Schubert that may be the cause of this: > >> > >>http://savannah.nongnu.org/bugs/?65002 > > > >Interesting. Can anyone replicate this bug? Michael, are you running > >on FreeBSD or something else? > > I haven't tried yet; it was on my list to look at. test/post/test-post-basic should catch it. David
Re: mysterious c0 80
>> What version of nmh are you using? There was a recent bug reported >> by Cy Schubert that may be the cause of this: >> >> http://savannah.nongnu.org/bugs/?65002 > >Interesting. Can anyone replicate this bug? Michael, are you running >on FreeBSD or something else? I haven't tried yet; it was on my list to look at. --Ken
Re: mysterious c0 80
Ken wrote: > What version of nmh are you using? There was a recent bug reported > by Cy Schubert that may be the cause of this: > > http://savannah.nongnu.org/bugs/?65002 Interesting. Can anyone replicate this bug? Michael, are you running on FreeBSD or something else? David
Re: mysterious c0 80
>Ralph Corderoy wrote: > >> The email I'm replying to: > >> $ hd `mhpath .` | tail -3 1ac0 6e 73 74 65 61 64 2e 0a 0a 0a >> c0 80 c0 80 c0 80 |nstead..| 1ad0 c0 80 c0 80 c0 80 >> c0 80 c0 80 0a 0a || 1adc $ > >> So perhaps look at your draft-template files in your `mhparam >> path`. > >So it has nothing to with GPG creation. It shows up in my outbox. If >it were in my template files, then it would show up in my emacs buffer >and draft files, right? It does not. What version of nmh are you using? There was a recent bug reported by Cy Schubert that may be the cause of this: http://savannah.nongnu.org/bugs/?65002 --Ken
Re: mysterious c0 80
Ralph Corderoy wrote: >> buffer and draft files, right? > If they're not hiding it from you to be helpful. > Try > LC_ALL=C sed -n $'/[^\t -~]/l' foo bar ... > to see if there is any output for files foo, bar, ... And to confirm this works: obiwan-[~/Mail](2.6.6) mcr 10038 %LC_ALL=C sed -n $'/[^\t -~]/l' inbox/244 \300\200\300\200\300\200\300\200\300\200\300\200\300\200\300\200$ where 244 was a message from me. signature.asc Description: PGP signature
Re: mysterious c0 80
Ralph Corderoy wrote: >> If it were in my template files, then it would show up in my emacs >> buffer and draft files, right? > If they're not hiding it from you to be helpful. > Try > LC_ALL=C sed -n $'/[^\t -~]/l' foo bar ... > to see if there is any output for files foo, bar, ... I used "hexdump -C * | grep c0" actually. I tried your method too, and nothing in any of my components/template files. signature.asc Description: PGP signature
Re: mysterious c0 80
Hi Michael, > If it were in my template files, then it would show up in my emacs > buffer and draft files, right? If they're not hiding it from you to be helpful. Try LC_ALL=C sed -n $'/[^\t -~]/l' foo bar ... to see if there is any output for files foo, bar, ... -- Cheers, Ralph.
mysterious c0 80
Ralph Corderoy wrote: > The email I'm replying to: > $ hd `mhpath .` | tail -3 > 1ac0 6e 73 74 65 61 64 2e 0a 0a 0a c0 80 c0 80 c0 80 |nstead..| > 1ad0 c0 80 c0 80 c0 80 c0 80 c0 80 0a 0a || > 1adc > $ > So perhaps look at your draft-template files in your `mhparam path`. So it has nothing to with GPG creation. It shows up in my outbox. If it were in my template files, then it would show up in my emacs buffer and draft files, right? It does not.
