Re: bounce/forward not working due to CR at line end
Gaute Hope writes: > Kai Harries writes on november 17, 2017 19:16: >> I am thinking about filing a ticket against gmailier because storing the >> emails with CRLF on a *nix is at least surprising. > > Hi, > > this was actually fixed in commit 71ce2b2 where line-endings are > converted to \n if that is the OS line seperator. I am not modifying the > file in any other way, but I think the DOS bomb should hopefully be set > correctly already if on Linux [0]. > > https://github.com/gauteh/gmailieer/blob/master/lieer/local.py#L326 > > GMail API supplies the content of the message base64 encoded (the whole > message not just any MIME parts) with CRLF line endings. > > Anyway, I realized I never made a release with that commit (I just did) > so unless you were running 'master' you would have been missing out on > that. Note that messages that were already downloaded before this change will > not be converted by gmailieer, but you can do that youself. All gmailieer > cares about is the file names. > > [0] https://en.wikipedia.org/wiki/Byte_order_mark#UTF-8 > > - gaute Hi, many thanks. My issue is indeed solved with the new release. Kai ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Re: bounce/forward not working due to CR at line end
Kai Harries writes on november 17, 2017 19:16: I am thinking about filing a ticket against gmailier because storing the emails with CRLF on a *nix is at least surprising. Hi, this was actually fixed in commit 71ce2b2 where line-endings are converted to \n if that is the OS line seperator. I am not modifying the file in any other way, but I think the DOS bomb should hopefully be set correctly already if on Linux [0]. https://github.com/gauteh/gmailieer/blob/master/lieer/local.py#L326 GMail API supplies the content of the message base64 encoded (the whole message not just any MIME parts) with CRLF line endings. Anyway, I realized I never made a release with that commit (I just did) so unless you were running 'master' you would have been missing out on that. Note that messages that were already downloaded before this change will not be converted by gmailieer, but you can do that youself. All gmailieer cares about is the file names. [0] https://en.wikipedia.org/wiki/Byte_order_mark#UTF-8 - gaute ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Re: bounce/forward not working due to CR at line end
Kai Harries writes: > David Bremner writes: >> >> Be that as it may, apparently your system is using CRLF. > > I am using gmailier [1] to sync my maildir. For testing purpose I just > installed offlineimap and in contrast to gmailiier the mails on > disk have no CRLF. > > I am thinking about filing a ticket against gmailier because storing the > emails with CRLF on a *nix is at least surprising. I think that's a good idea. I'd like to understand whether there's a good reason to support CRLF line endings in maildirs, or if it's just an oversight. I talked a bit to the neomutt maintainer on IRC, and such line endings cause problems for some operations in neomutt as well. d ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Re: bounce/forward not working due to CR at line end
David Bremner writes: > Kai Harries writes: > >> David Bremner writes: >> >>> Kai Harries writes: >>> David Bremner writes: > During off-list discussion, Kai mentioned id:87a7zsm5ol@gmail.com as > a message that he cannot bounce (and also not forward?). For me this > forwards fine, but does not bounce because it has a corrupted Cc header, > and my MTA rejects it. That doesn't sound related to the original > problem report (nothing about line endings). Can you please evaluate the following on your system: (call-process "notmuch" nil t nil "show" "--format=raw" "id:87a7zsm5ol@gmail.com") If I do this on my system, then the text that is inserted into the buffer has ^M (CR) at the line endings. Is this expected? >>> >>> No ^M line endings for me. Can you check the file on disk? Maybe "od -a" >>> lacking a better idea. >> >> `od -a` does also show that the lines are ending with 'cr' and 'nl' >> this is correct according to RFC 2822 [1]. > > That's not so clear. RFC2822 is explicitely not about disk storage; see > the Scope paragraph of RFC2822 or RFC522. I think most peoples unix mail > delivery agents use line feeds, otherwise more people would be having > problems resending mail. See e.g. point 2 in > > > https://www.exim.org/exim-html-current/doc/html/spec_html/ch-message_processing.html > > Be that as it may, apparently your system is using CRLF. I am using gmailier [1] to sync my maildir. For testing purpose I just installed offlineimap and in contrast to gmailiier the mails on disk have no CRLF. >> According to this [2] Emacs normally converts line endings to only 'nl' >> (newline) when opening a DOS file. This works on my system, if I open the >> file from disk no carriage-return (^M) is shown. >> >> Not working is inserting the output of `notmuch show` into a buffer. If >> I do this, then the carriage-return (^M) is shown. > > Yes, notmuch-show-resend-message uses notmuch-show-view-raw-message, > which intentionally turns off any conversion > >> >> David, I assume your system is also a Linux, does >> >> notmuch show --format=raw id:1472041345236.7...@de.bosch.com | od -a >> >> also show 'cr' and 'nl' as line ending? > > No, I don't have carriage returns in my actual files on disk, and > consequently not in the output of notmuch show --format=raw. > > I suspect notmuch is mostly fine with CRLF line endings, due to mainly > passing everything through gmime. But in the case of notmuch show > --format=raw, this is not the case. > > I did confirm that adding CRLF line endings to an existing mail file > causes the symptoms you originally described. > > I'm still not sure the best place to fix this; I'm pretty sure upstream > emacs won't accept your change (or equivalent) to mime-to-mml. > > d I am thinking about filing a ticket against gmailier because storing the emails with CRLF on a *nix is at least surprising. [1] https://github.com/gauteh/gmailieer ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Re: bounce/forward not working due to CR at line end
Kai Harries writes: > David Bremner writes: > >> Kai Harries writes: >> >>> David Bremner writes: >>> During off-list discussion, Kai mentioned id:87a7zsm5ol@gmail.com as a message that he cannot bounce (and also not forward?). For me this forwards fine, but does not bounce because it has a corrupted Cc header, and my MTA rejects it. That doesn't sound related to the original problem report (nothing about line endings). >>> >>> Can you please evaluate the following on your system: >>> >>> (call-process "notmuch" nil t nil "show" "--format=raw" >>> "id:87a7zsm5ol@gmail.com") >>> >>> If I do this on my system, then the text that is inserted into the >>> buffer has ^M (CR) at the line endings. Is this expected? >> >> No ^M line endings for me. Can you check the file on disk? Maybe "od -a" >> lacking a better idea. > > `od -a` does also show that the lines are ending with 'cr' and 'nl' > this is correct according to RFC 2822 [1]. That's not so clear. RFC2822 is explicitely not about disk storage; see the Scope paragraph of RFC2822 or RFC522. I think most peoples unix mail delivery agents use line feeds, otherwise more people would be having problems resending mail. See e.g. point 2 in https://www.exim.org/exim-html-current/doc/html/spec_html/ch-message_processing.html Be that as it may, apparently your system is using CRLF. > > According to this [2] Emacs normally converts line endings to only 'nl' > (newline) when opening a DOS file. This works on my system, if I open the > file from disk no carriage-return (^M) is shown. > > Not working is inserting the output of `notmuch show` into a buffer. If > I do this, then the carriage-return (^M) is shown. Yes, notmuch-show-resend-message uses notmuch-show-view-raw-message, which intentionally turns off any conversion > > David, I assume your system is also a Linux, does > > notmuch show --format=raw id:1472041345236.7...@de.bosch.com | od -a > > also show 'cr' and 'nl' as line ending? No, I don't have carriage returns in my actual files on disk, and consequently not in the output of notmuch show --format=raw. I suspect notmuch is mostly fine with CRLF line endings, due to mainly passing everything through gmime. But in the case of notmuch show --format=raw, this is not the case. I did confirm that adding CRLF line endings to an existing mail file causes the symptoms you originally described. I'm still not sure the best place to fix this; I'm pretty sure upstream emacs won't accept your change (or equivalent) to mime-to-mml. d ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Re: bounce/forward not working due to CR at line end
David Bremner writes: > Kai Harries writes: > >> David Bremner writes: >> >>> During off-list discussion, Kai mentioned id:87a7zsm5ol@gmail.com as >>> a message that he cannot bounce (and also not forward?). For me this >>> forwards fine, but does not bounce because it has a corrupted Cc header, >>> and my MTA rejects it. That doesn't sound related to the original >>> problem report (nothing about line endings). >> >> Can you please evaluate the following on your system: >> >> (call-process "notmuch" nil t nil "show" "--format=raw" >> "id:87a7zsm5ol@gmail.com") >> >> If I do this on my system, then the text that is inserted into the >> buffer has ^M (CR) at the line endings. Is this expected? > > No ^M line endings for me. Can you check the file on disk? Maybe "od -a" > lacking a better idea. `od -a` does also show that the lines are ending with 'cr' and 'nl' this is correct according to RFC 2822 [1]. According to this [2] Emacs normally converts line endings to only 'nl' (newline) when opening a DOS file. This works on my system, if I open the file from disk no carriage-return (^M) is shown. Not working is inserting the output of `notmuch show` into a buffer. If I do this, then the carriage-return (^M) is shown. David, I assume your system is also a Linux, does notmuch show --format=raw id:1472041345236.7...@de.bosch.com | od -a also show 'cr' and 'nl' as line ending? If No, then my `notmuch show` behaves different. If Yes, then my Emacs behaves different when inserting text with DOS line endings into a buffer. [1] https://tools.ietf.org/html/rfc2822#section-2.2 [2] https://www.gnu.org/software/emacs/manual/html_mono/emacs.html#Coding-Systems ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Re: bounce/forward not working due to CR at line end
On Wed, Nov 15 2017, David Bremner wrote: > Kai Harries writes: > >> David Bremner writes: >> >>> During off-list discussion, Kai mentioned id:87a7zsm5ol@gmail.com as >>> a message that he cannot bounce (and also not forward?). For me this >>> forwards fine, but does not bounce because it has a corrupted Cc header, >>> and my MTA rejects it. That doesn't sound related to the original >>> problem report (nothing about line endings). >> >> Can you please evaluate the following on your system: >> >> (call-process "notmuch" nil t nil "show" "--format=raw" >> "id:87a7zsm5ol@gmail.com") >> >> If I do this on my system, then the text that is inserted into the >> buffer has ^M (CR) at the line endings. Is this expected? > > No ^M line endings for me. Can you check the file on disk? Maybe "od -a" > lacking a better idea. $ notmuch search --output=files id:87a7zsm5ol@gmail.com outputs path to one file in my system. the mail content is base64-encoded and there is no cr characters in my system (I used base64 -d 'edited-mail ' | tr '\r' '$' | less to look) As Kai sent the mail, his local copy may be encoded differently... Tomi (Hoo, I just wondered whether there is base64-decode-region in emacs -- there is a function exactly that name... had to M-x read-only-mode (where did M-x toggle-read-only disappear) in notmuch-raw-id:... buffer before trying that (i.e. in emacs mua, in show buffer, first pressed 'V' to see raw message contents.) PS2: it is possible this mail doesn't reach Kai, as gmail is somewhat picky from where it accepts email... we may get this fixed any century now, though. ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Re: bounce/forward not working due to CR at line end
Kai Harries writes: > David Bremner writes: > >> During off-list discussion, Kai mentioned id:87a7zsm5ol@gmail.com as >> a message that he cannot bounce (and also not forward?). For me this >> forwards fine, but does not bounce because it has a corrupted Cc header, >> and my MTA rejects it. That doesn't sound related to the original >> problem report (nothing about line endings). > > Can you please evaluate the following on your system: > > (call-process "notmuch" nil t nil "show" "--format=raw" > "id:87a7zsm5ol@gmail.com") > > If I do this on my system, then the text that is inserted into the > buffer has ^M (CR) at the line endings. Is this expected? No ^M line endings for me. Can you check the file on disk? Maybe "od -a" lacking a better idea. ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Re: bounce/forward not working due to CR at line end
David Bremner writes: > During off-list discussion, Kai mentioned id:87a7zsm5ol@gmail.com as > a message that he cannot bounce (and also not forward?). For me this > forwards fine, but does not bounce because it has a corrupted Cc header, > and my MTA rejects it. That doesn't sound related to the original > problem report (nothing about line endings). Can you please evaluate the following on your system: (call-process "notmuch" nil t nil "show" "--format=raw" "id:87a7zsm5ol@gmail.com") If I do this on my system, then the text that is inserted into the buffer has ^M (CR) at the line endings. Is this expected? In contrast, if I evaluate the following: (require 'subr-x) (find-file (string-trim (shell-command-to-string "notmuch search --output=files id:87a7zsm5ol@gmail.com"))) then a buffer with the message is opened but with clean line endings (no ^M). > I guess we still need a good test case [1], but my best guess is that > something about Kai's settings. Kai, if you have the source unpacked > (even if not built), try > > $ ./devel/try-emacs-mua -Q > > and see if the problem is duplicated there. Yes, bounce is also not working in this mode. I only evaluated (setq message-send-mail-function (lambda () t)) (notmuch-hello) from the startup page (I have not loaded my ~/.emacs). But still the bounce fails with: Search failed: " " as message in the minibuffer. > [1]: I tried both the copy of that message I already have and the copy > Kai sent me, after changing it's message-id ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Re: bounce/forward not working due to CR at line end
Kai Harries writes: > Hello, > > I am new to notmuch so excuse if the fault is on my end. For me > resending and forwarding of emails are not working if > message-forward-show-mml is enabled. The problems are CR at the line > endings. With the attached two patches I got it working for me. Maybe > someone more knowledgeable can see from it what the right solution would > be. During off-list discussion, Kai mentioned id:87a7zsm5ol@gmail.com as a message that he cannot bounce (and also not forward?). For me this forwards fine, but does not bounce because it has a corrupted Cc header, and my MTA rejects it. That doesn't sound related to the original problem report (nothing about line endings). I guess we still need a good test case [1], but my best guess is that something about Kai's settings. Kai, if you have the source unpacked (even if not built), try $ ./devel/try-emacs-mua -Q and see if the problem is duplicated there. [1]: I tried both the copy of that message I already have and the copy Kai sent me, after changing it's message-id ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
bounce/forward not working due to CR at line end
Hello, I am new to notmuch so excuse if the fault is on my end. For me resending and forwarding of emails are not working if message-forward-show-mml is enabled. The problems are CR at the line endings. With the attached two patches I got it working for me. Maybe someone more knowledgeable can see from it what the right solution would be. My configuration: notmuch 0.25.1 Emacs 25.3.1 on GNU/Linux (NixOS 17.09) --- a/mml.el 2017-11-09 21:11:26.542291084 +0100 +++ b/mml.el 2017-11-09 21:12:18.838306315 +0100 @@ -929,10 +929,11 @@ (defun mime-to-mml (&optional handles) "Translate the current buffer (which should be a message) into MML. If HANDLES is non-nil, use it instead reparsing the buffer." ;; First decode the head. + (save-excursion (icalendar--clean-up-line-endings)) (save-restriction (message-narrow-to-head) (let ((rfc2047-quote-decoded-words-containing-tspecials t)) (mail-decode-encoded-word-region (point-min) (point-max (unless handles --- a/notmuch-show.el 2017-11-09 21:17:05.791414308 +0100 +++ b/notmuch-show.el 2017-11-09 21:17:57.958437388 +0100 @@ -2011,20 +2011,21 @@ (defun notmuch-show-view-raw-message () "View the original source of the current message." (interactive) (let* ((id (notmuch-show-get-message-id)) (buf (get-buffer-create (concat "*notmuch-raw-" id "*"))) (inhibit-read-only t)) (switch-to-buffer buf) (erase-buffer) (let ((coding-system-for-read 'no-conversion)) (call-process notmuch-command nil t nil "show" "--format=raw" id)) +(save-excursion (icalendar--clean-up-line-endings)) (goto-char (point-min)) (set-buffer-modified-p nil) (setq buffer-read-only t) (view-buffer buf 'kill-buffer-if-not-modified))) (defun notmuch-show-resume-message () "Resume EDITING the current draft message." (interactive) (notmuch-draft-resume (notmuch-show-get-message-id))) Best regards Kai Harries ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
