Re: Message security; protected header fields

[Steffen Nurpmeso]

Werner Koch wrote in
 <875xvoza5j@jacob.g10code.de>:
 |Thanks for the summary.  I fully agree add these 2 cents:
 |
 |In particular using a fixed subject is not going to work in any real
 |business because you are not able to ignore mails.  For my part, I even
 |use a auto-responder to tell that mails with a three-dot subject are
 |ignored.
 |
 |There is a simpler method than autocrypt to initially convey a key.  If
 |you can't MIME-attach it, include your key in the signature (gpg's
 |--include-key-block).  This is what S/MIME does for decades.  If you
 |don't have the recipient's key (i.e. no Web Key Directory), signing the
 |first message allows the recipient to reply encrypted.

That is the real thing!  That should be made a standard feature in
PGP, only the plain key without any Web of Trust noise, it is so
easy for S/MIME, even my one can simply use *SSL library provided
standard interfaces to take that and save it somewhere.
(And, to me, a real DNSSEC-secured DNS entry that can easily be
grasped by anyone, like the DKIM TXT record.)

--steffen
|
|Der Kragenbaer,The moon bear,
|der holt sich munter   he cheerfully and one by one
|einen nach dem anderen runter  wa.ks himself off
|(By Robert Gernhardt)

Re: Message security; protected header fields

[Steffen Nurpmeso]

Derek Martin wrote in
 <20240429203624.ge19...@bladeshadow.org>:
 |On Fri, Apr 26, 2024 at 06:45:57PM +0200, ilf wrote:
 |> The Autocrypt project worked on a draft for "Protected Headers for
 |> Cryptographic E-mail" [1]. That became the IETF draft "Header Protection \
 |> for
 |> Cryptographically Protected E-mail" [2]. draft-ietf-lamps-header-protect\
 |> ion
 |> is an Active Internet-Draft of the LAMPS WG, a "Proposed Standard" \
 |> and it is
 |> on track to become an RFC.
 |> 
 |> 1. https://github.com/autocrypt/protected-headers
 |> 2. https://datatracker.ietf.org/doc/draft-ietf-lamps-header-protection/
 |
 |Neat.  But this feature seems like a misfeature, making you
 |immediately susceptible to MITM.  It encourages users to forgo
 |establishing the trust of the keys so received.  What's to stop me
 |from sending you a forged e-mail that appears to be from someone else,
 |with an address and public key that I control?  Or, if I control
 |their/your local mail server, just replacing the key they gave with my
 |own?  Part of the point of using PGP/GPG is that you have taken the
 |time to verify the identity and key signature of the person presenting
 |the key, to prevent MITM attacks and similar.  This seems tailor-made
 |to encourage less savvy users (or really everyone) to do exactly the
 |wrong thing.

I *wholeheartly* agree!  S/MIME is so much better by concept!
This is why i like the new approach most PGP people now use, in
that they use a signed MIME multipart which includes the public
key as an attachment.

And, btw, i am in full support of the OpenPGP: header that can be
DKIM protected (plus by the "protected headers").  Unfortunately
that never made it to a standard.

  ...

For PGP there really should be better (ie: TXT-based; or like so)
SMIMEA/OPENPGKEY DNS entries, because what else one can have?
WKD, and HKPS.  I (and many others) use OpenPGP: and point via
https:// --- which is totally absurd given that the entire HTTPS
aka TLS community as it is of today uses CA pools that is based
upon commercial supermans.  No.

--steffen
|
|Der Kragenbaer,The moon bear,
|der holt sich munter   he cheerfully and one by one
|einen nach dem anderen runter  wa.ks himself off
|(By Robert Gernhardt)

Re: Message security; protected header fields

[Steffen Nurpmeso]

Derek Martin wrote in
 <20240425215214.gb19...@bladeshadow.org>:
 |On Thu, Apr 25, 2024 at 10:22:16PM +0200, Steffen Nurpmeso wrote:
 |> You talk to the wrong person given that other people added that
 |> mechanism and put it into practical use.
 |
 |Yeah unfortunately, as Kevin admitted, this feature was never
 |discussed here when it was implemented, so those who care about any of
 |that are having the conversation now.
 |
 |But that doesn't matter... I was specifically responding to YOUR
 |point, which was a logical fallacy.  You seemed to be making a point
 |that implied that if you aren't aware of clients that are negatively
 |impacted, then it doesn't matter.  That's bad logic.

So i really have no time for neither bikeshedding nor nitpicking,
but i quoted the standard, and there seems to be no
misinterpretation possible, for a simple brain like mine.
In email there is the main header, and that is anything that is
known; unless the agent is MIME capable, then it can interpret
multipart and "individual"-part sections.  By themselves, or
recursively descending/unfolding (decryption etc).
Which, btw, mutt implements in the only sensible way (with an
object tree, last i looked).
In *no* way, unless explicitly enabled, can a MIME header
influence the main message header.

I want to remark and always say that i think it is an unfortunate
decision to implement DKIM and all that mess the way it is done,
since our forefathers went the
envelope-into-new-envelope-then-seak approach for centuries or
even milleniums, and if it would be like that, then we would have
no problem with signatures breaking because of mailing-list
footers or adjusted subjects, and all that.

 |There's a difference between bugs caused by the implementer of the
 |receiver failing to correctly implement what is normal and expected,
 |and bugs caused by the implementer of the sender intentionally
 |implementing behavior that is outside what is normal and expected.
 |The former affects only the user of that software.  The latter affects
 |everyone else but the user of that software (at least potentially).
 |Responsible developers should strive to avoid both, but the latter
 |clearly has a more serious impact on more people, and exactly the
 |wrong people.  It is the whole reason the robustness principle exists.
 |Outlook is famous for this sort of thing, and it's one of the reasons
 |people hate Microsoft/Outlook.  
 |
 |> (And regarding this -- i think this works out fine.  Except for
 |> dumb clients like mine which do not get the thing and do not map
 |> those headers into the place of the main ones.  Yet.)
 |
 |You have no proof that this is true, among the hundreds if not
 |thousands of e-mail clients that exist.  That is the nature of

Yeah, i can plug one together with python modules or rust or what
in a minute that satisfies you.  Maybe.

 |undefined behavior.  There's nothing special about that term in the
 |context of ISO C or otherwise.  And since there isn't any sort of
 |official standard, it's probably rather unlikely that the majority of
 |clients will ever addopt this.  You can't call them dumb for not
 |implementing a standard that doesn't exist.
 |
 |> You can expect anything in it is what is said.  Of course, MUAs
 |> which do not understand maximally visualize those header lines in
 |> addition to the main ones (what mine does), or totally ignore them
 |> (what i expect graphical ones to do).  That is not "undefined
 |> behaviour".
 |
 |These are not the only possibilities, since again, the behavior is
 |undefined.  Headers could be incorrectly repeated (which could lead to

No one cares unless the signature breaks.
That, in general, is a GUI problem, again.  And everybody,
especially the browser based ones, will not do the right thing
upon it, whatever this is.

For example, some years ago, the ("some") GNU people were sending
out emails with two from:s i think it was (i maybe have saved it
somewhere), it was deliberately wrong (i told them).  This was
against the standard.
The email infrastructure delivered it :)

 |additional bugs, since that also is outside the spec), the "wrong"
 |version of the header could be the one the client uses.  Or the
 |structure you're using to hold the headers could overflow, causing
 |memory corruption and/or crashes.  Etc..  Those things might be
 |unlikely or impossible to happen in the absence of the undefined
 |behavior.  It's hard to be robust against problems you didn't forsee,
 |on account of them being well outside the norm.
 |
 |Undefined behavior is... undefined.  When the input data deviates from
 |what is expected, any variety of resulting bugs is LIKELY.

Interesting run, anyway.

--steffen
|
|Der Kragenbaer,The moon bear,
|der holt sich munter   he cheerfully and one by one
|einen nach dem anderen runter  wa.ks himself off
|(By Robert Gernhardt)

Re: Message security; protected header fields

[Steffen Nurpmeso]

Derek Martin wrote in
 <20240425190214.ga19...@bladeshadow.org>:
 |On Wed, Apr 24, 2024 at 01:07:10AM +0200, Steffen Nurpmeso wrote:
 |> Sirius via Mutt-dev wrote in
 |>|I would worry less about the users and more about breaking clients. The
 |>|method of "be liberal about what you receive and conservative about what
 |>|you send" is apt. Being standards-compliant is safe.
 |> 
 |> Ah, do you know about any one client which fails on headers it
 |> does not understand,
 |
 |Absence of evidence fallacy.  For this to be a non-concern (logically
 |speaking) you would need to prove that NO client has this problem.
 |Proving a negative is not always impossible, but given the number of
 |clients in existence, it's pretty impractical.

You talk to the wrong person given that other people added that
mechanism and put it into practical use.
(And regarding this -- i think this works out fine.  Except for
dumb clients like mine which do not get the thing and do not map
those headers into the place of the main ones.  Yet.)

 |Now, how that leaves you IS NOT with conclusive proof that you should
 |not do it this way... but rather a strong suggestion that IF you can
 |find a good alternative that doesn't have the same potential weakness
 |(nor other worse tradeoffs), choose that instead.
 |
 |But I'm repeating myself now.

Sure.  Anyhow it is plain that bugs are everywhere.
In January i have written a RFC 5322 parser and looked around
a bit (not mutt), and found not a single parser without problems.
Then since February DKIM, and here you are.
Also we have gained too many RFCs in the last one or two decades,
and every little one invents new rulesets, instead of building
upon either *822 or 2045, like the ones before them.  This
unfortunately even includes DKIM.

 |> ok i have reread 2045 and it says [...]
 |
 |Largely irrelevant, because of what it does NOT say...  For instance,
 |it does not describe what the behavior should be if standard RFC 822
 |headers appear BOTH in a mime header block AND in the actual message
 |headers.  This is what's known as undefined behavior.  Therein lies

You can expect anything in it is what is said.
Of course, MUAs which do not understand maximally visualize those
header lines in addition to the main ones (what mine does), or
totally ignore them (what i expect graphical ones to do).
That is not "undefined behaviour".

 |the path to non-interoperability--which Mutt intends (or, at least has
 |historically intended) to avoid.  I've already described how such a
 |problem might arise in a previous message.  Avoid forseeable
 |interoperability problems when possible.

You make problems when there are none.

 |Also, what the RFC does explicitly state is that headers in the MIME
 |header block that do not begin with "content-" "CAN HAVE NO MEANING"
 |[emph. mine], and "may be ignored."  It gives no indication that there
 |would be an exception to those conditions for headers that are
 |explicitly defined by RFC 822 (or its successors).  Therefore any
 |processing of them that you do is at best ambiguous, i.e. again,
 |undefined behavior, and at worst violates the spec (because processing
 |them gives them meaning that the spec says they can not have).

Not undefined behaviour.  In the sense of ISO C undefined at
least.  Their meaning is simply not defined in this context, so
you either ignore them or show them (in configurable parts), but
by no means would you treat them as a replacement to the main
header instances.  Where is the problem?

 |Which of course isn't to say that the standard can't be extended or
 |modified; but if you want to do that, then you should actually draft
 |the standard extension and get it approved, well before asking clients
 |whose central tenets include complying with standards (as Mutt's do)
 |to implement such extensions.  The reasons to do that are to establish
 |whether the relevant community even values the extension, and whether
 |better alternatives may exist or be found.

I think having a RFC which only defines that -- in the sense of
the Melnikov etc draft's i have posted, would be a good thing.
But these protected header duplicates have nothing to do with
autocrypt as such.
Unfortunately the S/MIME and PGP worlds seem to be enemies or
what, it would make very much sense to define this for both, and
say that in a signed message container header duplicates etc etc.
Imho, that is.

 |Derek D. Martinhttp://www.pizzashack.org/   GPG Key ID: 0xDFBEAD02
 |-=-=-=-=-
 |This message is posted from an invalid address.  Replying to it will \
 |result in
 |undeliverable mail due to spam prevention.  Sorry for the inconvenience.
 --End of <20240425190214.ga19...@bladeshadow.org>

--steffen
|
|Der Kragenbaer,The moon bear,
|der holt sich munter   he cheerfully and one by one
|einen nach dem anderen runter  wa.ks himself off
|(By Robert Gernhardt)

Re: Message security; protected header fields

[Steffen Nurpmeso]

Hello.

Apologizing for the very late reply..

Sirius via Mutt-dev wrote in
 :
 |In days of yore (Sat, 20 Apr 2024), Steffen Nurpmeso thus quoth: 
 |> Kurt Hackenberg wrote in
 |>|Agreed.
 |> 
 |> I do not, actually.  Especially since it already is actively used.
 |> The question always is "how do receivers act upon this", of
 |> course, and this especially means the graphical, even
 |> browser-based monsters which users cannot configure, more, which
 |> have uneducated -- or better say "unaware" -- user bases.
 |
 |I would worry less about the users and more about breaking clients. The
 |method of "be liberal about what you receive and conservative about what
 |you send" is apt. Being standards-compliant is safe.

Ah, do you know about any one client which fails on headers it
does not understand, instead of simply searching for the
separating empty line in between the header block and the
successive content?  I have not reread 2045-9, ... ok i have
reread 2045 and it says

  9.  Additional MIME Header Fields

 Future documents may elect to define additional MIME header fields
 for various purposes.  Any new header field that further describes
 the content of a message should begin with the string "Content-" to
 allow such fields which appear in a message header to be
 distinguished from ordinary RFC 822 message header fields.

This implies that ordinary header fields can be expected to occur,
unless someone who wants to read a standard supercorrect fails in
doing so.

 |> Those clients, and i do not even know how well GMail or Outlook
 |> (or Apple Mail) can deal with S/MIME or PGP signed or even
 ..
 |Outlook and Apple Mail are able to use S/MIME out of the box. GnuPG is
 |something you have to work on to get to a functioning state, but it is
 |possible. For GMail there is a browser-plugin called FlowCrypt that will
 |give you GnuPG encryption. For Apple Mail, using MacGPG will do the same.
 |Not sure about browser mail via icloud.com, but FlowCrypt should work
 |there too (I just have not tested it). Outlook and GnuPG I have not tried
 |to get working, but there is some GUI offering of GnuPG for Windows, so I
 |assume it is possible.

Interesting, thanks for the info.

 |>|I would like to hold off on this until the draft becomes an RFC, if \
 |>|it does.
 |
 |Agreed - though as Mutt can produce at least the Subject line this way
 |when signing messages, we can test different mail clients to see how they
 |behave. I will endeavour to test Mac Mail, Outlook (new) and web-UI for
 |Gmail, Outlook and Apple mail to see how they render it. I will also test
 |KMail.
 |
 |I do not envisage clients breaking, but never say never. Stranger things
 |have happened throughout history.
 |
 |-- 
 |Kind regards,
 |
 |/S
 --End of 

--steffen
|
|Der Kragenbaer,The moon bear,
|der holt sich munter   he cheerfully and one by one
|einen nach dem anderen runter  wa.ks himself off
|(By Robert Gernhardt)

Re: Message security; protected header fields

[Steffen Nurpmeso]

Steffen Nurpmeso wrote in
 <20240420230154.HauOMF4V@steffen%sdaoden.eu>:
 ...
 |But i thing we refer to different drafts now.  I think you are all
 |talking about draft-autocrypt-lamps-protected-headers-02, whereas
 ...

And i want to reiterate that i myself dislike autocrypt as yet one
another way to shit (sorry) masses of unused data into email
headers.  I still think that if you really want to communicate
with one securely then the normal thing is to send an email and
ask for it.  I mean hey, you want to have *encrypted
communication* with another human person, right.  Puh.
Other than that you can lookup keys.
The problem is solely that this automated fetching is shit (sorry)
as of today, except for WKD maybe, and those hkps which still
function, or not at all for S/MIME, that easily.  (And not by
default on German passports, not the one, not the other, and not
fetchable via German DNSSECured DNS records either.)

And all those DNS records which have been invented are the very
same brainfuckers (sorry), because no normal and mentally sane
person can use them, as they require specific DNS record
formatting that those web interfaces that the mentioned persona
has to use do not offer this, and, i guess, will never support.

Compare this with the intellectual penetration of reality that the
old good ones have proven to have, again, by looking at the DKIM
standard.  All you need is a TXT record, and almost everyone will
be able to place this.  DKIM is a good standard.  I have my heavy
doubts on most others.  But that is just me, of course.

I mean, what a pity.  Give me DNSSEC, give me RFC 7250 raw TLS
keys and DKIM certs and some better sort of SMIMEA and OPENPGP
through it, instead of also this .well-known trashbin and CA
certificate pools (get rid of the root server keys altogether
maybe, how about [1] instead, even if US does not like it?),
and more through it.

  [1] https://wander.science/paper/2017_Wander_Rootless_DNS.pdf

--steffen
|
|Der Kragenbaer,The moon bear,
|der holt sich munter   he cheerfully and one by one
|einen nach dem anderen runter  wa.ks himself off
|(By Robert Gernhardt)

Re: Message security; protected header fields

[Steffen Nurpmeso]

Steffen Nurpmeso wrote in
 <20240420191646.ZD-tN3eo@steffen%sdaoden.eu>:
 |Kurt Hackenberg wrote in
 | :
 ||On Fri, Apr 19, 2024 at 03:41:40PM -0400, Derek Martin wrote:
 ||>On Fri, Apr 19, 2024 at 09:05:23AM -0700, Will Yardley wrote:
 ||>> It's odd to me that, since OpenPGP and S/MIME both support MIME
 ||>> encapsulation that the draft standard wouldn't use a separate MIME part
 ||>> to handle the protected headers vs. stuffing it at the top of the
 ||>> message body, which just seems kind of kludgy at best.
 ||>
 ||>This also seems like a perfectly cromulent approach, and again better
 ||>than the proposed one which puts nonstandard headers in a place where
 ||>no standard says they can be. [Or perhaps the correct wording would
 ||>be, "...which nonstandardly puts standard headers in a place where no
 ||>standard says they should be."]
 ||
 ||Are you both thinking of defining a new MIME type to hold only the 
 ||protected headers?  I thought of that.  It seems cleaner...I see no 
 ||mention of that in the draft we're talking about, not even to reject 
 ||it.  I suppose old mail readers wouldn't know what to do with that new 
 ||body part.  They might display it if it's a subtype of text.
 ||
 ||Turns out an earlier way to protect headers was added in S/MIME 3.1.  
 ||It puts the whole message, including the header section, in a body part 
 ||of type message/rfc822, and wraps a crypto body part around that.  So 
 ||the message contains a copy of itself, sort of.
 |
 |There is also draft-melnikov-smime-header-signing(-02 i have) on
 |top of this.

And RFC 7508.  (At around the very same time..., well.)

 ||The draft calls that mechanism "wrapped".  The draft wants to replace 
 ||that with this header-stuffing thing because "legacy" mail readers are 
 ||sometimes confused by the wrapped message.  The draft says 
 ||header-stuffing is less likely to confuse mail readers that never heard 
 ||of either scheme.
 |
 |Regarding the topic itself, i see those h"eaders duplicated into
 |the signature-protected MIME header"s more and more often, it
 |seems some mailers can this produce regulary for some time.
 |The MUA i maintain needs at least two more years to get there, but
 |all in all i would wish that this mechanism of header duplication
 |would become a RFC by itself, maybe someone should talk with
 |Daniel Kahn Gillmor, i think it was, to split that part out.
 |(I have not kept the draft locally after reading it.)
 |
 ||>MIME header blocks are for MIME-specific metadata; even if no mail
 ||>clients actually break due to this, it still feels gross.
 ||
 ||Agreed.
 |
 |I do not, actually.  Especially since it already is actively used.

..and because it most likely is then placed in multipart/
"container" MIME parts.  I just got one.  

  [-- #1.1 96/5400 --]
  Content-Language: en-US
  Content-Type: multipart/signed; micalg=pgp-sha256; 
protocol="application/pgp-signature"; 
boundary="HM80rgQvS20JTDutBozx04k9"

  [-- #1.1.1 69/4207 --]
  Content-Type: multipart/mixed; 
boundary="eXGghbDaVc6gdEfE02LEf8h1"; protected-headers="v1"
  From:  <@...org>
  To: @org
  Message-ID: 
  Subject: Re: ..ing
  References: <...>
  In-Reply-To: <...>

 |The question always is "how do receivers act upon this", of
 |course, and this especially means the graphical, even
 |browser-based monsters which users cannot configure, more, which
 |have uneducated -- or better say "unaware" -- user bases.

 |Those clients, and i do not even know how well GMail or Outlook
 |(or Apple Mail) can deal with S/MIME or PGP signed or even
 |encrypted (hmm..) email,  would have to take and treat the secured
 |headers as the real ones.  But, quite the opposite, you hear
 |statements of participants on user complaints like "i cannot
 |[click-]open your attachment" and such for PGP etc detached
 |signatures etc etc etc.
 |(You also hear "please use > for quoting, my mailer does not
 |understand your |", even though the leading whitespace was the
 |very first quotation method ever used (in a RFC), and is still
 |standardized in POSIX .. whatever.)
 |
 ||I would like to hold off on this until the draft becomes an RFC, if \
 ||it does.
 | --End of 

But i thing we refer to different drafts now.  I think you are all
talking about draft-autocrypt-lamps-protected-headers-02, whereas
i was at draft-ietf-lamps-header-protection-20.txt, and i find
that terribly and needlessly excessive.  Note it also talks about
a future deprecation of any non-protected messages, which i find
too anticipatory, and needlessly so, too.

  #?0|kent:rfc$ wc -l draft-autocrypt-lamps-protected-headers-02.txt
  3864 draft-autocrypt-lamps-protected-headers-02.txt
  #?0|kent:rfc$ wc -l dr

Re: Message security; protected header fields

[Steffen Nurpmeso]

Kurt Hackenberg wrote in
 :
 |On Fri, Apr 19, 2024 at 03:41:40PM -0400, Derek Martin wrote:
 |>On Fri, Apr 19, 2024 at 09:05:23AM -0700, Will Yardley wrote:
 |>> It's odd to me that, since OpenPGP and S/MIME both support MIME
 |>> encapsulation that the draft standard wouldn't use a separate MIME part
 |>> to handle the protected headers vs. stuffing it at the top of the
 |>> message body, which just seems kind of kludgy at best.
 |>
 |>This also seems like a perfectly cromulent approach, and again better
 |>than the proposed one which puts nonstandard headers in a place where
 |>no standard says they can be. [Or perhaps the correct wording would
 |>be, "...which nonstandardly puts standard headers in a place where no
 |>standard says they should be."]
 |
 |Are you both thinking of defining a new MIME type to hold only the 
 |protected headers?  I thought of that.  It seems cleaner...I see no 
 |mention of that in the draft we're talking about, not even to reject 
 |it.  I suppose old mail readers wouldn't know what to do with that new 
 |body part.  They might display it if it's a subtype of text.
 |
 |Turns out an earlier way to protect headers was added in S/MIME 3.1.  
 |It puts the whole message, including the header section, in a body part 
 |of type message/rfc822, and wraps a crypto body part around that.  So 
 |the message contains a copy of itself, sort of.

There is also draft-melnikov-smime-header-signing(-02 i have) on
top of this.

 |The draft calls that mechanism "wrapped".  The draft wants to replace 
 |that with this header-stuffing thing because "legacy" mail readers are 
 |sometimes confused by the wrapped message.  The draft says 
 |header-stuffing is less likely to confuse mail readers that never heard 
 |of either scheme.

Regarding the topic itself, i see those h"eaders duplicated into
the signature-protected MIME header"s more and more often, it
seems some mailers can this produce regulary for some time.
The MUA i maintain needs at least two more years to get there, but
all in all i would wish that this mechanism of header duplication
would become a RFC by itself, maybe someone should talk with
Daniel Kahn Gillmor, i think it was, to split that part out.
(I have not kept the draft locally after reading it.)

 |>MIME header blocks are for MIME-specific metadata; even if no mail
 |>clients actually break due to this, it still feels gross.
 |
 |Agreed.

I do not, actually.  Especially since it already is actively used.
The question always is "how do receivers act upon this", of
course, and this especially means the graphical, even
browser-based monsters which users cannot configure, more, which
have uneducated -- or better say "unaware" -- user bases.

Those clients, and i do not even know how well GMail or Outlook
(or Apple Mail) can deal with S/MIME or PGP signed or even
encrypted (hmm..) email,  would have to take and treat the secured
headers as the real ones.  But, quite the opposite, you hear
statements of participants on user complaints like "i cannot
[click-]open your attachment" and such for PGP etc detached
signatures etc etc etc.
(You also hear "please use > for quoting, my mailer does not
understand your |", even though the leading whitespace was the
very first quotation method ever used (in a RFC), and is still
standardized in POSIX .. whatever.)

 |I would like to hold off on this until the draft becomes an RFC, if \
 |it does.
 --End of 

--steffen
|
|Der Kragenbaer,The moon bear,
|der holt sich munter   he cheerfully and one by one
|einen nach dem anderen runter  wa.ks himself off
|(By Robert Gernhardt)

Re: Message security; protected header fields

[Steffen Nurpmeso]

Derek Martin wrote in
 <20240419191717.ge2...@bladeshadow.org>:
 ...
 |Secondly, there is a standard mechanism for adding non-standard
 |headers to e-mail:  use the string "X-" before the thing, and add it

Not anymore since RFC 6648.

--steffen
|
|Der Kragenbaer,The moon bear,
|der holt sich munter   he cheerfully and one by one
|einen nach dem anderen runter  wa.ks himself off
|(By Robert Gernhardt)

Re: IMAP connection closed while Getting mailbox UIDVALIDITY

[Steffen Nurpmeso]

Kevin J. McCarthy wrote in
 :
 |On Mon, Apr 17, 2023, at 5:01 PM, Steffen Nurpmeso wrote:
 |> Kevin J. McCarthy wrote in
 |>  :
 |>
 |> Hi; just to drop in RFC 4551 (and 7162): support 64-bit
 |> UIDVALIDITY (but our IMAP is bad, i have forgotten about it and
 |> it surely is a flashbang).
 |
 |Thanks Steffen!
 |
 |Those added a 64-bit MODSEQ (which Mutt supports for condstore and \
 |qresync) but I don’t recall anything about a 64-bit uidvalidity in them.
 |
 |There might be a different extension for uidvalidity - I haven’t resea\
 |rched it, but if so I agree it wouldn’t be default on. 

A yes, RFC 9051 also uses 32-bit.

 |( to the list: I’m away from my computer the rest of this week so sorry \
 |for the poor formatting.)
 |
 |-Kevin
 --End of 

--steffen
|
|Der Kragenbaer,The moon bear,
|der holt sich munter   he cheerfully and one by one
|einen nach dem anderen runter  wa.ks himself off
|(By Robert Gernhardt)

Re: PLAIN auth mechanism fails with outlook.office365.com imap server

[Steffen Nurpmeso]

Steffen Nurpmeso wrote in
 <20230417193326.d_rw9%stef...@sdaoden.eu>:
 |Steffen Nurpmeso wrote in
 | <20230414004746.fn4a_%stef...@sdaoden.eu>:
 ||Ian Collier wrote in
 || :
 |||On Thu, Apr 13, 2023 at 05:05:31PM -0400, Craig Gallek wrote:
 |||> I've managed to get this to work with gmail:
 |||> https://gitlab.com/muttmua/mutt/-/blob/master/contrib/mutt_oauth2.py.RE\
 |||> A\
 |||> \
 |||> DME#L85
 |||
 |||I have used the mutt_oauth2.py script to authenticate against an institu\
 |||tional
 |||office365 account over IMAP (script is at URL above with .README removed\
 |||). \
 | ...
 ||| I
 |||changed exactly two things in the script: (a) the GPG identity, and (b):
 |||'client_id': '9e5f94bc-e8a4-4e73-b8be-63364c29d753'
 ...
 ||I can confirm that this one works, both IMAP and SMTP are
 ||possible, tenant=common!  However, they now forbid "devicecode"
 ||flow.  "auth" works.  ("redirect" not tried.  And tThis is all my
 ...
 |P.S.: after i changed the "tenant" of my own application ID to
 |common, i can access outlook via IMAP _and_ SMTP again.  Back in
 |last October it nonetheless worked with the tenant ID that the
 |application registration generated.

But i had to make yet another change to make my own script truly
work again.  Microsoft must have changed their software, because
one now must pass the "scope" around in all OAuth 2.0 requests,
otherwise you get only an access token, but the refresh_token is
missing.  (We update the configuration and take what they give
us.  They actively strip "offline_access" btw.)
This is one more divertion from the standard RFC 6749 that they
produced themselves.  And back in last October it was unnecessary.
As i am out of bandwidth i was unable to verify that Google and
Yandex still work with this change being implemented.

--steffen
|
|Der Kragenbaer,The moon bear,
|der holt sich munter   he cheerfully and one by one
|einen nach dem anderen runter  wa.ks himself off
|(By Robert Gernhardt)

Re: IMAP connection closed while Getting mailbox UIDVALIDITY

[Steffen Nurpmeso]

Kevin J. McCarthy wrote in
 :
 |On Mon, Apr 17, 2023 at 03:26:19PM +0200, Andrej Mikus wrote:
 |>[2023-04-17 09:23:48] 4< * OK [UIDVALIDITY 63817286416513] UIDs valid
 |>[2023-04-17 09:23:48] Getting mailbox UIDVALIDITY
 |>[2023-04-17 09:23:48] Closing connection to mail.slovensko.sk...
 |>[2023-04-17 09:23:48] 4> a0005 LOGOUT
 |>
 |>Is the number returned by server too big? Thunderbird has no issues.
 |
 |Yes, UIDVALIDITY is defined as a 32-bit value in RFC3501 (sec 2.3.1.1 
 |and sec 9, where nz-number is defined as a non-zero 32-bit integer).
 |
 |When Mutt tries to parse the number, it fails and so aborts the 
 |connection.
 |
 |If possible, I would suggest contacting the server owner to and seeing 
 |if a bug can be filed with the IMAP server developer.

Hi; just to drop in RFC 4551 (and 7162): support 64-bit
UIDVALIDITY (but our IMAP is bad, i have forgotten about it and
it surely is a flashbang).

--steffen
|
|Der Kragenbaer,The moon bear,
|der holt sich munter   he cheerfully and one by one
|einen nach dem anderen runter  wa.ks himself off
|(By Robert Gernhardt)

Re: PLAIN auth mechanism fails with outlook.office365.com imap server

[Steffen Nurpmeso]

Steffen Nurpmeso wrote in
 <20230414004746.fn4a_%stef...@sdaoden.eu>:
 |Ian Collier wrote in
 | :
 ||On Thu, Apr 13, 2023 at 05:05:31PM -0400, Craig Gallek wrote:
 ||> I've managed to get this to work with gmail:
 ||> https://gitlab.com/muttmua/mutt/-/blob/master/contrib/mutt_oauth2.py.REA\
 ||> \
 ||> DME#L85
 ||
 ||I have used the mutt_oauth2.py script to authenticate against an institu\
 ||tional
 ||office365 account over IMAP (script is at URL above with .README removed\
 ||). \
 ...
 || I
 ||changed exactly two things in the script: (a) the GPG identity, and (b):
 ||'client_id': '9e5f94bc-e8a4-4e73-b8be-63364c29d753'
 ||(that's nicked from a recent public version of Thunderbird, which I
 ||guess is not strictly kosher but it does work as long as you remember
 ||this when you see the authorisation message from Microsoft asking if
 ||Mozilla should be allowed access to your email.  The client secret is
 ||the empty string for this id.  It saves the faff of having to create
 ||an app registration and it allows the 'common' endpoints to work rather
 ||than needing your tenant ID).
 |
 |I can confirm that this one works, both IMAP and SMTP are
 |possible, tenant=common!  However, they now forbid "devicecode"
 |flow.  "auth" works.  ("redirect" not tried.  And tThis is all my
 ...

P.S.: after i changed the "tenant" of my own application ID to
common, i can access outlook via IMAP _and_ SMTP again.  Back in
last October it nonetheless worked with the tenant ID that the
application registration generated.

--steffen
|
|Der Kragenbaer,The moon bear,
|der holt sich munter   he cheerfully and one by one
|einen nach dem anderen runter  wa.ks himself off
|(By Robert Gernhardt)

Re: PLAIN auth mechanism fails with outlook.office365.com imap server

[Steffen Nurpmeso]

Ian Collier wrote in
 :
 |On Thu, Apr 13, 2023 at 05:05:31PM -0400, Craig Gallek wrote:
 |> I've managed to get this to work with gmail:
 |> https://gitlab.com/muttmua/mutt/-/blob/master/contrib/mutt_oauth2.py.REA\
 |> DME#L85
 |
 |I have used the mutt_oauth2.py script to authenticate against an institu\
 |tional
 |office365 account over IMAP (script is at URL above with .README removed). \
 | I
 |changed exactly two things in the script: (a) the GPG identity, and (b):
 |'client_id': '9e5f94bc-e8a4-4e73-b8be-63364c29d753'
 |(that's nicked from a recent public version of Thunderbird, which I
 |guess is not strictly kosher but it does work as long as you remember
 |this when you see the authorisation message from Microsoft asking if
 |Mozilla should be allowed access to your email.  The client secret is
 |the empty string for this id.  It saves the faff of having to create
 |an app registration and it allows the 'common' endpoints to work rather
 |than needing your tenant ID).

I can confirm that this one works, both IMAP and SMTP are
possible, tenant=common!  However, they now forbid "devicecode"
flow.  "auth" works.  ("redirect" not tried.  And tThis is all my
script thing.)  Interestingly there is no refresh_token no more!!


P.S.: "my thing", because i use that not the mutt contrib/ script
for "my MUA" not mutt is
  https://git.sdaoden.eu/browse/s-toolbox.git/plain/oauth-helper.py
ie
  curl -u moon:mars --basic -O 
https://git.sdaoden.eu/browse/s-toolbox.git/plain/oauth-helper.py

and config file (-R)

  
authorize_endpoint=https://login.microsoftonline.com/common/oauth2/v2.0/authorize
  
devicecode_endpoint=https://login.microsoftonline.com/common/oauth2/v2.0/devicecode
  token_endpoint=https://login.microsoftonline.com/common/oauth2/v2.0/token
  redirect_uri=https://login.microsoftonline.com/common/oauth2/nativeclient
  tenant=common
  scope=https://outlook.office.com/IMAP.AccessAsUser.All 
https://outlook.office.com/POP.AccessAsUser.All 
https://outlook.office.com/SMTP.Send
  flow=auth
  access_token=
  client_id=9e5f94bc-e8a4-4e73-b8be-63364c29d753
  login_hint=yourm...@outlook.com
  timeout=
  timestamp=

--steffen
|
|Der Kragenbaer,The moon bear,
|der holt sich munter   he cheerfully and one by one
|einen nach dem anderen runter  wa.ks himself off
|(By Robert Gernhardt)

Re: PLAIN auth mechanism fails with outlook.office365.com imap server

[Steffen Nurpmeso]

Craig Gallek wrote in
 :
 |On 2023-04-13 16:58, Sébastien Hinderer wrote:
 |> Ian Collier (2023/04/13 17:11 +0100):
 |>> On Thu, Apr 13, 2023 at 01:59:56PM +0200, Sébastien Hinderer wrote:
 |>>> I am unfortunate enough to have to use an Office365 e-mail account.
 |>> 
 |>>> I don't know how to proceed from here. I really would like PLAIN \
 |>>> to work
 |>>> because the only other mechanism supported is XAUTH2, which feels \
 |>>> like a
 |>>> hell.
 |>> 
 |>> The server is lying to you that it supports AUTH=PLAIN.  Microsoft has 
 |>> disabled
 |>> basic authentication across most of their services and you must use 
 |>> XOAUTH2.
 |>> 
 |>> See e.g.:
 |>> https://www.bleepingcomputer.com/news/microsoft/microsoft-will-turn-off-\
 |>> exchange-online-basic-auth-in-january/
 |> 
 |> Many thanks. That's very unfortunate.
 |> 
 |> Is there a reliable way to use XOAUTH2 support in mutt, then, please?
 |
 |I've managed to get this to work with gmail:
 |https://gitlab.com/muttmua/mutt/-/blob/master/contrib/mutt_oauth2.py.REA\
 |DME#L85

My oauth2 script (somewhat like that one) failed a month ago with
my Microsoft test accounts, whereas it worked just nicely in last
October.  They must have messed around with the "tenant"s i think,
since another user that has a company (or university or what)
account still can use my thing (and the mutt one) to use
Microsoft.  (That is, last i tried IMAP worked, only SMTP they
somehow messed up.)

 |Never tried with office365, though...
 --End of 

--steffen
|
|Der Kragenbaer,The moon bear,
|der holt sich munter   he cheerfully and one by one
|einen nach dem anderen runter  wa.ks himself off
|(By Robert Gernhardt)

Re: [PATCH 1/2] base64val: Add support to decode base64 safe URL.

[Steffen Nurpmeso]

Werner Koch wrote in
 <87lejvwnvc@wheatstone.g10code.de>:
 |On Sat,  4 Mar 2023 18:33, Sebastian Andrzej Siewior said:
 |> In the base64 safe URL dictionary the characters '+' and '/' are
 |> replaced by '-' and '_'.
 |
 |FWIW, using '-' for general base64 encoding would be a bad idea because
 |it would allow to create those PGP or PEM armor lines ("-BEGIN foo")
 |and thus break message parsing.

This is base64url as of RFC 4648 (+ -> -, / -> _).

--steffen
|
|Der Kragenbaer,The moon bear,
|der holt sich munter   he cheerfully and one by one
|einen nach dem anderen runter  wa.ks himself off
|(By Robert Gernhardt)

Re: [PATCH] Change default message-id format.

[Steffen Nurpmeso]

Sebastian Andrzej Siewior wrote in
 <20230307_213502_4a7zc...@breakpoint.cc>:
 |The default message-id uses a timestamp and a few random bytes encoded
 |as base64 (with safe URL dictionary) as the user part. This is fine
 |already.
 |It would be beneficial for the human parser to have the timestamp
 |encoded as a date string so that the date can be easily recognised as
 |part of the message id. The message-id can be visible as the URL in a
 |thread overview (in a web indexer) and the text is set to the subject of
 |the email.

Oh how much i agree with that!  That ML-archive randomization is
a pain in the a.se imho.
Yet -- isn't this about the format that mutt had before a very
long thread some years ago?

--steffen
|
|Der Kragenbaer,The moon bear,
|der holt sich munter   he cheerfully and one by one
|einen nach dem anderen runter  wa.ks himself off
|(By Robert Gernhardt)

Re: Enabling $rfc2047_parameters by default

[Steffen Nurpmeso]

Kevin J. McCarthy wrote in
 :
 |I'm thinking about enabling $rfc2047_parameters by default, and would 
 |like to hear any counter-arguments against this.
 |
 |Here we are in 2022, yet I still occasionally receive tickets, or most 
 |recently even a merge request (!154), about this.  Obviously some mail 
 |clients are *still* improperly applying 2047 encoding.
 |
 |Making the situation worse, the config variable name isn't intuitive (to 
 |someone who isn't familiar with the relevant rfcs), so the user usually 
 |has no idea how to fix the problem.  Even the merge request author, who 
 |was skilled enough to hack the code, wasn't aware there was already a 
 |variable.
 |
 |To me, enabling the variable has low potential risk and would definitely 
 |save user frustration, but I'd like to hear if there are potential 
 |downsides to doing this.

The MUA i maintain does

  /* We do have a result, but some (elder) software (S-nail 

Re: Use From-address fqdn for Message-ID Generation

[Steffen Nurpmeso]

Kevin J. McCarthy wrote in
 :
 |On Mon, Nov 22, 2021 at 11:22:33AM -0600, Aaron Poffenberger wrote:
 |>I know this is related to Message-ID because if I go into muttrc(5)
 |>and put `set hostname="example.com"` and send again to
 |>u...@sbcglobal.net, the message goes through without issue.
 |
 |Well, setting $hostname can affect other things too.
 |
 |Without the patch, if you *just* change your $message_id_format to 
 |"<%z...@example.com>" you are saying it goes through.  Are you using 
 |$smtp_url or a local MTA?  Are there any other possible variables?
 |
 |The behavior you are describing is highly unusual, as the MTA is usually 
 |just looking at the EHLO and envelope addresses.
 |
 |>Also, as noted in my original, setting the fqdn of the Message-ID to
 |>the fqdn of the From address make sense. When I'm sending email from
 |>my laptop through gmail.com, the Message-ID picks-up the fqdn of my
 |>laptop rather than using gmail.com.
 |
 |The purpose of the Message-ID is to be unique.  Setting to the RHS to 
 |the local machine reduces the scope of randomness needed on the left 
 |side.  There's no need or sense behind using the from domain.

Fwiw it likely is SMTP related.  My MUA has a smtp-hostname
variable in addition to deal with this:

   smtp-hostname
 [Option][v15-compat] Normally Mailx uses the variable from[432] to
 derive the necessary ‘USER@HOST’ information in order to issue a
 ‘MAIL FROM:<>’ SMTP mta[476] command.  Setting smtp-hostname[566]
 can be used to use the ‘USER’ from the SMTP account (mta[476] or
 the user[605] variable chain) and the given ‘HOST’ (hostname[443]
 if the empty string is given, or the local hostname as a last
 resort).  This often allows using an address that is itself valid
 but hosted by a provider other than from which (in from[432]) the
 message is sent.  Setting this variable also influences generated
 ‘Message-ID:’ and ‘Content-ID:’ header fields.  If the [Option]al
 IDNA support is available (see idna-disable[444]) variable assign‐
 ment is aborted when a necessary conversion fails.

The problem came up with Yandex i think disliking me sending mail
via my Sourceforge account by then, .. yes, the examples show

 # Here is a pretty large one which does not allow sending mails
 # if there is a domain name mismatch on the SMTP protocol level,
 # which would bite us if the value of from does not match, e.g.,
 # for people who have a seforge project and want to speak
 # with the mailing list under their project account (in from),
 # still sending the message through their normal mail provider
 define XandeX {
   set folder=~/spool/XandeX inbox=+syste.mbox sent=+sent
   set from='Your Name '

   shortcut pop %:pop3s://pop.yaXXex.com
   shortcut imap %:imaps://imap.yaXXex.com

   set mta=smtps://USER:p...@smtp.yaxxex.com:465 \
 hostname=yaXXex.com smtp-hostname=
 }

--steffen
|
|Der Kragenbaer,The moon bear,
|der holt sich munter   he cheerfully and one by one
|einen nach dem anderen runter  wa.ks himself off
|(By Robert Gernhardt)

Re: [PATCH] Option to clear the screen on quit

[Steffen Nurpmeso]

Vincent Lefevre wrote in
 <20211027151035.ga14...@cventin.lip.ens-lyon.fr>:
 |On 2021-10-25 14:44:32 -0500, Derek Martin wrote:
 |> There is an ANSI escape sequence to tee data to your printer, sure...
 |> but it can not be used retroactively copy data that is on your
 |> terminal to the printer.  It just copies data that is currently being
 |> displayed (i.e. since the sequence was emitted) to the printer.
 |
 |I was wondering whether this could occur when switching to the
 |alternate screen. But it seems that this is not the case, at least
 |not with Xterm's logging feature.
 |
 |So I assume that as long as the user doesn't use a virtual terminal
 |inside the real terminal, things are safe. Users of virtual terminals
 |(GNU Screen, etc.) should be careful, as older data are sent back to
 |the real terminal when switching a window, for instance. However, in
 |case of any issue, the real solution should be to ensure that the
 |printing feature is disabled.

Fwiw i implemented optional automatic clearance upon suspension (/
exit) when ca-mode (smcup..rmcup) is enabled for the mailer
i maintain, blindly trusting your words (i use the st terminal).
You and Oskari Pirhonen have a credit for this.  I thought it is
nice since the alternative screen cannot be reached via clear(1).
(Granted the MUA is not really ca-mode compatible.)

--steffen
|
|Der Kragenbaer,The moon bear,
|der holt sich munter   he cheerfully and one by one
|einen nach dem anderen runter  wa.ks himself off
|(By Robert Gernhardt)

Re: ctime, POSIX and ISO C

[Steffen Nurpmeso]

Vincent Lefevre wrote in
 <20211022104139.gb2...@cventin.lip.ens-lyon.fr>:
 |About the following commit:

(I have not seen this commit.)

 |commit 60ab5f117d813b6ea7bdd6dacc1a771ea63edc6d
 |Author: Kevin McCarthy 
 |Date:   2021-10-20 03:48:47 +0200
 |
 |Add internal mutt_ctime() implementation.
 |
 |ctime() is marked obsolescent in the POSIX guide, so we ought to stop
 |using it to ensure future portability.
 |
 |ctime() may be marked obsolescent in the POSIX guide, but it is still
 |specified by ISO C, and AFAIK, not marked obsolescent there, even in
 |the latest C2x draft N2176.
 |
 |Moreover, ISO C says: "It is equivalent to
 |
 |  asctime(localtime(timer))"
 |
 |So, if you want to replace ctime(), why no using just that?
 |... unless there is an issue with asctime(), in which case ctime()
 |would be affected as a consequence.

Wasn't this already here??  I want to point to the busybox
documentation as quoted, and note i *saw* that crash.
Don't mind the format please, this is old code.
Ciao.

char *
n_time_ctime(s64 secsepoch, struct tm const *localtime_or_nil){/* TODO err*/
   /* Problem is that secsepoch may be invalid for representation of ctime(3),
* which indeed is asctime(localtime(t)); musl libc says for asctime(3):
*ISO C requires us to use the above format string,
*even if it will not fit in the buffer. Thus asctime_r
*is _supposed_ to crash if the fields in tm are too large.
*We follow this behavior and crash "gracefully" to warn
*application developers that they may not be so lucky
*on other implementations (e.g. stack smashing..).
* So we need to do it on our own or the libc may kill us */
   static char buf[32]; /* TODO static buffer (-> datetime_to_format()) */

   s32 y, md, th, tm, ts;
   char const *wdn, *mn;
   struct tm const *tmp;
   NYD_IN;
   LCTA(FIELD_SIZEOF(struct time_current,tc_ctime) == sizeof(buf),
  "Buffers should have equal size");

   if((tmp = localtime_or_nil) == NIL){
  time_t t;

  t = (time_t)secsepoch;
jredo:
  if((tmp = localtime()) == NIL){
 /* TODO error log */
 t = 0;
 goto jredo;
  }
   }

   if(UNLIKELY((y = tmp->tm_year) < 0 || y >= /*S32_MAX*/ - 1900)){
  y = 1970;
  wdn = su_time_weekday_names_abbrev[su_TIME_WEEKDAY_THURSDAY];
  mn = su_time_month_names_abbrev[su_TIME_MONTH_JANUARY];
  md = 1;
  th = tm = ts = 0;
   }else{
  y += 1900;
  wdn = su_TIME_WEEKDAY_IS_VALID(tmp->tm_wday)
? su_time_weekday_names_abbrev[tmp->tm_wday] : n_qm;
  mn = su_TIME_MONTH_IS_VALID(tmp->tm_mon)
? su_time_month_names_abbrev[tmp->tm_mon] : n_qm;

  if((md = tmp->tm_mday) < 1 || md > 31)
 md = 1;

  if((th = tmp->tm_hour) < 0 || th > 23)
 th = 0;
  if((tm = tmp->tm_min) < 0 || tm > 59)
 tm = 0;
  if((ts = tmp->tm_sec) < 0 || ts > 60)
 ts = 0;
   }

   (void)snprintf(buf, sizeof buf, "%3s %3s%3d %.2d:%.2d:%.2d %d",
 wdn, mn, md, th, tm, ts, y);

   NYD_OU;
   return buf;
}

--steffen
|
|Der Kragenbaer,The moon bear,
|der holt sich munter   he cheerfully and one by one
|einen nach dem anderen runter  wa.ks himself off
|(By Robert Gernhardt)

Re: Change Message-ID generation to be more unique and leak less information

[Steffen Nurpmeso]

ilf wrote in
 :
 |Eric Wong:
 |> Without opening the above URLs, you can immediately tell it's 
 |> from April 6, 2016.
 |
 |Message-IDs are not supposed to be human-meaningful:

I am with Eric Wong here and agreed with almost everything he says
(except i hate git-send-email being standard now since i can do
the same with my MUA), this goes as far as with even the number of
random bytes (we use 5 except under a special condition, when we
use 8 -- these are "RFC4648-URL" encoded bytes, then).

 |> The "Message-ID:" field provides a unique message identifier that
 |> refers to a particular version of a particular message.  The
 |> uniqueness of the message identifier is guaranteed by the host that
 |> generates it (see below).  This message identifier is intended to be
 |> machine readable and not necessarily meaningful to humans.  A message
 |> identifier pertains to exactly one version of a particular message;
 |> subsequent revisions to the message each receive new message
 |> identifiers.
 |
 |https://tools.ietf.org/html/rfc5322#section-3.6.4
 |
 |If you want URLs to be human-meaningful, don't use Message-ID.
 |
 |I for one thing Mutt shouldn't invent its own, but just use a random 
 |UUID, like so many other MUAs.

I find it tremendously helpful and have started using this
date-included way of doing things years ago for exactly the given
reasoning.  For example i have found myself finding a related
thread more easily in an archive, too.
In fact i wondered why noone addressed this by then, but in the
end i am only an external listening here and the topic is mostly
about style.

--steffen
|
|Der Kragenbaer,The moon bear,
|der holt sich munter   he cheerfully and one by one
|einen nach dem anderen runter  wa.ks himself off
|(By Robert Gernhardt)

Re: [PATCH] Support for overriding permissions of saved files

[Steffen Nurpmeso]

Derek Martin wrote in
 <20200806234050.gb8...@bladeshadow.org>:
 |On Fri, Aug 07, 2020 at 12:56:34AM +0200, Vincent Lefevre wrote:
 |> On 2020-08-06 10:50:23 -0500, Derek Martin wrote:
 |>> On Wed, Jul 29, 2020 at 12:55:07PM -0500, Derek Martin wrote:
 |>>> On Tue, Jul 28, 2020 at 08:03:23PM +0200, sacham...@s0c4.net wrote:
 | The thread, and even older threads referenced there, is from 2007.
 | Since then, the security field have evolved - now we have SeLinux,
 | Apparmor and other techniques which are capable to provide even
 | better security than umask(077)
 |>>> 
 |>>> None of those changes affect this issue in any meaningful way.
 |>>> SELinux predates that thread by at least two years (longer, though it
 |>>> was not generally available to the public until ~2005).  The arguments
 |>>> made in those threads still stand, and I will not repeat them here.
 |>> 
 |>> And FWIW, here's a more precise and detailed description I posted MUCH
 |>> more recently than 2007, which explains why this is a bad idea.
 |>> Everything here remains true, regardless of any evolution you think
 |>> has happened in the security world.
 |>> 
 |>> https://www.mail-archive.com/mutt-users@mutt.org/msg49810.html
 ...
 |Are you serious, Vincent?  I'm pretty sure you well know that this is
 |a horrible idea, clearly contrary to best security practices, that no
 ...
 |> On such a system using umask (007) for secondary ids seems logical
 |> and safe.
 |
 |No, it doesn't. Even if someone were to run Mutt on such a horribly
 |mismanaged system, its system security is generally suspect, so it is
 |even more important for Mutt to make sure the files are never saved
 |readable by anyone other than the user who created them.  Regardless,
 |Mutt should stick to its guns concerning maintaining the security of
 |its users' files.
 |
 |And remember, what we're trading here is the, what, 3 seconds it takes
 |for the user to type "chmod 644 *" (or similar) if they really want to
 |do this.  It's a small price to pay for the best insurance available
 |that no one is ever able to read your sensitive mail attachments
 |without you explicitly taking action to allow them to.  You'll never
 |be able to blame Mutt for this.

I take that bait. As an outsider i nonetheless think this is very
much of an over- reaction.  Users have an umask, and they usually
have it for a reason.  Unless they do not, and inherit the 022
that is very often set in global default /etc/profile's.  There
you have a culprit.

  #?0|kent:sec.arena$ umask
  027

So for me my MUA instance integrates in my usual environment, and
thus i want it to embed itself the way i want.  And want 027.
That is, in fact i want

  if [ ${UID} -eq 0 ]; then
 umask 0022
  elif [ -n "${FACL_OK}" ]; then
 umask 0027
  elif [ -f ./.umask ]; then
 umask "`cat ./.umask`" || umask 0027
  else
 echo >&2 '.profile: no FACL_OK, no ~/.umask: umask 0077'
 umask 0077
  fi

Therefore i have cheerfully implemented *umask* functionality once
a(n actually non-) user asked for it, for the MUA i maintain.  It
applies to files which are explicitly saved.  It does not apply to
temporary data, never.  I do not really see a security threat,
since by default it will be set to 0077, until the user explicitly
changes it.  What is wrong with that?

--steffen
|
|Der Kragenbaer,The moon bear,
|der holt sich munter   he cheerfully and one by one
|einen nach dem anderen runter  wa.ks himself off
|(By Robert Gernhardt)

Re: [PATCH] Support for overriding permissions of saved files

[Steffen Nurpmeso]

sacham...@s0c4.net wrote in
  <20200728180323.GE15493@lucid>:
 |The thread, and even older threads referenced there, is from 2007. \
 |Since then, the security field have evolved - now we have SeLinux, \
 |Apparmor and other techniques which are capable to provide even better \
 |security than umask(077) - I would say that ignoring shell's umask \
 |and enforcing our own may be actually harmful in above mentioned contexts.
 |
 |Please don't take me wrong - I fully agree on default of 0600 for all \
 |created files (undoubtely mboxes). Yet even in my simple scenario (mutt \
 |got its own user profile) I need to be able to process stored attachments \
 |by other users (separate user for libreoffice, separate user for image \
 |viewer,...). Manually calling chmod *each and every time* is even more \
 |security-error-prone than being able to set umask once for time being.
 |> 
 |We may even make this patch available only as by default disabled config\
 |ure option, I can imagine something like --enable-umask-override. How \
 |does that differ from applying my patch manually? Simply one does not \
 |have to trust "random patch from the internet", but supported option \
 |available for users who know what/why they want.

fwiw, for the MUA i maintain i implemented an umask override:

   umask
 For a safe-by-default policy the process file mode creation mask
 umask(2)[762] will be set to ‘0077’ on program startup after the
 resource files have been loaded, and unless this variable is set.
 By assigning this an empty value the active setting will not be
 changed, otherwise the given value will be made the new file mode
 creation mask.  Child processes inherit the file mode creation mask
 of their parent.

  ? xv var umask
  #nodelete,initial-value,positive-number
set umask=''

--steffen
|
|Der Kragenbaer,The moon bear,
|der holt sich munter   he cheerfully and one by one
|einen nach dem anderen runter  wa.ks himself off
|(By Robert Gernhardt)

Re: Add XOAUTH2 support?

[Steffen Nurpmeso]

Will Yardley wrote in
<20200605171958.gb70...@aura.veggiechinese.net>:
 |On Wed, Apr 10, 2019 at 04:57:50PM -0500, Alexander Perlis wrote:
 |> In case it helps inform a decision, here's the OAuth2 status of several
 |> IMAP providers:
 |> 
 |>   OAUTHBEARER: Google, Yahoo, ATT, Comcast, Sky
 |>  XOAUTH2 only: Microsoft, AOL, Yandex
 |>   Neither: Apple, Cox, Zoho, Mail.com, GMX, FastMail, 1&1
 |
 |Any more news about this? Just wrote mutt-users too, to see if anyone's
 |got any solutions for Office365 yet

Google works with XOAUTH2 also, it is what my MUA supports
(falsely labeled as oauthbearer until v14.10. series is released).

A terrible auth mech that introduces usage of JSON to the
protocols (where A NUL B NUL C NUL NUL style would be better than
good enough), as well as compulsion to interact via HTTP, which by
itself would not be that hard, if there would not be the much more
complicated HTTP/2 around, with the HTTP/3 that even uses
a different transport coming along.

--steffen
|
|Der Kragenbaer,The moon bear,
|der holt sich munter   he cheerfully and one by one
|einen nach dem anderen runter  wa.ks himself off
|(By Robert Gernhardt)

Re: [PATCH 1/3] Use LFSR113 PRNG for mutt's internal random needs

[Steffen Nurpmeso]

Ian Collier wrote in
<20200530212040.gk1301...@cs.ox.ac.uk>:
 |On Mon, May 25, 2020 at 04:24:41PM +0200, Oswald Buddenhagen wrote:
 |> why not do something proper and use getentropy() instead?
 |
 |It's been previously suggested on here that a mail client shouldn't
 |consume entropy from the system each time it starts, because other
 |more important processes may want it.

As a non-mathematician i happily disagree and claim you cannot
"consume" entropy of a random pool that is stirred and mixed and
which mixes, where all this mixing is indeed done via
cryptographically advanced algorithms, itself into another pool
that finally serves you (whether directly but especially when also
being served indirectly via such an algorithm, which i think is
what now is done by all; NetBSD has gained a(gain an even more
improvied) tremendous amount of work just a few days/fewest weeks
ago, for example).

For elder code the MUA i maintain also plays fair with the old
per-user entropy ~/.rnd (by default) as managed via OpenSSL, in
that, if every program that uses that stirs the pool and saves the
file again, as we do (as is or was documented that it should be
done like that i think), then every program startup and usage even
"increases entropy", or, how i would say it, increases
unpredictability of the actual entropy data.

I mean, you have regular interrupts and timers and scheduler time
slices and system calls of programs and network traffic and other
I/O events, and all that stirs the pool a bit.  But like i said,
i am not a mathematician, i always wondered how such attacks can
work out at all, yet some did.  But then again we saw bugs in the
past like that only some low bits of time counters were used and/
or that no mixing through the entire pool was done etc.  Iirc.
That is me who only reads such things with only one eye.
But for example i wondered already about twenty years ago why SSL
was not used for some things, and i now wonder even more why
everybody wants to use it for exactly the very same things.  As if
the world had changed.  Hm.

--steffen
|
|Der Kragenbaer,The moon bear,
|der holt sich munter   he cheerfully and one by one
|einen nach dem anderen runter  wa.ks himself off
|(By Robert Gernhardt)

Re: locking mechanism

[Steffen Nurpmeso]

Derek Martin wrote in
<20200512021313.ge20...@bladeshadow.org>:
 |On Mon, May 11, 2020 at 08:38:19PM +0200, Steffen Nurpmeso wrote:
 |> Vincent Lefevre wrote in
 |> <20200510204809.ga71...@zira.vinc17.org>:
 |>|Related to commit 7bd57bc3c24adf97f1f57bd6bb2fd18347f8cbbd, is
 |>|dotlocking still used nowadays?
 |> 
 |> I find yes.  Or at least last i looked, some MTAs aka MDA or
 |> whatever the right name is (LDA?  postfix
 |> (configurable), i think OpenBSDs mail.local (which saw heavy
 |> modifications lately though)) create these files, and then i think
 |> it seems sensible to embed in this locking strategy.
 |
 |IIRC most of the MDAs support multiple locking mechanisms, and at
 |least some of them use multiple mechanism simultaneously.  But dot
 |locking is slow, which probably doesn't matter on your home e-mail
 |server, but would matter on a mail server that handles millions of
 |messages a day...  And it may still be unreliable, depending on the

Minus that dotlocking applies to local MBOX delivery only.  And
minus comparison to the quite complicated local delivery pipe
chains that many of you seem to use.  And also minus the very long
backing store synchronizations that many use, the dotlock file may
even never reach backing store.
That reminds me that the mailer i maintain does not use fsync() on
mailboxes, also something i never thought could happen in real
life.

 |exact mix of things you have.  If you have one mechanism that's known
 |to work reliably across your whole mail system, you should use that,
 |and hope that it's not dot locking.
 |
 |>|Let's recall that dotlocking alone is not safe with some file systems,
 |>|such as NFS, since even if the client has an exclusive access to the
 |>|mailbox file, there is no guarantee that the synchronization will be
 |>|done before another program accesses the mailbox. In short, a working
 |>|fcntl locking is needed. But if it is available, then dotlocking is
 |>|useless. So, IMHO, if still supported, dotlocking should just be seen
 |>|as a fallback, and if mutt_dotlock cannot be installed with the right
 |>|permissions, the installation of Mutt should not fail.
 |> 
 |> I do not think so.  fcntl is "the new way"
 |
 |I don't think anything that's been available for > 20 years in the
 |world of computing can be considered "new" anymore... =8^)
 |
 |> and used anyway (for my MUA), but embedding into the scheme used by
 |> others is crucial, and as long as they use dotlock files for
 |> synchronization i will use it.
 |
 |All MDAs that are still supported use POSIX locking.  So dot locking
 |is superfluous and only slows you down.
 |
 |But let's not also forget that NFS is not the only problem.  There are
 |a wide array of file systems, and you can't be sure they implement
 |your favorite locking mechanism.   But, t's more likely that if POSIX
 |locking doesn't work on those, since it's one call and its interface
 |expects that it might not be implemented everywhere, you'll likely
 |just get EINVAL back, whereas a dot lock might be more likely to fail
 |silently (as it did under Linux).
 |
 |> NFS locking is "now" ok i have heard (here "now" is maybe almost two
 |> decades),
 |
 |That depends on what you mean:
 |
 |If you're on Linux, and:
 |
 |  - If you mean POSIX locking, yes.  It's worked reliably since at
 |least as far back as 2.4, and I believe earlier. (There was a
 |really old bug in lockd that broke it, but that's been fixed for
 |about 20 years or something)...  AFAIK there have been no reported
 |bugs with it since.
 |
 |  - If you mean dot locking...  In Linux it's been "OK" since 2.6.5,
 |if all of your clients and server are running Linux and NFSv3 or
 |greater.  Maybe. :)
 |
 |If you mean NOT Linux, or you have a mix of things...  file locking
 |over NFS is a horror show, in general, and if your environment is not
 |100% homogenous all bets are off.  There have been various bugs in the
 |NFS and/or locking implementations of pretty much every platform ever,
 |and they don't generally interoperate well.
 |
 |If you have NFS in the picture, and your site isn't 100% homogenous,
 |you'd better know exactly what's supported reliably by all of them,
 |and it's pretty much guaranteed to be POSIX locking, by now--it's kind
 |of the point of POSIX.  Otherwise, you may as well assume data
 |corruption is guaranteed.  Better to avoid accessing your mail spool
 |over NFS, or use Maildir.
 |
 |I'm not sure that it makes sense to remove dot locking from Mutt
 |entirely, but it probably does make sense to turn it off unless
 |explicitly requested.

--steffen
|
|Der Kragenbaer,The moon bear,
|der holt sich munter   he cheerfully and one by one
|einen nach dem anderen runter  wa.ks himself off
|(By Robert Gernhardt)

Re: locking mechanism

[Steffen Nurpmeso]

Vincent Lefevre wrote in
<20200512004344.ga175...@zira.vinc17.org>:
 |On 2020-05-11 20:38:19 +0200, Steffen Nurpmeso wrote:
 |> Vincent Lefevre wrote in
 |> <20200510204809.ga71...@zira.vinc17.org>:
 |>|Related to commit 7bd57bc3c24adf97f1f57bd6bb2fd18347f8cbbd, is
 |>|dotlocking still used nowadays?
 |> 
 |> I find yes.  Or at least last i looked, some MTAs aka MDA or
 |> whatever the right name is (LDA?  postfix
 |> (configurable), i think OpenBSDs mail.local (which saw heavy
 |> modifications lately though)) create these files,
 |
 |OK, but it seems that all of them support fcntl locking. Under Debian,
 |procmail, postfix and exim seem to use both by default. Concerning
 |procmail, it is provided with a setgid /usr/bin/lockfile. For postfix,
 |its documentation says:
 |
 |  Note: The dotlock method requires that the recipient UID or GID has |  
write access to the parent directory of the mailbox file.
 |
 |So it seems that dotlocking does not always work.

Well, for my MUA i propagated a >2 decade old piece of code to
a different and standalone context, made it SETUID, and happily
introduced a local root CVE.  That is doable.

 |> and then i think it seems sensible to embed in this locking
 |> strategy. That is what i use for my MUA, though on OpenBSD, MacOS
 |> and some others (Fedora at least) OPT_DOTLOCK is disabled, and on
 |> Debian the maintainer has made the dotlock helper a SETGID instead
 |> of a SETUID program, which should be enough for the plain Debian
 |> mailspool, however.
 |
 |Yes, because the group can write into it:
 |
 |drwxrwsr-x 2 root mail 4096 2020-05-10 22:22:08 /var/mail/
 |
 |Now, even if dotlocking isn't dropped, I don't think that it
 |should be mandatory. IMHO,
 |  * one should be able to specify the location of the dotlock program
 |at run time (so that non-root users could install a more recent
 |version of Mutt in their home directory);

A real problem.

  #?0|kent:src$ ll /usr/local/libexec/s-nail-dotlock
  -r-sr-xr-x 1 root users 50520 Aug 18  2019 /usr/local/libexec/s-nail-dotlock*
  #?0|kent:src$ ll /usr/local/bin/s-nail
  -rwxr-xr-x 1 steffen steffen 4883544 Apr 26 03:01 /usr/local/bin/s-nail*

 |  * one should be able to use dotlocking in an optional way, i.e.
 |just as an additional security, in addition to fcntl: if there
 |is no sufficient directory permission to create a lock file,
 |then ignore dotlocking and just rely on fcntl (which should be
 |fine for most users).

I mean fcntl is more than enough, sure.  It is just that i think
(my MUA allows setting "dotlock-disable") that if you live on
a system where your local MDA uses fcntl+dotlock then you should
embed in fcntl+dotlock.  The situation as you find it on multiple
systems is that the MDA is configured to use dotlock by default
(like i said, postfix release does by default unless my memory
fools me, OpenBSD mail.local also did i think), but user software,
and if only through packager decisions, just does not care.  Too
many captains, too few soldiers, maybe.

--steffen
|
|Der Kragenbaer,The moon bear,
|der holt sich munter   he cheerfully and one by one
|einen nach dem anderen runter  wa.ks himself off
|(By Robert Gernhardt)

Re: locking mechanism

[Steffen Nurpmeso]

Vincent Lefevre wrote in
<20200510204809.ga71...@zira.vinc17.org>:
 |Related to commit 7bd57bc3c24adf97f1f57bd6bb2fd18347f8cbbd, is
 |dotlocking still used nowadays?

I find yes.  Or at least last i looked, some MTAs aka MDA or
whatever the right name is (LDA?  postfix
(configurable), i think OpenBSDs mail.local (which saw heavy
modifications lately though)) create these files, and then i think
it seems sensible to embed in this locking strategy.  That is what
i use for my MUA, though on OpenBSD, MacOS and some others
(Fedora at least) OPT_DOTLOCK is disabled, and on Debian the
maintainer has made the dotlock helper a SETGID instead of
a SETUID program, which should be enough for the plain Debian
mailspool, however.

 |Even if it is, I find annoying that for an end user who probably
 |won't need it, Mutt's installation will fail if he does not provide
 |the --with-homespool option.
 |
 |Let's recall that dotlocking alone is not safe with some file systems,
 |such as NFS, since even if the client has an exclusive access to the
 |mailbox file, there is no guarantee that the synchronization will be
 |done before another program accesses the mailbox. In short, a working
 |fcntl locking is needed. But if it is available, then dotlocking is
 |useless. So, IMHO, if still supported, dotlocking should just be seen
 |as a fallback, and if mutt_dotlock cannot be installed with the right
 |permissions, the installation of Mutt should not fail.

I do not think so.  fcntl is "the new way" and used anyway (for my
MUA), but embedding into the scheme used by others is crucial, and
as long as they use dotlock files for synchronization i will use
it.  NFS locking is "now" ok i have heard (here "now" is maybe
almost two decades), but i got a lot of test errors on the OpenCSW
cluster i have access to in the last year or so, because the NFS
creates a hidden file for each file it creates (links), maybe, but
anyway the rm(1) we do on the test file is not reflected fast
enough for the hidden (dot) file, i got "wrong link count"
failures there.  That is, bugs are everywhere.

--steffen
|
|Der Kragenbaer,The moon bear,
|der holt sich munter   he cheerfully and one by one
|einen nach dem anderen runter  wa.ks himself off
|(By Robert Gernhardt)

Re: [PATCH] Change Message-ID generation to be more unique and leak less information

[Steffen Nurpmeso]

Kevin J. McCarthy wrote in
<20200423170108.gy17...@afu.lan>:
 |On Thu, Apr 23, 2020 at 01:40:06PM +0200, Vincent Lefevre wrote:
 |>> The memory allocated and returned by mutt_gen_base64_enc_rand() is being
 |>> leaked.
 |>
 |>I'm thinking that there are two meanings of "leak".
 |>This can be confusing. :-)
 |
 |*facepalm*.  I did use the exact same word "leak" again, didn't I?  :-)
 |
 |Sorry Remco.  Hopefully it's clear enough now that by "leak" we mean the 
 |memory is not being free()'ed properly.

Unfortunately i feel to need to annotate that for the code of the
MUA i maintain aka that i posted this is not true since we use
auto-reclaimed memory since 1978 (called "string dope" by then).
(Sorry for that.)

Ciao from Germany,

--steffen
|
|Der Kragenbaer,The moon bear,
|der holt sich munter   he cheerfully and one by one
|einen nach dem anderen runter  wa.ks himself off
|(By Robert Gernhardt)

Re: [PATCH] Change Message-ID generation to be more unique and leak less information

[Steffen Nurpmeso]

Vincent Lefevre wrote in
<20200421205425.gb838...@zira.vinc17.org>:
 |On 2020-04-20 19:57:23 +0200, Gero Treuner wrote:
 ...
 |> If we don't want to be deterministic, then I don't see a major advantage
 |> of hash functions compared to random data.
 |
 |In this case you need to make sure that such random data cannot be
 |guessed. This may be difficult without using entropy. Using entropy
 |each time Mutt is started would not be a good idea, in case a system
 |would run Mutt several times a second to send mail (e.g. personalized
 |mail to its users).

Ah, what a total mess that topic is (random stuff that is)!
Regarding this, the MUA i maintain does

   snprintf(rv, i, "%04d%02d%02d%02d%02d%02d.%s%c%s",
  tmp->tm_year + 1900, tmp->tm_mon + 1, tmp->tm_mday,
  tmp->tm_hour, tmp->tm_min, tmp->tm_sec,
  mx_random_create_cp(rl, ), sep, h);

where "rl" is either 5 or 8, dependent on whether we include
a "from" address in the ID or only a hostname.  random_create_cp()
returns a Base64URL encoded random number string.  So far so good.

We have several random possibilities, the fallback is the old
well-known ARC4 digest on home-brewed seed (with time slice
release, gettimeofday / clock_gettime), but we can also seed via
getrandom (sys/libc), /dev/urandom, or entirely leave all the
random stuff up to arc4random or OpenSSL.  Older versions of the
latter were driven to save ~/.rnd, which should improve random
security when used more often, possibly, maybe, i would think.
Newer ones of the latter are now driven to not auto-seed
internally, which could cause random queries to fail (!!), i do
not know what they thought when they changed this.  (I would have
made the failure optional at best.  Funny that Linux now stops
taking care at all.)

Uff.
Ciao from Germany,

--steffen
|
|Der Kragenbaer,The moon bear,
|der holt sich munter   he cheerfully and one by one
|einen nach dem anderen runter  wa.ks himself off
|(By Robert Gernhardt)

Re: Do you have any plans to make mutt compatible with Microsoft's Exchange Online after they discontinue support for IMAP?

[Steffen Nurpmeso]

Arnt Gulbrandsen wrote in
<22bf0336-e861-4049-aa37-4d7d6e9a6...@gulbrandsen.priv.no>:
 |On Tuesday 11 February 2020 18:34:51 CET, Gerry O'Brien wrote:
 |>   Do you have any plans to make mutt compatible with 
 |> Microsoft's Exchange Online after they discontinue support for 
 |> IMAP 
 |> https://developer.microsoft.com/en-us/office/blogs/end-of-support-for-ba\
 |> sic-authentication-access-to-exchange-online-apis-for-office-365-custome\
 |> rs.
 |
 |As you've no doubt grasped now, you misunderstood that blog posting 
 |totally. IMAP is staying.
 |
 |But perhaps you want to know the point of the change: Most IMAP clients 
 |want to keep the user's password around in persistent storage, so they can 
 |reconnect without asking for a password. Which means that any antisocial 
 |miscreants who gain access to a user's comptuer also gain access to a very 
 |valuable password, which can be used to gain access to the user's Micros\
 |oft 
 |account at any time the miscreant wishes.
 |
 |So what Mirosooft is saying is: Effective October 2020, IMAP clients will 
 |only be permitted to use an authentication procedure that doesn't require 
 |storing high-value passowrds on poorly secured computers.

^_^
As its late, all i want to add here is that it is very neat that
XOAUTH2/OAUTHBEARER replaces that one account password with three
individual tokens, of which one is temporary, the access token,
which you can use in place of the normal account password for
accessing a specific service.

Even the cumulative access to the three tokens stored on your
poorly secured computer will not reveal the account password.

Different to a Kerberos ticket you gained with kinit(1) one cannot
render the access token invalid (kdestroy(1)), nor can you
fine-tune anything when creating it.  I think GMail times out an
access token after 60 minutes.

Refreshing the access token requires a HTTPS connection, and the
two other tokens.  Alpine MUA implemented that HTTPS thing itself,
it requires HTTPS (with HTTP/2 to come for sure...) and JSON.

Now.  Why can't you just use plain text authentication with this
access token, why must it be XOAUTH2/OAUTHBEARER?
Why do you need an OAuth framework if you do support these two?
Just offer a diversified set of passwords, one master, one for
each service.  Where is the difference?
Why don't you either introduce service additions to refresh the
access token, or extend the XOAUTH2/OAUTHBEARER "JSON" (why JSON?
why not \0 terminated sub\0strings\0\0, like for others?  Why
JSON?) query/response with that possibility?
(password=PASS\0refresh=yes\0\0?)

P.S.: i store my passwords in a GPG encrypted ~/.netrc.gpg file
(with the extensions my MUA supports).  That is actually a symlink
into a directory that is mounted via encfs.  I know this is not
perfect, it should reside on a filesystem that sits on block-
level encrypted hardware, including encrypted swap, i will do that
this year, somewhen, there is enough space for a 50:50 split when
doing it.

--steffen
|
|Der Kragenbaer,The moon bear,
|der holt sich munter   he cheerfully and one by one
|einen nach dem anderen runter  wa.ks himself off
|(By Robert Gernhardt)

Re: Do you have any plans to make mutt compatible with Microsoft's Exchange Online after they discontinue support for IMAP?

[Steffen Nurpmeso]

Kevin J. McCarthy wrote in
<20200211183132.ge23...@afu.lan>:
 |On Tue, Feb 11, 2020 at 06:46:32PM +0100, Steffen Nurpmeso wrote:
 |>Gerry O'Brien wrote in
 |><69f8180c-40de-b7b5-880b-0f0e9f998...@scss.tcd.ie>:
 |>> Do you have any plans to make mutt compatible with Microsoft's 
 |>>Exchange Online after they discontinue support for IMAP 
 |>>https://developer.microsoft.com/en-us/office/blogs/end-of-support-for-ba\
 |>>\ 
 |>>sic-authentication-access-to-exchange-online-apis-for-office-365-custome\
 |>>\ 
 |>>rs.
 ...
 |>I wonder whether they won't adapt the new, standardized JMAP 
 |>protocol, which will finally cover much more than just "I"MAP, but 
 |>also calenders etc.  And if it is for interoperability, and maybe 
 |>the possibility to use open source software instead of putting 
 |>development power into their own stuff.  I think JMAP is the 
 |>future, and i would almost bet: also for Microsoft.
 |
 |I've been aware of JMAP for a while, but haven't taken a serious look at 
 |it yet.  Is usage becoming widespread outside of Fastmail yet?

I think Cyrus comes with JMAP support (for mail), i have seen
cyrus-jmap message IDs flying by on a FreeBSD list already.
I do not now whether this is already shipout, or release-candidate
or something, though.  But quite some time already.
(The standard for the mail side is RFC 8621 from August 2019.)

--steffen
|
|Der Kragenbaer,The moon bear,
|der holt sich munter   he cheerfully and one by one
|einen nach dem anderen runter  wa.ks himself off
|(By Robert Gernhardt)

Re: Do you have any plans to make mutt compatible with Microsoft's Exchange Online after they discontinue support for IMAP?

[Steffen Nurpmeso]

Gerry O'Brien wrote in
<69f8180c-40de-b7b5-880b-0f0e9f998...@scss.tcd.ie>:
 |Hello,
 |
 |   Do you have any plans to make mutt compatible with Microsoft's 
 |Exchange Online after they discontinue support for IMAP 
 |https://developer.microsoft.com/en-us/office/blogs/end-of-support-for-ba\
 |sic-authentication-access-to-exchange-online-apis-for-office-365-custome\
 |rs.

Don't they support XOAUTH2/OAUTHBEARER?
In my opinion, and by the way, that article uses the same brain
washing that steamrolls the truth to rise OAuth on a level where
it does not belong, like i have seen so many times before.
Just my one cent.  Let it be two if they support XOAUTH2/
OAUTHBEARER.  Please.

 |In particular, do you plan to implement MAPI over HTTP?

I wonder whether they won't adapt the new, standardized JMAP
protocol, which will finally cover much more than just "I"MAP, but
also calenders etc.  And if it is for interoperability, and maybe
the possibility to use open source software instead of putting
development power into their own stuff.  I think JMAP is the
future, and i would almost bet: also for Microsoft.

 |   Regards,
 |     Gerry

Ciao,

--steffen
|
|Der Kragenbaer,The moon bear,
|der holt sich munter   he cheerfully and one by one
|einen nach dem anderen runter  wa.ks himself off
|(By Robert Gernhardt)

Re: MTA behavior with respect to Bcc headers

[Steffen Nurpmeso]

Kevin J. McCarthy wrote in <20191031224233.gj13...@qinghai.lan>:
 |I'm looking for a bit of history and discussion of the correct behavior 
 |of MTAs with respect to Bcc headers.
 |
 |Ticket #185  asserts that 
 |Courier MTA doesn't remove the Bcc header when recipients are passed on 
 |the command line.  I'm currently not in a position to verify this 
 |behavior, so I'm assuming the ticket is correct.
 |
 |It looks like Mutt provides $write_bcc, which allows the removal of the 
 |Bcc header from the message.  However, I believe this will also remove 
 |it from the Fcc copy.
 |
 |The ticket asks if there is a way to turn off passing the recipients on 
 |the command line.  I'm wondering if this would be a generally useful 
 |option.

Aside the specific issue, i found adding this option was the only
way to make my little MUA work with OpenSMTPD 6.0.2p1, as that
joined the command line arguments onto all receivers given in the
message itself, resulting in duplicate deliveries.
While this possibly was a bug, i added *mta-no-receiver-arguments*
in October 2017 to address this issue.

 |Thanks for any advice or input!

Hope this helps.

--steffen
|
|Der Kragenbaer,The moon bear,
|der holt sich munter   he cheerfully and one by one
|einen nach dem anderen runter  wa.ks himself off
|(By Robert Gernhardt)

Re: $TMPDIR (was Re: Security: Mutt and mailcap rules)

[Steffen Nurpmeso]

Derek Martin wrote in <20190624233654.gb13...@bladeshadow.org>:
 |On Tue, Jun 25, 2019 at 12:45:02AM +0200, Steffen Nurpmeso wrote:
 |> Hmm, while i totally support the $TMPDIR environment variable, and
 |> personally dislike it a lot if i set it and someone simply does
 |> not adhere to it, and if its only for testing purposes.., it shall
 |> be remarked that OpenBSD "removed support for $TMPDIR" in the base
 |> system, as far as i know and recall.  Are they young?  Well, yes..
 |
 |I think you must be mistaken, because A) that would be insane, B)
 |would require a lot of pointless work to remove it from many shell
 |utilities (and all of the shells), and C) I see references to it
 |being supported, for example in ssh and ssh-agent, the mktemp man
 |page, the ksh (openbsd's default shell) man page, etc..
 |
 |I did see reference to support for $TMPDIR being removed from crontab
 |("because it's not useful in crontab"), which seems kind of idiotic to
 |me, as well as sendbug and newfs (why would newfs need $TMPDIR anyway?
 |Though it seems useful in sendbug)... but that does not amount to it
 |being removed from the core system.

Well.. i think it was more about it, .. and grepping in the tree
it is true for all the in-base daemons and such, except ssh.
From grepping i see it still being used in tmpname.c of stdio,
which surprised me a bit.  My memory was about a ML thread where
usage of TMPDIR was "loudly" discouraged.

 |That said, none of it matters in the context of Mutt, unless they went
 |out of their way to remove support for it in their port... but even
 |then you can always just get the source.

It would have mattered from what comes in via the operating system
libraries, and tools _possibly_ called from within mutt, but
definetely those called during configuration/build, which is all
i wanted to say.

--steffen
|
|Der Kragenbaer,The moon bear,
|der holt sich munter   he cheerfully and one by one
|einen nach dem anderen runter  wa.ks himself off
|(By Robert Gernhardt)

Re: safe_rename() and verifying the result of link(2)

[Steffen Nurpmeso]

Vincent Lefevre wrote in <20180831155541.ga30...@cventin.lip.ens-lyon.fr>:
 |On 2018-08-24 18:54:16 +0200, Steffen Nurpmeso wrote:
 |> Oh, wait!  This was false rememberance, i referred to a message
 |> from Casper Dik of Oracle who wrote on 2015-12-31
 |> 
 |>>/* Create a unique file. O_EXCL does not really work over NFS so \
 |>>we follow
 |>> * the following trick (inspired by S.R. van den Berg):
 |>> * - make a mostly unique filename and try to create it
 |>> * - link the unique filename to our target
 |>> * - get the link count of the target
 |>> * - unlink the mostly unique filename
 |>> * - if the link count was 2, then we are ok; else we've failed */
 |> 
 |>   The problem of not being able to create a file with O_EXCL was, \
 |>   I think,
 |>   fixed in NFSv3 (if not, certainly in NFSv4)
 |> 
 |>   Casper
 |> 
 |> so this was not about link but about O_EXCL.
 |
 |Yes, a well-known issue.

Well.  RFC 1094 from 1989, section 2.2.10, Create File, says

   The file "name" is created in the directory given by "dir".  The
   initial attributes of the new file are given by "attributes".  A
   reply "status" of NFS_OK indicates that the file was created, and
   reply "file" and reply "attributes" are its file handle and
   attributes.  Any other reply "status" means that the operation failed
   and no file was created.

   Notes:  This routine should pass an exclusive create flag, meaning
   "create the file only if it is not already there".

So well-known yes, but i have no context of knowledge.  Maybe
i should ask or look in some NFS sources.

 |> About a year later
 |> (2016-11-02) there was a pair of message in between Stèphane and
 |> Jörg about links via NFS, as in "IIRC there were issues with ln on
 |> NFS for instance." and "Could you please explain what you have in
 |> mind?  I would like to understand whether there really is a NFS
 |> problem or whether there is just a NFS bug in Linux", but nothing
 |> more than that.
 |
 |Perhaps it could just be the one already mentioned:
 |
 |  On NFS filesystems, the return code may be wrong in case the NFS server
 |  performs the link creation and dies before it can say so.  Use  stat(2)
 |  to find out if the link got created.
 |
 |which, I suppose, is impossible to solve.

..The network connection can break at any time.  RFC 1094 says

  Also, most server failures occur between operations, not
  between the receipt of an operation and the response.  Finally,

A nice encouragement.

  although actual server failures may be rare, in complex networks,
  failures of any network, router, or bridge may be indistinguishable
  from a server failure.

One could also go and look locally whether the file exists or not.
No, i mean, it is hard to say and depends on the context.  If you
are linking/xy over a hundreds files in order an additional stat
is a problem except (maybe) for the last file, if you are only
doing one it may very well provide confidence.  Of course it is
still racy, some other process may surely have had its rights to
overwrite/remove/move away the target in the meantime.  Maybe
doing a stat on the target directory in order to verify that the
server is still alive would be it.  Since: which software is
actually flexible enough to create a (real) batch if batching as
above would be possible?

A nice Sunday i wish.

--steffen
|
|Der Kragenbaer,The moon bear,
|der holt sich munter   he cheerfully and one by one
|einen nach dem anderen runter  wa.ks himself off
|(By Robert Gernhardt)

Re: safe_rename() and verifying the result of link(2)

[Steffen Nurpmeso]

Vincent Lefevre wrote in <20180824091156.ga20...@zira.vinc17.org>:
 |On 2018-08-22 15:35:16 +0200, Steffen Nurpmeso wrote:
 |> Vincent Lefevre wrote in <20180821230229.ga16...@zira.vinc17.org>:
 |>|* It is not clear whether this has ever been usuful (nothing in
 |>|  comment or in the commit log).
 |> 
 |> According to some statement on a list you are also subscribed to
 |> this NFS problem has been fixed "at the early 90s".
 |
 |This is quite surprising. Even the discussion at
 |
 |  https://www.experts-exchange.com/questions/10078625/atomic-locking-over-\
 |  NFS-with-link-2-stat-2.html
 |
 |in 1998 doesn't mention this problem (it just mentions the case where
 |the link(2) call failed though the link was created).
 |
 |Several old documents say not to use the return value of the link(2)
 |call, but do not say whether this is only due to the case where
 |link(2) fails but the link was created, or the other way round.

Oh, wait!  This was false rememberance, i referred to a message
from Casper Dik of Oracle who wrote on 2015-12-31

  >/* Create a unique file. O_EXCL does not really work over NFS so we follow
  > * the following trick (inspired by S.R. van den Berg):
  > * - make a mostly unique filename and try to create it
  > * - link the unique filename to our target
  > * - get the link count of the target
  > * - unlink the mostly unique filename
  > * - if the link count was 2, then we are ok; else we've failed */

  The problem of not being able to create a file with O_EXCL was, I think,
  fixed in NFSv3 (if not, certainly in NFSv4)

  Casper

so this was not about link but about O_EXCL.  About a year later
(2016-11-02) there was a pair of message in between Stèphane and
Jörg about links via NFS, as in "IIRC there were issues with ln on
NFS for instance." and "Could you please explain what you have in
mind?  I would like to understand whether there really is a NFS
problem or whether there is just a NFS bug in Linux", but nothing
more than that.

--steffen
|
|Der Kragenbaer,The moon bear,
|der holt sich munter   he cheerfully and one by one
|einen nach dem anderen runter  wa.ks himself off
|(By Robert Gernhardt)

Re: safe_rename() and verifying the result of link(2)

[Steffen Nurpmeso]

Derek Martin wrote in <20180823050819.ga20...@bladeshadow.org>:
 |On Wed, Aug 22, 2018 at 11:12:39AM -0700, Kevin J. McCarthy wrote:
 |> On Wed, Aug 22, 2018 at 10:04:12AM -0500, Derek Martin wrote:
 ...
 |> Steffen's cautions apply to dotlock code, which is a different case and
 |> is not affected by this change.
 |
 |It's fundamentally the same thing though.  The mechanism for dotlock
 |works like this:
 |
 | - create a secure temporary file (with O_EXCL).
 |   This ensures that the file we're opening for the lock has not been
 |   subverted by another process, potentially an attacker.
 | - stat the file

This does not happen for the traditional BSD code.

 | - link the file to canonical name
 |   If the link succeeds, we have the lock, but the rc from link is
 |   unreliable, so...
 | - stat the file again using the new link
 |   Here, we compare the inode and/or make sure the link count has
 |   increased, to ensure we're really dealing with the same file...

No, instead stat(2) is called on the temporary file, and if that
has a link count of 2 then we have won the race on the lock file.

 | - write the PID to the lock file
 | - unlink the temporary file
 |
 |Exact details may vary slightly, but that's the essence of it.  This
 |is almost exactly what _maildir_commit_message() (and safe_rename())
 |does, for largely the same reasons, though the purpose of the file is
 |different.

I do not know.

--steffen
|
|Der Kragenbaer,The moon bear,
|der holt sich munter   he cheerfully and one by one
|einen nach dem anderen runter  wa.ks himself off
|(By Robert Gernhardt)

Re: safe_rename() and verifying the result of link(2)

[Steffen Nurpmeso]

Vincent Lefevre wrote in <20180821230229.ga16...@zira.vinc17.org>:
 |On 2018-08-21 07:07:15 -0700, Kevin J. McCarthy wrote:
 |> I don't like removing 20-year old safety checks, but I think it's okay
 |> to do so for the case where link() returns 0.
 |
 |I was also hesitant, but after thinking more about it:
 |
 |* That's an unusual "safety check" (I doubt other software does
 |  the same kind of check after link() with return value 0).

Quite the opposite is true.  The original BSD code for creating
dotlock files does this, always.

 |* It is not clear whether this has ever been usuful (nothing in
 |  comment or in the commit log).

According to some statement on a list you are also subscribed to
this NFS problem has been fixed "at the early 90s".  The BSD code
i refer to came up later as far as i now (without having digged to
thoroughly).

 |* It might even have been a mistake. I suspect that the goal was
 |  for NFS, but then the test should have been for non-zero return
 |  values instead of zero.

The code i refer to was not about renaming some file, but about
savely creating a temporary dotlock file, of course, where several
processes may create the very same file, and it must be ensured
that it is you who won the race.  Since i have no overview over
Unix systems but *BSD and Linux (2.0.x, 2.2.x, 2.4.x, then
a decade not, then 4.x), and Solaris via OpenCSW.org, and an old
UnixWare VM but which i have not used since my old machine died,
i would not dare to change this.  Maybe Christos Zoulas of NetBSD
would know more, but then i think i am not the right one to ask.

I for one will not change the code for the MUA i maintain, because
one of the (very far) future goals is (at least partial) support
of elder Unix from before Y2K (from before 1999-01-11, when
i first used Linux, to be exact).

--steffen
|
|Der Kragenbaer,The moon bear,
|der holt sich munter   he cheerfully and one by one
|einen nach dem anderen runter  wa.ks himself off
|(By Robert Gernhardt)

Re: Crash after failed [C]opy

[Steffen Nurpmeso]

Moritz Barsnick  wrote:

 |I don't know why a mutt from hg tip would crash, "for me" it seems
 |fixed. (No, I haven't had the time to try vanilla 1.8.0. Enough
 |correlating for now. ;-))

Yaah, i am very sorry.  No i don't have Mercurial no more at all:

  ?0[steffen@wales tmp]$ (cd /home/steffen/code.arena/mutt.tar_bomb_git;
  git loca)
  * db70827 (tag: refs/tags/v1.8.0, refs/heads/master) Import v1.8.0, 2017-02-24
  * d39448b (tag: refs/tags/v1.7.0) Import v1.7.0, 2016-08-18
  * ec22a9a (tag: refs/tags/v1.6.2) Import v1.6.2, 2016-08-03
  * 29d1cde (tag: refs/tags/v1.6.0) Import v1.6.0, 2016-04-19
  * 290e129 (tag: refs/tags/v1.5.24) Import v1.5.24, 2015-09-02
  * 419d57e (tag: refs/tags/v1.5.23) Import v1.5.23, 2015-02-24
  * a5cf704 (tag: refs/tags/v1.5.22) Import v1.5.22, 2015-02-24
  * 42282b5 (HEAD -> refs/heads/arena-manager-null) NULL

But since my main machine died i really use binary updates for
almost anything but the very small things, e.g., TinyCC.  E.g.,
i cannot even git garbage collect the groff repo because of memory
constraints in the VM i work in, i need to raise the limit and
restart for that, and then the rest of the system is unusable.
I really need to buy a Librem or still Zenbook, finally, and then
i can use CRUX-Linux more often again, juhu.  And such...
Sorry for the noise, then.

--steffen

Re: Crash after failed [C]opy

[Steffen Nurpmeso]

"Kevin J. McCarthy"  wrote:
 |On Fri, Apr 28, 2017 at 09:37:11PM +0200, Moritz Barsnick wrote:
 |> On Fri, Apr 28, 2017 at 11:25:28 -0700, Kevin J. McCarthy wrote:
 |>> I haven't been able to duplicate the segv yet, so I have a few questions
 |>> I'm hoping will help me track it down.
 |> 
 |> I can reproduce with my patched 1.8.0, but not with 1.8.2, patched or
 |> vanilla.
 |
 |Yes, that crash was something I fixed for 1.8.1:
 |https://dev.mutt.org/hg/mutt/rev/e3e47b2f1370
 |
 |But Steffen reported running 1.8.2.  I'm hoping he was perhaps mistaken.

Yes, pacman said 1.8.2, but mutt -v actually says 1.8.0.
Sorry again.

--steffen

Re: Crash after failed [C]opy

[Steffen Nurpmeso]

Good evening,

"Kevin J. McCarthy" <ke...@8t8.us> wrote:
 |On Fri, Apr 28, 2017 at 02:47:38PM +0200, Steffen Nurpmeso wrote:
 |>|For comparison purposes only, tried
 |>|
 |>|  [C] -> maildir:/tmp/x.maildir
 |>|
 |>|and got asked whether creation is desired, i said yes, and then:
 |>|
 |>|  maildir:/tmp/x.maildir: No such file or directory (errno = 2)Segmentatio\
 |>|  n \
 |>|  fault (core dumped)
 |>|  ?139[steffen@wales tmp]$
 |
 |I haven't been able to duplicate the segv yet, so I have a few questions
 |I'm hoping will help me track it down.
 |
 |What is your value of $mbox_type just before you attempt the copy?
 |:set ?mbox_type

  mbox_type=mbox

 |
 |What type of mailbox are you copying from (imap, mbox, maildir, mh)?

mbox.

  Copy to mailbox: maildir:/tmp/hi.mbox

Then "yes" for create and

  maildir:/tmp/hi.mbox: No such file or directory (errno = 2)Segmentation fault 
(core dumped)

 |Would you include the full output of 'mutt -v'?

Attached.

Eh, but, wait, please.  I see now this is indeed 1.8.0 (pacman
reports 1.8.2 but installed is still 1.8.0)!  If this has been
fixed in between, as the other two mails imply, then i am really
sorry for the noise.  I haven't updated this VM since about two
weeks because ArchLinux has switched to OpenSSL 1.1.0, and i was
near download limit, and that resets on the 22nd each month, and,
pfff, last Saturday i have found no time.  Please just ignore
this, then.  If after the update and reboot i can still reproduce
this (when you cannot today), then i would report this, ok?
Sorry.

--steffen