[Distutils] providing a way for pip to communicate extra info to users
On the pypa-dev Google group, a suggestion was raised about giving pip
a way to communicate extra info to users.
This was during a thread started by Matthew Brett about pip breaking
for certain macOS users due to certain TLS changes ("Impending silent
breakage of pip / macOS likely to cause severe confusion"). Donald
said this behavior is governed by PEP 503 and that the topic was best
discussed on distutils-sig:
https://groups.google.com/d/msg/pypa-dev/Oz6SGA7gefo/RRXQBQSBBAAJ
so I'm raising the suggestion here to continue the discussion.
One of Donald's comments in response to the idea (and that occurred to
me too and that I agree with) is that providing a way to communicate
messages to users introduces another possible avenue for attack.
A possible middle-ground could be to hard-code a message in pip. Pip
could display the message in certain circumstances, e.g. in response
to certain types of failures. For example, the message could tell
users to check a certain URL maintained by PyPA for further
information / possible announcements.
What do people think?
--Chris
___
Distutils-SIG maillist - Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] providing a way for pip to communicate extra info to users
On Tue, 10 Apr 2018, 05:17 Chris Jerdonek, wrote:
> On the pypa-dev Google group, a suggestion was raised about giving pip
> a way to communicate extra info to users.
>
> This was during a thread started by Matthew Brett about pip breaking
> for certain macOS users due to certain TLS changes ("Impending silent
> breakage of pip / macOS likely to cause severe confusion"). Donald
> said this behavior is governed by PEP 503 and that the topic was best
> discussed on distutils-sig:
> https://groups.google.com/d/msg/pypa-dev/Oz6SGA7gefo/RRXQBQSBBAAJ
> so I'm raising the suggestion here to continue the discussion.
>
> One of Donald's comments in response to the idea (and that occurred to
> me too and that I agree with) is that providing a way to communicate
> messages to users introduces another possible avenue for attack.
>
> A possible middle-ground could be to hard-code a message in pip. Pip
> could display the message in certain circumstances, e.g. in response
> to certain types of failures. For example, the message could tell
> users to check a certain URL maintained by PyPA for further
> information / possible announcements.
>
> What do people think?
>
I like the idea.
I think linking to a location where we can make informative comments would
be a good idea — ideally where we can show announcements reverse
chronologically.
I don’t know how relevant they are but scenarios where this would help,
that come to my mind are:
- Status Page: "pypi.org is undergoing an incident and installations may
fail. You can find more information at status.python.org."
- Major Features: for things like PEP 517 when it's released. "There's
news. Have a look at pypi.org/news" or something like that.
> --
> ___
> Distutils-SIG maillist - Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
___
Distutils-SIG maillist - Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] providing a way for pip to communicate extra info to users
On 11 April 2018 at 17:32, Pradyun Gedam wrote: > On Tue, 10 Apr 2018, 05:17 Chris Jerdonek, wrote: [...] >> A possible middle-ground could be to hard-code a message in pip. Pip >> could display the message in certain circumstances, e.g. in response >> to certain types of failures. For example, the message could tell >> users to check a certain URL maintained by PyPA for further >> information / possible announcements. >> >> What do people think? > > > I like the idea. > > I think linking to a location where we can make informative comments would > be a good idea — ideally where we can show announcements reverse > chronologically. > > I don’t know how relevant they are but scenarios where this would help, that > come to my mind are: > > - Status Page: "pypi.org is undergoing an incident and installations may > fail. You can find more information at status.python.org." For HTTP type responses, which is what I understood Chris' question to be about, this seems like a good approach to me - the index can supply a response that triggers pip to report a message. "The index XXX reported an issue - for more information see XXX/status". That would need a PEP 503 change to say that an index can trigger this message by sending a certain response code, and that if an index does send that code, it must provide additional information at its /status page. > - Major Features: for things like PEP 517 when it's released. "There's news. > Have a look at pypi.org/news" or something like that. For these, I was thinking about this in the context of how we announce releases like pip 10. Maybe something like this would better fit as an addition to the pip selfcheck code - so that as well as checking for a newer version, pip would also check for a "Message of the day" at a known URL and display it if there is one. That gives us a way to announce releases or betas, or upcoming deprecations, in a way that reaches every pip user (at least every one who's connected to the internet!) It's a bit intrusive, so I think it's critical that we use it sparingly, but it would be good to have at least some channel that reaches everyone. I don't think it's something we'd want to use for transient issues like pypi outages, for instance. Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] providing a way for pip to communicate extra info to users
On Mon, Apr 9, 2018, 16:47 Chris Jerdonek wrote: > > One of Donald's comments in response to the idea (and that occurred to > me too and that I agree with) is that providing a way to communicate > messages to users introduces another possible avenue for attack. I agree that this is worth thinking about, but having thought about it I'm having trouble coming up with a threat model where it creates additional exposure? If someone takes over package distribution, that's obviously a far more serious problem. A messaging mechanism could amplify such an attack by encouraging people to install the compromised packages – but pip's existing check for new pip versions can also do that. Or if we have a mechanism for securing package updates, like TUF, then presumably we can use it to protect the MOTD as well? -n ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] providing a way for pip to communicate extra info to users
On 11 April 2018 at 20:16, Dwight Hubbard wrote: > It would be useful as well for sites that run their own mirror > infrastructure to be able to add motd text to the pip commands as well. > > However I don't think this should be implemented via the response code from > a call to some rest api. It would be trivial to proxy the call to a > different location and send a different message. Any implementation would > need some way to sign and verify the message as authentic. -1 on explicit signing and verification of messages. The infrastructure needed for that is more than the feature warrants. HTTPS access to the index server is fundamental to pip - if an attacker can subvert that, they don't need to mess with a message, they can just replace packages. So I don't see that displaying a message that's available from that same index server is an additional vulnerability, surely? But I'm not a security expert - I'd defer to someone like Donald to comment on the security aspects of any proposal here. Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] providing a way for pip to communicate extra info to users
On 12 April 2018 at 07:01, Paul Moore wrote: > HTTPS access to the index server is fundamental to pip - if an > attacker can subvert that, they don't need to mess with a message, > they can just replace packages. So I don't see that displaying a > message that's available from that same index server is an additional > vulnerability, surely? But I'm not a security expert - I'd defer to > someone like Donald to comment on the security aspects of any proposal > here. Right now it doesn't create any additional vulnerabilities, since we're relying primarily on HTTPS for PyPI -> installer security. However, that changes once PEP 458 gets implemented, as that will switch the primary package level security mechanism over to TUF, which includes a range of mechanisms designed to detect tampering with the link to PyPI (including freeze attacks that keep you from checking for new packages, or attempting to lie about which versions are available). So the scenario we want to avoid is one where an attacker can present a notice that says "Please ignore that scary security warning your installer is giving you, we're having an issue with the metadata generation process on the server. To resolve the problem, please force upgrade pip". That's a solvable problem (e.g. only check for the MOTD *after* successfully retrieving a valid metadata file), but it's still something to take into account. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] providing a way for pip to communicate extra info to users
It would be useful as well for sites that run their own mirror infrastructure to be able to add motd text to the pip commands as well. However I don't think this should be implemented via the response code from a call to some rest api. It would be trivial to proxy the call to a different location and send a different message. Any implementation would need some way to sign and verify the message as authentic. On Wed, Apr 11, 2018 at 10:18 AM, Paul Moore wrote: > On 11 April 2018 at 17:32, Pradyun Gedam wrote: > > On Tue, 10 Apr 2018, 05:17 Chris Jerdonek, > wrote: > [...] > >> A possible middle-ground could be to hard-code a message in pip. Pip > >> could display the message in certain circumstances, e.g. in response > >> to certain types of failures. For example, the message could tell > >> users to check a certain URL maintained by PyPA for further > >> information / possible announcements. > >> > >> What do people think? > > > > > > I like the idea. > > > > I think linking to a location where we can make informative comments > would > > be a good idea — ideally where we can show announcements reverse > > chronologically. > > > > I don’t know how relevant they are but scenarios where this would help, > that > > come to my mind are: > > > > - Status Page: "pypi.org is undergoing an incident and installations may > > fail. You can find more information at status.python.org." > > For HTTP type responses, which is what I understood Chris' question to > be about, this seems like a good approach to me - the index can supply > a response that triggers pip to report a message. "The index XXX > reported an issue - for more information see XXX/status". That would > need a PEP 503 change to say that an index can trigger this message by > sending a certain response code, and that if an index does send that > code, it must provide additional information at its /status page. > > > - Major Features: for things like PEP 517 when it's released. "There's > news. > > Have a look at pypi.org/news" or something like that. > > For these, I was thinking about this in the context of how we announce > releases like pip 10. Maybe something like this would better fit as an > addition to the pip selfcheck code - so that as well as checking for a > newer version, pip would also check for a "Message of the day" at a > known URL and display it if there is one. That gives us a way to > announce releases or betas, or upcoming deprecations, in a way that > reaches every pip user (at least every one who's connected to the > internet!) It's a bit intrusive, so I think it's critical that we use > it sparingly, but it would be good to have at least some channel that > reaches everyone. I don't think it's something we'd want to use for > transient issues like pypi outages, for instance. > > Paul > ___ > Distutils-SIG maillist - Distutils-SIG@python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -- Dwight Hubbard Python Language Guy M: 408-646-9444 ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] providing a way for pip to communicate extra info to users
>From the TUF perspective it seems like it would be straightforward to make the MOTD a "package", whose "contents" is the MOTD text, and that we "upgrade" it to get the latest text before displaying anything. -n On Thu, Apr 12, 2018, 05:10 Nick Coghlan wrote: > On 12 April 2018 at 07:01, Paul Moore wrote: > > HTTPS access to the index server is fundamental to pip - if an > > attacker can subvert that, they don't need to mess with a message, > > they can just replace packages. So I don't see that displaying a > > message that's available from that same index server is an additional > > vulnerability, surely? But I'm not a security expert - I'd defer to > > someone like Donald to comment on the security aspects of any proposal > > here. > > Right now it doesn't create any additional vulnerabilities, since > we're relying primarily on HTTPS for PyPI -> installer security. > > However, that changes once PEP 458 gets implemented, as that will > switch the primary package level security mechanism over to TUF, which > includes a range of mechanisms designed to detect tampering with the > link to PyPI (including freeze attacks that keep you from checking for > new packages, or attempting to lie about which versions are > available). > > So the scenario we want to avoid is one where an attacker can present > a notice that says "Please ignore that scary security warning your > installer is giving you, we're having an issue with the metadata > generation process on the server. To resolve the problem, please force > upgrade pip". > > That's a solvable problem (e.g. only check for the MOTD *after* > successfully retrieving a valid metadata file), but it's still > something to take into account. > > Cheers, > Nick. > > -- > Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia > ___ > Distutils-SIG maillist - Distutils-SIG@python.org > https://mail.python.org/mailman/listinfo/distutils-sig > ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] providing a way for pip to communicate extra info to users
FYI: TUF has a custom metadata field in the targets metadata that could potentially be used for this purpose. We can explain more if there is interest... On Thu, Apr 12, 2018 at 8:26 AM, Nathaniel Smith wrote: > From the TUF perspective it seems like it would be straightforward to make > the MOTD a "package", whose "contents" is the MOTD text, and that we > "upgrade" it to get the latest text before displaying anything. > > -n > > On Thu, Apr 12, 2018, 05:10 Nick Coghlan wrote: > >> On 12 April 2018 at 07:01, Paul Moore wrote: >> > HTTPS access to the index server is fundamental to pip - if an >> > attacker can subvert that, they don't need to mess with a message, >> > they can just replace packages. So I don't see that displaying a >> > message that's available from that same index server is an additional >> > vulnerability, surely? But I'm not a security expert - I'd defer to >> > someone like Donald to comment on the security aspects of any proposal >> > here. >> >> Right now it doesn't create any additional vulnerabilities, since >> we're relying primarily on HTTPS for PyPI -> installer security. >> >> However, that changes once PEP 458 gets implemented, as that will >> switch the primary package level security mechanism over to TUF, which >> includes a range of mechanisms designed to detect tampering with the >> link to PyPI (including freeze attacks that keep you from checking for >> new packages, or attempting to lie about which versions are >> available). >> >> So the scenario we want to avoid is one where an attacker can present >> a notice that says "Please ignore that scary security warning your >> installer is giving you, we're having an issue with the metadata >> generation process on the server. To resolve the problem, please force >> upgrade pip". >> >> That's a solvable problem (e.g. only check for the MOTD *after* >> successfully retrieving a valid metadata file), but it's still >> something to take into account. >> >> Cheers, >> Nick. >> >> -- >> Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia >> ___ >> Distutils-SIG maillist - Distutils-SIG@python.org >> https://mail.python.org/mailman/listinfo/distutils-sig >> > > ___ > Distutils-SIG maillist - Distutils-SIG@python.org > https://mail.python.org/mailman/listinfo/distutils-sig > > ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] providing a way for pip to communicate extra info to users
A MOTD from anything but a signed package would be user-supplied input. Shell/terminal command ^[escaping would be necessary: https://stackoverflow.com/questions/6534556/how-to-remove-and-all-of-the-escape-sequences-in-a-file-using-linux-shell-sc Impact: Are additional requests and variable messages really necessary? Can static error messages simply say 'check /news for more information'? (thus saving: millions of requests per year and additional MOTD package signing overhead and bandwidth)? On Thursday, April 12, 2018, Justin Cappos wrote: > FYI: TUF has a custom metadata field in the targets metadata that could > potentially be used for this purpose. We can explain more if there is > interest... > > On Thu, Apr 12, 2018 at 8:26 AM, Nathaniel Smith wrote: > >> From the TUF perspective it seems like it would be straightforward to >> make the MOTD a "package", whose "contents" is the MOTD text, and that we >> "upgrade" it to get the latest text before displaying anything. >> >> -n >> >> On Thu, Apr 12, 2018, 05:10 Nick Coghlan wrote: >> >>> On 12 April 2018 at 07:01, Paul Moore wrote: >>> > HTTPS access to the index server is fundamental to pip - if an >>> > attacker can subvert that, they don't need to mess with a message, >>> > they can just replace packages. So I don't see that displaying a >>> > message that's available from that same index server is an additional >>> > vulnerability, surely? But I'm not a security expert - I'd defer to >>> > someone like Donald to comment on the security aspects of any proposal >>> > here. >>> >>> Right now it doesn't create any additional vulnerabilities, since >>> we're relying primarily on HTTPS for PyPI -> installer security. >>> >>> However, that changes once PEP 458 gets implemented, as that will >>> switch the primary package level security mechanism over to TUF, which >>> includes a range of mechanisms designed to detect tampering with the >>> link to PyPI (including freeze attacks that keep you from checking for >>> new packages, or attempting to lie about which versions are >>> available). >>> >>> So the scenario we want to avoid is one where an attacker can present >>> a notice that says "Please ignore that scary security warning your >>> installer is giving you, we're having an issue with the metadata >>> generation process on the server. To resolve the problem, please force >>> upgrade pip". >>> >>> That's a solvable problem (e.g. only check for the MOTD *after* >>> successfully retrieving a valid metadata file), but it's still >>> something to take into account. >>> >>> Cheers, >>> Nick. >>> >>> -- >>> Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia >>> ___ >>> Distutils-SIG maillist - Distutils-SIG@python.org >>> https://mail.python.org/mailman/listinfo/distutils-sig >>> >> >> ___ >> Distutils-SIG maillist - Distutils-SIG@python.org >> https://mail.python.org/mailman/listinfo/distutils-sig >> >> > ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
