Re: [Python-ideas] Remote package/module imports through HTTP/S

[Nick Coghlan]

On 24 August 2017 at 05:04, John Torakis  wrote:
> Dark times...
>
> So is it a "case closed", or is there any improvement that will make it
> worth it to be an stdlib module?

Not really, as even aside from the security concerns, there are simply
too many ways that it can fail that are outside of our control, but
would potentially lead to folks filing bug reports against CPython
without realising that the problem actually lies somewhere else (e.g.
with their network configuration).

For a third party module, that's not a problem:

- folks have to find out httpimport exists
- folks have to decide "I want this"
- folks have to explicitly install & enable it
- folks still get to keep all the very shiny pieces when it breaks
unexpectedly, but they also already know where to go for help :)

Being a third party utility means you can also update it on your own
timeline, rather than being limited to the standard library's
relatively slow update and rollout cycles.

>From a compatibility point of view, we also *like* having
sophisticated import system plugins like httpimport out in the wild,
as it means:

- it actually makes sense to define & maintain the import plugin APIs
that make it possible
- there's additional integration testing of those APIs happening
beyond our own test suite

Putting away my import system co-maintainer hat and donning my
commercial redistributor hat: it already bothers some of our (and our
customers') security folks that we ship package installation tools
that access unfiltered third party package repositories by default
(e.g. pip defaulting to querying PyPI).

As a result, I'm pretty sure that even if upstream said "httpimport is
in the Python standard library now!", we'd get explicit requests
asking us to take it out of our redistributed version and make it at
most an optional install (similar to what we do with IDLE and Tcl/Tk
support in general).

Cheers,
Nick.

P.S. As a potentially useful point of reference: "it's hard to debug
when it breaks" is the main reason we resisted adding native lazy
import support for so long, and that's just a matter of moving import
errors away from the import statement and instead raising them as a
side effect of an attribute access. It's also why we moved reload()
*out* of the builtins in the move to Python 3: while module reloading
is a fully supported operation, it also has a lot of subtleties that
make it easy to get wrong.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Remote package/module imports through HTTP/S

[Stephen J. Turnbull]

John Torakis writes:

 > But, as it seems like it is a very big feature (to me at least),

And "pip install httpimport" seems like it is a very small burden (to
me at least).

I agree with Paul Moore.  Putting this in the stdlib seems both
unnecessary, given pip, and an attractive nuisance for naive users.

>From the point of view of the blue team, checking for mere presence of
httpimport in the environment is indicative of danger if it's
pip-able, useless if it's in the stdlib.

With respect to "it just makes exec(urlopen()) easier", any code must
be audited for application of exec() to user input anyway, regardless
of whether it fetches stuff off the Internet.  Adding httpimport use
to the checklist adds a little bit of complexity to *every* security
check, and a fair amount of danger in security-oblivious environments
such as many university labs, and I would imagine many corporate
development groups as well.

YMMV, but from the point of view of the larger, security-conscious
organization, I would say -1.  It's an attractive nuisance unless
you're a security person, and then pip is not a big deal.

Steve

___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Remote package/module imports through HTTP/S

[Chris Angelico]

On Thu, Aug 24, 2017 at 12:13 PM, Stephen J. Turnbull
 wrote:
> Chris Angelico writes:
>
>  > If you're worried about the latter, don't use httpimport.
>
> I guarantee you that in my (university) environment, if httpimport is
> in the stdlib, its use will be rampant (and not just by students, but
> by security-oblivious faculty).  I want to be able to walk up to a
> student, say "may I?" and type "python -m httpimport" to determine if
> that particular risky behavior is a worry.  Because *I'm* liable for
> my students' PCs' behavior on the network.
>
> Personally speaking, +1 on PyPI, -100 on stdlib.

Agreed, and a VERY good reason for this to be an explicitly-installed
package. By its nature, it won't be a dependency of other packages, so
keeping it out of the stdlib pretty much guarantees that it'll only be
available if it's been called for by name.

ChrisA
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Remote package/module imports through HTTP/S

[Stephen J. Turnbull]

Chris Angelico writes:

 > If you're worried about the latter, don't use httpimport.

I guarantee you that in my (university) environment, if httpimport is
in the stdlib, its use will be rampant (and not just by students, but
by security-oblivious faculty).  I want to be able to walk up to a
student, say "may I?" and type "python -m httpimport" to determine if
that particular risky behavior is a worry.  Because *I'm* liable for
my students' PCs' behavior on the network.

Personally speaking, +1 on PyPI, -100 on stdlib.

Steve

___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Remote package/module imports through HTTP/S

[John Torakis]

On 23/08/2017 22:06, Chris Angelico wrote:
> On Thu, Aug 24, 2017 at 5:04 AM, John Torakis  wrote:
>> Dark times...
>>
>> So is it a "case closed", or is there any improvement that will make it
>> worth it to be an stdlib module?
>>
>> I mean, times have changed from 1995, and I am not referring to HTTPS
>> invention. This is the reason that makes httpimport just tolerable
>> security-wise.
>>
>> I'm talking about the need to rapidly test public code. I insist that
>> testing code available on Github (or other repos), without the
>> venv/clone/install hassle is a major improvement in my (and most sec
>> researchers' I know) Python workflow. It makes REPL prototyping million
>> times smoother.
>> We all have created small scripts that auto load modules from URLs anyway.
>> That's why I thought that this modules falls under the second category of
>> 20.2.1 in https://docs.python.org/devguide/stdlibchanges.html (I did my
>> homework before getting to mail in this list).
>>
>> So, if there is something that would make this module acceptable for stdlib,
>> please let me know! I'd more than happily reform it and make it comply with
>> Python stdlib requirements.
> Why can't people just "pip install httpimport" to make use of it? Why
> does it need to be in the stdlib?
It doesn't, strictly speaking, *need* to be in stdlib. Of course it
doesn't! Python is good enough without it.
But, as it seems like it is a very big feature (to me at least), it
feels right to be "officially" a Python feature. It feels right to be
officially supported and not just another module, as it extends core
import functionality. Just like zipimport does. Zipimport could just be
a 3rd party module too. But it is in the core, and I can see why.

Anyway, I will post it to PyPI when I finalize Github support and extend
the testing a little bit. I will then shoot a mail again and repropose
the module when it reaches full maturity.

>
> ChrisA
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/

Thank you all for your time!
John Torakis

___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Remote package/module imports through HTTP/S

[Chris Angelico]

On Thu, Aug 24, 2017 at 5:04 AM, John Torakis  wrote:
> Dark times...
>
> So is it a "case closed", or is there any improvement that will make it
> worth it to be an stdlib module?
>
> I mean, times have changed from 1995, and I am not referring to HTTPS
> invention. This is the reason that makes httpimport just tolerable
> security-wise.
>
> I'm talking about the need to rapidly test public code. I insist that
> testing code available on Github (or other repos), without the
> venv/clone/install hassle is a major improvement in my (and most sec
> researchers' I know) Python workflow. It makes REPL prototyping million
> times smoother.
> We all have created small scripts that auto load modules from URLs anyway.
> That's why I thought that this modules falls under the second category of
> 20.2.1 in https://docs.python.org/devguide/stdlibchanges.html (I did my
> homework before getting to mail in this list).
>
> So, if there is something that would make this module acceptable for stdlib,
> please let me know! I'd more than happily reform it and make it comply with
> Python stdlib requirements.

Why can't people just "pip install httpimport" to make use of it? Why
does it need to be in the stdlib?

ChrisA
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Remote package/module imports through HTTP/S

[John Torakis]

Dark times...

So is it a "case closed", or is there any improvement that will make it
worth it to be an stdlib module?

I mean, times have changed from 1995, and I am not referring to HTTPS
invention. This is the reason that makes httpimport just tolerable
security-wise.

I'm talking about the need to rapidly test public code. I insist that
testing code available on Github (or other repos), without the
venv/clone/install hassle is a major improvement in my (and most sec
researchers' I know) Python workflow. It makes REPL prototyping million
times smoother.
We all have created small scripts that auto load modules from URLs
anyway. That's why I thought that this modules falls under the second
category of 20.2.1 in
https://docs.python.org/devguide/stdlibchanges.html (I did my homework
before getting to mail in this list).

So, if there is something that would make this module acceptable for
stdlib, please let me know! I'd more than happily reform it and make it
comply with Python stdlib requirements.

John Torakis

On 23/08/2017 21:48, Guido van Rossum wrote:
> For security reasons. AFAIK HTTPS wasn't even invented at the time.
>
> On Wed, Aug 23, 2017 at 11:44 AM, John Torakis  > wrote:
>
>
>
> On 23/08/2017 21:41, Guido van Rossum wrote:
>> This isn't ever going to be a standard feature. It's available as
>> a third-party package and that's fine.
>>
>> I'd like to add a historic note -- this was first proposed around
>> 1995 by Michael McLay. (Sorry, I don't have an email sitting
>> around, but I'm sure he brought this up at or around the first
>> Python workshop at NIST in 1995 -- I was his guest at NIST for
>> several months at the time.)
>>
> Woah! I was 2 years old at that time! Little did I know!
> Can I ask why it got rejected the first time?
>> -- 
>> --Guido van Rossum (python.org/~guido )
>>
>>
>> ___
>> Python-ideas mailing list
>> Python-ideas@python.org 
>> https://mail.python.org/mailman/listinfo/python-ideas
>> 
>> Code of Conduct: http://python.org/psf/codeofconduct/
>> 
> ___ Python-ideas
> mailing list Python-ideas@python.org
> 
> https://mail.python.org/mailman/listinfo/python-ideas
>  Code of
> Conduct: http://python.org/psf/codeofconduct/
>  
>
> -- 
> --Guido van Rossum (python.org/~guido )
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Remote package/module imports through HTTP/S

[Guido van Rossum]

For security reasons. AFAIK HTTPS wasn't even invented at the time.

On Wed, Aug 23, 2017 at 11:44 AM, John Torakis 
wrote:

>
>
> On 23/08/2017 21:41, Guido van Rossum wrote:
>
> This isn't ever going to be a standard feature. It's available as a
> third-party package and that's fine.
>
> I'd like to add a historic note -- this was first proposed around 1995 by
> Michael McLay. (Sorry, I don't have an email sitting around, but I'm sure
> he brought this up at or around the first Python workshop at NIST in 1995
> -- I was his guest at NIST for several months at the time.)
>
> Woah! I was 2 years old at that time! Little did I know!
> Can I ask why it got rejected the first time?
>
> --
> --Guido van Rossum (python.org/~guido )
>
>
> ___
> Python-ideas mailing 
> listPython-ideas@python.orghttps://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
>
>
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
>


-- 
--Guido van Rossum (python.org/~guido)
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Remote package/module imports through HTTP/S

[John Torakis]

On 23/08/2017 21:41, Guido van Rossum wrote:
> This isn't ever going to be a standard feature. It's available as a
> third-party package and that's fine.
>
> I'd like to add a historic note -- this was first proposed around 1995
> by Michael McLay. (Sorry, I don't have an email sitting around, but
> I'm sure he brought this up at or around the first Python workshop at
> NIST in 1995 -- I was his guest at NIST for several months at the time.)
>
Woah! I was 2 years old at that time! Little did I know!
Can I ask why it got rejected the first time?
> -- 
> --Guido van Rossum (python.org/~guido )
>
>
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/

___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Remote package/module imports through HTTP/S

[John Torakis]

On 23/08/2017 21:24, Paul Moore wrote:
> On 23 August 2017 at 18:49, Chris Angelico  wrote:
>> Still -1 on this becoming a stdlib package, as there's nothing I've
>> yet seen that can't be done as a third-party package. But it's less
>> scary than I thought it was :)
> IMO, this would make a great 3rd party package (I note that it's not
> yet published on PyPI). It's possible that it would end up being
> extremely popular, and recognised as sufficiently secure - at which
> point it may be worth considering for core inclusion. But it's also
> possible that it remains niche, and/or people aren't willing to take
> the security risks that it implies, in which case it's still useful to
> those who do like it.
PyPI upload is scheduled when some more testing and commenting takes place.
> One aspect that hasn't been mentioned yet - as a 3rd party module, the
> user (or the organisation's security team) can control whether or not
> the ability to import over the web is available by controlling whether
> the module is allowed to be installed - whereas with a core module,
> it's there, like it or not, and *all* Python code has to be audited on
> the assumption that it might be used. 
True!
But you can urlopen()->exec() anything out there anyway! A ">>>" prompt
is all you need.
> I could easily imagine cases
> where the httpimport module was allowed on development machines and CI
> servers, but forbidden on production (and pre-production) systems.
> That option simply isn't available if the feature is in the core.
I agree that there are circumstances that this module should not be used
(regardless of security implications).
In a released product for example. Depending on the UP-ness of a remote
repository (e.g github), not to even mention the API
backward-compatibility of an upstream package, is just **BAD** for a
ready-released-deliverable product! This is why we have virtual
environments!

But it remains an option to use it or not! I, for example, find myself
REPLing more than scripting. When REPLing for something you plan to
implement sometime-somehow, this module is really what you need! But
when I finally create a script, I won't disable its offline
functionality just to use httpimport. That would be suicidal! When I
finally come with a working thing I will finally land the used packages
to disk and to a virtual environment.


My argument is that this module will add greatly to the Python's ad-hoc
testing capabilities! I find it elegant for such feature to be in the
stdlib of a language.
I don't doubt that it can survive as a 3rd party module, though.
>
> Paul
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/


___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Remote package/module imports through HTTP/S

[Guido van Rossum]

 This isn't ever going to be a standard feature. It's available as a
third-party package and that's fine.

I'd like to add a historic note -- this was first proposed around 1995 by
Michael McLay. (Sorry, I don't have an email sitting around, but I'm sure
he brought this up at or around the first Python workshop at NIST in 1995
-- I was his guest at NIST for several months at the time.)

-- 
--Guido van Rossum (python.org/~guido)
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Remote package/module imports through HTTP/S

[Paul Moore]

On 23 August 2017 at 18:49, Chris Angelico  wrote:
> Still -1 on this becoming a stdlib package, as there's nothing I've
> yet seen that can't be done as a third-party package. But it's less
> scary than I thought it was :)

IMO, this would make a great 3rd party package (I note that it's not
yet published on PyPI). It's possible that it would end up being
extremely popular, and recognised as sufficiently secure - at which
point it may be worth considering for core inclusion. But it's also
possible that it remains niche, and/or people aren't willing to take
the security risks that it implies, in which case it's still useful to
those who do like it.

One aspect that hasn't been mentioned yet - as a 3rd party module, the
user (or the organisation's security team) can control whether or not
the ability to import over the web is available by controlling whether
the module is allowed to be installed - whereas with a core module,
it's there, like it or not, and *all* Python code has to be audited on
the assumption that it might be used. I could easily imagine cases
where the httpimport module was allowed on development machines and CI
servers, but forbidden on production (and pre-production) systems.
That option simply isn't available if the feature is in the core.

Paul
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Remote package/module imports through HTTP/S

[John Torakis]

On 23/08/2017 21:11, Chris Angelico wrote:
> On Thu, Aug 24, 2017 at 4:04 AM, Bruce Leban  wrote:
>> On Wed, Aug 23, 2017 at 10:37 AM, John Torakis 
>> wrote:
>>>
>>> Github can be trusted 100% percent for example.
>>
>> This isn't even remotely close to true. While I'd agree with the statement
>> that the SSL cert on github is reasonably trustworthy, the *content* on
>> github is NOT trustworthy and that's where the security risk is.
>>
>> I agree that this is a useful feature and there is no way it should be on by
>> default. The right way IMHO to do this is to have a command line option
>> something like this:
>>
>> python --http-import somelib=https://github.com/someuser/somelib
> If you read his README, it's pretty explicit about URLs; the risk is
> that "https://github.com/someuser/somelib; can be intercepted, not
> that "someuser" is malicious. If you're worried about the latter,
> don't use httpimport.

Again, if https://github.com/someuser/somelib can be intercepted,
https://pypi.python.org/pypi can too.
If HTTPS is intercepted so easily (when not used from browsers) we are
f**ed...
>
> ChrisA
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/

___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Remote package/module imports through HTTP/S

[John Torakis]

On 23/08/2017 20:49, Chris Angelico wrote:
> On Thu, Aug 24, 2017 at 3:37 AM, John Torakis  wrote:
>>
>> On 23/08/2017 20:36, John Torakis wrote:
>>> Yeah, I am a security researcher, I am keen on backdoor programming and
>>> staging and all that! It is my official job and research topic! I go to
>>> the office and code such stuff! I am not a blackhat, nor a security
>>> enthusiast, it is my job.
>>>
>>>
>>> First of all, let's all agree that if someone can run Python code in
>>> your computer you are 100% hacked! It is irrelevant if "httpimport" is a
>>> core python feature or not in that case.
>>>
>>> Now, I agree that this can be exploited if used under plain HTTP, it is
>>> a MiTM -> Remote code execution case. I admit that this is not bright.
>>> But I mention that this can be used in testing.
>>>
>>> On the topic of HTTPS, man-in-the-middle is not possible without
>>> previous Trusted Certificate compromise. Github can be trusted 100%
>>> percent for example. A certificate check has to take place in the HTTPS
>>> remote loading for sure!
> Right, but that just pushes the problem one level further out: you
> need to have a 100% dependable certificate chain. And that means
> absolutely completely trusting all of your root certificates, and it
> also means either not needing to add any _more_ root certificates, or
> being able to configure the cert store. As we've seen elsewhere, this
> is nontrivial.
The centralized PKI as we know it is a pain altogether. Please let me
reference this XKCD strip here:
https://xkcd.com/1200/
Running code to your computer is among the low Impact things that can
happen to you if you have a compromised certificate store. Trust me on that!
In other words, if you can't trust your certificate store now, and you
are afraid of Remote code execution through HTTPS, stop using pip
altogether:
https://github.com/pypa/pip/issues/1168

And most Package Managers anyway..
>>> When I said a "core feature" I meant that the "httpimport" module would
>>> deliver with the core modules. Not that the Finder/Loader has to be in
>>> the list of Finders/Loaders that are used by default! For god sake, I
>>> wouldn't like my PC to start probing for modules just because I mistyped
>>> an import line!
> Glad we agree about that! I have seen people wanting all sorts of
> things to become core features (usually for the sake of interactive
> work), and a lot of it is MUCH better handled as a non-core feature.
>
> Though a lot of what you're saying here - especially this:
>
>>> I know that pip works nicely, especially when paired with virtual
>>> environments, but ad-hoc importing is another another thing. It isn't
>>> meant for delivering real projects. Just for testing modules without the
>>> need to download them, maybe install them, and all.
> could be equally well handled by pip-installing httpimport itself, and
> using that to bootstrap your testing procedures. Unless, of course,
> you're wanting to httpimport httpimport, in which case you're going to
> run into bootstrapping problems whichever way you do it :)
I will never open an issue for 'httpimporting the httpimport itself' It
is a promise!
>
> I think we're on the same page here, but it definitely needs some more
> text in the README to explain this - particularly how this is not a
> replacement for pip. For example, my first thought on seeing this was
> "wow, that's going to be abysmally slow unless it has a cache", but
> the answer to that is: if you need a cache, you probably should be
> using pip to install things properly.
>
> Still -1 on this becoming a stdlib package, as there's nothing I've
> yet seen that can't be done as a third-party package. But it's less
> scary than I thought it was :)
The reason I though that this could serve greatly as an stdlib package
is that it will broaden the horizon for importing arbitrary stuff just
to see if they are working as expected. "Testing" is the world!

> ChrisA
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/


___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Remote package/module imports through HTTP/S

[John Torakis]

On 23/08/2017 21:04, Bruce Leban wrote:
>
> On Wed, Aug 23, 2017 at 10:37 AM, John Torakis  > wrote:
>
>
> Github can be trusted 100% percent for example. 
>
>
> This isn't even remotely close to true. While I'd agree with the
> statement that the SSL cert on github is reasonably trustworthy, the
> *content* on github is NOT trustworthy and that's where the security
> risk is.

Do we trust code on github?

Do we trust code on PyPI?


This is why I **don't** want it ON by default. You have to explicitly
point the Finder/Loader to a repo that you created or you trust. And
provide a list of available modules/packages to import from that URL too.

If the developer isn't sure about the code she/he is importing then it
is her/his fault...

Same goes for pip installing though...

>
> I agree that this is a useful feature and there is no way it should be
> on by default. The right way IMHO to do this is to have a command line
> option something like this:
>
> python --http-import somelib=https://github.com/someuser/somelib
>
>
> which then redefines the import somelib command to import from that
> source. Along with your scenario, it allows people, for example, to
> replace a library with a different version without modifying source or
> installing a different version. That's pretty useful.
That's what I am thinking too! just provide the module so someone can
"python -m" it, or start a REPL in the context that some
packages/modules are available from a URL.


>
> --- Bruce

John Torakis
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Remote package/module imports through HTTP/S

[Chris Angelico]

On Thu, Aug 24, 2017 at 4:04 AM, Bruce Leban  wrote:
>
> On Wed, Aug 23, 2017 at 10:37 AM, John Torakis 
> wrote:
>>
>>
>> Github can be trusted 100% percent for example.
>
>
> This isn't even remotely close to true. While I'd agree with the statement
> that the SSL cert on github is reasonably trustworthy, the *content* on
> github is NOT trustworthy and that's where the security risk is.
>
> I agree that this is a useful feature and there is no way it should be on by
> default. The right way IMHO to do this is to have a command line option
> something like this:
>
> python --http-import somelib=https://github.com/someuser/somelib

If you read his README, it's pretty explicit about URLs; the risk is
that "https://github.com/someuser/somelib; can be intercepted, not
that "someuser" is malicious. If you're worried about the latter,
don't use httpimport.

ChrisA
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Remote package/module imports through HTTP/S

[Bruce Leban]

On Wed, Aug 23, 2017 at 10:37 AM, John Torakis 
wrote:

>
> Github can be trusted 100% percent for example.


This isn't even remotely close to true. While I'd agree with the statement
that the SSL cert on github is reasonably trustworthy, the *content* on
github is NOT trustworthy and that's where the security risk is.

I agree that this is a useful feature and there is no way it should be on
by default. The right way IMHO to do this is to have a command line option
something like this:

python --http-import somelib=https://github.com/someuser/somelib


which then redefines the import somelib command to import from that source.
Along with your scenario, it allows people, for example, to replace a
library with a different version without modifying source or installing a
different version. That's pretty useful.

--- Bruce
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Remote package/module imports through HTTP/S

[Chris Angelico]

On Thu, Aug 24, 2017 at 3:37 AM, John Torakis  wrote:
>
>
> On 23/08/2017 20:36, John Torakis wrote:
>> Yeah, I am a security researcher, I am keen on backdoor programming and
>> staging and all that! It is my official job and research topic! I go to
>> the office and code such stuff! I am not a blackhat, nor a security
>> enthusiast, it is my job.
>>
>>
>> First of all, let's all agree that if someone can run Python code in
>> your computer you are 100% hacked! It is irrelevant if "httpimport" is a
>> core python feature or not in that case.
>>
>> Now, I agree that this can be exploited if used under plain HTTP, it is
>> a MiTM -> Remote code execution case. I admit that this is not bright.
>> But I mention that this can be used in testing.
>>
>> On the topic of HTTPS, man-in-the-middle is not possible without
>> previous Trusted Certificate compromise. Github can be trusted 100%
>> percent for example. A certificate check has to take place in the HTTPS
>> remote loading for sure!

Right, but that just pushes the problem one level further out: you
need to have a 100% dependable certificate chain. And that means
absolutely completely trusting all of your root certificates, and it
also means either not needing to add any _more_ root certificates, or
being able to configure the cert store. As we've seen elsewhere, this
is nontrivial.

>> When I said a "core feature" I meant that the "httpimport" module would
>> deliver with the core modules. Not that the Finder/Loader has to be in
>> the list of Finders/Loaders that are used by default! For god sake, I
>> wouldn't like my PC to start probing for modules just because I mistyped
>> an import line!

Glad we agree about that! I have seen people wanting all sorts of
things to become core features (usually for the sake of interactive
work), and a lot of it is MUCH better handled as a non-core feature.

Though a lot of what you're saying here - especially this:

>> I know that pip works nicely, especially when paired with virtual
>> environments, but ad-hoc importing is another another thing. It isn't
>> meant for delivering real projects. Just for testing modules without the
>> need to download them, maybe install them, and all.

could be equally well handled by pip-installing httpimport itself, and
using that to bootstrap your testing procedures. Unless, of course,
you're wanting to httpimport httpimport, in which case you're going to
run into bootstrapping problems whichever way you do it :)

I think we're on the same page here, but it definitely needs some more
text in the README to explain this - particularly how this is not a
replacement for pip. For example, my first thought on seeing this was
"wow, that's going to be abysmally slow unless it has a cache", but
the answer to that is: if you need a cache, you probably should be
using pip to install things properly.

Still -1 on this becoming a stdlib package, as there's nothing I've
yet seen that can't be done as a third-party package. But it's less
scary than I thought it was :)

ChrisA
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Remote package/module imports through HTTP/S

[John Torakis]

Bounced back on list



 Forwarded Message 
Θέμα:   Re: [Python-ideas] Remote package/module imports through HTTP/S
Ημερομηνία: Wed, 23 Aug 2017 20:36:19 +0300
Από:John Torakis <john.tora...@gmail.com>
Προς:   Chris Angelico <ros...@gmail.com>



Yeah, I am a security researcher, I am keen on backdoor programming and
staging and all that! It is my official job and research topic! I go to
the office and code such stuff! I am not a blackhat, nor a security
enthusiast, it is my job.


First of all, let's all agree that if someone can run Python code in
your computer you are 100% hacked! It is irrelevant if "httpimport" is a
core python feature or not in that case.

Now, I agree that this can be exploited if used under plain HTTP, it is
a MiTM -> Remote code execution case. I admit that this is not bright.
But I mention that this can be used in testing.

On the topic of HTTPS, man-in-the-middle is not possible without
previous Trusted Certificate compromise. Github can be trusted 100%
percent for example. A certificate check has to take place in the HTTPS
remote loading for sure!

When I said a "core feature" I meant that the "httpimport" module would
deliver with the core modules. Not that the Finder/Loader has to be in
the list of Finders/Loaders that are used by default! For god sake, I
wouldn't like my PC to start probing for modules just because I mistyped
an import line!

I know that pip works nicely, especially when paired with virtual
environments, but ad-hoc importing is another another thing. It isn't
meant for delivering real projects. Just for testing modules without the
need to download them, maybe install them, and all.


Thank you for your time,
John Torakis


On 23/08/2017 20:17, Chris Angelico wrote:
> On Thu, Aug 24, 2017 at 2:55 AM, John Torakis <john.tora...@gmail.com> wrote:
>> Hello all!
>>
>> Today I opened an issue in bugs.python.org
>> (http://bugs.python.org/issue31264) proposing a module I created for
>> remote package/module imports through standard HTTP/S.
>>
>> The concept is that, if a directory is served through HTTP/S (the way
>> SimpleHTTPServer module serves directories), a Finder/Loader object can
>> fetch Python files from that directory using HTTP requests, and finally
>> load them as modules (or packages) in the running namespace.
>>
>> The repo containing a primitive (but working) version of the
>> Finder/Loader, also contains self explanatory examples (in the README.md):
>>
>> https://github.com/operatorequals/httpimport
>>
>>
>> My proposal is that this module can become a core Python feature,
>> providing a way to load modules even from Github.com repositories,
>> without the need to "git clone - setup.py install" them.
>>
>>
>> Other languages, like golang, provide this functionality from their
>> early days (day one?). Python development can be greatly improved if a
>> "try before pip installing" mechanism gets in place, as it will add a
>> lot to the REPL nature of the testing/experimenting process.
> As a core feature? No no no no no no no no. Absolutely do NOT WANT
> THIS. This is a security bug magnet; can you imagine trying to ensure
> that malicious code is not executed, in an arbitrary execution
> context? As an explicitly-enabled feature, it's a lot less hairy than
> a permanently-active one (can you IMAGINE how terrifying that would
> be?), but even so, trying to prove that addRemoteRepo (not a
> PEP8-compliant name, btw) is getting the correct code is not going to
> be easy. You have to (a) drop HTTP altogether and mandate SSL and (b)
> be absolutely sure that your certificate chains are 100% dependable,
> which - as we've seen recently - is a nontrivial task.
>
> The easiest way to add remote code is pip. For most packages, that's
> what you want to be using:
>
> pip install requests
>
> will make "import requests" functional. I don't see pip mentioned
> anywhere in your README, but you do mention the testing of pull
> requests, so at very least, this wants some explanatory screed.
>
> But I'm not entirely sure I want to support this. You're explicitly
> talking about using this with the creation of backdoors... in what,
> exactly? What are you actually getting at here?
>
> ChrisA
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/


___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Remote package/module imports through HTTP/S

[John Torakis]

On 23/08/2017 20:36, John Torakis wrote:
> Yeah, I am a security researcher, I am keen on backdoor programming and
> staging and all that! It is my official job and research topic! I go to
> the office and code such stuff! I am not a blackhat, nor a security
> enthusiast, it is my job.
>
>
> First of all, let's all agree that if someone can run Python code in
> your computer you are 100% hacked! It is irrelevant if "httpimport" is a
> core python feature or not in that case.
>
> Now, I agree that this can be exploited if used under plain HTTP, it is
> a MiTM -> Remote code execution case. I admit that this is not bright.
> But I mention that this can be used in testing.
>
> On the topic of HTTPS, man-in-the-middle is not possible without
> previous Trusted Certificate compromise. Github can be trusted 100%
> percent for example. A certificate check has to take place in the HTTPS
> remote loading for sure!
>
> When I said a "core feature" I meant that the "httpimport" module would
> deliver with the core modules. Not that the Finder/Loader has to be in
> the list of Finders/Loaders that are used by default! For god sake, I
> wouldn't like my PC to start probing for modules just because I mistyped
> an import line!
>
> I know that pip works nicely, especially when paired with virtual
> environments, but ad-hoc importing is another another thing. It isn't
> meant for delivering real projects. Just for testing modules without the
> need to download them, maybe install them, and all.
>
>
> Thank you for your time,
> John Torakis
>
>
> On 23/08/2017 20:17, Chris Angelico wrote:
>> On Thu, Aug 24, 2017 at 2:55 AM, John Torakis  wrote:
>>> Hello all!
>>>
>>> Today I opened an issue in bugs.python.org
>>> (http://bugs.python.org/issue31264) proposing a module I created for
>>> remote package/module imports through standard HTTP/S.
>>>
>>> The concept is that, if a directory is served through HTTP/S (the way
>>> SimpleHTTPServer module serves directories), a Finder/Loader object can
>>> fetch Python files from that directory using HTTP requests, and finally
>>> load them as modules (or packages) in the running namespace.
>>>
>>> The repo containing a primitive (but working) version of the
>>> Finder/Loader, also contains self explanatory examples (in the README.md):
>>>
>>> https://github.com/operatorequals/httpimport
>>>
>>>
>>> My proposal is that this module can become a core Python feature,
>>> providing a way to load modules even from Github.com repositories,
>>> without the need to "git clone - setup.py install" them.
>>>
>>>
>>> Other languages, like golang, provide this functionality from their
>>> early days (day one?). Python development can be greatly improved if a
>>> "try before pip installing" mechanism gets in place, as it will add a
>>> lot to the REPL nature of the testing/experimenting process.
>> As a core feature? No no no no no no no no. Absolutely do NOT WANT
>> THIS. This is a security bug magnet; can you imagine trying to ensure
>> that malicious code is not executed, in an arbitrary execution
>> context? As an explicitly-enabled feature, it's a lot less hairy than
>> a permanently-active one (can you IMAGINE how terrifying that would
>> be?), but even so, trying to prove that addRemoteRepo (not a
>> PEP8-compliant name, btw) is getting the correct code is not going to
>> be easy. You have to (a) drop HTTP altogether and mandate SSL and (b)
>> be absolutely sure that your certificate chains are 100% dependable,
>> which - as we've seen recently - is a nontrivial task.
>>
>> The easiest way to add remote code is pip. For most packages, that's
>> what you want to be using:
>>
>> pip install requests
>>
>> will make "import requests" functional. I don't see pip mentioned
>> anywhere in your README, but you do mention the testing of pull
>> requests, so at very least, this wants some explanatory screed.
>>
>> But I'm not entirely sure I want to support this. You're explicitly
>> talking about using this with the creation of backdoors... in what,
>> exactly? What are you actually getting at here?
>>
>> ChrisA
>> ___
>> Python-ideas mailing list
>> Python-ideas@python.org
>> https://mail.python.org/mailman/listinfo/python-ideas
>> Code of Conduct: http://python.org/psf/codeofconduct/
>

___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Remote package/module imports through HTTP/S

[Oleg Broytman]

Hi!

On Wed, Aug 23, 2017 at 07:55:00PM +0300, John Torakis  
wrote:
> Hello all!
> 
> Today I opened an issue in bugs.python.org
> (http://bugs.python.org/issue31264) proposing a module I created for
> remote package/module imports through standard HTTP/S.

   The issue is so big IMO it requires a PEP, not just an issue.

   Anyway I'm -1000 for reasons of security, connectivity (not all hosts
are connected), traffic cost and speed.

> The concept is that, if a directory is served through HTTP/S (the way
> SimpleHTTPServer module serves directories), a Finder/Loader object can
> fetch Python files from that directory using HTTP requests, and finally
> load them as modules (or packages) in the running namespace.
> 
> The repo containing a primitive (but working) version of the
> Finder/Loader, also contains self explanatory examples (in the README.md):
> 
> https://github.com/operatorequals/httpimport
> 
> 
> My proposal is that this module can become a core Python feature,
> providing a way to load modules even from Github.com repositories,
> without the need to "git clone - setup.py install" them.
> 
> 
> Other languages, like golang, provide this functionality from their

   AFAIK Go downloads modules at compile time, not run time. This is a
major distiction with Python.

> early days (day one?). Python development can be greatly improved if a
> "try before pip installing" mechanism gets in place, as it will add a
> lot to the REPL nature of the testing/experimenting process.
> 
> 
> 
> Thank you for your time,
> 
> John Torakis, IT Security Researcher
> 
> 
> 
> P.S: It is my first time in this mailing list and generally Python
> contribution. Please be tolerant!

Oleg.
-- 
 Oleg Broytmanhttp://phdru.name/p...@phdru.name
   Programmers don't die, they just GOSUB without RETURN.
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Remote package/module imports through HTTP/S

[Chris Angelico]

On Thu, Aug 24, 2017 at 2:55 AM, John Torakis  wrote:
> Hello all!
>
> Today I opened an issue in bugs.python.org
> (http://bugs.python.org/issue31264) proposing a module I created for
> remote package/module imports through standard HTTP/S.
>
> The concept is that, if a directory is served through HTTP/S (the way
> SimpleHTTPServer module serves directories), a Finder/Loader object can
> fetch Python files from that directory using HTTP requests, and finally
> load them as modules (or packages) in the running namespace.
>
> The repo containing a primitive (but working) version of the
> Finder/Loader, also contains self explanatory examples (in the README.md):
>
> https://github.com/operatorequals/httpimport
>
>
> My proposal is that this module can become a core Python feature,
> providing a way to load modules even from Github.com repositories,
> without the need to "git clone - setup.py install" them.
>
>
> Other languages, like golang, provide this functionality from their
> early days (day one?). Python development can be greatly improved if a
> "try before pip installing" mechanism gets in place, as it will add a
> lot to the REPL nature of the testing/experimenting process.

As a core feature? No no no no no no no no. Absolutely do NOT WANT
THIS. This is a security bug magnet; can you imagine trying to ensure
that malicious code is not executed, in an arbitrary execution
context? As an explicitly-enabled feature, it's a lot less hairy than
a permanently-active one (can you IMAGINE how terrifying that would
be?), but even so, trying to prove that addRemoteRepo (not a
PEP8-compliant name, btw) is getting the correct code is not going to
be easy. You have to (a) drop HTTP altogether and mandate SSL and (b)
be absolutely sure that your certificate chains are 100% dependable,
which - as we've seen recently - is a nontrivial task.

The easiest way to add remote code is pip. For most packages, that's
what you want to be using:

pip install requests

will make "import requests" functional. I don't see pip mentioned
anywhere in your README, but you do mention the testing of pull
requests, so at very least, this wants some explanatory screed.

But I'm not entirely sure I want to support this. You're explicitly
talking about using this with the creation of backdoors... in what,
exactly? What are you actually getting at here?

ChrisA
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/