Re: [Python-ideas] PEP 550 v2

[Guido van Rossum]

OK, I get it now. I really liked the analysis of existing uses in Django.
So no worries about this.

On Wed, Aug 23, 2017 at 5:36 PM, Yury Selivanov 
wrote:

> There's another "major" problem with theading.local()-like API for PEP
> 550: C API.
>
> threading.local() in C right now is PyThreadState_GetDict(), which
> returns a dictionary for the current thread, that can be
> queried/modified with PyDict_* functions.  For PEP 550 this would not
> work.
>
> The advantage of the current ContextKey solution is that the Python
> API and C API are essentially the same: [1]
>
> Another advantage, is that ContextKey implements a better caching,
> because it can have only one value cached in it, see [2] for details.
>
> [1] https://www.python.org/dev/peps/pep-0550/#new-apis
> [2] https://www.python.org/dev/peps/pep-0550/#contextkey-get-cache
>
> Yury
>



-- 
--Guido van Rossum (python.org/~guido)
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Remote package/module imports through HTTP/S

[Stephen J. Turnbull]

John Torakis writes:

 > But, as it seems like it is a very big feature (to me at least),

And "pip install httpimport" seems like it is a very small burden (to
me at least).

I agree with Paul Moore.  Putting this in the stdlib seems both
unnecessary, given pip, and an attractive nuisance for naive users.

>From the point of view of the blue team, checking for mere presence of
httpimport in the environment is indicative of danger if it's
pip-able, useless if it's in the stdlib.

With respect to "it just makes exec(urlopen()) easier", any code must
be audited for application of exec() to user input anyway, regardless
of whether it fetches stuff off the Internet.  Adding httpimport use
to the checklist adds a little bit of complexity to *every* security
check, and a fair amount of danger in security-oblivious environments
such as many university labs, and I would imagine many corporate
development groups as well.

YMMV, but from the point of view of the larger, security-conscious
organization, I would say -1.  It's an attractive nuisance unless
you're a security person, and then pip is not a big deal.

Steve

___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Remote package/module imports through HTTP/S

[Chris Angelico]

On Thu, Aug 24, 2017 at 12:13 PM, Stephen J. Turnbull
 wrote:
> Chris Angelico writes:
>
>  > If you're worried about the latter, don't use httpimport.
>
> I guarantee you that in my (university) environment, if httpimport is
> in the stdlib, its use will be rampant (and not just by students, but
> by security-oblivious faculty).  I want to be able to walk up to a
> student, say "may I?" and type "python -m httpimport" to determine if
> that particular risky behavior is a worry.  Because *I'm* liable for
> my students' PCs' behavior on the network.
>
> Personally speaking, +1 on PyPI, -100 on stdlib.

Agreed, and a VERY good reason for this to be an explicitly-installed
package. By its nature, it won't be a dependency of other packages, so
keeping it out of the stdlib pretty much guarantees that it'll only be
available if it's been called for by name.

ChrisA
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Remote package/module imports through HTTP/S

[Stephen J. Turnbull]

Chris Angelico writes:

 > If you're worried about the latter, don't use httpimport.

I guarantee you that in my (university) environment, if httpimport is
in the stdlib, its use will be rampant (and not just by students, but
by security-oblivious faculty).  I want to be able to walk up to a
student, say "may I?" and type "python -m httpimport" to determine if
that particular risky behavior is a worry.  Because *I'm* liable for
my students' PCs' behavior on the network.

Personally speaking, +1 on PyPI, -100 on stdlib.

Steve

___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] PEP 550 v2

[Yury Selivanov]

There's another "major" problem with theading.local()-like API for PEP
550: C API.

threading.local() in C right now is PyThreadState_GetDict(), which
returns a dictionary for the current thread, that can be
queried/modified with PyDict_* functions.  For PEP 550 this would not
work.

The advantage of the current ContextKey solution is that the Python
API and C API are essentially the same: [1]

Another advantage, is that ContextKey implements a better caching,
because it can have only one value cached in it, see [2] for details.

[1] https://www.python.org/dev/peps/pep-0550/#new-apis
[2] https://www.python.org/dev/peps/pep-0550/#contextkey-get-cache

Yury
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] PEP 550 v2

[Nathaniel Smith]

On Wed, Aug 23, 2017 at 8:41 AM, Guido van Rossum  wrote:
> If we're extending the analogy with thread-locals we should at least
> consider making each instantiation return a namespace rather than something
> holding a single value. We have
>
> log_state = threading.local()
> log_state.verbose = False
>
> def action(x):
> if log_state.verbose:
> print(x)
>
> def make_verbose():
> log_state.verbose = True
>
> It would be nice if we could upgrade this to make it PEP 550-aware so that
> only the first line needs to change:
>
> log_state = sys.AsyncLocal("log state")
> # The rest is the same

You can mostly implement this on top of the current PEP 550. Something like:

_tombstone = object()

class AsyncLocal:
def __getattribute__(self, name):
# if this raises AttributeError, we let it propagate
key = object.__getattribute__(self, name)
value = key.get()
if value is _tombstone:
raise AttributeError(name)
return value

def __setattr__(self, name, value):
try:
key = object.__getattribute__(self, name)
except AttributeError:
with some_lock:
 # double-checked locking pattern
 try:
 key = object.__getattribute__(self, name)
 except AttributeError:
 key = new_context_key()
 object.__setattr__(self, name, key)
key.set(value)

def __delattr__(self, name):
self.__setattr__(name, _tombstone)

def __dir__(self):
# filter out tombstoned values
return [name for name in object.__dir__(self) if hasattr(self, name)]

Issues:

Minor problem: On threading.local you can use .__dict__ to get the
dict. That doesn't work here. But this could be done by returning a
mapping proxy type, or maybe it's better not to support at all -- I
don't think it's a big issue.

Major problem: An attribute setting/getting API doesn't give any way
to solve the save/restore problem [1]. PEP 550 v3 doesn't have a
solution to this yet either, but we know we can do it by adding some
methods to context-key. Supporting this in AsyncLocal is kinda
awkward, since you can't use methods on the object -- I guess you
could have some staticmethods, like
AsyncLocal.save_state(my_async_local, name) and
AsyncLocal.restore_state(my_async_local, name, value)? In any case
this kinda spoils the sense of like "oh it's just an object with
attributes, I already know how this works".

Major problem: There are two obvious implementations. The above uses a
separate ContextKey for each entry in the dict; the other way would be
to have a single ContextKey that holds a dict. They have subtly
different semantics. Suppose you have a generator and inside it you
assign to my_async_local.a but not to my_async_local.b, then yield,
and then the caller assigns to my_async_local.b. Is this visible
inside the generator? In the ContextKey-holds-an-attribute approach,
the answer is "yes": each AsyncLocal is a bag of independent
attributes. In the ContextKey-holds-a-dict approach, the answer is
"no": each AsyncLocal is a single container holding a single piece of
(complex) state. It isn't obvious to me which of these semantics is
preferable – maybe it is if you're Dutch :-). But there's a danger
that either option leaves a bunch of people confused.

(Tangent: in the ContextKey-holds-a-dict approach, currently you have
to copy the dict before mutating it every time, b/c PEP 550 currently
doesn't provide a way to tell whether the value returned by get() came
from the top of the stack, and thus is private to you and can be
mutated in place, or somewhere deeper, and thus is shared and
shouldn't be mutated. But we should fix that anyway, and anyway
copy-the-mutate is a viable approach.)

Observation: I don't think there's any simpler way to implement
AsyncLocal other than to start with machinery like what PEP 550
already proposes, and then layer something like the above on top of
it. We could potentially hide the layers inside the interpreter and
only expose AsyncLocal, but I don't think it really simplifies the
implementation any.

Observation: I feel like many users of threading.local -- possibly the
majority -- only put a single attribute on each object anyway, so for
those users a raw ContextKey API is actually more natural and faster.
For example, looking through the core django repo, I see thread locals
in

- django.utils.timezone._active
- django.utils.translation.trans_real._active
- django.urls.base._prefixes
- django.urls.base._urlconfs
- django.core.cache._caches
- django.urls.resolvers.RegexURLResolver._local
- django.contrib.gis.geos.prototypes.threadsafe.thread_context
- django.contrib.gis.geos.prototypes.io.thread_context
- django.db.utils.ConnectionHandler._connections

Of these 9 thread-local objects, 7 of them have only a single
attribute; only the last 2 use multiple attributes. For the first 4,
that 

Re: [Python-ideas] Please consider adding context manager versions of setUp/tearDown to unittest.TestCase

[Chris Barker]

On Tue, Aug 22, 2017 at 7:05 PM, Neil Girdhar  wrote:

> Like you, I used nose and then switched to pytest.  The reason I proposed
> this for unittest is because pytest and nose and (I think) most of the
> other testing frameworks inherit from unittest,
>

not really -- they extend unittest -- in the sense that their test runners
can be used with unittest TestCases -- but they don't depend on unitest.

so improving unittest has downstream benefits.
>

only to those using unittest -- a lot of folks do use pytest or nose
primarily as a test runner, so those folks would benefit.

I may nevertheless propose this to the pytest people if this doesn't make
> it into unittest.
>

Anyway, I'm just being a curmudgeon -- if folks think it would be useful
and not disruptive, then why not?

-CHB







> On Tue, Aug 22, 2017 at 8:26 PM Chris Barker 
> wrote:
>
>> On Tue, Aug 22, 2017 at 5:19 PM, Chris Barker 
>> wrote:
>>
>>> anyway, that's enough ranting.
>>>
>>
>> Got carried away with the ranting, and didn't flesh out my point.
>>
>> My point is that unittest is a very static, not very pythonic framework
>> -- if you are productive with it, great, but I don't think it's worth
>> trying to add more pythonic niceties to. Chances are pytest (Or nose2?) may
>> already have them, or, if not, the simpler structure of pytest tests make
>> them easier to write yourself.
>>
>> -CHB
>>
>> --
>>
>> Christopher Barker, Ph.D.
>> Oceanographer
>>
>> Emergency Response Division
>> NOAA/NOS/OR(206) 526-6959   voice
>> 7600 Sand Point Way NE   (206) 526-6329   fax
>> Seattle, WA  98115   (206) 526-6317   main reception
>>
>> chris.bar...@noaa.gov
>>
>> --
>>
>> ---
>> You received this message because you are subscribed to a topic in the
>> Google Groups "python-ideas" group.
>> To unsubscribe from this topic, visit https://groups.google.com/d/
>> topic/python-ideas/cF_4IlJq698/unsubscribe.
>> To unsubscribe from this group and all its topics, send an email to
>> python-ideas+unsubscr...@googlegroups.com.
>> For more options, visit https://groups.google.com/d/optout.
>> ___
>> Python-ideas mailing list
>> Python-ideas@python.org
>> https://mail.python.org/mailman/listinfo/python-ideas
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
>> --
>>
>> ---
>> You received this message because you are subscribed to a topic in the
>> Google Groups "python-ideas" group.
>> To unsubscribe from this topic, visit https://groups.google.com/d/
>> topic/python-ideas/cF_4IlJq698/unsubscribe.
>> To unsubscribe from this group and all its topics, send an email to
>> python-ideas+unsubscr...@googlegroups.com.
>> For more options, visit https://groups.google.com/d/optout.
>>
>
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
>


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Please consider adding context manager versions of setUp/tearDown to unittest.TestCase

[Neil Girdhar]

On Wed, Aug 23, 2017 at 3:31 AM Nick Coghlan  wrote:

> On 23 August 2017 at 08:20, rym...@gmail.com  wrote:
> > TBH you're completely right. Every time I see someone using unittest
> > andItsHorriblyUnpythonicNames, I want to kill a camel.
> >
> > Sometimes, though, I feel like part of the struggle is the alternative.
> If
> > you dislike unittest, but pytest is too "magical" for you, what do you
> use?
> > Many Python testing tools like nose are just test *runners*, so you still
> > need something else. In the end, many just end up back at unittest, maybe
> > with nose on top.
>
> A snake_case helper API for unittest that I personally like is
> hamcrest, since that also separates out the definition of testing
> assertions from being part of a test case:
> https://pypi.python.org/pypi/PyHamcrest
>
> Introducing such a split natively into unittest is definitely
> attractive, but would currently be difficult due to the way that some
> features like self.maxDiff and self.subTest work.
>
> However, PEP 550's execution contexts may provide a way to track the
> test state reliably that's independent of being a method on a test
> case instance, in which case it would become feasible to offer a more
> procedural interface in addition to the current visibly
> object-oriented one.
>

If you have time, could you expand on that a little bit?


>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
> --
>
> ---
> You received this message because you are subscribed to a topic in the
> Google Groups "python-ideas" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/python-ideas/cF_4IlJq698/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> python-ideas+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Remote package/module imports through HTTP/S

[John Torakis]

On 23/08/2017 22:06, Chris Angelico wrote:
> On Thu, Aug 24, 2017 at 5:04 AM, John Torakis  wrote:
>> Dark times...
>>
>> So is it a "case closed", or is there any improvement that will make it
>> worth it to be an stdlib module?
>>
>> I mean, times have changed from 1995, and I am not referring to HTTPS
>> invention. This is the reason that makes httpimport just tolerable
>> security-wise.
>>
>> I'm talking about the need to rapidly test public code. I insist that
>> testing code available on Github (or other repos), without the
>> venv/clone/install hassle is a major improvement in my (and most sec
>> researchers' I know) Python workflow. It makes REPL prototyping million
>> times smoother.
>> We all have created small scripts that auto load modules from URLs anyway.
>> That's why I thought that this modules falls under the second category of
>> 20.2.1 in https://docs.python.org/devguide/stdlibchanges.html (I did my
>> homework before getting to mail in this list).
>>
>> So, if there is something that would make this module acceptable for stdlib,
>> please let me know! I'd more than happily reform it and make it comply with
>> Python stdlib requirements.
> Why can't people just "pip install httpimport" to make use of it? Why
> does it need to be in the stdlib?
It doesn't, strictly speaking, *need* to be in stdlib. Of course it
doesn't! Python is good enough without it.
But, as it seems like it is a very big feature (to me at least), it
feels right to be "officially" a Python feature. It feels right to be
officially supported and not just another module, as it extends core
import functionality. Just like zipimport does. Zipimport could just be
a 3rd party module too. But it is in the core, and I can see why.

Anyway, I will post it to PyPI when I finalize Github support and extend
the testing a little bit. I will then shoot a mail again and repropose
the module when it reaches full maturity.

>
> ChrisA
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/

Thank you all for your time!
John Torakis

___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Remote package/module imports through HTTP/S

[Chris Angelico]

On Thu, Aug 24, 2017 at 5:04 AM, John Torakis  wrote:
> Dark times...
>
> So is it a "case closed", or is there any improvement that will make it
> worth it to be an stdlib module?
>
> I mean, times have changed from 1995, and I am not referring to HTTPS
> invention. This is the reason that makes httpimport just tolerable
> security-wise.
>
> I'm talking about the need to rapidly test public code. I insist that
> testing code available on Github (or other repos), without the
> venv/clone/install hassle is a major improvement in my (and most sec
> researchers' I know) Python workflow. It makes REPL prototyping million
> times smoother.
> We all have created small scripts that auto load modules from URLs anyway.
> That's why I thought that this modules falls under the second category of
> 20.2.1 in https://docs.python.org/devguide/stdlibchanges.html (I did my
> homework before getting to mail in this list).
>
> So, if there is something that would make this module acceptable for stdlib,
> please let me know! I'd more than happily reform it and make it comply with
> Python stdlib requirements.

Why can't people just "pip install httpimport" to make use of it? Why
does it need to be in the stdlib?

ChrisA
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Remote package/module imports through HTTP/S

[John Torakis]

Dark times...

So is it a "case closed", or is there any improvement that will make it
worth it to be an stdlib module?

I mean, times have changed from 1995, and I am not referring to HTTPS
invention. This is the reason that makes httpimport just tolerable
security-wise.

I'm talking about the need to rapidly test public code. I insist that
testing code available on Github (or other repos), without the
venv/clone/install hassle is a major improvement in my (and most sec
researchers' I know) Python workflow. It makes REPL prototyping million
times smoother.
We all have created small scripts that auto load modules from URLs
anyway. That's why I thought that this modules falls under the second
category of 20.2.1 in
https://docs.python.org/devguide/stdlibchanges.html (I did my homework
before getting to mail in this list).

So, if there is something that would make this module acceptable for
stdlib, please let me know! I'd more than happily reform it and make it
comply with Python stdlib requirements.

John Torakis

On 23/08/2017 21:48, Guido van Rossum wrote:
> For security reasons. AFAIK HTTPS wasn't even invented at the time.
>
> On Wed, Aug 23, 2017 at 11:44 AM, John Torakis  > wrote:
>
>
>
> On 23/08/2017 21:41, Guido van Rossum wrote:
>> This isn't ever going to be a standard feature. It's available as
>> a third-party package and that's fine.
>>
>> I'd like to add a historic note -- this was first proposed around
>> 1995 by Michael McLay. (Sorry, I don't have an email sitting
>> around, but I'm sure he brought this up at or around the first
>> Python workshop at NIST in 1995 -- I was his guest at NIST for
>> several months at the time.)
>>
> Woah! I was 2 years old at that time! Little did I know!
> Can I ask why it got rejected the first time?
>> -- 
>> --Guido van Rossum (python.org/~guido )
>>
>>
>> ___
>> Python-ideas mailing list
>> Python-ideas@python.org 
>> https://mail.python.org/mailman/listinfo/python-ideas
>> 
>> Code of Conduct: http://python.org/psf/codeofconduct/
>> 
> ___ Python-ideas
> mailing list Python-ideas@python.org
> 
> https://mail.python.org/mailman/listinfo/python-ideas
>  Code of
> Conduct: http://python.org/psf/codeofconduct/
>  
>
> -- 
> --Guido van Rossum (python.org/~guido )
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Remote package/module imports through HTTP/S

[Guido van Rossum]

For security reasons. AFAIK HTTPS wasn't even invented at the time.

On Wed, Aug 23, 2017 at 11:44 AM, John Torakis 
wrote:

>
>
> On 23/08/2017 21:41, Guido van Rossum wrote:
>
> This isn't ever going to be a standard feature. It's available as a
> third-party package and that's fine.
>
> I'd like to add a historic note -- this was first proposed around 1995 by
> Michael McLay. (Sorry, I don't have an email sitting around, but I'm sure
> he brought this up at or around the first Python workshop at NIST in 1995
> -- I was his guest at NIST for several months at the time.)
>
> Woah! I was 2 years old at that time! Little did I know!
> Can I ask why it got rejected the first time?
>
> --
> --Guido van Rossum (python.org/~guido )
>
>
> ___
> Python-ideas mailing 
> listPython-ideas@python.orghttps://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
>
>
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
>


-- 
--Guido van Rossum (python.org/~guido)
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Remote package/module imports through HTTP/S

[John Torakis]

On 23/08/2017 21:41, Guido van Rossum wrote:
> This isn't ever going to be a standard feature. It's available as a
> third-party package and that's fine.
>
> I'd like to add a historic note -- this was first proposed around 1995
> by Michael McLay. (Sorry, I don't have an email sitting around, but
> I'm sure he brought this up at or around the first Python workshop at
> NIST in 1995 -- I was his guest at NIST for several months at the time.)
>
Woah! I was 2 years old at that time! Little did I know!
Can I ask why it got rejected the first time?
> -- 
> --Guido van Rossum (python.org/~guido )
>
>
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/

___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Remote package/module imports through HTTP/S

[John Torakis]

On 23/08/2017 21:24, Paul Moore wrote:
> On 23 August 2017 at 18:49, Chris Angelico  wrote:
>> Still -1 on this becoming a stdlib package, as there's nothing I've
>> yet seen that can't be done as a third-party package. But it's less
>> scary than I thought it was :)
> IMO, this would make a great 3rd party package (I note that it's not
> yet published on PyPI). It's possible that it would end up being
> extremely popular, and recognised as sufficiently secure - at which
> point it may be worth considering for core inclusion. But it's also
> possible that it remains niche, and/or people aren't willing to take
> the security risks that it implies, in which case it's still useful to
> those who do like it.
PyPI upload is scheduled when some more testing and commenting takes place.
> One aspect that hasn't been mentioned yet - as a 3rd party module, the
> user (or the organisation's security team) can control whether or not
> the ability to import over the web is available by controlling whether
> the module is allowed to be installed - whereas with a core module,
> it's there, like it or not, and *all* Python code has to be audited on
> the assumption that it might be used. 
True!
But you can urlopen()->exec() anything out there anyway! A ">>>" prompt
is all you need.
> I could easily imagine cases
> where the httpimport module was allowed on development machines and CI
> servers, but forbidden on production (and pre-production) systems.
> That option simply isn't available if the feature is in the core.
I agree that there are circumstances that this module should not be used
(regardless of security implications).
In a released product for example. Depending on the UP-ness of a remote
repository (e.g github), not to even mention the API
backward-compatibility of an upstream package, is just **BAD** for a
ready-released-deliverable product! This is why we have virtual
environments!

But it remains an option to use it or not! I, for example, find myself
REPLing more than scripting. When REPLing for something you plan to
implement sometime-somehow, this module is really what you need! But
when I finally create a script, I won't disable its offline
functionality just to use httpimport. That would be suicidal! When I
finally come with a working thing I will finally land the used packages
to disk and to a virtual environment.


My argument is that this module will add greatly to the Python's ad-hoc
testing capabilities! I find it elegant for such feature to be in the
stdlib of a language.
I don't doubt that it can survive as a 3rd party module, though.
>
> Paul
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/


___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Remote package/module imports through HTTP/S

[Guido van Rossum]

 This isn't ever going to be a standard feature. It's available as a
third-party package and that's fine.

I'd like to add a historic note -- this was first proposed around 1995 by
Michael McLay. (Sorry, I don't have an email sitting around, but I'm sure
he brought this up at or around the first Python workshop at NIST in 1995
-- I was his guest at NIST for several months at the time.)

-- 
--Guido van Rossum (python.org/~guido)
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Remote package/module imports through HTTP/S

[Paul Moore]

On 23 August 2017 at 18:49, Chris Angelico  wrote:
> Still -1 on this becoming a stdlib package, as there's nothing I've
> yet seen that can't be done as a third-party package. But it's less
> scary than I thought it was :)

IMO, this would make a great 3rd party package (I note that it's not
yet published on PyPI). It's possible that it would end up being
extremely popular, and recognised as sufficiently secure - at which
point it may be worth considering for core inclusion. But it's also
possible that it remains niche, and/or people aren't willing to take
the security risks that it implies, in which case it's still useful to
those who do like it.

One aspect that hasn't been mentioned yet - as a 3rd party module, the
user (or the organisation's security team) can control whether or not
the ability to import over the web is available by controlling whether
the module is allowed to be installed - whereas with a core module,
it's there, like it or not, and *all* Python code has to be audited on
the assumption that it might be used. I could easily imagine cases
where the httpimport module was allowed on development machines and CI
servers, but forbidden on production (and pre-production) systems.
That option simply isn't available if the feature is in the core.

Paul
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Remote package/module imports through HTTP/S

[John Torakis]

On 23/08/2017 21:11, Chris Angelico wrote:
> On Thu, Aug 24, 2017 at 4:04 AM, Bruce Leban  wrote:
>> On Wed, Aug 23, 2017 at 10:37 AM, John Torakis 
>> wrote:
>>>
>>> Github can be trusted 100% percent for example.
>>
>> This isn't even remotely close to true. While I'd agree with the statement
>> that the SSL cert on github is reasonably trustworthy, the *content* on
>> github is NOT trustworthy and that's where the security risk is.
>>
>> I agree that this is a useful feature and there is no way it should be on by
>> default. The right way IMHO to do this is to have a command line option
>> something like this:
>>
>> python --http-import somelib=https://github.com/someuser/somelib
> If you read his README, it's pretty explicit about URLs; the risk is
> that "https://github.com/someuser/somelib; can be intercepted, not
> that "someuser" is malicious. If you're worried about the latter,
> don't use httpimport.

Again, if https://github.com/someuser/somelib can be intercepted,
https://pypi.python.org/pypi can too.
If HTTPS is intercepted so easily (when not used from browsers) we are
f**ed...
>
> ChrisA
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/

___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Remote package/module imports through HTTP/S

[John Torakis]

On 23/08/2017 20:49, Chris Angelico wrote:
> On Thu, Aug 24, 2017 at 3:37 AM, John Torakis  wrote:
>>
>> On 23/08/2017 20:36, John Torakis wrote:
>>> Yeah, I am a security researcher, I am keen on backdoor programming and
>>> staging and all that! It is my official job and research topic! I go to
>>> the office and code such stuff! I am not a blackhat, nor a security
>>> enthusiast, it is my job.
>>>
>>>
>>> First of all, let's all agree that if someone can run Python code in
>>> your computer you are 100% hacked! It is irrelevant if "httpimport" is a
>>> core python feature or not in that case.
>>>
>>> Now, I agree that this can be exploited if used under plain HTTP, it is
>>> a MiTM -> Remote code execution case. I admit that this is not bright.
>>> But I mention that this can be used in testing.
>>>
>>> On the topic of HTTPS, man-in-the-middle is not possible without
>>> previous Trusted Certificate compromise. Github can be trusted 100%
>>> percent for example. A certificate check has to take place in the HTTPS
>>> remote loading for sure!
> Right, but that just pushes the problem one level further out: you
> need to have a 100% dependable certificate chain. And that means
> absolutely completely trusting all of your root certificates, and it
> also means either not needing to add any _more_ root certificates, or
> being able to configure the cert store. As we've seen elsewhere, this
> is nontrivial.
The centralized PKI as we know it is a pain altogether. Please let me
reference this XKCD strip here:
https://xkcd.com/1200/
Running code to your computer is among the low Impact things that can
happen to you if you have a compromised certificate store. Trust me on that!
In other words, if you can't trust your certificate store now, and you
are afraid of Remote code execution through HTTPS, stop using pip
altogether:
https://github.com/pypa/pip/issues/1168

And most Package Managers anyway..
>>> When I said a "core feature" I meant that the "httpimport" module would
>>> deliver with the core modules. Not that the Finder/Loader has to be in
>>> the list of Finders/Loaders that are used by default! For god sake, I
>>> wouldn't like my PC to start probing for modules just because I mistyped
>>> an import line!
> Glad we agree about that! I have seen people wanting all sorts of
> things to become core features (usually for the sake of interactive
> work), and a lot of it is MUCH better handled as a non-core feature.
>
> Though a lot of what you're saying here - especially this:
>
>>> I know that pip works nicely, especially when paired with virtual
>>> environments, but ad-hoc importing is another another thing. It isn't
>>> meant for delivering real projects. Just for testing modules without the
>>> need to download them, maybe install them, and all.
> could be equally well handled by pip-installing httpimport itself, and
> using that to bootstrap your testing procedures. Unless, of course,
> you're wanting to httpimport httpimport, in which case you're going to
> run into bootstrapping problems whichever way you do it :)
I will never open an issue for 'httpimporting the httpimport itself' It
is a promise!
>
> I think we're on the same page here, but it definitely needs some more
> text in the README to explain this - particularly how this is not a
> replacement for pip. For example, my first thought on seeing this was
> "wow, that's going to be abysmally slow unless it has a cache", but
> the answer to that is: if you need a cache, you probably should be
> using pip to install things properly.
>
> Still -1 on this becoming a stdlib package, as there's nothing I've
> yet seen that can't be done as a third-party package. But it's less
> scary than I thought it was :)
The reason I though that this could serve greatly as an stdlib package
is that it will broaden the horizon for importing arbitrary stuff just
to see if they are working as expected. "Testing" is the world!

> ChrisA
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/


___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Remote package/module imports through HTTP/S

[John Torakis]

On 23/08/2017 21:04, Bruce Leban wrote:
>
> On Wed, Aug 23, 2017 at 10:37 AM, John Torakis  > wrote:
>
>
> Github can be trusted 100% percent for example. 
>
>
> This isn't even remotely close to true. While I'd agree with the
> statement that the SSL cert on github is reasonably trustworthy, the
> *content* on github is NOT trustworthy and that's where the security
> risk is.

Do we trust code on github?

Do we trust code on PyPI?


This is why I **don't** want it ON by default. You have to explicitly
point the Finder/Loader to a repo that you created or you trust. And
provide a list of available modules/packages to import from that URL too.

If the developer isn't sure about the code she/he is importing then it
is her/his fault...

Same goes for pip installing though...

>
> I agree that this is a useful feature and there is no way it should be
> on by default. The right way IMHO to do this is to have a command line
> option something like this:
>
> python --http-import somelib=https://github.com/someuser/somelib
>
>
> which then redefines the import somelib command to import from that
> source. Along with your scenario, it allows people, for example, to
> replace a library with a different version without modifying source or
> installing a different version. That's pretty useful.
That's what I am thinking too! just provide the module so someone can
"python -m" it, or start a REPL in the context that some
packages/modules are available from a URL.


>
> --- Bruce

John Torakis
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Remote package/module imports through HTTP/S

[Chris Angelico]

On Thu, Aug 24, 2017 at 4:04 AM, Bruce Leban  wrote:
>
> On Wed, Aug 23, 2017 at 10:37 AM, John Torakis 
> wrote:
>>
>>
>> Github can be trusted 100% percent for example.
>
>
> This isn't even remotely close to true. While I'd agree with the statement
> that the SSL cert on github is reasonably trustworthy, the *content* on
> github is NOT trustworthy and that's where the security risk is.
>
> I agree that this is a useful feature and there is no way it should be on by
> default. The right way IMHO to do this is to have a command line option
> something like this:
>
> python --http-import somelib=https://github.com/someuser/somelib

If you read his README, it's pretty explicit about URLs; the risk is
that "https://github.com/someuser/somelib; can be intercepted, not
that "someuser" is malicious. If you're worried about the latter,
don't use httpimport.

ChrisA
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Remote package/module imports through HTTP/S

[Bruce Leban]

On Wed, Aug 23, 2017 at 10:37 AM, John Torakis 
wrote:

>
> Github can be trusted 100% percent for example.


This isn't even remotely close to true. While I'd agree with the statement
that the SSL cert on github is reasonably trustworthy, the *content* on
github is NOT trustworthy and that's where the security risk is.

I agree that this is a useful feature and there is no way it should be on
by default. The right way IMHO to do this is to have a command line option
something like this:

python --http-import somelib=https://github.com/someuser/somelib


which then redefines the import somelib command to import from that source.
Along with your scenario, it allows people, for example, to replace a
library with a different version without modifying source or installing a
different version. That's pretty useful.

--- Bruce
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Remote package/module imports through HTTP/S

[Chris Angelico]

On Thu, Aug 24, 2017 at 3:37 AM, John Torakis  wrote:
>
>
> On 23/08/2017 20:36, John Torakis wrote:
>> Yeah, I am a security researcher, I am keen on backdoor programming and
>> staging and all that! It is my official job and research topic! I go to
>> the office and code such stuff! I am not a blackhat, nor a security
>> enthusiast, it is my job.
>>
>>
>> First of all, let's all agree that if someone can run Python code in
>> your computer you are 100% hacked! It is irrelevant if "httpimport" is a
>> core python feature or not in that case.
>>
>> Now, I agree that this can be exploited if used under plain HTTP, it is
>> a MiTM -> Remote code execution case. I admit that this is not bright.
>> But I mention that this can be used in testing.
>>
>> On the topic of HTTPS, man-in-the-middle is not possible without
>> previous Trusted Certificate compromise. Github can be trusted 100%
>> percent for example. A certificate check has to take place in the HTTPS
>> remote loading for sure!

Right, but that just pushes the problem one level further out: you
need to have a 100% dependable certificate chain. And that means
absolutely completely trusting all of your root certificates, and it
also means either not needing to add any _more_ root certificates, or
being able to configure the cert store. As we've seen elsewhere, this
is nontrivial.

>> When I said a "core feature" I meant that the "httpimport" module would
>> deliver with the core modules. Not that the Finder/Loader has to be in
>> the list of Finders/Loaders that are used by default! For god sake, I
>> wouldn't like my PC to start probing for modules just because I mistyped
>> an import line!

Glad we agree about that! I have seen people wanting all sorts of
things to become core features (usually for the sake of interactive
work), and a lot of it is MUCH better handled as a non-core feature.

Though a lot of what you're saying here - especially this:

>> I know that pip works nicely, especially when paired with virtual
>> environments, but ad-hoc importing is another another thing. It isn't
>> meant for delivering real projects. Just for testing modules without the
>> need to download them, maybe install them, and all.

could be equally well handled by pip-installing httpimport itself, and
using that to bootstrap your testing procedures. Unless, of course,
you're wanting to httpimport httpimport, in which case you're going to
run into bootstrapping problems whichever way you do it :)

I think we're on the same page here, but it definitely needs some more
text in the README to explain this - particularly how this is not a
replacement for pip. For example, my first thought on seeing this was
"wow, that's going to be abysmally slow unless it has a cache", but
the answer to that is: if you need a cache, you probably should be
using pip to install things properly.

Still -1 on this becoming a stdlib package, as there's nothing I've
yet seen that can't be done as a third-party package. But it's less
scary than I thought it was :)

ChrisA
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Remote package/module imports through HTTP/S

[John Torakis]

Bounced back on list



 Forwarded Message 
Θέμα:   Re: [Python-ideas] Remote package/module imports through HTTP/S
Ημερομηνία: Wed, 23 Aug 2017 20:36:19 +0300
Από:John Torakis 
Προς:   Chris Angelico 



Yeah, I am a security researcher, I am keen on backdoor programming and
staging and all that! It is my official job and research topic! I go to
the office and code such stuff! I am not a blackhat, nor a security
enthusiast, it is my job.


First of all, let's all agree that if someone can run Python code in
your computer you are 100% hacked! It is irrelevant if "httpimport" is a
core python feature or not in that case.

Now, I agree that this can be exploited if used under plain HTTP, it is
a MiTM -> Remote code execution case. I admit that this is not bright.
But I mention that this can be used in testing.

On the topic of HTTPS, man-in-the-middle is not possible without
previous Trusted Certificate compromise. Github can be trusted 100%
percent for example. A certificate check has to take place in the HTTPS
remote loading for sure!

When I said a "core feature" I meant that the "httpimport" module would
deliver with the core modules. Not that the Finder/Loader has to be in
the list of Finders/Loaders that are used by default! For god sake, I
wouldn't like my PC to start probing for modules just because I mistyped
an import line!

I know that pip works nicely, especially when paired with virtual
environments, but ad-hoc importing is another another thing. It isn't
meant for delivering real projects. Just for testing modules without the
need to download them, maybe install them, and all.


Thank you for your time,
John Torakis


On 23/08/2017 20:17, Chris Angelico wrote:
> On Thu, Aug 24, 2017 at 2:55 AM, John Torakis  wrote:
>> Hello all!
>>
>> Today I opened an issue in bugs.python.org
>> (http://bugs.python.org/issue31264) proposing a module I created for
>> remote package/module imports through standard HTTP/S.
>>
>> The concept is that, if a directory is served through HTTP/S (the way
>> SimpleHTTPServer module serves directories), a Finder/Loader object can
>> fetch Python files from that directory using HTTP requests, and finally
>> load them as modules (or packages) in the running namespace.
>>
>> The repo containing a primitive (but working) version of the
>> Finder/Loader, also contains self explanatory examples (in the README.md):
>>
>> https://github.com/operatorequals/httpimport
>>
>>
>> My proposal is that this module can become a core Python feature,
>> providing a way to load modules even from Github.com repositories,
>> without the need to "git clone - setup.py install" them.
>>
>>
>> Other languages, like golang, provide this functionality from their
>> early days (day one?). Python development can be greatly improved if a
>> "try before pip installing" mechanism gets in place, as it will add a
>> lot to the REPL nature of the testing/experimenting process.
> As a core feature? No no no no no no no no. Absolutely do NOT WANT
> THIS. This is a security bug magnet; can you imagine trying to ensure
> that malicious code is not executed, in an arbitrary execution
> context? As an explicitly-enabled feature, it's a lot less hairy than
> a permanently-active one (can you IMAGINE how terrifying that would
> be?), but even so, trying to prove that addRemoteRepo (not a
> PEP8-compliant name, btw) is getting the correct code is not going to
> be easy. You have to (a) drop HTTP altogether and mandate SSL and (b)
> be absolutely sure that your certificate chains are 100% dependable,
> which - as we've seen recently - is a nontrivial task.
>
> The easiest way to add remote code is pip. For most packages, that's
> what you want to be using:
>
> pip install requests
>
> will make "import requests" functional. I don't see pip mentioned
> anywhere in your README, but you do mention the testing of pull
> requests, so at very least, this wants some explanatory screed.
>
> But I'm not entirely sure I want to support this. You're explicitly
> talking about using this with the creation of backdoors... in what,
> exactly? What are you actually getting at here?
>
> ChrisA
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/


___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Remote package/module imports through HTTP/S

[John Torakis]

On 23/08/2017 20:36, John Torakis wrote:
> Yeah, I am a security researcher, I am keen on backdoor programming and
> staging and all that! It is my official job and research topic! I go to
> the office and code such stuff! I am not a blackhat, nor a security
> enthusiast, it is my job.
>
>
> First of all, let's all agree that if someone can run Python code in
> your computer you are 100% hacked! It is irrelevant if "httpimport" is a
> core python feature or not in that case.
>
> Now, I agree that this can be exploited if used under plain HTTP, it is
> a MiTM -> Remote code execution case. I admit that this is not bright.
> But I mention that this can be used in testing.
>
> On the topic of HTTPS, man-in-the-middle is not possible without
> previous Trusted Certificate compromise. Github can be trusted 100%
> percent for example. A certificate check has to take place in the HTTPS
> remote loading for sure!
>
> When I said a "core feature" I meant that the "httpimport" module would
> deliver with the core modules. Not that the Finder/Loader has to be in
> the list of Finders/Loaders that are used by default! For god sake, I
> wouldn't like my PC to start probing for modules just because I mistyped
> an import line!
>
> I know that pip works nicely, especially when paired with virtual
> environments, but ad-hoc importing is another another thing. It isn't
> meant for delivering real projects. Just for testing modules without the
> need to download them, maybe install them, and all.
>
>
> Thank you for your time,
> John Torakis
>
>
> On 23/08/2017 20:17, Chris Angelico wrote:
>> On Thu, Aug 24, 2017 at 2:55 AM, John Torakis  wrote:
>>> Hello all!
>>>
>>> Today I opened an issue in bugs.python.org
>>> (http://bugs.python.org/issue31264) proposing a module I created for
>>> remote package/module imports through standard HTTP/S.
>>>
>>> The concept is that, if a directory is served through HTTP/S (the way
>>> SimpleHTTPServer module serves directories), a Finder/Loader object can
>>> fetch Python files from that directory using HTTP requests, and finally
>>> load them as modules (or packages) in the running namespace.
>>>
>>> The repo containing a primitive (but working) version of the
>>> Finder/Loader, also contains self explanatory examples (in the README.md):
>>>
>>> https://github.com/operatorequals/httpimport
>>>
>>>
>>> My proposal is that this module can become a core Python feature,
>>> providing a way to load modules even from Github.com repositories,
>>> without the need to "git clone - setup.py install" them.
>>>
>>>
>>> Other languages, like golang, provide this functionality from their
>>> early days (day one?). Python development can be greatly improved if a
>>> "try before pip installing" mechanism gets in place, as it will add a
>>> lot to the REPL nature of the testing/experimenting process.
>> As a core feature? No no no no no no no no. Absolutely do NOT WANT
>> THIS. This is a security bug magnet; can you imagine trying to ensure
>> that malicious code is not executed, in an arbitrary execution
>> context? As an explicitly-enabled feature, it's a lot less hairy than
>> a permanently-active one (can you IMAGINE how terrifying that would
>> be?), but even so, trying to prove that addRemoteRepo (not a
>> PEP8-compliant name, btw) is getting the correct code is not going to
>> be easy. You have to (a) drop HTTP altogether and mandate SSL and (b)
>> be absolutely sure that your certificate chains are 100% dependable,
>> which - as we've seen recently - is a nontrivial task.
>>
>> The easiest way to add remote code is pip. For most packages, that's
>> what you want to be using:
>>
>> pip install requests
>>
>> will make "import requests" functional. I don't see pip mentioned
>> anywhere in your README, but you do mention the testing of pull
>> requests, so at very least, this wants some explanatory screed.
>>
>> But I'm not entirely sure I want to support this. You're explicitly
>> talking about using this with the creation of backdoors... in what,
>> exactly? What are you actually getting at here?
>>
>> ChrisA
>> ___
>> Python-ideas mailing list
>> Python-ideas@python.org
>> https://mail.python.org/mailman/listinfo/python-ideas
>> Code of Conduct: http://python.org/psf/codeofconduct/
>

___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Remote package/module imports through HTTP/S

[Oleg Broytman]

Hi!

On Wed, Aug 23, 2017 at 07:55:00PM +0300, John Torakis  
wrote:
> Hello all!
> 
> Today I opened an issue in bugs.python.org
> (http://bugs.python.org/issue31264) proposing a module I created for
> remote package/module imports through standard HTTP/S.

   The issue is so big IMO it requires a PEP, not just an issue.

   Anyway I'm -1000 for reasons of security, connectivity (not all hosts
are connected), traffic cost and speed.

> The concept is that, if a directory is served through HTTP/S (the way
> SimpleHTTPServer module serves directories), a Finder/Loader object can
> fetch Python files from that directory using HTTP requests, and finally
> load them as modules (or packages) in the running namespace.
> 
> The repo containing a primitive (but working) version of the
> Finder/Loader, also contains self explanatory examples (in the README.md):
> 
> https://github.com/operatorequals/httpimport
> 
> 
> My proposal is that this module can become a core Python feature,
> providing a way to load modules even from Github.com repositories,
> without the need to "git clone - setup.py install" them.
> 
> 
> Other languages, like golang, provide this functionality from their

   AFAIK Go downloads modules at compile time, not run time. This is a
major distiction with Python.

> early days (day one?). Python development can be greatly improved if a
> "try before pip installing" mechanism gets in place, as it will add a
> lot to the REPL nature of the testing/experimenting process.
> 
> 
> 
> Thank you for your time,
> 
> John Torakis, IT Security Researcher
> 
> 
> 
> P.S: It is my first time in this mailing list and generally Python
> contribution. Please be tolerant!

Oleg.
-- 
 Oleg Broytmanhttp://phdru.name/p...@phdru.name
   Programmers don't die, they just GOSUB without RETURN.
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Remote package/module imports through HTTP/S

[Chris Angelico]

On Thu, Aug 24, 2017 at 2:55 AM, John Torakis  wrote:
> Hello all!
>
> Today I opened an issue in bugs.python.org
> (http://bugs.python.org/issue31264) proposing a module I created for
> remote package/module imports through standard HTTP/S.
>
> The concept is that, if a directory is served through HTTP/S (the way
> SimpleHTTPServer module serves directories), a Finder/Loader object can
> fetch Python files from that directory using HTTP requests, and finally
> load them as modules (or packages) in the running namespace.
>
> The repo containing a primitive (but working) version of the
> Finder/Loader, also contains self explanatory examples (in the README.md):
>
> https://github.com/operatorequals/httpimport
>
>
> My proposal is that this module can become a core Python feature,
> providing a way to load modules even from Github.com repositories,
> without the need to "git clone - setup.py install" them.
>
>
> Other languages, like golang, provide this functionality from their
> early days (day one?). Python development can be greatly improved if a
> "try before pip installing" mechanism gets in place, as it will add a
> lot to the REPL nature of the testing/experimenting process.

As a core feature? No no no no no no no no. Absolutely do NOT WANT
THIS. This is a security bug magnet; can you imagine trying to ensure
that malicious code is not executed, in an arbitrary execution
context? As an explicitly-enabled feature, it's a lot less hairy than
a permanently-active one (can you IMAGINE how terrifying that would
be?), but even so, trying to prove that addRemoteRepo (not a
PEP8-compliant name, btw) is getting the correct code is not going to
be easy. You have to (a) drop HTTP altogether and mandate SSL and (b)
be absolutely sure that your certificate chains are 100% dependable,
which - as we've seen recently - is a nontrivial task.

The easiest way to add remote code is pip. For most packages, that's
what you want to be using:

pip install requests

will make "import requests" functional. I don't see pip mentioned
anywhere in your README, but you do mention the testing of pull
requests, so at very least, this wants some explanatory screed.

But I'm not entirely sure I want to support this. You're explicitly
talking about using this with the creation of backdoors... in what,
exactly? What are you actually getting at here?

ChrisA
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Remote package/module imports through HTTP/S

[John Torakis]

Hello all!

Today I opened an issue in bugs.python.org
(http://bugs.python.org/issue31264) proposing a module I created for
remote package/module imports through standard HTTP/S.

The concept is that, if a directory is served through HTTP/S (the way
SimpleHTTPServer module serves directories), a Finder/Loader object can
fetch Python files from that directory using HTTP requests, and finally
load them as modules (or packages) in the running namespace.

The repo containing a primitive (but working) version of the
Finder/Loader, also contains self explanatory examples (in the README.md):

https://github.com/operatorequals/httpimport


My proposal is that this module can become a core Python feature,
providing a way to load modules even from Github.com repositories,
without the need to "git clone - setup.py install" them.


Other languages, like golang, provide this functionality from their
early days (day one?). Python development can be greatly improved if a
"try before pip installing" mechanism gets in place, as it will add a
lot to the REPL nature of the testing/experimenting process.



Thank you for your time,

John Torakis, IT Security Researcher



P.S: It is my first time in this mailing list and generally Python
contribution. Please be tolerant!


___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] PEP 550 v2

[Ethan Furman]

On 08/23/2017 08:41 AM, Guido van Rossum wrote:


If we're extending the analogy with thread-locals we should at least consider 
making each instantiation return a
namespace rather than something holding a single value.


+1

--
~Ethan~
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] PEP 550 v2

[Guido van Rossum]

On Wed, Aug 23, 2017 at 2:00 AM, Nick Coghlan  wrote:

> On 21 August 2017 at 07:01, Barry  wrote:
> > I'm not clear why there is a new_context_key which seems not to be a key.
> > It seems that the object is a container for a single value.
> >
> > Key.set( value ) does not feel right.
>
> It's basically borrowed from procedural thread local APIs, which tend
> to use APIs like "tss_set(key, value)".
>
> That said, in a separate discussion, Caleb Hattingh mentioned C#'s
> AsyncLocal API, and it occurred to me that "context local" might work
> well as the name of the context access API:
>
> my_implicit_state = sys.new_context_local('my_state')
> my_implicit_state.set('spam')
>
> # Later, to access the value of my_implicit_state:
> print(my_implicit_state.get())
>
> That way, we'd have 3 clearly defined kinds of local variables:
>
> * frame locals (the regular kind)
> * thread locals (threading.locals() et al)
> * context locals (PEP 550)
>
> The fact contexts can be nested, and a failed lookup in the active
> implicit context may then query outer namespaces in the current
> execution context would then be directly analogous to the way name
> lookups are resolved for frame locals.


If we're extending the analogy with thread-locals we should at least
consider making each instantiation return a namespace rather than something
holding a single value. We have

log_state = threading.local()
log_state.verbose = False

def action(x):
if log_state.verbose:
print(x)

def make_verbose():
log_state.verbose = True

It would be nice if we could upgrade this to make it PEP 550-aware so that
only the first line needs to change:

log_state = sys.AsyncLocal("log state")
# The rest is the same

We might even support the alternative notation where you can provide
default values and suggest a schema, similar to to threading.local:

class LogState(threading.local):
verbose = False

log_state = LogState()

(I think that for calls that construct empty instances of various types we
should just use the class name rather than some factory function. I also
think none of this should live in sys but that's separate.)

-- 
--Guido van Rossum (python.org/~guido)
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] PEP 550 v2

[Nick Coghlan]

On 21 August 2017 at 07:01, Barry  wrote:
> I'm not clear why there is a new_context_key which seems not to be a key.
> It seems that the object is a container for a single value.
>
> Key.set( value ) does not feel right.

It's basically borrowed from procedural thread local APIs, which tend
to use APIs like "tss_set(key, value)".

That said, in a separate discussion, Caleb Hattingh mentioned C#'s
AsyncLocal API, and it occurred to me that "context local" might work
well as the name of the context access API:

my_implicit_state = sys.new_context_local('my_state')
my_implicit_state.set('spam')

# Later, to access the value of my_implicit_state:
print(my_implicit_state.get())

That way, we'd have 3 clearly defined kinds of local variables:

* frame locals (the regular kind)
* thread locals (threading.locals() et al)
* context locals (PEP 550)

The fact contexts can be nested, and a failed lookup in the active
implicit context may then query outer namespaces in the current
execution context would then be directly analogous to the way name
lookups are resolved for frame locals.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Please consider adding context manager versions of setUp/tearDown to unittest.TestCase

[Nick Coghlan]

On 23 August 2017 at 08:20, rym...@gmail.com  wrote:
> TBH you're completely right. Every time I see someone using unittest
> andItsHorriblyUnpythonicNames, I want to kill a camel.
>
> Sometimes, though, I feel like part of the struggle is the alternative. If
> you dislike unittest, but pytest is too "magical" for you, what do you use?
> Many Python testing tools like nose are just test *runners*, so you still
> need something else. In the end, many just end up back at unittest, maybe
> with nose on top.

A snake_case helper API for unittest that I personally like is
hamcrest, since that also separates out the definition of testing
assertions from being part of a test case:
https://pypi.python.org/pypi/PyHamcrest

Introducing such a split natively into unittest is definitely
attractive, but would currently be difficult due to the way that some
features like self.maxDiff and self.subTest work.

However, PEP 550's execution contexts may provide a way to track the
test state reliably that's independent of being a method on a test
case instance, in which case it would become feasible to offer a more
procedural interface in addition to the current visibly
object-oriented one.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/