[Python-Dev] Why no venv in existing directory?
Hi All, While trying 3.3 beta I found that I cannot use my favorite virtualenv pattern with pyvenv: $ virtualenv . Installing.done. $ pyvenv . Error: Directory exists: /Users/stefan/sandbox/foo I appreciate that this behavior is documented and was in the PEP from the start: "If the target directory already exists an error will be raised, unless the --clear or --upgrade option was provided." Still, I am curious what the rationale behind this restriction is. For me, being able to "bless" an existing directory with a virtualenv has always been one of its attractions. Thanks, Stefan -- Stefan H. Holek [email protected] ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] venv scripts for fish and csh shells
virtualenv has virtualenv.csh and virtualenv.fish files. Is there any reason for restricting venv to bash/zsh only? -- Thanks, Andrew Svetlov ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Why no venv in existing directory?
Hi Stefan, On 07/19/2012 06:28 AM, Stefan H. Holek wrote: > While trying 3.3 beta I found that I cannot use my favorite > virtualenv pattern with pyvenv: > > $ virtualenv . Installing.done. > > $ pyvenv . Error: Directory exists: /Users/stefan/sandbox/foo > > I appreciate that this behavior is documented and was in the PEP from > the start: "If the target directory already exists an error will be > raised, unless the --clear or --upgrade option was provided." > > Still, I am curious what the rationale behind this restriction is. I'd have no problem with lifting the restriction. I don't recall any clear rationale; I think it was probably just the simplest implementation initially, and no one ever raised it as an issue in the PEP process. Carl ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] venv scripts for fish and csh shells
On 07/19/2012 10:26 AM, Andrew Svetlov wrote: > virtualenv has virtualenv.csh and virtualenv.fish files. > Is there any reason for restricting venv to bash/zsh only? No. As far as I'm concerned, a patch to port the virtualenv csh and fish activate scripts to pyvenv would be welcome (though I can't commit said patch, so it might be good to hear if Vinay has a different opinion). Carl ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Unbinding of methods
Hey, As part of pickle4, I found it interesting to add the possibility of pickling bound functions (instance methods). This is done by pickling f.__self__ and f.__func__ separately, and then adding a BIND opcode to tie them together. While this appears to work fine for python methods (non-builtin), some issues arise with builtins. These are partly caused because not all builtin function types support __func__, partly because not all of them fill __module__ when they should and partly because there are many (7) types a function can actually have: ClassMethodDescriptorType = type(??) BuiltinFunctionType = type(len) FunctionType = type(f) MethodType = type(A().f()) MethodDescriptorType = type(list.append) WrapperDescriptorType = type(list.__add__) MethodWrapperType = type([].__add__) AllFunctionTypes = (ClassMethodDescriptorType, BuiltinFunctionType, FunctionType, MethodType, MethodDescriptorType, WrapperDescriptorType, MethodWrapperType) repr(AllFunctionTypes) = ( , , , , , , ) I have created a patch at [1], which adds __func__ to some other function types, as well as: 1) adds AllFunctionTypes etc. to Lib/types.py 2) inspect.isanyfunction(), inspect.isanyboundfunction(), inspect.isanyunboundfunction() 3) functools.unbind Note that I am not knowledgeable of cpython internals and therefore the patch needs to be carefully reviewed. Possible issues: Should classmethods be considered bound or unbound? If cm is a classmethod, then should cm.__func__.__self__ = cm.__self__ or cm.__func__.__self__ = None? Currently does the latter: >>> cm.__self__, hasattr(cm,'__self__'), hasattr(cm.__func__, '__self__') (, True, False) This requires treating classmethods separately when pickling, so I'm not sure if this is ideal. Let me know if I should have opened an issue instead. I look forward to hearing your opinions/suggestions on this matter. Regards, Stefan M [1] https://gist.github.com/3145210 ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Unbinding of methods
On Thu, Jul 19, 2012 at 12:53 PM, M Stefan wrote: > Hey, > > As part of pickle4, I found it interesting to add the possibility > of pickling bound functions (instance methods). This is done by > pickling f.__self__ and f.__func__ separately, and then adding > a BIND opcode to tie them together. > > While this appears to work fine for python methods (non-builtin), some > issues arise with builtins. These are partly caused because > not all builtin function types support __func__, partly because > not all of them fill __module__ when they should and partly > because there are many (7) types a function can actually have: > > ClassMethodDescriptorType = type(??) > BuiltinFunctionType = type(len) > FunctionType = type(f) > MethodType = type(A().f()) > MethodDescriptorType = type(list.append) > WrapperDescriptorType = type(list.__add__) > MethodWrapperType = type([].__add__) > > AllFunctionTypes = (ClassMethodDescriptorType, BuiltinFunctionType, > FunctionType, MethodType, MethodDescriptorType, > WrapperDescriptorType, MethodWrapperType) > repr(AllFunctionTypes) = ( > , > , , > , , > , ) > > I have created a patch at [1], which adds __func__ to some other > function types, as well as: > 1) adds AllFunctionTypes etc. to Lib/types.py > 2) inspect.isanyfunction(), inspect.isanyboundfunction(), > inspect.isanyunboundfunction() > 3) functools.unbind > Note that I am not knowledgeable of cpython internals and therefore > the patch needs to be carefully reviewed. > > Possible issues: Should classmethods be considered bound or unbound? > If cm is a classmethod, then should > cm.__func__.__self__ = cm.__self__ or cm.__func__.__self__ = None? > Currently does the latter: > >>> cm.__self__, hasattr(cm,'__self__'), hasattr(cm.__func__, > '__self__') > (, True, False) > This requires treating classmethods separately when pickling, > so I'm not sure if this is ideal. > > Let me know if I should have opened an issue instead. I look > forward to hearing your opinions/suggestions on this matter. > Yes, open an issue for your patch and reply here with the issue #. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Unbinding of methods
The issue is http://bugs.python.org/issue15397 [Stefan accidentally replied privately to me] On Thu, Jul 19, 2012 at 1:00 PM, Brett Cannon wrote: > > > On Thu, Jul 19, 2012 at 12:53 PM, M Stefan wrote: > >> Hey, >> >> As part of pickle4, I found it interesting to add the possibility >> of pickling bound functions (instance methods). This is done by >> pickling f.__self__ and f.__func__ separately, and then adding >> a BIND opcode to tie them together. >> >> While this appears to work fine for python methods (non-builtin), some >> issues arise with builtins. These are partly caused because >> not all builtin function types support __func__, partly because >> not all of them fill __module__ when they should and partly >> because there are many (7) types a function can actually have: >> >> ClassMethodDescriptorType = type(??) >> BuiltinFunctionType = type(len) >> FunctionType = type(f) >> MethodType = type(A().f()) >> MethodDescriptorType = type(list.append) >> WrapperDescriptorType = type(list.__add__) >> MethodWrapperType = type([].__add__) >> >> AllFunctionTypes = (ClassMethodDescriptorType, BuiltinFunctionType, >> FunctionType, MethodType, MethodDescriptorType, >> WrapperDescriptorType, MethodWrapperType) >> repr(AllFunctionTypes) = ( >> , >> , , >> , , >> , ) >> >> I have created a patch at [1], which adds __func__ to some other >> function types, as well as: >> 1) adds AllFunctionTypes etc. to Lib/types.py >> 2) inspect.isanyfunction(), inspect.isanyboundfunction(), >> inspect.isanyunboundfunction() >> 3) functools.unbind >> Note that I am not knowledgeable of cpython internals and therefore >> the patch needs to be carefully reviewed. >> >> Possible issues: Should classmethods be considered bound or unbound? >> If cm is a classmethod, then should >> cm.__func__.__self__ = cm.__self__ or cm.__func__.__self__ = None? >> Currently does the latter: >> >>> cm.__self__, hasattr(cm,'__self__'), hasattr(cm.__func__, >> '__self__') >> (, True, False) >> This requires treating classmethods separately when pickling, >> so I'm not sure if this is ideal. >> >> Let me know if I should have opened an issue instead. I look >> forward to hearing your opinions/suggestions on this matter. >> > > Yes, open an issue for your patch and reply here with the issue #. > ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] clang
On Wed, 18 Jul 2012 17:15:18 +0200 Ronald Oussoren wrote: > > > >> I regularly run the 3.3 testsuite using the latest Xcode from the Appstore > >> on a OSX Lion machine and that works properly. > > > > I'm not actually using the latest Xcode. So if you could test my test > > program, that would be much appreciated. > > That bug in llvm-gcc still exists, and is unlikely to get fixed. That's a > bug in the integretion of the GCC frontend and LLVM backend, clang (LLVM > project frontend + LLVM backend) does work. Not only clang seems to work, but we have a stable buildbot running on it: http://buildbot.python.org/all/buildslaves/langa-lion Regards Antoine. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Use function names instead of functions for os.supports_dir_fd?
On Wed, 18 Jul 2012 02:26:14 +0200 Victor Stinner wrote: > >> Monkey patching is a common practice in Python. test_os.py replaces > >> os.exec*() functions temporary for example. > > > > Perhaps for testing, but I don't think monkey-patching is common in > > production code. Perhaps you are thinking of Ruby :) > > The gevent library does monkey-patch os.fork (and time.sleep and many > other functions), but gevent is maybe not ready for production? :-) Extensive monkey-patching of core OS functions would certainly make me weary of using such a third-party library, even if it claims to be "serious". Regards Antoine. -- Software development and contracting: http://pro.pitrou.net ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Unbinding of methods
On Thu, 19 Jul 2012 19:53:27 +0300 M Stefan wrote: > Hey, > > As part of pickle4, I found it interesting to add the possibility > of pickling bound functions (instance methods). This is done by > pickling f.__self__ and f.__func__ separately, and then adding > a BIND opcode to tie them together. Instead of a specific opcode, can't you use a suitable __reduce__ magic (or __getnewargs__, perhaps)? We want to limit the number of opcodes except for performance-critical types (and I don't think bound methods are performance-critical for the purpose of serialization). > I have created a patch at [1], which adds __func__ to some other > function types, as well as: > 1) adds AllFunctionTypes etc. to Lib/types.py > 2) inspect.isanyfunction(), inspect.isanyboundfunction(), > inspect.isanyunboundfunction() > 3) functools.unbind That sounds like a lot of changes if the goal is simply to make those types picklable. Regards Antoine. -- Software development and contracting: http://pro.pitrou.net ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Unbinding of methods
On 19/07/2012 7:54pm, Antoine Pitrou wrote:
> Instead of a specific opcode, can't you use a suitable __reduce__
> magic (or __getnewargs__, perhaps)? We want to limit the number of
> opcodes except for performance-critical types (and I don't think
> bound methods are performance-critical for the purpose of
> serialization).
The one wrinkle is that BuiltinFunctionType is special cased to be
pickled with save_global, and has no fallback to
__reduce__/__reduce_ex__/copyreg. (The C implementation for
FunctionType *does* have such a fallback, whereas the Python
implementation doesn't -- see bug http://bugs.python.org/issue14336.)
If the fallback is added for BuiltinFunctionType then "__reduce__ magic"
should be enough.
The following code works as expected:
import pickle
import copyreg
class A(object):
def f(self):
pass
@classmethod
def g(cls):
pass
def f(self):
pass
ClassMethodDescriptorType = type(A.g)
BuiltinFunctionType = type(len)
FunctionType = type(f)
MethodType = type(A().f)
MethodDescriptorType = type(list.append)
WrapperDescriptorType = type(list.__add__)
MethodWrapperType = type([].__add__)
obj_list = [A.g, len, f, A().f, list.append, list.__add__, [].__add__]
assert ClassMethodDescriptorType is MethodType
def reduce_self(self):
return getattr, (self.__self__, self.__name__)
def reduce_objclass(self):
return getattr, (self.__objclass__, self.__name__)
copyreg.pickle(MethodType, reduce_self)
copyreg.pickle(BuiltinFunctionType, reduce_self)
copyreg.pickle(MethodWrapperType, reduce_self)
copyreg.pickle(MethodDescriptorType, reduce_objclass)
copyreg.pickle(WrapperDescriptorType, reduce_objclass)
for obj in obj_list:
data = pickle.dumps(obj)
new_obj = pickle.loads(data)
print('%s\n%s\n' % (obj, new_obj))
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Unbinding of methods
On 7/19/2012 9:54 PM, Antoine Pitrou wrote: On Thu, 19 Jul 2012 19:53:27 +0300 M Stefan wrote: Hey, As part of pickle4, I found it interesting to add the possibility of pickling bound functions (instance methods). This is done by pickling f.__self__ and f.__func__ separately, and then adding a BIND opcode to tie them together. Instead of a specific opcode, can't you use a suitable __reduce__ magic (or __getnewargs__, perhaps)? We want to limit the number of opcodes except for performance-critical types (and I don't think bound methods are performance-critical for the purpose of serialization). Yes, I agree that doing it with __reduce__ would be better approach than adding a new opcode, I'll consider switching. I have created a patch at [1], which adds __func__ to some other function types, as well as: 1) adds AllFunctionTypes etc. to Lib/types.py 2) inspect.isanyfunction(), inspect.isanyboundfunction(), inspect.isanyunboundfunction() 3) functools.unbind That sounds like a lot of changes if the goal is simply to make those types picklable. Regards Antoine. Indeed they are, I just thought there may be a chance this code would be used elsewhere too. It's a bit weird that you can use inspect to check for certain types of functions but not others, as well as be able to "unbind" certain types of methods but not others. Admittedly, these changes have little use-case and are not a priority. Yours, Stefan M ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Runtime forks through monkeypatching (was Re: Use function names instead of functions for os.supports_dir_fd?)
On Fri, Jul 20, 2012 at 4:48 AM, Antoine Pitrou wrote: > On Wed, 18 Jul 2012 02:26:14 +0200 > Victor Stinner wrote: > >> >> Monkey patching is a common practice in Python. test_os.py replaces >> >> os.exec*() functions temporary for example. >> > >> > Perhaps for testing, but I don't think monkey-patching is common in >> > production code. Perhaps you are thinking of Ruby :) >> >> The gevent library does monkey-patch os.fork (and time.sleep and many >> other functions), but gevent is maybe not ready for production? :-) > > Extensive monkey-patching of core OS functions would certainly make me > weary of using such a third-party library, even if it claims to be > "serious". gevent is one of the well-behaved players in that game - it doesn't monkeypatch anything unless you explicitly tell it to [1]. You do need to have some idea what you're doing if using the monkey patching aspect (as not all third party modules will behave well in that case - just as not all third party modules behave well on other Python implementations), but you can also just use gevent as a normal library without monkey patching anything (however, adhering strictly to that rule leaves you in the same situation as Twisted, where modules that use the standard blocking APIs are completely useless to you. Thus, if your goal is "pure async" code without any need for monkeypatching, Twisted is a much better bet than gevent, just because it's been playing that game for longer and has more fully async aware components available). That's why I call features like the gevent.monkey module "runtime forks" - when they're implemented well, they effectively give you two Python implementations without the complexity of actually having parallel Python installations (compare the complexity of using Stackless, a traditional fork, vs gevent.monkey, a runtime fork - there's a reason the core greenlet component was lifted out of Stackless and made available as a CPython extension module). Applications can decide at startup which Python variant (standard CPython, or CPython+gevent) they want to use without affecting other Python applications installed on the system and without needing to duplicating all installed Python modules into a second interpreter. It's only when libraries monkeypatch standard interfaces *implicitly* as a side effect of import that you start to get seriously broken "action at a distance" behaviour. Or, as happened with distutils, you get a third party project (setuptools) effectively freezing implementation details of the stdlib version - when monkeypatching starts delving too deep into implementation details, it's time to switch to a traditional fork and start figuring out better ways to solve the problem (which process is now grinding away on the distutils2/packaging front). Monkeypatching is a powerful tool - used well, it lets you achieve impressive things that would otherwise be substantially more difficult to either create in the first place or maintain in the long run. As with many powerful tools though, used badly it can lead to a lot of pain. Static languages like Java decide the risk of action at a distance from monkeypatching isn't worth the power it gives you to work around issues in third party libraries without resorting to creating a traditional fork, so they strictly enforce access controls and have completely closed class definitions. Python permits monkeypatching at a language level (because when you need it, you really need it), but discourages it a social level (because there is usually a less magical technique, such as creating and using a wrapper or subclass, that will achieve the desired effect). Cheers, Nick. [1] http://www.gevent.org/intro.html#monkey-patching P.S. for a non-monkeypatching approach to reusing a synchronous library when creating an asynchronous equivalent, take a look at 10gen's attempt to create a *maintainable* Tornado-compatible async wrapper for PyMongo by using greenlets ([2][3]). It's a clever idea, but still a lot more work than just using PyMongo itself with gevent if you're only interested in the "async" part rather than the "Tornado-compatible" part. [2] http://emptysquare.net/blog/motor-internals-how-i-asynchronized-a-synchronous-library/ [3] http://emptysquare.net/blog/motor-four-strategies-for-maintainability/ -- Nick Coghlan | [email protected] | Brisbane, Australia ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
