[issue24400] Awaitable ABC incompatible with functools.singledispatch
Stefan Behnel added the comment: I originally planned to make the next Cython release patch the Generator and Coroutine ABCs into collections.abc, but I now think it would be worth uploading an abc_backports package to PyPI instead that does that and on which asyncio, tornado and other packages could simply depend with a try-import. Done: https://pypi.python.org/pypi/backports_abc Feedback welcome. Stefan -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue24400 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24400] Awaitable ABC incompatible with functools.singledispatch
Stefan Behnel added the comment: I added three more comments to the review. The rest looks good to me, too. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue24400 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24469] Py2.x int free list can grow without bounds
New submission from Stefan Behnel:
A Cython user noticed a memory leak when C-inheriting from int.
http://thread.gmane.org/gmane.comp.python.cython.devel/15689
The Cython code to reproduce this is simply this:
cdef class ExtendedInt(int): pass
for j in xrange(1000):
ExtendedInt(j)
The problem is due to the free-list of the int type. It uses this code for
deallocation:
static void
int_dealloc(PyIntObject *v)
{
if (PyInt_CheckExact(v)) {
Py_TYPE(v) = (struct _typeobject *)free_list;
free_list = v;
}
else
Py_TYPE(v)-tp_free((PyObject *)v);
}
static void
int_free(PyIntObject *v)
{
Py_TYPE(v) = (struct _typeobject *)free_list;
free_list = v;
}
Now, when C-inheriting from PyInt_Type without providing an own tp_free
implementation, PyType_Ready() will inherit the supertype's tp_free slot, which
means that int_dealloc() above will end up calling int_free() in all cases, not
only for the exact-int case. Thus, whether or not it's exactly int or a
subtype, the object will always be added to the free-list on deallocation.
However, in the subtype case, the free-list is actually ignored by int_new()
and int_subtype_new(), so that as long as the user code only creates tons of
int subtype instances and not plain int instances, the freelist will keep
growing without bounds by pushing dead subtype objects onto it that are never
reused.
There are two problems here:
1) int subtypes should not be added to the freelist, or at least not unless
they have exactly the same struct size as PyIntObject (which the ExtendedInt
type above does but other subtypes may not)
2) if a free-list is used, it should be used in all instantiation cases, not
just in PyInt_FromLong().
Fixing 1) by adding a type check to int_free() would be enough to fix the
overall problem. Here's a quickly hacked up change that seems to work for me:
static void
int_free(PyIntObject *v)
{
if (PyInt_CheckExact(v)) {
Py_TYPE(v) = (struct _typeobject *)free_list;
free_list = v;
}
else if (Py_TYPE(v)-tp_flags Py_TPFLAGS_HAVE_GC)
PyObject_GC_Del(v); // untested by probably necessary
else
PyObject_Del(v);
}
--
components: Interpreter Core
messages: 245492
nosy: scoder
priority: normal
severity: normal
status: open
title: Py2.x int free list can grow without bounds
type: crash
versions: Python 2.7
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue24469
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24450] Add gi_yieldfrom calculated property to generator object
Stefan Behnel added the comment: No problem for Cython either. The change in issue 24400 that makes coroutines real Awaitables also removes surprises for a cr_await return value being a coroutine and previously *not* an Awaitable. The contract for gi_yieldfrom is only that the returned value is an Iterator, IIUC? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue24450 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24450] Add gi_yieldfrom calculated property to generator object
Changes by Stefan Behnel sco...@users.sourceforge.net: -- nosy: +scoder ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue24450 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24469] Py2.x int free list can grow without bounds
Changes by Stefan Behnel sco...@users.sourceforge.net: -- nosy: +pitrou, serhiy.storchaka ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue24469 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24469] Py2.x int free list can grow without bounds
Stefan Behnel added the comment:
1) The intended solution is to require that int subclasses override tp_free.
In the way I showed? By calling either PyObject_GC_Del() or PyObject_Del()
directly? I'll happily do that (Py2.7 and Py2 int being dead ends makes
that pretty future proof), but it's neither obvious nor very clean.
2) I don't see any constructors that don't call PyInt_FromLong() -- what am I
missing?
Not sure what you mean. int_new() immediately calls int_subtype_new(),
which then calls type-tp_alloc(). No call to PyInt_FromLong() involved.
static PyObject *
int_new(PyTypeObject *type, PyObject *args, PyObject *kwds)
{
PyObject *x = NULL;
int base = -909;
static char *kwlist[] = {x, base, 0};
if (type != PyInt_Type)
return int_subtype_new(type, args, kwds); /* Wimp out */
...
static PyObject *
int_subtype_new(PyTypeObject *type, PyObject *args, PyObject *kwds)
{
...
tmp = int_new(PyInt_Type, args, kwds);
if (tmp == NULL)
return NULL;
if (!PyInt_Check(tmp)) {
ival = PyLong_AsLong(tmp);
if (ival == -1 PyErr_Occurred()) {
Py_DECREF(tmp);
return NULL;
}
} else {
ival = ((PyIntObject *)tmp)-ob_ival;
}
newobj = type-tp_alloc(type, 0);
if (newobj == NULL) {
Py_DECREF(tmp);
return NULL;
}
((PyIntObject *)newobj)-ob_ival = ival;
Py_DECREF(tmp);
return newobj;
}
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue24469
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24400] Awaitable ABC incompatible with functools.singledispatch
Stefan Behnel added the comment: I agree that a typedef is a good idea. It doesn't cost anything but gives us more freedom for future changes. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue24400 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
Re: Java to Python autoconverters
Sebastian M Cheung via Python-list schrieb am 12.06.2015 um 13:36: Are these available? Any good ones to recommend? I recommend not doing that. You'd end up with ugly and unidiomatic Python code that's impossible to maintain, whereas you now (hopefully) have somewhat idiomatic Java code that should be reasonably maintainable. If you want to integrate Python code with Java code, take a look at Jython instead. If that's not what you want, then feel free to unveil your intentions. Stefan -- https://mail.python.org/mailman/listinfo/python-list
[issue23996] _PyGen_FetchStopIterationValue() crashes on unnormalised exceptions
Changes by Stefan Behnel sco...@users.sourceforge.net: Added file: http://bugs.python.org/file39692/fix_stopiteration_value.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23996 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23996] _PyGen_FetchStopIterationValue() crashes on unnormalised exceptions
Stefan Behnel added the comment: Here are two patches that fix this case, one with special casing, one without. Please choose and apply one. -- Added file: http://bugs.python.org/file39691/fix_stopiteration_value_slow.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23996 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24400] Awaitable ABC incompatible with functools.singledispatch
Stefan Behnel added the comment: I currently do isinstance(x, types.GeneratorType), which will fail if x is actually a GeneratorWrapper. I can change this to use collections.abc.Generator when it exists, but then I'd have to have some conditional logic to switch between collections.abc and types depending on what version I'm running on. I originally planned to make the next Cython release patch the Generator and Coroutine ABCs into collections.abc, but I now think it would be worth uploading an abc_backports package to PyPI instead that does that and on which asyncio, tornado and other packages could simply depend with a try-import. It would be nice if that were encapsulated in inspect.isgenerator(). +1, code that needs exactly a generator, e.g. for gi_running and friends, can still test for isinstance(obj, types.GeneratorType) in a completely backwards compatible way, which is more explicit anyway. More generally, the inconsistency between isgenerator() and iscoroutine() is kind of odd. I would expect that either all inspect functions or none of them would use a suitable ABC if one exists. Yes, it's odd. Either way would work (types is for types only vs. types includes protocols), but having both disagree seems wrong. I think the mere fact that there is a higher level function than isinstance() suggests that it should really be more high level and not just doing exactly the same for all times. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue24400 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24017] Implemenation of the PEP 492 - Coroutines with async and await syntax
Stefan Behnel added the comment: See issue 24400 regarding a split of yield-based generators and async-def coroutines at a type level. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue24017 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24400] Awaitable ABC incompatible with functools.singledispatch
Stefan Behnel added the comment: I added some review comments. The main thing is that the coroutine type should be awaitable. For reference, here's my current implementation for Cython. It's a bit more involved as it needs to support Python 2.6+. I may also add some special casing for CPython's own coroutine type when compiling in Py3.5 if this change makes it in. https://github.com/cython/cython/blob/bb0dec2fab91cbde443e6756c3dc29ee009caba7/Cython/Utility/Coroutine.c -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue24400 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
Re: Embedded Python and C Callback functions
doc.mefi...@gmail.com schrieb am 07.06.2015 um 10:56: And I can't use Cython, because I have C++ module, and I have to use it. That's not a valid reason. Cython supports C++ code just fine. http://docs.cython.org/src/userguide/wrapping_CPlusPlus.html Stefan -- https://mail.python.org/mailman/listinfo/python-list
[issue24383] consider implementing __await__ on concurrent.futures.Future
Stefan Behnel added the comment: (Copying my review comment here so that it doesn't get lost as it's more of a general design thing than a specific review comment.) Having Future know and deal with asyncio seems wrong to me. asyncio should be able to deal *somehow* with a Future that is being awaited, but a Future shouldn't require asyncio in order to be awaited. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue24383 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24079] xml.etree.ElementTree.Element.text does not conform to the documentation
Stefan Behnel added the comment: Looks good to me. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue24079 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
Re: Sorting in reverse is not the same as sorting then reversing
Steven D'Aprano schrieb am 05.06.2015 um 16:07: Sorting in reverse does not give the same result as sorting then reversing. It's easiest to see with a key function: py a = ['fox', 'dog', 'DOG', 'cat', 'ape'] py b = a[:] py a.sort(key=str.lower, reverse=True) py b.sort(key=str.lower) py b.reverse() py a ['fox', 'dog', 'DOG', 'cat', 'ape'] py b ['fox', 'DOG', 'dog', 'cat', 'ape'] Sorting in reverse keeps the initial order of any equal elements unchanged. Sorting, then reversing, reverses them. (Thanks to Tim Peters for the tip.) ... and for implementing this in the first place. :) For those of you who didn't know and now got interested, the relevant term here is stable sorting. It means that elements that compare equal keep their relative order. That's a general property of Python's sort algorithm. All that reverse=True does is to change lower than into greater than and vice versa for elements that compare unequal. It does not change the behaviour for elements that compare equal, which means that they keep the same relative order in both cases (reversed/non-reversed). Stefan -- https://mail.python.org/mailman/listinfo/python-list
[issue24004] avoid explicit generator type check in asyncio
Stefan Behnel added the comment: Cython functions that return a generator aren't special in any way, so it's actually correct that isgeneratorfunction() always returns False. I mean, I could set the CO_COROUTINE flag on the code object that we fake, but that wouldn't help much as Cython's functions are not Python functions, so isfunction() will also return False for them and no-one would look at the flag in the first place. And finally, a Cython generator is not a Python generator (with byte code position etc.), so isgenerator() also correctly returns False. At least iscoroutine() and iswaitable() will return True for Cython Coroutines now. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue24004 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
Re: Creating a reliable sandboxed Python environment
Laura Creighton schrieb am 30.05.2015 um 13:24: As a point of fact, We've _already got_ Topaz, a Ruby interpreter, Hippy, a PHP interpreter, a Prolog interpreter, a Smalltalk interpeter, and a javascript interpreter. Recently we got Pyket a Racket compiler. There also exist plenty of experimental languages written by academic langauge designers, and other crazy people who like such things. But don't ask the PyPy project about hard is it to sandbox one versus the other. From our point of view, they all cost the same -- free, as in _already done for you_, same as you get a JIT for free, and pluggable garbage collectors for free, etc. etc. So here the cost of security is actually rewriting the entire language runtime and potentially also major parts of its ecosystem? Not exactly a cheap price either. Stefan -- https://mail.python.org/mailman/listinfo/python-list
Re: Building an extension module with SWIG
garyr schrieb am 30.05.2015 um 18:22:
I'm trying to create an extension module using SWIG. I've
succeeded in generating a pyd file but when I import the module I get the
error message: SystemError: dynamic module not initialized properly. I
added an initfoo() function but that didn't solve the problem. Below are the
various files, a slightly modified version of a SWIG exmaple.
I'm using Python 2.7
What am I missing?
//foo.c:
#include foo.h
double Foo;
void initfoo()
{
Foo = 3.0;
}
This is wrong and you also won't need that.
int gcd(int x, int y) {
int g;
g = y;
while (x 0) {
g = x;
x = y % x;
y = g;
}
return g;
}
[...]
Just in case you're not bound to SWIG yet, here's a Cython [1] version of
your code:
# put this in a file called foo.pyx
def gcd(int x, int y):
while x 0:
y, x = x, y % x
return y
Compile it (cythonize -b foo.pyx) and you'll get an extension module that
executes faster than what SWIG would give you and keeps everything in one
file to improve readability.
Stefan
[1] http://cython.org/
--
https://mail.python.org/mailman/listinfo/python-list
[issue24004] avoid explicit generator type check in asyncio
Stefan Behnel added the comment: Hmm, I just noticed that this only started failing when I disabled the asyncio module patching for 3.5beta1+, assuming that it now all works. A side effect of that was that also the inspect module is no longer patched into returning True from isgenerator() for Cython generators. While there might be code that fails when it tries to inspect Cython generators as Python generators (as the first don't have byte code), I think it's still the easiest fix to simply keep the inspect patching. That would then trigger the right code path here without changes in asyncio. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue24004 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24004] avoid explicit generator type check in asyncio
Stefan Behnel added the comment: ... except that the returned object may not actually be a Cython generator but a GeneratorWrapper, now that the patch from issue 24316 is in place. :) Then I guess we're back to the point where duck-typing and calling __await__() is a good idea. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue24004 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24004] avoid explicit generator type check in asyncio
Stefan Behnel added the comment: Works for me. Why is the AwaitableABC type check needed, in addition to looking up the relevant method? IIUC, the type check will just do a lookup of the same method if the type hasn't been registered as Awaitable explicitly. -- status: pending - open ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue24004 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
Re: Building an extension module with SWIG
garyr schrieb am 30.05.2015 um 22:48: *snip* Compile it (cythonize -b foo.pyx) and you'll get an extension module that executes faster than what SWIG would give you and keeps everything in one file to improve readability. [1] http://cython.org/ Thanks for your reply. My interest is not in computing the gcd but to learn how build an extension module. I have some much more complicated C code I wish to use. You can do that with Cython, too. http://docs.cython.org/src/tutorial/external.html http://docs.cython.org/src/tutorial/clibraries.html I might be a bit biased as a core developer, but if the parts of you C library's API for which you have an immediate use are not so tremendously huge that it's entirely infeasible for you to write a nicely usable Python API for them, I'd always recommend using Cython over a wrapper generator like SWIG. Once you get to the points where it becomes interesting, you'll always end up having more fun writing a Cython based integration layer than fighting your up-hill battle against the way the wrapper generator wants you to design it. Stefan -- https://mail.python.org/mailman/listinfo/python-list
[issue24004] avoid explicit generator type check in asyncio
Stefan Behnel added the comment: I found one more place in asyncio.coroutines, around line 190 in the coroutine() decorator: if inspect.isgeneratorfunction(func): coro = func else: @functools.wraps(func) def coro(*args, **kw): res = func(*args, **kw) if isinstance(res, futures.Future) or inspect.isgenerator(res): res = yield from res return res The wrapped isgenerator() check should be replaced by an Awaitable ABC check, e.g. this works for me: res = func(*args, **kw) if isinstance(res, futures.Future) or inspect.isgenerator(res): res = yield from res +else: +await_res = getattr(res, '__await__', None) +if await_res is not None: +res = yield from await_res() return res -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue24004 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24325] Speedup types.coroutine()
Stefan Behnel added the comment: I would still like to see a patch in Py3.5 that only flips the bit in C when _types is available and otherwise falls back to the existing Python code that creates a new code object. This part is much cleaner and faster to do in C than in the current Python implementation. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue24325 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
Re: Creating a reliable sandboxed Python environment
Chris Angelico schrieb am 28.05.2015 um 20:51: On Fri, May 29, 2015 at 4:41 AM, Stefan Behnel wrote: davidf...@gmail.com schrieb am 26.05.2015 um 04:24: Has anyone on this list attempted to sandbox Python programs in a serious fashion? I'd be interested to hear your approach. Not quite sandboxing Python, but I've seen people use my Lupa [1] library for this. They're writing all their code in Python, and then let users embed their own Lua code into it to script their API. The Lua runtime is apparently quite good at sandboxing, and it's really small, just some 600KB or so. Lupa then lets you easily control the access to your Python code at a whitelist level by intercepting all Python attribute lookups. It doesn't add much to your application to embed Lua (or even LuaJIT) in Python, and it gives users a nicely object oriented language to call and orchestrate your Python objects. Lua's a much weaker language than Python is, though. Can it handle arbitrary-precision integers? Unicode? Dare I even ask, arbitrary-precision rationals (fractions.Fraction)? All of those and way more, as long as you use it embedded in Python. Security comes at a price, I guess. Sure, but features aren't the price here. Stefan -- https://mail.python.org/mailman/listinfo/python-list
Re: Creating a reliable sandboxed Python environment
Chris Angelico schrieb am 29.05.2015 um 09:41: On Fri, May 29, 2015 at 4:18 PM, Stefan Behnel wrote: Lua's a much weaker language than Python is, though. Can it handle arbitrary-precision integers? Unicode? Dare I even ask, arbitrary-precision rationals (fractions.Fraction)? All of those and way more, as long as you use it embedded in Python. Okay, so how would you go about using Lua-embedded-in-Python to manipulate Unicode text? Lua only supports byte strings, so Lupa will encode and decode them for you. If that's not enough, you'll have to work with Python Unicode string objects through the language interface. (And I just noticed that the handling can be improved here by overloading Lua operators with Python operators - not currently implemented.) Looks to me as if Lua doesn't have integers at all The standard number type in Lua is a C double float, i.e. the steady integer range is somewhere within +/-2^53. That tends to be enough for a *lot* of use cases. You could change that type in the Lua C code (e.g. to a 64 bit int), but that's usually a bad idea. The same comment as above applies: if you need Python object features, use Python objects. Embedding Lua in Python gives you access to all of Python's objects and ecosystem. It may not always be as cool to use as from Python, but in that case, why not code it in Python in the first place? You wouldn't use Lua/Lupa to write whole applications, just the user defined parts of them. The rest can happily remain in Python. And should, for your own sanity. Stefan -- https://mail.python.org/mailman/listinfo/python-list
[issue24268] PEP 489 -- Multi-phase extension module initialization
Stefan Behnel added the comment: I'm seeing crashes with this assertion in Cython test modules: python: ./Python/importdl.c:75: get_encoded_name: Assertion `(((PyObject*)(encoded))-ob_refcnt) == 1' failed. The problem seems to be that the module name is only one character long (a), and single character byte strings are interned. -- nosy: +scoder ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue24268 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24268] PEP 489 -- Multi-phase extension module initialization
Stefan Behnel added the comment: Patch LGTM and it fixes the problem (tried it on my side). Please make sure it gets applied in time for beta 2. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue24268 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24017] Implemenation of the PEP 492 - Coroutines with async and await syntax
Stefan Behnel added the comment: Tried it, works for me. Thanks! -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue24017 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24316] Fix types.coroutine to accept objects from Cython
Stefan Behnel added the comment:
I just noticed that I hadn't used the real types.coroutine in my Py3.5 tests
when reporting back in issue 24017. When I pass a Cython generator through it,
I get
Traceback (most recent call last):
File tests/run/test_coroutines_pep492.pyx, line 245, in
test_coroutines_pep492.CoroutineTest.test_func_5
(test_coroutines_pep492.c:13445)
for el in bar():
File /opt/python3.5/lib/python3.5/types.py, line 197, in wrapped
'non-coroutine: {!r}'.format(coro))
TypeError: callable wrapped with types.coroutine() returned non-coroutine:
generator object at 0x7f178c458898
This is actually obvious, given that the sole purpose of the decorator is to
turn something that is a Generator and *not* a Coroutine into something that is
a Coroutine, as a means for the user to say but I know better. So checking
for the return value being a Coroutine is wrong. Instead, it should check that
it's a Generator and if it's not an Awaitable, wrap it as a self-returning
Awaitable. That's more or less what my proposed implementation in issue 24017
did:
class types_coroutine(object):
def __init__(self, gen):
self._gen = gen
class as_coroutine(object):
def __init__(self, gen):
self._gen = gen
self.send = gen.send
self.throw = gen.throw
self.close = gen.close
def __await__(self):
return self._gen
def __call__(self, *args, **kwargs):
return self.as_coroutine(self._gen(*args, **kwargs))
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue24316
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24316] Fix types.coroutine to accept objects from Cython
Stefan Behnel added the comment: One failing test in test_coroutines: test_func_5. The reason is that the GeneratorWrapper is not iterable (and there is no reason it shouldn't be, given that it wraps a Generator). That was my fault, I had already added an __iter__ method but didn't copy it in my previous message. Adding it as follows fixes the test for me: def __iter__(self): return self.__wrapped__ Alternatively, __iter__ = __await__ would do the same. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue24316 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24316] Fix types.coroutine to accept objects from Cython
Stefan Behnel added the comment: BTW, it's not only for compiled generators but also for normal Python functions that construct Python generators internally and return them, or that delegate the generator creation in some way. With this change, it's enough to decorate the constructor function and not each of the potential generators that it returns. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue24316 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24079] xml.etree.ElementTree.Element.text does not conform to the documentation
Stefan Behnel added the comment: IMHO less clear and less correct than the previous suggestions. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue24079 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24079] xml.etree.ElementTree.Element.text does not conform to the documentation
Stefan Behnel added the comment: Seems like a good idea to explain text and tail in one section, though. That makes tail easier to find for those who are not used to this kind of split (and that's basically everyone who needs to read the docs in the first place). -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue24079 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24316] Fix types.coroutine to accept objects from Cython
Stefan Behnel added the comment: Ok, now the problem with *this* patch is that __iter__ and __await__ are special methods that are being looked up on the type, not the instance. Similarly __next__, I think, as it also has its own (type) slot. But I think you're right that __next__ is also needed here. I'm attaching a patch that works for me. -- Added file: http://bugs.python.org/file39556/types_coroutine.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue24316 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24316] Fix types.coroutine to accept objects from Cython
Stefan Behnel added the comment: Your latest patch works for me. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue24316 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
Re: Creating a reliable sandboxed Python environment
davidf...@gmail.com schrieb am 26.05.2015 um 04:24: Has anyone on this list attempted to sandbox Python programs in a serious fashion? I'd be interested to hear your approach. Not quite sandboxing Python, but I've seen people use my Lupa [1] library for this. They're writing all their code in Python, and then let users embed their own Lua code into it to script their API. The Lua runtime is apparently quite good at sandboxing, and it's really small, just some 600KB or so. Lupa then lets you easily control the access to your Python code at a whitelist level by intercepting all Python attribute lookups. It doesn't add much to your application to embed Lua (or even LuaJIT) in Python, and it gives users a nicely object oriented language to call and orchestrate your Python objects. Stefan [1] https://pypi.python.org/pypi/lupa -- https://mail.python.org/mailman/listinfo/python-list
[issue24017] Implemenation of the PEP 492 - Coroutines with async and await syntax
Stefan Behnel added the comment: Yield-from iterates, and a coroutine is not supposed to be iterable, only awaitable (at least, that's what all error messages tell me when I try it). So why should yield from work on them? What if foo() was not an Iterable but a Coroutine? Should yield from then call __await__ on it internally? I would find that *really* surprising, but given the above, I think it would be necessary to achieve consistency. This is a special backwards-compatibility thing. That only answers the half-serious first part of my question. ;) This code only works if foo() returns an Iterable, including a (yield) coroutine: @types.coroutine def bar(): return (yield from foo()) It does not work for arbitrary Coroutines as they are not iterable, but it might trick people into writing code that fails for non-coroutine Coroutines. I'd rather like to have this either work for any Coroutine or not at all. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue24017 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24017] Implemenation of the PEP 492 - Coroutines with async and await syntax
Stefan Behnel added the comment: BTW, given that iter(iterator) works and returns the iterator, should we also allow await x.__await__() to work? I guess that would be tricky to achieve given that __await__() is only required to return any kind of arbitrary Iterator, and Iterators cannot be awaited due to deliberate restrictions. But it might be nice to have for wrapping purposes. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue24017 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24017] Implemenation of the PEP 492 - Coroutines with async and await syntax
Stefan Behnel added the comment: Can't your Coroutine object return itself from its __await__, and implement __next__? Like genobject in CPython simply returns self from its __iter__. That was my first try, sure, and it mostly worked. It has a drawback, though: it's an incomplete implementation of the Iterator protocol. It's still (mostly) an Iterator, but not an Iterable, so it depends on how you use it whether you notice or not, and whether it works at all with other code or not. There's a test for a failing next(coro) in your test suite, for example, which would then not fail in Cython. OTOH, code cannot assume that calling iter() or for-looping over on an Iterable is a sane thing to do, because it doesn't work for Python's generator type based coroutine either, so we might get away with it... All of these little details make this trick appear like a potential source of subtle inconsistencies or incompatibilities. But given that there will almost certainly remain inconsistencies for compiled Python code, I'm not sure yet which approach is better. It's not impossible that I'll end up going back to the original design. I guess I'll eventually have to include some benchmarks in the decision. On a related note, my testing made me stumble over this code in asyncio.tasks.Task(): if coro.__class__ is types.GeneratorType: self._coro = coro else: self._coro = iter(coro) # Use the iterator just in case. This seems wrong regardless of how you look at it. And it definitely fails with both designs. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue24017 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24017] Implemenation of the PEP 492 - Coroutines with async and await syntax
Stefan Behnel added the comment: Thanks Yury, I'll give it a try ASAP. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue24017 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24017] Implemenation of the PEP 492 - Coroutines with async and await syntax
Stefan Behnel added the comment: It *is* correct, see PEP 492. Awaitable is either a coroutine *or* an object with an __await__ method. coroutine, yes. But Coroutine? Shouldn't the Coroutine ABC then require __await__ to be implemented? Maybe even by inheriting from Awaitable? Just implement tp_await/__await__ for coroutine-like objects coming from C-API or Cython. Sure, that's how it's done. (Specifically, Coroutine is not an Iterable/Iterator, but its __await__() returns a thin Iterator that simply calls into the Generator code. A bit annoying and slowish, but that's what it takes.) I was just wondering how Cython should compile Python code that makes use of this decorator. The Coroutine and Generator types are separated in Cython now, and I think that's actually the right thing to do. This types.coroutine() decorator and special casing in CPython's code base gets a bit in the way here. In general, iteration protocol is still the foundation for Future-like objects That's not really reflected in the ABCs, is it? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue24017 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23275] Can assign [] = (), but not () = []
Changes by Stefan Behnel sco...@users.sourceforge.net: -- nosy: +scoder ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23275 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24017] Implemenation of the PEP 492 - Coroutines with async and await syntax
Stefan Behnel added the comment: I added a couple of review comments to patch 6, but since no-one has responded so far, I guess they simply haven't been noticed. So I'll just repeat them here. 1) getawaitablefunc / aiternextfunc / getaiterfunc Is there a reason why these need to have their specific C type name instead of just reusing unaryfunc, or at least the existing iternextfunc / getiterfunc? They are unprefixed global names in the C namespace and I think we should be careful when adding more of those. 2) Awaitable.register(Coroutine) I think this is incorrect. A Coroutine is not Awaitable unless it also implements __await__. How else should it be awaited? 3) I propose to use this wrapping code as a fallback for types.coroutine() in the case that a Generator (ABC) is passed instead of a generator (yield): class types_coroutine(object): def __init__(self, gen): self._gen = gen class as_coroutine(object): def __init__(self, gen): self._gen = gen self.send = gen.send self.throw = gen.throw self.close = gen.close def __await__(self): return self._gen def __call__(self, *args, **kwargs): return self.as_coroutine(self._gen(*args, **kwargs)) Note that the resulting Awaitable Coroutine type is not an Iterable. This differs from a (yield) coroutine, but it matches the Coroutine and Awaitable protocols, and the intention to separate both in order to avoid mistakes on user side. Additionally, regarding the tests: 4) def test_func_2(self): async def foo(): raise StopIteration with self.assertRaisesRegex( RuntimeError, generator raised StopIteration): run_async(foo()) Why is this actually necessary? I'm aware that it's also mentioned in the PEP, but is there an actual reason why a coroutine should behave the same as a generator here? Is it just an implementation detail for legacy reasons because generators and coroutines happen to share the same type implementation? (I don't think they need to, BTW.) 5) def test_func_8(self): @types.coroutine def bar(): return (yield from foo()) async def foo(): return 'spam' self.assertEqual(run_async(bar()), ([], 'spam') ) I find it surprising that this works at all. Yield-from iterates, and a coroutine is not supposed to be iterable, only awaitable (at least, that's what all error messages tell me when I try it). So why should yield from work on them? What if foo() was not an Iterable but a Coroutine? Should yield from then call __await__ on it internally? I would find that *really* surprising, but given the above, I think it would be necessary to achieve consistency. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue24017 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24017] Implemenation of the PEP 492 - Coroutines with async and await syntax
Stefan Behnel added the comment: Another question: is it ok if Cython implements and uses the tp_as_async slot in all Py3.x versions (3.2+)? It shouldn't hurt, but it would speed up async/await in Cython at least under Py3.x. Only Py2.6/7 would then have to resort to calling __await__() etc. at the Python level. One drawback is that Py3.5 currently (needlessly) checks that tp_reserved and tp_richcompare are both implemented, but that can be worked around by also implementing the latter... -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue24017 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24287] Let ElementTree prolog include comments and processing instructions
Stefan Behnel added the comment: FTR, lxml's Element class has addnext() and addprevious() methods which are commonly used for this purpose. But ET can't adopt those due to its different tree model. I second Martin's comment that ET.append() is a misleading name. It suggests adding stuff to the end, whereas things are actually being inserted before the root element here. I do agree, however, that this is a helpful feature and that the ElementTree class is a good place to expose it. I propose to provide a prolog (that's the spec's spelling) property holding a list that users can fill and modify any way they wish. The serialiser would then validate that all content is proper XML prologue content, and would serialise it in order. My guess is that lxml would eventually use a MutableSequence here that maps changes directly to the underlying tree (and would thus validate them during modification), but ET can be more lenient, just like it allows arbitrary objects in the text and tail properties which only the serialiser rejects. Note that newlines can easily be generated on user side by setting the tail of a PI/Comment to \n. (The serialiser must also validate that the tail text is only allowed whitespace.) For reference: http://www.w3.org/TR/REC-xml/#sec-prolog-dtd -- components: +XML ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue24287 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23996] _PyGen_FetchStopIterationValue() crashes on unnormalised exceptions
Stefan Behnel added the comment: I noticed that my patch isn't entirely correct. If the exception value is a tuple, both PyErr_SetObject() and PyErr_NormalizeException() use it directly as *argument tuple* for the exception instantiation call, i.e. they essentially unpack it into separate arguments. The StopIteration value is then only the first item of that tuple. I wonder if it's worth repeating this, uhm, surprising special case in yet another place, or if we should just always instantiate the exception. -- status: closed - open ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23996 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue17239] XML vulnerabilities in Python
Changes by Stefan Behnel sco...@users.sourceforge.net: -- nosy: +scoder ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue17239 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24270] PEP 485 (math.isclose) implementation
Stefan Behnel added the comment: Eventually, I think a corresponding function should be added to cmath. math.isclose() shouldn't deal with complex values by itself (other than rejecting them as non-floatish input). -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue24270 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24270] PEP 485 (math.isclose) implementation
Stefan Behnel added the comment: The cast is correct and required (the actual signature is determined by the METH_* flags). Patch LGTM, FWIW. -- components: +Library (Lib) nosy: +scoder type: - enhancement ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue24270 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24221] Clean-up and optimization for heapq siftup() and siftdown()
Stefan Behnel added the comment: There are calls to PyObject_RichCompareBool() in the loops, which means that user code might get executed. What if that's evil code that modifies the heap list? Couldn't that lead to list resizing and thus invalidation of the list item pointer? -- nosy: +scoder ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue24221 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
Re: Strategies for using cffi with C++ code?
Skip Montanaro schrieb am 22.05.2015 um 19:15: 2015-05-22 12:05 GMT-05:00 Lele Gaifax: Maybe http://docs.cython.org/src/userguide/wrapping_CPlusPlus.html? Thanks for the reference, Lele. I was thinking in terms of cffi, but this might work as well. (Shouldn't cffi interfaces be the thinnest?) Thin in what sense? cffi is a generic library (at least on CPython), so the interface can never be as thin as what a dedicated compiler can generate for a specific piece of interface code. PyPy's cffi can mitigate this by applying runtime optimisations, but you can't do that in CPython without running some kind of native code generator, be it a JIT compiler or a static compiler. cffi can apply the latter (run a C compiler) to a certain extent, but then you end up with a dependency on a C compiler at *runtime*. The term thin really doesn't apply to that dependency. And even if you accept to go down that route, you'd still get better results from runtime compilation with Cython, as it will additionally optimise your interface code (and thus make it thinner). Being a Cython core developer, I'm certainly a somewhat biased expert, but using Cython to generate a statically compiled and optimised C++ wrapper is really your best choice. IMHO, it provides the best results/tradeoffs in terms of developer effort, runtime performance and overall end user experience. Stefan -- https://mail.python.org/mailman/listinfo/python-list
[issue24221] Clean-up and optimization for heapq siftup() and siftdown()
Stefan Behnel added the comment: Ah, sorry, yes. I somehow misread the arguments being passed *into* richcompare as subsequent array access. LGTM too. Sorry for the noise. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue24221 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
Re: Calling a function is faster than not calling it?
Steven D'Aprano schrieb am 10.05.2015 um 11:58: Why is calling a function faster than bypassing the function object and evaluating the code object itself? And not by a little, but by a lot? Here I have a file, eval_test.py: # === cut === from timeit import Timer def func(): a = 2 b = 3 c = 4 return (a+b)*(a-b)/(a*c + b*c) code = func.__code__ assert func() == eval(code) t1 = Timer(eval; func(), setup=from __main__ import func) t2 = Timer(eval(code), setup=from __main__ import code) # Best of 10 trials. print (min(t1.repeat(repeat=10))) print (min(t2.repeat(repeat=10))) # === cut === Note that both tests include a name lookup for eval, so that as much as possible I am comparing the two pieces of code on an equal footing. Here are the results I get: [steve@ando ~]$ python2.7 eval_test.py 0.804041147232 1.74012994766 [steve@ando ~]$ python3.3 eval_test.py 0.7233301624655724 1.7154695875942707 Directly eval'ing the code object is easily more than twice as expensive than calling the function, but calling the function has to eval the code object. Well, yes, but it does so directly in C code. What you are essentially doing here is replacing a part of the fast C code path for executing a Python function by some mostly equivalent but more general Python code. So, you're basically replacing a function call by another function call to eval(), plus some extra generic setup overhead. Python functions know exactly what they have to do internally in order to execute. eval() cannot make the same streamlined assumptions. Stefan -- https://mail.python.org/mailman/listinfo/python-list
[issue24165] Free list for single-digits ints
Stefan Behnel added the comment: Well, as I've shown in issue 24076 (I'm copying the numbers here), even simple arithmetic expressions can benefit from a free-list. Basically anything that uses temporary integer results. Original: $ ./python -m timeit 'sum(range(1, 10))' 1000 loops, best of 3: 1.86 msec per loop $ ./python -m timeit -s 'l = list(range(1000, 1))' '[(i*2+5) // 7 for i in l]' 1000 loops, best of 3: 1.05 msec per loop With freelist: $ ./python -m timeit 'sum(range(1, 10))' 1000 loops, best of 3: 1.52 msec per loop $ ./python -m timeit -s 'l = list(range(1000, 1))' '[(i*2+5) // 7 for i in l]' 1000 loops, best of 3: 931 usec per loop -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue24165 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24076] sum() several times slower on Python 3
Stefan Behnel added the comment: Issue 24165 was created to pursue the path of a free-list for PyLong objects. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue24076 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24165] Free list for single-digits ints
Stefan Behnel added the comment: I got similar results on 64bits for my original patch (very similar to what Serhiy used now). The numbers are not really conclusive. Report on Linux leppy 3.13.0-46-generic #77-Ubuntu SMP Mon Mar 2 18:23:39 UTC 2015 x86_64 x86_64 Total CPU cores: 4 ### 2to3 ### 6.885334 - 6.829016: 1.01x faster ### etree_process ### Min: 0.249504 - 0.253876: 1.02x slower Med: 0.252730 - 0.258274: 1.02x slower Avg: 0.254332 - 0.261100: 1.03x slower Significant (t=-5.99) Stddev: 0.00478 - 0.00640: 1.3391x larger ### fastpickle ### Min: 0.402085 - 0.416765: 1.04x slower Med: 0.405595 - 0.424729: 1.05x slower Avg: 0.405882 - 0.429707: 1.06x slower Significant (t=-12.45) Stddev: 0.00228 - 0.01334: 5.8585x larger ### json_dump_v2 ### Min: 2.611031 - 2.522507: 1.04x faster Med: 2.678369 - 2.544085: 1.05x faster Avg: 2.706089 - 2.552111: 1.06x faster Significant (t=10.40) Stddev: 0.09551 - 0.04290: 2.2265x smaller ### nbody ### Min: 0.217901 - 0.214968: 1.01x faster Med: 0.224340 - 0.216781: 1.03x faster Avg: 0.226012 - 0.216981: 1.04x faster Significant (t=6.03) Stddev: 0.01049 - 0.00142: 7.4102x smaller ### regex_v8 ### Min: 0.040856 - 0.039377: 1.04x faster Med: 0.041847 - 0.040082: 1.04x faster Avg: 0.042468 - 0.040726: 1.04x faster Significant (t=3.20) Stddev: 0.00291 - 0.00252: 1.1549x smaller The following not significant results are hidden, use -v to show them: etree_generate, etree_iterparse, etree_parse, fastunpickle, json_load. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue24165 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24017] Implemenation of the PEP 492 - Coroutines with async and await syntax
Stefan Behnel added the comment: We also need a Coroutine ABC. Both the GeneratorType and CO_COROUTINE checks are too restrictive. Also see issue 24018, which this one should in fact depend on. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue24017 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24018] add a Generator ABC
Stefan Behnel added the comment: This is blocking issue 24017 (async/await syntax). -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue24018 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24018] add a Generator ABC
Stefan Behnel added the comment: Thanks! Minor grouch: it should say collections.*abc*.Generator in the NEWS entry. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue24018 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue2292] Missing *-unpacking generalizations
Stefan Behnel added the comment:
I get a test failure in Cython's compatibility tests which seems to be
attributable to this change:
def sideeffect(x):
... L.append(x)
... return x
def unhashable(x):
... L.append(x)
... return [x]
L = []
{1:2, sideeffect(2): 3, 3: 4, unhashable(4): 5, sideeffect(5): 6}#
doctest: +ELLIPSIS
Traceback (most recent call last):
TypeError: ...unhashable...
L
[2, 4]
Instead, L ends up being [2, 4, 5]. Is this intended? Or acceptable?
--
nosy: +scoder
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue2292
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24138] Speed up range() by caching and modifying long objects
Stefan Behnel added the comment: See issue 24076. -- nosy: +scoder ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue24138 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22881] show median in benchmark results
Stefan Behnel added the comment: I'm actually not sure how it relates to the minimum. The more runs you have, the higher the chance of hitting the actual minimum at least once. And if none of the runs hits the real minimum, you're simply out of luck. However, it should tend to give a much better result than the (currently printed) average, which suffers from outliers. And outliers are almost always too high for benchmarks and never too low, due to various external influences. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22881 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22881] show median in benchmark results
Stefan Behnel added the comment: Well, we can apply a kludge, or apply statistics. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22881 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22881] show median in benchmark results
Stefan Behnel added the comment: for the even number case, I think you shouldn't do // 2, but / 2. Right. I updated the patch. -- Added file: http://bugs.python.org/file39279/show_median.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22881 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24076] sum() several times slower on Python 3
Stefan Behnel added the comment: I tried implementing a freelist. Patch attached, mostly adapted from the one in dictobject.c, but certainly needs a bit of cleanup. The results are not bad, about 10-20% faster: Original: $ ./python -m timeit 'sum(range(1, 10))' 1000 loops, best of 3: 1.86 msec per loop $ ./python -m timeit -s 'l = list(range(1000, 1))' '[(i*2+5) // 7 for i in l]' 1000 loops, best of 3: 1.05 msec per loop With freelist: $ ./python -m timeit 'sum(range(1, 10))' 1000 loops, best of 3: 1.52 msec per loop $ ./python -m timeit -s 'l = list(range(1000, 1))' '[(i*2+5) // 7 for i in l]' 1000 loops, best of 3: 931 usec per loop -- keywords: +patch Added file: http://bugs.python.org/file39245/pylong_freelist.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue24076 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24076] sum() several times slower on Python 3
Stefan Behnel added the comment: I don't think it's irrelevant. Throw-away integers are really not uncommon. For-loops use them quite often, non-trivial arithmetic expressions can create a lot of intermediate temporaries. Speeding up the create-delete cycle of PyLong sounds like a very obvious thing to do. Imagine some code that iterates over a list of integers, applies some calculation to them, and then stores them in a new list, maybe even using a list comprehension or so. If you could speed up the intermediate calculation by avoiding overhead in creating temporary PyLong objects, such code could benefit a lot. I suspect that adding a free-list for single-digit PyLong objects (the most common case) would provide some visible benefit. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue24076 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24017] Implemenation of the PEP 492 - Coroutines with async and await syntax
Stefan Behnel added the comment: I don't think it's necessary to have slots for __aiter__, __anext__, __aenter__ and __aexit__. Async iteration will never be as fast as regular iteration, and there is plenty overhead in it. You seem to be assuming that the outer loop is the asyncio I/O loop. That might not be the case. It might be a thread-pool executor, it might be an in-memory task switcher, or it might just be something passing on items from a list. At least __anext__ is worth being fast, IMHO. Also note that one advantage of slots is that the user can cache their value to keep calling the same C function multiple times. That is not the case for a Python method, which can easily be replaced. Some iterators do that with their __next__ method, and it's perfectly valid. Having to look up the Python method on each iteration and calling through it sounds like unnecessary overhead. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue24017 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24079] xml.etree.ElementTree.Element.text does not conform to the documentation
Stefan Behnel added the comment: I agree that the wording in the documentation isn't great: text The text attribute can be used to hold additional data associated with the element. As the name implies this attribute is usually a string but may be any application-specific object. If the element is created from an XML file the attribute will contain any text found between the element tags. tail The tail attribute can be used to hold additional data associated with the element. This attribute is usually a string but may be any application-specific object. If the element is created from an XML file the attribute will contain any text found after the element’s end tag and before the next tag. Special cases that no-one uses (sticking non-string objects into text/tail) are given too much space and the difference isn't explained as needed. Since the distinction between text and tail is a (great but) rather special feature of ElementTree, it needs to be given more room in the docs. Proposal: text The text attribute holds the immediate text content of the element. It contains any text found up to either the closing tag if the element has no children, or the next opening child tag within the element. For text following an element, see the `tail` attribute. To collect the entire text content of a subtree, see `tostring`. Applications may store arbitrary objects in this attribute. tail The tail attribute holds any text that directly follows the element. For example, in a document like ``aTextb/BTailc/CTail/a``, the `text` attribute of the ``a`` element holds the string Text, and the tail attributes of ``b`` and ``c`` hold the strings BTail and CTail respectively. Applications may store arbitrary objects in this attribute. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue24079 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24076] sum() several times slower on Python 3
Stefan Behnel added the comment: there are three ingredients here - sum, (x)range and the integer addition that sum will be performing at each iteration. ... not to forget the interpreter startup time on his machine. :) I did a tiny bit of profiling and about 90% of the time seems to be spent creating and deallocating throw-away PyLong objects. My guess is that it simply lacks a free-list in _PyLong_New(). -- nosy: +scoder ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue24076 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24018] add a Generator ABC
Stefan Behnel added the comment: Please either 1) drop the throw() method entirely or 2) make throw an abstractmethod() Ok, as I already said, I think it's important to provide the complete protocol as code will usually expect that. Also, close() has a helpful implementation, but it depends on throw(). Thus, I'll go with 2) and make throw() abstract. It's easy enough to call super(), but then it's explicit. I agree that that's better here. Patch updated. -- Added file: http://bugs.python.org/file39213/generator_abc.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue24018 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24053] Define EXIT_SUCCESS and EXIT_FAILURE constants in sys
Stefan Behnel added the comment: why not spell them sys.exit.FAILURE and sys.exit.SUCCESS ? -- nosy: +scoder ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue24053 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24053] Define EXIT_SUCCESS and EXIT_FAILURE constants in sys
Stefan Behnel added the comment: Actually, my guess is also that these constants will not be used. Not because they are not helpful, but because they'd only be available in Python 3.5+. Meaning that if you use them, your code won't work with long time supported CPython versions like 3.4 for the next decade or so. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue24053 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22486] Add math.gcd()
Stefan Behnel added the comment: Any more comments on this? The deadlines for new features in Py3.5 are getting closer. It seems we're just discussing details here, but pretty much everyone wants this feature. So, what are the things that still need to be done? Serhiy submitted working patches months ago. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22486 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20485] Enable non-ASCII extension module names
Stefan Behnel added the comment: PEP 489 (Redesigning extension module loading) includes the proposal to fix this by using punycode. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20485 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22881] show median in benchmark results
Stefan Behnel added the comment: Any more comments on the patch, or can it be applied? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22881 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23996] _PyGen_FetchStopIterationValue() crashes on unnormalised exceptions
Changes by Stefan Behnel sco...@users.sourceforge.net: -- nosy: +pitrou ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23996 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24017] Implemenation of the PEP 492 - Coroutines with async and await syntax
Stefan Behnel added the comment: Could we have type slots for the new special methods? Otherwise, implementing the protocol in C would be fairly inefficient, especially for async iteration. I'm asking because Cython's generator type is not Python's generator type, but implementing the rest of the proposed protocol doesn't seem to be all that difficult. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue24017 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24018] add a Generator ABC
Stefan Behnel added the comment: PEP 342 isn't really conclusive here as it intended to define the protocol based on the de-facto design of a yield-based generator function. Trying to abstract from that poses the question how a class based generator implementation should look like. Specifically, it is not clear in this case what raise an exception *inside* the generator even means. As Antoine pointed out, there definitely has to be more than an Iterator. However, it's not clear that throw() and close() are always required for an implementation. Simple Generators might get away without special error handling and cleanup. Similarly, a Generator may not allow (or expect) values to be passed in and only make use of close() for cleanup (and potentially also throw()). So, there seem to be at least three use cases for the Generator protocol here: 1) an Iterator that can accept input values via send() 2) an Iterator that needs to safely do certain cleanup operations after use (throw/close) 3) a complete Generator that accepts input values and cleans up after it Given that there used to be only one implementation, I think we can assume that a lot of code depends on the complete protocol. Some recipients will certainly be aware of the exact subset of the protocol that the Generator they have in their hands makes use of, but if they don't, they'll just have to assume it supports everything and requires proper cleanup on errors and on termination. Therefore, I think it's important to cover the complete protocol in the Generator ABC. I also think it's helpful to not require users to override throw() in a subclass, as they might not need it. Throwing an exception inside of them might simply not have a meaning for them. If they do need it, however, then the current implementation might help them to properly raise the exception, given that the 3-argument raise statement is no longer available in Py3. Both reasons (whether you need throw() or not) make me think that it should not be abstract. The same applies to close(), which has a meaningful implementation now that subclasses can make use of in order to avoid having to implement both close() and throw(). And yes, this is about sneaking past generator type checks because most of them are actually *not* there for a reason. Asyncio is just one example. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue24018 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22881] show median in benchmark results
Stefan Behnel added the comment: In general, wouldn't it be good to let the statistics module do all the stats calculations? It's not available in older Python versions, e.g. 2.6. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22881 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24018] add a Generator ABC
Stefan Behnel added the comment: FYI, here's the patch implementation for Cython: https://github.com/cython/cython/blob/cf63ff71c06b16c3a30facdc7859743f4cd495f6/Cython/Utility/Generator.c#L849 The only difference is that it takes care of changing __next__ to next in Py2.x. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue24018 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24018] add a Generator ABC
Stefan Behnel added the comment: Searching for Python code that seems to implement the Generator protocol doesn't return much: https://code.openhub.net/search?s=%22def%20throw%28%22%20%22def%20send%28%22%20%22def%20close%28%22pp=0fl=Pythonmp=1ml=1me=1md=1ff=1filterChecked=true But at least it pointed me to this: https://hg.python.org/cpython/file/5c0247a6f98a/Lib/multiprocessing/managers.py#l922 So, wrapping generators (apparently for remote calls in this case) seems to be another use case. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue24018 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24018] add a Generator ABC
Stefan Behnel added the comment: Adding a patch for the inspect docs that refers to the Right Way To Do It. -- Added file: http://bugs.python.org/file39179/inspect_docs.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue24018 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24018] add a Generator ABC
Changes by Stefan Behnel sco...@users.sourceforge.net: Removed file: http://bugs.python.org/file39179/inspect_docs.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue24018 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24018] add a Generator ABC
Changes by Stefan Behnel sco...@users.sourceforge.net: Added file: http://bugs.python.org/file39180/inspect_docs.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue24018 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24018] add a Generator ABC
Stefan Behnel added the comment: Good catch with the RuntimeError. Patch updated. -- Added file: http://bugs.python.org/file39157/generator_abc.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue24018 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24018] add a Generator ABC
Stefan Behnel added the comment: should inspect.isgenerator() be changed? Replying to myself, one thing that speaks against this approach is that inspect also has the functions getgeneratorlocals() and getgeneratorstate(), which depend on gen.gi_frame. Cython could emulate that, but normal user code usually can't. It's definitely not part of the Generator protocol but an implementation detail of GeneratorType. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue24018 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24018] add a Generator ABC
Stefan Behnel added the comment: Yes, and there are certainly other compilers and tools. However, it's going to be a short list, so if Py3.5 takes the lead, they can all just agree on the one way to do it and let the first patcher win. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue24018 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24018] add a Generator ABC
Stefan Behnel added the comment: Yes, code usually doesn't fall from the sky with a new CPython release. If something wants to make use of this ABC, it still has to find ways to also work with older CPythons. What would be a good fallback? A backport on PyPI? I wouldn't even mind shipping this class with Cython and dropping it right into collections.abc if it's missing. Cython already contains lots of compatibility code, and we need to patch *something* in all current CPythons either way. Then your code snippet would be the right thing to do for user code (with the twist that Py2.x doesn't have collections.abc...). -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue24018 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24018] add a Generator ABC
Stefan Behnel added the comment: Here's a new patch that addresses the review comments. I kept throw() and close() non-abstract and added an example to the tests instead that implements a minimal generator by inheriting these methods from the Generator base class, using it as a mixin. It only needs to implement send(). I don't think it's unreasonable to assume that there are use cases where a generator that is implemented in a class instead of a yield-function does not require special cleanup actions in its close()/throw(). Or is it *required* that a generator stops working when close() is called? There are also iterators that raise StopIteration at some point to signal that they are temporarily exhausted, but then eventually restart returning values from their __next__() method when they have received more to return. Avoids recreating the iterator object after each exhaustion cycle. I extended the default implementation of close() to call throw(GeneratorExit), though, because that's how the protocol is defined. Unless users implement throw(), it won't make a difference for them. And if they do, they may even get away with not reimplementing close(). -- Added file: http://bugs.python.org/file39156/generator_abc.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue24018 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24018] add a Generator ABC
Stefan Behnel added the comment: Next question: should inspect.isgenerator() be changed? Or should its usages be replaced by isinstance(obj, Generator) ? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue24018 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24018] add a Generator ABC
Stefan Behnel added the comment: Is it a problem that the check can't be done in a fast way from C code? C code can still quickly special case the generator type in the positive case, and it will usually be interested in an exact type match anyway. Determining that an object is *not* (compatible with) a generator might lead to some degradation, but in most cases, at least in the interpreter core, this would be an error case anyway, so being slower here should rarely hurt. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue24018 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24004] avoid explicit generator type check in asyncio
Stefan Behnel added the comment: I was (silently) hoping that this patching would eventually not be necessary anymore because the one place (currently inspect.isgenerator()) would be adapted to check for the generator protocol rather than the generator type. But that was going to go into a separate ticket. :) I'm not sure inspect.isgenerator() is the right place, though, as the module tends to deal with types, not protocols. I think the right fix overall would be a Generator ABC class that defines the protocol. What Cython does now in order to make asyncio work with Cython compiled generators is this: https://github.com/cython/cython/blob/4af42443bd37f6207143f8870904f780a65bb3e3/Cython/Utility/Generator.c#L824 https://github.com/cython/cython/blob/4af42443bd37f6207143f8870904f780a65bb3e3/Cython/Utility/Generator.c#L906 Aweful, but currently required due to the type check, which prevents objects that implement the generator protocol from being handled correctly by asyncio. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue24004 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24004] avoid explicit generator type check in asyncio
Stefan Behnel added the comment: I created issue 24018 for adding a Generator ABC to collections.abc. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue24004 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24017] Implemenation of the PEP 492 - Coroutines with async and await syntax
Changes by Stefan Behnel sco...@users.sourceforge.net: -- nosy: +scoder type: - enhancement ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue24017 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24018] add a Generator ABC
Stefan Behnel added the comment: Ok, sure. Here's a new patch that adds tests and docs. -- Added file: http://bugs.python.org/file39150/generator_abc.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue24018 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24018] add a Generator ABC
Changes by Stefan Behnel sco...@users.sourceforge.net: Removed file: http://bugs.python.org/file39146/generator_abc.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue24018 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
