Re: [Python-Dev] New calling convention to avoid temporarily tuples when calling functions
Oh, I found a nice pice of CPython history in Modules/_pickle.c. Extract of Python 3.3: - /* A temporary cleaner API for fast single argument function call. XXX: Does caching the argument tuple provides any real performance benefits? A quick benchmark, on a 2.0GHz Athlon64 3200+ running Linux 2.6.24 with glibc 2.7, tells me that it takes roughly 20,000,000 PyTuple_New(1) calls when the tuple is retrieved from the freelist (i.e, call PyTuple_New() then immediately DECREF it) and 1,200,000 calls when allocating brand new tuples (i.e, call PyTuple_New() and store the returned value in an array), to save one second (wall clock time). Either ways, the loading time a pickle stream large enough to generate this number of calls would be massively overwhelmed by other factors, like I/O throughput, the GC traversal and object allocation overhead. So, I really doubt these functions provide any real benefits. On the other hand, oprofile reports that pickle spends a lot of time in these functions. But, that is probably more related to the function call overhead, than the argument tuple allocation. XXX: And, what is the reference behavior of these? Steal, borrow? At first glance, it seems to steal the reference of 'arg' and borrow the reference of 'func'. */ static PyObject * _Pickler_FastCall(PicklerObject *self, PyObject *func, PyObject *arg) - Extract of Python 3.4 (same function): - /* Note: this function used to reuse the argument tuple. This used to give a slight performance boost with older pickle implementations where many unbuffered reads occurred (thus needing many function calls). However, this optimization was removed because it was too complicated to get right. It abused the C API for tuples to mutate them which led to subtle reference counting and concurrency bugs. Furthermore, the introduction of protocol 4 and the prefetching optimization via peek() significantly reduced the number of function calls we do. Thus, the benefits became marginal at best. */ - It recalls me the story of property_descr_get() optimizations :-) I hope that the new generic "fastcall" functions will provide a safe and reliable optimization for the pickle module, property_descr_get() and others optimized functions. Victor ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] New calling convention to avoid temporarily tuples when calling functions
2016-08-22 10:01 GMT+02:00 Victor Stinner : > The next step is to support keyword parameters. In fact, it's already > supported in all cases except of Python functions: > https://bugs.python.org/issue27809 Serhiy Storchaka proposed to use a single C array for positional and keyword arguments. Keyword arguments are passed as (key, value) pairs. I just added this function: PyAPI_FUNC(PyObject *) _PyObject_FastCallKeywords( PyObject *func, PyObject **stack, Py_ssize_t nargs, Py_ssize_t nkwargs); The function is not used yet. Serhiy proposed to enhance the functions to parse arguments to support this format to pass arguments which would allow to avoid the creation of a temporary dictionary in many cases. I proposed to use this format (instead of (PyObject **stack, Py_ssize_t nargs, PyObject *kwargs)) for a new METH_FASTCALL calling convention for C functions: https://bugs.python.org/issue27810 Victor ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 525
On 24.08.2016 21:05, Yury Selivanov wrote: Sorry for making you irritated. Please feel free to remind me about any concrete changes to the PEP that I promised to add on python-ideas. I'll go and re-read that thread right now anyways. No problem as it seems I wasn't the only one. So, it doesn't matter anymore. :) Best, Sven ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 525
On 24.08.2016 21:00, Yury Selivanov wrote: For an async generator there are two cases: either it tries to yield another value (the first time this happens you can throw an error back into it) or it tries to await -- in that case you can also throw an error back into it, and if the error comes out unhandled you can print the error (in both cases actually). It's probably to specify all this behavior using some kind of default finalizer (though you don't have to implement it that way). Hopefully there will be other discussion as well, otherwise I'll have to accept the PEP once this issue is cleared up. :-) Curious to hear your thoughts on two different approaches to finalization. At this point, I'm inclined to change the PEP to use the second approach. I think it gives much more power to event loops, and basically means that any kind of APIs to control AG (or to finalize the loop) is possible. I think your alternative approach is the better one. It feels more integrated even though it's harder for event loop implementors (which are rarer than normal event loop users). Also AG finalization is something that's not really custom to each AG but makes more sense at the event loop level, I think. Best, Sven ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 525
On 2016-08-24 3:01 PM, Sven R. Kunze wrote:
On 24.08.2016 18:35, Guido van Rossum wrote:
On Wed, Aug 24, 2016 at 8:17 AM, Yury Selivanov
mailto:yselivanov...@gmail.com>> wrote:
On 2016-08-23 10:38 PM, Rajiv Kumar wrote:
I was playing with your implementation to gain a better
understanding of the operation of asend() and friends. Since
I was explicitly trying to "manually" advance the generators,
I wasn't using asyncio or other event loop. This meant that
the first thing I ran into with my toy code was the
RuntimeError ("cannot iterate async generator without
finalizer set").
As you have argued elsewhere, in practice the finalizer is
likely to be set by the event loop. Since the authors of
event loops are likely to know that they should set the
finalizer, would it perhaps be acceptable to merely issue a
warning instead of an error if the finalizer is not set? That
way there isn't an extra hoop to jump through for simple
examples.
In my case, I just called
sys.set_asyncgen_finalizer(lambda g: 1)
to get around the error and continue playing :) (I realize
that's a bad thing to do but it didn't matter for the toy cases)
Yeah, maybe warning would be sufficient. I just find it's highly
unlikely that a lot of people would use async generators without
a loop/coroutine runner, as it's a very tedious process.
Heh, I had the same reaction as Rajiv. I think the tediousness is
actually a good argument that there's no reason to forbid this. I
don't even think a warning is needed. People who don't use a
coroutine runner are probably just playing around (maybe even in the
REPL) and they shouldn't get advice unasked.
I also was irritated as Yury said there were absolutely no changes
after python-ideas. He said he might consider a clearer warning for
those examples at the beginning of the PEP to make them work for the
reader.
Sorry for making you irritated. Please feel free to remind me about any
concrete changes to the PEP that I promised to add on python-ideas.
I'll go and re-read that thread right now anyways.
Yury
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 525
On 24.08.2016 18:35, Guido van Rossum wrote:
On Wed, Aug 24, 2016 at 8:17 AM, Yury Selivanov
mailto:yselivanov...@gmail.com>> wrote:
On 2016-08-23 10:38 PM, Rajiv Kumar wrote:
I was playing with your implementation to gain a better
understanding of the operation of asend() and friends. Since I
was explicitly trying to "manually" advance the generators, I
wasn't using asyncio or other event loop. This meant that the
first thing I ran into with my toy code was the RuntimeError
("cannot iterate async generator without finalizer set").
As you have argued elsewhere, in practice the finalizer is
likely to be set by the event loop. Since the authors of event
loops are likely to know that they should set the finalizer,
would it perhaps be acceptable to merely issue a warning
instead of an error if the finalizer is not set? That way
there isn't an extra hoop to jump through for simple examples.
In my case, I just called
sys.set_asyncgen_finalizer(lambda g: 1)
to get around the error and continue playing :) (I realize
that's a bad thing to do but it didn't matter for the toy cases)
Yeah, maybe warning would be sufficient. I just find it's highly
unlikely that a lot of people would use async generators without a
loop/coroutine runner, as it's a very tedious process.
Heh, I had the same reaction as Rajiv. I think the tediousness is
actually a good argument that there's no reason to forbid this. I
don't even think a warning is needed. People who don't use a coroutine
runner are probably just playing around (maybe even in the REPL) and
they shouldn't get advice unasked.
I also was irritated as Yury said there were absolutely no changes after
python-ideas. He said he might consider a clearer warning for those
examples at the beginning of the PEP to make them work for the reader.
Would it be possible to print a warning only when an async generator
is being finalized and doesn't run straight to the end without
suspending or yielding? For regular generators we have a similar
exception (although I don't recall whether we actually warn) -- if you
call close() and it tries to yield another value it is just GC'ed
without giving the frame more control. For an async generator there
are two cases: either it tries to yield another value (the first time
this happens you can throw an error back into it) or it tries to await
-- in that case you can also throw an error back into it, and if the
error comes out unhandled you can print the error (in both cases
actually).
It's probably to specify all this behavior using some kind of default
finalizer (though you don't have to implement it that way).
Does a default finalizer solve the "event loop does not know its AGs"
problem?
Hopefully there will be other discussion as well, otherwise I'll have
to accept the PEP once this issue is cleared up. :-)
--
--Guido van Rossum (python.org/~guido )
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/srkunze%40mail.de
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 525
On 2016-08-24 12:35 PM, Guido van Rossum wrote:
On Wed, Aug 24, 2016 at 8:17 AM, Yury Selivanov
mailto:yselivanov...@gmail.com>> wrote:
On 2016-08-23 10:38 PM, Rajiv Kumar wrote:
I was playing with your implementation to gain a better
understanding of the operation of asend() and friends. Since I
was explicitly trying to "manually" advance the generators, I
wasn't using asyncio or other event loop. This meant that the
first thing I ran into with my toy code was the RuntimeError
("cannot iterate async generator without finalizer set").
As you have argued elsewhere, in practice the finalizer is
likely to be set by the event loop. Since the authors of event
loops are likely to know that they should set the finalizer,
would it perhaps be acceptable to merely issue a warning
instead of an error if the finalizer is not set? That way
there isn't an extra hoop to jump through for simple examples.
In my case, I just called
sys.set_asyncgen_finalizer(lambda g: 1)
to get around the error and continue playing :) (I realize
that's a bad thing to do but it didn't matter for the toy cases)
Yeah, maybe warning would be sufficient. I just find it's highly
unlikely that a lot of people would use async generators without a
loop/coroutine runner, as it's a very tedious process.
Heh, I had the same reaction as Rajiv. I think the tediousness is
actually a good argument that there's no reason to forbid this. I
don't even think a warning is needed. People who don't use a coroutine
runner are probably just playing around (maybe even in the REPL) and
they shouldn't get advice unasked.
Good point.
Would it be possible to print a warning only when an async generator
is being finalized and doesn't run straight to the end without
suspending or yielding? For regular generators we have a similar
exception (although I don't recall whether we actually warn) -- if you
call close() and it tries to yield another value it is just GC'ed
without giving the frame more control.
Yes, we can implement the exact same semantics for AGs:
- A ResourceWarning will be issued if an AG is GCed and cannot be
synchronously closed (that will happen if no finalizer is set and there
are 'await' expressions in 'finally').
- A RuntimeError is issued when an AG is yielding (asynchronously) in
its 'finally' block.
I think both of those things are already there in the reference
implementation. So we can just lift the requirement for asynchronous
finalizer being set before you iterate an AG.
For an async generator there are two cases: either it tries to yield
another value (the first time this happens you can throw an error back
into it) or it tries to await -- in that case you can also throw an
error back into it, and if the error comes out unhandled you can print
the error (in both cases actually).
It's probably to specify all this behavior using some kind of default
finalizer (though you don't have to implement it that way).
Hopefully there will be other discussion as well, otherwise I'll have
to accept the PEP once this issue is cleared up. :-)
Curious to hear your thoughts on two different approaches to
finalization. At this point, I'm inclined to change the PEP to use the
second approach. I think it gives much more power to event loops, and
basically means that any kind of APIs to control AG (or to finalize the
loop) is possible.
Thank you,
Yury
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 525
On Wed, Aug 24, 2016 at 8:17 AM, Yury Selivanov
wrote:
> On 2016-08-23 10:38 PM, Rajiv Kumar wrote:
>
>> I was playing with your implementation to gain a better understanding of
>> the operation of asend() and friends. Since I was explicitly trying to
>> "manually" advance the generators, I wasn't using asyncio or other event
>> loop. This meant that the first thing I ran into with my toy code was the
>> RuntimeError ("cannot iterate async generator without finalizer set").
>>
>> As you have argued elsewhere, in practice the finalizer is likely to be
>> set by the event loop. Since the authors of event loops are likely to know
>> that they should set the finalizer, would it perhaps be acceptable to
>> merely issue a warning instead of an error if the finalizer is not set?
>> That way there isn't an extra hoop to jump through for simple examples.
>>
>> In my case, I just called
>> sys.set_asyncgen_finalizer(lambda g: 1)
>> to get around the error and continue playing :) (I realize that's a bad
>> thing to do but it didn't matter for the toy cases)
>>
>
> Yeah, maybe warning would be sufficient. I just find it's highly unlikely
> that a lot of people would use async generators without a loop/coroutine
> runner, as it's a very tedious process.
>
Heh, I had the same reaction as Rajiv. I think the tediousness is actually
a good argument that there's no reason to forbid this. I don't even think a
warning is needed. People who don't use a coroutine runner are probably
just playing around (maybe even in the REPL) and they shouldn't get advice
unasked.
Would it be possible to print a warning only when an async generator is
being finalized and doesn't run straight to the end without suspending or
yielding? For regular generators we have a similar exception (although I
don't recall whether we actually warn) -- if you call close() and it tries
to yield another value it is just GC'ed without giving the frame more
control. For an async generator there are two cases: either it tries to
yield another value (the first time this happens you can throw an error
back into it) or it tries to await -- in that case you can also throw an
error back into it, and if the error comes out unhandled you can print the
error (in both cases actually).
It's probably to specify all this behavior using some kind of default
finalizer (though you don't have to implement it that way).
Hopefully there will be other discussion as well, otherwise I'll have to
accept the PEP once this issue is cleared up. :-)
--
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] File system path encoding on Windows
On 23Aug2016 2150, Stephen J. Turnbull wrote: Steve Dower writes: > * Stephen sees "no reason not to change locale.getpreferredencoding()" > (default encoding for open()) at the same time with the same switches, > while I'm not quite as confident. Do users generally specify an encoding > these days? I know I always put utf-8 there. I was insufficiently specific. "No reason not to" depends on separate switches for file system encoding and preferred encoding. That makes things somewhat more complicated for implementation, and significantly so for users. Yes, it does, but it's about the only possible migration path. I know Nick and Victor like the idea of a -X flag (or a direct -utf8 flag), but I prefer more specific environment variables: - PYTHONWINDOWSLEGACYSTDIO (for the console changes) - PYTHONWINDOWSLEGACYPATHENCODING (assuming getfilesystemencoding() is utf8) - PYTHONWINDOWSLEGACYLOCALEENCODING (assuming getpreferredencoding() is utf8) I'm open to dropping "WINDOWS" from these if others don't think it's necessary. Swap "LEGACY" for "UNICODE" if we just offer the option to upgrade now rather than changing the default. (I could also see the possibility of PYTHONWINDOWSSTRICT* options to use the default encoding but raise if decoding bytes fails - mbcs:strict rather than mbcs:replace. For utf-8 mode I'd want to use surrogatepass throughout, so it will always raise on invalid encoding, but there will be far fewer occurrences than for mbcs.) I'll transform my earlier post into a PEP (or probably three PEPs - one for each change), since it seems like the paper trail is going to be more valuable than discussing it now. Without an actual build to test it's pretty hard to evaluate the impact. Cheers, Steve ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 525
Hi Rajiv,
On 2016-08-23 10:38 PM, Rajiv Kumar wrote:
Hi Yury,
I was playing with your implementation to gain a better understanding
of the operation of asend() and friends. Since I was explicitly trying
to "manually" advance the generators, I wasn't using asyncio or other
event loop. This meant that the first thing I ran into with my toy
code was the RuntimeError ("cannot iterate async generator without
finalizer set").
As you have argued elsewhere, in practice the finalizer is likely to
be set by the event loop. Since the authors of event loops are likely
to know that they should set the finalizer, would it perhaps be
acceptable to merely issue a warning instead of an error if the
finalizer is not set? That way there isn't an extra hoop to jump
through for simple examples.
In my case, I just called
sys.set_asyncgen_finalizer(lambda g: 1)
to get around the error and continue playing :) (I realize that's a
bad thing to do but it didn't matter for the toy cases)
Yeah, maybe warning would be sufficient. I just find it's highly
unlikely that a lot of people would use async generators without a
loop/coroutine runner, as it's a very tedious process.
Thank you,
Yury
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
