[Python-ideas] Re: Less is more? Smaller code and data to fit more into the CPU cache?
Barry Scott schrieb am 27.03.22 um 22:23: On 22 Mar 2022, at 15:57, Jonathan Fine wrote: As you may have seen, AMD has recently announced CPUs that have much larger L3 caches. Does anyone know of any work that's been done to research or make critical Python code and data smaller so that more of it fits in the CPU cache? I'm particularly interested in measured benefits. I few years ago (5? 10?) there was a blog about making the python eval loop fit into L1 cache. The author gave up on the work as he claimed it was too hard to contribute any changes to python at the time. I have not kept a link to the blog post sadly. What I recall is that the author found that GCC was producing far more code then was required to implement sections of ceval.c. Fixing that would shrink the ceval code by 50% I recall was the claim. He had a PoC that showed the improvements. Might be worth trying out if "gcc -Os" changes anything for ceval.c. Can also be enabled temporarily with a pragma (and MSVC has a similar option). We use it in Cython for the (run once) module init code to reduce the binary module size, but it might have an impact on cache usage as well. Stefan ___ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/QQVYUUKOKN472N4OLNCAA76HLVFXMKLB/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] Re: Fractions vs. floats - let's have the cake and eat it
Martin Teichmann schrieb am 18.05.21 um 23:41: > >>> solve(x**2-2/3) > [-0.816496580927726, 0.816496580927726] > > That went wrong. We wanted to have a symbolic result, not a numeric solution! > How did that come about? Well, that's the point of discussion: 2/3 gets > immediately turned into 0.6..., and sympy has no clue where this is > coming from. So, do I understand correctly that you want a way to disable the compile-time optimiser in Python that does the constant folding? Stefan ___ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/DPHABHUG337IOIWJ3PU6XK5K7X4FAIUV/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] Re: PEP 637 - support for indexing with keyword arguments
Stefan Behnel schrieb am 29.09.20 um 11:48: > Salve Stefano, > > Stefano Borini schrieb am 23.09.20 um 22:55: >> "Support for indexing with keyword arguments" has now been merged with >> the assigned PEP number 637. > > Cool, this looks like a great addition to the language! > > One thing that I'm missing from the PEP is the C side of things, though. > How are C extension types going to implement this? > > Will there be two new slot methods for them? Something like (but better > named than) "mp_subscript_kw()" and "mp_ass_subscript_kw()"? > > Would the extension types signal their availability with a new "TP_*" class > feature flag? > > Are the slot methods going to use the vectorcall calling convention, i.e. > pass the keyword names as a tuple, or should they accept (and thus, require > the overhead of) a Python dict as argument? > > This design must clearly be part of the PEP. What's the current status of > the discussion there? So, should I conclude from the silence that there has not been any discussion of this so far? Stefan ___ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/3RCUDRAM7Y22I6JUWCLD7MIIB7K7IU4Z/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] Re: PEP 637 - support for indexing with keyword arguments
Salve Stefano, Stefano Borini schrieb am 23.09.20 um 22:55: > "Support for indexing with keyword arguments" has now been merged with > the assigned PEP number 637. Cool, this looks like a great addition to the language! One thing that I'm missing from the PEP is the C side of things, though. How are C extension types going to implement this? Will there be two new slot methods for them? Something like (but better named than) "mp_subscript_kw()" and "mp_ass_subscript_kw()"? Would the extension types signal their availability with a new "TP_*" class feature flag? Are the slot methods going to use the vectorcall calling convention, i.e. pass the keyword names as a tuple, or should they accept (and thus, require the overhead of) a Python dict as argument? This design must clearly be part of the PEP. What's the current status of the discussion there? Stefan ___ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/GIHORXWACHNHF3G4CQBAW3DEO3EMY7SK/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] Re: Accelerate zlib and gzip libraries with Intel's Storage Acceleration Libraries
Steven D'Aprano schrieb am 17.08.20 um 17:00: > On Mon, Aug 17, 2020 at 04:08:54PM +0200, Stefan Behnel wrote: > >> I re-opened the ticket to allow for some discussion over there in order to >> understand the implications better. But I agree that a third-party package >> on PyPI seems like a good first step, also as a backport. > > Perhaps I have misunderstood, but isn't this a pure implementation > change, with no user visible API changes and backward compatible output? > > So why does it need to go on PyPI first? It isn't as if we need to wait > to see whether there is demand for faster reading and writing of gzip > files, or for the API to settle down. I didn't say that it won't be accepted into CPython. That depends a lot on how easy it is to integrate and what implications that would have. That will be decided as part of the discussion in the ticket. However, even if it gets accepted, then that would be a change for CPython 3.10 at the earliest, maybe later, depending on when the patch is ready. Older CPython versions could still benefit from the faster (de-)compression by having a third-party module in PyPI. Basically, wrapping a zlib compatible library, e.g. in Cython or even as a copy of CPython's own zlib code, seems rather straight forward to me. I'm more worried about the build time dependencies and setup that arise here, which would need tackling regardless of where/how we integrate it. Having a third-party module available would show how easy or difficult it is to build such a module, thus giving an indication for the effort required to integrate and ship it with the CPython code base. It's a good first step with a readily usable outcome, whatever the decision in CPython will be. Stefan ___ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/IYWBCNV3BTC5ZUH4YKFF7ZURFXAOJEVG/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] Re: Accelerate zlib and gzip libraries with Intel's Storage Acceleration Libraries
Antoine Pitrou schrieb am 17.08.20 um 15:00: > On Mon, 17 Aug 2020 08:49:23 - > "Ruben Vorderman" wrote: >> Dear python developers, >> >> As a bioinformatician I work a lot with gzip-compressed data. Recently I >> discovered Intel's Storage Acceleration Libraries at >> https://github.com/intel/isa-l. These people implemented the DEFLATE and >> INFLATE algorithms in assembly language. As a result it is much faster than >> zlib. >> >> I have posted a few benchmarks in this python bug >> https://bugs.python.org/issue41566. (I just discovered bugs.python.org is >> the wrong place for feature requests. I am sorry, I am still learning about >> the proper way of doing this things, as this is my first feature proposal). >> The TLDR is that it can speed up compression by 5x while speeding up >> compression by 3x compared to standard gzip. >> >> Isa-l is bsd-3-clause licensed and as such I see no licensing issues when >> using it in CPython. > > In any case, it should be simple enough to post a package on PyPI that > exposes the desired wrapper APIs. I re-opened the ticket to allow for some discussion over there in order to understand the implications better. But I agree that a third-party package on PyPI seems like a good first step, also as a backport. Stefan ___ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/6SOR62YK7A32L62V26VVEE7EZSIM5E6U/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] Re: Tracebacks for C code in Python
Ram Rachum schrieb am 15.08.20 um 21:08: > Here's something that's been bugging me for years. I'll suggest something, > but since I'm a total newbie about this area, it's possible that everything > I'm saying is impossible or doesn't make sense. > > I'm working with some Pandas code now, and there's an exception because I'm > doing something wrong. I get a traceback, but some of the frames are in pyd > files (C code I guess?) so I don't see the code for them. > > This is frustrating, because the exception message isn't that clear, so I > would at least like to know what the code was trying to do when it got the > exception. Maybe this will give me more hints about what's going wrong. > > *Would it be possible to have Python tracebacks include code for C code > that's called in Python?* > > I know very little about how the C-to-Python interaction works, and I > assume we'd need something complicated like packaging the source code with > the binaries in some way that lets Python get the right line of C code to > put in the traceback. This can get complicated. > > Do you think it's possible? Pandas is actually not implemented in C (or just a bit of that) but in Cython. That is why you get tracebacks that include line numbers from extension modules at all. C implemented extensions do not normally provide this. The reason why the source code lines are not displayed is probably just that Pandas does not ship its source code but only the compiled modules. Remember that the reason why you get Python source code lines in tracebacks is that the Python code file is sitting right there in the installed package. If Pandas did the same thing, you'd probably also get code lines printed in tracebacks. Stefan ___ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/P7ONR64HMS27XABUH4BBXPISGJ2WNLUR/ Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] dict.merge(d1, d2, ...) (Counter proposal for PEP 584)
Steven D'Aprano schrieb am 21.03.19 um 17:21: > On Thu, Mar 21, 2019 at 09:11:18AM -0700, Guido van Rossum wrote: > >> I don't find it easy to understand or remember that d1.update(d2) modifies >> d1 in place, while d1.merge(d2) first copies d1. >> >> Maybe the name can indicate the copying stronger? Like we did with sorting: >> l.sort() sorts in-place, while sorted(l) returns a sorted copy. > > How about dict.merged(*args, **kw)? Or dict.updated()? And then users would accidentally type d.updated(items) and lack the tests to detect that this didn't do anything (except wasting some time and memory). Stefan ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] PEP: Dict addition and subtraction
INADA Naoki schrieb am 05.03.19 um 08:03:> On Tue, Mar 5, 2019 at 12:02 AM
Stefan Behnel wrote:
>> INADA Naoki schrieb am 04.03.19 um 11:15:
>>> Why statement is not enough?
>>
>> I'm not sure I understand why you're asking this, but a statement is
>> "not enough" because it's a statement and not an expression. It does
>> not replace the convenience of an expression.
>
> It seems tautology and say nothing.
That's close to what I thought when I read your question. :)
> What is "convenience of an expression"?
It's the convenience of being able to write an expression that generates
the thing you need, rather than having to split code into statements that
create it step by step before you can use it.
Think of comprehensions versus for-loops. Comprehensions are expressions
that don't add anything to the language that a for-loop cannot achieve.
Still, everyone uses them because they are extremely convenient.
> Is it needed to make Python more readable language?
No, just like comprehensions, it's not "needed". It's just convenient.
> Anyway, If "there is expression" is the main reason for this proposal,
> symbolic operator is not necessary.
As said, "needed" is not the right word. Being able to use a decorator
closes a gap in the language. Just like list comprehensions fit generator
expressions and vice versa. There is no "need" for being able to write
[x**2 for x in seq]
{x**2 for x in seq}
when you can equally well write
list(x**2 for x in seq)
set(x**2 for x in seq)
But I certainly wouldn't complain about that redundancy in the language.
> `new = d1.updated(d2)` or `new = dict.merge(d1, d2)` are enough. Python
> preferred name over symbol in general. Symbols are readable and
> understandable only when it has good math metaphor.
>
> Sets has symbol operator because it is well known in set in math, not
> because set is frequently used.
>
> In case of dict, there is no simple metaphor in math.
So then, if "list+list" and "tuple+tuple" wasn't available through an
operator, would you also reject the idea of adding it, argueing that we
could use this:
L = L1.extended(L2)
I honestly do not see the math relation in concatenation via "+".
But, given that "+" and "|" already have the meaning of "merging two
containers into one" in Python, I think it makes sense to allow that also
for dicts.
> It just cryptic and hard to Google.
I honestly doubt that it's something people would have to search for any
more than they have to search for the list "+" operation. My guess is that
it's pretty much what most people would try first when they have the need
to merge two dicts, and only failing that, they would start a web search.
In comparison, very few users would be able to come up with "{**d1, **d2}"
on their own, or even "d1.updated(d2)".
My point is, given the current language, "dict+dict" is a gap that is worth
closing.
Stefan
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] PEP: Dict addition and subtraction
INADA Naoki schrieb am 04.03.19 um 11:15: > Why statement is not enough? I'm not sure I understand why you're asking this, but a statement is "not enough" because it's a statement and not an expression. It does not replace the convenience of an expression. Stefan ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] PEP: Dict addition and subtraction
Jimmy Girardet schrieb am 04.03.19 um 10:12:
> I'm not old on this list but every time there is a proposal, the answer
> is "what are you trying to solve ?".
>
> Since
>
> |z ={**x,**y} and z.update(y) Exists, I can"t find the answer.
I think the main intentions is to close a gap in the language.
[1,2,3] + [4,5,6]
works for lists and tuples,
{1,2,3} | {4,5,6}
works for sets, but joining two dicts isn't simply
{1:2, 3:4} + {5:6}
but requires either some obscure syntax or a statement instead of a simple
expression.
The proposal is to enable the obvious syntax for something that should be
obvious.
Stefan
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] PEP: Dict addition and subtraction
James Lu schrieb am 04.03.19 um 03:28: > I propose that the + sign merge two python dictionaries such that if there > are conflicting keys, a KeyError is thrown. Please, no. That would be really annoying. If you need that feature, it can become a new method on dicts. Stefan ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Dict joining using + and +=
Rémi Lapeyre schrieb am 01.03.19 um 16:44: > Le 1 mars 2019 à 16:04:47, Stefan Behnel a écrit: >> I think the behaviour makes sense when you know how it's implemented (keys >> are stored separately from values). > > Is a Python user expected to know the implementation details of all mappings > thought? No, it just helps _me_ in explaining the behaviour to myself. Feel free to look it up in the documentation if you prefer. >> I would have been less surprised if the >> keys had also been reordered, but well, this is how it is now in Py3.6+, so >> this is how it's going to work also for the operator. >> >> No *additional* surprises here. ;) > > There is never any surprises left once all details have been carefully worked > out but having `+` for mappings make it looks like an easy operation whose > meaning is non ambiguous and obvious. > > I’m still not convinced that it the meaning is obvious, and gave an example > in my other message where I think it could be ambiguous. What I meant was that it's obvious in the sense that it is no new behaviour at all. It just provides an operator for behaviour that is already there. We are not discussing the current behaviour here. That ship has long sailed with the release of Python 3.6 beta 1 back in September 2016. The proposal that is being discussed here is the new operator. Stefan ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Dict joining using + and +=
Eric V. Smith schrieb am 01.03.19 um 15:49:
> I understand Inada to be saying that each value on the LHS (as shown above)
> affects the result on the RHS. That's the case with addition of ints and
> other types, but not so with the proposed dict addition. As he says, the
> {a:1} doesn't affect the result. The result would be the same if this key
> wasn't present in the first dict, or if the key had a different value.
>
> This doesn't bother me, personally.
+1
Stefan
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Dict joining using + and +=
Rémi Lapeyre schrieb am 01.03.19 um 15:50:
> Le 1 mars 2019 à 15:41:52, Stefan Behnel a écrit:
>
>> Rémi Lapeyre schrieb am 01.03.19 um 15:06:
>>> I’m having issues to understand the semantics of d1 + d2.
>>>
>>> I think mappings are more complicated than sequences it some things
>>> seems not obvious to me.
>>>
>>> What would be OrderedDict1 + OrderedDict2, in which positions would be
>>> the resulting keys, which value would be used if the same key is
>>> present in both?
>>
>> The only reasonable answer I can come up with is:
>>
>> 1) unique keys from OrderedDict1 are in the same order as before
>> 2) duplicate keys and new keys from OrderedDict2 come after the keys from
>> d1, in their original order in d2 since they replace keys in d1.
>>
>> Basically, the expression says: "take a copy of d1 and add the items from
>> d2 to it". That's exactly what you should get, whether the mappings are
>> ordered or not (and dict are ordered by insertion in Py3.6+).
>
> Thanks Stefan for your feedback, unless I’m mistaken this does not work like
> Rhodri suggested, he said:
>
> I can tell it's pretty straightforward:
>
> d = d1 + d2 is equivalent to:
>
> >>> d = d1.copy()
> >>> d.update(d2)
>
> But doing this:
>
> >>> d1 = OrderedDict({"a": 1, "b": 2, "c": 3})
> >>> d2 = OrderedDict({"d": 4, "b": 5})
> >>> d = d1.copy()
> >>> d.update(d2)
> >>> d
> OrderedDict([('a', 1), ('b', 5), ('c', 3), ('d', 4)])
>
> It looks like that the semantics are either not straightforward or what you
> proposed is not the only reasonable answer. Am I missing something?
No, I was, apparently. In Py3.7:
>>> d1 = {"a": 1, "b": 2, "c": 3}
>>> d1
{'a': 1, 'b': 2, 'c': 3}
>>> d2 = {"d": 4, "b": 5}
>>> d = d1.copy()
>>> d.update(d2)
>>> d
{'a': 1, 'b': 5, 'c': 3, 'd': 4}
I think the behaviour makes sense when you know how it's implemented (keys
are stored separately from values). I would have been less surprised if the
keys had also been reordered, but well, this is how it is now in Py3.6+, so
this is how it's going to work also for the operator.
No *additional* surprises here. ;)
Stefan
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Dict joining using + and +=
Rémi Lapeyre schrieb am 01.03.19 um 15:06: > I’m having issues to understand the semantics of d1 + d2. > > I think mappings are more complicated than sequences it some things > seems not obvious to me. > > What would be OrderedDict1 + OrderedDict2, in which positions would be > the resulting keys, which value would be used if the same key is > present in both? The only reasonable answer I can come up with is: 1) unique keys from OrderedDict1 are in the same order as before 2) duplicate keys and new keys from OrderedDict2 come after the keys from d1, in their original order in d2 since they replace keys in d1. Basically, the expression says: "take a copy of d1 and add the items from d2 to it". That's exactly what you should get, whether the mappings are ordered or not (and dict are ordered by insertion in Py3.6+). > What would be defaultdict1 + defaultdict2? No surprises here, the result is a copy of defaultdict1 (using the same missing-key function) with all items from defaultdict2 added. Remember that the order of the two operands matters. The first always defines the type of the result, the second is only added to it. > It seems to me that subclasses of dict are complex mappings for which > « merging » may be less obvious than for sequences. It's the same for subclasses of sequences. Stefan ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Mention more alternative implementations on the PSF website
Eric V. Smith schrieb am 03.02.19 um 23:41: >> On Feb 3, 2019, at 5:31 PM, James Lu wrote: >> https://www.python.org/download/alternatives/ should possibly mention: >> - Mention the possibility of compiling Python to WASM >>- WASM allows Web and Mobile use of Python at possibly native speed. >> Though not mature yet, Pyodide is a working implementation. > > I suggest opening an issue at https://github.com/python/pythondotorg/issues > > Although personally I think it’s too early to mention WASM. I agree about the maturity problem, but you could argue that Pyodide is essentially a "repackaging of CPython for WASM", which would belong into the second list on the page, once it reaches a level of compatibility and reliability that makes it relevant for users. Stefan ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Clearer communication
James Lu schrieb am 01.02.19 um 17:40: > A lot of the traffic on this email list is people saying “I don’t understand” > or “that’s not what I meant” or trying to re-explain. A lot of “-1”s are > really “I don’t see the usefulness of this”. > > So I want an open discussion on: How can we communicate clearer? While I agree with others that this is a good question worth asking, I also think that it's somewhat the nature of this list that a) the topics discussed here often require a good understanding of Python and the workings of the language, b) the described use cases and ideas are often novel and therefore may not immediately ring bells in everyone's ears, and c) many topics are recurring, so people who give a quick "-1" may just be reluctant to discuss them all over again without expecting new facets to appear (which rarely happens, in fact, although there are famously PEP-ed exceptions). And, last but not least, "I don't see the usefulness of this" is a perfectly valid counter-argument, especially for a well designed language like Python. I can think of at least one programming language that was not so lucky to have enough people say this. Stefan ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] No need to add a regex pattern literal
Ma Lin schrieb am 31.12.18 um 14:02:
> On 18-12-31 19:47, Antoine Pitrou wrote:
>> The complaint is that the global cache is still too costly.
>> See measurements in https://bugs.python.org/issue35559
>
> In this issue, using a global variable `_has_non_base16_digits` [1] will
> accelerate 30%.
> Is re module's internal cache [2] so bad?
>
> If rewrite re module's cache with C and use a custom data structure, maybe
> we will get a small speedup.
>
> [1] `_has_non_base16_digits` in PR11287
> [1] https://github.com/python/cpython/pull/11287/files
>
> [2] re module's internal cache code:
> [2] https://github.com/python/cpython/blob/master/Lib/re.py#L268-L295
>
> _cache = {} # ordered!
> _MAXCACHE = 512
> def _compile(pattern, flags):
> # internal: compile pattern
> if isinstance(flags, RegexFlag):
> flags = flags.value
> try:
> return _cache[type(pattern), pattern, flags]
> except KeyError:
> pass
> ...
I wouldn't be surprised if the slowest part here was the isinstance()
check. Maybe the RegexFlag class could implement "__hash__()" as "return
hash(self.value)" ?
Stefan
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Add regex pattern literal p""
Ma Lin schrieb am 27.12.18 um 14:15:
>> It'd be good to know just how much benefit this precompilation actually
> grants.
>
> As far as I know, Pattern objects in regex module can be pickled, don't
> know if it's useful.
>
import pickle
import regex
That's from the external regex package, not the stdlib re module.
p = regex.compile('[a-z]')
b = pickle.dumps(p)
p = pickle.loads(b)
Look a little closer:
>>> import pickle, re
>>> p = re.compile("[abc]")
>>> pickle.dumps(p)
b'\x80\x03cre\n_compile\nq\x00X\x05\x00\x00\x00[abc]q\x01K \x86q\x02Rq\x03.'
What this does, essentially, is to make the pickle loader pass the original
regex pattern string into re.compile() to "unpickle" it. Meaning, it
compiles the regex on the way in. Thus, there isn't much to gain from using
(the current form of) regex pickling here.
I'm not saying that this can't be changed, but personally, this is exactly
what I would do if I was asked to make a compiled regex picklable.
Everything else would probably get you into portability hell.
Stefan
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Why shouldn't Python be better at implementing Domain Specific Languages?
Matthew Einhorn schrieb am 31.08.2018 um 20:57: > with model: > with Dense(): > units = 64 > activation = 'relu' > input_dim = 100 > > with Dense(): > units = 10 > activation = 'softmax' This looks like it could use 'class' instead of 'with'. Stefan ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Does jargon make learning more difficult?
Michael Selik schrieb am 14.08.2018 um 21:42: > This is supported by education research. Some light Googling found a > study on the topic [0] that is consistent with my own observations. OTx2, and no offence, but … this is supported by research as well. People tend to search just long enough to find the link that backs their … observations. :) Stefan ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Syntactic sugar to declare partial functions
Michel Desmoulin schrieb am 09.08.2018 um 18:59: > I'd rather have functools.partial() to be added as a new method on > function objects. > >> from functools import partial >> >> def add(x:int,y:int)->int: >> returnx +y >> >> add_2 = partial(add,2) > > Would become: > > add_2 = add.partial(2) Except that this only works for functions, not for other callables. Meaning, code that uses this for anything but its self-defined functions will break as soon as someone passes in a callable object that is not a function. Stefan ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] A better (simpler) approach to PEP 505
David Mertz schrieb am 23.07.2018 um 16:12: > The need addressed by PEP 505 is real; it's also MUCH more niche and > uncommon than something that would merit new syntax. Moreover, the actual > legitimate purpose served by the PEP 505 syntax is easily served by > existing Python simply by using a wrapper class. The discussion so far made it clear to me that a) there is a use case for this feature, although I never needed it myself b) throwing new syntax at it is not the right solution Especially since there seem to be slightly diverging ideas about the exact details in behaviour. Since this can be done in form of a library, people should just drop it into a couple of competing libraries and let users choose what they like better in their specific situation. And since we already have a PEP now, let's continue to use it as a basis for discussion about how these libraries should best behave in general and what mistakes they should avoid. Stefan ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] slice[] to get more complex slices
Stephan Hoyer schrieb am 23.07.2018 um 18:01: > On Mon, Jul 23, 2018 at 4:24 AM Paul Moore wrote: > >> I thought the reason the proposal got nowhere was because it's pretty >> simple to define it yourself: >> >> >>> class SliceHelper: >> ... def __getitem__(self, slice): >> ... return slice >> ... >> >>> SH = SliceHelper() >> >>> SH[1::3] >> slice(1, None, 3) >> >> Did I miss something significant about why this wasn't sufficient? > > > I think a SliceHelper class like this is a reasonable solution, but there > would be a lot of value having it a standard place somewhere in the > standard library (e.g., operator.subscript). > > Both pandas and NumPy include this helper object under different names > (pandas.IndexSlice and numpy.index_exp / numpy.s_), but it would be > surprising/unexpected for pandas/numpy specific helpers to show up when not > using one of those libraries. I do the exact same sorts of indexing > manipulations with xarray, dask and TensorFlow. > > Given that this is basically a simple feature to make it easier to work > with Python syntax (so there's no danger it will change in the future), I > think there is a lot to be said for putting it in the standard library in > one place so it's obvious what to use and users don't have to relearn that > name for this object and/or reimplement it. Please copy that comment into the ticket and ask for it to be reopened. https://bugs.python.org/issue24379 Stefan ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] slice[] to get more complex slices
Todd schrieb am 22.07.2018 um 21:03: > For basic slices, the normal "slice(start, stop, step)" syntax works well. > But it becomes much more verbose to create more complicated slices that you > want to re-use for multiple multidimensional data structures, like numpy, > pandas, xarray, etc. > > One idea I had was to allow creating slices by using indexing on the slice > class. So for example: > > x = slice[5:1:-1, 10:20:2, 5:end] > > Would be equivalent to: > > x = (slice(5, 1, -1), slice(10, 20, 2), slice(5, None)) > > Note that this wouldn't be done on a slice instance, it would be done on > the slice class. The basic idea is that it would simply return whatever is > given to __getitem__. Personally, I always likes that idea, but it would be worth looking up the previous discussions in the list archive to find out if they lead to any conclusion. AFAICT, "slice.literal" was the latest such proposal that was discussed. Stefan ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Performance improvements via static typing
Michael Hall schrieb am 19.07.2018 um 15:51:
> While I am aware of projects like Cython and mypy, it seems to make sense
> for CPython to allow optional enforcement of type hints, with compiler
> optimizations related to it to be used. While this would not receive the
> same level of performance benefits as using ctypes directly, there do
> appear to be various gains available here.
Well, first of all, a C level type check at runtime is quite fast compared
to a byte code dispatch with stack argument handling, and can be done
regardless of any type hints. There are various code patterns that would
suggest a certain type ("x.append()" probably appends to a list, "x.get()"
will likely be a dict lookup, etc.), and that can be optimised for without
static type declarations and even without any type hints.
Then, the mere fact that user code says "a: List" does not help in any way.
Even "a: list" would not help, because any behaviour of that "list" might
have been overridden by the list subtype that the function eventually receives.
The same applies to "x: float". Here, in order to gain speed, the compiler
would have to generate two separate code paths through the entire function,
one for C double computations, and one for Python float subtypes. And then
exponentiate that by the number of other typed arguments that may or may
not contain subtypes. Quite some overhead.
It's unclear if the gain would have any reasonable relation to the effort
in the end. Sure, type hints could be used as a bare trigger for certain
optimistic optimisations, but then, it's cheap enough to always apply these
optimisations regardless, via a C type check.
Stefan
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] OT: slowing rotation [was: High Precision datetime]
Greg Ewing schrieb am 18.05.2018 um 10:05: > Ethan Furman wrote: >> How long before the earth stops rotating? > > Apparently about 1.9 trillion years. So, does that mean we now need to hold our breath for 1.9 british trillion years or 1.9 american trillion years? Assuming you were referring to the French-Latin-Arabic based numbers and naming systems at all, that is... And anyway, what's that point doing there, right between the "1" and the "9" ? Stefan ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] PEP proposal: unifying function/method classes
Jeroen Demeyer schrieb am 24.03.2018 um 22:38: > On 2018-03-24 09:09, Nick Coghlan wrote: >> If it was just about introspection, then we could define a new protocol >> method or attribute, update the inspect module to respect it, and call >> it done. > > True. In fact, I got a long way by just defining inspect.isfunction as > > def isfunction(obj): > return hasattr(type(obj), "__code__") This relies on the fact that Cython attaches an (empty) code object to its C implemented functions, which could be considered a hack but is needed for things like source file reporting, and also line reporting (although we don't currently use that, but an even worse hack). The main problem, though, is that a lot of code out there does not use the "inspect" module but "types" to distinguish its input. That probably got a bit better over time with the ABCs, but for things like functions, generators and coroutines, I doubt that people would easily consider them. "types" just seems too obvious at first thought. Which suggests that it could help to add a visible note to the CPython docs of the "types" module that it's often better to use "inspect" or the ABCs. Seems worth a PR. Stefan ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Why CPython is still behind in performance for some widely used patterns ?
Nick Coghlan schrieb am 02.02.2018 um 06:47: > to make the various extension module authoring tools > easier to discover, rather than having folks assuming that handcrafted > calls directly into the CPython C API is their only option. Or even a competitive option. Tools like Cython or pybind11 go to great length to shave off every bit of overhead from C-API calls, commonly replacing high-level C-API functionality with macros and direct access to data structures. The C/C++ code that they generate is so complex and tuned that it would be infeasible to write and maintain something like that by hand, but it can perfectly be generated, and it usually performs visibly better than most hand-written modules, definitely much better than anything a non-expert could write. Basically, by not learning the C-API you can benefit from all that highly tuned and specialised code written by C-API experts that the documentation doesn't even tell you about. Stefan ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Why not picoseconds?
Antoine Pitrou schrieb am 16.10.2017 um 10:20: > On Sun, 15 Oct 2017 22:00:10 -0700 > Guido van Rossum wrote: >> On Sun, Oct 15, 2017 at 8:40 PM, Nick Coghlan wrote: >> >>> Hopefully by the time we decide it's worth worrying about picoseconds in >>> "regular" code, compiler support for decimal128 will be sufficiently >>> ubiquitous that we'll be able to rely on that as our 3rd generation time >>> representation (where the first gen is seconds as a 64 bit binary float and >>> the second gen is nanoseconds as a 64 bit integer). >> >> I hope we'll never see time_ns() and friends as the second generation -- >> it's a hack that hopefully we can retire in those glorious days of hardware >> decimal128 support. > > Given the implementation costs, hardware decimal128 will only become > mainstream if there's a strong incentive for it, which I'm not sure > exists or will ever exist ;-) Then we shouldn't implement the new nanosecond API at all, in order to keep pressure on the hardware developers. Stefan :o) ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Add time.time_ns(): system clock with nanosecond resolution
Nick Coghlan schrieb am 14.10.2017 um 17:46: > On 14 October 2017 at 18:21, Antoine Pitrou wrote: >> On Sat, 14 Oct 2017 10:49:11 +0300 >> Serhiy Storchaka wrote: >>> I don't like the idea of adding a parallel set of functions. >>> >>> In the list of alternatives in PEP 410 there is no an idea about fixed >>> precision float type with nanoseconds precision. It can be implemented >>> internally as a 64-bit integer, but provide all methods required for >>> float-compatible number. It would be simpler and faster than general >>> Decimal. >> >> I agree a parallel set of functions is not ideal, but I think a parallel >> set of functions is still more appropriate than a new number type >> specific to the time module. >> >> Also, if you change existing functions to return a new type, you risk >> breaking compatibility even if you are very careful about designing the >> new type. >> > > Might it make more sense to have a parallel *module* that works with a > different base data type rather than parallel functions within the existing > API? > > That is, if folks wanted to switch to 64-bit nanosecond time, they would > use: > > * time_ns.time() > * time_ns.monotonic() > * time_ns.perf_counter() > * time_ns.clock_gettime() > * time_ns.clock_settime() > > The idea here would be akin to the fact we have both math and cmath as > modules, where the common APIs conceptually implement the same algorithms, > they just work with a different numeric type (floats vs complex numbers). I thought of that, too. People are used to rename things on import, so this would provide a very easy way for them to switch. OTOH, I would guess that "from time import time" covers more than 90% of the use cases of the time module and it doesn't really matter if we make people change the first or the second part of that import statement... But the real point here is that the data type which the current time module deals with is really (semantically) different from what is proposed now. All functionality in the time module assumes to work with "seconds", and accepts fractions of seconds for better precision. But the common semantic ground is "seconds". That suggests to me that "nanoseconds" really fits best into a new module which clearly separates the semantics of the two data types. (That being said, I'm a big fan of fractions, so I wonder if a Fraction with a constant nano denominator wouldn't fit in here...) Stefan ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Add time.time_ns(): system clock with nanosecond resolution
Victor Stinner schrieb am 13.10.2017 um 16:12: > I would like to add new functions to return time as a number of > nanosecond (Python int), especially time.time_ns(). I might have missed it while skipping through your post, but could you quickly explain why improving the precision of time.time() itself wouldn't help already? Would double FP precision not be accurate enough here? Stefan ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Make map() better
Jason H schrieb am 13.09.2017 um 17:54: > I'm rather surprised that there isn't a Iterable class which dict and list > derive from. > If that were added to just dict and list, I think it would cover 98% of > cases, and adding Iterable would be reasonable in the remaining scenarios. Would you then always have to inherit from that class in order to make a type iterable? That would be fairly annoying... The iterable and iterator protocols are extremely simple, and that's a great feature. And look, map() even works with all of them, without inheritance, registration, and whatnot. It's so easy! Stefan ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] PEP 563: Postponed Evaluation of Annotations, first draft
Lukasz Langa schrieb am 11.09.2017 um 21:25: > I remember mostly Stefan Behnel's concerns about Cython's annotations, I'm currently reimplementing the annotation typing in Cython to be compatible with PEP-484, so that concern is pretty much out of the way. This PEP still has an impact on Cython, because we'd have to implement the same thing, and also make the interface available in older Python versions (2.6+) for Cython compiled modules. Stefan ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] PEP 563: Postponed Evaluation of Annotations, first draft
Ryan Gonzalez schrieb am 11.09.2017 um 19:16: > One thing I want to point out: there are a lot of really useful Python > libraries that have come to rely on annotations being objects, ranging > from plac to fbuild to many others. I could understand something that > delays the evaluation of annotations until they are accessed, but this > seems really extreme. I guess there could be some helper that would allow you to say "here's an annotation or a function, here's the corresponding module globals(), please give me the annotation instances". But the "__annotations__" mapping could probably also be self-expanding on first request, if it remembers its globals(). Stefan ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] PEP 550 v2
Nathaniel Smith schrieb am 16.08.2017 um 09:18:
> On Tue, Aug 15, 2017 at 4:55 PM, Yury Selivanov wrote:
>> Here's the PEP 550 version 2.
> Awesome!
+1
>> Backwards Compatibility
>> ===
>>
>> This proposal preserves 100% backwards compatibility.
>
> While this is mostly true in the strict sense, in practice this PEP is
> useless if existing thread-local users like decimal and numpy can't
> migrate to it without breaking backcompat. So maybe this section
> should discuss that?
>
> (For example, one constraint on the design is that we can't provide
> only a pure push/pop API, even though that's what would be most
> convenient context managers like decimal.localcontext or
> numpy.errstate, because we also need to provide some backcompat story
> for legacy functions like decimal.setcontext and numpy.seterr.)
I agree with Nathaniel that many projects that can benefit from this
feature will need to keep supporting older Python versions as well. In the
case of Cython, that's Py2.6+. We already have the problem that the
asynchronous finalisation of async generators cannot be supported in older
Python versions ("old" as in Py3.5 and before), so we end up with a
language feature that people can use in Py2.6, but not completely/safely.
I can't say yet how difficult it will be to integrate the new
infrastructure that this PEP proposes into a backwards compatible code
base, but if there's something we can think of now in order to help
projects keep supporting older Python versions in the same code base, given
the constraints of their existing APIs and semantics - that would be great.
Stefan
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Generator syntax hooks?
Soni L. schrieb am 08.08.2017 um 01:56: > On 2017-08-07 08:35 PM, Steven D'Aprano wrote: >> Hi Soni, and welcome! >> >> On Mon, Aug 07, 2017 at 04:30:05PM -0300, Soni L. wrote: >> >>> What if, (x for x in integers if 1000 <= x < 100), was syntax sugar >>> for (x for x in range(1000, 100))? >> If you want the integers from 1000 to 100, use: >> >> range(1000, 100) >> >> Don't waste your time slowing down the code with an unnecessary and >> pointless wrapper that does nothing but pass every value on unchanged: >> >> (x for x in range(1000, 100)) # waste of time and effort > > Actually, those have different semantics! > x = range(1, 10) list(x) > [1, 2, 3, 4, 5, 6, 7, 8, 9] list(x) > [1, 2, 3, 4, 5, 6, 7, 8, 9] > x = (x for x in range(1, 10)) list(x) > [1, 2, 3, 4, 5, 6, 7, 8, 9] list(x) > [] In that case, use iter(range(1000, 100)). range() creates an iterable, which is iterable more than once. iter(range()) creates an iterator from that iterable, which has the semantics that you apparently wanted. Stefan ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] PEP: Hide implementation details in the C API
Victor Stinner schrieb am 11.07.2017 um 12:19: > Split the ``Include/`` directory of CPython: > > * ``python`` API: ``Include/Python.h`` remains the default C API > * ``core`` API: ``Include/core/Python.h`` is a new C API designed for > building Python > * ``stable`` API: ``Include/stable/Python.h`` is the stable ABI > [...] > Step 3: first pass of implementation detail removal > --- > > Modify the ``python`` API: > > * Add a new ``API`` subdirectory in the Python source code which will > "implement" the Python C API > * Replace macros with functions. The implementation of new functions > will be written in the ``API/`` directory. For example, Py_INCREF() > becomes the function ``void Py_INCREF(PyObject *op)`` and its > implementation will be written in the ``API`` directory. > * Slowly remove more and more implementation details from this API. >From a Cython perspective, it's (not great but) ok if these "implementation details" were moved somewhere else, but it would be a problem if they became entirely unavailable for external modules. Cython uses some of the internals for performance reasons, and we adapt it to changes of these internals whenever necessary. The question then arises if this proposal fulfills its intended purpose if Cython based tools like NumPy or lxml continued to use internal implementation details in their Cython generated C code. Specifically because that code is generated, I find it acceptable that it actively exploits non-portable details, because it already takes care of adapting to different Python platforms anyway. Cython has incorporated support for CPython, PyPy and Pyston that way, adding others is probably not difficult, and optimising for a specific one (usually CPython) is also easy. The general rule of thumb in Cython core development is that it's ok to exploit internals as long as there is a generic fallback through some C-API operations which can be used in other Python implementations. I'd be happy if that continued to be supported by CPython in the future. Exposing CPython internals is a good thing! :) Stefan ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Make partial a built-in
אלעזר schrieb am 19.09.2016 um 17:59: > If at all, it should be function.bind(). It was discussed and dropped; I > don't remember why, but one problem is that it looks like an in-place > modification. IMHO, the main drawback of that solution is that it only works for functions and not for any other callable in Python. It would introduce an inconsistency without solving the problem in general. Stefan ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
