Re: [Python-Dev] Compile-time resolution of packages [Was: Another update for PEP 394...]

2019-02-26 Thread Stephen J. Turnbull
Barry Warsaw writes: >`/usr/bin/env python` is great for development and terrible for deployment. Developers of `six` and `2to3`, you mean? Steve ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev

Re: [Python-Dev] Possible performance regression

2019-02-26 Thread Stephen J. Turnbull
Raymond Hettinger writes: > We're trying to keep performant the ones that people actually use. > For the Mac, I think there are only four that matter: > > 1) The one we distribute on the python.org > website at > https://www.python.org/ftp/python/3.8.0/python-3.8.0a2-macosx10.9.pkg

Re: [Python-Dev] [bpo-35155] Requesting a review

2019-02-26 Thread Denton Liu
On Tue, Feb 12, 2019 at 02:14:55AM -0800, Denton Liu wrote: > Hello all, > > A couple months back, I reported bpo-35155[1] and I submitted a PR for > consideration[2]. After a couple of reviews, it seems like progress has > stalled. Would it be possible for someone to review this? > > Thanks, >

Re: [Python-Dev] datetime.timedelta total_microseconds

2019-02-26 Thread Fred Drake
On Tue, Feb 26, 2019 at 10:20 PM Terry Reedy wrote: > To me, total_x implies that there is a summation of multiple timedeltas, > and there is not. Do you believe this is a particularly dominant perception? I don't, but specific backgrounds probably play into this heavily. I'd expect to total a

Re: [Python-Dev] Register-based VM [Was: Possible performance regression]

2019-02-26 Thread Greg Ewing
Victor Stinner wrote: Using a different register may require an explicit "CLEAR_REG R1" (decref the reference to the builtin range function) which is less efficient. Maybe the source operand fields of the bytecodes could have a flag indicating whether to clear the register after use. -- Greg

Re: [Python-Dev] datetime.timedelta total_microseconds

2019-02-26 Thread Terry Reedy
On 2/26/2019 7:03 PM, Chris Barker via Python-Dev wrote: So: it would be good to provide a correct, simple,  intuitive, and discoverable way to do that. timedelta.total_seconds() To me, total_x implies that there is a summation of multiple timedeltas, and there is not. So not intuitive to

Re: [Python-Dev] Possible performance regression

2019-02-26 Thread Raymond Hettinger
On Feb 26, 2019, at 2:28 PM, Neil Schemenauer wrote: > > Are you compiling with --enable-optimizations (i.e. PGO)? In my > experience, that is needed to get meaningful results. I'm not and I would worry that PGO would give less stable comparisons because it is highly sensitive to changes its

Re: [Python-Dev] Register-based VM [Was: Possible performance regression]

2019-02-26 Thread Dino Viehland
On Tue, Feb 26, 2019 at 3:56 PM Neil Schemenauer wrote: > Right. I wonder though, could we avoid allocating the Python frame > object until we actually need it? Two situations when you need a > heap allocated frame come to mind immediately: generators that are > suspended and frames as part of

Re: [Python-Dev] Another update for PEP 394 -- The "python" Command on Unix-Like Systems

2019-02-26 Thread Chris Barker via Python-Dev
On Tue, Feb 26, 2019 at 3:25 PM Barry Warsaw wrote: > > Who gets to decide on PEP 394 changes? > > Honestly, I think it’s the active distro maintainers who need to make this > decision. They have the pulse of their own communities and users, and can > make the best decisions and compromises for

Re: [Python-Dev] datetime.timedelta total_microseconds

2019-02-26 Thread Chris Barker via Python-Dev
This thread petered out, seemingly with a consensus that we should update the docs -- is anyone doing that? But anyway, I'd like to offer a counterpoint: >From the OP, it is clear that: * Folks have a need for timedeltas to be converted to numbers values, with units other than seconds

Re: [Python-Dev] Register-based VM [Was: Possible performance regression]

2019-02-26 Thread Neil Schemenauer
On 2019-02-27, Greg Ewing wrote: > Joe Jevnik via Python-Dev wrote: > > If Python switched to a global stack and global registers we may be able > > to eliminate a lot of instructions that just shuffle data from the > > caller's stack to the callee's stack. > > That would make implementing

Re: [Python-Dev] OpenSSL 1.1.1 update for 3.7/3.8

2019-02-26 Thread Wes Turner
Thanks, as always On Tue, Feb 26, 2019 at 4:45 PM Christian Heimes wrote: > On 26/02/2019 21.31, Wes Turner wrote: > >> IMHO it's > > fine to ship the last 2.7 build with an OpenSSL version that was EOLed > > just 24h earlier. > > > > Is this a time / cost issue or a branch policy issue? > > >

Re: [Python-Dev] Compact ordered set

2019-02-26 Thread Barry Warsaw
On Feb 26, 2019, at 13:02, Raymond Hettinger wrote: > * I gave up on ordering right away. If we care about performance, keys can > be stored in the order added; but no effort should be expended to maintain > order if subsequent deletions occur. Likewise, to keep set-to-set operations >

Re: [Python-Dev] Compile-time resolution of packages [Was: Another update for PEP 394...]

2019-02-26 Thread Barry Warsaw
On Feb 26, 2019, at 14:04, Neil Schemenauer wrote: > > Interesting idea. I remember when I was helping develop Debian > packaging guides for Python software. I had to fight with people > to convince them that Debian packages should use > >#!/usr/bin/pythonX.Y > > rather than > >

Re: [Python-Dev] Possible performance regression

2019-02-26 Thread Victor Stinner
Le mer. 27 févr. 2019 à 00:17, Victor Stinner a écrit : > My sad story with code placement: > https://vstinner.github.io/analysis-python-performance-issue.html > > tl; dr Use PGO. Hum wait, this article isn't complete. You have to see the follow-up: https://bugs.python.org/issue28618#msg286662

Re: [Python-Dev] Another update for PEP 394 -- The "python" Command on Unix-Like Systems

2019-02-26 Thread Barry Warsaw
On Feb 26, 2019, at 13:34, Petr Viktorin wrote: > I have two very different questions in mind for moving this forward. > > Who gets to decide on PEP 394 changes? Honestly, I think it’s the active distro maintainers who need to make this decision. They have the pulse of their own communities

Re: [Python-Dev] Possible performance regression

2019-02-26 Thread Victor Stinner
Hi, PGO compilation is very slow. I tried very hard to avoid it. I started to annotate the C code with various GCC attributes like "inline", "always_inline", "hot", etc.. I also experimented likely/unlikely Linux macros which use __builtin_expect(). At the end... my efforts were worthless. I

Re: [Python-Dev] Register-based VM [Was: Possible performance regression]

2019-02-26 Thread Skip Montanaro
> I uploaded a tarfile I had on my PC to my web site: > > http://python.ca/nas/python/rattlesnake20010813/ > > It seems his name doesn't appear in the readme or source but I think > Rattlesnake was Skip Montanaro's project. I suppose my idea of > unifying the local variables and the registers

Re: [Python-Dev] Another update for PEP 394 -- The "python" Command on Unix-Like Systems

2019-02-26 Thread Chris Barker via Python-Dev
On Tue, Feb 26, 2019 at 9:58 AM Barry Warsaw wrote: > I see this question as having several parts, and the conflation of them is > part of the reason why the unversioned `python` command is so problematic. > Python is used for: > > * OS functionality > * to run applications that aren’t critical

Re: [Python-Dev] Register-based VM [Was: Possible performance regression]

2019-02-26 Thread Greg Ewing
Victor Stinner wrote: LOAD_CONST_REG R0, 2 (const#2) LOAD_GLOBAL_REG R1, 'range' (name#0) CALL_FUNCTION_REG4, R1, R1, R0, 'n' Out of curiosity, why is the function being passed twice here? -- Greg ___ Python-Dev mailing list

Re: [Python-Dev] Register-based VM [Was: Possible performance regression]

2019-02-26 Thread Greg Ewing
Joe Jevnik via Python-Dev wrote: If Python switched to a global stack and global registers we may be able to eliminate a lot of instructions that just shuffle data from the caller's stack to the callee's stack. That would make implementing generators more complicated. -- Greg

Re: [Python-Dev] Another update for PEP 394 -- The "python" Command on Unix-Like Systems

2019-02-26 Thread Steve Dower
On 2/26/2019 1:20 PM, Gregory P. Smith wrote: For an OS distro provided interpreter, being able to restrict its use to only OS distro provided software would be ideal (so ideal that people who haven't learned the hard distro maintenance lessons may hate me for it). Such a restriction could be

Re: [Python-Dev] Another update for PEP 394 -- The "python" Command on Unix-Like Systems

2019-02-26 Thread Victor Stinner
Le mar. 26 févr. 2019 à 22:24, Gregory P. Smith a écrit : > A feature that I find missing from posix-y OSes that support #! lines is an > ability to restrict what can use a given interpreter. Fedora runs system tools (like "/usr/bin/semanage", tool to manager SELinux) with "python3 -Es": $

Re: [Python-Dev] Possible performance regression

2019-02-26 Thread Neil Schemenauer
On 2019-02-26, Raymond Hettinger wrote: > That said, I'm only observing the effect when building with the > Mac default Clang (Apple LLVM version 10.0.0 (clang-1000.11.45.5). > When building GCC 8.3.0, there is no change in performance. My guess is that the code in _PyEval_EvalFrameDefault() got

Re: [Python-Dev] Register-based VM [Was: Possible performance regression]

2019-02-26 Thread Joe Jevnik via Python-Dev
METH_FASTCALL passing arguments on the stack doesn't necessarily mean it will be slow. In x86 there are calling conventions that read all the arguments from the stack, but the rest of the machine is register based. Python could also look at ABI calling conventions for inspiration, like x86-64

Re: [Python-Dev] Register-based VM [Was: Possible performance regression]

2019-02-26 Thread Greg Ewing
Jeroen Demeyer wrote: Let me just say that the code for METH_FASTCALL function/method calls is optimized for a stack layout: a piece of the stack is used directly for calling METH_FASTCALL functions We might be able to get some ideas for dealing with this kind of thing from register-window

Re: [Python-Dev] Register-based VM [Was: Possible performance regression]

2019-02-26 Thread Victor Stinner
Le mar. 26 févr. 2019 à 21:58, Neil Schemenauer a écrit : > It seems his name doesn't appear in the readme or source but I think > Rattlesnake was Skip Montanaro's project. I suppose my idea of > unifying the local variables and the registers could have came from > Rattlesnake. Very little new

[Python-Dev] Compile-time resolution of packages [Was: Another update for PEP 394...]

2019-02-26 Thread Neil Schemenauer
On 2019-02-26, Gregory P. Smith wrote: > On Tue, Feb 26, 2019 at 9:55 AM Barry Warsaw wrote: > For an OS distro provided interpreter, being able to restrict its use to > only OS distro provided software would be ideal (so ideal that people who > haven't learned the hard distro maintenance lessons

Re: [Python-Dev] Possible performance regression

2019-02-26 Thread Victor Stinner
Le mar. 26 févr. 2019 à 22:45, Raymond Hettinger a écrit : > Victor said he generally doesn't care about 5% regressions. That makes sense > for odd corners of Python. The reason I was concerned about this one is that > it hits the eval-loop and seems to effect every single op code. The >

Re: [Python-Dev] Register-based VM [Was: Possible performance regression]

2019-02-26 Thread Victor Stinner
Hum, I read again my old REGISTERVM.txt that I wrote a few years ago. A little bit more context. In my "registervm" fork I also tried to implement further optimizations like moving invariants out of the loop. Some optimizations could change the Python semantics, like remove "duplicated"

Re: [Python-Dev] Register-based VM [Was: Possible performance regression]

2019-02-26 Thread Jeroen Demeyer
Let me just say that the code for METH_FASTCALL function/method calls is optimized for a stack layout: a piece of the stack is used directly for calling METH_FASTCALL functions (without any copying any PyObject* pointers). So this would probably be slower with a register-based VM (which

Re: [Python-Dev] OpenSSL 1.1.1 update for 3.7/3.8

2019-02-26 Thread Christian Heimes
On 26/02/2019 21.31, Wes Turner wrote: >> IMHO it's > fine to ship the last 2.7 build with an OpenSSL version that was EOLed > just 24h earlier. > > Is this a time / cost issue or a branch policy issue? > > If someone was to back port the forthcoming 1.1.1 to 2.7 significantly > before the EOL

Re: [Python-Dev] Register-based VM [Was: Possible performance regression]

2019-02-26 Thread Victor Stinner
No, I wasn't aware of this project. My starting point was: http://static.usenix.org/events/vee05/full_papers/p153-yunhe.pdf Yunhe Shi, David Gregg, Andrew Beatty, M. Anton Ertl, 2005 See also my email to python-dev that I sent in 2012:

Re: [Python-Dev] Register-based VM [Was: Possible performance regression]

2019-02-26 Thread Guido van Rossum
Yes, this should totally be attempted. All the stack manipulation opcodes could be dropped if we just made (nearly) everything use 3-address codes, e.g. ADD would take the names of three registers, left, right and result. The compiler would keep track of which registers contain a live object (for

Re: [Python-Dev] Possible performance regression

2019-02-26 Thread Raymond Hettinger
On Feb 25, 2019, at 8:23 PM, Eric Snow wrote: > > So it looks like commit ef4ac967 is not responsible for a performance > regression. I did narrow it down to that commit and I can consistently reproduce the timing differences. That said, I'm only observing the effect when building with the

Re: [Python-Dev] Compact ordered set

2019-02-26 Thread Victor Stinner
Le mar. 26 févr. 2019 à 17:33, INADA Naoki a écrit : > My company gives me dedicated Linux machine with Core(TM) i7-6700. > So I think it's not issue of my machine. Oh great :-) > perf shows this line caused many page fault. >

Re: [Python-Dev] Another update for PEP 394 -- The "python" Command on Unix-Like Systems

2019-02-26 Thread Petr Viktorin
On 2/26/19 6:54 PM, Barry Warsaw wrote: There haven't been many new ideas since this summary – mostly it was explaining and re-hashing what's been mentioned before. Thanks for the summary Petr. Here’s another way to think about the problem. I know Nick and I have talked about this before,

Re: [Python-Dev] Another update for PEP 394 -- The "python" Command on Unix-Like Systems

2019-02-26 Thread Gregory P. Smith
On Tue, Feb 26, 2019 at 9:55 AM Barry Warsaw wrote: > > There haven't been many new ideas since this summary – mostly it was > explaining and re-hashing what's been mentioned before. > > Thanks for the summary Petr. > > Here’s another way to think about the problem. I know Nick and I have >

Re: [Python-Dev] Another update for PEP 394 -- The "python" Command on Unix-Like Systems

2019-02-26 Thread Gregory P. Smith
On Tue, Feb 26, 2019 at 8:59 AM INADA Naoki wrote: > > > > With *Homebrew*, `python` points to Homebrew’s Python 2.7.x (if > > installed) otherwise the macOS system Python. That's exactly according > > to the PEP. They tried to switch python to python3 before, and got > > rather nasty backlash

Re: [Python-Dev] Compact ordered set

2019-02-26 Thread Raymond Hettinger
Quick summary of what I found when I last ran experiments with this idea: * To get the same lookup performance, the density of index table would need to go down to around 25%. Otherwise, there's no way to make up for the extra indirection and the loss of cache locality. * There was a small win

[Python-Dev] Register-based VM [Was: Possible performance regression]

2019-02-26 Thread Neil Schemenauer
On 2019-02-26, Victor Stinner wrote: > I made an attempt once and it was faster: > https://faster-cpython.readthedocs.io/registervm.html Interesting. I don't think I have seen that before. Were you aware of "Rattlesnake" before you started on that? It seems your approach is similar. Probably

Re: [Python-Dev] Possible performance regression

2019-02-26 Thread Victor Stinner
I made an attempt once and it was faster: https://faster-cpython.readthedocs.io/registervm.html But I had bugs and I didn't know how to implement correctly a compiler. Victor Le mardi 26 février 2019, Neil Schemenauer a écrit : > On 2019-02-25, Eric Snow wrote: >> So it looks like commit

Re: [Python-Dev] OpenSSL 1.1.1 update for 3.7/3.8

2019-02-26 Thread Wes Turner
> IMHO it's fine to ship the last 2.7 build with an OpenSSL version that was EOLed just 24h earlier. Is this a time / cost issue or a branch policy issue? If someone was to back port the forthcoming 1.1.1 to 2.7 significantly before the EOL date, could that be merged? There are all sorts of

Re: [Python-Dev] Possible performance regression

2019-02-26 Thread Neil Schemenauer
On 2019-02-25, Eric Snow wrote: > So it looks like commit ef4ac967 is not responsible for a performance > regression. I did a bit of exploration myself and that was my conclusion as well. Perhaps others would be interested in how to use "perf" so I did a little write up:

Re: [Python-Dev] Another update for PEP 394 -- The "python" Command on Unix-Like Systems

2019-02-26 Thread Barry Warsaw
> There haven't been many new ideas since this summary – mostly it was > explaining and re-hashing what's been mentioned before. Thanks for the summary Petr. Here’s another way to think about the problem. I know Nick and I have talked about this before, but I don’t think any distros have

Re: [Python-Dev] Compact ordered set

2019-02-26 Thread Raymond Hettinger
> On Feb 26, 2019, at 3:30 AM, INADA Naoki wrote: > > I'm working on compact and ordered set implementation. > It has internal data structure similar to new dict from Python 3.6. > > On Feb 26, 2019, at 3:30 AM, INADA Naoki wrote: > > I'm working on compact and ordered set implementation.

Re: [Python-Dev] Another update for PEP 394 -- The "python" Command on Unix-Like Systems

2019-02-26 Thread INADA Naoki
> > With *Homebrew*, `python` points to Homebrew’s Python 2.7.x (if > installed) otherwise the macOS system Python. That's exactly according > to the PEP. They tried to switch python to python3 before, and got > rather nasty backlash citing PEP 394. I assume they will follow the PEP > quite

Re: [Python-Dev] Compact ordered set

2019-02-26 Thread INADA Naoki
On Wed, Feb 27, 2019 at 12:37 AM Victor Stinner wrote: > > Le mar. 26 févr. 2019 à 12:33, INADA Naoki a écrit : > > - unpickle_list: 8.48 us +- 0.09 us -> 12.8 us +- 0.5 us: 1.52x slower > > (+52%)> ... > > ... > > unpickle and unpickle_list shows massive slowdown. I suspect this slowdown > >

[Python-Dev] New member

2019-02-26 Thread Aratz Manterola Lasa via Python-Dev
I do not know if I should be doing this, maybe I have been cheated, but this is my introduction: Hello, my name is Aratz, I am a Computer Engineering Bachelor's (not, not science) student on my 3rd grade. I love Python's gramatic efficiency, but most of all, how much I learn thanks to its

Re: [Python-Dev] [RELEASE] Python 3.8.0a1 is now available for testing

2019-02-26 Thread Victor Stinner
Armin Rigo released https://pypi.org/project/cffi/1.12.2/ which is compatible with Python 3.8.0a2. The issue was related to the PyInterpreterState change: https://bugs.python.org/issue35886#msg336501 Note: "[RELEASE] Python 3.8.0a1 is now available for testing" the correct version is 3.8.0a2 :-)

[Python-Dev] OpenSSL 1.1.1 update for 3.7/3.8

2019-02-26 Thread Christian Heimes
Hi, today's OpenSSL release of 1.0.2r and 1.1.1b reminded me of OpenSSL's release strategy [1]. OpenSSL 1.0.2 will reach EOL on 2019-12-31, 1.1.0 will reach EOL on 2019-09-11 (one year after release of OpenSSL 1.1.1). First the good news: There is no need to take any action for 2.7 to 3.6. As of

Re: [Python-Dev] Compact ordered set

2019-02-26 Thread Victor Stinner
Le mar. 26 févr. 2019 à 12:33, INADA Naoki a écrit : > - unpickle_list: 8.48 us +- 0.09 us -> 12.8 us +- 0.5 us: 1.52x slower > (+52%)> ... > ... > unpickle and unpickle_list shows massive slowdown. I suspect this slowdown > is not caused from set change. Linux perf shows many pagefault is

Re: [Python-Dev] Another update for PEP 394 -- The "python" Command on Unix-Like Systems

2019-02-26 Thread Petr Viktorin
On 2/14/19 9:56 AM, Petr Viktorin wrote: On 2/13/19 4:24 PM, Petr Viktorin wrote: I think it's time for another review. [...] Please see this PR for details and a suggested change: https://github.com/python/peps/pull/893 Summary of the thread so far. Antoine Pitrou noted that the PEP

Re: [Python-Dev] [RELEASE] Python 3.8.0a1 is now available for testing

2019-02-26 Thread Stephane Wirtel
Hi Łukasz, Thank you for your job. I have created a Merge Request for the docker image of Barry [1]. I also filled an issue [2] for brotlipy (used by httpbin and requests). The problem is with PyInterpreterState. Via Twitter, I have proposed to the community to fix the issue [2]. [1]:

[Python-Dev] Compact ordered set

2019-02-26 Thread INADA Naoki
Hello, folks. I'm working on compact and ordered set implementation. It has internal data structure similar to new dict from Python 3.6. It is still work in progress. Comments, tests, and documents should be updated. But it passes existing tests excluding test_sys and test_gdb (both tests

Re: [Python-Dev] Possible performance regression

2019-02-26 Thread Victor Stinner
Hi, Le mar. 26 févr. 2019 à 05:27, Eric Snow a écrit : > I ran the "performance" suite (https://github.com/python/performance), > which has 57 different benchmarks. Ah yes, by the way: I also ran manually performance on speed.python.org yesterday: it added a new dot at Feb 25. > In the