[Python-Dev] Python 3.5.4rc1 and 3.4.7rc1 slipping by a day, to July 24 2017

[Larry Hastings]

Release engineering for 3.5.4rc1 and 3.4.7rc1 took a lot longer than 
expected, because this is the first release using "blurb", and it turned 
out there was a lot of work left to do and a couple dark corners yet to 
stumble over.  3.5.4rc1 and 3.4.7rc1 will be released Monday, July 24, 
2017.  The release dates for 3.5.4 final and 3.4.7 final are not 
expected to change.


Sorry about that,


//arry/
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Is Windows XP still supported on Python 2.7?

[Victor Stinner]

Hi,

We have a Windows XP buildbot for Python 2.7, run by David Bolen:
http://buildbot.python.org/all/builders/x86%20Windows%20XP%202.7/

test_bsddb3 fails randomly on this buildbot:
http://bugs.python.org/issue30778

But Windows XP clearly reached its end-of-life, Microsoft doesn't
support it anymore. So my question is if it makes sense to spend time
on it?

We have a rule for new x.y.0 released, but not if a Microsoft Windows
support expires during the lifetime of a Python release (2.7 here):
https://www.python.org/dev/peps/pep-0011/#microsoft-windows

Firefox made great efforts to support Windows XP last years, but they
decided to drop support last March with Firefox 52, last release
supporting XP and Visa:
https://support.mozilla.org/en-US/kb/end-support-windows-xp-and-vista

Victor
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Is Windows XP still supported on Python 2.7?

[Alex Walters]

The promise that PEP-11 is making is that as long as a python was released
while Microsoft still supported that OS, and that python is still supported,
there will still be a python that works for you.  So, yes, Windows XP is
long since unsupported by Microsoft, but a disturbing number of people still
run it.  (I think the NHS in the UK still runs embedded windows XP, just to
name a big one).  Yes, it's a support burden, but it's on the support burden
version of python anyways.  2.7 is a very slow moving branch so it shouldn't
be THAT big of a pain for the last 2 years of python 2 support.

> -Original Message-
> From: Python-Dev [mailto:python-dev-bounces+tritium-
> list=sdamon@python.org] On Behalf Of Victor Stinner
> Sent: Monday, July 24, 2017 5:05 AM
> To: Python Dev ; David Bolen
> 
> Subject: [Python-Dev] Is Windows XP still supported on Python 2.7?
> 
> Hi,
> 
> We have a Windows XP buildbot for Python 2.7, run by David Bolen:
> http://buildbot.python.org/all/builders/x86%20Windows%20XP%202.7/
> 
> test_bsddb3 fails randomly on this buildbot:
> http://bugs.python.org/issue30778
> 
> But Windows XP clearly reached its end-of-life, Microsoft doesn't
> support it anymore. So my question is if it makes sense to spend time
> on it?
> 
> We have a rule for new x.y.0 released, but not if a Microsoft Windows
> support expires during the lifetime of a Python release (2.7 here):
> https://www.python.org/dev/peps/pep-0011/#microsoft-windows
> 
> Firefox made great efforts to support Windows XP last years, but they
> decided to drop support last March with Firefox 52, last release
> supporting XP and Visa:
> https://support.mozilla.org/en-US/kb/end-support-windows-xp-and-vista
> 
> Victor
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/tritium-
> list%40sdamon.com

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Is Windows XP still supported on Python 2.7?

[Victor Stinner]

2017-07-24 11:38 GMT+02:00 Alex Walters :
> The promise that PEP-11 is making is that as long as a python was released
> while Microsoft still supported that OS, and that python is still supported,
> there will still be a python that works for you.  So, yes, Windows XP is
> long since unsupported by Microsoft, but a disturbing number of people still
> run it.  (I think the NHS in the UK still runs embedded windows XP, just to
> name a big one).  Yes, it's a support burden, but it's on the support burden
> version of python anyways.  2.7 is a very slow moving branch so it shouldn't
> be THAT big of a pain for the last 2 years of python 2 support.

Python 2.7.13 which was released at 2016-12-17 still supports Windows
XP. It's not like you cannot install on Windows XP anymore.

The question is who will fix bugs specific to Windows XP or Visa until
2020. Having a working Windows XP is probably required to debug and
fix such bugs.

I don't want to install Windows XP on my network. The last time I ran
XP was something like 4 years ago, and I was reading an article saying
that malwares are able to attack XP even during the installation time.
So you cannot be sure that installed Windows is safe of malwares or
not...

Victor
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Cython compiled stdlib modules - Re: Python startup time

[Nick Coghlan]

On 22 July 2017 at 06:43, Stefan Behnel  wrote:
> Nick Coghlan schrieb am 21.07.2017 um 08:23:
>> I'll also note that in these cases where the import overhead is
>> proportionally significant for always-imported modules, we may want to look
>> at the benefits of freezing them (if they otherwise remain as pure Python
>> modules), or compiling them as builtin modules (if we switch them over to
>> Cython), in addition to looking at ways to make the modules themselves
>> faster.
>
> Just for the sake of it, I gave the Cython compilation a try. I had to
> apply the attached hack to Lib/typing.py to get the test passing, because
> it uses frame call offsets in some places and Cython functions do not
> create frames when being called (they only create them for exception
> traces). I also had to disable the import of "abc" in the Cython generated
> module to remove the circular self dependency at startup when the "abc"
> module is compiled. That shouldn't have an impact on the runtime
> performance, though.

[snip]

> As it stands, the gain is probably not worth the increase in library file
> size, which also translates to a higher bottom line for the memory
> consumption. At least not for these two modules. Manually optimising the
> files would likely also reduce the .so file size in addition to giving
> better speedups, though, because the generated code would become less generic.

Thanks for trying the experiment! I agree with your conclusion that
the file size impact likely rules it out as a general technique.

Selective freezing may still be interesting though, since that at
least avoids the import path searches and merges the disk read into
the initial loading of the executable.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] dictionaries in Dataframe column

[Katherine Bobrovnik]

Hello guys!
I've stuck at this:
I have pandas Dataframe with a lot of columns. One column contains
dictionaries with emoji like {'count': 1, 'name': 'fire'}. My goal is
to sort rows of this Dataframe by the number from 'count'. Like first
row will be with  {'count': 49, 'name': '+1'}, the last -  {'count':
1, 'name': 'fire'}, for instance. Help, please :)

Best regards,
Kateryna
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] dictionaries in Dataframe column

[Oleg Broytman]

Hello.

   This mailing list is to work on developing Python (adding new
features to Python itself and fixing bugs); if you're having problems
learning, understanding or using Python, please find another forum.
Probably python-list/comp.lang.python mailing list/news group is the
best place; there are Python developers who participate in it; you may
get a faster, and probably more complete, answer there. See
http://www.python.org/community/ for other lists/news groups/fora. Thank
you for understanding.

On Mon, Jul 24, 2017 at 11:04:48AM +0300, Katherine Bobrovnik 
 wrote:
> Hello guys!
> I've stuck at this:
> I have pandas Dataframe with a lot of columns. One column contains
> dictionaries with emoji like {'count': 1, 'name': 'fire'}. My goal is
> to sort rows of this Dataframe by the number from 'count'. Like first
> row will be with  {'count': 49, 'name': '+1'}, the last -  {'count':
> 1, 'name': 'fire'}, for instance. Help, please :)
> 
> Best regards,
> Kateryna
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/phd%40phdru.name

Oleg.
-- 
 Oleg Broytmanhttp://phdru.name/p...@phdru.name
   Programmers don't die, they just GOSUB without RETURN.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] for...else

[Kiuhnm via Python-Dev]

Hello,

I think that the expression "for...else" or "while...else" is completely 
counter-intuitive. Wouldn't it be possible to make it clearer? Maybe 
something like


break in for i in range(n):
  ...
  if cond:
break
else:
  ...

I'm not an English native speaker so I don't know whether "break in" is 
acceptable English in this context or can only mean "to get into a 
building by force".


Kiuhnm
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] for...else

[Steven D'Aprano]

Hello Kiuhnm, and welcome.

On Mon, Jul 24, 2017 at 05:35:03PM +0200, Kiuhnm via Python-Dev wrote:
> Hello,
> 
> I think that the expression "for...else" or "while...else" is completely 
> counter-intuitive.


You may be right -- this has been discussed many, many times before. In 
my personal opinion, the best (and only accurate!) phrase would have 
been:

for item in sequence:
# block
then:
# block

If you look at the byte-code generated by a for...else statement, you 
see that the "else" block is unconditionally executed after the for loop 
completes, unless something causes a jump outside of the entire 
statement: return, break, or raise. So it is more like:

- run the loop;
- *then* run the following block

rather than:

- run the loop;
- otherwise ("else") run the following block.

Others disagree and would prefer other keywords. But regardless, 
backwards compatibility means that we must keep "for...else", so I'm 
afraid that discussing alternatives is *almost certainly* a waste of 
time.


> Wouldn't it be possible to make it clearer? Maybe 
> something like

At this point, no, it is not practical to change the syntax used. Maybe 
when Python 3.0 was first introduced, but that ship has long sailed. It 
is very, very unlikely that the syntax for this will ever change, but if 
it does, it probably won't be until something in the distant future like 
Python 5.

But not Python 4: Guido has already ruled that Python 4 will not include 
major backwards-incompatible changes. Going from 3 to 4 will not be as 
disruptive as going from 2 to 3.

So depending on how you look at it: discussing alternative syntax to 
for...else is either ten years too late or ten years too early.



-- 
Steve
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] for...else

[Ben Hoyt]

This is more of a python-ideas discussion, and Steven's answer is good.

I'll just add one thing. Maybe it's obvious to others, but I've liked
for...else since I found a kind of mnemonic to help me remember when the
"else" part happens: I think of it not as "for ... else" but as "break ...
else" -- saying it this way makes it clear to me that the break goes with
the else. "If this condition inside the loop is true, break. ... *else* if
we didn't break, do this other thing after the loop."

-Ben

On Mon, Jul 24, 2017 at 12:14 PM, Steven D'Aprano 
wrote:

> Hello Kiuhnm, and welcome.
>
> On Mon, Jul 24, 2017 at 05:35:03PM +0200, Kiuhnm via Python-Dev wrote:
> > Hello,
> >
> > I think that the expression "for...else" or "while...else" is completely
> > counter-intuitive.
>
>
> You may be right -- this has been discussed many, many times before. In
> my personal opinion, the best (and only accurate!) phrase would have
> been:
>
> for item in sequence:
> # block
> then:
> # block
>
> If you look at the byte-code generated by a for...else statement, you
> see that the "else" block is unconditionally executed after the for loop
> completes, unless something causes a jump outside of the entire
> statement: return, break, or raise. So it is more like:
>
> - run the loop;
> - *then* run the following block
>
> rather than:
>
> - run the loop;
> - otherwise ("else") run the following block.
>
> Others disagree and would prefer other keywords. But regardless,
> backwards compatibility means that we must keep "for...else", so I'm
> afraid that discussing alternatives is *almost certainly* a waste of
> time.
>
>
> > Wouldn't it be possible to make it clearer? Maybe
> > something like
>
> At this point, no, it is not practical to change the syntax used. Maybe
> when Python 3.0 was first introduced, but that ship has long sailed. It
> is very, very unlikely that the syntax for this will ever change, but if
> it does, it probably won't be until something in the distant future like
> Python 5.
>
> But not Python 4: Guido has already ruled that Python 4 will not include
> major backwards-incompatible changes. Going from 3 to 4 will not be as
> disruptive as going from 2 to 3.
>
> So depending on how you look at it: discussing alternative syntax to
> for...else is either ten years too late or ten years too early.
>
>
>
> --
> Steve
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> benhoyt%40gmail.com
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] for...else

[Alexander Belopolsky]

On Mon, Jul 24, 2017 at 12:23 PM, Ben Hoyt  wrote:
> .. I found a kind of mnemonic to help me remember when the
> "else" part happens: I think of it not as "for ... else" but as "break ...
> else" -- saying it this way makes it clear to me that the break goes with
> the else. "If this condition inside the loop is true, break. ... *else* if
> we didn't break, do this other thing after the loop."

Note that since break itself is typically guarded by an "if" as in

for i in x:
...
if cond(i):
break
...
else:
...

you can match the "else" above to the "if" inside the loop.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Is Windows XP still supported on Python 2.7?

[Terry Reedy]

On 7/24/2017 5:04 AM, Victor Stinner wrote:


We have a Windows XP buildbot for Python 2.7, run by David Bolen:
http://buildbot.python.org/all/builders/x86%20Windows%20XP%202.7/

test_bsddb3 fails randomly on this buildbot:
http://bugs.python.org/issue30778


If that turns out to be an unfixable intermittent failure of two 
particular functions, then it becomes expected.  To keep buildbots 
green, skip the one that crashes and turn the failure of the other into 
a skip.



But Windows XP clearly reached its end-of-life, Microsoft doesn't
support it anymore. So my question is if it makes sense to spend time
on it?


When an entity *sells* 'support', that means an active effort to fix. 
For free, volunteer-built Python, 'support' for system Z means that we 
(continue to) allow system Z specific code, new patches, and a system Z 
buildbot.  If we make an installer, it installs on system Z.


In that minimal sense, xp is still generally supported, although experts 
for specific modules may refuse to review and merge patches for 'their' 
modules.  But that exception typically has nothing to do with xp in 
particular.  But it could.


When XP became unsupported in 3.5, xp-specific code was removed, the xp 
buildbot was removed, we started rejecting xp-specific patches, and our 
windows installer refused to install on XP.  Are you are proposing this 
for 2.7?



We have a rule for new x.y.0 released, but not if a Microsoft Windows
support expires during the lifetime of a Python release (2.7 here):
https://www.python.org/dev/peps/pep-0011/#microsoft-windows


> Firefox made great efforts to support Windows XP last years, but they
> decided to drop support last March with Firefox 52, last release
> supporting XP and Visa:
> https://support.mozilla.org/en-US/kb/end-support-windows-xp-and-vista

The link says that they continue with security patches until next 
September, at which point they will review installation numbers.  The 
extra 5 years of support for 2.7, above the originally intended 5 years, 
is mainly for security fixes, although some people continue with routine 
non-security bugfixes.


--
Terry Jan Reedy

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Is Windows XP still supported on Python 2.7?

[Zachary Ware]

On Mon, Jul 24, 2017 at 11:48 AM, Terry Reedy  wrote:
> On 7/24/2017 5:04 AM, Victor Stinner wrote:
>> We have a Windows XP buildbot for Python 2.7, run by David Bolen:
>> http://buildbot.python.org/all/builders/x86%20Windows%20XP%202.7/
>>
>> test_bsddb3 fails randomly on this buildbot:
>> http://bugs.python.org/issue30778
>
>
> If that turns out to be an unfixable intermittent failure of two particular
> functions, then it becomes expected.  To keep buildbots green, skip the one
> that crashes and turn the failure of the other into a skip.

We are committed to support back to Windows 2000 in Python 2.7.  In
general, that just means "don't commit something that makes the
platform unsupportable and accept a patch if somebody fixes something
on that platform".  In this case, considering that it's a test of a
2.x-only module on an out-of-vendor-support OS, skipping the tests
(possibly even the entirety of test_bsddb3) on XP sounds just fine to
me.

-- 
Zach
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Is Windows XP still supported on Python 2.7?

[Victor Stinner]

2017-07-24 19:05 GMT+02:00 Zachary Ware :
> In this case, considering that it's a test of a
> 2.x-only module on an out-of-vendor-support OS, skipping the tests
> (possibly even the entirety of test_bsddb3) on XP sounds just fine to
> me.

Oh ok. Since Terry and you agree on that, I will skip the test on Windows XP.

Victor
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] for...else

[Isaac Morland]

The way I remember it is to observe that the following are *almost* exactly
the same thing:

if C:
T
else:
E

while C:
T
else:
E

The *only* differences are:

1) where execution jumps if it reaches the end of the T: in the "while", it
jumps back to the while itself, resulting in the condition being rechecked,
whereas in the "if" execution skips over the "else" to whatever follows; and

2) in the "while", inside the T "break" and "continue" relate to this
control structure rather than to a containing loop.

(At least I don't think I'm missing any other differences!)

Seen this way, the meaning of the "else" is easy to understand and to
remember.

And the for loop else is like the while loop else.


On 24 July 2017 at 11:35, Kiuhnm via Python-Dev 
wrote:

> Hello,
>
> I think that the expression "for...else" or "while...else" is completely
> counter-intuitive. Wouldn't it be possible to make it clearer? Maybe
> something like
>
> break in for i in range(n):
>   ...
>   if cond:
> break
> else:
>   ...
>
> I'm not an English native speaker so I don't know whether "break in" is
> acceptable English in this context or can only mean "to get into a building
> by force".
>
> Kiuhnm
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/isaac.
> morland%40gmail.com
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Program runs in 12s on Python 2.7, but 5s on Python 3.5 -- why so much difference?

[Wang, Peter Xihong]

Hi Ben,

Out of curiosity with a quick experiment, I ran your pentomino.py with 2.7.12 
PGO+LTO build (Ubuntu OS 16.04.2 LTS default at /usr/bin/python), and compared 
with 3.7.0 alpha1 PGO+LTO (which I built a while ago), on my SkyLake processor 
based desktop, and 2.7 outperforms 3.7 by 3.5%.
On your 2.5 GHz i7 system, I'd recommend making sure the 2 Python binaries you 
are comparing are in equal footings (compiled with same optimization PGO+LTO).

Thanks,

Peter



-Original Message-
From: Python-Dev 
[mailto:python-dev-bounces+peter.xihong.wang=intel@python.org] On Behalf Of 
Nathaniel Smith
Sent: Tuesday, July 18, 2017 7:00 PM
To: Ben Hoyt 
Cc: Python-Dev 
Subject: Re: [Python-Dev] Program runs in 12s on Python 2.7, but 5s on Python 
3.5 -- why so much difference?

I'd probably start with a regular C-level profiler, like perf or callgrind. 
They're not very useful for comparing two versions of code written in Python, 
but here the Python code is the same (modulo changes in the stdlib), and it's 
changes in the interpreter's C code that probably make the difference.

On Tue, Jul 18, 2017 at 9:03 AM, Ben Hoyt  wrote:
> Hi folks,
>
> (Not entirely sure this is the right place for this question, but 
> hopefully it's of interest to several folks.)
>
> A few days ago I posted a note in response to Victor Stinner's 
> articles on his CPython contributions, noting that I wrote a program 
> that ran in 11.7 seconds on Python 2.7, but only takes 5.1 seconds on 
> Python 3.5 (on my 2.5 GHz macOS i7), more than 2x as fast. Obviously 
> this is a Good Thing, but I'm curious as to why there's so much difference.
>
> The program is a pentomino puzzle solver, and it works via code 
> generation, generating a ton of nested "if" statements, so I believe 
> it's exercising the Python bytecode interpreter heavily. Obviously 
> there have been some big optimizations to make this happen, but I'm 
> curious what the main improvements are that are causing this much difference.
>
> There's a writeup about my program here, with benchmarks at the bottom:
> http://benhoyt.com/writings/python-pentomino/
>
> This is the generated Python code that's being exercised:
> https://github.com/benhoyt/python-pentomino/blob/master/generated_solv
> e.py
>
> For reference, on Python 3.6 it runs in 4.6 seconds (same on Python 
> 3.7 alpha). This smallish increase from Python 3.5 to Python 3.6 was 
> more expected to me due to the bytecode changing to wordcode in 3.6.
>
> I tried using cProfile on both Python versions, but that didn't say 
> much, because the functions being called aren't taking the majority of the 
> time.
> How does one benchmark at a lower level, or otherwise explain what's 
> going on here?
>
> Thanks,
> Ben
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/njs%40pobox.com
>



--
Nathaniel J. Smith -- https://vorpus.org 
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/peter.xihong.wang%40intel.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Program runs in 12s on Python 2.7, but 5s on Python 3.5 -- why so much difference?

[Ben Hoyt]

Thanks for testing.

Oddly, I just tested it in Linux (Ubuntu), and get the same results as you
-- Python 2.7.13 outperforms 3 (3.5.3 in my case) by a few percent. And
even under a Virtualbox VM it takes 3.4 and 3.6 seconds, compared to ~5s on
the host macOS operating system. Very odd. I guess that means Virtualbox is
very good, and that clang/LLVM is not as good at optimizing the Python VM
as gcc is.

I can't find anything majorly different about my macOS Python 2 and 3
builds. Both look like they have PGO turned on (from
sysconfig.get_config_vars()). Both have HAVE_COMPUTED_GOTOS=1 but
USE_COMPUTED_GOTOS=0 for some reason. My Python 2 version is the macOS
system version (/usr/local/bin/python2), whereas my Python3 version is from
"brew install", so that's probably the difference, though still doesn't
explain exactly why.

-Ben

On Mon, Jul 24, 2017 at 1:49 PM, Wang, Peter Xihong <
peter.xihong.w...@intel.com> wrote:

> Hi Ben,
>
> Out of curiosity with a quick experiment, I ran your pentomino.py with
> 2.7.12 PGO+LTO build (Ubuntu OS 16.04.2 LTS default at /usr/bin/python),
> and compared with 3.7.0 alpha1 PGO+LTO (which I built a while ago), on my
> SkyLake processor based desktop, and 2.7 outperforms 3.7 by 3.5%.
> On your 2.5 GHz i7 system, I'd recommend making sure the 2 Python binaries
> you are comparing are in equal footings (compiled with same optimization
> PGO+LTO).
>
> Thanks,
>
> Peter
>
>
>
> -Original Message-
> From: Python-Dev [mailto:python-dev-bounces+peter.xihong.wang=intel.com@
> python.org] On Behalf Of Nathaniel Smith
> Sent: Tuesday, July 18, 2017 7:00 PM
> To: Ben Hoyt 
> Cc: Python-Dev 
> Subject: Re: [Python-Dev] Program runs in 12s on Python 2.7, but 5s on
> Python 3.5 -- why so much difference?
>
> I'd probably start with a regular C-level profiler, like perf or
> callgrind. They're not very useful for comparing two versions of code
> written in Python, but here the Python code is the same (modulo changes in
> the stdlib), and it's changes in the interpreter's C code that probably
> make the difference.
>
> On Tue, Jul 18, 2017 at 9:03 AM, Ben Hoyt  wrote:
> > Hi folks,
> >
> > (Not entirely sure this is the right place for this question, but
> > hopefully it's of interest to several folks.)
> >
> > A few days ago I posted a note in response to Victor Stinner's
> > articles on his CPython contributions, noting that I wrote a program
> > that ran in 11.7 seconds on Python 2.7, but only takes 5.1 seconds on
> > Python 3.5 (on my 2.5 GHz macOS i7), more than 2x as fast. Obviously
> > this is a Good Thing, but I'm curious as to why there's so much
> difference.
> >
> > The program is a pentomino puzzle solver, and it works via code
> > generation, generating a ton of nested "if" statements, so I believe
> > it's exercising the Python bytecode interpreter heavily. Obviously
> > there have been some big optimizations to make this happen, but I'm
> > curious what the main improvements are that are causing this much
> difference.
> >
> > There's a writeup about my program here, with benchmarks at the bottom:
> > http://benhoyt.com/writings/python-pentomino/
> >
> > This is the generated Python code that's being exercised:
> > https://github.com/benhoyt/python-pentomino/blob/master/generated_solv
> > e.py
> >
> > For reference, on Python 3.6 it runs in 4.6 seconds (same on Python
> > 3.7 alpha). This smallish increase from Python 3.5 to Python 3.6 was
> > more expected to me due to the bytecode changing to wordcode in 3.6.
> >
> > I tried using cProfile on both Python versions, but that didn't say
> > much, because the functions being called aren't taking the majority of
> the time.
> > How does one benchmark at a lower level, or otherwise explain what's
> > going on here?
> >
> > Thanks,
> > Ben
> >
> > ___
> > Python-Dev mailing list
> > Python-Dev@python.org
> > https://mail.python.org/mailman/listinfo/python-dev
> > Unsubscribe:
> > https://mail.python.org/mailman/options/python-dev/njs%40pobox.com
> >
>
>
>
> --
> Nathaniel J. Smith -- https://vorpus.org __
> _
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> peter.xihong.wang%40intel.com
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Program runs in 12s on Python 2.7, but 5s on Python 3.5 -- why so much difference?

[Wang, Peter Xihong]

I believe we have evaluated clang vs gcc before (long time ago), and gcc won at 
that time.

PGO might have overshadowed impact from computed goto, and thus the latter may 
no longer be needed.

When the performance difference is as large as 50%, there could be various 
options to nail down the root cause, including bytecode analysis.  However, 
coming down to 3.6 sec vs 3.4 sec, a delta of ~5%, it could be hard to find 
out.  Internally we use sampling based performance tools for micro-architecture 
level analysis.  Or generic Linux based and open source tool “perf” is very 
good to use.   You could also do a disassembly analysis/comparison of the 
object files such as the main loop, ceval.o, looking at the efficiency of the 
generated codes (which gives generic info regarding to Python2 and 3, but may 
not tell you the run time behavior with respect your specific app, 
pentomino.py).

Hope that helps.

Peter


From: Ben Hoyt [mailto:benh...@gmail.com]
Sent: Monday, July 24, 2017 12:35 PM
To: Wang, Peter Xihong 
Cc: Nathaniel Smith ; Python-Dev 
Subject: Re: [Python-Dev] Program runs in 12s on Python 2.7, but 5s on Python 
3.5 -- why so much difference?

Thanks for testing.

Oddly, I just tested it in Linux (Ubuntu), and get the same results as you -- 
Python 2.7.13 outperforms 3 (3.5.3 in my case) by a few percent. And even under 
a Virtualbox VM it takes 3.4 and 3.6 seconds, compared to ~5s on the host macOS 
operating system. Very odd. I guess that means Virtualbox is very good, and 
that clang/LLVM is not as good at optimizing the Python VM as gcc is.

I can't find anything majorly different about my macOS Python 2 and 3 builds. 
Both look like they have PGO turned on (from sysconfig.get_config_vars()). Both 
have HAVE_COMPUTED_GOTOS=1 but USE_COMPUTED_GOTOS=0 for some reason. My Python 
2 version is the macOS system version (/usr/local/bin/python2), whereas my 
Python3 version is from "brew install", so that's probably the difference, 
though still doesn't explain exactly why.

-Ben

On Mon, Jul 24, 2017 at 1:49 PM, Wang, Peter Xihong 
mailto:peter.xihong.w...@intel.com>> wrote:
Hi Ben,

Out of curiosity with a quick experiment, I ran your pentomino.py with 2.7.12 
PGO+LTO build (Ubuntu OS 16.04.2 LTS default at /usr/bin/python), and compared 
with 3.7.0 alpha1 PGO+LTO (which I built a while ago), on my SkyLake processor 
based desktop, and 2.7 outperforms 3.7 by 3.5%.
On your 2.5 GHz i7 system, I'd recommend making sure the 2 Python binaries you 
are comparing are in equal footings (compiled with same optimization PGO+LTO).

Thanks,

Peter



-Original Message-
From: Python-Dev 
[mailto:python-dev-bounces+peter.xihong.wang=intel@python.org]
 On Behalf Of Nathaniel Smith
Sent: Tuesday, July 18, 2017 7:00 PM
To: Ben Hoyt mailto:benh...@gmail.com>>
Cc: Python-Dev mailto:python-dev@python.org>>
Subject: Re: [Python-Dev] Program runs in 12s on Python 2.7, but 5s on Python 
3.5 -- why so much difference?

I'd probably start with a regular C-level profiler, like perf or callgrind. 
They're not very useful for comparing two versions of code written in Python, 
but here the Python code is the same (modulo changes in the stdlib), and it's 
changes in the interpreter's C code that probably make the difference.

On Tue, Jul 18, 2017 at 9:03 AM, Ben Hoyt 
mailto:benh...@gmail.com>> wrote:
> Hi folks,
>
> (Not entirely sure this is the right place for this question, but
> hopefully it's of interest to several folks.)
>
> A few days ago I posted a note in response to Victor Stinner's
> articles on his CPython contributions, noting that I wrote a program
> that ran in 11.7 seconds on Python 2.7, but only takes 5.1 seconds on
> Python 3.5 (on my 2.5 GHz macOS i7), more than 2x as fast. Obviously
> this is a Good Thing, but I'm curious as to why there's so much difference.
>
> The program is a pentomino puzzle solver, and it works via code
> generation, generating a ton of nested "if" statements, so I believe
> it's exercising the Python bytecode interpreter heavily. Obviously
> there have been some big optimizations to make this happen, but I'm
> curious what the main improvements are that are causing this much difference.
>
> There's a writeup about my program here, with benchmarks at the bottom:
> http://benhoyt.com/writings/python-pentomino/
>
> This is the generated Python code that's being exercised:
> https://github.com/benhoyt/python-pentomino/blob/master/generated_solv
> e.py
>
> For reference, on Python 3.6 it runs in 4.6 seconds (same on Python
> 3.7 alpha). This smallish increase from Python 3.5 to Python 3.6 was
> more expected to me due to the bytecode changing to wordcode in 3.6.
>
> I tried using cProfile on both Python versions, but that didn't say
> much, because the functions being called aren't taking the majority of the 
> time.
> How does one benchmark at a lower level, or otherwise explain what's
> going on 

Re: [Python-Dev] Program runs in 12s on Python 2.7, but 5s on Python 3.5 -- why so much difference?

[Gregory P. Smith]

On Mon, Jul 24, 2017 at 1:49 PM Wang, Peter Xihong <
peter.xihong.w...@intel.com> wrote:

> I believe we have evaluated clang vs gcc before (long time ago), and gcc
> won at that time.
>
>
>
> PGO might have overshadowed impact from computed goto, and thus the latter
> may no longer be needed.
>

Computed goto is still needed.  PGO does not magically replace it.  A PGO
build with computed goto is faster than one without computed goto.

... as tested on gcc 4.9 a couple years ago. I doubt that has changed or
changes between compilers; PGO and computed goto are different types of
optimizations.

-gps
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Appending a link back to bugs.python.org in GitHub PRs

[Mariatta Wijaya]

Thanks for working on this, Kushal and Brett.

Works great!

Mariatta Wijaya

On Fri, Jul 21, 2017 at 2:28 PM, Brett Cannon  wrote:

> Thanks to Kushal Das we now have one of the most requested features since
> the transition: a link in PRs back to bugs.python.org (in a more
> discoverable way since we have had them since Bedevere launched :) . When a
> pull request comes in with an issue number in the title (or one gets
> added), a link to bugs.python.org will be appended to the PR's body (the
> message you fill out when creating a PR). There's no logic to remove the
> link if the issue number is removed from the title, changed, or for
> multiple issue numbers since basically those cases are all rare and it was
> easier to launch without that kind of support.
>
> P.S.: Berker Peksag is working on providing commit emails with diffs in
> them which is the other most requested feature since the transition.
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> mariatta.wijaya%40gmail.com
>
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] for...else

[Nick Coghlan]

On 25 July 2017 at 02:23, Ben Hoyt  wrote:
> This is more of a python-ideas discussion, and Steven's answer is good.
>
> I'll just add one thing. Maybe it's obvious to others, but I've liked
> for...else since I found a kind of mnemonic to help me remember when the
> "else" part happens: I think of it not as "for ... else" but as "break ...
> else" -- saying it this way makes it clear to me that the break goes with
> the else. "If this condition inside the loop is true, break. ... *else* if
> we didn't break, do this other thing after the loop."

For folks looking for a more in-depth explanation of the
"if-break-else" approach to thinking about this construct:
http://python-notes.curiousefficiency.org/en/latest/python_concepts/break_else.html

That article also has a note explaining that we're unlikely to ever
change this: 
http://python-notes.curiousefficiency.org/en/latest/python_concepts/break_else.html#but-couldn-t-python-be-different

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com