After much testing I found what is causing the regression in 16.04 and later. There are several distinct causes which are attributed to the choices made in debian/rules and the changes in GCC.
Cause #1: the decision to compile `Modules/_math.c` with `-fPIC` *and* link it statically into the python executable [1]. This causes the majority of the slowdown. This may be a bug in GCC or simply a constraint, I didn't find anything specific on this topic, although there are a lot of old bug reports regarding the interaction of -fPIC with -flto. Cause #2: the enablement of `fpectl` [2], specifically the passage of `--with-fpectl` to `configure`. fpectl is disabled in python.org builds by default and its use is discouraged. Yet, Debian builds enable it unconditionally, and it seems to cause a significant performance degradation. It's much less noticeable on 14.04 with GCC 4.8.0, but on more recent releases the performance difference seems to be larger. Plausible Cause #3: stronger stack smashing protection in 16.04, which uses --fstack-protector-strong, wherease 14.04 and earlier used --fstack-protector (with lesser performance overhead). Also, debian/rules limits the scope of PGO's PROFILE_TASK to 377 test suites vs upstream's 397, which affects performance somewhat negatively, but this is not definitive. What are the reasons behind the trimming of the tests used for PGO? Without fpectl, and without -fPIC on _math.c, 2.7.12 built on 16.04 is slower than stock 2.7.6 on 14.04 by about 0.9% in my pyperformance runs [3]. This is in contrast to a whopping 7.95% slowdown when comparing stock versions. Finally, a vanilla Python 2.7.12 build using GCC 5.4.0, default CFLAGS, default PROFILE_TASK and default Modules/Setup.local consistently runs faster in benchmarks than 2.7.6 (by about 0.7%), but I was not able to pinpoint the exact reason for that. Note: the percentages above are the relative change in the geometric mean of pyperformance benchmark results. [1] https://git.launchpad.net/~usd-import-team/ubuntu/+source/python2.7/tree/debian/rules?h=ubuntu/xenial-updates#n421 [2] https://git.launchpad.net/~usd-import- team/ubuntu/+source/python2.7/tree/debian/rules?h=ubuntu/xenial- updates#n117 [3] https://docs.google.com/spreadsheets/d/1L3_gxe- AOYJsXFwGZgFko8jaChB0dFPjK5oMO5T5vj4/edit?usp=sharing -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to python2.7 in Ubuntu. https://bugs.launchpad.net/bugs/1638695 Title: Python 2.7.12 performance regression Status in python2.7 package in Ubuntu: Confirmed Bug description: I work on the OpenStack-Ansible project and we've noticed that testing jobs on 16.04 take quite a bit longer to complete than on 14.04. They complete within an hour on 14.04 but they normally take 90 minutes or more on 16.04. We use the same version of Ansible with both versions of Ubuntu. After more digging, I tested python performance (using the 'performance' module) on 14.04 (2.7.6) and on 16.04 (2.7.12). There is a significant performance difference between each version of python. That is detailed in a spreadsheet[0]. I began using perf to dig into the differences when running the python performance module and when using Ansible playbooks. CPU migrations (as measured by perf) are doubled in Ubuntu 16.04 when running the same python workloads. I tried changing some of the kerne.sched sysctl configurables but they had very little effect on the results. I compiled python 2.7.12 from source on 14.04 and found the performance to be unchanged there. I'm not entirely sure where the problem might be now. We also have a bug open in OpenStack-Ansible[1] that provides additional detail. Thanks in advance for any help you can provide! [0] https://docs.google.com/spreadsheets/d/18MmptS_DAd1YP3OhHWQqLYVA9spC3xLt4PS3STI6tds/edit?usp=sharing [1] https://bugs.launchpad.net/openstack-ansible/+bug/1637494 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/python2.7/+bug/1638695/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
