[issue30468] Propagate zipfile.py pypy issue #905 patch to CPython 3.7
Alecsandru Patrascu added the comment: Serhiy, I am curious why did you have chosen to compute the CRC32 table everytime? It is standard (the generator polynomial does not change) and always will output the same values. And it is also less computational intensive to loading a precomputed array vs calculating it each time a zip archive is extracted. -- nosy: +alecsandru.patrascu ___ Python tracker <http://bugs.python.org/issue30468> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue26359] CPython build options for out-of-the box performance
Alecsandru Patrascu added the comment: The patches in discussions are the v02 ones. -- ___ Python tracker <http://bugs.python.org/issue26359> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue26359] CPython build options for out-of-the box performance
Alecsandru Patrascu added the comment: I've updated the patch for 2.7 branch to contain the LTO part merged by Gregory. -- Added file: http://bugs.python.org/file43142/cpython2_with_optimizations_v02.patch ___ Python tracker <http://bugs.python.org/issue26359> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue26359] CPython build options for out-of-the box performance
Alecsandru Patrascu added the comment: Hello Brett, I saw that there was no other activity on this issue so far. Can you please update this, as I do not have rights? -- ___ Python tracker <http://bugs.python.org/issue26359> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue26359] CPython build options for out-of-the box performance
Alecsandru Patrascu added the comment: Brett, I tried to add you in the "assigned to", but I do not have any rights to update those :( -- ___ Python tracker <http://bugs.python.org/issue26359> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue26359] CPython build options for out-of-the box performance
Alecsandru Patrascu added the comment: I've attached a cleaner version for CPython 3 and one for CPython 2. The problem with CPython 2 is that Gregory did not yet merged the LTO on this branch (#25702), but as soon as he does it, I'll post updated patches. Regards, Alecsandru -- ___ Python tracker <http://bugs.python.org/issue26359> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue26359] CPython build options for out-of-the box performance
Changes by Alecsandru Patrascu : Added file: http://bugs.python.org/file42872/cpython3_with_optimizations_v02.patch ___ Python tracker <http://bugs.python.org/issue26359> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue26359] CPython build options for out-of-the box performance
Changes by Alecsandru Patrascu : Added file: http://bugs.python.org/file42871/cpython2_with_optimizations_v01.patch ___ Python tracker <http://bugs.python.org/issue26359> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue26359] CPython build options for out-of-the box performance
Alecsandru Patrascu added the comment: I've added the --with-optimizations option to configure, for CPython3 only at this point. If it looks good I'll create a version also for CPython2. -- Added file: http://bugs.python.org/file42576/cpython3_with_optimizations_v01.patch ___ Python tracker <http://bugs.python.org/issue26359> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue26788] test_gdb fails all tests on a profile-opt build configured --with-lto
Alecsandru Patrascu added the comment: I uploaded a patch that skips these tests if optimizations are available. -- keywords: +patch Added file: http://bugs.python.org/file42574/issue26788.patch ___ Python tracker <http://bugs.python.org/issue26788> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue26787] test_distutils fails when configured --with-lto
Alecsandru Patrascu added the comment: This patch is for CPython 3, and if it looks good I will post also the CPython 2 version. -- ___ Python tracker <http://bugs.python.org/issue26787> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue26787] test_distutils fails when configured --with-lto
Alecsandru Patrascu added the comment: Hello, You can find a patch for this issue attached. Basically, since LTO needs LD flags when used, I modified the makefile template file and the test to push the flags up to the sysconfig module. -- keywords: +patch Added file: http://bugs.python.org/file42504/issue26787.patch ___ Python tracker <http://bugs.python.org/issue26787> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue26788] test_gdb fails all tests on a profile-opt build configured --with-lto
Alecsandru Patrascu added the comment: I will investigate this issue to understand what is happening there and submit a fix. Thank you for pointing out the exact OS and toolchain used. -- ___ Python tracker <http://bugs.python.org/issue26788> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue26787] test_distutils fails when configured --with-lto
Alecsandru Patrascu added the comment: I will investigate this and submit a fix. At a first glance, it seems the test is failing because it does not have the knowledge of the LTO flags. -- ___ Python tracker <http://bugs.python.org/issue26787> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue26359] CPython build options for out-of-the box performance
Alecsandru Patrascu added the comment: That sounds like a good idea and plan, to have everything enabled with just one dedicated configure flag. I can work on this and post the patches as soon as they are done. Thank you -- ___ Python tracker <http://bugs.python.org/issue26359> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25702] Link Time Optimizations support for GCC and CLANG
Alecsandru Patrascu added the comment: Maybe an workflow like the one proposed in issue #26359 can be helpful in these development phases. -- ___ Python tracker <http://bugs.python.org/issue25702> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25702] Link Time Optimizations support for GCC and CLANG
Alecsandru Patrascu added the comment: @Stefan and @Marc, you say that people don't want to wait for PGO to build when running ./configure && make, but why? Even though many developers use it, this mode is not intended for development, it is production level and should be run once (or at leas a limited number or times), when the developers are sure that everything is fine in the debug mode. As Victor previously said, we should have all the *good* stuff (PGO, LTO, etc) enabled by default, regardless the time needed to do it. @Victor, indeed, LTO is not yet good enough to use it stand-alone in CPython. That is the reason why it is enabled only with PGO, because applied over it, we obtain further speedups than PGO alone. Also Ubuntu uses PGO and LTO in their releases. But in the end maybe `./configure --with-lto && make profile-opt` will have to do for everybody. -- ___ Python tracker <http://bugs.python.org/issue25702> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue26359] CPython build options for out-of-the box performance
Alecsandru Patrascu added the comment: Maybe that end users are lazy and want to type just `make` and let things happen magically behind the scenes :-) -- ___ Python tracker <http://bugs.python.org/issue26359> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue26359] CPython build options for out-of-the box performance
Alecsandru Patrascu added the comment: Hello Victor, Indeed, the best outcome is to have PGO, LTO, etc enabled by default when running ./configure && make, for production level. I also feel that they should be on by default, if only developers would use the debug version. Do you think that modifying the patches attached to this issue and enabling all optimizations by default could be considered acceptable by the Python community? Thank you, Alecsandru -- ___ Python tracker <http://bugs.python.org/issue26359> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25702] Link Time Optimizations support for GCC and CLANG
Alecsandru Patrascu added the comment: You are doing measurements on a virtual machine... For sure you are not the only user that has active workloads on the physical machine while you do benchmarks :) On the other hand, the path you are going with just LTO is nice for experiments, but for real-world usages is not feasible. Using it in conjunction with PGO is the way to have the best Python interpreter, and I strongly recommend for you to use the v04 versions of the patches. -- ___ Python tracker <http://bugs.python.org/issue25702> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25702] Link Time Optimizations support for GCC and CLANG
Alecsandru Patrascu added the comment: >From our experience, pybench only is not a representative benchmark. Instead, >if you like to measure performance close to real workloads, you can run the >Grand Unified Python Benchmark suite, that is more complete. Also, you need to take into consideration the hardware and software environment. For this, you can read the initial comment at this issue, section "Hardware and OS Configuration", to see the approach we have here at Intel. -- ___ Python tracker <http://bugs.python.org/issue25702> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25702] Link Time Optimizations support for GCC and CLANG
Alecsandru Patrascu added the comment: I understand now your question. LTO is not enabled when running just `make`, only in `make profile-opt` -- ___ Python tracker <http://bugs.python.org/issue25702> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25702] Link Time Optimizations support for GCC and CLANG
Alecsandru Patrascu added the comment: Yes, you can use LTO without PGO, but the proposed ways it's more efficient and makes more sense for CPython builds. -- ___ Python tracker <http://bugs.python.org/issue25702> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue26285] Garbage collection of unused input sections from CPython binaries
Changes by Alecsandru Patrascu : -- nosy: +gregory.p.smith, lemburg, scoder, steve.dower, zach.ware ___ Python tracker <http://bugs.python.org/issue26285> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue26359] CPython build options for out-of-the box performance
Changes by Alecsandru Patrascu : -- nosy: +brett.cannon, gregory.p.smith, lemburg, pitrou, r.david.murray, scoder, skrah, steve.dower, zach.ware ___ Python tracker <http://bugs.python.org/issue26359> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue26307] no PGO for built-in modules with `make profile-opt`
Changes by Alecsandru Patrascu : Added file: http://bugs.python.org/file42000/pgofix-cpython3.patch ___ Python tracker <http://bugs.python.org/issue26307> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue26307] no PGO for built-in modules with `make profile-opt`
Alecsandru Patrascu added the comment: I've added the patch here also. -- keywords: +patch Added file: http://bugs.python.org/file41999/pgofix-cpython2.patch ___ Python tracker <http://bugs.python.org/issue26307> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue26382] List object memory allocator
Changes by Alecsandru Patrascu : -- nosy: +alecsandru.patrascu ___ Python tracker <http://bugs.python.org/issue26382> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue26249] Change PyMem_Malloc to use PyObject_Malloc allocator?
Changes by Alecsandru Patrascu : -- nosy: +alecsandru.patrascu ___ Python tracker <http://bugs.python.org/issue26249> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue26359] CPython build options for out-of-the box performance
Changes by Alecsandru Patrascu : Added file: http://bugs.python.org/file41919/cpython3_pgo_default-v02.patch ___ Python tracker <http://bugs.python.org/issue26359> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue26359] CPython build options for out-of-the box performance
New submission from Alecsandru Patrascu: Hello, This is Alecsandru from the Dynamic Scripting Languages Optimization Team at Intel Corporation. I would like to submit a patch as a response to the discussion thread opened in Python-Dev (https://mail.python.org/pipermail/python-dev/2016-February/143215.html), regarding the way CPython is built, mainly the options that are available to the programmers. Analyzing the CPython ecosystem we can see that there are a lot of users that just download the sources and hit the commands "./configure", "make" and "make install" once and then continue using it with their Python scripts. One of the problems with this workflow it that the users do not benefit from the entire optimization features that are existing in the build system, such as PGO and LTO. Therefore, I propose a workflow, like the following. Assume some work has to be done into the CPython interpreter, a developer can do the following steps: A. Implementation and debugging phase. 1. The command "./configure PYDIST=debug" is ran once. It will enable the Py_DEBUG, -O0 and -g flags 2. The command "make" is ran once or multiple times B. Testing the implementation from step A, in a pre-release environment 1. The command "./configure PYDIST=devel" is ran once. It will disable the Py_DEBUG flags and will enable the -O3 and -g flags, and it is just like the current implementation in CPython 2. The command "make" is ran once or multiple times C. For any other CPython usage, for example distributing the interpreter, installing it inside an operating system, or just the majority of users who are not CPython developers and only want to compile it once and use it as-is: 1. The command "./configure" is ran once. Alternatively, the command "./configure PYDIST=release" can be used. It will disable all debugging functionality, enable the -O3 flag and will enable PGO and LTO. 2. The command "make" is ran once Thank you, Alecsandru -- components: Build files: cpython2_pgo_default-v01.patch keywords: patch messages: 260271 nosy: alecsandru.patrascu priority: normal severity: normal status: open title: CPython build options for out-of-the box performance type: performance versions: Python 2.7, Python 3.5, Python 3.6 Added file: http://bugs.python.org/file41918/cpython2_pgo_default-v01.patch ___ Python tracker <http://bugs.python.org/issue26359> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue26307] no PGO for built-in modules with `make profile-opt`
Alecsandru Patrascu added the comment: I've added a fix for the PGO builds after the issue pointed out by you. Can you please test it also? -- ___ Python tracker <http://bugs.python.org/issue26307> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24915] Profile Guided Optimization improvements (better training, llvm support, etc)
Changes by Alecsandru Patrascu : -- resolution: fixed -> status: closed -> open ___ Python tracker <http://bugs.python.org/issue24915> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24915] Profile Guided Optimization improvements (better training, llvm support, etc)
Changes by Alecsandru Patrascu : Added file: http://bugs.python.org/file41917/pgofix-cpython3.patch ___ Python tracker <http://bugs.python.org/issue24915> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24915] Profile Guided Optimization improvements (better training, llvm support, etc)
Alecsandru Patrascu added the comment: I've added a fix for the PGO builds after a issue pointed out in #26307. Thank you Christos for your observation! -- Added file: http://bugs.python.org/file41916/pgofix-cpython2.patch ___ Python tracker <http://bugs.python.org/issue24915> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue26307] no PGO for built-in modules with `make profile-opt`
Alecsandru Patrascu added the comment: Thank you for noticing and signaling this issue. Since I proposed the PGO patches, I will fix it in another patch, as I don't want it to break any builds. -- ___ Python tracker <http://bugs.python.org/issue26307> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue26307] no PGO for built-in modules with `make profile-opt`
Changes by Alecsandru Patrascu : -- nosy: +alecsandru.patrascu ___ Python tracker <http://bugs.python.org/issue26307> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24915] Profile Guided Optimization improvements (better training, llvm support, etc)
Alecsandru Patrascu added the comment: For my responses, I modified locally the Makefile so that it will not remove the build directory and any of the gcda files. I will make some more tests tomorrow, but i think that this problem will solve simpler if the removal of the build directory is deleted and all the profile info is kept. -- ___ Python tracker <http://bugs.python.org/issue24915> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24915] Profile Guided Optimization improvements (better training, llvm support, etc)
Alecsandru Patrascu added the comment: That's interesting. Even on CPython3, I still don't see any gcda's inside the build directory, nor the tree structure you are seeing there. Can you please give me a couple of details regarding your environment (os, distribution, gcc version, 32/64 bit, cross compilation, etc)? If this is a general issue, I can add another patch to fix it. -- ___ Python tracker <http://bugs.python.org/issue24915> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24915] Profile Guided Optimization improvements (better training, llvm support, etc)
Alecsandru Patrascu added the comment: Hello and thank you for your feedback. For CPython this does not apply because due to the structure of the build system, inside the "build" directory there are no PGO profiles saved. You can run find . -name '*.gc??' to see. -- ___ Python tracker <http://bugs.python.org/issue24915> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue26285] Garbage collection of unused input sections from CPython binaries
Alecsandru Patrascu added the comment: I attached the list for CPython3 (gc-removed-zones-cpython3.txt), but now split in two sections (core and parser), for more clarity as from what and where it is removed. As you can see, the reason why the module works is because the API that can be used by modules remains untouched. What I was trying to say and prove before is that these GCC/LD flags are safe to use in CPython (and for any other software project) and will not break any compatibility with existing or future modules. -- Added file: http://bugs.python.org/file41851/gc-removed-zones-cpython3.txt ___ Python tracker <http://bugs.python.org/issue26285> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue26285] Garbage collection of unused input sections from CPython binaries
Alecsandru Patrascu added the comment: I've done again the experiments on larger workloads, such as our OpenStack Swift cluster, and it works without any issues. Also, I've attached an archive with a simple external module in CPython3 that uses PyMem_RawMalloc. The output is ok, and it's copied bellow. u@palecsandru:~/w/experimente/c_ext3$ /home/u/w/cpython3_deadcode/python setup.py build_ext --inplace running build_ext building 'mytest' extension creating build creating build/temp.linux-x86_64-3.6 gcc -pthread -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fdata-sections -ffunction-sections -Wl,--gc-sections -fPIC -I/home/u/w/cpython3_deadcode/Include -I/home/u/w/cpython3_deadcode -c mytest.c -o build/temp.linux-x86_64-3.6/mytest.o gcc -pthread -shared build/temp.linux-x86_64-3.6/mytest.o -o /home/u/w/experimente/c_ext3/mytest.cpython-36m-x86_64-linux-gnu.so u@palecsandru:~/w/experimente/c_ext3$ ll total 40 drwxrwxr-x 3 u u 4096 Feb 5 14:29 ./ drwxr-xr-x 12 u u 4096 Feb 5 14:00 ../ drwxrwxr-x 3 u u 4096 Feb 5 14:29 build/ -rw-rw-r-- 1 u u 619 Feb 5 14:16 mytest.c -rwxrwxr-x 1 u u 17856 Feb 5 14:29 mytest.cpython-36m-x86_64-linux-gnu.so* -rw-rw-r-- 1 u u 132 Feb 5 14:15 setup.py u@palecsandru:~/w/experimente/c_ext3$ /home/u/w/cpython3_deadcode/python Python 3.6.0a0 (default:87dfadd61e0d+, Feb 5 2016, 14:22:57) [GCC 5.2.1 20151010] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import mytest >>> mytest.mytest() 'test' >>> -- Added file: http://bugs.python.org/file41819/c_ext3.zip ___ Python tracker <http://bugs.python.org/issue26285> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue26285] Garbage collection of unused input sections from CPython binaries
Alecsandru Patrascu added the comment: Sure, I attached them as files because they have a lot of lines for posting here (~90 in total). The linker offers the possibility to show what piece of data/functions was removed, but I intentionally omitted it in order not to clutter the build trace. If you think it will be useful for the user to see it, I can add them to the patch also. -- Added file: http://bugs.python.org/file41808/gc-removed-cpython2.txt ___ Python tracker <http://bugs.python.org/issue26285> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue26285] Garbage collection of unused input sections from CPython binaries
Changes by Alecsandru Patrascu : Added file: http://bugs.python.org/file41809/gc-removed-cpython3.txt ___ Python tracker <http://bugs.python.org/issue26285> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue26285] Garbage collection of unused input sections from CPython binaries
Alecsandru Patrascu added the comment: I realize now that I should have explained a bit more the background of this patch. I'll do this now, for everyone to be clear of what is the effect of those flags. This issue was revealed after running the coverage target over various workloads, for both CPython2 and CPython3. After running, it can be observed that there are functions in the interpreter that are not called at all over the lifespan of the interpreter. Even more, these functions occupy space in the resulting binary file, and the CPU is forced to jump to longer offsets than it is required. Furthermore, for production level binaries, it is a good idea to remove these stubs, as they bring no benefit. Now, in order to do this, in the first step, every function or data item must exist in its own section (and the flags -ffunction-sections and -fdata-sections come to help in GCC). In the second step, the linker comes into play and because it has the entire picture of every piece of data or function, it is able to see if there are functions that are never called for the current build (and the flag --gc-sections come to help). This functionality is not unique or new and are used by default in other interpreters, such as V8/Node.JS in their Release target, to achieve exactly the same goal. Another example for behind the scene usage of this functionality is the Microsoft's compiler, which does it automatically in their interprocedural optimization phase. To compress all of the above, the main reason for this speedup is the reduction of the code path length and having the useful function close together, so that the CPU will be able to prefetch them in advance and use them instead of trowing them away because they are not used. -- ___ Python tracker <http://bugs.python.org/issue26285> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue26285] Garbage collection of unused input sections from CPython binaries
New submission from Alecsandru Patrascu: Hi all, This is Alecsandru from the Dynamic Scripting Languages Optimization Team at Intel Corporation. I would like to submit a patch that enables garbage collection of unused input sections from the CPython2 and CPython3 binaries, by using the "--gc-sections" linker flag, which decides which input sections are used by examining symbols and relocations. In order for this to work, GCC must place each function or data item into its own section in the output file, thus dedicated flags are used. With this technique, an average of 1% is gained in both interpreters, with a few small regressions. Steps: == 1. Get the CPython source codes hg clone https://hg.python.org/cpython cpython cd cpython hg update 2.7 (for CPython2) 2. Build the binary a) Default: ./configure make b) Unused input sections patch Copy the attached patch files hg import --no-commit cpython2-deadcode-v01.patch.patch (for CPython3) hg import --no-commit cpython2-deadcode-v01.patch (for CPython2) ./configure make Hardware and OS Configuration = Hardware: Intel XEON (Haswell-EP) 18 Cores BIOS settings: Intel Turbo Boost Technology: false Hyper-Threading: false OS: Ubuntu 14.04.3 LTS Server OS configuration: Address Space Layout Randomization (ASLR) disabled to reduce run to run variation by echo 0 > /proc/sys/kernel/randomize_va_space CPU frequency set fixed at 2.6GHz GCC version:GCC version 4.9.2 Benchmark: Grand Unified Python Benchmark from https://hg.python.org/benchmarks/ Measurements and Results CPython2 and CPython3 sample results, measured using GUPB on a Haswell platform, can be viewed in Table 1 and 2. On the first column (Benchmark) you can see the benchmark name and on the second (%S) the speedup compared with the default version; a higher value is better. Table 1. CPython3 results: Benchmark %S -- telco 11 etree_parse 7 call_simple 6 etree_iterparse 5 regex_v84 meteor_contest 3 etree_process 3 call_method_unknown 3 json_dump_v23 formatted_logging 2 hexiom2 2 chaos 2 richards2 django_v3 2 nbody 2 etree_generate 2 pickle_list 1 go 1 nqueens 1 call_method 1 mako_v2 1 raytrace1 chameleon_v21 silent_logging 0 fastunpickle0 2to30 float 0 regex_effbot0 pidigits0 json_load 0 simple_logging 0 normal_startup 0 startup_nosite 0 fastpickle 0 tornado_http0 regex_compile 0 fannkuch0 spectral_norm 0 pickle_dict 0 unpickle_list 0 call_method_slots 0 pathlib -2 unpack_sequence -2 Table 2. CPython2 results: Benchmark %S -- simple_logging 4 formatted_logging 3 slowpickle 2 silent_logging 2 pickle_dict 1 chameleon_v21 hg_startup 1 pickle_list 1 call_method_unknown 1 pidigits1 regex_effbot1 regex_v81 html5lib0 normal_startup 0 regex_compile 0 etree_parse 0 spambayes 0 html5lib_warmup 0 unpack_sequence 0 richards0 rietveld0 startup_nosite 0 raytrace0 etree_iterparse 0 json_dump_v20 fastpickle 0 slowspitfire0 slowunpickle0 call_simple 0 float 0 2to30 bzr_startup 0 json_load 0 hexiom2 0 chaos 0 unpickle_list 0 call_method_slots 0 tornado_http0 fastunpickle0 etree_process 0 spectral_norm 0 meteor_contest 0 pybench 0 go 0 etree_generate 0 mako_v2 0 django_v3 0 fannkuch0 nbody 0 nqueens 0 telco -1 call_method -2 pathlib -3 Thank you, Alecsandru -- components: Build files: cpython2-deadcode-v01.patch keywords: patch messages: 259572 nosy: alecsandru.patrascu priority: normal severity: normal status: open title: Garbage collection of unused input sections from CPython binaries type: performance versions: Python 2.7, Python 3.6 Added file: http://bugs.python.org/file41805/cpython2-deadcode-v01.patch ___ Python tracker <http://bugs.python.org/issue26285> ___ _
[issue26285] Garbage collection of unused input sections from CPython binaries
Changes by Alecsandru Patrascu : Added file: http://bugs.python.org/file41806/cpython3-deadcode-v01.patch ___ Python tracker <http://bugs.python.org/issue26285> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue26110] Speedup method calls 1.2x
Alecsandru Patrascu added the comment: Yury, thank you for the heads up! Here at Intel, in the Dynamic Scripting Languages Optimization Team, we can help the community with reviewing and measuring this patch in our quiet and stable environment, the same one that we use to provide public CPython daily measurements. We will wait for your update. -- ___ Python tracker <http://bugs.python.org/issue26110> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue26110] Speedup method calls 1.2x
Changes by Alecsandru Patrascu : -- nosy: +alecsandru.patrascu ___ Python tracker <http://bugs.python.org/issue26110> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25702] Link Time Optimizations support for GCC and CLANG
Alecsandru Patrascu added the comment: As Steve mentioned, the Microsoft compiler uses LTO (they call it Link-Time Code Generation) and the flags are used when compiling CPython on Windows systems. Thus our proposal to enable it on GCC and CLANG also. -- ___ Python tracker <http://bugs.python.org/issue25702> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25702] Link Time Optimizations support for GCC and CLANG
Changes by Alecsandru Patrascu : Added file: http://bugs.python.org/file41666/lto-cpython3-v04.patch ___ Python tracker <http://bugs.python.org/issue25702> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25702] Link Time Optimizations support for GCC and CLANG
Alecsandru Patrascu added the comment: Thank you for your feedback, I've updated the patches and now LTO flags are used only when building with PGO (v04). CFLAGS/LDFLAGS remain untouched, as Antoine and Victor suggested is better. -- Added file: http://bugs.python.org/file41665/lto-cpython2-v04.patch ___ Python tracker <http://bugs.python.org/issue25702> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25702] Link Time Optimizations support for GCC and CLANG
Changes by Alecsandru Patrascu : Added file: http://bugs.python.org/file41494/lto-cpython3-v03.patch ___ Python tracker <http://bugs.python.org/issue25702> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25702] Link Time Optimizations support for GCC and CLANG
Alecsandru Patrascu added the comment: Hello, I've added an updated set of patches (v03) for the current CPython2 and CPython3 codebase. Also made some small changes to reduce the number of places where the flags are set. -- Added file: http://bugs.python.org/file41493/lto-cpython2-v03.patch ___ Python tracker <http://bugs.python.org/issue25702> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25702] Link Time Optimizations support for GCC and CLANG
Alecsandru Patrascu added the comment: I'm adding Brett, Gregory, Stefan and Victor as nosy because this issue might be interesting for them also. -- nosy: +brett.cannon, gregory.p.smith, haypo, scoder, skrah ___ Python tracker <http://bugs.python.org/issue25702> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25925] Coverage support for CPython 2
Alecsandru Patrascu added the comment: Thank you for the reviews. I've added a new patch (v02) to correct the issues. -- Added file: http://bugs.python.org/file41386/coverage-2.7-v02.patch ___ Python tracker <http://bugs.python.org/issue25925> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25925] Coverage support for CPython 2
New submission from Alecsandru Patrascu: Hi All, This is Alecsandru from Server Scripting Languages Optimization team at Intel Corporation. I would like to submit a patch that adds extended support for coverage, in the same way that it is used in CPython3. Used in conjunction with the regrtests, it provides a broader view of the hot and cold zones in the interpreter. Thank you, Alecsandru -- components: Build files: coverage-2.7-v01.patch keywords: patch messages: 256839 nosy: alecsandru.patrascu priority: normal severity: normal status: open title: Coverage support for CPython 2 type: enhancement versions: Python 2.7 Added file: http://bugs.python.org/file41385/coverage-2.7-v01.patch ___ Python tracker <http://bugs.python.org/issue25925> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25827] Support ICC in configure
Alecsandru Patrascu added the comment: Thank you David for including me in this issue. On ICC, when executing the instrumented applications, it will generate dynamic information file that has a unique name and .dyn suffix. From my previous experience with GCC and CLANG PGO patch (issue24915), also the output of the autoconf will have to be included in the final diff. Regarding the place for adding the '-fp-model strict' flags, I had the same dillema when working on the LTO patch (issue25702), because I needed the flags to be propagated to _all_ compilation and linking phases, and finally decided to add them to the CONFIGURE_CFLAGS, CONFIGURE_CFLAGS_NODIST, CONFIGURE_CPPFLAGS, CONFIGURE_LDFLAGS variables in the Makefile.pre.in file. You can try there also and keep the BASECFLAGS clean . -- ___ Python tracker <http://bugs.python.org/issue25827> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue17781] optimize compilation options
Alecsandru Patrascu added the comment: Hi, I added a dedicated issue just for LTO only when using GCC and CLANG (http://bugs.python.org/issue25702), that works well with PGO also. -- nosy: +alecsandru.patrascu ___ Python tracker <http://bugs.python.org/issue17781> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25702] Link Time Optimizations support for GCC and CLANG
Changes by Alecsandru Patrascu : Added file: http://bugs.python.org/file41138/lto-cpython3-v02.patch ___ Python tracker <http://bugs.python.org/issue25702> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25702] Link Time Optimizations support for GCC and CLANG
Changes by Alecsandru Patrascu : Added file: http://bugs.python.org/file41137/lto-cpython2-v02.patch ___ Python tracker <http://bugs.python.org/issue25702> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25702] Link Time Optimizations support for GCC and CLANG
Alecsandru Patrascu added the comment: Meanwhile I've added the patches (v02) for LTO enabled only if the "./configure --with-lto" command is issued. -- ___ Python tracker <http://bugs.python.org/issue25702> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25702] Link Time Optimizations support for GCC and CLANG
Alecsandru Patrascu added the comment: LTO exists in GCC since version 4.5, but it is true that only recent versions (>=4.8) perform it in good conditions. It is not enabled by default in this patch, it is only available when building with PGO support. Running just "make" will not activate the LTO flags. Do you see it as an configure option (using, for example, an explicit --with-lto flag) rather than using it automatically? -- ___ Python tracker <http://bugs.python.org/issue25702> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25702] Link Time Optimizations support for GCC and CLANG
Changes by Alecsandru Patrascu : -- keywords: +patch Added file: http://bugs.python.org/file41130/lto-cpython2-v01.patch ___ Python tracker <http://bugs.python.org/issue25702> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25702] Link Time Optimizations support for GCC and CLANG
Changes by Alecsandru Patrascu : Added file: http://bugs.python.org/file41131/lto-cpython3-v01.patch ___ Python tracker <http://bugs.python.org/issue25702> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25702] Link Time Optimizations support for GCC and CLANG
New submission from Alecsandru Patrascu: Title: Link Time Optimizations support for GCC and CLANG Hi All, This is Alecsandru from Server Scripting Languages Optimization team at Intel Corporation. I would like to submit a patch that adds support for Link Time Optimization (LTO) when using GCC and CLANG to compile CPython2 and CPython3. LTO is a compiler assisted optimization technique that is performed by the compiler at link time. Combined with Profile Guided Optimization (PGO), enabled when running "make profile-opt", and running the Grand Unified Python Benchmark (GUPB), a speedup up to 11%, with a few regressions, was observed comparing with PGO only. Compared with a default build, a performance gain as high as 26% was observed from PGO+LTO. In addition, we are also seeing 2% boost in throughput rate from our OpenStack Swift setup comparing with PGO only. Our GUPB performance evaluation was conducted on Intel SkyLake/Broadwell systems running CentOS/Ubuntu, with CLANG/LLVM and GCC 4.*/5.*. Our OpenStack Swift performance was done on various systems consisting of XEON and Avoton processors. Steps: == 1. Get the CPython source codes hg clone https://hg.python.org/cpython cpython cd cpython hg update 2.7 (for CPython2) 2. Build the binary a) Default: ./configure make b) PGO: ./configure make profile-opt c) PGO+LTO: Copy the attached patch files hg import --no-commit lto-cpython3-v01.patch (for CPython3) hg import --no-commit lto-cpython2-v01.patch (for CPython2) ./configure make profile-opt Hardware and OS Configuration = Hardware: Intel XEON (Broadwell-DE) 8 Cores BIOS settings: Intel Turbo Boost Technology: false Hyper-Threading: false OS: Ubuntu 14.04.3 LTS Server OS configuration: Address Space Layout Randomization (ASLR) disabled to reduce run to run variation by echo 0 > /proc/sys/kernel/randomize_va_space CPU frequency set fixed at 2.6GHz GCC version:GCC version 4.9.2 Benchmark: Grand Unified Python Benchmark from https://hg.python.org/benchmarks/ Measurements and Results A. Repository: GUPB Benchmark: hg id : 2979f5ce6a0c tip hg --debug id -i : 2979f5ce6a0cee994d5485401945d8457bb0afac CPython3: hg id : 21a28f6de358 hg id -r 'ancestors(.) and tag()': 374f501f4567 (3.5) v3.5.0 hg --debug id -i : 21a28f6de3582833652c958b8fd6ae8448b61c7c CPython2: hg id : a37ea1d56e98 (2.7) hg id -r 'ancestors(.) and tag()': 15c95b7d81dc (2.7) v2.7.10 hg --debug id -i : a37ea1d56e98eb158750d3e495a5cf524e8c3980 B. Results: CPython2 and CPython3 sample results, measured on a Broadwell platform, can be viewed in Table 1 and 2. On the first column (Benchmark) you can see the benchmark name, on the second (%D) the speedup compared with the default version and on the third column (%PGO) the speedup compared with just PGO; a higher value is better. Table 1. CPython2 results: Benchmark %D %PGO raytrace18 3 chaos 16 5 django_v2 16 6 mako16 6 pathlib 15 3 simple_logging 15 1 slowpickle 15 5 django 14 4 go 14 4 richards13 -1 float 12 4 slowunpickle12 4 etree_process 11 3 fastunpickle11 6 formatted_logging 11 3 nqueens 11 1 regex_compile 11 3 etree_iterparse 10 4 mako_v2 10 3 telco 10 5 pybench 9 1 hexiom2 9 1 html5lib_warmup 9 3 meteor_contest 9 4 pickle_list 9 5 2to38 2 bzr_startup 8 2 chameleon 8 0 etree_generate 8 2 regex_v88 3 silent_logging 8 1 fannkuch7 1 html5lib7 3 json_load 7 -5 tornado_http7 3 call_method_slots 6 3 json_dump_v26 -4 spambayes 6 2 unpickle_list 6 0 etree_parse 5 3 fastpickle 5 4 rietveld5 1 call_method 4 -1 normal_startup 4 2 startup_nosite 4 2 slowspitfire3 0 ssbench 4 2 call_method_unknown 1 -6 json_dump 1 -4 nbody 1 1 pidigits1 -10 pickle_dict 0 -1 regex_effbot0 -2 spectral_norm 0
[issue25188] regrtest.py improvement for Profile Guided Optimization builds
Alecsandru Patrascu added the comment: Thank you Brett for your effort to merge all the changes made to regrtest despite the changes appeared on the way! -- ___ Python tracker <http://bugs.python.org/issue25188> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25188] regrtest.py improvement for Profile Guided Optimization builds
Alecsandru Patrascu added the comment: Thank you for the tweaks. I tested the patch and it is all working fine. Without failure count the PGO run is even more transparent to the user. I tested your final version of the patch on several Linux server and regular desktop distributions and software configurations and I saw that test_eintr does not block and the entire PGO build finishes in good conditions every time. -- ___ Python tracker <http://bugs.python.org/issue25188> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25188] regrtest.py improvement for Profile Guided Optimization builds
Alecsandru Patrascu added the comment: Thank you for the link. I modified regrtest.py for Python3 according to the specifications. -- ___ Python tracker <http://bugs.python.org/issue25188> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25188] regrtest.py improvement for Profile Guided Optimization builds
Changes by Alecsandru Patrascu : Added file: http://bugs.python.org/file40547/regrtest_36_v02.patch ___ Python tracker <http://bugs.python.org/issue25188> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25188] regrtest.py improvement for Profile Guided Optimization builds
Alecsandru Patrascu added the comment: The patch does not display dots, as they are not that intuitive and users cannot make an estimate of the time left or the position in the training phase. Instead, the regrtest.py, will display the progress as [X/Y], like the default version, but without any warnings, failed tests, skipped tests. Also the start and end report are suppressed. I don't understand what do you mean by "keyword-only parameter". I added a new flag (-P/--pgo) and propagated it along all the functions that need it. Another approach would have been to use a global variable, lets say "pgo" and use it directly. This is closer to what you had in mind? -- ___ Python tracker <http://bugs.python.org/issue25188> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25188] regrtest.py improvement for Profile Guided Optimization builds
Alecsandru Patrascu added the comment: I've added the patches for Python default and 2.7. To apply them please use the command "hg import --no-commit". PGO flag is implemented for both variants (single and multi process). Default in profile-opt will be single process. -- ___ Python tracker <http://bugs.python.org/issue25188> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25188] regrtest.py improvement for Profile Guided Optimization builds
Changes by Alecsandru Patrascu : -- keywords: +patch Added file: http://bugs.python.org/file40532/regrtest_27.patch ___ Python tracker <http://bugs.python.org/issue25188> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25188] regrtest.py improvement for Profile Guided Optimization builds
Changes by Alecsandru Patrascu : Added file: http://bugs.python.org/file40533/regrtest_36.patch ___ Python tracker <http://bugs.python.org/issue25188> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25188] regrtest.py improvement for Profile Guided Optimization builds
New submission from Alecsandru Patrascu: This issue adds improved support for Profile Guided Optimization builds in the regression test suite. Mainly, we keep the visual progress status in this process and suppress errors or any other unnecessary output that can alarm users. -- components: Library (Lib) messages: 251140 nosy: alecsandru.patrascu priority: normal severity: normal status: open title: regrtest.py improvement for Profile Guided Optimization builds type: performance versions: Python 2.7, Python 3.5, Python 3.6 ___ Python tracker <http://bugs.python.org/issue25188> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24915] Profile Guided Optimization improvements (better training, llvm support, etc)
Alecsandru Patrascu added the comment: I can work on modifying the existing regrtest and adding a distinct flag, --pgo for example, as Antoine suggested. Indeed, it will not be trivial as regrtest has a dual approach (single process and multi process), but I will give it a try and post a patch as soon as possible. I also suggest that I open a new issue for this case as it is somehow a distinct implementation than pure PGO and definitively will be some iterations on regrtest.py for both versions of Python until we reach a common ground. It is ok for everyone? -- ___ Python tracker <http://bugs.python.org/issue24915> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24915] Profile Guided Optimization improvements (better training, llvm support, etc)
Alecsandru Patrascu added the comment: Thank you for upstreaming this in both branches of Python! Do you think that a different version of regrtest.py, that will be used only for PGO training, should be better in this case? I mean, by implementing a custom version, I think we can control better the output and errors shown on screen. -- ___ Python tracker <http://bugs.python.org/issue24915> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25106] Hash computation speedup for {buffer, string, unicode}object
Alecsandru Patrascu added the comment: Please hold this patch for a while, while I analyze the received feedback. -- ___ Python tracker <http://bugs.python.org/issue25106> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25106] Hash computation speedup for {buffer, string, unicode}object
Alecsandru Patrascu added the comment: Yes, it doesn't break the compatibility with existing applications. To make sure, we tested it in various scenarios and workloads. -- ___ Python tracker <http://bugs.python.org/issue25106> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25106] Hash computation speedup for {buffer, string, unicode}object
Alecsandru Patrascu added the comment: This patch is not applicable to Python 3, as it already has a better hash function and has the same unrolling technique applied. As Brett well observed, it is a backport of an optimization technique used in Python 3, applied to the existing hash function in 2.7. -- ___ Python tracker <http://bugs.python.org/issue25106> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25106] Hash computation speedup for {buffer, string, unicode}object
Alecsandru Patrascu added the comment: I've submitted a more compact version of the previous code (hash8-v02.patch) -- ___ Python tracker <http://bugs.python.org/issue25106> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25106] Hash computation speedup for {buffer, string, unicode}object
Changes by Alecsandru Patrascu : Added file: http://bugs.python.org/file40460/hash8-v02.patch ___ Python tracker <http://bugs.python.org/issue25106> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25106] Hash computation speedup for {buffer, string, unicode}object
Alecsandru Patrascu added the comment: Yes, sure it can be implemented in smaller space, as Serhiy and Chris well pointed out. I submitted the 01 version like that just to be clear that I don't want to re-implement a new hash computing value and just unroll the loop in the existing one. I will submit a new version based on this observations. -- ___ Python tracker <http://bugs.python.org/issue25106> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25106] Hash computation speedup for {buffer, string, unicode}object
New submission from Alecsandru Patrascu:
Hi All,
This is Alecsandru from Server Scripting Languages Optimization team at Intel
Corporation.
I would like to submit a patch that improves the performance of the hash
computation code on stringobject, bufferobject and unicodeobject. As can be
seen from the attached sample performance results from the Grand Unified Python
Benchmark, speedups up to 40% were observed. Furthermore, we see a 5-7%
performance on OpenStack/Swift, where most of the code is in Python 2.7.
Attached is the patch that modifies Object/stringobject.c,
Object/bufferobject.c and Object/unicodeobject.c files. We built and tested
this patch for Python 2.7 on our Linux machines (CentOS 7/Ubuntu Server 14.04,
Intel Xeon Haswell/Broadwell with 18/8 cores).
Steps to apply the patch:
1. hg clone https://hg.python.org/cpython cpython
2. cd cpython
3. hg update 2.7
4. Copy hash8.patch to the current directory
5. hg import --no-commit hash8.patch
6. ./configure
7. make
In the following, please find our sample performance results measured on a XEON
Haswell machine.
Hardware (HW): Intel XEON (Haswell) 18 Cores
BIOS settings: Intel Turbo Boost Technology: false
Hyper-Threading: false
Operating System: Ubuntu 14.04.3 LTS trusty
OS configuration: CPU freq set at fixed: 2.0GHz by
echo 200 >
/sys/devices/system/cpu/cpu*/cpufreq/scaling_min_freq
echo 200 >
/sys/devices/system/cpu/cpu*/cpufreq/scaling_max_freq
Address Space Layout Randomization (ASLR) disabled (to
reduce run to run variation) by
echo 0 > /proc/sys/kernel/randomize_va_space
GCC version:gcc version 4.8.4 (Ubuntu 4.8.4-2ubuntu1~14.04)
Benchmark: Grand Unified Python Benchmark (GUPB)
GUPB Source: https://hg.python.org/benchmarks/
Python2.7 results:
Python source: hg clone https://hg.python.org/cpython cpython
Python Source: hg update 2.7
Benchmarks Speedup(%)
unpack_sequence 40.32733766
chaos 24.84002537
chameleon 23.01392651
silent_logging 22.27202911
django 20.83842317
etree_process 20.46968294
nqueens 20.34234985
pathlib 19.63445919
pidigits19.34722148
etree_generate 19.25836634
pybench 19.06895825
django_v2 18.06073108
etree_iterparse 17.3797149
fannkuch17.08120879
pickle_list 16.60363602
raytrace16.0316265
slowpickle 15.86611184
pickle_dict 15.30447114
call_simple 14.42909032
richards14.2949594
simple_logging 13.6522626
etree_parse 13.38113097
json_dump_v212.2655
float 11.88164311
mako11.20606516
spectral_norm 11.04356684
hg_startup 10.57686164
mako_v2 10.37912648
slowunpickle10.24030714
go 10.03567319
meteor_contest 9.956231435
normal_startup 9.607401586
formatted_logging 9.601244811
html5lib9.082603748
2to38.741557816
html5lib_warmup 8.268150981
nbody 7.507012306
regex_compile 7.153922724
bzr_startup 7.140244739
telco 6.869411927
slowspitfire5.746323922
tornado_http5.24360121
rietveld3.865704876
regex_v83.777622219
hexiom2 3.586305282
json_dump 3.477551682
spambayes 3.183991854
fastunpickle2.971645347
fastpickle 0.673086656
regex_effbot0.127946837
json_load 0.023727176
Thank you,
Alecsandru
--
components: Interpreter Core
files: hash8-v01.patch
keywords: patch
messages: 250639
nosy: alecsandru.patrascu
priority: normal
severity: normal
status: open
title: Hash computation speedup for {buffer,string,unicode}object
type: enhancement
versions: Python 2.7
Added file: http://bugs.python.org/file40456/hash8-v01.patch
___
Python tracker
<http://bugs.python.org/issue25106>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24915] Profile Guided Optimization improvements (better training, llvm support, etc)
Alecsandru Patrascu added the comment: If you are talking just about the 3.6 patch, it is called this way to emphasize the fact that it is intended for the development branch. It is perfectly compatible with 3.5, therefore it is not needed for packagers to maintain two distinct versions. I've tested with: hg update 3.5 ; hg import --no-commit python3.6-pgo-v07.patch ; ./configure ; make profile-opt I also renamed the profile task makefile name. -- ___ Python tracker <http://bugs.python.org/issue24915> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24915] Profile Guided Optimization improvements (better training, llvm support, etc)
Changes by Alecsandru Patrascu : Added file: http://bugs.python.org/file40392/python3.6-pgo-v07.patch ___ Python tracker <http://bugs.python.org/issue24915> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24915] Profile Guided Optimization improvements (better training, llvm support, etc)
Changes by Alecsandru Patrascu : Removed file: http://bugs.python.org/file40388/python3.6-pgo-v07.patch ___ Python tracker <http://bugs.python.org/issue24915> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24915] Profile Guided Optimization improvements (better training, llvm support, etc)
Changes by Alecsandru Patrascu : Removed file: http://bugs.python.org/file40387/python2.7-pgo-v07.patch ___ Python tracker <http://bugs.python.org/issue24915> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24915] Profile Guided Optimization improvements (better training, llvm support, etc)
Changes by Alecsandru Patrascu : Added file: http://bugs.python.org/file40391/python2.7-pgo-v07.patch ___ Python tracker <http://bugs.python.org/issue24915> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24915] Profile Guided Optimization improvements (better training, llvm support, etc)
Alecsandru Patrascu added the comment: I named this task PROFILE_TASK_X86 because it is rigorously tested and we have proven that regrtest performs better on this architecture. Until any other clear evidence and solid measurements that regrtest is performing better on other architectures exists, I'd keep it this way. Even though this does not threaten the stability of the interpreter in any way, the dual aspect you mentioned appears because CPython 2 and 3 have slightly different makefile rule format. To create a common patch working cross-versions will create a very tangled Makefile. If you all agree that having an unified patch for both versions is acceptable, I will work on that. -- ___ Python tracker <http://bugs.python.org/issue24915> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24915] Profile Guided Optimization improvements (better training, llvm support, etc)
Changes by Alecsandru Patrascu : Added file: http://bugs.python.org/file40390/README3.6-pgo-v02.patch ___ Python tracker <http://bugs.python.org/issue24915> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24915] Profile Guided Optimization improvements (better training, llvm support, etc)
Changes by Alecsandru Patrascu : Added file: http://bugs.python.org/file40388/python3.6-pgo-v07.patch ___ Python tracker <http://bugs.python.org/issue24915> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24915] Profile Guided Optimization improvements (better training, llvm support, etc)
Changes by Alecsandru Patrascu : Added file: http://bugs.python.org/file40389/README2.7-pgo-v02.patch ___ Python tracker <http://bugs.python.org/issue24915> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24915] Profile Guided Optimization improvements (better training, llvm support, etc)
Changes by Alecsandru Patrascu : Added file: http://bugs.python.org/file40387/python2.7-pgo-v07.patch ___ Python tracker <http://bugs.python.org/issue24915> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24915] Profile Guided Optimization improvements (better training, llvm support, etc)
Alecsandru Patrascu added the comment: I've updated the patches with redirect to /dev/null, as is it is more clearer to the user what is our intent, without having him to necessarily read the regrtest documentation. I've also added a warning message regarding the output and ported all these lines to 3.6 and to the README files also. -- ___ Python tracker <http://bugs.python.org/issue24915> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24915] Profile Guided Optimization improvements (better training, llvm support, etc)
Changes by Alecsandru Patrascu : Added file: http://bugs.python.org/file40299/README2.7-pgo-v01.patch ___ Python tracker <http://bugs.python.org/issue24915> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24915] Profile Guided Optimization improvements (better training, llvm support, etc)
Changes by Alecsandru Patrascu : Added file: http://bugs.python.org/file40300/README3.6-pgo-v01.patch ___ Python tracker <http://bugs.python.org/issue24915> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24915] Profile Guided Optimization improvements (better training, llvm support, etc)
Alecsandru Patrascu added the comment: That's a good point Skip. I added another set of patches, just for the README files, explaining the entire procedure, so now anyone reading it will see that PGO is available, what are the steps involved and a brief comment about the warning. -- ___ Python tracker <http://bugs.python.org/issue24915> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24915] Profile Guided Optimization improvements (better training, llvm support, etc)
Changes by Alecsandru Patrascu : Added file: http://bugs.python.org/file40288/python2.7-pgo-v06.patch ___ Python tracker <http://bugs.python.org/issue24915> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24915] Profile Guided Optimization improvements (better training, llvm support, etc)
Changes by Alecsandru Patrascu : Added file: http://bugs.python.org/file40289/python3.6-pgo-v06.patch ___ Python tracker <http://bugs.python.org/issue24915> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
