Re: [yocto] [opkg-utils][PATCH 00/22] Partially prepare for python3 and fix old Packages file reuse
Op 26 apr. 2012, om 20:52 heeft Martin Jansa het volgende geschreven: On Thu, Apr 26, 2012 at 06:21:45PM +0200, Koen Kooi wrote: Op 26 apr. 2012, om 12:43 heeft Richard Purdie het volgende geschreven: On Fri, 2012-03-30 at 00:21 +0200, Martin Jansa wrote: First 5 patches are taken from oe-core. Then there are some cleanups and fixes for issues I've found while looking for cause of very long package-index calls. Some patches uses compatible changes from 2to3 (tested with python2.7), but python3 support is not 100% complete, but were closer. And in the end I've found the cause.. printing long fields (e. g. description) with opkg.py was producing empty lines in Packages file (which are Packages entry separators) so only first entries till first empty line were processed from old Packages file and the rest was always extracted from control files in packaged .ipk. And this also fixes Packages.filelist generation. I don't use python very much, so please review. And the speedup? Almost 60 times :) [..] I reviewed these and they all look like good improvements. I've merged it into master, thanks! After some more time in oe-core master for testing we should seriously consider cherry-picking the SRCREV bump to the denzil branch. Agreed, saves about 4 hours per day (13 images each with own package-index run and extra run for feed if built after iamges) on my buildhost for last month or so :) I merged in denzil-next locally and grabbed 26465 from patchwork. Let's see how much time it saves on my machine :) ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
Re: [yocto] [opkg-utils][PATCH 00/22] Partially prepare for python3 and fix old Packages file reuse
Op 27 apr. 2012, om 08:39 heeft Koen Kooi het volgende geschreven: Op 26 apr. 2012, om 20:52 heeft Martin Jansa het volgende geschreven: On Thu, Apr 26, 2012 at 06:21:45PM +0200, Koen Kooi wrote: Op 26 apr. 2012, om 12:43 heeft Richard Purdie het volgende geschreven: On Fri, 2012-03-30 at 00:21 +0200, Martin Jansa wrote: First 5 patches are taken from oe-core. Then there are some cleanups and fixes for issues I've found while looking for cause of very long package-index calls. Some patches uses compatible changes from 2to3 (tested with python2.7), but python3 support is not 100% complete, but were closer. And in the end I've found the cause.. printing long fields (e. g. description) with opkg.py was producing empty lines in Packages file (which are Packages entry separators) so only first entries till first empty line were processed from old Packages file and the rest was always extracted from control files in packaged .ipk. And this also fixes Packages.filelist generation. I don't use python very much, so please review. And the speedup? Almost 60 times :) [..] I reviewed these and they all look like good improvements. I've merged it into master, thanks! After some more time in oe-core master for testing we should seriously consider cherry-picking the SRCREV bump to the denzil branch. Agreed, saves about 4 hours per day (13 images each with own package-index run and extra run for feed if built after iamges) on my buildhost for last month or so :) I merged in denzil-next locally and grabbed 26465 from patchwork. Let's see how much time it saves on my machine :) With a relatively small deploy area: 7 minutes saved per image build! ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
Re: [yocto] [opkg-utils][PATCH 00/22] Partially prepare for python3 and fix old Packages file reuse
On Thu, Apr 26, 2012 at 06:21:45PM +0200, Koen Kooi wrote: Op 26 apr. 2012, om 12:43 heeft Richard Purdie het volgende geschreven: On Fri, 2012-03-30 at 00:21 +0200, Martin Jansa wrote: First 5 patches are taken from oe-core. Then there are some cleanups and fixes for issues I've found while looking for cause of very long package-index calls. Some patches uses compatible changes from 2to3 (tested with python2.7), but python3 support is not 100% complete, but were closer. And in the end I've found the cause.. printing long fields (e. g. description) with opkg.py was producing empty lines in Packages file (which are Packages entry separators) so only first entries till first empty line were processed from old Packages file and the rest was always extracted from control files in packaged .ipk. And this also fixes Packages.filelist generation. I don't use python very much, so please review. And the speedup? Almost 60 times :) [..] I reviewed these and they all look like good improvements. I've merged it into master, thanks! After some more time in oe-core master for testing we should seriously consider cherry-picking the SRCREV bump to the denzil branch. Agreed, saves about 4 hours per day (13 images each with own package-index run and extra run for feed if built after iamges) on my buildhost for last month or so :) Cheers, -- Martin 'JaMa' Jansa jabber: martin.ja...@gmail.com signature.asc Description: Digital signature ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
Re: [yocto] [opkg-utils][PATCH 00/22] Partially prepare for python3 and fix old Packages file reuse
Martin, Just wanted to let you know that this was received, I am holding off doing anything about it right now until we get through the current release process, this will be build and tested with early in the 1.3 as appropriate. Thanks Sau! On 03/29/2012 03:21 PM, Martin Jansa wrote: First 5 patches are taken from oe-core. Then there are some cleanups and fixes for issues I've found while looking for cause of very long package-index calls. Some patches uses compatible changes from 2to3 (tested with python2.7), but python3 support is not 100% complete, but were closer. And in the end I've found the cause.. printing long fields (e. g. description) with opkg.py was producing empty lines in Packages file (which are Packages entry separators) so only first entries till first empty line were processed from old Packages file and the rest was always extracted from control files in packaged .ipk. And this also fixes Packages.filelist generation. I don't use python very much, so please review. And the speedup? Almost 60 times :) before: real20m10.484s user18m32.335s sys 0m11.885s after real0m24.492s user0m23.748s sys 0m0.565s The following changes since commit 002d29bc605d7c2d02e4cf20a43c5277c15f5597: [opkg-utils] fix install fail problem Thanks for khorben's patch :-) https://docs.openmoko.org/trac/attachment/ticket/2072/patch-opkg-utils_Makefile_install_path.diff (2008-11-03 03:59:59 +) are available in the git repository at: git://github.com/shr-project/opkg-utils jansa/pull https://github.com/shr-project/opkg-utils/tree/jansa/pull Christopher Larson (1): Use python via the PATH, rather than hardcoding /usr/bin/python Enrico Scholz (1): opkg-make-index: convert mtime to int before comparing it Khem Raj (1): opkg.py: Add knowledge about License field in ipk headers Martin Jansa (17): opkg.py: use hashlib instead of old md5 module opkg.py, arfile.py: report which file has wrong format and use the same test arfile: decode read lines as ascii string before getting fields from it 2to3: use subprocess instead of commands 2to3: print fixes 2to3: dictionary fixes 2to3: exception handling fixes opkg-make-index: show OSError/IOError opkg.py: use string funtcions directly on string variable opkg.py: catch TypeError when reading control file, to show which one is failing arfile: fix test opkg-make-index: don't use stdout for Packages output and use __str__ instead of __repr__ opkg.py: cast lines from controlfile as string opkg.py: computeFileMD5 only when we have fn, otherwise fails to read None file opkg.py: fix write_package when called from main test opkg.py: use textwrap for description writing opkg.py: improve test so it prints temporary control file with long description and then reads it back Richard Purdie (1): opkg-make-index: don't error out when some package disappears Scott Anderson (1): arfile.py: handle six digit UIDs arfile.py| 55 +- makePackage |2 +- opkg-compare-indexes | 22 opkg-list-fields |4 +- opkg-make-index | 74 +++-- opkg-show-deps | 28 +-- opkg-unbuild |4 +- opkg-update-index|5 +- opkg.py | 125 + 9 files changed, 186 insertions(+), 133 deletions(-) ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
[yocto] [opkg-utils][PATCH 00/22] Partially prepare for python3 and fix old Packages file reuse
First 5 patches are taken from oe-core. Then there are some cleanups and fixes for issues I've found while looking for cause of very long package-index calls. Some patches uses compatible changes from 2to3 (tested with python2.7), but python3 support is not 100% complete, but were closer. And in the end I've found the cause.. printing long fields (e. g. description) with opkg.py was producing empty lines in Packages file (which are Packages entry separators) so only first entries till first empty line were processed from old Packages file and the rest was always extracted from control files in packaged .ipk. And this also fixes Packages.filelist generation. I don't use python very much, so please review. And the speedup? Almost 60 times :) before: real20m10.484s user18m32.335s sys 0m11.885s after real0m24.492s user0m23.748s sys 0m0.565s The following changes since commit 002d29bc605d7c2d02e4cf20a43c5277c15f5597: [opkg-utils] fix install fail problem Thanks for khorben's patch :-) https://docs.openmoko.org/trac/attachment/ticket/2072/patch-opkg-utils_Makefile_install_path.diff (2008-11-03 03:59:59 +) are available in the git repository at: git://github.com/shr-project/opkg-utils jansa/pull https://github.com/shr-project/opkg-utils/tree/jansa/pull Christopher Larson (1): Use python via the PATH, rather than hardcoding /usr/bin/python Enrico Scholz (1): opkg-make-index: convert mtime to int before comparing it Khem Raj (1): opkg.py: Add knowledge about License field in ipk headers Martin Jansa (17): opkg.py: use hashlib instead of old md5 module opkg.py, arfile.py: report which file has wrong format and use the same test arfile: decode read lines as ascii string before getting fields from it 2to3: use subprocess instead of commands 2to3: print fixes 2to3: dictionary fixes 2to3: exception handling fixes opkg-make-index: show OSError/IOError opkg.py: use string funtcions directly on string variable opkg.py: catch TypeError when reading control file, to show which one is failing arfile: fix test opkg-make-index: don't use stdout for Packages output and use __str__ instead of __repr__ opkg.py: cast lines from controlfile as string opkg.py: computeFileMD5 only when we have fn, otherwise fails to read None file opkg.py: fix write_package when called from main test opkg.py: use textwrap for description writing opkg.py: improve test so it prints temporary control file with long description and then reads it back Richard Purdie (1): opkg-make-index: don't error out when some package disappears Scott Anderson (1): arfile.py: handle six digit UIDs arfile.py| 55 +- makePackage |2 +- opkg-compare-indexes | 22 opkg-list-fields |4 +- opkg-make-index | 74 +++-- opkg-show-deps | 28 +-- opkg-unbuild |4 +- opkg-update-index|5 +- opkg.py | 125 + 9 files changed, 186 insertions(+), 133 deletions(-) -- 1.7.8.5 ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
