Bug#888572: git-buildpackage: creating xz tarballs should be done multi-threaded
On 2024-04-03 19:16:25 [+0200], Carsten Schoenert wrote: > I did a few more working on some of the packages I maintain and I did not > encounter similar problems with wrong hash ids like a few days back. Given > the troubled times around the liblzma versions I'm not sure the initial > issue about the report topic years back is still solved as we are back to a > version before 5.6.0 again. 5.6.0+ did -T0 by default while the current 5.4.x series does -T1 by default. You could set the enviroment variable XZ_DEFAULTS=-T0 to get the same defaults since (based on my understanding) gbp relies on xz's defaults. I recommend to use multiple threads for compression/ decompression for the speed boost. > My typical "problematic" packages are thunderbird and especially > kicad-packages3d, I wasn't needed to package a newer version of any of these > source packages lately. Sebastian
Bug#888572: git-buildpackage: creating xz tarballs should be done multi-threaded
Hi Guido, Am 22.03.24 um 16:22 schrieb Carsten Schoenert: I need to check this again and in detail, because I had recently some issues as the expected hash ids were different while working with gbp import and later gbp buildpackage on some package from the kicad universe. But I think these problems are currently related to the t64 transition. The last work on TB at the start of the work did work flawless. @Guido Please poke me again latest 2 weeks in case I don't have give feedback until then. I did a few more working on some of the packages I maintain and I did not encounter similar problems with wrong hash ids like a few days back. Given the troubled times around the liblzma versions I'm not sure the initial issue about the report topic years back is still solved as we are back to a version before 5.6.0 again. My typical "problematic" packages are thunderbird and especially kicad-packages3d, I wasn't needed to package a newer version of any of these source packages lately. -- Regards Carsten
Bug#888572: git-buildpackage: creating xz tarballs should be done multi-threaded
Hi Carsten, all, On Fri Mar 22, 2024 at 11:22 AM EDT, Carsten Schoenert wrote: [...] > > Thanks a lot for the follow up. Carsten can you confirm with TB if this > > works as > > expected? > > I need to check this again and in detail, because I had recently some > issues as the expected hash ids were different while working with gbp > import and later gbp buildpackage on some package from the kicad > universe. But I think these problems are currently related to the t64 > transition. [...] I'd run into a similar issue with gbp and I, too, initially suspected t64, but I believe it was actually an issue with pristine-tar, that Sebastian has sinced fixed, per https://bugs.debian.org/1065751. I haven't had the chance to specifically try reproducing it with pristine-tar 1.50+nmu2, but per Sebastian and my colleague Jeremy it should now be fixed with that version of pristine-tar. Otherwise if you can still reproduce it please open a bug against pristine-tar and let Sebastian know. Thanks, -a
Bug#888572: git-buildpackage: creating xz tarballs should be done multi-threaded
Hi Guido, Am 22.03.24 um 09:35 schrieb Guido Günther: ... I got pointed here by Amin. As mentioned, pristine-tar can handle multi-block xz images so this is not an issue. Further xz-utils v5.6.0+ (currently in unstable/ testing) is using multi threaded compression by default. ahh, that would explain why my last work on kicad-packages3d and in detail the pristine-tar commit did not take the usual hour and did take only about 10-15min! :-) So this *might* be considered fixed. > Thanks a lot for the follow up. Carsten can you confirm with TB if this works as expected? I need to check this again and in detail, because I had recently some issues as the expected hash ids were different while working with gbp import and later gbp buildpackage on some package from the kicad universe. But I think these problems are currently related to the t64 transition. The last work on TB at the start of the work did work flawless. @Guido Please poke me again latest 2 weeks in case I don't have give feedback until then. -- Mit freundlichen Grüßen Carsten Schönert
Bug#888572: git-buildpackage: creating xz tarballs should be done multi-threaded
Hi Sebastian, Carsten, On Fri, Mar 22, 2024 at 08:34:29AM +0100, Sebastian Andrzej Siewior wrote: > On 2019-03-08 22:10:10 [+0100], Carsten Schoenert wrote: > > Hello Guido, > Hi, > > > On Tue, Jan 30, 2018 at 07:19:48AM +0100, Carsten Schoenert wrote: > > > > We should not do more options. Multi threaded should be on when: > > > > > > > > - not using pristine-tar > > > > - iff pristine-tar can handle it > > > > > > > > The first one is simple but what about the second? > > > > the mentioned issue #888572 was closed some time ago so pristine-tar > > shouldn't be a problem here anymore. > > > > I will try to tweak gbp locally now again while working on a new version > > of kicad-packages3d which will be about 5GB now to archive. ;) > > I got pointed here by Amin. > As mentioned, pristine-tar can handle multi-block xz images so this is > not an issue. Further xz-utils v5.6.0+ (currently in unstable/ testing) > is using multi threaded compression by default. So this *might* be > considered fixed. Thanks a lot for the follow up. Carsten can you confirm with TB if this works as expected? Cheers, -- Guido
Bug#888572: git-buildpackage: creating xz tarballs should be done multi-threaded
On 2019-03-08 22:10:10 [+0100], Carsten Schoenert wrote: > Hello Guido, Hi, > On Tue, Jan 30, 2018 at 07:19:48AM +0100, Carsten Schoenert wrote: > > > We should not do more options. Multi threaded should be on when: > > > > > > - not using pristine-tar > > > - iff pristine-tar can handle it > > > > > > The first one is simple but what about the second? > > the mentioned issue #888572 was closed some time ago so pristine-tar > shouldn't be a problem here anymore. > > I will try to tweak gbp locally now again while working on a new version > of kicad-packages3d which will be about 5GB now to archive. ;) I got pointed here by Amin. As mentioned, pristine-tar can handle multi-block xz images so this is not an issue. Further xz-utils v5.6.0+ (currently in unstable/ testing) is using multi threaded compression by default. So this *might* be considered fixed. > Regards > Carsten Sebastian
Bug#888572: git-buildpackage: creating xz tarballs should be done multi-threaded
Hello Guido, On Tue, Jan 30, 2018 at 07:19:48AM +0100, Carsten Schoenert wrote: > > We should not do more options. Multi threaded should be on when: > > > > - not using pristine-tar > > - iff pristine-tar can handle it > > > > The first one is simple but what about the second? the mentioned issue #888572 was closed some time ago so pristine-tar shouldn't be a problem here anymore. I will try to tweak gbp locally now again while working on a new version of kicad-packages3d which will be about 5GB now to archive. ;) Regards Carsten
Bug#888572: git-buildpackage: creating xz tarballs should be done multi-threaded
Hello Guido, On Mon, Jan 29, 2018 at 01:02:30PM +0100, Guido Günther wrote: > > If possible gbp could become a at least a option for setting up the > > number of threads the xz commend will use as long pristine-tar isn't > > working properly with all kinds xz archives? > > We should not do more options. Multi threaded should be on when: > > - not using pristine-tar > - iff pristine-tar can handle it > > The first one is simple but what about the second? I agree on not adding more options and put the build/create metrics into gbp's internal logic. I was also thinking about a useful co-exitence between gbp and pristine-tar as it's pointless to add functionality of multi-threaded xz archive creation into gbp but pristine-tar can't care such tarballs correctly. I'm not fully sure how to proceed here, as it's currently more a problem of pristine-tar I think (given the timespan #869191 is opened). #869191 should be tagged to affect this report firstly? Regards Carsten
Bug#888572: git-buildpackage: creating xz tarballs should be done multi-threaded
Hi,
On Sat, Jan 27, 2018 at 10:38:12AM +0100, Carsten Schoenert wrote:
> Package: git-buildpackage
> Version: 0.9.6
> Severity: wishlist
>
> Hi,
>
> while doing packaging work on packages which have several hundreds or
> gigabytes if source code files it takes currently a long time to import
> new upstream tarballs which uses xz for compression. The same amount of
> time is needed if gbp needs to recreate the tarball from the upstream
> branch. This happens because the xz command is called without option
> '-T|--threads [x]' and thus only one thread is used to do the
> compression.
>
> I've hacked compressor.py a bit to let gbp call 'xz -T 0 ...' for using
> the maximum of threads and this is decreasing the time xz needs
> significantly. On my quad core machine about about a quarter of the time
> as before.
>
> diff --git a/gbp/pkg/compressor.py b/gbp/pkg/compressor.py
> index b1b795f..a345b34 100644
> --- a/gbp/pkg/compressor.py
> +++ b/gbp/pkg/compressor.py
> @@ -24,7 +24,7 @@ class Compressor(object):
> Opts = {'gzip': '-n',
> 'bzip2': '',
> 'lzma': '',
> -'xz': ''}
> +'xz': '"-T 0"'}
>
> Exts = {'gzip': 'gz',
> 'bzip2': 'bz2',
>
> But currently pristine-tar may stumble about tarballs which created by
> this option, please see #869191 for details. The provided patch in
> #869191 is fixing the problems so far for me.
>
> If possible gbp could become a at least a option for setting up the
> number of threads the xz commend will use as long pristine-tar isn't
> working properly with all kinds xz archives?
We should not do more options. Multi threaded should be on when:
- not using pristine-tar
- iff pristine-tar can handle it
The first one is simple but what about the second?
Cheers,
-- Guido
Bug#888572: git-buildpackage: creating xz tarballs should be done multi-threaded
Package: git-buildpackage
Version: 0.9.6
Severity: wishlist
Hi,
while doing packaging work on packages which have several hundreds or
gigabytes if source code files it takes currently a long time to import
new upstream tarballs which uses xz for compression. The same amount of
time is needed if gbp needs to recreate the tarball from the upstream
branch. This happens because the xz command is called without option
'-T|--threads [x]' and thus only one thread is used to do the
compression.
I've hacked compressor.py a bit to let gbp call 'xz -T 0 ...' for using
the maximum of threads and this is decreasing the time xz needs
significantly. On my quad core machine about about a quarter of the time
as before.
diff --git a/gbp/pkg/compressor.py b/gbp/pkg/compressor.py
index b1b795f..a345b34 100644
--- a/gbp/pkg/compressor.py
+++ b/gbp/pkg/compressor.py
@@ -24,7 +24,7 @@ class Compressor(object):
Opts = {'gzip': '-n',
'bzip2': '',
'lzma': '',
-'xz': ''}
+'xz': '"-T 0"'}
Exts = {'gzip': 'gz',
'bzip2': 'bz2',
But currently pristine-tar may stumble about tarballs which created by
this option, please see #869191 for details. The provided patch in
#869191 is fixing the problems so far for me.
If possible gbp could become a at least a option for setting up the
number of threads the xz commend will use as long pristine-tar isn't
working properly with all kinds xz archives?
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=869191
Regards
Carsten
-- System Information:
Debian Release: buster/sid
APT prefers testing
APT policy: (500, 'testing'), (1, 'experimental')
Architecture: amd64 (x86_64)
Foreign Architectures: i386
Kernel: Linux 4.14.0-3-amd64 (SMP w/4 CPU cores)
Locale: LANG=de_DE.UTF-8, LC_CTYPE=de_DE.UTF-8 (charmap=UTF-8),
LANGUAGE=de_DE.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /usr/bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled
Versions of packages git-buildpackage depends on:
ii devscripts 2.17.12
ii git1:2.15.1-3
ii man-db 2.7.6.1-4
ii python33.6.4-1
ii python3-dateutil 2.6.1-1
ii python3-pkg-resources 38.2.4-2
Versions of packages git-buildpackage recommends:
ii cowbuilder0.86
ii pbuilder 0.229
ii pristine-tar 1.42
ii python3-requests 2.18.4-1
Versions of packages git-buildpackage suggests:
pn python3-notify2
ii sudo 1.8.21p2-3
ii unzip6.0-21
-- no debconf information
