Re: [ptxdist] [PATCH] image-root-cpio.in: Use multithreaded XZ (revisited).
On 3/7/24 17:04, Michael Olbrich wrote: > Hi, > > On Sun, Feb 25, 2024 at 03:49:06PM +0100, Christian Melki wrote: >> So, after asking questions about the default multithread mode >> of xz 5.6.0 and determinism, it turned out that there >> is a common misconception about determinism in xz. >> >> XZ compression varies between the two compressors. Ie, the single >> thread and multi thread variant. T0 always uses the multithread >> one, even if used on one core. But T1 uses the singlethread >> compressor. >> >> This has caused a very common misconception about determinism >> variability with number of threads. I've seen this discussion >> on a lot of places. So albeit true, it only applies to the >> distinct compressors, not how many threads are used in >> multithread mode. >> >> Now after giving this some thought I've decided to reenable >> threaded XZ, using T0. T0 should produce the same results always, >> on any target. Even unicore machines. And with the multithread >> compressor now being the default, users should end up with the >> same results unless they are actively using T1. > > So if I understand it correctly then multithreaded compression results i a > slightly larger output file. But it also means that it can be extracted > with multiple threads. > The question is: will that actually happen? The initramfs is extracted by > the kernel and I'm not sure if it supports that. > > Michael > Not sure I follow. You're asking if a multithreaded compressed archive can be decompressed by the in kernel (single thread, assumed) decompressor? It would seem it's very bad if it wasn't capable of doing so, so I never think about that. I just assume it. Regardless. You may drop this if you like, regarding slightly larger archives. It's a very trivial local change for those who thinks it's useful. Regards, Christian >> Signed-off-by: Christian Melki >> --- >> platforms/image-root-cpio.in | 2 +- >> 1 file changed, 1 insertion(+), 1 deletion(-) >> >> diff --git a/platforms/image-root-cpio.in b/platforms/image-root-cpio.in >> index 0338175a0..546dda7fb 100644 >> --- a/platforms/image-root-cpio.in >> +++ b/platforms/image-root-cpio.in >> @@ -84,7 +84,7 @@ config IMAGE_ROOT_CPIO_COMPRESSION_UTIL >> default "" if IMAGE_ROOT_CPIO_COMPRESSION_MODE_NONE >> default "gzip" if IMAGE_ROOT_CPIO_COMPRESSION_MODE_GZ >> default "zstd -T0 -19" if IMAGE_ROOT_CPIO_COMPRESSION_MODE_ZSTD >> -default "xz --check=crc32" if IMAGE_ROOT_CPIO_COMPRESSION_MODE_XZ >> +default "xz -T0 --check=crc32" if IMAGE_ROOT_CPIO_COMPRESSION_MODE_XZ >> default "lzop" if IMAGE_ROOT_CPIO_COMPRESSION_MODE_LZOP >> >> endif >> -- >> 2.34.1 >> >> >> >
Re: [ptxdist] [PATCH] image-root-cpio.in: Use multithreaded XZ (revisited).
Hi, On Sun, Feb 25, 2024 at 03:49:06PM +0100, Christian Melki wrote: > So, after asking questions about the default multithread mode > of xz 5.6.0 and determinism, it turned out that there > is a common misconception about determinism in xz. > > XZ compression varies between the two compressors. Ie, the single > thread and multi thread variant. T0 always uses the multithread > one, even if used on one core. But T1 uses the singlethread > compressor. > > This has caused a very common misconception about determinism > variability with number of threads. I've seen this discussion > on a lot of places. So albeit true, it only applies to the > distinct compressors, not how many threads are used in > multithread mode. > > Now after giving this some thought I've decided to reenable > threaded XZ, using T0. T0 should produce the same results always, > on any target. Even unicore machines. And with the multithread > compressor now being the default, users should end up with the > same results unless they are actively using T1. So if I understand it correctly then multithreaded compression results i a slightly larger output file. But it also means that it can be extracted with multiple threads. The question is: will that actually happen? The initramfs is extracted by the kernel and I'm not sure if it supports that. Michael > Signed-off-by: Christian Melki > --- > platforms/image-root-cpio.in | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) > > diff --git a/platforms/image-root-cpio.in b/platforms/image-root-cpio.in > index 0338175a0..546dda7fb 100644 > --- a/platforms/image-root-cpio.in > +++ b/platforms/image-root-cpio.in > @@ -84,7 +84,7 @@ config IMAGE_ROOT_CPIO_COMPRESSION_UTIL > default "" if IMAGE_ROOT_CPIO_COMPRESSION_MODE_NONE > default "gzip" if IMAGE_ROOT_CPIO_COMPRESSION_MODE_GZ > default "zstd -T0 -19" if IMAGE_ROOT_CPIO_COMPRESSION_MODE_ZSTD > - default "xz --check=crc32" if IMAGE_ROOT_CPIO_COMPRESSION_MODE_XZ > + default "xz -T0 --check=crc32" if IMAGE_ROOT_CPIO_COMPRESSION_MODE_XZ > default "lzop" if IMAGE_ROOT_CPIO_COMPRESSION_MODE_LZOP > > endif > -- > 2.34.1 > > > -- Pengutronix e.K. | | Steuerwalder Str. 21 | http://www.pengutronix.de/ | 31137 Hildesheim, Germany | Phone: +49-5121-206917-0| Amtsgericht Hildesheim, HRA 2686 | Fax: +49-5121-206917- |
[ptxdist] [PATCH] image-root-cpio.in: Use multithreaded XZ (revisited).
So, after asking questions about the default multithread mode of xz 5.6.0 and determinism, it turned out that there is a common misconception about determinism in xz. XZ compression varies between the two compressors. Ie, the single thread and multi thread variant. T0 always uses the multithread one, even if used on one core. But T1 uses the singlethread compressor. This has caused a very common misconception about determinism variability with number of threads. I've seen this discussion on a lot of places. So albeit true, it only applies to the distinct compressors, not how many threads are used in multithread mode. Now after giving this some thought I've decided to reenable threaded XZ, using T0. T0 should produce the same results always, on any target. Even unicore machines. And with the multithread compressor now being the default, users should end up with the same results unless they are actively using T1. Signed-off-by: Christian Melki --- platforms/image-root-cpio.in | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/platforms/image-root-cpio.in b/platforms/image-root-cpio.in index 0338175a0..546dda7fb 100644 --- a/platforms/image-root-cpio.in +++ b/platforms/image-root-cpio.in @@ -84,7 +84,7 @@ config IMAGE_ROOT_CPIO_COMPRESSION_UTIL default "" if IMAGE_ROOT_CPIO_COMPRESSION_MODE_NONE default "gzip" if IMAGE_ROOT_CPIO_COMPRESSION_MODE_GZ default "zstd -T0 -19" if IMAGE_ROOT_CPIO_COMPRESSION_MODE_ZSTD - default "xz --check=crc32" if IMAGE_ROOT_CPIO_COMPRESSION_MODE_XZ + default "xz -T0 --check=crc32" if IMAGE_ROOT_CPIO_COMPRESSION_MODE_XZ default "lzop" if IMAGE_ROOT_CPIO_COMPRESSION_MODE_LZOP endif -- 2.34.1
