Re: [arch-general] Multi-threaded mkinitpcio

[Lone_Wolf]

On 06-04-2020 12:43, Ralph Corderoy wrote:

Hi Lone Wolf,


  $ perf stat -e instructions gzip /dev/null
  $ perf stat -e instructions lzop /dev/null

Those outputs appear to be from unpacking initramfs.
I do think OP and Giancarlo were talking about creating an initramfs .

Oh, sorry for the noise if I'm wrong, but
/boot/initramfs-linux-lts-fallback.img is gzip-compressed here.
I uncompressed it to give the initramfs-linux-lts-fallback used above.
I thought the compression of it was the step OP and Giancarlo were
discussing, as you say.



Guess I misinterpretated the command, ralph .

Below are some data from my own system (threadripper 1920x , NvME2 ssd) .

lzop is about twice as fast as gzip , while xz is very slow.

Lone_Wolf


$ perf stat -e instructions mkinitcpio --generate testing --compress 
gzip > /dev/null


 Performance counter stats for 'mkinitcpio --generate testing 
--compress gzip':


    16,585,047,999 instructions:u

   4.706074377 seconds time elapsed

   4.000137000 seconds user
   1.18722 seconds sys


$ perf stat -e instructions mkinitcpio --generate testing --compress xz 
> /dev/null


 Performance counter stats for 'mkinitcpio --generate testing 
--compress xz':


    84,360,010,752 instructions:u

  17.415348248 seconds time elapsed

  16.75674 seconds user
   1.500338000 seconds sys


$ perf stat -e instructions mkinitcpio --generate testing --compress 
lzop > /dev/null


 Performance counter stats for 'mkinitcpio --generate testing 
--compress lzop':


 5,316,618,410 instructions:u

   2.426944720 seconds time elapsed

   1.695263000 seconds user
   1.027651000 seconds sys


$

Re: [arch-general] Multi-threaded mkinitpcio

[Ralph Corderoy]

Hi Lone Wolf,

> >  $ perf stat -e instructions gzip  > >/dev/null
> >  $ perf stat -e instructions lzop  > >/dev/null
>
> Those outputs appear to be from unpacking initramfs.
> I do think OP and Giancarlo were talking about creating an initramfs .

Oh, sorry for the noise if I'm wrong, but
/boot/initramfs-linux-lts-fallback.img is gzip-compressed here.
I uncompressed it to give the initramfs-linux-lts-fallback used above.
I thought the compression of it was the step OP and Giancarlo were
discussing, as you say.

-- 
Cheers, Ralph.

Re: [arch-general] Multi-threaded mkinitpcio

[Lone_Wolf]

On 06-04-2020 11:47, Ralph Corderoy wrote:

Hi Giancarlo,


Of course lzop will be faster than xz, but mkinitcpio's default is gz,
which should be comparable, at least in speed, with lzop.

A slap-dash test.

 $ perf stat -e instructions gzip /dev/null
  Performance counter stats for 'gzip':
19,533,249,541  instructions:u
  19.642066695 seconds time elapsed
  19.540737000 seconds user
   0.13000 seconds sys
 $ perf stat -e instructions lzop /dev/null
  Performance counter stats for 'lzop':
 1,233,520,108  instructions:u
   1.129177816 seconds time elapsed
   1.092271000 seconds user
   0.036743000 seconds sys
 $

gzip is about twenty times slower.

 1,233,520,108 / 19,533,249,541 = 0.0631497644777874596
   1.129177816 / 19.642066695   = 0.05748772944994829119

Yes, my computer is probably slower than most, but then perhaps so is
the OP's.



Those outputs appear to be from unpacking initramfs.

I do think OP and Giancarlo were talking about creating an initramfs .

LW

Re: [arch-general] Multi-threaded mkinitpcio

[Ralph Corderoy]

Hi Giancarlo,

> Of course lzop will be faster than xz, but mkinitcpio's default is gz,
> which should be comparable, at least in speed, with lzop.

A slap-dash test.

$ perf stat -e instructions gzip /dev/null
 Performance counter stats for 'gzip':
19,533,249,541  instructions:u  

  19.642066695 seconds time elapsed
  19.540737000 seconds user
   0.13000 seconds sys
$ perf stat -e instructions lzop /dev/null
 Performance counter stats for 'lzop':
 1,233,520,108  instructions:u  

   1.129177816 seconds time elapsed
   1.092271000 seconds user
   0.036743000 seconds sys
$

gzip is about twenty times slower.

1,233,520,108 / 19,533,249,541 = 0.0631497644777874596
  1.129177816 / 19.642066695   = 0.05748772944994829119

Yes, my computer is probably slower than most, but then perhaps so is
the OP's.

-- 
Cheers, Ralph.

Re: [arch-general] Multi-threaded mkinitpcio

[Pascal via arch-general]

thank you for these details.

regard, lacsaP.

Le lun. 6 avr. 2020 à 05:39, Giancarlo Razzolini via arch-general <
arch-general@archlinux.org> a écrit :

> Em abril 4, 2020 12:13 Jelle van der Waa escreveu:
> >
> > multi-threaded compression does not create a predictable reproducible
> > archive (for xz/zstd).
> >
>
> Zstd is multi-thread safe, as far as I know, xz is not. But the kernel does
> not support zstd.
>
> >
> > Sure, has nothing to do with reproducibility however.
> >
>
> It has, because lzop is the only algorithm that produces unreproducible
> builds
> due to them adding their own timestamp, which there's no flag to remove.
> Of course
> lzop will be faster than xz, but mkinitcpio's default is gz, which should
> be comparable,
> at least in speed, with lzop.
>
> Regards,
> Giancarlo Razzolini

Re: [arch-general] Multi-threaded mkinitpcio

[Giancarlo Razzolini via arch-general]

Em abril 4, 2020 12:13 Jelle van der Waa escreveu:


multi-threaded compression does not create a predictable reproducible
archive (for xz/zstd).



Zstd is multi-thread safe, as far as I know, xz is not. But the kernel does
not support zstd.



Sure, has nothing to do with reproducibility however.



It has, because lzop is the only algorithm that produces unreproducible builds
due to them adding their own timestamp, which there's no flag to remove. Of 
course
lzop will be faster than xz, but mkinitcpio's default is gz, which should be 
comparable,
at least in speed, with lzop.

Regards,
Giancarlo Razzolini

pgpW22pQ7tQDH.pgp
Description: PGP signature

Re: [arch-general] Multi-threaded mkinitpcio

[Jelle van der Waa]

On 04/04/2020 12:11, Pascal via arch-general wrote:
> I don't quite understand where the reproducibility is broken ?

multi-threaded compression does not create a predictable reproducible
archive (for xz/zstd).

> Amin Vakil says that rebuilding his initrd (when updating kernel and/or
> modules) takes a lot of time and the final compression phase can be reduced
> by switching to lz4 or lzop. when you go from xz (which may be his case) to
> lzop, the difference is remarkable.

Sure, has nothing to do with reproducibility however.

> 
> Le ven. 3 avr. 2020 à 14:47, Giancarlo Razzolini 
> a écrit :
> 
>> Em abril 3, 2020 3:17 Pascal via arch-general escreveu:
>>> hi,
>>>
>>> change the COMPRESSION variable to lz4 ou lzop in your
>> /etc/mkinitcpio.conf
>>> : it will considerably reduce the compression time.
>>>
>>
>> And break reproducibility. Also, I wouldn't say considerably. Might be
>> faster,
>> but not leaps and bounds faster.
>>
>> Regards,
>> Giancarlo Razzolini
>>

Re: [arch-general] Multi-threaded mkinitpcio

[Pascal via arch-general]

I don't quite understand where the reproducibility is broken ?

Amin Vakil says that rebuilding his initrd (when updating kernel and/or
modules) takes a lot of time and the final compression phase can be reduced
by switching to lz4 or lzop. when you go from xz (which may be his case) to
lzop, the difference is remarkable.

Le ven. 3 avr. 2020 à 14:47, Giancarlo Razzolini 
a écrit :

> Em abril 3, 2020 3:17 Pascal via arch-general escreveu:
> > hi,
> >
> > change the COMPRESSION variable to lz4 ou lzop in your
> /etc/mkinitcpio.conf
> > : it will considerably reduce the compression time.
> >
>
> And break reproducibility. Also, I wouldn't say considerably. Might be
> faster,
> but not leaps and bounds faster.
>
> Regards,
> Giancarlo Razzolini
>

Re: [arch-general] Multi-threaded mkinitpcio

[Giancarlo Razzolini via arch-general]

Em abril 3, 2020 9:28 Amin Vakil escreveu:

time and saving space. Also I'm planning to read wiki on how to safely
disable fallback. I'm using LVM on LUKS method and I suppose fallback
image cannot boot my Arch, so it doesn't help in case of an accident.



The fallback image does help. But if you keep an arch iso usb stick around,
and make sure you update it every few months, you can disable the fallback.

Regards,
Giancarlo Razzolini

pgp3GDTKQDSKW.pgp
Description: PGP signature

Re: [arch-general] Multi-threaded mkinitpcio

[Giancarlo Razzolini via arch-general]

Em abril 3, 2020 3:17 Pascal via arch-general escreveu:

hi,

change the COMPRESSION variable to lz4 ou lzop in your /etc/mkinitcpio.conf
: it will considerably reduce the compression time.



And break reproducibility. Also, I wouldn't say considerably. Might be faster,
but not leaps and bounds faster.

Regards,
Giancarlo Razzolini


pgpvxCf7l2Bta.pgp
Description: PGP signature

Re: [arch-general] Multi-threaded mkinitpcio

[Amish via arch-general]

On 03/04/20 5:58 pm, Amin Vakil wrote:

As it's true for pretty much all update process for pretty much every OS
in the world, things can break if the update is interrupted.

I meant as other linux distros keep multiple kernel versions, if
something happens during kernel update you can always boot to another
kernel which I've done on CentOS servers before when update has broken
during kernel update.


In case of Arch Linux when kernel updates it deletes all initramfs and 
vmlinuz related to that kernel just before the update.


Via: /usr/share/libalpm/hooks/60-mkinitcpio-remove.hook

So when there is update to linux as well as linux-lts and something 
happens while upgrade is running, none of the kernels will boot.


So best way is to, update each kernel separately. (I normally use pacman 
-Syu --ignore 'linux-lts*')


Amish.

Re: [arch-general] Multi-threaded mkinitpcio

[Amin Vakil]

> To sum it up I will enable lz4 compression to benefit from both reduced
> time and saving space. Also I'm planning to read wiki on how to safely
> disable fallback. I'm using LVM on LUKS method and I suppose fallback
> image cannot boot my Arch, so it doesn't help in case of an accident.

Removing fallback from /etc/mkinitpcio.d/linux.preset &
/etc/lmkinitpcio.d/linux-hardened.preset PRESET which makes it like this
and enabling lz4 compression reduced my time from 100 seconds to 20 seconds.

PRESETS=('default')

real0m19.104s
user0m12.970s
sys 0m7.856s



signature.asc
Description: OpenPGP digital signature

Re: [arch-general] Multi-threaded mkinitpcio

[Amin Vakil]

> 
> As it's true for pretty much all update process for pretty much every OS
> in the world, things can break if the update is interrupted.
> 
I meant as other linux distros keep multiple kernel versions, if
something happens during kernel update you can always boot to another
kernel which I've done on CentOS servers before when update has broken
during kernel update.
> 
>   For my particular setup, running `mkinitcpio -P` yields those times:
> 
> real  1m8.320s
> user  0m24.742s
> sys   0m12.806s
> 
> Half of the time mkinitcpio was waiting for disk I/O. Most likely
> `find`, but that?s only mu guess.
> 
As my Arch is on SSD I suppose hard drive is not a bottleneck.
> 
> Even while it's unlikely that only mkinitcpio suffers from a damaged
> HDD and even while smartctl not necessarily does show issues of a
> already damaged aged HDD, I would run smartctl and take a look, if it
> shows something fishy.
> 
Good point, I've checked and my SSD is OK.
> 
> change the COMPRESSION variable to lz4 ou lzop in your /etc/mkinitcpio.conf
> : it will considerably reduce the compression time.
> 
Great suggestion, I've compared with gzip and it's really considerably
faster. (Four images are built, linux (default, fallback),
linux-hardened (default, fallback))

gzip:
real1m42.140s
user1m22.717s
sys 0m23.493s

lz4:
real1m1.120s
user0m44.310s
sys 0m21.785s

cat (without compression):
real0m59.477s
user0m43.050s
sys 0m21.946s

To sum it up I will enable lz4 compression to benefit from both reduced
time and saving space. Also I'm planning to read wiki on how to safely
disable fallback. I'm using LVM on LUKS method and I suppose fallback
image cannot boot my Arch, so it doesn't help in case of an accident.

Thank you all



signature.asc
Description: OpenPGP digital signature

Re: [arch-general] Multi-threaded mkinitpcio

[Pascal via arch-general]

hi,

change the COMPRESSION variable to lz4 ou lzop in your /etc/mkinitcpio.conf
: it will considerably reduce the compression time.

grep COMP /etc/mkinitcpio.conf
# COMPRESSION
#COMPRESSION="gzip"
#COMPRESSION="bzip2"
#COMPRESSION="lzma"
#COMPRESSION="xz"
#COMPRESSION="lzop"
COMPRESSION="lz4"
# COMPRESSION_OPTIONS
#COMPRESSION_OPTIONS=()

see here

for quick  benchmarks...

regards, lacsaP.

Le ven. 3 avr. 2020 à 08:08, Ralf Mardorf via arch-general <
arch-general@archlinux.org> a écrit :

> On Fri, 3 Apr 2020 04:36:37 +0200, mpan wrote:
> >Half of the time mkinitcpio was waiting for disk I/O.
>
> Even while it's unlikely that only mkinitcpio suffers from a damaged
> HDD and even while smartctl not necessarily does show issues of a
> already damaged aged HDD, I would run smartctl and take a look, if it
> shows something fishy.
>

Re: [arch-general] Multi-threaded mkinitpcio

[Ralf Mardorf via arch-general]

On Fri, 3 Apr 2020 04:36:37 +0200, mpan wrote:
>Half of the time mkinitcpio was waiting for disk I/O.

Even while it's unlikely that only mkinitcpio suffers from a damaged
HDD and even while smartctl not necessarily does show issues of a
already damaged aged HDD, I would run smartctl and take a look, if it
shows something fishy.

Re: [arch-general] Multi-threaded mkinitpcio

[mpan]

> Actually this 2 minutes bothers me this much that I'm emailing Arch
> General Mailing List is because I'm afraid my Arch would be broken if my
> laptop shuts down in this process and I have to rescue it by live USB.
  For nearly a decade of using this particular Arch installation alone,
I never had a single case of losing power while mkinitcpio was running.
I wouldn’t be afraid of that scenario. And laptops are better than my PC
in that manner, because you have backup power from the battery.

> Does it help if it gets executed over multi threads or the bottleneck is
> somewhere else?
  For my particular setup, running `mkinitcpio -P` yields those times:

real1m8.320s
user0m24.742s
sys 0m12.806s

Half of the time mkinitcpio was waiting for disk I/O. Most likely
`find`, but that’s only mu guess.

  This is only a single computer. Unlike modern laptops it uses spinning
rust for storage, but OTOH it is also running a CPU that remembers 2000s
and DDR2-800 RAM. So both would affect the outcome.



signature.asc
Description: OpenPGP digital signature

Re: [arch-general] Multi-threaded mkinitpcio

[Giancarlo Razzolini via arch-general]

Em abril 2, 2020 22:28 Amin Vakil escreveu:

So every update of linux kernels and modules which need mkinitpcio to be
executed, takes too much unnecessary time (maybe 2 min) which could be
handled if it executes on multi threads of CPU.



Which compression algorithm are you using?


Actually this 2 minutes bothers me this much that I'm emailing Arch
General Mailing List is because I'm afraid my Arch would be broken if my
laptop shuts down in this process and I have to rescue it by live USB.



As it's true for pretty much all update process for pretty much every OS
in the world, things can break if the update is interrupted.



So my question is why it doesn't execute over all threads or at least
have the option to do that?



mkinitcpio is a bunch of scripts copying files to a temporary dir and then
compressing the result afterwards. I bet most of this time you're seeing is
on compression's fault, not mkinicpio itself.


Does it help if it gets executed over multi threads or the bottleneck is
somewhere else?



Depending on the compression algorithm, reproducibility might not work if
it's using multiple threads. Also, I see multiple issues that could happen
if the build hooks ran out of order.

Regards,
Giancarlo Razzolini

pgpPId6xm3A5S.pgp
Description: PGP signature

[arch-general] Multi-threaded mkinitpcio

[Amin Vakil]

So every update of linux kernels and modules which need mkinitpcio to be
executed, takes too much unnecessary time (maybe 2 min) which could be
handled if it executes on multi threads of CPU.

Actually this 2 minutes bothers me this much that I'm emailing Arch
General Mailing List is because I'm afraid my Arch would be broken if my
laptop shuts down in this process and I have to rescue it by live USB.

I have read the mkinitpcio wiki page and I can't find anything regarding
this matter:
https://wiki.archlinux.org/index.php/Mkinitcpio

I've checked and my cpu usage and it shows me that it uses maximum power
of a core when mkinitpcio is running, so I assumed it could be helped if
it runs on multi threads.

So my question is why it doesn't execute over all threads or at least
have the option to do that?

Does it help if it gets executed over multi threads or the bottleneck is
somewhere else?



signature.asc
Description: OpenPGP digital signature