Re: Fedora 32 system-wide change proposal: reduce installation media size by improving the compression ratio of SquashFS filesystem

[Bohdan Khomutskyi]

Hello,

I have implemented the necessary changes in Pungi, the software that 
creates Fedora compose. So this change has a potential of moving forward.


I have created 2 pull-requests for release engineering for this change:

https://pagure.io/pungi-fedora/pull-request/871

https://pagure.io/pungi-fedora/pull-request/872

Fedora release engineering would have the ability to test this change 
and provide fresh test results after Pungi 4.2.4 is released.


I hope this change could land in Fedora 33.

As a reminder, here is the change proposal: 
https://fedoraproject.org/wiki/Changes/OptimizeSquashFS


Regards,

On 13/05/2020 13:32, Bohdan Khomutskyi wrote:


Hello,

It was a long time since the last message in this change proposal.

Recently I was working to reduce the impact of the increased 
compression ratio on the installation image size for Fedora. I have 
achieved outstanding results -- working proof of concept. With the 
following change: https://github.com/rhinstaller/anaconda/pull/2292 , 
not only the higher compression does not impact the installation time. 
In certain cases, the installation time is even reduced. This is 
because of the fact the filesystem internal structure aware process is 
used to install the system from the SquashFS. The new process also 
allows for taking advantage of the multi-core architecture of the 
system during installation -- does the decompression on multiple 
processors in parallel.


The combination of https://github.com/rhinstaller/anaconda/pull/2292 
and https://fedoraproject.org/wiki/Category:Changes/OptimizeSquashFS 
should reduce _both_ the image size and the installation time. The 
installation time will be reduced in case the system is installed from 
the SquashFS. This is the case in Fedora Workstation.


For optimization of the SquashFS, I will work on requesting the 
support of the required functionality in the Pungi compose build software.


On 25/01/2020 16:36, Bohdan Khomutskyi wrote:

Hello,

I opened a request to anaconda team with a draft patch to use 
unsquashfs instead of rsync: 
https://github.com/rhinstaller/anaconda/pull/2292.

That should lower the installation time from Live media.
Adjustments should be made to make this patch work, I was not able to 
install the image using this patch. Anaconda installer crashes.


Chris,

> Could this be read amplification?
> That's probably mitigated with unsquashfs.
That could be read amplification, it will be mitigated with unsquashfs.
The memory usage should not be a problem: xz (1) manual page, states 
that 2 MiB of memory required to decompress an archive with 1MiB 
block size. My previous observations found that the squashfs is 
currently decompressed in single thread. I welcome additional 
testing, but I think the memory limit will not be a problem.


> If image size is a significant consideration, then evaluation of erofs
> seems indicated. It promises both significant compression and CPU
> performance. The intended use case is for Android device read-only
> partitions with both limited storage and CPU/power capacity.

I briefly reviewed the document and found that LZ4 is the only 
supported algorithm in EROFS, even if EROFS has perfect layout of the 
filesystem, it's to good to be true it can outperform the XZ in 
compression ratio tests. The difference for SquashFS LZ4hc 1M vs XZ 
1M is >40%.


On Fri, Jan 24, 2020 at 2:17 AM Chris Murphy > wrote:


On Mon, Jan 20, 2020 at 1:38 AM Bohdan Khomutskyi
mailto:bkhom...@redhat.com>> wrote:
>
> In my previous message, I mentioned that CPU is underutilized
during installation. I haven't investigated further why, but I
suspect it's due to the inefficiency caused by the usage of the
loop device and/or inefficiency in the rsync itself.

Could this be read amplification?

This paper on erofs suggests read amplification can be a significant
side effect with squashfs. It could be exacerbated with random reads,
and I expect it gets worse with larger block size. That's probably
mitigated with unsquashfs.

Specifically page 4, 2nd paragraph.
https://www.usenix.org/system/files/atc19-gao.pdf

This also makes me wonder about the memory consumption effect of a 1M
block size, especially for Fedora ARM where it looks like
Raspberry Pi
2B

Most of the ARM images are raw.xz but some are bootable ISOs, dvd and
netinstall. And those contain a squashfs sysroot. Even if there's no
out of memory problem, it could result in paging. All ISOs setup
swap-on-ZRAM these days, lives, DVD, and netinstall. I think the ARM
case needs testing before committing to 1M block size across all
ISOs,
or implementing changes in Fedora release engineering.


-- 
Chris Murphy




--
Bohdan Khomutskyi, RHCE
Release configuration management engineer, PnT DevOps
Red Hat Czech s.r.o
T: +420532270289 IRC: bkhomuts

--
Bohdan Khomutskyi

Red Hat


--
Bohdan 

Re: Fedora 32 system-wide change proposal: reduce installation media size by improving the compression ratio of SquashFS filesystem

[Chris Murphy]

On Wed, May 13, 2020 at 6:03 PM Chris Murphy  wrote:
>
> On Wed, May 13, 2020 at 5:32 AM Bohdan Khomutskyi  wrote:
> >
> > Hello,
> >
> > It was a long time since the last message in this change proposal.
> >
> > Recently I was working to reduce the impact of the increased compression 
> > ratio on the installation image size for Fedora. I have achieved 
> > outstanding results -- working proof of concept. With the following change: 
> > https://github.com/rhinstaller/anaconda/pull/2292 , not only the higher 
> > compression does not impact the installation time. In certain cases, the 
> > installation time is even reduced. This is because of the fact the 
> > filesystem internal structure aware process is used to install the system 
> > from the SquashFS. The new process also allows for taking advantage of the 
> > multi-core architecture of the system during installation -- does the 
> > decompression on multiple processors in parallel.
> >
> > The combination of https://github.com/rhinstaller/anaconda/pull/2292 and 
> > https://fedoraproject.org/wiki/Category:Changes/OptimizeSquashFS should 
> > reduce _both_ the image size and the installation time. The installation 
> > time will be reduced in case the system is installed from the SquashFS. 
> > This is the case in Fedora Workstation.
> >
> > For optimization of the SquashFS, I will work on requesting the support of 
> > the required functionality in the Pungi compose build software.
>
> Hi, since the feedback was that a higher emphasis be placed on install
> time being reduced, even if there was some increase in ISO size (not
> without limit, it's a balancing act), I'm still curious how the change
> compares when using zstd, all other things being equal.
>
> For example Solus recently changed from xz to zstd in squashfs, and
> claim 3-4x faster install times, with some increase in image size.
> https://getsol.us/2020/01/25/solus-4-1-released/

From anaconda.log for a default/auto LVM+ext4 install using
Fedora-Workstation-Live-x86_64-32-1.6.iso
19:57:16 DBG ui.gui.spokes.installation_progress: The installation has finished.
19:51:52 DBG ui.gui.spokes.installation_progress: The installation has started.
00:05:24

This is not an exact comparison to using a plain squashfs image and
writing out (I'm guessing) 30,000 files to the install target. But,
using unsquashfs to extract the root.img and write it to the same
target:

real 0m50.315s user 2m18.318s sys 0m6.569s

I'm extracting just one file, the embedded ext4. But (a) unsquashfs is
parallelizing at about 270% CPU for a 3 virtual core VM and (b)
/dev/loop1 isn't busy at all. Does unsquashfs and ext4 slow down when
handed 30K files to write out instead of one big one? Dunno. But as
prior testing suggests this is a CPU bound problem, not a disk
contention problem - I'm definitely in the "tell me more" position.

2m18s is a lot better than 5m24s. And honestly 5m isn't bad, it's
takes a lot longer to install Windows 10 and macOS.

I still think that zstd would get even better decompression rates with
less of a CPU, and thus power hit, it could be splitting hairs. I'm
not sure.


-- 
Chris Murphy
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Fedora 32 system-wide change proposal: reduce installation media size by improving the compression ratio of SquashFS filesystem

[Bruno Wolff III]

On Wed, May 13, 2020 at 13:32:19 +0200,
 Bohdan Khomutskyi  wrote:


For optimization of the SquashFS, I will work on requesting the 
support of the required functionality in the Pungi compose build 
software.


Note that squashfs-tools 4.4 just went into rawhide a couple of days ago. 
By default it does reproducible builds and this might affect performance 
since the files need to be added in a consistent order. This probably 
increases the wall clock time when creating images. I wouldn't expect 
it to have much affect on reading data from images, but you might see 
some changes.

___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Fedora 32 system-wide change proposal: reduce installation media size by improving the compression ratio of SquashFS filesystem

[Chris Murphy]

On Wed, May 13, 2020 at 5:32 AM Bohdan Khomutskyi  wrote:
>
> Hello,
>
> It was a long time since the last message in this change proposal.
>
> Recently I was working to reduce the impact of the increased compression 
> ratio on the installation image size for Fedora. I have achieved outstanding 
> results -- working proof of concept. With the following change: 
> https://github.com/rhinstaller/anaconda/pull/2292 , not only the higher 
> compression does not impact the installation time. In certain cases, the 
> installation time is even reduced. This is because of the fact the filesystem 
> internal structure aware process is used to install the system from the 
> SquashFS. The new process also allows for taking advantage of the multi-core 
> architecture of the system during installation -- does the decompression on 
> multiple processors in parallel.
>
> The combination of https://github.com/rhinstaller/anaconda/pull/2292 and 
> https://fedoraproject.org/wiki/Category:Changes/OptimizeSquashFS should 
> reduce _both_ the image size and the installation time. The installation time 
> will be reduced in case the system is installed from the SquashFS. This is 
> the case in Fedora Workstation.
>
> For optimization of the SquashFS, I will work on requesting the support of 
> the required functionality in the Pungi compose build software.

Hi, since the feedback was that a higher emphasis be placed on install
time being reduced, even if there was some increase in ISO size (not
without limit, it's a balancing act), I'm still curious how the change
compares when using zstd, all other things being equal.

For example Solus recently changed from xz to zstd in squashfs, and
claim 3-4x faster install times, with some increase in image size.
https://getsol.us/2020/01/25/solus-4-1-released/

-- 
Chris Murphy
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Fedora 32 system-wide change proposal: reduce installation media size by improving the compression ratio of SquashFS filesystem

[Kamil Paral]

On Wed, May 13, 2020 at 1:33 PM Bohdan Khomutskyi 
wrote:

> Hello,
>
> It was a long time since the last message in this change proposal.
>
> Recently I was working to reduce the impact of the increased compression
> ratio on the installation image size for Fedora. I have achieved
> outstanding results -- working proof of concept. With the following change:
> https://github.com/rhinstaller/anaconda/pull/2292 , not only the higher
> compression does not impact the installation time. In certain cases, the
> installation time is even reduced. This is because of the fact the
> filesystem internal structure aware process is used to install the system
> from the SquashFS. The new process also allows for taking advantage of the
> multi-core architecture of the system during installation -- does the
> decompression on multiple processors in parallel.
>

This sounds very good. Thanks for working on this.
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Fedora 32 system-wide change proposal: reduce installation media size by improving the compression ratio of SquashFS filesystem

[Bohdan Khomutskyi]

Hello,

It was a long time since the last message in this change proposal.

Recently I was working to reduce the impact of the increased compression 
ratio on the installation image size for Fedora. I have achieved 
outstanding results -- working proof of concept. With the following 
change: https://github.com/rhinstaller/anaconda/pull/2292 , not only the 
higher compression does not impact the installation time. In certain 
cases, the installation time is even reduced. This is because of the 
fact the filesystem internal structure aware process is used to install 
the system from the SquashFS. The new process also allows for taking 
advantage of the multi-core architecture of the system during 
installation -- does the decompression on multiple processors in parallel.


The combination of https://github.com/rhinstaller/anaconda/pull/2292 and 
https://fedoraproject.org/wiki/Category:Changes/OptimizeSquashFS should 
reduce _both_ the image size and the installation time. The installation 
time will be reduced in case the system is installed from the SquashFS. 
This is the case in Fedora Workstation.


For optimization of the SquashFS, I will work on requesting the support 
of the required functionality in the Pungi compose build software.


On 25/01/2020 16:36, Bohdan Khomutskyi wrote:

Hello,

I opened a request to anaconda team with a draft patch to use 
unsquashfs instead of rsync: 
https://github.com/rhinstaller/anaconda/pull/2292.

That should lower the installation time from Live media.
Adjustments should be made to make this patch work, I was not able to 
install the image using this patch. Anaconda installer crashes.


Chris,

> Could this be read amplification?
> That's probably mitigated with unsquashfs.
That could be read amplification, it will be mitigated with unsquashfs.
The memory usage should not be a problem: xz (1) manual page, states 
that 2 MiB of memory required to decompress an archive with 1MiB block 
size. My previous observations found that the squashfs is currently 
decompressed in single thread. I welcome additional testing, but I 
think the memory limit will not be a problem.


> If image size is a significant consideration, then evaluation of erofs
> seems indicated. It promises both significant compression and CPU
> performance. The intended use case is for Android device read-only
> partitions with both limited storage and CPU/power capacity.

I briefly reviewed the document and found that LZ4 is the only 
supported algorithm in EROFS, even if EROFS has perfect layout of the 
filesystem, it's to good to be true it can outperform the XZ in 
compression ratio tests. The difference for SquashFS LZ4hc 1M vs XZ 1M 
is >40%.


On Fri, Jan 24, 2020 at 2:17 AM Chris Murphy > wrote:


On Mon, Jan 20, 2020 at 1:38 AM Bohdan Khomutskyi
mailto:bkhom...@redhat.com>> wrote:
>
> In my previous message, I mentioned that CPU is underutilized
during installation. I haven't investigated further why, but I
suspect it's due to the inefficiency caused by the usage of the
loop device and/or inefficiency in the rsync itself.

Could this be read amplification?

This paper on erofs suggests read amplification can be a significant
side effect with squashfs. It could be exacerbated with random reads,
and I expect it gets worse with larger block size. That's probably
mitigated with unsquashfs.

Specifically page 4, 2nd paragraph.
https://www.usenix.org/system/files/atc19-gao.pdf

This also makes me wonder about the memory consumption effect of a 1M
block size, especially for Fedora ARM where it looks like Raspberry Pi
2B

Most of the ARM images are raw.xz but some are bootable ISOs, dvd and
netinstall. And those contain a squashfs sysroot. Even if there's no
out of memory problem, it could result in paging. All ISOs setup
swap-on-ZRAM these days, lives, DVD, and netinstall. I think the ARM
case needs testing before committing to 1M block size across all ISOs,
or implementing changes in Fedora release engineering.


-- 
Chris Murphy




--
Bohdan Khomutskyi, RHCE
Release configuration management engineer, PnT DevOps
Red Hat Czech s.r.o
T: +420532270289 IRC: bkhomuts


--
Bohdan Khomutskyi

Red Hat

___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Fedora 32 system-wide change proposal: reduce installation media size by improving the compression ratio of SquashFS filesystem

[Bohdan Khomutskyi]

Hello,

I opened a request to anaconda team with a draft patch to use unsquashfs
instead of rsync: https://github.com/rhinstaller/anaconda/pull/2292.
That should lower the installation time from Live media.
Adjustments should be made to make this patch work, I was not able to
install the image using this patch. Anaconda installer crashes.

Chris,

> Could this be read amplification?
> That's probably mitigated with unsquashfs.
That could be read amplification, it will be mitigated with unsquashfs.
The memory usage should not be a problem: xz (1) manual page, states that 2
MiB of memory required to decompress an archive with 1MiB block size. My
previous observations found that the squashfs is currently decompressed in
single thread. I welcome additional testing, but I think the memory limit
will not be a problem.

> If image size is a significant consideration, then evaluation of erofs
> seems indicated. It promises both significant compression and CPU
> performance. The intended use case is for Android device read-only
> partitions with both limited storage and CPU/power capacity.

I briefly reviewed the document and found that LZ4 is the only supported
algorithm in EROFS, even if EROFS has perfect layout of the filesystem,
it's to good to be true it can outperform the XZ in compression ratio
tests. The difference for SquashFS LZ4hc 1M vs XZ 1M is >40%.

On Fri, Jan 24, 2020 at 2:17 AM Chris Murphy 
wrote:

> On Mon, Jan 20, 2020 at 1:38 AM Bohdan Khomutskyi 
> wrote:
> >
> > In my previous message, I mentioned that CPU is underutilized during
> installation. I haven't investigated further why, but I suspect it's due to
> the inefficiency caused by the usage of the loop device and/or inefficiency
> in the rsync itself.
>
> Could this be read amplification?
>
> This paper on erofs suggests read amplification can be a significant
> side effect with squashfs. It could be exacerbated with random reads,
> and I expect it gets worse with larger block size. That's probably
> mitigated with unsquashfs.
>
> Specifically page 4, 2nd paragraph.
> https://www.usenix.org/system/files/atc19-gao.pdf
>
> This also makes me wonder about the memory consumption effect of a 1M
> block size, especially for Fedora ARM where it looks like Raspberry Pi
> 2B
>
> Most of the ARM images are raw.xz but some are bootable ISOs, dvd and
> netinstall. And those contain a squashfs sysroot. Even if there's no
> out of memory problem, it could result in paging. All ISOs setup
> swap-on-ZRAM these days, lives, DVD, and netinstall. I think the ARM
> case needs testing before committing to 1M block size across all ISOs,
> or implementing changes in Fedora release engineering.
>
>
> --
> Chris Murphy
>
>

-- 
Bohdan Khomutskyi, RHCE
Release configuration management engineer, PnT DevOps
Red Hat Czech s.r.o
T: +420532270289 IRC: bkhomuts
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Fedora 32 system-wide change proposal: reduce installation media size by improving the compression ratio of SquashFS filesystem

[Chris Murphy]

On Mon, Jan 20, 2020 at 1:38 AM Bohdan Khomutskyi  wrote:
>
> In my previous message, I mentioned that CPU is underutilized during 
> installation. I haven't investigated further why, but I suspect it's due to 
> the inefficiency caused by the usage of the loop device and/or inefficiency 
> in the rsync itself.

Could this be read amplification?

This paper on erofs suggests read amplification can be a significant
side effect with squashfs. It could be exacerbated with random reads,
and I expect it gets worse with larger block size. That's probably
mitigated with unsquashfs.

Specifically page 4, 2nd paragraph.
https://www.usenix.org/system/files/atc19-gao.pdf

This also makes me wonder about the memory consumption effect of a 1M
block size, especially for Fedora ARM where it looks like Raspberry Pi
2B

Most of the ARM images are raw.xz but some are bootable ISOs, dvd and
netinstall. And those contain a squashfs sysroot. Even if there's no
out of memory problem, it could result in paging. All ISOs setup
swap-on-ZRAM these days, lives, DVD, and netinstall. I think the ARM
case needs testing before committing to 1M block size across all ISOs,
or implementing changes in Fedora release engineering.


-- 
Chris Murphy
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Fedora 32 system-wide change proposal: reduce installation media size by improving the compression ratio of SquashFS filesystem

[Chris Murphy]

On Mon, Jan 20, 2020 at 1:38 AM Bohdan Khomutskyi  wrote:

> In my previous message, I mentioned that CPU is underutilized during 
> installation. I haven't investigated further why, but I suspect it's due to 
> the inefficiency caused by the usage of the loop device and/or inefficiency 
> in the rsync itself.

In all installations with xz compression, I see the loop1 device
pegged at 100% CPU (single thread), and perf shows this is almost
entirely lzma decompression. Better utilization would happen with
parallelized decompression threads. But what about the use cases where
there's only one CPU? VMs may not assign multiple CPUs, and what about
the ARM boards we support?

A zstd compressed squashfs image, loop1 uses 30% CPU or less, IO of
either the source or target is near 100% utilization, therefore I'm
not sure parallelization in this case would improve things by much.

> In fact, I have an optimization to file next weekend on my to do list.
>
> > All of the Live installations use rsync.
>
> And that's what I propose to change: to use unsquashfs instead of rsync, 
> preliminary benchmarks show 8x improvement in decompressing speed on local 
> media for XZ on local storage.

I had not considered unsquashfs, so that's an interesting optimization.

> Yes, Zstd consumes 12.24x less CPU user time while unsquashfs, but let's 
> consider the practical application.

I am. There's an electricity cost when there's enough heat generated
by an installation that my computer sounds like a hair dryer.

Therefore, I'm still biased against the heavy CPU cost hit for xz for
an insignificant reduction in ISO size, multiplied by thousands of
installations per week (real, virtual and test) quite a lot of which
aren't USB sticks as sources.

>
> Will Zstd decrease the installation time, given the constraints and 
> optimization above -- that's what I plan to investigate in upcoming weekends.
>
> My proposal focuses on reducing the installation media size, and recommends 
> to use certain compression options. But, I think, the final decision is to be 
> made by FESCO.

If image size is a significant consideration, then evaluation of erofs
seems indicated. It promises both significant compression and CPU
performance. The intended use case is for Android device read-only
partitions with both limited storage and CPU/power capacity.

-- 
Chris Murphy
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Fedora 32 system-wide change proposal: reduce installation media size by improving the compression ratio of SquashFS filesystem

[John M. Harris Jr]

On Sunday, January 19, 2020 4:41:06 PM MST Chris Murphy wrote:
> I admit I'm biased toward the two endpoints: create and consume, not
> distribution ,i.e the mirror donors. Their storage and bandwidth
> concerns were evaluated with the RPM change from xz to zstd. So I'm
> mystified by the bias for image size.

End consumers also benefit from reduced image size, namely in the amount of 
space needed for installation media.

-- 
John M. Harris, Jr.
Splentity

___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Fedora 32 system-wide change proposal: reduce installation media size by improving the compression ratio of SquashFS filesystem

[Chris Murphy]

On Mon, Jan 20, 2020 at 4:24 AM Adam Williamson
 wrote:
>
> On Sun, 2020-01-12 at 15:02 -0700, Chris Murphy wrote:
> > Do you have any tests to compare plain squashfs xz with zstd? The nested
> > ext4 stuff is really pointless now because Fedora hasn't used 'dd' +
> > resizing the ext4 file system as an installation method in a long time
> > (going back to Fedora 18 I think). All of the Live installations use rsync.
>
> I am not entirely clear on what this proposal covers (it would actually
> be nice if this was specified on the Change page; it's not immediately
> clear *which of Fedora's images* are affected by this Change), but are
> you sure of this? Do we not do it even for Cloud image deployments or
> ARM disk image deployments? ISTR there being a filesystem resize
> involved there, which is why I ask...

Cloud images don't use squashfs so they can be ignored. (But yes, some
cloud related images do fs resize on first boot.)

With one exception [1], all ISO images I've looked at have a squashfs
image file on them containing a LiveOS used as system root. If the
change proposal implements plain squashfs images, startup assembly
will use overlayfs. The current ISOs have a nested ext4 in that
squashfs image, and startup assembly uses device-mapper.

Also with one exception [1], any ISO with the word "Live" in the
filename, might switch from rsync based installation to unsquashfs.
The netinstalls and DVD ISOs are unaffected.


[1]
Fedora CoreOS, fedora-coreos-31.20200113.3.1-live.x86_64.iso
(congratulations on the final release btw), which has an initramfs
based LiveOS, uses dd for installation, with ignition doing
provisioning on first boot that also includes GPT fixup and fs resize.


--
Chris Murphy
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Fedora 32 system-wide change proposal: reduce installation media size by improving the compression ratio of SquashFS filesystem

[Kamil Paral]

On Sun, Jan 19, 2020 at 4:42 PM Bohdan Khomutskyi 
wrote:

> Hello,
>
> Thanks everyone for posting feedback.
> More benchmarking results are available at
> https://fedoraproject.org/wiki/Category:Changes/OptimizeSquashFS,
> including the 'plain' SquashFS filesystem.
> After performing the tests, I personally recommend to use xz compression
> with 1MiB block size, without bcj, on a 'plain' squash filesystem -- this
> will lead to a reduction of 142MiB on the ISO, compared to the stock Fedora
> 31 Workstation x86_64 image.
> Alternative compression options, such as Zstd, are also mentioned in the
> change proposal.
>

Hmm, and I see I've been completely confused in my last reply, and I
considered the numbers in the first image [1] to be installation times.
Instead they are image creation times. OK. Looking at the new second image
[2], Zstd seems like a clear winner to me at least for QA purposes. With
fine tuning the compression level, we can achieve almost the same file size
for a great installation speedup.

[1]
https://fedoraproject.org/wiki/File:Compression_vs_SquashFS_creation_time.png
[2] https://fedoraproject.org/wiki/File:Compression_vs_installation_time.png
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Fedora 32 system-wide change proposal: reduce installation media size by improving the compression ratio of SquashFS filesystem

[Adam Williamson]

On Sun, 2020-01-12 at 15:02 -0700, Chris Murphy wrote:
> Do you have any tests to compare plain squashfs xz with zstd? The nested
> ext4 stuff is really pointless now because Fedora hasn't used 'dd' +
> resizing the ext4 file system as an installation method in a long time
> (going back to Fedora 18 I think). All of the Live installations use rsync.

I am not entirely clear on what this proposal covers (it would actually
be nice if this was specified on the Change page; it's not immediately
clear *which of Fedora's images* are affected by this Change), but are
you sure of this? Do we not do it even for Cloud image deployments or
ARM disk image deployments? ISTR there being a filesystem resize
involved there, which is why I ask...
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora
http://www.happyassassin.net
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Fedora 32 system-wide change proposal: reduce installation media size by improving the compression ratio of SquashFS filesystem

[Richard W.M. Jones]

On Mon, Jan 20, 2020 at 09:37:32AM +0100, Bohdan Khomutskyi wrote:
> Chris,
> 
> Thanks for your feedback and comments, it's very valuable to me.
> 
> In my previous message, I mentioned that CPU is *underutilized* during
> installation. I haven't investigated further why, but I suspect it's due to
> the inefficiency caused by the usage of the *loop* device and/or
> inefficiency in the rsync itself.

We might be able to do somewith nbdkit inside the installer to
transparently uncompress and loop mount at the same time.  (Although
this would require xz, not zstd because of a missing feature in zstd -
see my other comment in this thread).

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
libguestfs lets you edit virtual machines.  Supports shell scripting,
bindings from many languages.  http://libguestfs.org
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Fedora 32 system-wide change proposal: reduce installation media size by improving the compression ratio of SquashFS filesystem

[Richard W.M. Jones]

On Sun, Jan 05, 2020 at 10:08:07AM -0700, Chris Murphy wrote:
> In my testing, xz does provide better compression ratios, well suited
> for seldom used images like archives. But it really makes the
> installation experience worse by soaking the CPU, times thousands of
> installations (openQA tests on every single nightly, every human QA
> tester for nightlies, betas, and then the final released product used
> by Fedora end users).
> 
> Has zstandard been evaluated? In my testing of images compressed with
> zstd, the CPU hit is cut by more than 50%, and is no longer a
> bottleneck during installations. Image size does increase, although I
> haven't tested mksquashfs block size higher than 256K. Using zstd with
> Fedora images also builds on prior evaluation, testing, and effort
> moving RPM from xz to zstd.

Blocked-based decompression works with xz, but not with zstd.  We do
use this feature.  Here's the github issue to get block-based
decompression supported in zstd, and a bit of background about how we
use the feature:

https://github.com/facebook/zstd/issues/395#issuecomment-535875379

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
libguestfs lets you edit virtual machines.  Supports shell scripting,
bindings from many languages.  http://libguestfs.org
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Fedora 32 system-wide change proposal: reduce installation media size by improving the compression ratio of SquashFS filesystem

[Bohdan Khomutskyi]

Chris,

Thanks for your feedback and comments, it's very valuable to me.

In my previous message, I mentioned that CPU is *underutilized* during
installation. I haven't investigated further why, but I suspect it's due to
the inefficiency caused by the usage of the *loop* device and/or
inefficiency in the rsync itself.
In fact, I have an optimization to file next weekend on my to do list.

> All of the Live installations use rsync.

And that's what I propose to change: to use unsquashfs instead of rsync,
preliminary benchmarks show 8x improvement in decompressing speed on local
media for XZ on local storage.

On my PC configuration, that will require approx. 52.98 MiB/s sequential
read performance from local media and approx. 181.19MiB writing speed to
the destination media.
That level performance is not common among today's USB drives or optical
media -- the installation speed will not be capped due to the CPU limits,
but otherwise limited by the sequential read speed of the installation
media. It means, selecting an algorithm with better compression ratio
should reduce the installation time from commonly used USB storage.

Yes, Zstd consumes 12.24x less CPU user time while unsquashfs, but let's
consider the practical application.

Will Zstd decrease the installation time, given the constraints and
optimization above -- that's what I plan to investigate in upcoming
weekends.

My proposal focuses on reducing the installation media size, and *recommends
*to use certain compression options. But, I think, the final decision is to
be made by FESCO.

On Mon, Jan 20, 2020 at 12:42 AM Chris Murphy 
wrote:

> On Sun, Jan 19, 2020 at 8:41 AM Bohdan Khomutskyi 
> wrote:
> >
> > Hello,
> >
> > Thanks everyone for posting feedback.
> > More benchmarking results are available at
> https://fedoraproject.org/wiki/Category:Changes/OptimizeSquashFS,
> including the 'plain' SquashFS filesystem.
> > After performing the tests, I personally recommend to use xz compression
> with 1MiB block size, without bcj, on a 'plain' squash filesystem -- this
> will lead to a reduction of 142MiB on the ISO, compared to the stock Fedora
> 31 Workstation x86_64 image.
> > Alternative compression options, such as Zstd, are also mentioned in the
> change proposal.
>
> Thanks for all the tests.
>
> While I see the meaningfully reduced CPU hit of xz compressed images,
> the proposal leaves a lot of performance improvement on the table by
> not also enabling zstd as an option in the compose process. The tests
> show zstd results, but the proposal doesn't mention zstd at all.
>
> In particular for Workstation ISO, the CPU hit isn't worth the size
> savings for regular users, let alone the recurring hit for releng
> composes and QA's automated installation tests. It's a lot of CPU burn
> at both ends of the candle, for not a lot of size savings. I'm not
> convinced it's worth the extra hit on the create side for Zstd level
> 22, compared to Zstd 15 or 17.
>
> I admit I'm biased toward the two endpoints: create and consume, not
> distribution ,i.e the mirror donors. Their storage and bandwidth
> concerns were evaluated with the RPM change from xz to zstd. So I'm
> mystified by the bias for image size.
>
> Anyway, I approve of the change but disappointed if it really doesn't
> let Fedora release engineering the ability to choose (possibly based
> on image type - maybe there's some benefit to using xz for raw and
> qcow2 images).
>
>
>
> --
> Chris Murphy
> ___
> devel mailing list -- devel@lists.fedoraproject.org
> To unsubscribe send an email to devel-le...@lists.fedoraproject.org
> Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
> https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
>


-- 
Bohdan Khomutskyi, RHCE
Release configuration management engineer, PnT DevOps
Red Hat Czech s.r.o
T: +420532270289 IRC: bkhomuts
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Fedora 32 system-wide change proposal: reduce installation media size by improving the compression ratio of SquashFS filesystem

[Chris Murphy]

On Sun, Jan 19, 2020 at 8:41 AM Bohdan Khomutskyi  wrote:
>
> Hello,
>
> Thanks everyone for posting feedback.
> More benchmarking results are available at 
> https://fedoraproject.org/wiki/Category:Changes/OptimizeSquashFS, including 
> the 'plain' SquashFS filesystem.
> After performing the tests, I personally recommend to use xz compression with 
> 1MiB block size, without bcj, on a 'plain' squash filesystem -- this will 
> lead to a reduction of 142MiB on the ISO, compared to the stock Fedora 31 
> Workstation x86_64 image.
> Alternative compression options, such as Zstd, are also mentioned in the 
> change proposal.

Thanks for all the tests.

While I see the meaningfully reduced CPU hit of xz compressed images,
the proposal leaves a lot of performance improvement on the table by
not also enabling zstd as an option in the compose process. The tests
show zstd results, but the proposal doesn't mention zstd at all.

In particular for Workstation ISO, the CPU hit isn't worth the size
savings for regular users, let alone the recurring hit for releng
composes and QA's automated installation tests. It's a lot of CPU burn
at both ends of the candle, for not a lot of size savings. I'm not
convinced it's worth the extra hit on the create side for Zstd level
22, compared to Zstd 15 or 17.

I admit I'm biased toward the two endpoints: create and consume, not
distribution ,i.e the mirror donors. Their storage and bandwidth
concerns were evaluated with the RPM change from xz to zstd. So I'm
mystified by the bias for image size.

Anyway, I approve of the change but disappointed if it really doesn't
let Fedora release engineering the ability to choose (possibly based
on image type - maybe there's some benefit to using xz for raw and
qcow2 images).



-- 
Chris Murphy
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Fedora 32 system-wide change proposal: reduce installation media size by improving the compression ratio of SquashFS filesystem

[Bohdan Khomutskyi]

Hello,

Thanks everyone for posting feedback.
More benchmarking results are available at
https://fedoraproject.org/wiki/Category:Changes/OptimizeSquashFS, including
the 'plain' SquashFS filesystem.
After performing the tests, I personally recommend to use xz compression
with 1MiB block size, without bcj, on a 'plain' squash filesystem -- this
will lead to a reduction of 142MiB on the ISO, compared to the stock Fedora
31 Workstation x86_64 image.
Alternative compression options, such as Zstd, are also mentioned in the
change proposal.

Select re-packaged ISOs of Fedora 31 Workstation x86-64 is available for
download at https://khomutsky.com/fedora-dvd/

On Mon, Jan 13, 2020 at 5:34 PM Kamil Paral  wrote:

> On Sun, Jan 12, 2020 at 5:46 PM Bohdan Khomutskyi 
> wrote:
>
>> Hello,
>>
>> I posted more benchmark results in this article:
>> https://fedoraproject.org/wiki/Category:Changes/OptimizeSquashFS
>>
>> In short, bigger block size and higher compression ratio does not
>> increase the installation time for Fedora Workstation.  I saw the
>> opposite effect.
>> The Zstd compression performed worse than XZ in the compression test. On
>> the other hand, 40% lower installation time for Zstd, was documented. Along
>> with the CPU consumption 37% lower.
>> All installation tests were performed from and to local NVMe storage.
>> Which I consider far from real life scenario.
>>
>
> This is very interesting, thank you!
>
> The "CPU user time" should be independent on the number of CPU cores you
> have, is that correct? I.e. the number should be always roughly the same,
> whether you run it on 1 core, 2 cores or 8 cores, right? I'm asking because
> our QA tests often use 1-2 cores for installation, and I assume you used
> all your available cores (if I read it correctly, you seem to have a 4 core
> system), therefore the "real time" value is applicable just to your system,
> but the "cpu user time" should be better comparable to other systems.
>
> How exactly did you measure those numbers, can you please provide
> reproduction steps?
>
> I'm quite surprised that plain squashfs is a bit smaller, but also a bit
> slower than squashfs+ext4. Our expectations were that it would be faster.
>
> Looking at compressions, the most interesting results for me are:
> -comp xz, without -Xbcj x86 --- cutting CPU time by 50% at the expense of
> 30MB is awesome
> -Xdict-size 1M -b 1M, without -Xbcj x86 (optionally with hardlinking) ---
> 33% speedup while also saving 110 MB
> -comp zstd -Xcompression 15 -b 1M --- blazing fast installation with
> cutting CPU time by 80%, but also increasing the size by 150 MB
>
> I'm sure different people will have different priorities regarding size
> and installation time, but these are really interesting numbers, thanks for
> benchmarking.
>
> ___
> devel mailing list -- devel@lists.fedoraproject.org
> To unsubscribe send an email to devel-le...@lists.fedoraproject.org
> Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
> https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
>


-- 
Bohdan Khomutskyi, RHCE
Release configuration management engineer, PnT DevOps
Red Hat Czech s.r.o
T: +420532270289 IRC: bkhomuts
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Fedora 32 system-wide change proposal: reduce installation media size by improving the compression ratio of SquashFS filesystem

[Kamil Paral]

On Sun, Jan 12, 2020 at 5:46 PM Bohdan Khomutskyi 
wrote:

> Hello,
>
> I posted more benchmark results in this article:
> https://fedoraproject.org/wiki/Category:Changes/OptimizeSquashFS
>
> In short, bigger block size and higher compression ratio does not increase
> the installation time for Fedora Workstation.  I saw the opposite effect.
> The Zstd compression performed worse than XZ in the compression test. On
> the other hand, 40% lower installation time for Zstd, was documented. Along
> with the CPU consumption 37% lower.
> All installation tests were performed from and to local NVMe storage.
> Which I consider far from real life scenario.
>

This is very interesting, thank you!

The "CPU user time" should be independent on the number of CPU cores you
have, is that correct? I.e. the number should be always roughly the same,
whether you run it on 1 core, 2 cores or 8 cores, right? I'm asking because
our QA tests often use 1-2 cores for installation, and I assume you used
all your available cores (if I read it correctly, you seem to have a 4 core
system), therefore the "real time" value is applicable just to your system,
but the "cpu user time" should be better comparable to other systems.

How exactly did you measure those numbers, can you please provide
reproduction steps?

I'm quite surprised that plain squashfs is a bit smaller, but also a bit
slower than squashfs+ext4. Our expectations were that it would be faster.

Looking at compressions, the most interesting results for me are:
-comp xz, without -Xbcj x86 --- cutting CPU time by 50% at the expense of
30MB is awesome
-Xdict-size 1M -b 1M, without -Xbcj x86 (optionally with hardlinking) ---
33% speedup while also saving 110 MB
-comp zstd -Xcompression 15 -b 1M --- blazing fast installation with
cutting CPU time by 80%, but also increasing the size by 150 MB

I'm sure different people will have different priorities regarding size and
installation time, but these are really interesting numbers, thanks for
benchmarking.
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Fedora 32 system-wide change proposal: reduce installation media size by improving the compression ratio of SquashFS filesystem

[Chris Murphy]

On Sun, Jan 12, 2020 at 3:53 PM Samuel Sieb  wrote:
>
> On 1/7/20 11:16 AM, Kevin Kofler wrote:
> > Chris Murphy wrote:
> >> Stacked images on the same media functionality is in the kernel, it's
> >> not complicated, it's well tested, doesn't require any gymnastics in
> >> the initramfs - your bootloader entries can each point to different
> >> root=UUIDs and image assembly is figured out entirely in kernel code,
> >> no special handling in the client side deliverable. Yes the image
> >> creator needs to know some things to achieve this.
> >>
> >> Why stacked images? Consider a single base.img that's maybe 1G, and
> >> now you don't have to do separate composes for server, cloud, GNOME,
> >> KDE, Cinnamon, LXQt, Astronomy that repeat a lot of the same steps,
> >> including expensive steps like compressing the same things over and
> >> over again. Just do a 'dnf group install' tacked onto that base.img,
> >> the work being done is custom for that output, rather than repetitive.
> >> Not complicated. It would be fast enough that the high level variants
> >> could be composed on demand. Seconds. It'd be fast enough to queue it
> >> for download within the hour.
> >
> > Then how do you deliver the stacked images? Either the user still needs to
> > download base.img + the specific image the user actually wants, either as 2
> > downloads (but then how does the user reliably get them onto bootable media?
> > Surely you don't want to require 2 media!) or as 1 combined download, or you
> > ship one image with base.img and all the specific layers at once, which will
> > waste a lot of download size for all the images the user does not care
> > about. I do not see what use case would be served by stacked images.
> >
> > What would be a much more useful feature is hybrid netinstall, i.e.,
> > allowing liveinst to netinstall additional packages on top of the installed
> > live image. See the Calamares netinstall module (e.g., on my old Kannolo 27
> > images, as long as I don't have newer ones) for how the user experience can
> > look like. And that requires only installer support, no file system or
> > compression support.
>
> At first I did think the same thing as you did, that it would have all
> of them.  But my understanding of what he was suggesting is similar to
> your suggestion.  There's a base image and then for each spin, there's
> an extra stacked image that contains whatever extra is needed for that
> spin.  So there's a separate iso for each spin, but each one has the
> base image plus (at least) one other.

The ISO contains a base.img + the diff.img that makes that particular
edition/spin ISO unique. Plausibly there's more than one diff file on
an ISO depending on the desired level of granularity, and whether it's
useful to have more than one bootable option per ISO. A single USB
stick could get imaged with a single ISO file, capable of installing
all the editions.

But yeah, netinstall is a much faster and easier way to get to a one
off custom "spin". A portal that basically helps create a kickstart or
updates.img file for Anaconda to consume. And viola. Custom installer.
The gotcha there is: it's not Live demo media, and to install multiple
times you either take multiple download hits, or you have to setup a
local mirror.

-- 
Chris Murphy
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Fedora 32 system-wide change proposal: reduce installation media size by improving the compression ratio of SquashFS filesystem

[Samuel Sieb]

On 1/7/20 11:16 AM, Kevin Kofler wrote:

Chris Murphy wrote:

Stacked images on the same media functionality is in the kernel, it's
not complicated, it's well tested, doesn't require any gymnastics in
the initramfs - your bootloader entries can each point to different
root=UUIDs and image assembly is figured out entirely in kernel code,
no special handling in the client side deliverable. Yes the image
creator needs to know some things to achieve this.

Why stacked images? Consider a single base.img that's maybe 1G, and
now you don't have to do separate composes for server, cloud, GNOME,
KDE, Cinnamon, LXQt, Astronomy that repeat a lot of the same steps,
including expensive steps like compressing the same things over and
over again. Just do a 'dnf group install' tacked onto that base.img,
the work being done is custom for that output, rather than repetitive.
Not complicated. It would be fast enough that the high level variants
could be composed on demand. Seconds. It'd be fast enough to queue it
for download within the hour.


Then how do you deliver the stacked images? Either the user still needs to
download base.img + the specific image the user actually wants, either as 2
downloads (but then how does the user reliably get them onto bootable media?
Surely you don't want to require 2 media!) or as 1 combined download, or you
ship one image with base.img and all the specific layers at once, which will
waste a lot of download size for all the images the user does not care
about. I do not see what use case would be served by stacked images.

What would be a much more useful feature is hybrid netinstall, i.e.,
allowing liveinst to netinstall additional packages on top of the installed
live image. See the Calamares netinstall module (e.g., on my old Kannolo 27
images, as long as I don't have newer ones) for how the user experience can
look like. And that requires only installer support, no file system or
compression support.


At first I did think the same thing as you did, that it would have all 
of them.  But my understanding of what he was suggesting is similar to 
your suggestion.  There's a base image and then for each spin, there's 
an extra stacked image that contains whatever extra is needed for that 
spin.  So there's a separate iso for each spin, but each one has the 
base image plus (at least) one other.

___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Fedora 32 system-wide change proposal: reduce installation media size by improving the compression ratio of SquashFS filesystem

[Chris Murphy]

On Sun, Jan 12, 2020 at 9:45 AM Bohdan Khomutskyi 
wrote:

> Hello,
>
> I posted more benchmark results in this article:
> https://fedoraproject.org/wiki/Category:Changes/OptimizeSquashFS
>

Cool!

Do you have any tests to compare plain squashfs xz with zstd? The nested
ext4 stuff is really pointless now because Fedora hasn't used 'dd' +
resizing the ext4 file system as an installation method in a long time
(going back to Fedora 18 I think). All of the Live installations use rsync.


The Zstd compression performed worse than XZ in the compression test. On
the other hand, 40% lower installation time for Zstd, was documented. Along
with the CPU consumption 37% lower.
All installation tests were performed from and to local NVMe storage. Which
I consider far from real life scenario.

Fedora QA nightly tests are real and I think it'll make a meaningful impact
for both the creation of the ISOs, as well as their consumption, in a lot
of cases. Even if it doesn't impact USB installations. I do VM installs on
both SSD and NVMe and and it matters there. But also the power consumption
of xz I think is relevant whether baremetal or virtual.

Thanks!


-- 
Chris Murphy
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Fedora 32 system-wide change proposal: reduce installation media size by improving the compression ratio of SquashFS filesystem

[Kevin Fenzi]

On Sun, Jan 12, 2020 at 05:44:33PM +0100, Bohdan Khomutskyi wrote:
> Hello,
> 
> I posted more benchmark results in this article:
> https://fedoraproject.org/wiki/Category:Changes/OptimizeSquashFS
> 
> In short, bigger block size and higher compression ratio does not increase
> the installation time for Fedora Workstation.  I saw the opposite effect.
> The Zstd compression performed worse than XZ in the compression test. On
> the other hand, 40% lower installation time for Zstd, was documented. Along
> with the CPU consumption 37% lower.
> All installation tests were performed from and to local NVMe storage. Which
> I consider far from real life scenario.
> 
> I plan to perform more testing, with slower installation media, and will
> post the results next weekend.
> I did not evaluate CONFIG_SQUASHFS_DECOMP_MULTI this weekend.

Many thanks for doing this testing. It's really appreciated as well as
driving this change forward. :) 

kevin


signature.asc
Description: PGP signature
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Fedora 32 system-wide change proposal: reduce installation media size by improving the compression ratio of SquashFS filesystem

[Bohdan Khomutskyi]

Hello,

I posted more benchmark results in this article:
https://fedoraproject.org/wiki/Category:Changes/OptimizeSquashFS

In short, bigger block size and higher compression ratio does not increase
the installation time for Fedora Workstation.  I saw the opposite effect.
The Zstd compression performed worse than XZ in the compression test. On
the other hand, 40% lower installation time for Zstd, was documented. Along
with the CPU consumption 37% lower.
All installation tests were performed from and to local NVMe storage. Which
I consider far from real life scenario.

I plan to perform more testing, with slower installation media, and will
post the results next weekend.
I did not evaluate CONFIG_SQUASHFS_DECOMP_MULTI this weekend.

On Sat, Jan 11, 2020 at 4:38 PM Bohdan Khomutskyi 
wrote:

> Thanks everyone for your comments. These are all valid concerns.
>
> I filed a new change proposal at
> https://fedoraproject.org/wiki/Category:Changes/OptimizeSquashFS.
>
> I'll run more benchmarks, including using Zstd compression algorithm, and
> will post results.
> Hopefully this weekend, I'll also try to measure the impact of the
> compression, block size versus installation time.
>
> Also, the decompression of SquashFS is currently happening in single
> thread. This could be changed with CONFIG_SQUASHFS_DECOMP_MULTI_PERCPU
> kernel build-time configuration option. I'll file a new change proposal for
> it, along with benchmarks.
>
> On Thu, Jan 9, 2020 at 11:13 AM Lukas Ruzicka  wrote:
>
>>
>>
>>> Why stacked images? Consider a single base.img that's maybe 1G, and
>>> now you don't have to do separate composes for server, cloud, GNOME,
>>> KDE, Cinnamon, LXQt, Astronomy that repeat a lot of the same steps,
>>> including expensive steps like compressing the same things over and
>>> over again. Just do a 'dnf group install' tacked onto that base.img,
>>> the work being done is custom for that output, rather than repetitive.
>>> Not complicated. It would be fast enough that the high level variants
>>> could be composed on demand. Seconds. It'd be fast enough to queue it
>>> for download within the hour.
>>>
>>
>> Well, I would like that.
>> +1
>>
>>
>>
>>>
>>>
>>> --
>>> Chris Murphy
>>> ___
>>> devel mailing list -- devel@lists.fedoraproject.org
>>> To unsubscribe send an email to devel-le...@lists.fedoraproject.org
>>> Fedora Code of Conduct:
>>> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>>> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>>> List Archives:
>>> https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
>>>
>>
>>
>> --
>>
>> Lukáš Růžička
>>
>> FEDORA QE, RHCE
>>
>> Red Hat
>>
>> 
>>
>> Purkyňova 115
>>
>> 612 45 Brno - Královo Pole
>>
>> lruzi...@redhat.com
>> TRIED AND PERSONALLY TESTED, ERGO TRUSTED. 
>> ___
>> devel mailing list -- devel@lists.fedoraproject.org
>> To unsubscribe send an email to devel-le...@lists.fedoraproject.org
>> Fedora Code of Conduct:
>> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>> List Archives:
>> https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
>>
>
>
> --
> Bohdan Khomutskyi, RHCE
> Release configuration management engineer, PnT DevOps
> Red Hat Czech s.r.o
> T: +420532270289 IRC: bkhomuts
>


-- 
Bohdan Khomutskyi, RHCE
Release configuration management engineer, PnT DevOps
Red Hat Czech s.r.o
T: +420532270289 IRC: bkhomuts
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Fedora 32 system-wide change proposal: reduce installation media size by improving the compression ratio of SquashFS filesystem

[Bohdan Khomutskyi]

Thanks everyone for your comments. These are all valid concerns.

I filed a new change proposal at
https://fedoraproject.org/wiki/Category:Changes/OptimizeSquashFS.

I'll run more benchmarks, including using Zstd compression algorithm, and
will post results.
Hopefully this weekend, I'll also try to measure the impact of the
compression, block size versus installation time.

Also, the decompression of SquashFS is currently happening in single
thread. This could be changed with CONFIG_SQUASHFS_DECOMP_MULTI_PERCPU
kernel build-time configuration option. I'll file a new change proposal for
it, along with benchmarks.

On Thu, Jan 9, 2020 at 11:13 AM Lukas Ruzicka  wrote:

>
>
>> Why stacked images? Consider a single base.img that's maybe 1G, and
>> now you don't have to do separate composes for server, cloud, GNOME,
>> KDE, Cinnamon, LXQt, Astronomy that repeat a lot of the same steps,
>> including expensive steps like compressing the same things over and
>> over again. Just do a 'dnf group install' tacked onto that base.img,
>> the work being done is custom for that output, rather than repetitive.
>> Not complicated. It would be fast enough that the high level variants
>> could be composed on demand. Seconds. It'd be fast enough to queue it
>> for download within the hour.
>>
>
> Well, I would like that.
> +1
>
>
>
>>
>>
>> --
>> Chris Murphy
>> ___
>> devel mailing list -- devel@lists.fedoraproject.org
>> To unsubscribe send an email to devel-le...@lists.fedoraproject.org
>> Fedora Code of Conduct:
>> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>> List Archives:
>> https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
>>
>
>
> --
>
> Lukáš Růžička
>
> FEDORA QE, RHCE
>
> Red Hat
>
> 
>
> Purkyňova 115
>
> 612 45 Brno - Královo Pole
>
> lruzi...@redhat.com
> TRIED AND PERSONALLY TESTED, ERGO TRUSTED. 
> ___
> devel mailing list -- devel@lists.fedoraproject.org
> To unsubscribe send an email to devel-le...@lists.fedoraproject.org
> Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
> https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
>


-- 
Bohdan Khomutskyi, RHCE
Release configuration management engineer, PnT DevOps
Red Hat Czech s.r.o
T: +420532270289 IRC: bkhomuts
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Fedora 32 system-wide change proposal: reduce installation media size by improving the compression ratio of SquashFS filesystem

[Lukas Ruzicka]

>
> Why stacked images? Consider a single base.img that's maybe 1G, and
> now you don't have to do separate composes for server, cloud, GNOME,
> KDE, Cinnamon, LXQt, Astronomy that repeat a lot of the same steps,
> including expensive steps like compressing the same things over and
> over again. Just do a 'dnf group install' tacked onto that base.img,
> the work being done is custom for that output, rather than repetitive.
> Not complicated. It would be fast enough that the high level variants
> could be composed on demand. Seconds. It'd be fast enough to queue it
> for download within the hour.
>

Well, I would like that.
+1



>
>
> --
> Chris Murphy
> ___
> devel mailing list -- devel@lists.fedoraproject.org
> To unsubscribe send an email to devel-le...@lists.fedoraproject.org
> Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
> https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
>


-- 

Lukáš Růžička

FEDORA QE, RHCE

Red Hat



Purkyňova 115

612 45 Brno - Královo Pole

lruzi...@redhat.com
TRIED AND PERSONALLY TESTED, ERGO TRUSTED. 
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Fedora 32 system-wide change proposal: reduce installation media size by improving the compression ratio of SquashFS filesystem

[Richard Hughes]

On Tue, 7 Jan 2020 at 18:07, Martin Kolman  wrote:
> IIRC the speedups in compression and decompression
> speed we got for RPMs[0] with zstd were pretty nice

If it helps the argument, at the moment 99.7% of the time building the
AppStream metadata is spent decompressing the RPMs. If zstd helps with
that *at all*, I'm a huge proponent.

Richard
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Fedora 32 system-wide change proposal: reduce installation media size by improving the compression ratio of SquashFS filesystem

[Colin Walters]

On Sun, Jan 5, 2020, at 12:08 PM, Chris Murphy wrote:

> I've pretty much concluded Fedora is best off dropping the nested ext4
> in favor of plain squashfs, and using zstd. 

Fedora CoreOS already uses zstd for squashfs:
https://github.com/coreos/coreos-assembler/blob/master/src/cmd-buildextend-installer#L57

(I'm also working to rebase Fedora Silverblue on Fedora CoreOS' toolchain and 
I'd like the same to happen for IoT, which would mean this proposal should more 
clearly call out it's affecting Fedora images that use Anaconda)
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Fedora 32 system-wide change proposal: reduce installation media size by improving the compression ratio of SquashFS filesystem

[Kamil Paral]

On Tue, Jan 7, 2020 at 4:21 PM Kevin Kofler  wrote:

> Kamil Paral wrote:
> > Well for the general user, everything is one-time. One download, one
> write
> > to USB, one install. Saving a minute in one step and adding it to a
> > different step doesn't really matter, it's the same sum overall (unless
> > you pay considerable money for the extra downloaded data, of course).
>
> But the larger download will take several minutes extra even on a low-end
> "broadband" connection. On slower connections, which are still standard in
> parts of the world, it will take hours longer.
>

Sorry, but your argument is just wrong. We've had a similar discussion
regarding RPM payload compression and so we know we're talking about small
percent number increase by changing the compression, if any. That means a
few tens of MBs for e.g. the Workstation image. And if you spend *hours* to
download the extra 50 MBs, that means you're on a dial-up connection and
the whole image would take you a *week* to download. This whole example is
simply unrealistic. Anyone who has a problem to download extra 50-100 MBs
can hardly use Fedora at all, because even the first dnf metadata update
will consume exactly this amount of data, and then they will be presented
with 1 GB worth of system updates.

Not to mention we're talking here about removing the nested ext4
filesystem, which is likely to *reduce* the image size (and combined with
changing the compression type can equal out to no change at all).

It makes no sense to pre-emptively hate the discussed changes. Let's try
them and then discuss the cost/benefit of the output with actual numbers in
hand.
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Fedora 32 system-wide change proposal: reduce installation media size by improving the compression ratio of SquashFS filesystem

[Kevin Kofler]

Chris Murphy wrote:
> Even at 8% bigger it would be worth it. And probably 16%.

I disagree. We need to stop treating bloat like a feature.

And please see my other replies for why this is a particularly bad tradeoff 
in this particular case.

> Gaining additional features, like on the fly checksumming is worth
> considering (at least not making it harder to implement in the future,
> by taking it into account with the work implied in this proposal). The
> monolithic ISO check is terrible. It's dog slow. It's optional. And
> it's a one time check. Typically real optical media tends to work or
> persistently fail; whereas USB sticks can have transient bad reads
> (explicit or silently corrupt).

Can't we just drop the mediacheck entirely? It is optional for a reason.

> Stacked images on the same media functionality is in the kernel, it's
> not complicated, it's well tested, doesn't require any gymnastics in
> the initramfs - your bootloader entries can each point to different
> root=UUIDs and image assembly is figured out entirely in kernel code,
> no special handling in the client side deliverable. Yes the image
> creator needs to know some things to achieve this.
>
> Why stacked images? Consider a single base.img that's maybe 1G, and
> now you don't have to do separate composes for server, cloud, GNOME,
> KDE, Cinnamon, LXQt, Astronomy that repeat a lot of the same steps,
> including expensive steps like compressing the same things over and
> over again. Just do a 'dnf group install' tacked onto that base.img,
> the work being done is custom for that output, rather than repetitive.
> Not complicated. It would be fast enough that the high level variants
> could be composed on demand. Seconds. It'd be fast enough to queue it
> for download within the hour.

Then how do you deliver the stacked images? Either the user still needs to 
download base.img + the specific image the user actually wants, either as 2 
downloads (but then how does the user reliably get them onto bootable media? 
Surely you don't want to require 2 media!) or as 1 combined download, or you 
ship one image with base.img and all the specific layers at once, which will 
waste a lot of download size for all the images the user does not care 
about. I do not see what use case would be served by stacked images.

What would be a much more useful feature is hybrid netinstall, i.e., 
allowing liveinst to netinstall additional packages on top of the installed 
live image. See the Calamares netinstall module (e.g., on my old Kannolo 27 
images, as long as I don't have newer ones) for how the user experience can 
look like. And that requires only installer support, no file system or 
compression support.

Kevin Kofler
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Fedora 32 system-wide change proposal: reduce installation media size by improving the compression ratio of SquashFS filesystem

[Kevin Kofler]

Gary Buhrmaster wrote:
> While not exactly the same, the measured increase in size
> by the Arch community for their packaging by moving from
> xz to zstd was ~0.8% (and gaining a huge reduction in CPU
> utilization at the decompress end).

I don't know what xz settings Arch was using, but in the case of our RPMs, 
xz was being used with very conservative settings, mainly so that applying 
DeltaRPMs (which recompresses the RPMs) can be done in a reasonable time 
(and by the way, IIRC, the switch to zstd actually slows down that use 
case!), though decompression time was also a criterion. That's why switching 
to zstd was not a huge size increase. But we could have saved significant 
size by using higher xz compression rates.

If Arch was using similar settings, that would explain the relatively small 
size increase. But it is still a size increase.

Kevin Kofler
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Fedora 32 system-wide change proposal: reduce installation media size by improving the compression ratio of SquashFS filesystem

[Kevin Kofler]

Chris Murphy wrote:
> It's untenable to consider ISO size alone. It is a legitimate concern, but
> it can't be reasonable to soak every single CPU, times  thousands. You're
> willing to exchange less download time for longer install time and higher
> energy demand, but there are quite a lot of other uses occurring that are
> relevant.

Downloads also require energy, on the client, on the server, on the 
intermediate hops, and even for the actual information transmission in the 
cables. So I am not convinced that you are going to save any energy by 
making the image significantly larger just so that it is faster to 
decompress.

(As for the time factor, I already explained how that does not compute 
either, at least in large, less-privileged parts of the world.)

Kevin Kofler
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Fedora 32 system-wide change proposal: reduce installation media size by improving the compression ratio of SquashFS filesystem

[Kevin Kofler]

Brian C. Lane wrote:
> Yes, according to the manpage it supports xattrs.

Does that include file capabilities and ACLs?

Kevin Kofler
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Fedora 32 system-wide change proposal: reduce installation media size by improving the compression ratio of SquashFS filesystem

[Brian C. Lane]

On Tue, Jan 07, 2020 at 09:56:21AM +0100, Kevin Kofler wrote:
> Brian C. Lane wrote:
> > I agree with Chris here, I think we should make the switch to plain
> > squashfs unless someone can come up something dramatic that it will
> > break :)
> 
> Does SquashFS support all the advanced features that are needed, such as 
> extended attributes (used at least by SELinux), file system capabilities, 
> etc.?

Yes, according to the manpage it supports xattrs.

-- 
Brian C. Lane (PST8PDT) - weldr.io - lorax - parted - pykickstart
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Fedora 32 system-wide change proposal: reduce installation media size by improving the compression ratio of SquashFS filesystem

[Chris Murphy]

On Tue, Jan 7, 2020 at 9:58 AM Gary Buhrmaster
 wrote:
>
> On Tue, Jan 7, 2020 at 9:00 AM Kevin Kofler  wrote:
>
> > I think increasing the size of the live images, also affecting the download
> > time and the time to write the image to media (even USB sticks are not
> > instant), to get a one-time installation speedup is a very bad tradeoff.
>
> While not exactly the same, the measured increase in size
> by the Arch community for their packaging by moving from
> xz to zstd was ~0.8% (and gaining a huge reduction in CPU
> utilization at the decompress end).
>
> If those (approximate) numbers hold for this use case
> (someone would clearly have to test to confirm or
> refute those numbers) I would have to suggest that is
> likely a good tradeoff that is worth further consideration.

Even at 8% bigger it would be worth it. And probably 16%.

Gaining additional features, like on the fly checksumming is worth
considering (at least not making it harder to implement in the future,
by taking it into account with the work implied in this proposal). The
monolithic ISO check is terrible. It's dog slow. It's optional. And
it's a one time check. Typically real optical media tends to work or
persistently fail; whereas USB sticks can have transient bad reads
(explicit or silently corrupt).

Stacked images on the same media functionality is in the kernel, it's
not complicated, it's well tested, doesn't require any gymnastics in
the initramfs - your bootloader entries can each point to different
root=UUIDs and image assembly is figured out entirely in kernel code,
no special handling in the client side deliverable. Yes the image
creator needs to know some things to achieve this.

Why stacked images? Consider a single base.img that's maybe 1G, and
now you don't have to do separate composes for server, cloud, GNOME,
KDE, Cinnamon, LXQt, Astronomy that repeat a lot of the same steps,
including expensive steps like compressing the same things over and
over again. Just do a 'dnf group install' tacked onto that base.img,
the work being done is custom for that output, rather than repetitive.
Not complicated. It would be fast enough that the high level variants
could be composed on demand. Seconds. It'd be fast enough to queue it
for download within the hour.


-- 
Chris Murphy
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Fedora 32 system-wide change proposal: reduce installation media size by improving the compression ratio of SquashFS filesystem

[Martin Kolman]

On Tue, 2020-01-07 at 11:20 -0700, Chris Murphy wrote:
> On Tue, Jan 7, 2020 at 11:07 AM Martin Kolman  wrote:
> > On Mon, 2020-01-06 at 16:35 -0800, Brian C. Lane wrote:
> > > On Sun, Jan 05, 2020 at 10:08:07AM -0700, Chris Murphy wrote:
> > > > I've pretty much concluded Fedora is best off dropping the nested ext4
> > > > in favor of plain squashfs, and using zstd. It's not required to do
> > > > both, but the benefit is additive and significant. The work in dracut
> > > > and lorax to support plain squashfs, assembling it using overlayfs
> > > > instead of device-mapper is already done, and tested.
> > > 
> > > I agree with Chris here, I think we should make the switch to plain
> > > squashfs unless someone can come up something dramatic that it will
> > > break :) Tweaking the current settings would be fine if we didn't have a
> > > better, simpler, solution.
> > > 
> > > A side note about the xz bcj compression -- in some experiments I
> > > noticed that enabling x86 and armthumb resulted in further reduction
> > > (about 400k with the default block size). My guess was due to use of ARM
> > > instructions in the firmware blobs.
> > Also does squashfs support zstd compression ?
> 
> Yes, that's what I was referring to in the first sentence quoted above.
Oh right, now I see it. :D In any case, nice! :)

> 
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Fedora 32 system-wide change proposal: reduce installation media size by improving the compression ratio of SquashFS filesystem

[Chris Murphy]

On Tue, Jan 7, 2020 at 11:07 AM Martin Kolman  wrote:
>
> On Mon, 2020-01-06 at 16:35 -0800, Brian C. Lane wrote:
> > On Sun, Jan 05, 2020 at 10:08:07AM -0700, Chris Murphy wrote:
> > > I've pretty much concluded Fedora is best off dropping the nested ext4
> > > in favor of plain squashfs, and using zstd. It's not required to do
> > > both, but the benefit is additive and significant. The work in dracut
> > > and lorax to support plain squashfs, assembling it using overlayfs
> > > instead of device-mapper is already done, and tested.
> >
> > I agree with Chris here, I think we should make the switch to plain
> > squashfs unless someone can come up something dramatic that it will
> > break :) Tweaking the current settings would be fine if we didn't have a
> > better, simpler, solution.
> >
> > A side note about the xz bcj compression -- in some experiments I
> > noticed that enabling x86 and armthumb resulted in further reduction
> > (about 400k with the default block size). My guess was due to use of ARM
> > instructions in the firmware blobs.
> Also does squashfs support zstd compression ?

Yes, that's what I was referring to in the first sentence quoted above.

-- 
Chris Murphy
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Fedora 32 system-wide change proposal: reduce installation media size by improving the compression ratio of SquashFS filesystem

[Martin Kolman]

On Mon, 2020-01-06 at 16:35 -0800, Brian C. Lane wrote:
> On Sun, Jan 05, 2020 at 10:08:07AM -0700, Chris Murphy wrote:
> > I've pretty much concluded Fedora is best off dropping the nested ext4
> > in favor of plain squashfs, and using zstd. It's not required to do
> > both, but the benefit is additive and significant. The work in dracut
> > and lorax to support plain squashfs, assembling it using overlayfs
> > instead of device-mapper is already done, and tested.
> 
> I agree with Chris here, I think we should make the switch to plain
> squashfs unless someone can come up something dramatic that it will
> break :) Tweaking the current settings would be fine if we didn't have a
> better, simpler, solution.
> 
> A side note about the xz bcj compression -- in some experiments I
> noticed that enabling x86 and armthumb resulted in further reduction
> (about 400k with the default block size). My guess was due to use of ARM
> instructions in the firmware blobs.
Also does squashfs support zstd compression ? IIRC the speedups in compression 
and decompression
speed we got for RPMs[0] with zstd were pretty nice, so having the same for the 
media would
be nice as well. :)

[0] https://fedoraproject.org/wiki/Changes/Switch_RPMs_to_zstd_compression

> 
> -- 
> Brian C. Lane (PST8PDT) - weldr.io - lorax - parted - pykickstart
> ___
> devel mailing list -- devel@lists.fedoraproject.org
> To unsubscribe send an email to devel-le...@lists.fedoraproject.org
> Fedora Code of Conduct: 
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: 
> https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Fedora 32 system-wide change proposal: reduce installation media size by improving the compression ratio of SquashFS filesystem

[Gary Buhrmaster]

On Tue, Jan 7, 2020 at 9:00 AM Kevin Kofler  wrote:

> I think increasing the size of the live images, also affecting the download
> time and the time to write the image to media (even USB sticks are not
> instant), to get a one-time installation speedup is a very bad tradeoff.

While not exactly the same, the measured increase in size
by the Arch community for their packaging by moving from
xz to zstd was ~0.8% (and gaining a huge reduction in CPU
utilization at the decompress end).

If those (approximate) numbers hold for this use case
(someone would clearly have to test to confirm or
refute those numbers) I would have to suggest that is
likely a good tradeoff that is worth further consideration.
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Fedora 32 system-wide change proposal: reduce installation media size by improving the compression ratio of SquashFS filesystem

[Chris Murphy]

It's untenable to consider ISO size alone. It is a legitimate concern, but it 
can't be reasonable to soak every single CPU, times  thousands. You're willing 
to exchange less download time for longer install time and higher energy 
demand, but there are quite a lot of other uses occurring that are relevant.

I don't have an immediate breakdown but RPM switching from xz to zstd increased 
RPM sizes somewhat, but the installation performance makes up for the extra 
download time.

--
Chris Murphy
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Fedora 32 system-wide change proposal: reduce installation media size by improving the compression ratio of SquashFS filesystem

[Kevin Kofler]

Kamil Paral wrote:
> Well for the general user, everything is one-time. One download, one write
> to USB, one install. Saving a minute in one step and adding it to a
> different step doesn't really matter, it's the same sum overall (unless
> you pay considerable money for the extra downloaded data, of course).

But the larger download will take several minutes extra even on a low-end 
"broadband" connection. On slower connections, which are still standard in 
parts of the world, it will take hours longer.

Kevin Kofler
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Fedora 32 system-wide change proposal: reduce installation media size by improving the compression ratio of SquashFS filesystem

[Kamil Paral]

On Tue, Jan 7, 2020 at 10:01 AM Kevin Kofler  wrote:

> Chris Murphy wrote:
> > Has zstandard been evaluated? In my testing of images compressed with
> > zstd, the CPU hit is cut by more than 50%, and is no longer a
> > bottleneck during installations. Image size does increase, although I
> > haven't tested mksquashfs block size higher than 256K.
>
> I think increasing the size of the live images, also affecting the
> download
> time and the time to write the image to media (even USB sticks are not
> instant), to get a one-time installation speedup is a very bad tradeoff.
>

Well for the general user, everything is one-time. One download, one write
to USB, one install. Saving a minute in one step and adding it to a
different step doesn't really matter, it's the same sum overall (unless you
pay considerable money for the extra downloaded data, of course). Where it
matters is when you do a high amount of operations. And those operations
are likely to be installations. Either in some school lab, where you
install 20 machines, or, in our very specific example, when you perform
automated testing/CI as part of the release process, and perform tens or
hundreds of installation every single day. The time difference (and CPU
usage difference) saved during installation gets really noticeable in such
cases.
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Fedora 32 system-wide change proposal: reduce installation media size by improving the compression ratio of SquashFS filesystem

[Kevin Kofler]

Chris Murphy wrote:
> Has zstandard been evaluated? In my testing of images compressed with
> zstd, the CPU hit is cut by more than 50%, and is no longer a
> bottleneck during installations. Image size does increase, although I
> haven't tested mksquashfs block size higher than 256K.

I think increasing the size of the live images, also affecting the download 
time and the time to write the image to media (even USB sticks are not 
instant), to get a one-time installation speedup is a very bad tradeoff.

Kevin Kofler
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Fedora 32 system-wide change proposal: reduce installation media size by improving the compression ratio of SquashFS filesystem

[Kevin Kofler]

Brian C. Lane wrote:
> I agree with Chris here, I think we should make the switch to plain
> squashfs unless someone can come up something dramatic that it will
> break :)

Does SquashFS support all the advanced features that are needed, such as 
extended attributes (used at least by SELinux), file system capabilities, 
etc.?

Kevin Kofler
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Fedora 32 system-wide change proposal: reduce installation media size by improving the compression ratio of SquashFS filesystem

[Brian C. Lane]

On Sun, Jan 05, 2020 at 10:08:07AM -0700, Chris Murphy wrote:
> I've pretty much concluded Fedora is best off dropping the nested ext4
> in favor of plain squashfs, and using zstd. It's not required to do
> both, but the benefit is additive and significant. The work in dracut
> and lorax to support plain squashfs, assembling it using overlayfs
> instead of device-mapper is already done, and tested.

I agree with Chris here, I think we should make the switch to plain
squashfs unless someone can come up something dramatic that it will
break :) Tweaking the current settings would be fine if we didn't have a
better, simpler, solution.

A side note about the xz bcj compression -- in some experiments I
noticed that enabling x86 and armthumb resulted in further reduction
(about 400k with the default block size). My guess was due to use of ARM
instructions in the firmware blobs.

-- 
Brian C. Lane (PST8PDT) - weldr.io - lorax - parted - pykickstart
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Fedora 32 system-wide change proposal: reduce installation media size by improving the compression ratio of SquashFS filesystem

[Kamil Paral]

On Sun, Jan 5, 2020 at 1:24 PM Bohdan Khomutskyi 
wrote:

> Summary
>
> Improve compression ratio of SquashFS filesystem on the installation media.
>

Hi Bohdan, as a member of QA, I'll happily support any proposal that
improves the installation speed (the image size is not that important from
my POV). Chris found that dropping the nested ext4 filesystem can get us
substantial gains in this area, as well as changing xz to zstd. I see he
already provided you with links to the respective tickets. Perhaps you can
work with him to make this change a reality (and benefit in both areas -
speed and size)? QA would really appreciate it :-)
Thanks!
Kamil
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Fedora 32 system-wide change proposal: reduce installation media size by improving the compression ratio of SquashFS filesystem

[Ben Cotton]

On Sun, Jan 5, 2020 at 7:24 AM Bohdan Khomutskyi  wrote:
>
> I was unable to create an article in Fedora wiki system.
>
Since you don't have any non-CLA groups in FAS, I have added you to
the wikiedit group. Please add this to the wiki ASAP. This proposal is
past the deadline for Fedora 32 System-Wide Change proposals, but
since it is only a few days late, I'll continue shepherding it and
FESCo can decide if it's worth granting an exception.

-- 
Ben Cotton
He / Him / His
Fedora Program Manager
Red Hat
TZ=America/Indiana/Indianapolis
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Fedora 32 system-wide change proposal: reduce installation media size by improving the compression ratio of SquashFS filesystem

[Vít Ondruch]

Hi,

it does not look you follow the formal "Changes policy". The process for
submitting change proposals is described here:

https://docs.fedoraproject.org/en-US/program_management/changes_policy/


Vít


Dne 05. 01. 20 v 13:23 Bohdan Khomutskyi napsal(a):
>
>
>   Summary
>
> Improve compression ratio of SquashFS filesystem on the installation
> media.
>
>
>   Owner
>
> Name: Bohdan Khomutskyi
>
> Email: bkhom...@redhat.com 
>
>
>   Current status
>
> Targeted release: I propose this change for Fedora 32
>
> Last updated: Jan 5 2020
>
> Pagure.io issue: https://pagure.io/releng/issue/9127
>
> I was unable to create an article in Fedora wiki system.
>
>
>   Detailed Description
>
> As of Fedora 31, the LiveOS/squashfs.img file on the installation
> image, is compressed with default settings of mksquashfs. The standard
> configuration is set to XZ algorithm with block size of 128k andBCJ
> filter enabled. Those parameters can be adjusted which will lead to a
> better compression ratio and/or reduction of the CPU usage at build time.
>
> This is simple to achieve. Recently, Lorax has gotten support[1] for
> adjusting the compression options for mksquashfs via the configuration
> file. The file should be altered as following:
>
> [compression]
> bcj = yes
> args = -b 1M -Xdict-size 1M -no-recovery
>
> Where -b 1M and -Xdict-size 1M are block and dictionary sizes
> respectively. Could be adjusted.
>
>
>   Benefit to Fedora
>
>  *
>
> Reduction of the installation media size and the cost of storing
> and distributing Fedora.
>
>  *
>
> Reduction of the CPU usage at build time. Depending on which
> compression parameters chosen.
>
>  *
>
> See a graphical detail at https://pagure.io/releng/issue/9127.
>
>
>   Scope
>
>  *
>
> Proposal owners:
>
> The build environment should have support for adjusting the Lorax
> configuration file.. Lorax is a program that produces the
> LiveOS/squashfs.img file on the installation media.
>
> One of the way to allow for such customization, is to add a feature in
> Pungi, to allow for passing -c option to Lorax.
>
>  *
>
> Release engineering: #9127 
>
>  *
>
> Policies and guidelines: N/A
>
>  *
>
> Trademark approval: N/A
>
>
>   Upgrade/compatibility impact
>
>  *
>
> This change comes at a cost of higher memory usage during the
> installation. Based on my personal estimations, this should not be
> the issue. Since the decompression should require up to 1MiB per
> thread.
>
>
>   User Experience
>
>  *
>
> Increasing the block size on the current configuration with EXT4
> file system, should increase latency while accessing the EXT4
> filesystem. The exact impact is to be evaluated.
>
>  *
>
> The impact of latency will be reduced, if the plain SquashFS
> option is be choosen.
>
>
>   Dependencies
>
>  *
>
> N/A
>
>
>   Contingency Plan
>
>  *
>
> N/A
>
>
>   Documentation
>
> https://pagure.io/releng/issue/9127.
>
> mksquashfs(1)
>
>
>   Release notes
>
>
>   See also
>
> https://pagure.io/releng/issue/8646
>
> --
>
> Bohdan Khomutskyi
>
> Release Configuration Management engineer
> Red Hat
>
>
> ___
> devel mailing list -- devel@lists.fedoraproject.org
> To unsubscribe send an email to devel-le...@lists.fedoraproject.org
> Fedora Code of Conduct: 
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: 
> https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Fedora 32 system-wide change proposal: reduce installation media size by improving the compression ratio of SquashFS filesystem

[John M. Harris Jr]

On Sunday, January 5, 2020 5:23:12 AM MST Bohdan Khomutskyi wrote:
>  Summary
> 
> Improve compression ratio of SquashFS filesystem on the installation media.
> Owner
> 
> Name: Bohdan Khomutskyi
> 
> Email: bkhom...@redhat.com
> Current status
> 
> Targeted release: I propose this change for Fedora 32
> 
> Last updated: Jan 5 2020
> 
> Pagure.io issue: https://pagure.io/releng/issue/9127
> 
> I was unable to create an article in Fedora wiki system.
> Detailed Description
> 
> As of Fedora 31, the LiveOS/squashfs.img file on the installation image, is
> compressed with default settings of mksquashfs. The standard configuration
> is set to XZ algorithm with block size of 128k and BCJ filter enabled.
> Those parameters can be adjusted which will lead to a better compression
> ratio and/or reduction of the CPU usage at build time.
> 
> This is simple to achieve. Recently, Lorax has gotten support[1] for
> adjusting the compression options for mksquashfs via the configuration
> file. The file should be altered as following:
> 
> [compression]
> bcj = yes
> args = -b 1M -Xdict-size 1M -no-recovery
> 
> Where -b 1M and -Xdict-size 1M are block and dictionary sizes respectively.
> Could be adjusted.
> Benefit to Fedora
> 
>-
> 
>Reduction of the installation media size and the cost of storing and
>distributing Fedora.
>-
> 
>Reduction of the CPU usage at build time. Depending on which compression
>parameters chosen.
>-
> 
>See a graphical detail at https://pagure.io/releng/issue/9127.
> 
> Scope
> 
>-
> 
>Proposal owners:
> 
> The build environment should have support for adjusting the Lorax
> configuration file.. Lorax is a program that produces the
> LiveOS/squashfs.img file on the installation media.
> 
> One of the way to allow for such customization, is to add a feature in
> Pungi, to allow for passing -c option to Lorax.
> 
>-
> 
>Release engineering: #9127 
>-
> 
>Policies and guidelines: N/A
>-
> 
>Trademark approval: N/A
> 
> Upgrade/compatibility impact
> 
>-
> 
>This change comes at a cost of higher memory usage during the
>installation. Based on my personal estimations, this should not be the
>issue. Since the decompression should require up to 1MiB per thread.
> 
> User Experience
> 
>-
> 
>Increasing the block size on the current configuration with EXT4 file
>system, should increase latency while accessing the EXT4 filesystem. The
>exact impact is to be evaluated.
>-
> 
>The impact of latency will be reduced, if the plain SquashFS option is
>be choosen.
> 
> Dependencies
> 
>-
> 
>N/A
> 
> Contingency Plan
> 
>-
> 
>N/A
> 
> Documentation
> https://pagure.io/releng/issue/9127.
> 
> mksquashfs(1)
> Release notes See also
> 
> https://pagure.io/releng/issue/8646
> 
> --
> 
> Bohdan Khomutskyi
> 
> Release Configuration Management engineer
> Red Hat

This looks like an excellent Change, especially as these images grow.

-- 
John M. Harris, Jr.
Splentity

___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Fedora 32 system-wide change proposal: reduce installation media size by improving the compression ratio of SquashFS filesystem

[Chris Murphy]

On Sun, Jan 5, 2020 at 5:24 AM Bohdan Khomutskyi  wrote:
>
> Summary
>
> Improve compression ratio of SquashFS filesystem on the installation media.

On the issues of Fedora ISOs using excessive CPU, related to lzma decompression:
https://bugzilla.redhat.com/show_bug.cgi?id=1717728
https://pagure.io/releng/issue/8581

Create images using plain squashfs (without nested ext4)
https://pagure.io/releng/issue/8646

And koji issue to enhance it so it can accept configurable rootfs
types (plain squashfs and configurable compression)
https://pagure.io/koji/issue/1622

I'm wondering if you can relate this feature proposal to those issues
and feature requests?

In my testing, xz does provide better compression ratios, well suited
for seldom used images like archives. But it really makes the
installation experience worse by soaking the CPU, times thousands of
installations (openQA tests on every single nightly, every human QA
tester for nightlies, betas, and then the final released product used
by Fedora end users).

Has zstandard been evaluated? In my testing of images compressed with
zstd, the CPU hit is cut by more than 50%, and is no longer a
bottleneck during installations. Image size does increase, although I
haven't tested mksquashfs block size higher than 256K. Using zstd with
Fedora images also builds on prior evaluation, testing, and effort
moving RPM from xz to zstd.

My testing with mksquashfs block size suggests compression ratio
improves but latency gets worse, and becomes somewhat pathological
with a nested ext4 in it: my best guess is the random access nature of
ext4, many 4KiB seeks turn into larger 128KiB seeks; and also squashfs
and ext4 probably have different localities (where data is placed in
relation to their metadata, in attempt to optimize). Dropping the
nested ext4 image also improved performance quite a bit, independent
of compression algorithm. I forget how much exactly but it may be
~30%.

I've pretty much concluded Fedora is best off dropping the nested ext4
in favor of plain squashfs, and using zstd. It's not required to do
both, but the benefit is additive and significant. The work in dracut
and lorax to support plain squashfs, assembling it using overlayfs
instead of device-mapper is already done, and tested.

-- 
Chris Murphy
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org