Re: [OpenWrt-Devel] [RFC PATCH] build: re-enable parallelism for mksquashfs
Ahh, thank you both for the clarification. On 11/08/2018 01:35 AM, Felix Fietkau wrote: > On 2018-11-08 02:20, Daniel Santos wrote: >> On 11/07/2018 01:52 PM, Felix Fietkau wrote: >>> On 2018-11-05 00:19, Daniel Santos wrote: This was disabled by commit dcd0e4a6727611f03eb3d3a75f073235f5f1229c due to a threading bug back in 2009. The specifics of the bug are not given in the commit message and squashfs-tools has had several updates to it's parallelism since this time. There are currently no open issues related to parallelism in their issue tracker: https://github.com/plougher/squashfs-tools/issues It now "works for me" with 16 threads, and while this is a terrible test for a race condition I still propose we remove this work-around unless and until we have specific knowledge of a current bug. Signed-off-by: Daniel Santos >>> Are the images still reproducible after that change? >>> If I remember correctly, threading would break that. >> Hello. I'm not sure what you mean by the images being reproducible. >> I've been building with TARGET_ROOTFS_SQUASHFS=y and >> TARGET_ROOTFS_SQUASHFS=256 for a few weeks now without any problem. I'm >> presuming they've fixed it, but I didn't see specific mention of this >> bug in their github issue tracker. They may have discussed it elsewhere >> prior to porting to github. > Reproducible as in https://reproducible-builds.org/ > You're supposed to be able to generate the binary-identical rootfs image > on different machines, as long as you're using the same version and the > same configuration. > >From my understanding, using multiple threads for the squashfs build can > make the order in which data appears in the filesystem image > non-deterministic. > > - Felix I see. I have definitely not tested that. I would assume they could vary from one build to another based upon my understanding of how multi-threaded compression is done in general. How about a patch that allows multi-threaded compression via a .config option? Thanks, Daniel ___ openwrt-devel mailing list openwrt-devel@lists.openwrt.org https://lists.openwrt.org/mailman/listinfo/openwrt-devel
Re: [OpenWrt-Devel] [RFC PATCH] build: re-enable parallelism for mksquashfs
Op do, 8 nov 2018 om 2:20 , schreef Daniel Santos : On 11/07/2018 01:52 PM, Felix Fietkau wrote: On 2018-11-05 00:19, Daniel Santos wrote: This was disabled by commit dcd0e4a6727611f03eb3d3a75f073235f5f1229c due to a threading bug back in 2009. The specifics of the bug are not given in the commit message and squashfs-tools has had several updates to it's parallelism since this time. There are currently no open issues related to parallelism in their issue tracker: https://github.com/plougher/squashfs-tools/issues It now "works for me" with 16 threads, and while this is a terrible test for a race condition I still propose we remove this work-around unless and until we have specific knowledge of a current bug. Signed-off-by: Daniel Santos Are the images still reproducible after that change? If I remember correctly, threading would break that. - Felix Hello. I'm not sure what you mean by the images being reproducible. Reproducible means you can compile twice (or more) and it will result in identical (bit by bit, same checksum) binaries every time. Multithreading apparently broke that, as Felix stated. While reproducibility might not be an issue for you personally, I believe it's one of the project's goals - it's a way to ensure the builds that are publicly available have not been tampered with. Stijn [...] Daniel ___ openwrt-devel mailing list openwrt-devel@lists.openwrt.org https://lists.openwrt.org/mailman/listinfo/openwrt-devel ___ openwrt-devel mailing list openwrt-devel@lists.openwrt.org https://lists.openwrt.org/mailman/listinfo/openwrt-devel
Re: [OpenWrt-Devel] [RFC PATCH] build: re-enable parallelism for mksquashfs
On 2018-11-08 02:20, Daniel Santos wrote: > On 11/07/2018 01:52 PM, Felix Fietkau wrote: >> On 2018-11-05 00:19, Daniel Santos wrote: >>> This was disabled by commit dcd0e4a6727611f03eb3d3a75f073235f5f1229c due >>> to a threading bug back in 2009. The specifics of the bug are not given >>> in the commit message and squashfs-tools has had several updates to it's >>> parallelism since this time. There are currently no open issues related >>> to parallelism in their issue tracker: >>> https://github.com/plougher/squashfs-tools/issues >>> >>> It now "works for me" with 16 threads, and while this is a terrible test >>> for a race condition I still propose we remove this work-around >>> unless and until we have specific knowledge of a current bug. >>> >>> Signed-off-by: Daniel Santos >> Are the images still reproducible after that change? >> If I remember correctly, threading would break that. > Hello. I'm not sure what you mean by the images being reproducible. > I've been building with TARGET_ROOTFS_SQUASHFS=y and > TARGET_ROOTFS_SQUASHFS=256 for a few weeks now without any problem. I'm > presuming they've fixed it, but I didn't see specific mention of this > bug in their github issue tracker. They may have discussed it elsewhere > prior to porting to github. Reproducible as in https://reproducible-builds.org/ You're supposed to be able to generate the binary-identical rootfs image on different machines, as long as you're using the same version and the same configuration. From my understanding, using multiple threads for the squashfs build can make the order in which data appears in the filesystem image non-deterministic. - Felix ___ openwrt-devel mailing list openwrt-devel@lists.openwrt.org https://lists.openwrt.org/mailman/listinfo/openwrt-devel
Re: [OpenWrt-Devel] [RFC PATCH] build: re-enable parallelism for mksquashfs
Hi, > I'm not sure what you mean by the images being reproducible. It means that different mksquashfs runs with different amounts of CPUs on different hosts should yield bit-identical images having the same checksum given the same input. ~ Jo signature.asc Description: OpenPGP digital signature ___ openwrt-devel mailing list openwrt-devel@lists.openwrt.org https://lists.openwrt.org/mailman/listinfo/openwrt-devel
Re: [OpenWrt-Devel] [RFC PATCH] build: re-enable parallelism for mksquashfs
On 11/07/2018 01:52 PM, Felix Fietkau wrote: > On 2018-11-05 00:19, Daniel Santos wrote: >> This was disabled by commit dcd0e4a6727611f03eb3d3a75f073235f5f1229c due >> to a threading bug back in 2009. The specifics of the bug are not given >> in the commit message and squashfs-tools has had several updates to it's >> parallelism since this time. There are currently no open issues related >> to parallelism in their issue tracker: >> https://github.com/plougher/squashfs-tools/issues >> >> It now "works for me" with 16 threads, and while this is a terrible test >> for a race condition I still propose we remove this work-around >> unless and until we have specific knowledge of a current bug. >> >> Signed-off-by: Daniel Santos > Are the images still reproducible after that change? > If I remember correctly, threading would break that. > > - Felix > Hello. I'm not sure what you mean by the images being reproducible. I've been building with TARGET_ROOTFS_SQUASHFS=y and TARGET_ROOTFS_SQUASHFS=256 for a few weeks now without any problem. I'm presuming they've fixed it, but I didn't see specific mention of this bug in their github issue tracker. They may have discussed it elsewhere prior to porting to github. Daniel ___ openwrt-devel mailing list openwrt-devel@lists.openwrt.org https://lists.openwrt.org/mailman/listinfo/openwrt-devel
Re: [OpenWrt-Devel] [RFC PATCH] build: re-enable parallelism for mksquashfs
On 2018-11-05 00:19, Daniel Santos wrote: > This was disabled by commit dcd0e4a6727611f03eb3d3a75f073235f5f1229c due > to a threading bug back in 2009. The specifics of the bug are not given > in the commit message and squashfs-tools has had several updates to it's > parallelism since this time. There are currently no open issues related > to parallelism in their issue tracker: > https://github.com/plougher/squashfs-tools/issues > > It now "works for me" with 16 threads, and while this is a terrible test > for a race condition I still propose we remove this work-around > unless and until we have specific knowledge of a current bug. > > Signed-off-by: Daniel Santos Are the images still reproducible after that change? If I remember correctly, threading would break that. - Felix ___ openwrt-devel mailing list openwrt-devel@lists.openwrt.org https://lists.openwrt.org/mailman/listinfo/openwrt-devel
[OpenWrt-Devel] [RFC PATCH] build: re-enable parallelism for mksquashfs
This was disabled by commit dcd0e4a6727611f03eb3d3a75f073235f5f1229c due to a threading bug back in 2009. The specifics of the bug are not given in the commit message and squashfs-tools has had several updates to it's parallelism since this time. There are currently no open issues related to parallelism in their issue tracker: https://github.com/plougher/squashfs-tools/issues It now "works for me" with 16 threads, and while this is a terrible test for a race condition I still propose we remove this work-around unless and until we have specific knowledge of a current bug. Signed-off-by: Daniel Santos --- include/image.mk | 1 - 1 file changed, 1 deletion(-) diff --git a/include/image.mk b/include/image.mk index f2a85f6feb..dcf1e3572a 100644 --- a/include/image.mk +++ b/include/image.mk @@ -203,7 +203,6 @@ define Image/mkfs/squashfs $(STAGING_DIR_HOST)/bin/mksquashfs4 $(call mkfs_target_dir,$(1)) $@ \ -nopad -noappend -root-owned \ -comp $(SQUASHFSCOMP) $(SQUASHFSOPT) \ - -processors 1 \ $(if $(SOURCE_DATE_EPOCH),-fixed-time $(SOURCE_DATE_EPOCH)) endef -- 2.16.4 ___ openwrt-devel mailing list openwrt-devel@lists.openwrt.org https://lists.openwrt.org/mailman/listinfo/openwrt-devel
