Hi, I'm pleased to announce the release of Squashfs tools 4.4. This is the first release in over 5 years, and there are substantial improvements: reproducible builds, new compressors, CVE fixes, security hardening and new options for Mksquashfs/Unsquashfs.
The new release can be downloaded here: http://sourceforge.net/projects/squashfs/files/latest/download?source=files Summary of changes in Squashfs tools 4.4 ---------------------------------------- 1. Mksquashfs now generates reproducible images by default. Mkfs time and file timestamps can also be specified. 2. Support for the Zstandard (ZSTD) compression algorithm has been added. 3. Pseudo files now support symbolic links. 4. CVE-2015-4645 and CVE-2015-4646 have been fixed. 5. Unsquashfs has been further hardened against corrupted filestems. 6. Unsquashfs is now more strict about error handling. 7. Miscellaneous new options and major bug fixes for Mksquashfs. 8. Miscellaneous new options and major bug fixes for Unsquashfs. 9. Squashfs-tools 4.4 is compatible with all earlier 4.x filesystems and releases. 1. Introducing reproducible builds ---------------------------------- Ever since Mksquashfs was parallelised back in 2006, there has been a certain randomness in how fragments and multi-block files are ordered in the output filesystem even if the input remains the same. This is because the multiple parallel threads can be scheduled differently between Mksquashfs runs. For example, the thread given fragment 10 to compress may finish before the thread given fragment 9 to compress on one run (writing fragment 10 to the output filesystem before fragment 9), but, on the next run it could be vice-versa. There are many different scheduling scenarios here, all of which can have a knock on effect causing different scheduling and ordering later in the filesystem too. Mkquashfs doesn't care about the ordering of fragments and multi-block files within the filesystem, as this does not affect the correctness of the filesystem. In fact not caring about the ordering, as it doesn't matter, allows Mksquashfs to run as fast as possible, maximising CPU and I/O performance. But, in the last couple of years, Squashfs has become used in scenarios (cloud etc) where this randomness is causing problems. Specifically this appears to be where downloaders, installers etc. try to work out the differences between Squashfs filesystem updates to minimise the amount of data that needs to transferred to update an image. Additionally, in the last couple of years has arisen the notion of reproducible builds, that is the same source and build environment etc should be able to (re-)generate identical output. This is usually for verification and security, allowing binaries/distributions to be checked for malicious activity. See https://reproducible-builds.org/ for more information. Mksquashfs now generates reproducible images by default. Images generated by Mksquashfs will be ordered identically to previous runs if the same input has been supplied, and the same options used. 1.1.1 Dealing with timestamps Timestamps embedded in the filesystem will stiil cause differences. Each new run of Mksquashfs will produce a different mkfs (make filesystem) timestamp in the super-block. Moreover if any file timestamps have changed (even if the content hasn't), this will produce a difference. To prevent timestamps from producing differences, the following new Mksquashfs options have been added. 1.1.2 -mkfs-time <time> This option takes a positive time value (which is the number of seconds since the epoch of 1970-01-01 00:00:00 UTC), and sets the file system timestamp to that. Squashfs uses an unsigned 32-bit integer to store time, and the time given should be in that range. Obviously you can use the date command to convert dates into this value, i.e. % mksquashfs source source.sqsh -mkfs-time $(date +%s -d "Jan 1 2019 19:00") 1.1.3 -all-time <time> This option takes a positive time value (which is the number of seconds since the epoch of 1970-01-01 00:00:00 UTC), and sets the timestamp on all files to that (but not the mkfs time). 1.1.4 environment variable SOURCE_DATE_EPOCH As an alternative to the above command line options, you can set the environment variable SOURCE_DATE_EPOCH to a time value. This value will be used to set the mkfs time. Also any file timestamps which are after SOURCE_DATE_EPOCH will be clamped to SOURCE_DATE_EPOCH. See https://reproducible-builds.org/docs/source-date-epoch/ for more information. Note: both SOURCE_DATE_EPOCH and the command line options cannot be used at the same time. They are different ways to do the same thing, and both have FORCE sematics which mean they can't be over-ridden elsewhere (otherwise it would defeat the purpose). 1.1.5 -not-reproducible This option tells Mksquashfs that the files do not have to be strictly ordered. This will make Mksquashfs behave like version 4.3. 2. Zstandard (ZSTD) compression added ------------------------------------- This is named zstd. It supports the following compression options. zstd -Xcompression-level <compression-level> <compression-level> should be 1 .. 22 (default 15) 3. Pseudo file symbolic link support added ------------------------------------------ Pseudo definition Filename s mode uid gid symlink uid and gid can be either specified as a decimal number, or by name. Note mode is ignored, as symlinks always have "rwxrwxrwx" permissions. For example: symlink s 0 root root example creates a symlink "symlink" to file "example" with root uid/gid. 4. CVE-2015-2015-4645 and CVE-2015-4646 --------------------------------------- These CVEs were raised due to Unsquashfs having variable overflow and stack overflow in a number of vulnerable functions. All instances of variable overflow and stack overflow have been removed. 5. Unsquashfs hardened against corrupted filestems -------------------------------------------------- The filesystem super-block values and filesystem metadata tables are further sanity checked. More importantly, these values are now checked for validity against other values in the metadata tables, and these values must match. 6. Unsquashfs is now more strict about error handling ----------------------------------------------------- Unsquashfs splits errors into two categories: fatal errors and non-fatal errors. In this release a significant number of errors that were previously non-fatal have been hardened to fatal. Fatal errors are those which cause Unsquashfs to abort instantly. These are generally due to failure to read the filesystem (corruption), and/or failure to write files to the output filesystem, due to I/O error or out of space. Generally anything which is unexpected is a fatal error. Non-fatal errors are generally where support is lacking in the output filesystem, and it can be considered to be an expected failure. This includes the inability to write extended attributes (xattrs) to a filesystem that doesn't support them, the inability to create files on filesystem that doesn't support them (i.e. symbolic links on VFAT), and the inability to execute privileged operations as a user-process. The user may well know the filesystem cannot support certain operations and would prefer Unsquashfs to ignore then without aborting. Two new options have been added: 6.1. -ignore-errors This makes Unsquashfs behave like previous versions, and treats more errors as non-fatal. 6.2. -strict-errors This makes Unsquashfs treat every error as fatal, and it will abort instantly. 7. Miscellaneous new options and major bug fixes for Mksquashfs --------------------------------------------------------------- 7.1. -root-mode <mode> This sets the permissions of the root directory to the octal <mode>. This is mostly intended for when multiple sources are specified on the command line. In this instance Mksquashfs produces a dummy top level directory with permissions 0777 (rwxrwxrwx). This option allows the permissions to be changed. But the option can also be used when a single source is specified. 7.2. -quiet This suppresses all output from Mksquashfs, except the progress bar. The progress bar can disabled with -no-progress to produce completely silent output. This new option is useful for scripts. 7.3. -noId This is similar to the pre-existing -noI option, except, it specifies that only the Id table (uids and gids) should be uncompressed. This option was added to enable a use-case where uids and gids need to be updated after filesystem generation. 7.4. -offset <offset> This option skips <offset> bytes at the beginning of the output filesystem. Optionally a suffix of K, M or G can be given to specify Kbytes, Mbytes or Gbytes respectively. 7.5. Update lz4 wrapper to use new functions introduced in 1.7.0 7.5. Bug fix, don't allow "/" pseudo filenames 7.6. Bug fix, allow quoting of pseudo files, to better handle filenames with spaces 7.7. Fix compilation with glibc 2.25+ 8. Miscellaneous new options and major bug fixes for Unsquashfs --------------------------------------------------------------- 8.1. -lln[umeric] This is similar to the "-lls" option but displays uids and gids numerically. 8.2. -lc option This is similar to the "-ls" option except it only displays files and empty directories. 8.3. -llc option As "-llc" option but displays file attributes. 8.4. -offset <offset> This option skips <offset> bytes at the beginning of the input filesystem. Optionally a suffix of K, M or G can be given to specify Kbytes, Mbytes or Gbytes respectively. 8.5. -quiet This suppresses all output from Unsquashfs, except the progress bar. The progress bar can disabled with -no-progress to produce completely silent output. This new option is useful for scripts. 8.6. -UTC This option makes Unsquashfs display all times in the UTC time zone rather than using the default local time zone. 8.7. Update lz4 wrapper to use new functions introduced in 1.7.0 8.8. Bug fix, fatal and non-fatal errors now set the exit code to 1 8.9. Bug fix, fix time setting for symlinks 8.10. Bug fix, try to set sticky-bit when running as a user process 8.11. Fix compilation with glibc 2.25+ 9. Compatiblity --------------- Mksquashfs 4.4 generates 4.0 filesystems. These filesystems are fully compatible/interchangable with filesystems generated by Mksquashfs 4.x and are mountable on 2.6.29 and later kernels.
