[PATCH] btrfs: Call kunmap if zlib_inflateInit2 fails

If zlib_inflateInit2 fails, the input page is never unmapped. Add a call to kunmap when it fails. Signed-off-by: Nick Terrell <nickrterr...@gmail.com> --- fs/btrfs/zlib.c | 1 + 1 file changed, 1 insertion(+) diff --git a/fs/btrfs/zlib.c b/fs/btrfs/zlib.c index 88d274e..30e19c9 100644 --

Re: [PATCH v5 3/5] btrfs: Add zstd support

On 8/10/17, 7:13 PM, "Adam Borowski" <kilob...@angband.pl> wrote: > On Wed, Aug 09, 2017 at 07:39:02PM -0700, Nick Terrell wrote: >> Add zstd compression and decompression support to BtrFS. > > Re-tested on arm64, amd64 and i386, this time everything seems fi

Re: [PATCH v5 2/5] lib: Add zstd modules

On 8/10/17, 10:48 AM, "Austin S. Hemmelgarn" <ahferro...@gmail.com> wrote: >On 2017-08-10 13:24, Eric Biggers wrote: >>On Thu, Aug 10, 2017 at 07:32:18AM -0400, Austin S. Hemmelgarn wrote: >>>On 2017-08-10 04:30, Eric Biggers wrote: >>>>On Wed, Aug 09

Re: [PATCH v5 2/5] lib: Add zstd modules

On 8/10/17, 1:30 AM, "Eric Biggers" <ebigge...@gmail.com> wrote: > On Wed, Aug 09, 2017 at 07:35:53PM -0700, Nick Terrell wrote: >> >> It can compress at speeds approaching lz4, and quality approaching lzma. > > Well, for a very loose definition of "app

Re: [PATCH v2 3/4] btrfs: Add zstd support

On 7/6/17, 9:32 AM, "Adam Borowski" <kilob...@angband.pl> wrote: > On Thu, Jun 29, 2017 at 12:41:07PM -0700, Nick Terrell wrote: >> Add zstd compression and decompression support to BtrFS. zstd at its >> fastest level compresses almost as well as zlib, while offe

Re: [PATCH v2 3/4] btrfs: Add zstd support

On 7/10/17, 9:57 PM, "Nick Terrell" <terre...@fb.com> wrote: > The problem is caused by a gcc-7 bug [1]. It miscompiles > ZSTD_wildcopy(void *dst, void const *src, ptrdiff_t len) when len is 0. > It only happens when it can't analyze ZSTD_copy8(), which is the case in >

Re: [PATCH v2 3/4] btrfs: Add zstd support

On 7/10/17, 5:36 AM, "Austin S. Hemmelgarn" <ahferro...@gmail.com> wrote: > On 2017-07-07 23:07, Adam Borowski wrote: >> On Sat, Jul 08, 2017 at 01:40:18AM +0200, Adam Borowski wrote: >>> On Fri, Jul 07, 2017 at 11:17:49PM +, Nick Terrell wrote: >>>&

Re: [PATCH v2 3/4] btrfs: Add zstd support

On 7/10/17, 5:36 AM, "Austin S. Hemmelgarn" <ahferro...@gmail.com> wrote: > On 2017-07-07 23:07, Adam Borowski wrote: >> On Sat, Jul 08, 2017 at 01:40:18AM +0200, Adam Borowski wrote: >>> On Fri, Jul 07, 2017 at 11:17:49PM +, Nick Terrell wrote: >>>&

[PATCH v3 0/4] Add xxhash and zstd modules

Hi all, This patch set adds xxhash, zstd compression, and zstd decompression modules. It also adds zstd support to BtrFS and SquashFS. Each patch has relevant summaries, benchmarks, and tests. Best, Nick Terrell Changelog: v1 -> v2: - Make pointer in lib/xxhash.c:394 non-const (1/4) -

[PATCH v3 1/4] lib: Add xxhash module

repository: https://github.com/cyan4973/xxhash Signed-off-by: Nick Terrell <terre...@fb.com> --- v1 -> v2: - Make pointer in lib/xxhash.c:394 non-const include/linux/xxhash.h | 236 +++ lib/Kconfig| 3 + lib/Makefile | 1 + lib/xxhash.c

[PATCH v3 4/4] squashfs: Add zstd support

source repository: https://github.com/facebook/zstd Cc: Sean Purcell <m...@seanp.xyz> Signed-off-by: Nick Terrell <terre...@fb.com> --- fs/squashfs/Kconfig| 14 + fs/squashfs/Makefile | 1 + fs/squashfs/decompressor.c | 7 +++ fs/squashfs/decompressor.h | 4 ++ fs/squa

[PATCH v3 3/4] btrfs: Add zstd support

[3] http://sun.aei.polsl.pl/~sdeor/index.php?page=silesia [4] https://cdn.kernel.org/pub/linux/kernel/v4.x/linux-4.11.6.tar.xz zstd source repository: https://github.com/facebook/zstd Signed-off-by: Nick Terrell <terre...@fb.com> --- v2 -> v3: - Port upstream BtrFS commits e1ddce71d6, 3

[PATCH] btrfs: Keep one more workspace around

and zstd. > I'm on linus:4.12-rc7 with only a handful of btrfs patches (v3 of Qu's chunk > check, some misc crap) -- I guess I should use at least btrfs-for-4.13. Or > would you prefer full-blown next? Whatever is convenient for you. The relevant code in BtrFS hasn't changed for a

[PATCH v2 1/4] lib: Add xxhash module

repository: https://github.com/cyan4973/xxhash Signed-off-by: Nick Terrell <terre...@fb.com> --- v1 -> v2: - Make pointer in lib/xxhash.c:394 non-const include/linux/xxhash.h | 236 +++ lib/Kconfig| 3 + lib/Makefile | 1 + lib/xxhash.c

[PATCH] btrfs: Keep one more workspace around

for zstd compression. Signed-off-by: Nick Terrell <terre...@fb.com> --- fs/btrfs/compression.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/fs/btrfs/compression.c b/fs/btrfs/compression.c index 3beb0d0..1a0ef55 100644 --- a/fs/btrfs/compression.c +++ b/fs/btrfs/compressi

[PATCH v2 3/4] btrfs: Add zstd support

[3] http://sun.aei.polsl.pl/~sdeor/index.php?page=silesia [4] https://cdn.kernel.org/pub/linux/kernel/v4.x/linux-4.11.6.tar.xz zstd source repository: https://github.com/facebook/zstd Signed-off-by: Nick Terrell <terre...@fb.com> --- fs/btrfs/Kconfig | 2 + fs/btrfs/Ma

[PATCH v2 0/4] Add xxhash and zstd modules

Hi all, This patch set adds xxhash, zstd compression, and zstd decompression modules. It also adds zstd support to BtrFS and SquashFS. Each patch has relevant summaries, benchmarks, and tests. Best, Nick Terrell Changelog: v1 -> v2: - Make pointer in lib/xxhash.c:394 non-const (1/4) -

[PATCH v2 4/4] squashfs: Add zstd support

source repository: https://github.com/facebook/zstd Cc: Sean Purcell <m...@seanp.xyz> Signed-off-by: Nick Terrell <terre...@fb.com> --- fs/squashfs/Kconfig| 14 + fs/squashfs/Makefile | 1 + fs/squashfs/decompressor.c | 7 +++ fs/squashfs/decompressor.h | 4 ++ fs/squa

Re: [PATCH v2 0/4] Add xxhash and zstd modules

> If i understood all correctly, > zstd can compress (decompress) data in way compatible with gzip (zlib) > Do that also true for in kernel library? The zstd command line tool can decompress gzip/zlib compressed data, and can compress in the gzip format. However, the zstd format is not compatible

Re: [PATCH v2 3/4] btrfs: Add zstd support

>> If we're going to make that configurable, there are some things to >> consider: >> >> * the underlying compressed format -- does not change for different >>levels This is true for zlib and zstd. lzo in the kernel only supports one compression level. >> * the configuration interface --

Re: [PATCH v2 3/4] btrfs: Add zstd support

On 7/5/17, 12:57 PM, "Austin S. Hemmelgarn" wrote: > It's the slower compression speed that has me arguing for the > possibility of configurable levels on zlib. 11MB/s is painfully slow > considering that most decent HDD's these days can get almost 5-10x that > speed

Re: [PATCH v2 3/4] btrfs: Add zstd support

On 7/5/17, 11:45 AM, "Austin S. Hemmelgarn" <ahferro...@gmail.com> wrote: >On 2017-07-05 14:18, Adam Borowski wrote: >> On Wed, Jul 05, 2017 at 07:43:27AM -0400, Austin S. Hemmelgarn >> wrote: >>> On 2017-06-30 19:01, Nick Terrell wrote: >>>

[PATCH v4 1/5] lib: Add xxhash module

repository: https://github.com/cyan4973/xxhash Signed-off-by: Nick Terrell <terre...@fb.com> --- v1 -> v2: - Make pointer in lib/xxhash.c:394 non-const include/linux/xxhash.h | 236 +++ lib/Kconfig| 3 + lib/Makefile | 1 + lib/xxhash.c

[PATCH v4 0/5] Add xxhash and zstd modules

Hi all, This patch set adds xxhash, zstd compression, and zstd decompression modules. It also adds zstd support to BtrFS and SquashFS. Each patch has relevant summaries, benchmarks, and tests. Best, Nick Terrell Changelog: v1 -> v2: - Make pointer in lib/xxhash.c:394 non-const (1/5) -

[PATCH v4 4/5] squashfs: Add zstd support

source repository: https://github.com/facebook/zstd Cc: Sean Purcell <m...@seanp.xyz> Signed-off-by: Nick Terrell <terre...@fb.com> --- v3 -> v4: - Fix minor linter warnings fs/squashfs/Kconfig| 14 + fs/squashfs/Makefile | 1 + fs/squashfs/decompressor.c |

[PATCH v4 5/5] crypto: Add zstd support

Adds zstd support to crypto and scompress. Only supports the default level. Signed-off-by: Nick Terrell <terre...@fb.com> --- crypto/Kconfig | 9 ++ crypto/Makefile | 1 + crypto/testmgr.c | 10 +++ crypto/testmgr.h | 71 +++ crypto/zstd.c

[PATCH v4 3/5] btrfs: Add zstd support

[3] http://sun.aei.polsl.pl/~sdeor/index.php?page=silesia [4] https://cdn.kernel.org/pub/linux/kernel/v4.x/linux-4.11.6.tar.xz zstd source repository: https://github.com/facebook/zstd Signed-off-by: Nick Terrell <terre...@fb.com> --- v2 -> v3: - Port upstream BtrFS commits e1ddce71d6, 3

Re: [PATCH preview] btrfs: allow to set compression level for zlib

On 07/25/2017 01:29 AM, David Sterba wrote: > Preliminary support for setting compression level for zlib, the > following works: Thanks for working on this, I think it is a great feature. I have a few comments relating to how it would work with zstd. > > $ mount -o compess=zlib

Re: [PATCH v4 4/5] squashfs: Add zstd support

On 8/4/17, 3:10 PM, "linus...@gmail.com on behalf of Linus Torvalds" <linus...@gmail.com on behalf of torva...@linux-foundation.org> wrote: > On Fri, Aug 4, 2017 at 1:19 PM, Nick Terrell <terre...@fb.com> wrote: > > > > This patch was written by

Re: [PATCH preview] btrfs: allow to set compression level for zlib

On 8/4/17, 6:27 PM, "Adam Borowski" <kilob...@angband.pl> wrote: > On Fri, Aug 04, 2017 at 09:51:44PM +, Nick Terrell wrote: > > On 07/25/2017 01:29 AM, David Sterba wrote: > > > Preliminary support for setting compression level for zlib, the > > > f

Re: [PATCH v3 4/4] squashfs: Add zstd support

On 7/30/17, 6:50 PM, "Phillip Lougher" <phillip.loug...@gmail.com> wrote: > On Thu, Jul 20, 2017 at 10:27 PM, Nick Terrell <terre...@fb.com> wrote: >> Add zstd compression and decompression support to SquashFS. zstd is a >> great fit for SquashFS because it

[PATCH v5 1/5] lib: Add xxhash module

repository: https://github.com/cyan4973/xxhash Signed-off-by: Nick Terrell <terre...@fb.com> --- v1 -> v2: - Make pointer in lib/xxhash.c:394 non-const include/linux/xxhash.h | 236 +++ lib/Kconfig| 3 + lib/Makefile | 1 + lib/xxhash.c

[PATCH v5 0/5] Add xxhash and zstd modules

Hi all, This patch set adds xxhash, zstd compression, and zstd decompression modules. It also adds zstd support to BtrFS and SquashFS. Each patch has relevant summaries, benchmarks, and tests. Best, Nick Terrell Changelog: v1 -> v2: - Make pointer in lib/xxhash.c:394 non-const (1/5) -

[PATCH v5 3/5] btrfs: Add zstd support

[3] http://sun.aei.polsl.pl/~sdeor/index.php?page=silesia [4] https://cdn.kernel.org/pub/linux/kernel/v4.x/linux-4.11.6.tar.xz zstd source repository: https://github.com/facebook/zstd Signed-off-by: Nick Terrell <terre...@fb.com> --- v2 -> v3: - Port upstream BtrFS commits e1ddce71d6, 3

[PATCH 3/4] btrfs: Add zstd support

[3] http://sun.aei.polsl.pl/~sdeor/index.php?page=silesia [4] https://cdn.kernel.org/pub/linux/kernel/v4.x/linux-4.11.6.tar.xz zstd source repository: https://github.com/facebook/zstd Signed-off-by: Nick Terrell <terre...@fb.com> --- fs/btrfs/Kconfig | 2 + fs/btrfs/Ma

[PATCH 1/4] lib: Add xxhash module

repository: https://github.com/cyan4973/xxhash Signed-off-by: Nick Terrell <terre...@fb.com> --- include/linux/xxhash.h | 236 +++ lib/Kconfig| 3 + lib/Makefile | 1 + lib/xxhash.c | 500 + 4

[PATCH 4/4] squashfs: Add zstd support

source repository: https://github.com/facebook/zstd Cc: Sean Purcell <m...@seanp.xyz> Signed-off-by: Nick Terrell <terre...@fb.com> --- fs/squashfs/Kconfig| 14 + fs/squashfs/Makefile | 1 + fs/squashfs/decompressor.c | 7 +++ fs/squashfs/decompressor.h | 4 ++ fs/squa

Re: [PATCH 3/4] btrfs: Add zstd support

ick, > > > > url: https://github.com/0day-ci/linux/commits/Nick-Terrell/lib-Add-xxhash-module/20170625-214344 > > config: i386-allmodconfig (attached as .config) > > compiler: gcc-6 (Debian 6.2.0-3) 6.2.0 20160901 > > reproduce: >

Re: [PATCH] lib/zstd: use div_u64() to let it build on 32-bit

Adam, I’ve applied the same patch in my tree. I’ll send out the update [1] once it's reviewed, since I also reduced the stack usage of functions using over 1 KB of stack space. You’re right that div_u64() will work, since the FSE functions are only called on blocks of at most 128 KB at a time.

Re: [PATCH] lib/zstd: use div_u64() to let it build on 32-bit

> Please don't top post. Sorry about that. > Which function needs 1KB of stack space? That's quite a lot. FSE_buildCTable_wksp(), FSE_compress_wksp(), and HUF_readDTableX4() required over 1 KB of stack space. > I can see in [1] that there are some on-stack buffers replaced by > pointers to the

Re: [PATCH preview] btrfs: allow to set compression level for zlib

On 9/15/17, 7:53 AM, "David Sterba" wrote: > On Sun, Aug 20, 2017 at 06:38:50PM -0600, Chris Murphy wrote: > > On Fri, Aug 18, 2017 at 10:08 AM, David Sterba wrote: > > > > > That's quite a lot, in kernel. IIRC zlib and lzo use less than 200kb, > > > zstd wants

[PATCH v2] btrfs-progs: Add zstd support

std compressed file. The patch is also available in my fork of btrfs-progs [1], which passes Travis-CI with the new tests. The prebuilt binary is available there. I haven't updated Android.mk. [1] https://github.com/terrelln/btrfs-progs/tree/devel Signed-off-by: Nick Terrell <terre...@fb.com> ---

[PATCH] btrfs-progs: Add zstd support

am not certain if it is used. [1] https://github.com/terrelln/btrfs-progs/tree/devel [2] https://gist.github.com/terrelln/4136a369c5d10092956781433eed0a23 Signed-off-by: Nick Terrell <terre...@fb.com> --- .travis.yml | 9 +++ Documentation/btrfs-filesystem.asci

Re: [GIT PULL] zstd support (lib, btrfs, squashfs)

On 9/8/17, 6:36 PM, "Herbert Xu" wrote: > On Fri, Sep 08, 2017 at 03:33:05PM -0400, Chris Mason wrote: > > > > crypto/Kconfig |9 + > > crypto/Makefile|1 + > > crypto/testmgr.c | 10 + > > crypto/testmgr.h | 71

Re: \o/ compsize

On 9/4/17, 8:12 AM, "Adam Borowski" wrote: > Hi! > Here's an utility to measure used compression type + ratio on a set of files > or directories: https://github.com/kilobyte/compsize > > It should be of great help for users, and also if you: > * muck with compression levels

Re: [PATCH] btrfs-progs: Add zstd support

On 9/4/17, 11:03 AM, "linux-btrfs-ow...@vger.kernel.org on behalf of David Sterba" <linux-btrfs-ow...@vger.kernel.org on behalf of dste...@suse.cz> wrote: > On Wed, Aug 30, 2017 at 02:53:22PM -0700, Nick Terrell wrote: > > Adds zstd support to the btrfs program, an

Re: Read before you deploy btrfs + zstd

On 11/15/17, 6:41 AM, "David Sterba" wrote: > The branch is now in a state that can be tested. Turns out the memory > requirements are too much for grub, so the boot fails with "not enough > memory". The calculated value > > ZSTD_BTRFS_MAX_INPUT: 131072 >

Re: [PATCH] btrfs: move some zstd work data from stack to workspace

ompression and decompression test and everything worked. Is there a case where these 24 bytes matter, or is this just an easy optimization? Reviewed-by: Nick Terrell <terre...@fb.com>

Re: [btrfs_mount] general protection fault: 0000 [#1] SMP

> On Nov 29, 2017, at 6:21 PM, Fengguang Wu wrote: > > Hello, > > FYI this happens in mainline kernel 4.15.0-rc1. > It looks like a new regression. Bisect is in progress. > > It occurs in 11 out of 11 xfstests run. > > [ 1456.361614] > [ 1456.918942] BTRFS info

Re: Read before you deploy btrfs + zstd

> On Nov 28, 2017, at 3:49 PM, David Sterba <dste...@suse.cz> wrote: > > On Tue, Nov 28, 2017 at 09:31:57PM +0000, Nick Terrell wrote: >> >>> On Nov 21, 2017, at 8:22 AM, David Sterba <dste...@suse.cz> wrote: >>> >>> On Wed, Nov 15, 2017

Re: Btrfs progs pre-release 4.14-rc1

> On Nov 17, 2017, at 8:48 AM, Uli Heller wrote: > > I tried to compile these on ubuntu-14.04 running 4.4.0-98-generic > - I had to install libzstd-dev > - I had to disable the zstd tests > > Do you think this ends up in a working binary? Or am I doing something

Re: Read before you deploy btrfs + zstd

> On Dec 5, 2017, at 7:54 AM, David Sterba wrote: > > I had a different branch with patches from openSUSE, so the diffs apply with > minimal efforts to the package. The branch btrfs-zstd has been synced up. The > ENOMEM error was not from the file decompression but from the

Re: [PATCH] btrfs: fix default compression value when set by SETFLAGS ioctl

set fs_info->compress_type=BTRFS_COMPRESS_NONE and the ioctl() call set the compression type to zstd before your patch, and zlib after, as expected. Tested-by: Nick Terrell <terre...@fb.com> > The incompat bit for zstd is not set, so this could lead to a filesystem > with a zstd compre

Re: [PATCH v2] btrfs: add zstd compression level support

> On Nov 1, 2018, at 5:57 AM, Timofey Titovets wrote: > > ср, 31 окт. 2018 г. в 21:12, Nick Terrell : >> >> From: Jennifer Liu >> >> Adds zstd compression level support to btrfs. Zstd requires >> different amounts of memory for each level, so th

[PATCH v2] btrfs: add zstd compression level support

2.9510 MB/s111 MB/s2.3 MB 14 2.997 MB/s110 MB/s2.6 MB 15 3.036 MB/s110 MB/s2.6 MB [0] http://sun.aei.polsl.pl/~sdeor/index.php?page=silesia Signed-off-by: Jennifer Liu Signed-off-by: Nick Terrell Reviewed-by: Omar Sandoval

[PATCH] btrfs: add zstd compression level support

2.9510 MB/s111 MB/s2.3 MB 14 2.997 MB/s110 MB/s2.6 MB 15 3.036 MB/s110 MB/s2.6 MB [0] http://sun.aei.polsl.pl/~sdeor/index.php?page=silesia Signed-off-by: Jennifer Liu Signed-off-by: Nick Terrell --- fs/btrfs

Re: [PATCH v2] btrfs: add zstd compression level support

> On Nov 12, 2018, at 4:33 PM, David Sterba wrote: > > On Wed, Oct 31, 2018 at 11:11:08AM -0700, Nick Terrell wrote: >> From: Jennifer Liu >> >> Adds zstd compression level support to btrfs. Zstd requires >> different amounts of memory for each level, so

Re: [PATCH v2] btrfs: add zstd compression level support

> On Nov 13, 2018, at 5:29 AM, Timofey Titovets wrote: > > вт, 13 нояб. 2018 г. в 04:52, Nick Terrell : >> >> >> >>> On Nov 12, 2018, at 4:33 PM, David Sterba wrote: >>> >>> On Wed, Oct 31, 2018 at 11:11:08AM -0700, Nick Terrel