Re: [PATCH 1/4] lib: Update LZ4 compressor module
Hi Sven,
[auto build test WARNING on linus/master]
[also build test WARNING on v4.10-rc4 next-20170120]
[if your patch is applied to the wrong git tree, please drop us a note to help
improve the system]
url:
https://github.com/0day-ci/linux/commits/Sven-Schmidt/Update-LZ4-compressor-module/20170121-231418
reproduce:
# apt-get install sparse
make ARCH=x86_64 allmodconfig
make C=1 CF=-D__CHECK_ENDIAN__
sparse warnings: (new ones prefixed by >>)
include/linux/compiler.h:253:8: sparse: attribute 'no_sanitize_address':
unknown attribute
>> lib/lz4/lz4_decompress.c:521:21: sparse: incompatible types for operation (>)
lib/lz4/lz4_decompress.c:521:21:left side has type unsigned long
[usertype] *src_len
lib/lz4/lz4_decompress.c:521:21:right side has type int
vim +521 lib/lz4/lz4_decompress.c
505 return -1;
506 }
507 EXPORT_SYMBOL(lz4_decompress_unknownoutputsize);
508
509 int lz4_decompress(const unsigned char *src, size_t *src_len,
510 unsigned char *dest, size_t actual_dest_len) {
511 *src_len = LZ4_decompress_fast(src, dest, (int)actual_dest_len);
512
513 /*
514 * Prior lz4_decompress will return
515 * 0 for success and a negative result for error
516 * new LZ4_decompress_fast returns
517 * - the length of data read on success
518 * - and also a negative result on error
519 * meaning when result > 0, we just return 0 here
520 */
> 521 if (src_len > 0)
522 return 0;
523 else
524 return -1;
525 }
526 EXPORT_SYMBOL(lz4_decompress);
527
528 MODULE_LICENSE("Dual BSD/GPL");
529 MODULE_DESCRIPTION("LZ4 decompressor");
---
0-DAY kernel test infrastructureOpen Source Technology Center
https://lists.01.org/pipermail/kbuild-all Intel Corporation
--
To unsubscribe from this list: send the line "unsubscribe linux-crypto" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 1/4] lib: Update LZ4 compressor module
Hi Sven,
[auto build test ERROR on linus/master]
[also build test ERROR on v4.10-rc4 next-20170120]
[if your patch is applied to the wrong git tree, please drop us a note to help
improve the system]
url:
https://github.com/0day-ci/linux/commits/Sven-Schmidt/Update-LZ4-compressor-module/20170121-231418
config: x86_64-lkp (attached as .config)
compiler: gcc-6 (Debian 6.2.0-3) 6.2.0 20160901
reproduce:
# save the attached .config to linux build tree
make ARCH=x86_64
Note: the
linux-review/Sven-Schmidt/Update-LZ4-compressor-module/20170121-231418 HEAD
0472409e2a1c442b51502961aa6d83b866218953 builds fine.
It only hurts bisectibility.
All errors (new ones prefixed by >>):
In file included from lib/decompress_unlz4.c:19:0:
lib/decompress_unlz4.c: In function 'unlz4':
>> lib/decompress_unlz4.c:75:22: error: implicit declaration of function
>> 'lz4_compressbound' [-Werror=implicit-function-declaration]
inp = large_malloc(lz4_compressbound(uncomp_chunksize));
^
include/linux/decompress/mm.h:83:33: note: in definition of macro
'large_malloc'
#define large_malloc(a) vmalloc(a)
^
cc1: some warnings being treated as errors
vim +/lz4_compressbound +75 lib/decompress_unlz4.c
e76e1fdf Kyungsik Lee 2013-07-08 13 #include "lz4/lz4_decompress.c"
e76e1fdf Kyungsik Lee 2013-07-08 14 #else
e76e1fdf Kyungsik Lee 2013-07-08 15 #include
e76e1fdf Kyungsik Lee 2013-07-08 16 #endif
e76e1fdf Kyungsik Lee 2013-07-08 17 #include
e76e1fdf Kyungsik Lee 2013-07-08 18 #include
e76e1fdf Kyungsik Lee 2013-07-08 @19 #include
e76e1fdf Kyungsik Lee 2013-07-08 20 #include
e76e1fdf Kyungsik Lee 2013-07-08 21
e76e1fdf Kyungsik Lee 2013-07-08 22 #include
e76e1fdf Kyungsik Lee 2013-07-08 23
e76e1fdf Kyungsik Lee 2013-07-08 24 /*
e76e1fdf Kyungsik Lee 2013-07-08 25 * Note: Uncompressed chunk size is used
in the compressor side
e76e1fdf Kyungsik Lee 2013-07-08 26 * (userspace side for compression).
e76e1fdf Kyungsik Lee 2013-07-08 27 * It is hardcoded because there is not
proper way to extract it
e76e1fdf Kyungsik Lee 2013-07-08 28 * from the binary stream which is
generated by the preliminary
e76e1fdf Kyungsik Lee 2013-07-08 29 * version of LZ4 tool so far.
e76e1fdf Kyungsik Lee 2013-07-08 30 */
e76e1fdf Kyungsik Lee 2013-07-08 31 #define
LZ4_DEFAULT_UNCOMPRESSED_CHUNK_SIZE (8 << 20)
e76e1fdf Kyungsik Lee 2013-07-08 32 #define ARCHIVE_MAGICNUMBER 0x184C2102
e76e1fdf Kyungsik Lee 2013-07-08 33
d97b07c5 Yinghai Lu 2014-08-08 34 STATIC inline int INIT unlz4(u8 *input,
long in_len,
d97b07c5 Yinghai Lu 2014-08-08 35long
(*fill)(void *, unsigned long),
d97b07c5 Yinghai Lu 2014-08-08 36long
(*flush)(void *, unsigned long),
d97b07c5 Yinghai Lu 2014-08-08 37u8 *output,
long *posp,
e76e1fdf Kyungsik Lee 2013-07-08 38void (*error)
(char *x))
e76e1fdf Kyungsik Lee 2013-07-08 39 {
e76e1fdf Kyungsik Lee 2013-07-08 40int ret = -1;
e76e1fdf Kyungsik Lee 2013-07-08 41size_t chunksize = 0;
e76e1fdf Kyungsik Lee 2013-07-08 42size_t uncomp_chunksize =
LZ4_DEFAULT_UNCOMPRESSED_CHUNK_SIZE;
e76e1fdf Kyungsik Lee 2013-07-08 43u8 *inp;
e76e1fdf Kyungsik Lee 2013-07-08 44u8 *inp_start;
e76e1fdf Kyungsik Lee 2013-07-08 45u8 *outp;
d97b07c5 Yinghai Lu 2014-08-08 46long size = in_len;
e76e1fdf Kyungsik Lee 2013-07-08 47 #ifdef PREBOOT
e76e1fdf Kyungsik Lee 2013-07-08 48size_t out_len =
get_unaligned_le32(input + in_len);
e76e1fdf Kyungsik Lee 2013-07-08 49 #endif
e76e1fdf Kyungsik Lee 2013-07-08 50size_t dest_len;
e76e1fdf Kyungsik Lee 2013-07-08 51
e76e1fdf Kyungsik Lee 2013-07-08 52
e76e1fdf Kyungsik Lee 2013-07-08 53if (output) {
e76e1fdf Kyungsik Lee 2013-07-08 54outp = output;
e76e1fdf Kyungsik Lee 2013-07-08 55} else if (!flush) {
e76e1fdf Kyungsik Lee 2013-07-08 56error("NULL output pointer and
no flush function provided");
e76e1fdf Kyungsik Lee 2013-07-08 57goto exit_0;
e76e1fdf Kyungsik Lee 2013-07-08 58} else {
e76e1fdf Kyungsik Lee 2013-07-08 59outp =
large_malloc(uncomp_chunksize);
e76e1fdf Kyungsik Lee 2013-07-08 60if (!outp) {
e76e1fdf Kyungsik Lee 2013-07-08 61error("Could not
allocate output buffer");
e76e1fdf Kyungsik Lee 2013-07-08 62goto exit_0;
e76e1fdf Kyungsik Lee 2013-07-08 63}
e76e1fdf Kyungsik Lee 2013-07-08 64}
e76e1fdf Kyungsik Lee 2013-07-08 65
e76e1fdf Kyungsik Lee 2013-07-08 66if (input && fill) {
e76e1fdf Kyungsik Lee 2013-07-08 67error("Both input pointer and
fill function provided,");
e76e1fdf Kyungsik Lee 2013-07-08 68goto exit
Re: [PATCH 1/4] lib: Update LZ4 compressor module
Hi Sven,
[auto build test ERROR on linus/master]
[also build test ERROR on v4.10-rc4 next-20170120]
[if your patch is applied to the wrong git tree, please drop us a note to help
improve the system]
url:
https://github.com/0day-ci/linux/commits/Sven-Schmidt/Update-LZ4-compressor-module/20170121-231418
config: i386-randconfig-r0-201703 (attached as .config)
compiler: gcc-5 (Debian 5.4.1-2) 5.4.1 20160904
reproduce:
# save the attached .config to linux build tree
make ARCH=i386
Note: the
linux-review/Sven-Schmidt/Update-LZ4-compressor-module/20170121-231418 HEAD
0472409e2a1c442b51502961aa6d83b866218953 builds fine.
It only hurts bisectibility.
All errors (new ones prefixed by >>):
fs/pstore/platform.c: In function 'allocate_lz4':
>> fs/pstore/platform.c:369:20: error: implicit declaration of function
>> 'lz4_compressbound' [-Werror=implicit-function-declaration]
big_oops_buf_sz = lz4_compressbound(psinfo->bufsize);
^
cc1: some warnings being treated as errors
vim +/lz4_compressbound +369 fs/pstore/platform.c
8cfc8ddc Geliang Tang 2016-02-18 363
8cfc8ddc Geliang Tang 2016-02-18 364 return outlen;
8cfc8ddc Geliang Tang 2016-02-18 365 }
8cfc8ddc Geliang Tang 2016-02-18 366
8cfc8ddc Geliang Tang 2016-02-18 367 static void allocate_lz4(void)
8cfc8ddc Geliang Tang 2016-02-18 368 {
8cfc8ddc Geliang Tang 2016-02-18 @369 big_oops_buf_sz =
lz4_compressbound(psinfo->bufsize);
8cfc8ddc Geliang Tang 2016-02-18 370 big_oops_buf = kmalloc(big_oops_buf_sz,
GFP_KERNEL);
8cfc8ddc Geliang Tang 2016-02-18 371 if (big_oops_buf) {
8cfc8ddc Geliang Tang 2016-02-18 372 workspace =
kmalloc(LZ4_MEM_COMPRESS, GFP_KERNEL);
:: The code at line 369 was first introduced by commit
:: 8cfc8ddc99df9509a46043b14af81f5c6a223eab pstore: add lzo/lz4 compression
support
:: TO: Geliang Tang <geliangt...@163.com>
:: CC: Kees Cook <keesc...@chromium.org>
---
0-DAY kernel test infrastructureOpen Source Technology Center
https://lists.01.org/pipermail/kbuild-all Intel Corporation
.config.gz
Description: application/gzip
[PATCH 3/4] crypto: Change LZ4 modules to work with new LZ4 module version
This patch updates the crypto modules using LZ4 compression
to work with the new LZ4 module version.
Signed-off-by: Sven Schmidt <4ssch...@informatik.uni-hamburg.de>
---
crypto/lz4.c | 21 -
crypto/lz4hc.c | 21 -
2 files changed, 16 insertions(+), 26 deletions(-)
diff --git a/crypto/lz4.c b/crypto/lz4.c
index 99c1b2c..40fd2c2 100644
--- a/crypto/lz4.c
+++ b/crypto/lz4.c
@@ -66,15 +66,13 @@ static void lz4_exit(struct crypto_tfm *tfm)
static int __lz4_compress_crypto(const u8 *src, unsigned int slen,
u8 *dst, unsigned int *dlen, void *ctx)
{
- size_t tmp_len = *dlen;
- int err;
+ int out_len = LZ4_compress_default(src, dst,
+ slen, (int)((size_t)dlen), ctx);
- err = lz4_compress(src, slen, dst, _len, ctx);
-
- if (err < 0)
+ if (!out_len)
return -EINVAL;
- *dlen = tmp_len;
+ *dlen = out_len;
return 0;
}
@@ -96,16 +94,13 @@ static int lz4_compress_crypto(struct crypto_tfm *tfm,
const u8 *src,
static int __lz4_decompress_crypto(const u8 *src, unsigned int slen,
u8 *dst, unsigned int *dlen, void *ctx)
{
- int err;
- size_t tmp_len = *dlen;
- size_t __slen = slen;
+ int out_len = LZ4_decompress_safe(src, dst, slen, (int)((size_t)dlen));
- err = lz4_decompress_unknownoutputsize(src, __slen, dst, _len);
- if (err < 0)
+ if (out_len < 0)
return -EINVAL;
- *dlen = tmp_len;
- return err;
+ *dlen = out_len;
+ return out_len;
}
static int lz4_sdecompress(struct crypto_scomp *tfm, const u8 *src,
diff --git a/crypto/lz4hc.c b/crypto/lz4hc.c
index 75ffc4a..6f16f96 100644
--- a/crypto/lz4hc.c
+++ b/crypto/lz4hc.c
@@ -65,15 +65,13 @@ static void lz4hc_exit(struct crypto_tfm *tfm)
static int __lz4hc_compress_crypto(const u8 *src, unsigned int slen,
u8 *dst, unsigned int *dlen, void *ctx)
{
- size_t tmp_len = *dlen;
- int err;
+ int out_len = LZ4_compress_HC(src, dst, slen,
+ (int)((size_t)dlen), LZ4HC_DEFAULT_CLEVEL, ctx);
- err = lz4hc_compress(src, slen, dst, _len, ctx);
-
- if (err < 0)
+ if (out_len == 0)
return -EINVAL;
- *dlen = tmp_len;
+ *dlen = out_len;
return 0;
}
@@ -97,16 +95,13 @@ static int lz4hc_compress_crypto(struct crypto_tfm *tfm,
const u8 *src,
static int __lz4hc_decompress_crypto(const u8 *src, unsigned int slen,
u8 *dst, unsigned int *dlen, void *ctx)
{
- int err;
- size_t tmp_len = *dlen;
- size_t __slen = slen;
+ int out_len = LZ4_decompress_safe(src, dst, slen, (int)((size_t)dlen));
- err = lz4_decompress_unknownoutputsize(src, __slen, dst, _len);
- if (err < 0)
+ if (out_len < 0)
return -EINVAL;
- *dlen = tmp_len;
- return err;
+ *dlen = out_len;
+ return out_len;
}
static int lz4hc_sdecompress(struct crypto_scomp *tfm, const u8 *src,
--
2.1.4
--
To unsubscribe from this list: send the line "unsubscribe linux-crypto" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH 4/4] fs/pstore: fs/squashfs: Change usage of LZ4 to work with new LZ4 version
This patch updates fs/pstore and fs/squashfs to use the updated
functions from the new LZ4 module.
Signed-off-by: Sven Schmidt <4ssch...@informatik.uni-hamburg.de>
---
fs/pstore/platform.c | 14 +++---
fs/squashfs/lz4_wrapper.c | 12 ++--
2 files changed, 13 insertions(+), 13 deletions(-)
diff --git a/fs/pstore/platform.c b/fs/pstore/platform.c
index 729677e..85c4c58 100644
--- a/fs/pstore/platform.c
+++ b/fs/pstore/platform.c
@@ -342,31 +342,31 @@ static int compress_lz4(const void *in, void *out, size_t
inlen, size_t outlen)
{
int ret;
- ret = lz4_compress(in, inlen, out, , workspace);
- if (ret) {
+ ret = LZ4_compress_default(in, out, inlen, outlen, workspace);
+ if (!ret) {
pr_err("lz4_compress error, ret = %d!\n", ret);
return -EIO;
}
- return outlen;
+ return ret;
}
static int decompress_lz4(void *in, void *out, size_t inlen, size_t outlen)
{
int ret;
- ret = lz4_decompress_unknownoutputsize(in, inlen, out, );
- if (ret) {
+ ret = LZ4_decompress_safe(in, out, inlen, outlen);
+ if (ret < 0) {
pr_err("lz4_decompress error, ret = %d!\n", ret);
return -EIO;
}
- return outlen;
+ return ret;
}
static void allocate_lz4(void)
{
- big_oops_buf_sz = lz4_compressbound(psinfo->bufsize);
+ big_oops_buf_sz = LZ4_compressBound(psinfo->bufsize);
big_oops_buf = kmalloc(big_oops_buf_sz, GFP_KERNEL);
if (big_oops_buf) {
workspace = kmalloc(LZ4_MEM_COMPRESS, GFP_KERNEL);
diff --git a/fs/squashfs/lz4_wrapper.c b/fs/squashfs/lz4_wrapper.c
index ff4468b..95da653 100644
--- a/fs/squashfs/lz4_wrapper.c
+++ b/fs/squashfs/lz4_wrapper.c
@@ -97,7 +97,6 @@ static int lz4_uncompress(struct squashfs_sb_info *msblk,
void *strm,
struct squashfs_lz4 *stream = strm;
void *buff = stream->input, *data;
int avail, i, bytes = length, res;
- size_t dest_len = output->length;
for (i = 0; i < b; i++) {
avail = min(bytes, msblk->devblksize - offset);
@@ -108,12 +107,13 @@ static int lz4_uncompress(struct squashfs_sb_info *msblk,
void *strm,
put_bh(bh[i]);
}
- res = lz4_decompress_unknownoutputsize(stream->input, length,
- stream->output, _len);
- if (res)
+ res = LZ4_decompress_safe(stream->input, stream->output,
+ length, output->length);
+
+ if (res < 0)
return -EIO;
- bytes = dest_len;
+ bytes = res;
data = squashfs_first_page(output);
buff = stream->output;
while (data) {
@@ -128,7 +128,7 @@ static int lz4_uncompress(struct squashfs_sb_info *msblk,
void *strm,
}
squashfs_finish_page(output);
- return dest_len;
+ return res;
}
const struct squashfs_decompressor squashfs_lz4_comp_ops = {
--
2.1.4
--
To unsubscribe from this list: send the line "unsubscribe linux-crypto" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH 2/4] lib/decompress_unlz4: Change module to work with new LZ4 module version
This patch updates the unlz4 wrapper to work with the
updated LZ4 kernel module version.
Signed-off-by: Sven Schmidt <4ssch...@informatik.uni-hamburg.de>
---
lib/decompress_unlz4.c | 13 -
1 file changed, 8 insertions(+), 5 deletions(-)
diff --git a/lib/decompress_unlz4.c b/lib/decompress_unlz4.c
index 036fc88..1b0baf3 100644
--- a/lib/decompress_unlz4.c
+++ b/lib/decompress_unlz4.c
@@ -72,7 +72,7 @@ STATIC inline int INIT unlz4(u8 *input, long in_len,
error("NULL input pointer and missing fill function");
goto exit_1;
} else {
- inp = large_malloc(lz4_compressbound(uncomp_chunksize));
+ inp = large_malloc(LZ4_compressBound(uncomp_chunksize));
if (!inp) {
error("Could not allocate input buffer");
goto exit_1;
@@ -136,7 +136,7 @@ STATIC inline int INIT unlz4(u8 *input, long in_len,
inp += 4;
size -= 4;
} else {
- if (chunksize > lz4_compressbound(uncomp_chunksize)) {
+ if (chunksize > LZ4_compressBound(uncomp_chunksize)) {
error("chunk length is longer than allocated");
goto exit_2;
}
@@ -152,11 +152,14 @@ STATIC inline int INIT unlz4(u8 *input, long in_len,
out_len -= dest_len;
} else
dest_len = out_len;
- ret = lz4_decompress(inp, , outp, dest_len);
+
+ ret = LZ4_decompress_fast(inp, outp, dest_len);
+ chunksize = ret;
#else
dest_len = uncomp_chunksize;
- ret = lz4_decompress_unknownoutputsize(inp, chunksize, outp,
- _len);
+
+ ret = LZ4_decompress_safe(inp, outp, chunksize, dest_len);
+ dest_len = ret;
#endif
if (ret < 0) {
error("Decoding failed");
--
2.1.4
--
To unsubscribe from this list: send the line "unsubscribe linux-crypto" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH v3 0/4] Update LZ4 compressor module
This patchset is for updating the LZ4 compression module to a version based on LZ4 v1.7.3 allowing to use the fast compression algorithm aka LZ4 fast which provides an "acceleration" parameter as a tradeoff between high compression ratio and high compression speed. We want to use LZ4 fast in order to support compression in lustre and (mostly, based on that) investigate data reduction techniques in behalf of storage systems. Also, it will be useful for other users of LZ4 compression, as with LZ4 fast it is possible to enable applications to use fast and/or high compression depending on the usecase. For instance, ZRAM is offering a LZ4 backend and could benefit from an updated LZ4 in the kernel. LZ4 homepage: http://www.lz4.org/ LZ4 source repository: https://github.com/lz4/lz4 Source version: 1.7.3 Benchmark (taken from [1], Core i5-4300U @1.9GHz): |--||-- Compressor | Compression | Decompression | Ratio |--||-- memcpy | 4200 MB/s | 4200 MB/s | 1.000 LZ4 fast 50 | 1080 MB/s | 2650 MB/s | 1.375 LZ4 fast 17 | 680 MB/s | 2220 MB/s | 1.607 LZ4 fast 5 | 475 MB/s | 1920 MB/s | 1.886 LZ4 default | 385 MB/s | 1850 MB/s | 2.101 [1] http://fastcompression.blogspot.de/2015/04/sampling-or-faster-lz4.html [PATCHv2 1/4] lib: Update LZ4 compressor module based on LZ4 v1.7.2 [PATCHv2 2/4] lib: Update decompress_unlz4 wrapper to work with new LZ4 module [PATCHv2 3/4] crypto: Change lz4 modules to work with new LZ4 module [PATCHv2 4/4] fs/pstore: fs/squashfs: Change LZ4 compressor functions to work with new LZ4 module v2: - Changed order of the patches since in the initial patchset the lz4.h was in the last patch but was referenced by the other ones - Split lib/decompress_unlz4.c in an own patch - Fixed errors reported by the buildbot - Further refactorings - Added more appropriate copyright note to include/linux/lz4.h v3: - Adjusted the code to satisfy kernel coding style (checkpatch.pl) - Made sure the changes to LZ4 in Kernel (overflow checks etc.) are included in the new module (they are) - Removed the second LZ4_compressBound function with related name but different return type - Corrected version number (was LZ4 1.7.3) -- To unsubscribe from this list: send the line "unsubscribe linux-crypto" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
