вт, 30 мар. 2021 г. в 17:44, Willy Tarreau <w...@1wt.eu>: > On Tue, Mar 30, 2021 at 02:25:14PM +0500, ???? ??????? wrote: > > I would wait for feedback from some other high load projects. From my > > observation it is significant benefit. > > Originally slz was written for use in haproxy to solve the huge memory > consumption that comes with zlib (~288kB of context per stream IIRC). > SLZ needs much less and uses ~40 bytes or so per stream, and by keeping > less context, has less opportunities for high compression ratios. This > simplified its lookup algorithms, and as a byproduct made it significantly > faster. >
for example, silesia xml, Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz lzbench suite [root@localhost lzbench]# ./lzbench -ezlib,1/slz_zlib,1 silezia/xml lzbench 1.8 (64-bit Linux) Assembled by P.Skibinski Compressor name Compress. Decompress. Compr. size Ratio Filename memcpy 10948 MB/s 15766 MB/s 5345280 100.00 silezia/xml zlib 1.2.11 -1 125 MB/s 429 MB/s 965248 18.06 silezia/xml slz_zlib 1.2.0 -1 329 MB/s 331 MB/s 1294302 24.21 silezia/xml done... (cIters=1 dIters=1 cTime=1.0 dTime=2.0 chunkSize=1706MB cSpeed=0MB) [root@localhost lzbench]# I observe that slz is more than 2 times faster compared to zlib, while compressed file size is 20% bigger. but I did not yet checked zlib-ng. also, I read that slz stops compressing when CPU level reaches some threshold. is it related to all gzip, including zlib ? if so, we can safely stick with any compression lib (but I agree that having benchmarks would help people) > > The only sane way I've seen to use zlib is to limit its CPU usage. When it > reaches a target ratio (typically 80%), compression automatically disables > itself, so overall the savings-to-cpu usage makes it less interesting for > zlib in our use cases. The only case where zlib is more interesting is for > those having a small line and who prefer to spend more CPU cycles to get > more savings. > > One reasonable approach we could think of would consist in importing SLZ > directly into haproxy now. I didn't want to do it initially because the > code was young and I wanted it to live its own life and get its own fixes. > Now the burn-in test is long finished, and the only updates over the last > 4 years were for performance improvements or fixes for use cases outside > of haproxy. That's only 1500 lines of code to integrate and it would > certainly simplify a lot of things. We could even imagine to always build > with compression enabled, SLZ when not specified otherwise zlib when > asked for it. > > Just a few ideas. > Willy >
