Re: zlib vs slz (perfoarmance)

[Dinko Korunic]

> […]
> 
> 
> Hi Lukas,
> 
> I am maintaining haproxytech Docker images and I can easily make that (slz 
> being used) happen, if that’s what community would like to see.
> 

Hi,

Given quite a few positive responses from community about including SLZ by 
default, all new haproxytech/haproxy* Docker images have SLZ enabled and 
statically linked in the haproxy binary.


Kind regards,
D.

-- 
Dinko Korunic   ** Standard disclaimer applies **
Sent from OSF1 osf1v4b V4.0 564 alpha

Re: zlib vs slz (perfoarmance)

[Willy Tarreau]

On Tue, Mar 30, 2021 at 05:57:55PM +0500,  ??? wrote:
> also, I read that slz stops compressing when CPU level reaches some
> threshold. is it related to all gzip, including zlib ?

It's neither, it's haproxy which does this, based on "maxcompcpuusage"
(but it defaults to 100, i.e. unlimited). And similarly you can limit
zlib's memory usage using maxzlibmem, which will make it refrain from
compressing once the limit is reached. For me the memory is the real
problem here. 30k concurrent streams will consume 9 GB.

> if so, we can safely stick with any compression lib (but I agree that
> having benchmarks would help people)

There are a few benchmarks on the site and the readme, and some can be
found in dedicated tools. We could indeed produce some more detailed
tests on cheap dedicated servers with limited bandwidth/unlimited data
to see which one is better for which case, but I recall a few figures
such as "use zlib below 50 Mbps output link size on Atoms if you've
got lots of RAM otherwise use slz".

Willy

Re: zlib vs slz (perfoarmance)

[Илья Шипицин]

вт, 30 мар. 2021 г. в 17:44, Willy Tarreau :

> On Tue, Mar 30, 2021 at 02:25:14PM +0500,  ??? wrote:
> > I would wait for feedback from some other high load projects. From my
> > observation it is significant benefit.
>
> Originally slz was written for use in haproxy to solve the huge memory
> consumption that comes with zlib (~288kB of context per stream IIRC).
> SLZ needs much less and uses ~40 bytes or so per stream, and by keeping
> less context, has less opportunities for high compression ratios. This
> simplified its lookup algorithms, and as a byproduct made it significantly
> faster.
>


for example, silesia xml, Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz

lzbench suite

[root@localhost lzbench]# ./lzbench -ezlib,1/slz_zlib,1  silezia/xml
lzbench 1.8 (64-bit Linux)   Assembled by P.Skibinski
Compressor name Compress. Decompress. Compr. size  Ratio Filename
memcpy  10948 MB/s 15766 MB/s 5345280 100.00 silezia/xml
zlib 1.2.11 -1125 MB/s   429 MB/s  965248  18.06 silezia/xml
slz_zlib 1.2.0 -1 329 MB/s   331 MB/s 1294302  24.21 silezia/xml
done... (cIters=1 dIters=1 cTime=1.0 dTime=2.0 chunkSize=1706MB cSpeed=0MB)
[root@localhost lzbench]#


I observe that slz is more than 2 times faster compared to zlib, while
compressed file size is 20% bigger.
but I did not yet checked zlib-ng.

also, I read that slz stops compressing when CPU level reaches some
threshold. is it related to all gzip, including zlib ?
if so, we can safely stick with any compression lib (but I agree that
having benchmarks would help people)


>
> The only sane way I've seen to use zlib is to limit its CPU usage. When it
> reaches a target ratio (typically 80%), compression automatically disables
> itself, so overall the savings-to-cpu usage makes it less interesting for
> zlib in our use cases. The only case where zlib is more interesting is for
> those having a small line and who prefer to spend more CPU cycles to get
> more savings.
>
> One reasonable approach we could think of would consist in importing SLZ
> directly into haproxy now. I didn't want to do it initially because the
> code was young and I wanted it to live its own life and get its own fixes.
> Now the burn-in test is long finished, and the only updates over the last
> 4 years were for performance improvements or fixes for use cases outside
> of haproxy. That's only 1500 lines of code to integrate and it would
> certainly simplify a lot of things. We could even imagine to always build
> with compression enabled, SLZ when not specified otherwise zlib when
> asked for it.
>
> Just a few ideas.
> Willy
>

Re: zlib vs slz (perfoarmance)

[Willy Tarreau]

On Tue, Mar 30, 2021 at 02:25:14PM +0500,  ??? wrote:
> I would wait for feedback from some other high load projects. From my
> observation it is significant benefit.

Originally slz was written for use in haproxy to solve the huge memory
consumption that comes with zlib (~288kB of context per stream IIRC).
SLZ needs much less and uses ~40 bytes or so per stream, and by keeping
less context, has less opportunities for high compression ratios. This
simplified its lookup algorithms, and as a byproduct made it significantly
faster.

The only sane way I've seen to use zlib is to limit its CPU usage. When it
reaches a target ratio (typically 80%), compression automatically disables
itself, so overall the savings-to-cpu usage makes it less interesting for
zlib in our use cases. The only case where zlib is more interesting is for
those having a small line and who prefer to spend more CPU cycles to get
more savings.

One reasonable approach we could think of would consist in importing SLZ
directly into haproxy now. I didn't want to do it initially because the
code was young and I wanted it to live its own life and get its own fixes.
Now the burn-in test is long finished, and the only updates over the last
4 years were for performance improvements or fixes for use cases outside
of haproxy. That's only 1500 lines of code to integrate and it would
certainly simplify a lot of things. We could even imagine to always build
with compression enabled, SLZ when not specified otherwise zlib when
asked for it.

Just a few ideas.
Willy

Re: zlib vs slz (perfoarmance)

[Илья Шипицин]

I would wait for feedback from some other high load projects. From my
observation it is significant benefit.

On Tue, Mar 30, 2021, 12:18 PM Dinko Korunic 
wrote:

> On 30.03.2021., at 08:17, Илья Шипицин  wrote:
> >
> > I would really like to know whether zlib was chosen for purpose or by
> chance.
> >
> > And yes, some marketing campaign makes sense
> >
>
> Hi Илья,
>
> People tend to spawn Docker images in dozens and/or even hundreds and we
> always have to consider that adding a single library on top of what’s
> already present in the minimal Docker base image (libz is pretty much
> always present in base images) will result in additional size which is in
> general frown upon by Docker users.
>
> But then again, if the community (aka you) thinks that pros (performance)
> outweigh the cons (increased target size), I’ll take care of it for
> haproxytech images and these changes would be also propagated to Ingress
> Controller image as well.
>
>
> Kind regards,
> D.
>
> --
> Dinko Korunic   ** Standard disclaimer applies **
> Sent from OSF1 osf1v4b V4.0 564 alpha
>
>
>

Re: zlib vs slz (perfoarmance)

[Dinko Korunic]

On 30.03.2021., at 08:17, Илья Шипицин  wrote:
> 
> I would really like to know whether zlib was chosen for purpose or by chance.
> 
> And yes, some marketing campaign makes sense
> 

Hi Илья,

People tend to spawn Docker images in dozens and/or even hundreds and we always 
have to consider that adding a single library on top of what’s already present 
in the minimal Docker base image (libz is pretty much always present in base 
images) will result in additional size which is in general frown upon by Docker 
users.

But then again, if the community (aka you) thinks that pros (performance) 
outweigh the cons (increased target size), I’ll take care of it for haproxytech 
images and these changes would be also propagated to Ingress Controller image 
as well.


Kind regards,
D.

-- 
Dinko Korunic   ** Standard disclaimer applies **
Sent from OSF1 osf1v4b V4.0 564 alpha

Re: zlib vs slz (perfoarmance)

[Aleksandar Lazic]

+1

On 30.03.21 08:17, Илья Шипицин wrote:

I would really like to know whether zlib was chosen for purpose or by chance.

And yes, some marketing campaign makes sense

On Tue, Mar 30, 2021, 10:35 AM Dinko Korunic mailto:dinko.koru...@gmail.com>> wrote:


 > On 29.03.2021., at 23:06, Lukas Tribus mailto:lu...@ltri.eu>> wrote:
 >

[…]

 > Like I said last year, this needs a marketing campaign:
 > https://www.mail-archive.com/haproxy@formilux.org/msg38044.html 

 >
 >
 > What about the docker images from haproxytech? Are those zlib or slz
 > based? Perhaps that would be a better starting point?
 >
 > https://hub.docker.com/r/haproxytech/haproxy-alpine 




Hi Lukas,

I am maintaining haproxytech Docker images and I can easily make that (slz 
being used) happen, if that’s what community would like to see.


Kind regards,
D.

-- 
Dinko Korunic                   ** Standard disclaimer applies **

Sent from OSF1 osf1v4b V4.0 564 alpha

Re: zlib vs slz (perfoarmance)

[Илья Шипицин]

I would really like to know whether zlib was chosen for purpose or by
chance.

And yes, some marketing campaign makes sense

On Tue, Mar 30, 2021, 10:35 AM Dinko Korunic 
wrote:

>
> > On 29.03.2021., at 23:06, Lukas Tribus  wrote:
> >
>
> […]
>
> > Like I said last year, this needs a marketing campaign:
> > https://www.mail-archive.com/haproxy@formilux.org/msg38044.html
> >
> >
> > What about the docker images from haproxytech? Are those zlib or slz
> > based? Perhaps that would be a better starting point?
> >
> > https://hub.docker.com/r/haproxytech/haproxy-alpine
>
>
>
> Hi Lukas,
>
> I am maintaining haproxytech Docker images and I can easily make that (slz
> being used) happen, if that’s what community would like to see.
>
>
> Kind regards,
> D.
>
> --
> Dinko Korunic   ** Standard disclaimer applies **
> Sent from OSF1 osf1v4b V4.0 564 alpha
>
>
>

Re: zlib vs slz (perfoarmance)

[Dinko Korunic]

> On 29.03.2021., at 23:06, Lukas Tribus  wrote:
> 

[…]

> Like I said last year, this needs a marketing campaign:
> https://www.mail-archive.com/haproxy@formilux.org/msg38044.html
> 
> 
> What about the docker images from haproxytech? Are those zlib or slz
> based? Perhaps that would be a better starting point?
> 
> https://hub.docker.com/r/haproxytech/haproxy-alpine



Hi Lukas,

I am maintaining haproxytech Docker images and I can easily make that (slz 
being used) happen, if that’s what community would like to see.


Kind regards,
D.

-- 
Dinko Korunic   ** Standard disclaimer applies **
Sent from OSF1 osf1v4b V4.0 564 alpha

Re: zlib vs slz (perfoarmance)

[Lukas Tribus]

Hello,


On Mon, 29 Mar 2021 at 20:54, Илья Шипицин  wrote:
>> > Dear list,
>> >
>> > on browser load (html + js + css) I observe 80% of cpu spent on gzip.
>> > also, I observe that zlib is probably one of the slowest implementation
>> > my personal benchmark correlate with https://github.com/inikep/lzbench
>> >
>> > if so, should'n we switch to slz by default ? or am I missing something ?
>>
>> There is no default.
>>
>> zlib is optional.
>> slz is optional.
>>
>> Haproxy is compiled with either zlib, slz or no compression library at all.
>>
>>
>> What specifically are you suggesting to change in the haproxy source tree?
>
>
> for example, docker image
> https://hub.docker.com/_/haproxy

So nothing we control directly.

Docker images, package repositories, etc. This means filing requests
at those places, convincing other people to switch from a well known
library to an unknown library that isn't even packaged yet in most
places, that those maintainers then have to support. I'm not sure how
realistic that is.

Like I said last year, this needs a marketing campaign:
https://www.mail-archive.com/haproxy@formilux.org/msg38044.html


What about the docker images from haproxytech? Are those zlib or slz
based? Perhaps that would be a better starting point?

https://hub.docker.com/r/haproxytech/haproxy-alpine


Lukas

Re: zlib vs slz (perfoarmance)

[Илья Шипицин]

пн, 29 мар. 2021 г. в 19:35, Lukas Tribus :

> Hi Ilya,
>
>
> On Mon, 29 Mar 2021 at 15:34, Илья Шипицин  wrote:
> >
> > Dear list,
> >
> > on browser load (html + js + css) I observe 80% of cpu spent on gzip.
> > also, I observe that zlib is probably one of the slowest implementation
> > my personal benchmark correlate with https://github.com/inikep/lzbench
> >
> > if so, should'n we switch to slz by default ? or am I missing something ?
>
> There is no default.
>
> zlib is optional.
> slz is optional.
>
> Haproxy is compiled with either zlib, slz or no compression library at all.
>
>
> What specifically are you suggesting to change in the haproxy source tree?
>

for example, docker image
https://hub.docker.com/_/haproxy


>
>
> Regards,
> Lukas
>

Re: zlib vs slz (perfoarmance)

[Lukas Tribus]

Hi Ilya,


On Mon, 29 Mar 2021 at 15:34, Илья Шипицин  wrote:
>
> Dear list,
>
> on browser load (html + js + css) I observe 80% of cpu spent on gzip.
> also, I observe that zlib is probably one of the slowest implementation
> my personal benchmark correlate with https://github.com/inikep/lzbench
>
> if so, should'n we switch to slz by default ? or am I missing something ?

There is no default.

zlib is optional.
slz is optional.

Haproxy is compiled with either zlib, slz or no compression library at all.


What specifically are you suggesting to change in the haproxy source tree?


Regards,
Lukas