Re: [squid-users] rock storage and max-swap-rate

[Alex Rousskov]

On 01/18/2018 05:35 PM, Ivan Larionov wrote:

> * Multiple squid swap in or swap out ops reading/writing contiguous
> blocks could be merged into one 256KB IO operation.

> * Random squid operations could be handled as single 32KB IO operation.

FWIW, on a busy Squid with a large disk cache in a stable state, there
should be virtually no mergeable/contiguous swap requests (first bullet)
due to random nature of rock slot assignment.

Alex.


> On Thu, Jan 18, 2018 at 3:20 PM, Amos Jeffries wrote:
> 
> On 19/01/18 12:04, Ivan Larionov wrote:
> 
> Thank you for the fast reply!
> 
> read_ops and write_ops is AWS EBS metric and in general it
> correlates with OS-level reads/s writes/s stats which iostat shows.
> 
> So if I understand you correctly max-swap-rate doesn't limit
> disk IOPS but limits squid swap ops instead and every squid
> operation could in theory use more than 1 disk IO operation.
> This means we can't really say "limit swap ops to 1500 because
> our disk can handle 1500 iops" but should figure out the number
> after testing different values.
> 
> Ok, I suppose I'll just do what Rock documentation says – will
> test different values and figure out what works for us.
> 
> 
> 
> If you know what the OS level IOP size is (eg usually 4KB) and the
> Squid rock IOP size Alex mentioned of 32KB. That should give you a
> number to divide the disk IOPS limit you want with to get a rough
> estimate for the appropriate Squid setting.
> 
> The tuning bit is just to see how much variance from that is caused
> by your traffic objects being different from the 32KB slot size.
> 
> 
> Amos
> 
> ___
> squid-users mailing list
> squid-users@lists.squid-cache.org
> 
> http://lists.squid-cache.org/listinfo/squid-users
> 
> 
> 
> 
> 
> -- 
> With best regards, Ivan Larionov.
> 
> 
> ___
> squid-users mailing list
> squid-users@lists.squid-cache.org
> http://lists.squid-cache.org/listinfo/squid-users
> 

___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] rock storage and max-swap-rate

[Ivan Larionov]

Thanks Amos.

According to AWS docs:

> I/O size is capped at 256 KiB for SSD volumes
> When small I/O operations are physically contiguous, Amazon EBS attempts
to merge them into a single I/O up to the maximum size. For example, for
SSD volumes, a single 1,024 KiB I/O operation counts as 4 operations
(1,024÷256=4), while 8 contiguous I/O operations at 32 KiB each count as
1operation (8×32=256). However, 8 random I/O operations at 32 KiB each
count as 8 operations. Each I/O operation under 32 KiB counts as 1
operation.

So it's not so easy to figure out correlation between squid swap ops and
AWS EBS ops. What I see from here is:

* Multiple squid swap in or swap out ops reading/writing contiguous blocks
could be merged into one 256KB IO operation.
* Random squid operations could be handled as single 32KB IO operation.

On Thu, Jan 18, 2018 at 3:20 PM, Amos Jeffries  wrote:

> On 19/01/18 12:04, Ivan Larionov wrote:
>
>> Thank you for the fast reply!
>>
>> read_ops and write_ops is AWS EBS metric and in general it correlates
>> with OS-level reads/s writes/s stats which iostat shows.
>>
>> So if I understand you correctly max-swap-rate doesn't limit disk IOPS
>> but limits squid swap ops instead and every squid operation could in theory
>> use more than 1 disk IO operation. This means we can't really say "limit
>> swap ops to 1500 because our disk can handle 1500 iops" but should figure
>> out the number after testing different values.
>>
>> Ok, I suppose I'll just do what Rock documentation says – will test
>> different values and figure out what works for us.
>>
>>
>
> If you know what the OS level IOP size is (eg usually 4KB) and the Squid
> rock IOP size Alex mentioned of 32KB. That should give you a number to
> divide the disk IOPS limit you want with to get a rough estimate for the
> appropriate Squid setting.
>
> The tuning bit is just to see how much variance from that is caused by
> your traffic objects being different from the 32KB slot size.
>
>
> Amos
>
> ___
> squid-users mailing list
> squid-users@lists.squid-cache.org
> http://lists.squid-cache.org/listinfo/squid-users
>



-- 
With best regards, Ivan Larionov.
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] rock storage and max-swap-rate

[Alex Rousskov]

On 01/18/2018 04:04 PM, Ivan Larionov wrote:

> So if I understand you correctly max-swap-rate doesn't limit disk IOPS

Correct. Squid does not know what the disk- or OS-level stats are.


> but limits squid swap ops instead 

If you define a "swap op" as reading or writing a single I/O page for
the purpose of swapping in or swapping out a cached object, then yes.
Processing an HTTP transaction may involve reading and writing many I/O
pages. See my original response for a more precise definition.


> and every squid operation could in
> theory use more than 1 disk IO operation. 

Yes, depending on the file system, disk, etc.

Ideally, rock cache_dirs should use raw disk partitions, but we do not
have a sponsor for that work.


> This means we can't really say
> "limit swap ops to 1500 because our disk can handle 1500 iops" but
> should figure out the number after testing different values.

Correct.


> Ok, I suppose I'll just do what Rock documentation says – will test
> different values and figure out what works for us.

Sounds like a good plan! You may even find a good/stable correlation
between Squid swap I/O ops and low-level disk I/O ops. Please consider
reporting your findings for others to reuse.


Cheers,

Alex.


> On Thu, Jan 18, 2018 at 2:54 PM, Alex Rousskov wrote:
> 
> On 01/18/2018 03:16 PM, Ivan Larionov wrote:
> 
> > cache_dir max-swap-rate documentation says that swap in requests
> > contribute to measured swap rate. However in our squid 4 load test we
> > see that read_ops + write_ops significantly exceeds the max-swap-rate we
> > set and squid doesn't limit it.
> 
> In this context, a single Squid "op" is a read or write request from
> worker to the disker process. These requests are up to one I/O page in
> size. A single I/O page is 32*1024 bytes. See Ipc::Mem::PageSize().
> 
> * A single read request usually ends up being a single pread(2) system
> call that reads at most one I/O page worth of data from disk. See
> diskerRead().
> 
> * A single write request usually ends up being a single pwrite(2) system
> call that writes at most one I/O page worth of data to disk. However, if
> that single pwrite() does not write everything a worker has requested to
> write, then Squid will make more pwrite() calls, up to 10 calls total.
> See diskerWrite().
> 
> Within a single cache miss transaction, the rock code should accumulate
> small swapout requests from Store into page-size write requests to
> disker, but I do not remember how complete those optimizations are: It
> is possible that smaller-than-page writes get through to diskers,
> increasing the number of write requests. Same for reading cache hits.
> 
> 
> What is the "op" in read_ops and write_ops you have measured?
> 
> 
> Since Squid does not (and does not want to) have access to low-level
> disk stats and since Squid cannot assume exlusive disk ownership, the
> rate-limiting feature for rock cache_dirs does not know how many
> low-level disk operations the disk is doing and how those operations
> correspond to what Squid is asking the disk to do.
> 
> 
> HTH,
> 
> Alex.
> 
> 
> > I tried to set it to 200 to confirm that it actually works and saw
> that
> > it does. Squid started warning about exceeding max-swap-rate. But
> looks
> > like real limit is higher than the value we set in configuration.
> >
> > Hardware:
> >
> > AWS GP2 EBS (SSD) 600GB, 1500 iops baseline performance, 3000 iops
> > burstable.
> >
> > Config:
> >
> > cache_dir rock /mnt/services/squid/cache 435200 swap-timeout=500
> > max-swap-rate=1200 slot-size=16384
> >
> > IOPS squid pushes under our load test:
> >
> > read ~800 ops/sec
> > write ~1100 ops/sec
> >
> > In summary it gives us ~1900 ops/sec which exceeds AWS limit of 1500
> > ops/sec and after spending too much "burst balance" we started getting
> > throttled from AWS side.
> >
> > Could you please comment on this behavior? What the limit should
> we set
> > to stay under 1500 ops/sec for swap in + swap out operations?
> >
> > Thanks.
> >
> > Squid version:
> >
> > Squid Cache: Version 4.0.22
> > Service Name: squid
> > configure options:  '--program-prefix=' '--prefix=/usr'
> > '--exec-prefix=/usr' '--bindir=/usr/sbin' '--sbindir=/usr/sbin'
> > '--sysconfdir=/etc/squid' '--libdir=/usr/lib'
> > '--libexecdir=/usr/lib/squid' '--includedir=/usr/include'
> > '--datadir=/usr/share' '--sharedstatedir=/usr/com'
> > '--localstatedir=/var' '--mandir=/usr/share/man'
> > '--infodir=/usr/share/info' '--enable-epoll'
> > '--enable-removal-policies=heap,lru' '--enable-storeio=aufs,rock'
> > '--enable-delay-pools' '--with-pthreads' '--enable-cache-digests'
> > '--with-large-files' '--with-maxfd=16384' '--enable-htcp'
> >
>   

Re: [squid-users] rock storage and max-swap-rate

[Amos Jeffries]

On 19/01/18 12:04, Ivan Larionov wrote:

Thank you for the fast reply!

read_ops and write_ops is AWS EBS metric and in general it correlates 
with OS-level reads/s writes/s stats which iostat shows.


So if I understand you correctly max-swap-rate doesn't limit disk IOPS 
but limits squid swap ops instead and every squid operation could in 
theory use more than 1 disk IO operation. This means we can't really say 
"limit swap ops to 1500 because our disk can handle 1500 iops" but 
should figure out the number after testing different values.


Ok, I suppose I'll just do what Rock documentation says – will test 
different values and figure out what works for us.





If you know what the OS level IOP size is (eg usually 4KB) and the Squid 
rock IOP size Alex mentioned of 32KB. That should give you a number to 
divide the disk IOPS limit you want with to get a rough estimate for the 
appropriate Squid setting.


The tuning bit is just to see how much variance from that is caused by 
your traffic objects being different from the 32KB slot size.



Amos
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] rock storage and max-swap-rate

[Ivan Larionov]

Thank you for the fast reply!

read_ops and write_ops is AWS EBS metric and in general it correlates with
OS-level reads/s writes/s stats which iostat shows.

So if I understand you correctly max-swap-rate doesn't limit disk IOPS but
limits squid swap ops instead and every squid operation could in theory use
more than 1 disk IO operation. This means we can't really say "limit swap
ops to 1500 because our disk can handle 1500 iops" but should figure out
the number after testing different values.

Ok, I suppose I'll just do what Rock documentation says – will test
different values and figure out what works for us.

Thanks again.

On Thu, Jan 18, 2018 at 2:54 PM, Alex Rousskov <
rouss...@measurement-factory.com> wrote:

> On 01/18/2018 03:16 PM, Ivan Larionov wrote:
>
> > cache_dir max-swap-rate documentation says that swap in requests
> > contribute to measured swap rate. However in our squid 4 load test we
> > see that read_ops + write_ops significantly exceeds the max-swap-rate we
> > set and squid doesn't limit it.
>
> In this context, a single Squid "op" is a read or write request from
> worker to the disker process. These requests are up to one I/O page in
> size. A single I/O page is 32*1024 bytes. See Ipc::Mem::PageSize().
>
> * A single read request usually ends up being a single pread(2) system
> call that reads at most one I/O page worth of data from disk. See
> diskerRead().
>
> * A single write request usually ends up being a single pwrite(2) system
> call that writes at most one I/O page worth of data to disk. However, if
> that single pwrite() does not write everything a worker has requested to
> write, then Squid will make more pwrite() calls, up to 10 calls total.
> See diskerWrite().
>
> Within a single cache miss transaction, the rock code should accumulate
> small swapout requests from Store into page-size write requests to
> disker, but I do not remember how complete those optimizations are: It
> is possible that smaller-than-page writes get through to diskers,
> increasing the number of write requests. Same for reading cache hits.
>
>
> What is the "op" in read_ops and write_ops you have measured?
>
>
> Since Squid does not (and does not want to) have access to low-level
> disk stats and since Squid cannot assume exlusive disk ownership, the
> rate-limiting feature for rock cache_dirs does not know how many
> low-level disk operations the disk is doing and how those operations
> correspond to what Squid is asking the disk to do.
>
>
> HTH,
>
> Alex.
>
>
> > I tried to set it to 200 to confirm that it actually works and saw that
> > it does. Squid started warning about exceeding max-swap-rate. But looks
> > like real limit is higher than the value we set in configuration.
> >
> > Hardware:
> >
> > AWS GP2 EBS (SSD) 600GB, 1500 iops baseline performance, 3000 iops
> > burstable.
> >
> > Config:
> >
> > cache_dir rock /mnt/services/squid/cache 435200 swap-timeout=500
> > max-swap-rate=1200 slot-size=16384
> >
> > IOPS squid pushes under our load test:
> >
> > read ~800 ops/sec
> > write ~1100 ops/sec
> >
> > In summary it gives us ~1900 ops/sec which exceeds AWS limit of 1500
> > ops/sec and after spending too much "burst balance" we started getting
> > throttled from AWS side.
> >
> > Could you please comment on this behavior? What the limit should we set
> > to stay under 1500 ops/sec for swap in + swap out operations?
> >
> > Thanks.
> >
> > Squid version:
> >
> > Squid Cache: Version 4.0.22
> > Service Name: squid
> > configure options:  '--program-prefix=' '--prefix=/usr'
> > '--exec-prefix=/usr' '--bindir=/usr/sbin' '--sbindir=/usr/sbin'
> > '--sysconfdir=/etc/squid' '--libdir=/usr/lib'
> > '--libexecdir=/usr/lib/squid' '--includedir=/usr/include'
> > '--datadir=/usr/share' '--sharedstatedir=/usr/com'
> > '--localstatedir=/var' '--mandir=/usr/share/man'
> > '--infodir=/usr/share/info' '--enable-epoll'
> > '--enable-removal-policies=heap,lru' '--enable-storeio=aufs,rock'
> > '--enable-delay-pools' '--with-pthreads' '--enable-cache-digests'
> > '--with-large-files' '--with-maxfd=16384' '--enable-htcp'
> >
> > --
> > With best regards, Ivan Larionov.
> >
> >
> > ___
> > squid-users mailing list
> > squid-users@lists.squid-cache.org
> > http://lists.squid-cache.org/listinfo/squid-users
> >
>
>


-- 
With best regards, Ivan Larionov.
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] rock storage and max-swap-rate

[Alex Rousskov]

On 01/18/2018 03:16 PM, Ivan Larionov wrote:

> cache_dir max-swap-rate documentation says that swap in requests
> contribute to measured swap rate. However in our squid 4 load test we
> see that read_ops + write_ops significantly exceeds the max-swap-rate we
> set and squid doesn't limit it.

In this context, a single Squid "op" is a read or write request from
worker to the disker process. These requests are up to one I/O page in
size. A single I/O page is 32*1024 bytes. See Ipc::Mem::PageSize().

* A single read request usually ends up being a single pread(2) system
call that reads at most one I/O page worth of data from disk. See
diskerRead().

* A single write request usually ends up being a single pwrite(2) system
call that writes at most one I/O page worth of data to disk. However, if
that single pwrite() does not write everything a worker has requested to
write, then Squid will make more pwrite() calls, up to 10 calls total.
See diskerWrite().

Within a single cache miss transaction, the rock code should accumulate
small swapout requests from Store into page-size write requests to
disker, but I do not remember how complete those optimizations are: It
is possible that smaller-than-page writes get through to diskers,
increasing the number of write requests. Same for reading cache hits.


What is the "op" in read_ops and write_ops you have measured?


Since Squid does not (and does not want to) have access to low-level
disk stats and since Squid cannot assume exlusive disk ownership, the
rate-limiting feature for rock cache_dirs does not know how many
low-level disk operations the disk is doing and how those operations
correspond to what Squid is asking the disk to do.


HTH,

Alex.


> I tried to set it to 200 to confirm that it actually works and saw that
> it does. Squid started warning about exceeding max-swap-rate. But looks
> like real limit is higher than the value we set in configuration.
> 
> Hardware:
> 
> AWS GP2 EBS (SSD) 600GB, 1500 iops baseline performance, 3000 iops
> burstable.
> 
> Config:
> 
> cache_dir rock /mnt/services/squid/cache 435200 swap-timeout=500
> max-swap-rate=1200 slot-size=16384
> 
> IOPS squid pushes under our load test:
> 
> read ~800 ops/sec
> write ~1100 ops/sec
> 
> In summary it gives us ~1900 ops/sec which exceeds AWS limit of 1500
> ops/sec and after spending too much "burst balance" we started getting
> throttled from AWS side.
> 
> Could you please comment on this behavior? What the limit should we set
> to stay under 1500 ops/sec for swap in + swap out operations?
> 
> Thanks.
> 
> Squid version:
> 
> Squid Cache: Version 4.0.22
> Service Name: squid
> configure options:  '--program-prefix=' '--prefix=/usr'
> '--exec-prefix=/usr' '--bindir=/usr/sbin' '--sbindir=/usr/sbin'
> '--sysconfdir=/etc/squid' '--libdir=/usr/lib'
> '--libexecdir=/usr/lib/squid' '--includedir=/usr/include'
> '--datadir=/usr/share' '--sharedstatedir=/usr/com'
> '--localstatedir=/var' '--mandir=/usr/share/man'
> '--infodir=/usr/share/info' '--enable-epoll'
> '--enable-removal-policies=heap,lru' '--enable-storeio=aufs,rock'
> '--enable-delay-pools' '--with-pthreads' '--enable-cache-digests'
> '--with-large-files' '--with-maxfd=16384' '--enable-htcp'
> 
> -- 
> With best regards, Ivan Larionov.
> 
> 
> ___
> squid-users mailing list
> squid-users@lists.squid-cache.org
> http://lists.squid-cache.org/listinfo/squid-users
> 

___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

[squid-users] rock storage and max-swap-rate

[Ivan Larionov]

Hello.

cache_dir max-swap-rate documentation says that swap in requests contribute
to measured swap rate. However in our squid 4 load test we see that
read_ops + write_ops significantly exceeds the max-swap-rate we set and
squid doesn't limit it.

I tried to set it to 200 to confirm that it actually works and saw that it
does. Squid started warning about exceeding max-swap-rate. But looks like
real limit is higher than the value we set in configuration.

Hardware:

AWS GP2 EBS (SSD) 600GB, 1500 iops baseline performance, 3000 iops
burstable.

Config:

cache_dir rock /mnt/services/squid/cache 435200 swap-timeout=500
max-swap-rate=1200 slot-size=16384

IOPS squid pushes under our load test:

read ~800 ops/sec
write ~1100 ops/sec

In summary it gives us ~1900 ops/sec which exceeds AWS limit of 1500
ops/sec and after spending too much "burst balance" we started getting
throttled from AWS side.

Could you please comment on this behavior? What the limit should we set to
stay under 1500 ops/sec for swap in + swap out operations?

Thanks.

Squid version:

Squid Cache: Version 4.0.22
Service Name: squid
configure options:  '--program-prefix=' '--prefix=/usr'
'--exec-prefix=/usr' '--bindir=/usr/sbin' '--sbindir=/usr/sbin'
'--sysconfdir=/etc/squid' '--libdir=/usr/lib' '--libexecdir=/usr/lib/squid'
'--includedir=/usr/include' '--datadir=/usr/share'
'--sharedstatedir=/usr/com' '--localstatedir=/var'
'--mandir=/usr/share/man' '--infodir=/usr/share/info' '--enable-epoll'
'--enable-removal-policies=heap,lru' '--enable-storeio=aufs,rock'
'--enable-delay-pools' '--with-pthreads' '--enable-cache-digests'
'--with-large-files' '--with-maxfd=16384' '--enable-htcp'

-- 
With best regards, Ivan Larionov.
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

[squid-users] [squid-announce] RFC: Squid ESI processor changes

[Amos Jeffries]

The Squid team are planning to remove the Custom XML parser used for ESI 
processing from the next Squid version.


At first this seemed like a simple removal if unused functionality. 
However during review of the changes it turns out this functionality may 
be used in many situations when it should not have been.



Can people using ESI please provide answers for these questions:


What is your setting for esi_parser squid.conf directive?

 a) did not set it
 b) "expat"
 c) "libxml2"
 d) "custom"

If you answered (d), why choose that one?


If you answer (a) does setting it explicitly to either of the other 
parsers cause a large impact on your traffic performance?


  Yes/No

If yes, which did you choose?

 ... and how much of an impact was seen?



If you are not comfortable answering this to the squid-dev mailing list 
please feel free to send your responses directly to me.


Amos Jeffries
The Squid Software Foundation

___
squid-announce mailing list
squid-annou...@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-announce
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] TCP out of memory

[Vieri]

From: Amos Jeffries 
>
> Sorry I have a bit of a distraction going on ATM so have not got to that 

> detailed check yet. Good to hear you found a slightly better situation > 
> though.
[...]
> In normal network conditions it should rise and fall with your peak vs 
> off-peak traffic times. I expect with your particular trouble it will 
> mostly just go upwards.


No worries. I'd like to confirm that I'm still seeing the same issue with 
c-icap-modules, even though it's slightly better in that the FD numbers grow 
slower, at least at first.
I must say that it seems to be growing faster now. I had 4k two days ago, now I 
have:
Largest file desc currently in use:   6664
Number of file desc currently in use: 6270
So it seesm that the more days go by, the faster the FD numbers rise.

Vieri
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users