[squid-users] Squid3: 100 % CPU load during object caching

[Jens Offenbach]

I am running Squid3 3.3.8 on Ubuntu 14.04. Squid3 has been installed from the 
Ubuntu package repository. In my scenario, Squid has to cache big files >= 1 
GB. At the moment, I am getting very bad transfer rates lower that 1 MB/sec. I 
have checked the connectivity using iperf3. It gives my a bandwith of 853 
Mbits/sec between the nodes.

I have tried to investigate the problem and recognized that when there is no 
cache hit for a requested object, the Squid process reaches shortly after 
startup 100 % of one CPU core. The download rate drops down to 1 MB/sec. When I 
have a cache hit, I only get 30 MB/sec in my download.

Is there someting wrong with my config? I have already used Squid 3.3.14. I get 
the same result. Unfortunately, I was not able to build Squid 3.5.5 and 3.5.6.

Here is my squid.conf:
# ACCESS CONTROLS
# 
  acl intranetsrc 139.2.0.0/16
  acl intranetsrc 193.96.112.0/21
  acl intranetsrc 192.109.216.0/24
  acl intranetsrc 100.1.4.0/22
  acl localnetsrc 10.0.0.0/8
  acl localnetsrc 172.16.0.0/12
  acl localnetsrc 192.168.0.0/16
  acl localnetsrc fc00::/7
  acl localnetsrc fe80::/10
  acl to_intranet dst 139.2.0.0/16
  acl to_intranet dst 193.96.112.0/21
  acl to_intranet dst 192.109.216.0/24
  acl to_intranet dst 100.1.4.0/22
  acl to_localnet dst 10.0.0.0/8
  acl to_localnet dst 172.16.0.0/12
  acl to_localnet dst 192.168.0.0/16
  acl to_localnet dst fc00::/7
  acl to_localnet dst fe80::/10
  http_access allow manager localhost
  http_access deny  manager
  http_access allow localnet
  http_access allow localhost
  http_access deny all

# NETWORK OPTIONS
# 
  http_port 0.0.0.0:3128

# OPTIONS WHICH AFFECT THE NEIGHBOR SELECTION ALGORITHM
# 
  cache_peer proxy.mycompany.de parent 8080 0 no-query no-digest

# MEMORY CACHE OPTIONS
# 
  maximum_object_size_in_memory 1 GB
  memory_replacement_policy heap LFUDA
  cache_mem 4 GB

# DISK CACHE OPTIONS
# 
  maximum_object_size 10 GB
  cache_replacement_policy heap GDSF
  cache_dir aufs /var/cache/squid3 88894 16 256 max-size=10737418240

# LOGFILE OPTIONS
# 
  access_log daemon:/var/log/squid3/access.log squid
  cache_store_log daemon:/var/log/squid3/store.log

# OPTIONS FOR TROUBLESHOOTING
# 
  cache_log /var/log/squid3/cache.log
  coredump_dir /var/log/squid3

# OPTIONS FOR TUNING THE CACHE
# 
  cache allow localnet
  cache allow localhost
  cache allow intranet
  cache deny  all
  refresh_pattern ^ftp:  144020%10080
  refresh_pattern ^gopher:   1440 0% 1440
  refresh_pattern -i (/cgi-bin/|\?) 0 0%0
  refresh_pattern . 020% 4320

# HTTP OPTIONS
# 
  via off

# ADMINISTRATIVE PARAMETERS
# 
  cache_effective_user proxy
  cache_effective_group proxy

# ICP OPTIONS
# 
  icp_port 0

# OPTIONS INFLUENCING REQUEST FORWARDING 
# 
  nonhierarchical_direct on
  prefer_direct off
  always_direct allow to_localnet
  always_direct allow to_localhost
  always_direct allow to_intranet
  never_direct  allow all

# MISCELLANEOUS
# 
  memory_pools off
  forwarded_for off
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] Squid3: 100 % CPU load during object caching

[Amos Jeffries]

On 21/07/2015 7:59 p.m., Jens Offenbach wrote:
> I am running Squid3 3.3.8 on Ubuntu 14.04. Squid3 has been installed from the 
> Ubuntu package repository. In my scenario, Squid has to cache big files >= 1 
> GB. At the moment, I am getting very bad transfer rates lower that 1 MB/sec. 
> I have checked the connectivity using iperf3. It gives my a bandwith of 853 
> Mbits/sec between the nodes.
> 
> I have tried to investigate the problem and recognized that when there is no 
> cache hit for a requested object, the Squid process reaches shortly after 
> startup 100 % of one CPU core. The download rate drops down to 1 MB/sec. When 
> I have a cache hit, I only get 30 MB/sec in my download.
> 
> Is there someting wrong with my config? I have already used Squid 3.3.14. I 
> get the same result. Unfortunately, I was not able to build Squid 3.5.5 and 
> 3.5.6.
> 

Squid-3 is better able to cope with large objects than Squid-2 was. But
there are still significant problems.


Firstly, you only have space in memory for 4x 1GB objects. Total. if you
are dealing with such large objects at any regular frequency, you need
at least a much larger cache_mem setting.


Secondly, consider that Squid-3.3 places *all* active transactions into
cache_mem. 4GB of memory cache can store ~4 million x 1KB transactions,
or only 4 x 1GB transactions.

 If you have a cache full of small objects happily sitting in memory.
Then a requests for a 1GB object comes in, a hugh number of those small
objects need to be pushed out of memory cache onto disk, the memory
reallocated for use by the big one, and possibly 1GB object loaded from
disk into memory cache.

Then consider that GB sized object sitting in cache as it gets near to
being the oldest in memory. The next request is probably a puny little
0-1KB object, Squid may have to repeat all the GB size shufflings to and
from disk just to make memory space for that KB.

As you can imagine any one part of that process takes a lot of work and
time with a big object involved as compared to only small objects being
involved. The whole set of actions can be excruciatingly painful if the
proxy is busy.


Thirdly, you also only have 88GB of disk cache total. Neither that nor
the memory cache is sufficient to be trying to cache GB sized objects.
The tradeoff is whether one GB size object is going to get enough HITs
often enough to be worth not caching the million or so smaller objects
that could be taking its place. For most uses the tradeoff only makes
sense with high traffic on the large objects and/or TB of disk space.


My rule-of-thumb advice for caching is to keep it so that you can store
at least a few thousand maximum-sized objects at once in the allocated size.

So 4GB memory cache reasonable for 1MB size objects, 80GB disk cache is
reasonable for ~100MB sized objects.

That keeps almost all web page traffic able to be in memory, bigger but
popular media/video objects on disk. And the big things like Windows
Service Packs or whole DVD downloads get slower network fetches as
needed. If those latter are actually a problem for you get a bigger disk
cache, you *will* need it.


And a free audit for your config...


> Here is my squid.conf:
> # ACCESS CONTROLS
> # 
>   acl intranetsrc 139.2.0.0/16
>   acl intranetsrc 193.96.112.0/21
>   acl intranetsrc 192.109.216.0/24
>   acl intranetsrc 100.1.4.0/22
>   acl localnetsrc 10.0.0.0/8
>   acl localnetsrc 172.16.0.0/12
>   acl localnetsrc 192.168.0.0/16
>   acl localnetsrc fc00::/7
>   acl localnetsrc fe80::/10
>   acl to_intranet dst 139.2.0.0/16
>   acl to_intranet dst 193.96.112.0/21
>   acl to_intranet dst 192.109.216.0/24
>   acl to_intranet dst 100.1.4.0/22
>   acl to_localnet dst 10.0.0.0/8
>   acl to_localnet dst 172.16.0.0/12
>   acl to_localnet dst 192.168.0.0/16
>   acl to_localnet dst fc00::/7
>   acl to_localnet dst fe80::/10

The intended purpose behind the localnet and to_localnet ACLs is that
they are matching your intranet / LAN  / local network ranges.

The ones we distribute are just common standard ranges. You can simplify
your config by adding the intranet ranges to localnet and dropping all
the 'intranet' ACLs.

... BUT ...


>   http_access allow manager localhost
>   http_access deny  manager
>   http_access allow localnet
>   http_access allow localhost
>   http_access deny all

... noting how the intranet ACLs are not used to permit access through
the proxy. Maybe just dropping them entirely is better. If this is a
working proxy they are not being used.


> 
> # NETWORK OPTIONS
> # 
>   http_port 0.0.0.0:3128
> 
> # OPTIONS WHICH AFFECT THE NEIGHBOR SELECTION ALGORITHM
> # 
>   cache_peer proxy.mycompany.de parent 8080 0 no-query no-digest
> 
> # MEMORY CACHE OPTI

Re: [squid-users] Squid3: 100 % CPU load during object caching

[Eliezer Croitoru]

On 21/07/2015 10:59, Jens Offenbach wrote:

Is there someting wrong with my config? I have already used Squid 3.3.14. I get 
the same result. Unfortunately, I was not able to build Squid 3.5.5 and 3.5.6.


What was the issue?
I am using 3.5.6 on 14.04.2 64 bit.

Eliezer

___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] Squid3: 100 % CPU load during object caching

[Amos Jeffries]

On 22/07/2015 12:31 a.m., Jens Offenbach wrote:
> Thank you very much for your detailed explainations. We want to use Squid in 
> order to accelerate our automated software setup processes via Puppet. 
> Actually 
> Squid will host only a very short amount of large objects (10-20). Its 
> purpose 
> is not to cache web traffic or little objects.

Ah, Squid does not "host", it caches. The difference may seem trivial at
first glance but it is the critical factor between whether a proxy or a
local web server is the best tool for the job.

From my own experiences with Puppet, yes Squid is the right tool. But
only because the Puppet server was using relatively slow python code to
generate objects and not doing server-side caching on its own. If that
situation has changed in recent years then Squids usefulness will also
have changed.


> The hit-ratio for all the hosted 
> objects will be very high, because most of our VMs require the same software 
> stack.
> I will update mit config regarding to your comments! Thanks a lot!
> But actually I have still no idea, why the download rates are so 
> unsatisfying. 
> We are sill in the test phase. We have only one client that requests a large 
> object from Squid and the transfer rates are lower than 1 MB/sec during cache 
> build-up without any form of concurrency. Have vou got an idea what could be 
> the 
> source of the problem here? Why causes the Squid process 100 % CPU usage.

I did not see any config causing the known 100% CPU bugs to be
encountered in your case (eg. HTTPS going through delay pools guarantees
100% CPU). Which leads me to think its probably related to memory
shuffling. ( appears
to be the same and still unidentified)

As for speed, if the CPU is maxed out by one particular action Squid
wont have time for much other work. So things go slow.

On the other hand Squid is also optimized for relatively high traffic
usage. For very small client counts (such as under-10) it is effectively
running in idle mode 99% of the time. The I/O event loop starts pausing
for 10ms blocks waiting to see if some more useful amount of work can be
done at the end of the wait. That can lead to apparent network slowdown
as TCP gets up to 10ms delay per packet. But that should not be visible
in CPU numbers.


That said, 1 client can still max out Squid CPU and/or NIC throughput
capacity on a single request if its pushing/pulling packets fast enough.


If you can attach the strace tool to Squid when its consuming the CPU
there might be some better hints about where to look.


Cheers
Amos

___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] Squid3: 100 % CPU load during object caching

[Jens Offenbach]

I checked the bug you have mentioned and I think I am confronted with the same issue. I was able to build and test Squid 3.5.6 on Ubuntu 14.04.2 x84_64. I observed the same behavior. I have tested an 8 GB archive file and I get 100 % CPU usage and a download rate of nearly 500 KB/sec when the object gets cached. I have attached strace to the running process, but I killed it after 30 minutes. The whole takes hours, although we have a 1-GBit ethernet:

Process 4091 attached
Process 4091 detached
% time seconds usecs/call calls errors syscall
-- --- --- - - 
78.83 2.622879 1 1823951 write
12.29 0.408748 2 228029 2 read
6.18 0.205663 0 912431 1 epoll_wait
2.58 0.085921 0 456020 epoll_ctl
0.09 0.002919 0 6168 brk
0.02 0.000623 2 356 openat
0.01 0.000286 0 712 getdents
0.00 0.71 1 91 getrusage
0.00 0.38 0 362 close
0.00 0.03 2 2 sendto
0.00 0.01 0 3 1 recvfrom
0.00 0.00 0 2 open
0.00 0.00 0 3 stat
0.00 0.00 0 1 1 rt_sigreturn
0.00 0.00 0 1 kill
0.00 0.00 0 4 fcntl
0.00 0.00 0 2 2 unlink
0.00 0.00 0 1 getppid
-- --- --- - - 
100.00 3.327152 3428139 7 total

Can I do anything that helps to get ride of this problem?
 

Gesendet: Dienstag, 21. Juli 2015 um 17:37 Uhr
Von: "Amos Jeffries" 
An: "Jens Offenbach" , "squid-users@lists.squid-cache.org" 
Betreff: Re: Aw: Re: [squid-users] Squid3: 100 % CPU load during object caching
On 22/07/2015 12:31 a.m., Jens Offenbach wrote:
> Thank you very much for your detailed explainations. We want to use Squid in
> order to accelerate our automated software setup processes via Puppet. Actually
> Squid will host only a very short amount of large objects (10-20). Its purpose
> is not to cache web traffic or little objects.

Ah, Squid does not "host", it caches. The difference may seem trivial at
first glance but it is the critical factor between whether a proxy or a
local web server is the best tool for the job.

>From my own experiences with Puppet, yes Squid is the right tool. But
only because the Puppet server was using relatively slow python code to
generate objects and not doing server-side caching on its own. If that
situation has changed in recent years then Squids usefulness will also
have changed.


> The hit-ratio for all the hosted
> objects will be very high, because most of our VMs require the same software stack.
> I will update mit config regarding to your comments! Thanks a lot!
> But actually I have still no idea, why the download rates are so unsatisfying.
> We are sill in the test phase. We have only one client that requests a large
> object from Squid and the transfer rates are lower than 1 MB/sec during cache
> build-up without any form of concurrency. Have vou got an idea what could be the
> source of the problem here? Why causes the Squid process 100 % CPU usage.

I did not see any config causing the known 100% CPU bugs to be
encountered in your case (eg. HTTPS going through delay pools guarantees
100% CPU). Which leads me to think its probably related to memory
shuffling. (<http://bugs.squid-cache.org/show_bug.cgi?id=3189> appears
to be the same and still unidentified)

As for speed, if the CPU is maxed out by one particular action Squid
wont have time for much other work. So things go slow.

On the other hand Squid is also optimized for relatively high traffic
usage. For very small client counts (such as under-10) it is effectively
running in idle mode 99% of the time. The I/O event loop starts pausing
for 10ms blocks waiting to see if some more useful amount of work can be
done at the end of the wait. That can lead to apparent network slowdown
as TCP gets up to 10ms delay per packet. But that should not be visible
in CPU numbers.


That said, 1 client can still max out Squid CPU and/or NIC throughput
capacity on a single request if its pushing/pulling packets fast enough.


If you can attach the strace tool to Squid when its consuming the CPU
there might be some better hints about where to look.


Cheers
Amos
 
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] Squid3: 100 % CPU load during object caching

[Eliezer Croitoru]

Can you share the relevant squid.conf settings? Just to reproduce..

I have a dedicated testing server here which I can test the issue on.
8GB archive which might be an ISO and can be cached on AUFS\UFS and 
LARGE ROCK cache types.


I am pretty sure that the maximum cache object size is one thing to 
change and waht more?


From What I understand it should not be different for 2GB cached 
archive and to 8 GB cached archive.

I have a local copy of centos 7 ISO which should be a test worthy object.
Anything more you can add to the test subject?

Eliezer

On 22/07/2015 16:24, Jens Offenbach wrote:

I checked the bug you have mentioned and I think I am confronted with the same
issue. I was able to build and test Squid 3.5.6 on Ubuntu 14.04.2 x84_64. I
observed the same behavior. I have tested an 8 GB archive file and I get 100 %
CPU usage and a download rate of nearly 500 KB/sec when the object gets cached.
I have attached strace to the running process, but I killed it after 30 minutes.
The whole takes hours, although we have a 1-GBit ethernet:

Process 4091 attached
Process 4091 detached
% time seconds usecs/call calls errors syscall
-- --- --- - - 
78.83 2.622879 1 1823951 write
12.29 0.408748 2 228029 2 read
6.18 0.205663 0 912431 1 epoll_wait
2.58 0.085921 0 456020 epoll_ctl
0.09 0.002919 0 6168 brk
0.02 0.000623 2 356 openat
0.01 0.000286 0 712 getdents
0.00 0.71 1 91 getrusage
0.00 0.38 0 362 close
0.00 0.03 2 2 sendto
0.00 0.01 0 3 1 recvfrom
0.00 0.00 0 2 open
0.00 0.00 0 3 stat
0.00 0.00 0 1 1 rt_sigreturn
0.00 0.00 0 1 kill
0.00 0.00 0 4 fcntl
0.00 0.00 0 2 2 unlink
0.00 0.00 0 1 getppid
-- --- --- - - 
100.00 3.327152 3428139 7 total

Can I do anything that helps to get ride of this problem?


Gesendet: Dienstag, 21. Juli 2015 um 17:37 Uhr
Von: "Amos Jeffries" 
An: "Jens Offenbach" , "squid-users@lists.squid-cache.org"

Betreff: Re: Aw: Re: [squid-users] Squid3: 100 % CPU load during object caching
On 22/07/2015 12:31 a.m., Jens Offenbach wrote:
  > Thank you very much for your detailed explainations. We want to use Squid in
  > order to accelerate our automated software setup processes via Puppet. 
Actually
  > Squid will host only a very short amount of large objects (10-20). Its 
purpose
  > is not to cache web traffic or little objects.

Ah, Squid does not "host", it caches. The difference may seem trivial at
first glance but it is the critical factor between whether a proxy or a
local web server is the best tool for the job.

  From my own experiences with Puppet, yes Squid is the right tool. But
only because the Puppet server was using relatively slow python code to
generate objects and not doing server-side caching on its own. If that
situation has changed in recent years then Squids usefulness will also
have changed.


  > The hit-ratio for all the hosted
  > objects will be very high, because most of our VMs require the same software
stack.
  > I will update mit config regarding to your comments! Thanks a lot!
  > But actually I have still no idea, why the download rates are so 
unsatisfying.
  > We are sill in the test phase. We have only one client that requests a large
  > object from Squid and the transfer rates are lower than 1 MB/sec during 
cache
  > build-up without any form of concurrency. Have vou got an idea what could 
be the
  > source of the problem here? Why causes the Squid process 100 % CPU usage.

I did not see any config causing the known 100% CPU bugs to be
encountered in your case (eg. HTTPS going through delay pools guarantees
100% CPU). Which leads me to think its probably related to memory
shuffling. (<http://bugs.squid-cache.org/show_bug.cgi?id=3189
<https://3c.gmx.net/mail/client/dereferrer?redirectUrl=http%3A%2F%2Fbugs.squid-cache.org%2Fshow_bug.cgi%3Fid%3D3189>>
appears
to be the same and still unidentified)

As for speed, if the CPU is maxed out by one particular action Squid
wont have time for much other work. So things go slow.

On the other hand Squid is also optimized for relatively high traffic
usage. For very small client counts (such as under-10) it is effectively
running in idle mode 99% of the time. The I/O event loop starts pausing
for 10ms blocks waiting to see if some more useful amount of work can be
done at the end of the wait. That can lead to apparent network slowdown
as TCP gets up to 10ms delay per packet. But that should not be visible
in CPU numbers.


That said, 1 client can still max out Squid CPU and/or NIC throughput
capacity on a single request if its pushing/pulling packets fast enough.


If you can attach the strace tool to Squid when its consuming the CPU
there might be some better hints about where to look.


Cheers
Amos



___
squid-users mailing 

Re: [squid-users] Squid3: 100 % CPU load during object caching

[Jens Offenbach]

I will send you my current settings tomorrow. I have used AUFS as caching 
format, but I have also tested UFS. The format seems to have no influence on 
the issue.

I have tested the 1 GB Ubuntu 15.04 image (ubuntu-15.04-desktop-amd64.iso). 
This is the link 
http://releases.ubuntu.com/15.04/ubuntu-15.04-desktop-amd64.iso.

If you want to stress caching more with large files. You can use one of those:
https://surfer.nmr.mgh.harvard.edu/fswiki/Download

But I think the Centos 7 ISO are large enough, In my test scenario, I have put 
all files on an internal web server with gives them in stable 120 MB/sec. So 
the problem does not come from a slow network connection. I have checked 
network connectivity with Iperf3 (>= 900 MBit/sec) and made a direct wget 
without Squid. The file gets downloaded in high speed. Adding Squid in the 
communication flow which caches the file on the first request and the issue 
occurrs. After some minutes, the download rate drops to 500 KByte/sec and stays 
on this level together with 100 % CPU load. The download rate corresponds with 
the disk IO. The file gets written with 500 KByte/sec.

Thank you very much! 
 

Gesendet: Mittwoch, 22. Juli 2015 um 18:28 Uhr
Von: "Eliezer Croitoru" 
An: squid-users@lists.squid-cache.org
Betreff: Re: [squid-users] Squid3: 100 % CPU load during object caching
Can you share the relevant squid.conf settings? Just to reproduce..

I have a dedicated testing server here which I can test the issue on.
8GB archive which might be an ISO and can be cached on AUFS\UFS and
LARGE ROCK cache types.

I am pretty sure that the maximum cache object size is one thing to
change and waht more?

From What I understand it should not be different for 2GB cached
archive and to 8 GB cached archive.
I have a local copy of centos 7 ISO which should be a test worthy object.
Anything more you can add to the test subject?

Eliezer

On 22/07/2015 16:24, Jens Offenbach wrote:
> I checked the bug you have mentioned and I think I am confronted with the same
> issue. I was able to build and test Squid 3.5.6 on Ubuntu 14.04.2 x84_64. I
> observed the same behavior. I have tested an 8 GB archive file and I get 100 %
> CPU usage and a download rate of nearly 500 KB/sec when the object gets 
> cached.
> I have attached strace to the running process, but I killed it after 30 
> minutes.
> The whole takes hours, although we have a 1-GBit ethernet:
>
> Process 4091 attached
> Process 4091 detached
> % time seconds usecs/call calls errors syscall
> -- --- --- - - 
> 78.83 2.622879 1 1823951 write
> 12.29 0.408748 2 228029 2 read
> 6.18 0.205663 0 912431 1 epoll_wait
> 2.58 0.085921 0 456020 epoll_ctl
> 0.09 0.002919 0 6168 brk
> 0.02 0.000623 2 356 openat
> 0.01 0.000286 0 712 getdents
> 0.00 0.71 1 91 getrusage
> 0.00 0.38 0 362 close
> 0.00 0.03 2 2 sendto
> 0.00 0.01 0 3 1 recvfrom
> 0.00 0.00 0 2 open
> 0.00 0.00 0 3 stat
> 0.00 0.00 0 1 1 rt_sigreturn
> 0.00 0.00 0 1 kill
> 0.00 0.00 0 4 fcntl
> 0.00 0.00 0 2 2 unlink
> 0.00 0.00 0 1 getppid
> -- --- --- - - 
> 100.00 3.327152 3428139 7 total
>
> Can I do anything that helps to get ride of this problem?
>
>
> Gesendet: Dienstag, 21. Juli 2015 um 17:37 Uhr
> Von: "Amos Jeffries" 
> An: "Jens Offenbach" , "squid-users@lists.squid-cache.org"
> 
> Betreff: Re: Aw: Re: [squid-users] Squid3: 100 % CPU load during object 
> caching
> On 22/07/2015 12:31 a.m., Jens Offenbach wrote:
> > Thank you very much for your detailed explainations. We want to use Squid in
> > order to accelerate our automated software setup processes via Puppet. 
> > Actually
> > Squid will host only a very short amount of large objects (10-20). Its 
> > purpose
> > is not to cache web traffic or little objects.
>
> Ah, Squid does not "host", it caches. The difference may seem trivial at
> first glance but it is the critical factor between whether a proxy or a
> local web server is the best tool for the job.
>
> From my own experiences with Puppet, yes Squid is the right tool. But
> only because the Puppet server was using relatively slow python code to
> generate objects and not doing server-side caching on its own. If that
> situation has changed in recent years then Squids usefulness will also
> have changed.
>
>
> > The hit-ratio for all the hosted
> > objects will be very high, because most of our VMs require the same software
> stack.
> > I will update mit config regarding to your comments! Thanks a lot!
> > But actually I have still no idea, why the download rates are so 
> > unsatisfying.
> > We are sill in the test phas

Re: [squid-users] Squid3: 100 % CPU load during object caching

[Eliezer Croitoru]

Hey Jens,

I have tested the issue with LARGE ROCK and not AUFS or UFS.
Using squid or not my connection to the server is about 2.5 MBps (20Mbps).
Squid is sitting on an intel atom with SSD drive and on a HIT case the 
download speed is more then doubled to 4.5 MBps(36Mbps).

I have not tried it with AUFS yet.

My testing machine is an ARCH linux with self compiled squid with the 
replacement of diskd to rock from the compilation options of arch linux.


You can take a look at the HIT log at:
http://paste.ngtech.co.il/pnhkglgsu

Eliezer

On 22/07/2015 21:07, Jens Offenbach wrote:

I will send you my current settings tomorrow. I have used AUFS as caching 
format, but I have also tested UFS. The format seems to have no influence on 
the issue.

I have tested the 1 GB Ubuntu 15.04 image (ubuntu-15.04-desktop-amd64.iso). 
This is the link 
http://releases.ubuntu.com/15.04/ubuntu-15.04-desktop-amd64.iso.

If you want to stress caching more with large files. You can use one of those:
https://surfer.nmr.mgh.harvard.edu/fswiki/Download

But I think the Centos 7 ISO are large enough, In my test scenario, I have put all 
files on an internal web server with gives them in stable 120 MB/sec. So the 
problem does not come from a slow network connection. I have checked network 
connectivity with Iperf3 (>= 900 MBit/sec) and made a direct wget without 
Squid. The file gets downloaded in high speed. Adding Squid in the communication 
flow which caches the file on the first request and the issue occurrs. After some 
minutes, the download rate drops to 500 KByte/sec and stays on this level together 
with 100 % CPU load. The download rate corresponds with the disk IO. The file gets 
written with 500 KByte/sec.

Thank you very much!


Gesendet: Mittwoch, 22. Juli 2015 um 18:28 Uhr
Von: "Eliezer Croitoru" 
An: squid-users@lists.squid-cache.org
Betreff: Re: [squid-users] Squid3: 100 % CPU load during object caching
Can you share the relevant squid.conf settings? Just to reproduce..

I have a dedicated testing server here which I can test the issue on.
8GB archive which might be an ISO and can be cached on AUFS\UFS and
LARGE ROCK cache types.

I am pretty sure that the maximum cache object size is one thing to
change and waht more?

 From What I understand it should not be different for 2GB cached
archive and to 8 GB cached archive.
I have a local copy of centos 7 ISO which should be a test worthy object.
Anything more you can add to the test subject?

Eliezer

On 22/07/2015 16:24, Jens Offenbach wrote:

I checked the bug you have mentioned and I think I am confronted with the same
issue. I was able to build and test Squid 3.5.6 on Ubuntu 14.04.2 x84_64. I
observed the same behavior. I have tested an 8 GB archive file and I get 100 %
CPU usage and a download rate of nearly 500 KB/sec when the object gets cached.
I have attached strace to the running process, but I killed it after 30 minutes.
The whole takes hours, although we have a 1-GBit ethernet:

Process 4091 attached
Process 4091 detached
% time seconds usecs/call calls errors syscall
-- --- --- - - 
78.83 2.622879 1 1823951 write
12.29 0.408748 2 228029 2 read
6.18 0.205663 0 912431 1 epoll_wait
2.58 0.085921 0 456020 epoll_ctl
0.09 0.002919 0 6168 brk
0.02 0.000623 2 356 openat
0.01 0.000286 0 712 getdents
0.00 0.71 1 91 getrusage
0.00 0.38 0 362 close
0.00 0.03 2 2 sendto
0.00 0.01 0 3 1 recvfrom
0.00 0.00 0 2 open
0.00 0.00 0 3 stat
0.00 0.00 0 1 1 rt_sigreturn
0.00 0.00 0 1 kill
0.00 0.00 0 4 fcntl
0.00 0.00 0 2 2 unlink
0.00 0.00 0 1 getppid
-- --- --- - - 
100.00 3.327152 3428139 7 total

Can I do anything that helps to get ride of this problem?


Gesendet: Dienstag, 21. Juli 2015 um 17:37 Uhr
Von: "Amos Jeffries" 
An: "Jens Offenbach" , "squid-users@lists.squid-cache.org"

Betreff: Re: Aw: Re: [squid-users] Squid3: 100 % CPU load during object caching
On 22/07/2015 12:31 a.m., Jens Offenbach wrote:

Thank you very much for your detailed explainations. We want to use Squid in
order to accelerate our automated software setup processes via Puppet. Actually
Squid will host only a very short amount of large objects (10-20). Its purpose
is not to cache web traffic or little objects.


Ah, Squid does not "host", it caches. The difference may seem trivial at
first glance but it is the critical factor between whether a proxy or a
local web server is the best tool for the job.

 From my own experiences with Puppet, yes Squid is the right tool. But
only because the Puppet server was using relatively slow python code to
generate objects and not doing server-side caching on its own. If that
situation has changed in recent years then Squids usefulness will also
have changed.



The hit-ratio for all the hosted
objects will be very high, because most of our VMs require the same so

Re: [squid-users] Squid3: 100 % CPU load during object caching

[Eliezer Croitoru]

On 22/07/2015 21:59, Eliezer Croitoru wrote:

Hey Jens,

I have tested the issue with LARGE ROCK and not AUFS or UFS.
Using squid or not my connection to the server is about 2.5 MBps (20Mbps).
Squid is sitting on an intel atom with SSD drive and on a HIT case the
download speed is more then doubled to 4.5 MBps(36Mbps).
I have not tried it with AUFS yet.



And I must admit that AUFS beats rock cache with speed.
I have tried rock with basic "cache_dir rock /var/spool/squid 8000" vs 
"cache_dir aufs /var/spool/squid 8000 16 256" and the aufs cache HIT 
results more then doubles 3 the speed rock gave with default settings.


So about 15MBps which is 120Mbps.
I do not seem to feel what Jens feels but the 100% CPU might be because 
of spinning disk hangs while reading the file from disk.


Amos, I remember that there were some suggestions how to tune large rock.
Any hints?
I can test it and make it a suggestion for big files.

Eliezer

___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] Squid3: 100 % CPU load during object caching

[Jens Offenbach]

I have not tested rock yet, but I will give it a try and will report the 
results as soon as possible.

I have tested Squid 3.5.6 in three different virtualized environments (VMware 
vSphere, OpenStack Icehouse, OpenStack Kilo).

For clarification:
The 100 % CPU usage and the low download speed occurs only when there is NO 
cache hit and when the object is added to disk cache FOR THE FIRST TIME. When 
the object is in the cache, the download speed and the CPU usage of Squid 
(3.3.8 and 3.5.6) is optimal. I have tested it with an Ubuntu ISO image (>= 1 
GB).

This is my current squid.config.

# ACCESS CONTROLS
# -
  acl localnetsrc 139.2.0.0/16
  acl localnetsrc 193.96.112.0/21
  acl localnetsrc 192.109.216.0/24
  acl localnetsrc 100.1.4.0/22
  acl localnetsrc 10.0.0.0/8
  acl localnetsrc 172.16.0.0/12
  acl localnetsrc 192.168.0.0/16
  http_access allow manager localhost
  http_access deny  manager
  http_access allow localnet
  http_access allow localhost
  http_access deny all

# NETWORK OPTIONS
# -
  http_port 0.0.0.0:3128

# MEMORY CACHE OPTIONS
# -
  maximum_object_size_in_memory 128 MB
  memory_replacement_policy heap LFUDA
  cache_mem 4 GB

# DISK CACHE OPTIONS
# -
  maximum_object_size 10 GB
  cache_replacement_policy heap GDSF
  cache_dir aufs /var/cache/squid3 88894 16 256 max-size=10737418240

# LOGFILE OPTIONS
# -
  access_log daemon:/var/log/squid3/access.log squid
  cache_store_log daemon:/var/log/squid3/store.log

# OPTIONS FOR TROUBLESHOOTING
# -
  cache_log /var/log/squid3/cache.log
  coredump_dir /var/log/squid3
  
# OPTIONS FOR TUNING THE CACHE
# -
  cache allow all
  
# OPTIONS INFLUENCING REQUEST FORWARDING 
# -
  always_direct allow all

# MISCELLANEOUS
# -
  memory_pools off


Thank you very much for your help! It is good to know that you will take care 
of my problem.

Regards,
Jens
 

Gesendet: Mittwoch, 22. Juli 2015 um 21:47 Uhr
Von: "Eliezer Croitoru" 
An: squid-users@lists.squid-cache.org
Cc: "Amos Jeffries" 
Betreff: Re: [squid-users] Squid3: 100 % CPU load during object caching
On 22/07/2015 21:59, Eliezer Croitoru wrote:
> Hey Jens,
>
> I have tested the issue with LARGE ROCK and not AUFS or UFS.
> Using squid or not my connection to the server is about 2.5 MBps (20Mbps).
> Squid is sitting on an intel atom with SSD drive and on a HIT case the
> download speed is more then doubled to 4.5 MBps(36Mbps).
> I have not tried it with AUFS yet.


And I must admit that AUFS beats rock cache with speed.
I have tried rock with basic "cache_dir rock /var/spool/squid 8000" vs
"cache_dir aufs /var/spool/squid 8000 16 256" and the aufs cache HIT
results more then doubles 3 the speed rock gave with default settings.

So about 15MBps which is 120Mbps.
I do not seem to feel what Jens feels but the 100% CPU might be because
of spinning disk hangs while reading the file from disk.

Amos, I remember that there were some suggestions how to tune large rock.
Any hints?
I can test it and make it a suggestion for big files.

Eliezer
 
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] Squid3: 100 % CPU load during object caching

[Jens Offenbach]

A test with ROCK "cache_dir rock /var/cache/squid3 51200" gives very confusing 
results.
 
I cleared the cache:
rm -rf /var/cache/squid3/*
squid -z
squid
http_proxy=http://139.2.57.120:3128/ wget 
http://test-server/freesurfer-Linux-centos6_x86_64-stable-pub-v5.3.0.tar

The download starts with 10 MB/sec and stays constant for 1 minutes, then it 
drops gradually to 1 MB/sec and stays there for some time. After 5 minutes the 
download rate returns back to 10 MB/sec very quickly and drops again 
step-by-step to 1 MB/sec. After 5-6 minutes the download rates rises again to 
10 MB/sec and drops again gradually to 1 MB/sec.

During caching progress, we have 100 % CPU usage and a disk IO that is 
corresponds with the download rate.

For further investigations I give you my build properties:
squid -v
Squid Cache: Version 3.5.6
Service Name: squid
configure options:  '--build=x86_64-linux-gnu' '--prefix=/usr' 
'--includedir=/include' '--mandir=/share/man' '--infodir=/share/info' 
'--sysconfdir=/etc' '--localstatedir=/var' '--libexecdir=/lib/squid3' 
'--srcdir=.' '--disable-maintainer-mode' '--disable-dependency-tracking' 
'--disable-silent-rules' '--datadir=/usr/share/squid3' 
'--sysconfdir=/etc/squid3' '--mandir=/usr/share/man' '--enable-inline' 
'--enable-async-io=8' '--enable-storeio=ufs,aufs,diskd,rock' 
'--enable-removal-policies=lru,heap' '--enable-delay-pools' 
'--enable-cache-digests' '--enable-underscores' '--enable-icap-client' 
'--enable-follow-x-forwarded-for' 
'--enable-auth-basic=DB,fake,getpwnam,LDAP,NCSA,NIS,PAM,POP3,RADIUS,SASL,SMB' 
'--enable-auth-digest=file,LDAP' '--enable-auth-negotiate=kerberos,wrapper' 
'--enable-auth-ntlm=fake,smb_lm' 
'--enable-external-acl-helpers=file_userip,kerberos_ldap_group,LDAP_group,session,SQL_session,unix_group,wbinfo_group'
 '--enable-url-rewrite-helpers=fake' '--enable-eui' '--enable-esi' 
'--enable-icmp' '--enable-zph-qos' '--enable-ecap' '--disable-translation' 
'--with-swapdir=/var/cache/squid3' '--with-logdir=/var/log/squid3' 
'--with-pidfile=/var/run/squid3.pid' '--with-filedescriptors=65536' 
'--with-large-files' '--with-default-user=proxy' '--enable-linux-netfilter' 
'build_alias=x86_64-linux-gnu' 'CFLAGS=-g -O2 -fPIE -fstack-protector 
--param=ssp-buffer-size=4 -Wformat -Werror=format-security -Wall' 
'LDFLAGS=-Wl,-Bsymbolic-functions -fPIE -pie -Wl,-z,relro -Wl,-z,now' 
'CPPFLAGS=-D_FORTIFY_SOURCE=2' 'CXXFLAGS=-g -O2 -fPIE -fstack-protector 
--param=ssp-buffer-size=4 -Wformat -Werror=format-security'
 

Gesendet: Mittwoch, 22. Juli 2015 um 21:47 Uhr
Von: "Eliezer Croitoru" 
An: squid-users@lists.squid-cache.org
Betreff: Re: [squid-users] Squid3: 100 % CPU load during object caching
On 22/07/2015 21:59, Eliezer Croitoru wrote:
> Hey Jens,
>
> I have tested the issue with LARGE ROCK and not AUFS or UFS.
> Using squid or not my connection to the server is about 2.5 MBps (20Mbps).
> Squid is sitting on an intel atom with SSD drive and on a HIT case the
> download speed is more then doubled to 4.5 MBps(36Mbps).
> I have not tried it with AUFS yet.


And I must admit that AUFS beats rock cache with speed.
I have tried rock with basic "cache_dir rock /var/spool/squid 8000" vs
"cache_dir aufs /var/spool/squid 8000 16 256" and the aufs cache HIT
results more then doubles 3 the speed rock gave with default settings.

So about 15MBps which is 120Mbps.
I do not seem to feel what Jens feels but the 100% CPU might be because
of spinning disk hangs while reading the file from disk.

Amos, I remember that there were some suggestions how to tune large rock.
Any hints?
I can test it and make it a suggestion for big files.

Eliezer

___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] Squid3: 100 % CPU load during object caching

[Marcus Kool]

I am not sure if it is relevant, maybe it is:

I am developing an ICAP daemon and after the ICAP server sends a "100 continue"
Squid sends the object to the ICAP server in small chunks of varying sizes:
4095, 5813, 1448, 4344, 1448, 1448, 2896, etc.
Note that the interval of receiving the chunks is 1/1000th of a second.
It seems that Squid forwards the object to the ICAP server every time it 
receives
one or a few TCP packets.

I have a suspicion that in the scenario of 100% CPU, large #write calls and low 
throughput a similar thing is happening:
Squid physically stores a small part of the object many times, i.e. every time 
one or a few TCP packets arrive.

Amos, is there a debug setting that can confirm/reject this suspicion?

Marcus


On 07/23/2015 04:25 AM, Jens Offenbach wrote:

A test with ROCK "cache_dir rock /var/cache/squid3 51200" gives very confusing 
results.

I cleared the cache:
rm -rf /var/cache/squid3/*
squid -z
squid
http_proxy=http://139.2.57.120:3128/ wget 
http://test-server/freesurfer-Linux-centos6_x86_64-stable-pub-v5.3.0.tar

The download starts with 10 MB/sec and stays constant for 1 minutes, then it 
drops gradually to 1 MB/sec and stays there for some time. After 5 minutes the 
download rate returns back to 10 MB/sec very quickly and drops again 
step-by-step to 1 MB/sec. After 5-6 minutes the download rates rises again to 
10 MB/sec and drops again gradually to 1 MB/sec.

During caching progress, we have 100 % CPU usage and a disk IO that is 
corresponds with the download rate.

For further investigations I give you my build properties:
squid -v
Squid Cache: Version 3.5.6
Service Name: squid
configure options:  '--build=x86_64-linux-gnu' '--prefix=/usr' 
'--includedir=/include' '--mandir=/share/man' '--infodir=/share/info' 
'--sysconfdir=/etc' '--localstatedir=/var' '--libexecdir=/lib/squid3' 
'--srcdir=.' '--disable-maintainer-mode' '--disable-dependency-tracking' 
'--disable-silent-rules' '--datadir=/usr/share/squid3' 
'--sysconfdir=/etc/squid3' '--mandir=/usr/share/man' '--enable-inline' 
'--enable-async-io=8' '--enable-storeio=ufs,aufs,diskd,rock' 
'--enable-removal-policies=lru,heap' '--enable-delay-pools' 
'--enable-cache-digests' '--enable-underscores' '--enable-icap-client' 
'--enable-follow-x-forwarded-for' 
'--enable-auth-basic=DB,fake,getpwnam,LDAP,NCSA,NIS,PAM,POP3,RADIUS,SASL,SMB' 
'--enable-auth-digest=file,LDAP' '--enable-auth-negotiate=kerberos,wrapper' 
'--enable-auth-ntlm=fake,smb_lm' 
'--enable-external-acl-helpers=file_userip,kerberos_ldap_group,LDAP_group,session,SQL_session,unix_group,wbinfo_group'
 '--enable-url-rewrite-helpers=fake' '--enable-eui' '--enable-e

s
i' '--enable-icmp' '--enable-zph-qos' '--enable-ecap' '--disable-translation' 
'--with-swapdir=/var/cache/squid3' '--with-logdir=/var/log/squid3' 
'--with-pidfile=/var/run/squid3.pid' '--with-filedescriptors=65536' 
'--with-large-files' '--with-default-user=proxy' '--enable-linux-netfilter' 
'build_alias=x86_64-linux-gnu' 'CFLAGS=-g -O2 -fPIE -fstack-protector 
--param=ssp-buffer-size=4 -Wformat -Werror=format-security -Wall' 
'LDFLAGS=-Wl,-Bsymbolic-functions -fPIE -pie -Wl,-z,relro -Wl,-z,now' 
'CPPFLAGS=-D_FORTIFY_SOURCE=2' 'CXXFLAGS=-g -O2 -fPIE -fstack-protector 
--param=ssp-buffer-size=4 -Wformat -Werror=format-security'



Gesendet: Mittwoch, 22. Juli 2015 um 21:47 Uhr
Von: "Eliezer Croitoru" 
An: squid-users@lists.squid-cache.org
Betreff: Re: [squid-users] Squid3: 100 % CPU load during object caching
On 22/07/2015 21:59, Eliezer Croitoru wrote:

Hey Jens,

I have tested the issue with LARGE ROCK and not AUFS or UFS.
Using squid or not my connection to the server is about 2.5 MBps (20Mbps).
Squid is sitting on an intel atom with SSD drive and on a HIT case the
download speed is more then doubled to 4.5 MBps(36Mbps).
I have not tried it with AUFS yet.



And I must admit that AUFS beats rock cache with speed.
I have tried rock with basic "cache_dir rock /var/spool/squid 8000" vs
"cache_dir aufs /var/spool/squid 8000 16 256" and the aufs cache HIT
results more then doubles 3 the speed rock gave with default settings.

So about 15MBps which is 120Mbps.
I do not seem to feel what Jens feels but the 100% CPU might be because
of spinning disk hangs while reading the file from disk.

Amos, I remember that there were some suggestions how to tune large rock.
Any hints?
I can test it and make it a suggestion for big files.

Eliezer

___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users


___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] Squid3: 100 % CPU load during object caching

[Jens Offenbach]

I have attached strace to Squid and waited until the download rate has decreased to 500 KB/sec.
 

I used "cache_dir aufs /var/cache/squid3 88894 16 256 max-size=10737418240".


 

Here is the download link:

http://w1.wikisend.com/node-fs/download/6a004a416f65b4cdf7f8eff4ff961199/squid.strace

 

I hope it can help you.

 


Gesendet: Donnerstag, 23. Juli 2015 um 13:29 Uhr
Von: "Marcus Kool" 
An: "Jens Offenbach" , "Eliezer Croitoru" , "Amos Jeffries" , squid-users@lists.squid-cache.org
Betreff: Re: [squid-users] Squid3: 100 % CPU load during object caching

I am not sure if it is relevant, maybe it is:

I am developing an ICAP daemon and after the ICAP server sends a "100 continue"
Squid sends the object to the ICAP server in small chunks of varying sizes:
4095, 5813, 1448, 4344, 1448, 1448, 2896, etc.
Note that the interval of receiving the chunks is 1/1000th of a second.
It seems that Squid forwards the object to the ICAP server every time it receives
one or a few TCP packets.

I have a suspicion that in the scenario of 100% CPU, large #write calls and low throughput a similar thing is happening:
Squid physically stores a small part of the object many times, i.e. every time one or a few TCP packets arrive.

Amos, is there a debug setting that can confirm/reject this suspicion?

Marcus


On 07/23/2015 04:25 AM, Jens Offenbach wrote:
> A test with ROCK "cache_dir rock /var/cache/squid3 51200" gives very confusing results.
>
> I cleared the cache:
> rm -rf /var/cache/squid3/*
> squid -z
> squid
> http_proxy=http://139.2.57.120:3128/ wget http://test-server/freesurfer-Linux-centos6_x86_64-stable-pub-v5.3.0.tar
>
> The download starts with 10 MB/sec and stays constant for 1 minutes, then it drops gradually to 1 MB/sec and stays there for some time. After 5 minutes the download rate returns back to 10 MB/sec very quickly and drops again step-by-step to 1 MB/sec. After 5-6 minutes the download rates rises again to 10 MB/sec and drops again gradually to 1 MB/sec.
>
> During caching progress, we have 100 % CPU usage and a disk IO that is corresponds with the download rate.
>
> For further investigations I give you my build properties:
> squid -v
> Squid Cache: Version 3.5.6
> Service Name: squid
> configure options: '--build=x86_64-linux-gnu' '--prefix=/usr' '--includedir=/include' '--mandir=/share/man' '--infodir=/share/info' '--sysconfdir=/etc' '--localstatedir=/var' '--libexecdir=/lib/squid3' '--srcdir=.' '--disable-maintainer-mode' '--disable-dependency-tracking' '--disable-silent-rules' '--datadir=/usr/share/squid3' '--sysconfdir=/etc/squid3' '--mandir=/usr/share/man' '--enable-inline' '--enable-async-io=8' '--enable-storeio=ufs,aufs,diskd,rock' '--enable-removal-policies=lru,heap' '--enable-delay-pools' '--enable-cache-digests' '--enable-underscores' '--enable-icap-client' '--enable-follow-x-forwarded-for' '--enable-auth-basic=DB,fake,getpwnam,LDAP,NCSA,NIS,PAM,POP3,RADIUS,SASL,SMB' '--enable-auth-digest=file,LDAP' '--enable-auth-negotiate=kerberos,wrapper' '--enable-auth-ntlm=fake,smb_lm' '--enable-external-acl-helpers=file_userip,kerberos_ldap_group,LDAP_group,session,SQL_session,unix_group,wbinfo_group' '--enable-url-rewrite-helpers=fake' '--enable-eui' '--enable-e
s
i' '--enable-icmp' '--enable-zph-qos' '--enable-ecap' '--disable-translation' '--with-swapdir=/var/cache/squid3' '--with-logdir=/var/log/squid3' '--with-pidfile=/var/run/squid3.pid' '--with-filedescriptors=65536' '--with-large-files' '--with-default-user=proxy' '--enable-linux-netfilter' 'build_alias=x86_64-linux-gnu' 'CFLAGS=-g -O2 -fPIE -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security -Wall' 'LDFLAGS=-Wl,-Bsymbolic-functions -fPIE -pie -Wl,-z,relro -Wl,-z,now' 'CPPFLAGS=-D_FORTIFY_SOURCE=2' 'CXXFLAGS=-g -O2 -fPIE -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security'
>
>
> Gesendet: Mittwoch, 22. Juli 2015 um 21:47 Uhr
> Von: "Eliezer Croitoru" 
> An: squid-users@lists.squid-cache.org
> Betreff: Re: [squid-users] Squid3: 100 % CPU load during object caching
> On 22/07/2015 21:59, Eliezer Croitoru wrote:
>> Hey Jens,
>>
>> I have tested the issue with LARGE ROCK and not AUFS or UFS.
>> Using squid or not my connection to the server is about 2.5 MBps (20Mbps).
>> Squid is sitting on an intel atom with SSD drive and on a HIT case the
>

Re: [squid-users] Squid3: 100 % CPU load during object caching

[Amos Jeffries]

On 23/07/2015 11:29 p.m., Marcus Kool wrote:
> I am not sure if it is relevant, maybe it is:
> 
> I am developing an ICAP daemon and after the ICAP server sends a "100
> continue"
> Squid sends the object to the ICAP server in small chunks of varying sizes:
> 4095, 5813, 1448, 4344, 1448, 1448, 2896, etc.
> Note that the interval of receiving the chunks is 1/1000th of a second.
> It seems that Squid forwards the object to the ICAP server every time it
> receives
> one or a few TCP packets.
> 
> I have a suspicion that in the scenario of 100% CPU, large #write calls
> and low throughput a similar thing is happening:
> Squid physically stores a small part of the object many times, i.e.
> every time one or a few TCP packets arrive.

If testing with low traffic (one connection) that guess is correct.
Squid is so fast that it cycles through the whole
allocate-memory->read()->process->write()-to-icap cycle in under a
millisecond.

It takes a few dozen busy clients in parallel to slow Slow down enough
to increase the chunk sizes.


> 
> Amos, is there a debug setting that can confirm/reject this suspicion?

Not that I recall at the moment.

Amos

___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] Squid3: 100 % CPU load during object caching

[Jens Offenbach]

Ok. I am sorry,

Please, try this one:

http://wikisend.com/download/413650/squid.strace

Regards,
Jens
 

Gesendet: Donnerstag, 23. Juli 2015 um 16:25 Uhr
Von: "Marcus Kool" 
An: "Jens Offenbach" 
Betreff: Re: Aw: Re: [squid-users] Squid3: 100 % CPU load during object caching
I get a 403 Access Denied error trying to download the trace file

Marcus


On 07/23/2015 10:39 AM, Jens Offenbach wrote:
> I have attached strace to Squid and waited until the download rate has 
> decreased to 500 KB/sec.
> I used "cache_dir aufs /var/cache/squid3 88894 16 256 max-size=10737418240".
> Here is the download link:
> http://w1.wikisend.com/node-fs/download/6a004a416f65b4cdf7f8eff4ff961199/squid.strace
> I hope it can help you.
> *Gesendet:* Donnerstag, 23. Juli 2015 um 13:29 Uhr
> *Von:* "Marcus Kool" 
> *An:* "Jens Offenbach" , "Eliezer Croitoru" 
> , "Amos Jeffries" , 
> squid-users@lists.squid-cache.org
> *Betreff:* Re: [squid-users] Squid3: 100 % CPU load during object caching
> I am not sure if it is relevant, maybe it is:
>
> I am developing an ICAP daemon and after the ICAP server sends a "100 
> continue"
> Squid sends the object to the ICAP server in small chunks of varying sizes:
> 4095, 5813, 1448, 4344, 1448, 1448, 2896, etc.
> Note that the interval of receiving the chunks is 1/1000th of a second.
> It seems that Squid forwards the object to the ICAP server every time it 
> receives
> one or a few TCP packets.
>
> I have a suspicion that in the scenario of 100% CPU, large #write calls and 
> low throughput a similar thing is happening:
> Squid physically stores a small part of the object many times, i.e. every 
> time one or a few TCP packets arrive.
>
> Amos, is there a debug setting that can confirm/reject this suspicion?
>
> Marcus
>
>
> On 07/23/2015 04:25 AM, Jens Offenbach wrote:
> > A test with ROCK "cache_dir rock /var/cache/squid3 51200" gives very 
> > confusing results.
> >
> > I cleared the cache:
> > rm -rf /var/cache/squid3/*
> > squid -z
> > squid
> > http_proxy=http://139.2.57.120:3128/[http://139.2.57.120:3128/] wget 
> > http://test-server/freesurfer-Linux-centos6_x86_64-stable-pub-v5.3.0.tar
> >
> > The download starts with 10 MB/sec and stays constant for 1 minutes, then 
> > it drops gradually to 1 MB/sec and stays there for some time. After 5 
> > minutes the download rate returns back to 10 MB/sec
> very quickly and drops again step-by-step to 1 MB/sec. After 5-6 minutes the 
> download rates rises again to 10 MB/sec and drops again gradually to 1 MB/sec.
> >
> > During caching progress, we have 100 % CPU usage and a disk IO that is 
> > corresponds with the download rate.
> >
> > For further investigations I give you my build properties:
> > squid -v
> > Squid Cache: Version 3.5.6
> > Service Name: squid
> > configure options: '--build=x86_64-linux-gnu' '--prefix=/usr' 
> > '--includedir=/include' '--mandir=/share/man' '--infodir=/share/info' 
> > '--sysconfdir=/etc' '--localstatedir=/var'
> '--libexecdir=/lib/squid3' '--srcdir=.' '--disable-maintainer-mode' 
> '--disable-dependency-tracking' '--disable-silent-rules' 
> '--datadir=/usr/share/squid3' '--sysconfdir=/etc/squid3'
> '--mandir=/usr/share/man' '--enable-inline' '--enable-async-io=8' 
> '--enable-storeio=ufs,aufs,diskd,rock' '--enable-removal-policies=lru,heap' 
> '--enable-delay-pools' '--enable-cache-digests'
> '--enable-underscores' '--enable-icap-client' 
> '--enable-follow-x-forwarded-for' 
> '--enable-auth-basic=DB,fake,getpwnam,LDAP,NCSA,NIS,PAM,POP3,RADIUS,SASL,SMB' 
> '--enable-auth-digest=file,LDAP'
> '--enable-auth-negotiate=kerberos,wrapper' '--enable-auth-ntlm=fake,smb_lm' 
> '--enable-external-acl-helpers=file_userip,kerberos_ldap_group,LDAP_group,session,SQL_session,unix_group,wbinfo_group'
> '--enable-url-rewrite-helpers=fake' '--enable-eui' '--enable-e
> s
> i' '--enable-icmp' '--enable-zph-qos' '--enable-ecap' '--disable-translation' 
> '--with-swapdir=/var/cache/squid3' '--with-logdir=/var/log/squid3' 
> '--with-pidfile=/var/run/squid3.pid'
> '--with-filedescriptors=65536' '--with-large-files' 
> '--with-default-user=proxy' '--enable-linux-netfilter' 
> 'build_alias=x86_64-linux-gnu&

Re: [squid-users] Squid3: 100 % CPU load during object caching

[Marcus Kool]

The strace output shows this loop:

   Squid reads 16K-1 bytes from FD 13webserver
   Squid writes 4 times 4K to FD 17  /var/cache/squid3/00/00/
   Squid writes 4 times 4K to FD 12  browser

But this loop does not explain the 100% CPU usage...

Does Squid do a buffer reshuffle when it reads 16K-1 and writes 16K ?

I did the download test with Squid 3.4.12 AUFS on an idle system with a 500 
mbit connection and 1 CPU with 4 cores @ 3.7 GHz.
The first download used 35% of 1 CPU core with a steady download speed of 62 
MB/sec.
The second (cached) download used 50% of 1 CPU core with a steady download 
speed of 87 MB/sec.
I never looked at Squid CPU usage and do not know what is reasonable but it 
feels high.

With respect to the 100% CPU issue of Jens, one factor is that Squid runs in a 
virtual machine.
Squid in a virtual machine cannot be compared with a wget test since Squid 
allocates a lot of memory that the host must manage.
This is a possible explanation for the fact that you see the performance going 
down and up.
Can you do the same test on the host (i.e. not inside a VM).

Marcus



On 07/23/2015 10:39 AM, Jens Offenbach wrote:

I have attached strace to Squid and waited until the download rate has 
decreased to 500 KB/sec.
I used "cache_dir aufs /var/cache/squid3 88894 16 256 max-size=10737418240".
Here is the download link:
http://w1.wikisend.com/node-fs/download/6a004a416f65b4cdf7f8eff4ff961199/squid.strace
I hope it can help you.
*Gesendet:* Donnerstag, 23. Juli 2015 um 13:29 Uhr
*Von:* "Marcus Kool" 
*An:* "Jens Offenbach" , "Eliezer Croitoru" , 
"Amos Jeffries" , squid-users@lists.squid-cache.org
*Betreff:* Re: [squid-users] Squid3: 100 % CPU load during object caching
I am not sure if it is relevant, maybe it is:

I am developing an ICAP daemon and after the ICAP server sends a "100 continue"
Squid sends the object to the ICAP server in small chunks of varying sizes:
4095, 5813, 1448, 4344, 1448, 1448, 2896, etc.
Note that the interval of receiving the chunks is 1/1000th of a second.
It seems that Squid forwards the object to the ICAP server every time it 
receives
one or a few TCP packets.

I have a suspicion that in the scenario of 100% CPU, large #write calls and low 
throughput a similar thing is happening:
Squid physically stores a small part of the object many times, i.e. every time 
one or a few TCP packets arrive.

Amos, is there a debug setting that can confirm/reject this suspicion?

Marcus


On 07/23/2015 04:25 AM, Jens Offenbach wrote:
 > A test with ROCK "cache_dir rock /var/cache/squid3 51200" gives very 
confusing results.
 >
 > I cleared the cache:
 > rm -rf /var/cache/squid3/*
 > squid -z
 > squid
 > http_proxy=http://139.2.57.120:3128/ wget 
http://test-server/freesurfer-Linux-centos6_x86_64-stable-pub-v5.3.0.tar
 >
 > The download starts with 10 MB/sec and stays constant for 1 minutes, then it 
drops gradually to 1 MB/sec and stays there for some time. After 5 minutes the 
download rate returns back to 10 MB/sec
very quickly and drops again step-by-step to 1 MB/sec. After 5-6 minutes the 
download rates rises again to 10 MB/sec and drops again gradually to 1 MB/sec.
 >
 > During caching progress, we have 100 % CPU usage and a disk IO that is 
corresponds with the download rate.
 >
 > For further investigations I give you my build properties:
 > squid -v
 > Squid Cache: Version 3.5.6
 > Service Name: squid
 > configure options: '--build=x86_64-linux-gnu' '--prefix=/usr' 
'--includedir=/include' '--mandir=/share/man' '--infodir=/share/info' 
'--sysconfdir=/etc' '--localstatedir=/var'
'--libexecdir=/lib/squid3' '--srcdir=.' '--disable-maintainer-mode' 
'--disable-dependency-tracking' '--disable-silent-rules' 
'--datadir=/usr/share/squid3' '--sysconfdir=/etc/squid3'
'--mandir=/usr/share/man' '--enable-inline' '--enable-async-io=8' 
'--enable-storeio=ufs,aufs,diskd,rock' '--enable-removal-policies=lru,heap' 
'--enable-delay-pools' '--enable-cache-digests'
'--enable-underscores' '--enable-icap-client' '--enable-follow-x-forwarded-for' 
'--enable-auth-basic=DB,fake,getpwnam,LDAP,NCSA,NIS,PAM,POP3,RADIUS,SASL,SMB' 
'--enable-auth-digest=file,LDAP'
'--enable-auth-negotiate=kerberos,wrapper' '--enable-auth-ntlm=fake,smb_lm' 
'--enable-external-acl-helpers=file_userip,kerberos_ldap_group,LDAP_group,session,SQL_session,unix_group,wbinfo_group'
'--enable-url-rewrite-helpers=fake' '--enable-eui' '--enable-e
s
i' '--enable-icmp' '--enable-zph-qos' '--enable-ecap' '--disable-translation' 
&#

Re: [squid-users] Squid3: 100 % CPU load during object caching

[Jens Offenbach]

It is not easy for me, but I have tested Squid 3.3.8 from the Ubuntu packaging 
on a "real" physical infrastructure. I get the same results on the physical 
machine (1x Intel(R) Xeon(R) CPU E3-1225 V2 @ 3.20GHz, 32 GB RAM, 1 TB disk) 
where Squid is running: 100 % CPU usage, 500 KB/sec download rate. All machines 
are idle and we have 1 GBit ethernet.

The strace log from the physical test scenario can be found here, but I think 
it does not differ from the "virtual" test scenario:
http://wikisend.com/download/293856/squid.strace2

@Marcus:
Have you verified that the file does not fit into memory and gets cached on 
disk? On which OS is Squid running? What are your build options of Squid (squid 
-v)? Is it possible that the issue is not part of 3.4.12? Do we have a 
regression?

@Amos, Eliezer
Is someone able to reproduce the disk caching effect?

Regards,
Jens


Gesendet: Donnerstag, 23. Juli 2015 um 20:08 Uhr
Von: "Marcus Kool" 
An: "Jens Offenbach" , "Amos Jeffries" 
, "Eliezer Croitoru" , 
squid-users@lists.squid-cache.org
Betreff: Re: Aw: Re: [squid-users] Squid3: 100 % CPU load during object caching
The strace output shows this loop:

Squid reads 16K-1 bytes from FD 13 webserver
Squid writes 4 times 4K to FD 17 /var/cache/squid3/00/00/
Squid writes 4 times 4K to FD 12 browser

But this loop does not explain the 100% CPU usage...

Does Squid do a buffer reshuffle when it reads 16K-1 and writes 16K ?

I did the download test with Squid 3.4.12 AUFS on an idle system with a 500 
mbit connection and 1 CPU with 4 cores @ 3.7 GHz.
The first download used 35% of 1 CPU core with a steady download speed of 62 
MB/sec.
The second (cached) download used 50% of 1 CPU core with a steady download 
speed of 87 MB/sec.
I never looked at Squid CPU usage and do not know what is reasonable but it 
feels high.

With respect to the 100% CPU issue of Jens, one factor is that Squid runs in a 
virtual machine.
Squid in a virtual machine cannot be compared with a wget test since Squid 
allocates a lot of memory that the host must manage.
This is a possible explanation for the fact that you see the performance going 
down and up.
Can you do the same test on the host (i.e. not inside a VM).

Marcus



On 07/23/2015 10:39 AM, Jens Offenbach wrote:
> I have attached strace to Squid and waited until the download rate has 
> decreased to 500 KB/sec.
> I used "cache_dir aufs /var/cache/squid3 88894 16 256 max-size=10737418240".
> Here is the download link:
> http://w1.wikisend.com/node-fs/download/6a004a416f65b4cdf7f8eff4ff961199/squid.strace
> I hope it can help you.
> *Gesendet:* Donnerstag, 23. Juli 2015 um 13:29 Uhr
> *Von:* "Marcus Kool" 
> *An:* "Jens Offenbach" , "Eliezer Croitoru" 
> , "Amos Jeffries" , 
> squid-users@lists.squid-cache.org
> *Betreff:* Re: [squid-users] Squid3: 100 % CPU load during object caching
> I am not sure if it is relevant, maybe it is:
>
> I am developing an ICAP daemon and after the ICAP server sends a "100 
> continue"
> Squid sends the object to the ICAP server in small chunks of varying sizes:
> 4095, 5813, 1448, 4344, 1448, 1448, 2896, etc.
> Note that the interval of receiving the chunks is 1/1000th of a second.
> It seems that Squid forwards the object to the ICAP server every time it 
> receives
> one or a few TCP packets.
>
> I have a suspicion that in the scenario of 100% CPU, large #write calls and 
> low throughput a similar thing is happening:
> Squid physically stores a small part of the object many times, i.e. every 
> time one or a few TCP packets arrive.
>
> Amos, is there a debug setting that can confirm/reject this suspicion?
>
> Marcus
>
>
> On 07/23/2015 04:25 AM, Jens Offenbach wrote:
> > A test with ROCK "cache_dir rock /var/cache/squid3 51200" gives very 
> > confusing results.
> >
> > I cleared the cache:
> > rm -rf /var/cache/squid3/*
> > squid -z
> > squid
> > http_proxy=http://139.2.57.120:3128/[http://139.2.57.120:3128/] wget 
> > http://test-server/freesurfer-Linux-centos6_x86_64-stable-pub-v5.3.0.tar
> >
> > The download starts with 10 MB/sec and stays constant for 1 minutes, then 
> > it drops gradually to 1 MB/sec and stays there for some time. After 5 
> > minutes the download rate returns back to 10 MB/sec
> very quickly and drops again step-by-step to 1 MB/sec. After 5-6 minutes the 
> download rates rises again to 10 MB/sec and drops again gradually to 1 MB/sec.
> >
> > During caching progress, we have 100 % CPU usage and a disk IO that is 
> > corresponds with the download rate.
> >
> > For further investigations I give you my build properties:
> > squid -v
> > Squid Cache: Version 3.5.6
> >

Re: [squid-users] Squid3: 100 % CPU load during object caching

[Jens Offenbach]

I have made a quick test of Squid 3.3.8 on Ubuntu 15.04 and I get the same 
problem: 100 % CPU usage, 500 KB/sec download rate.
 

Gesendet: Freitag, 24. Juli 2015 um 07:54 Uhr
Von: "Jens Offenbach" 
An: "Marcus Kool" , "Eliezer Croitoru" 
, "Amos Jeffries" , 
squid-users@lists.squid-cache.org
Betreff: Re: [squid-users] Squid3: 100 % CPU load during object caching
It is not easy for me, but I have tested Squid 3.3.8 from the Ubuntu packaging 
on a "real" physical infrastructure. I get the same results on the physical 
machine (1x Intel(R) Xeon(R) CPU E3-1225 V2 @ 3.20GHz, 32 GB RAM, 1 TB disk) 
where Squid is running: 100 % CPU usage, 500 KB/sec download rate. All machines 
are idle and we have 1 GBit ethernet.

The strace log from the physical test scenario can be found here, but I think 
it does not differ from the "virtual" test scenario:
http://wikisend.com/download/293856/squid.strace2

@Marcus:
Have you verified that the file does not fit into memory and gets cached on 
disk? On which OS is Squid running? What are your build options of Squid (squid 
-v)? Is it possible that the issue is not part of 3.4.12? Do we have a 
regression?

@Amos, Eliezer
Is someone able to reproduce the disk caching effect?

Regards,
Jens


Gesendet: Donnerstag, 23. Juli 2015 um 20:08 Uhr
Von: "Marcus Kool" 
An: "Jens Offenbach" , "Amos Jeffries" 
, "Eliezer Croitoru" , 
squid-users@lists.squid-cache.org
Betreff: Re: Aw: Re: [squid-users] Squid3: 100 % CPU load during object caching
The strace output shows this loop:

Squid reads 16K-1 bytes from FD 13 webserver
Squid writes 4 times 4K to FD 17 /var/cache/squid3/00/00/
Squid writes 4 times 4K to FD 12 browser

But this loop does not explain the 100% CPU usage...

Does Squid do a buffer reshuffle when it reads 16K-1 and writes 16K ?

I did the download test with Squid 3.4.12 AUFS on an idle system with a 500 
mbit connection and 1 CPU with 4 cores @ 3.7 GHz.
The first download used 35% of 1 CPU core with a steady download speed of 62 
MB/sec.
The second (cached) download used 50% of 1 CPU core with a steady download 
speed of 87 MB/sec.
I never looked at Squid CPU usage and do not know what is reasonable but it 
feels high.

With respect to the 100% CPU issue of Jens, one factor is that Squid runs in a 
virtual machine.
Squid in a virtual machine cannot be compared with a wget test since Squid 
allocates a lot of memory that the host must manage.
This is a possible explanation for the fact that you see the performance going 
down and up.
Can you do the same test on the host (i.e. not inside a VM).

Marcus



On 07/23/2015 10:39 AM, Jens Offenbach wrote:
> I have attached strace to Squid and waited until the download rate has 
> decreased to 500 KB/sec.
> I used "cache_dir aufs /var/cache/squid3 88894 16 256 max-size=10737418240".
> Here is the download link:
> http://w1.wikisend.com/node-fs/download/6a004a416f65b4cdf7f8eff4ff961199/squid.strace[http://w1.wikisend.com/node-fs/download/6a004a416f65b4cdf7f8eff4ff961199/squid.strace]
> I hope it can help you.
> *Gesendet:* Donnerstag, 23. Juli 2015 um 13:29 Uhr
> *Von:* "Marcus Kool" 
> *An:* "Jens Offenbach" , "Eliezer Croitoru" 
> , "Amos Jeffries" , 
> squid-users@lists.squid-cache.org
> *Betreff:* Re: [squid-users] Squid3: 100 % CPU load during object caching
> I am not sure if it is relevant, maybe it is:
>
> I am developing an ICAP daemon and after the ICAP server sends a "100 
> continue"
> Squid sends the object to the ICAP server in small chunks of varying sizes:
> 4095, 5813, 1448, 4344, 1448, 1448, 2896, etc.
> Note that the interval of receiving the chunks is 1/1000th of a second.
> It seems that Squid forwards the object to the ICAP server every time it 
> receives
> one or a few TCP packets.
>
> I have a suspicion that in the scenario of 100% CPU, large #write calls and 
> low throughput a similar thing is happening:
> Squid physically stores a small part of the object many times, i.e. every 
> time one or a few TCP packets arrive.
>
> Amos, is there a debug setting that can confirm/reject this suspicion?
>
> Marcus
>
>
> On 07/23/2015 04:25 AM, Jens Offenbach wrote:
> > A test with ROCK "cache_dir rock /var/cache/squid3 51200" gives very 
> > confusing results.
> >
> > I cleared the cache:
> > rm -rf /var/cache/squid3/*
> > squid -z
> > squid
> > http_proxy=http://139.2.57.120:3128/[http://139.2.57.120:3128/][http://139.2.57.120:3128/[http://139.2.57.120:3128/]]
> >  wget 
> > http://test-server/freesurfer-Linux-centos6_x86_64-stable-pub-v5.3.0.tar
> >
> > The download starts with 10 MB/sec and stays constant for 1 minutes, then 
> > it drops gradual

Re: [squid-users] Squid3: 100 % CPU load during object caching

[Jens Offenbach]

I have found something out... Hopefully, it helps to reproduce and solve the 
issue. 

I got it working with a good download rate, but very high CPU usage on Squid 
3.3.8 and Squid 3.5.6. There seems to be problem with large files that get 
cached on disk in combination with memory caching. When I use these settings, 
memory usage of Squid grows step-by-step with 100% CPU usage and 500 KB/sec 
download rate:

# MEMORY CACHE OPTIONS
# -
  maximum_object_size_in_memory 1 GB
  memory_replacement_policy heap LFUDA
  cache_mem 4 GB

# DISK CACHE OPTIONS
# -
  maximum_object_size 10 GB
  cache_replacement_policy heap GDSF
  cache_dir aufs /var/cache/squid3 25600 16 256

I decided to turn off memory caching completely and used the following settings:

# MEMORY CACHE OPTIONS
# -
  maximum_object_size_in_memory 0 GB
  memory_replacement_policy heap LFUDA
  cache_mem 0 GB

# DISK CACHE OPTIONS
# -
  maximum_object_size 10 GB
  cache_replacement_policy heap GDSF
  cache_dir aufs /var/cache/squid3 25600 16 256

Now, I get stable and high download rates even on a cache miss.

@Marcus:
Could you please post your squid.config
 

Gesendet: Freitag, 24. Juli 2015 um 08:25 Uhr
Von: "Jens Offenbach" 
An: squid-users@lists.squid-cache.org
Betreff: Re: [squid-users] Squid3: 100 % CPU load during object caching
I have made a quick test of Squid 3.3.8 on Ubuntu 15.04 and I get the same 
problem: 100 % CPU usage, 500 KB/sec download rate.
 

Gesendet: Freitag, 24. Juli 2015 um 07:54 Uhr
Von: "Jens Offenbach" 
An: "Marcus Kool" , "Eliezer Croitoru" 
, "Amos Jeffries" , 
squid-users@lists.squid-cache.org
Betreff: Re: [squid-users] Squid3: 100 % CPU load during object caching
It is not easy for me, but I have tested Squid 3.3.8 from the Ubuntu packaging 
on a "real" physical infrastructure. I get the same results on the physical 
machine (1x Intel(R) Xeon(R) CPU E3-1225 V2 @ 3.20GHz, 32 GB RAM, 1 TB disk) 
where Squid is running: 100 % CPU usage, 500 KB/sec download rate. All machines 
are idle and we have 1 GBit ethernet.

The strace log from the physical test scenario can be found here, but I think 
it does not differ from the "virtual" test scenario:
http://wikisend.com/download/293856/squid.strace2

@Marcus:
Have you verified that the file does not fit into memory and gets cached on 
disk? On which OS is Squid running? What are your build options of Squid (squid 
-v)? Is it possible that the issue is not part of 3.4.12? Do we have a 
regression?

@Amos, Eliezer
Is someone able to reproduce the disk caching effect?

Regards,
Jens


Gesendet: Donnerstag, 23. Juli 2015 um 20:08 Uhr
Von: "Marcus Kool" 
An: "Jens Offenbach" , "Amos Jeffries" 
, "Eliezer Croitoru" , 
squid-users@lists.squid-cache.org
Betreff: Re: Aw: Re: [squid-users] Squid3: 100 % CPU load during object caching
The strace output shows this loop:

Squid reads 16K-1 bytes from FD 13 webserver
Squid writes 4 times 4K to FD 17 /var/cache/squid3/00/00/
Squid writes 4 times 4K to FD 12 browser

But this loop does not explain the 100% CPU usage...

Does Squid do a buffer reshuffle when it reads 16K-1 and writes 16K ?

I did the download test with Squid 3.4.12 AUFS on an idle system with a 500 
mbit connection and 1 CPU with 4 cores @ 3.7 GHz.
The first download used 35% of 1 CPU core with a steady download speed of 62 
MB/sec.
The second (cached) download used 50% of 1 CPU core with a steady download 
speed of 87 MB/sec.
I never looked at Squid CPU usage and do not know what is reasonable but it 
feels high.

With respect to the 100% CPU issue of Jens, one factor is that Squid runs in a 
virtual machine.
Squid in a virtual machine cannot be compared with a wget test since Squid 
allocates a lot of memory that the host must manage.
This is a possible explanation for the fact that you see the performance going 
down and up.
Can you do the same test on the host (i.e. not inside a VM).

Marcus



On 07/23/2015 10:39 AM, Jens Offenbach wrote:
> I have attached strace to Squid and waited until the download rate has 
> decreased to 500 KB/sec.
> I used "cache_dir aufs /var/cache/squid3 88894 16 256 max-size=10737418240".
> Here is the download link:
> http://w1.wikisend.com/node-fs/download/6a004a416f65b4cdf7f8eff4ff961199/squid.strace[http://w1.wikisend.com/node-fs/download/6a004a416f65b4cdf7f8eff4ff961199/squid.strace][http://w1.wikisend.com/node-fs/download/6a004a416f65b4cdf7f8eff4ff961199/squid.strace[http://w1.wikisend.com/node-fs/download/6a004a416f65b4cdf7f8eff4ff961199/squid.strace]]
> I hope it can help you.
> *Gesendet:*

Re: [squid-users] Squid3: 100 % CPU load during object caching

[Amos Jeffries]

On 24/07/2015 6:08 a.m., Marcus Kool wrote:
> The strace output shows this loop:
> 
>Squid reads 16K-1 bytes from FD 13webserver
>Squid writes 4 times 4K to FD 17  /var/cache/squid3/00/00/
>Squid writes 4 times 4K to FD 12  browser
> 
> But this loop does not explain the 100% CPU usage...
> 
> Does Squid do a buffer reshuffle when it reads 16K-1 and writes 16K ?

Yes, several (UFS / AUFS 3, or diskd 5).

TCP buffer -> FD 13 read buffer

FD 13 read buffer -> 4x mem_node (4KB each)
 ** walk the length of the in-memory part of the object to find where to
attach the mem_node. (once per each node?)
  - this has been a big CPU hog in the past (Squid-2 did it twice per
node insertion)

4x mem_node -> SHM memory buffer
  - diskd only, AUFS uses mem_node directly

SHM memory buffer -> FD 17 disk write latency
  - happens with both diskd (single treaded) and AUFS (x64 threads)
  - wait latency until completion event is seen by Squid ...

4x mem_node write() copy to FD 12 TCP buffer (OS dependent)


If you are doing any kind of ICAP processing you can add +3 copies per
service processing the transaction body.


> 
> I did the download test with Squid 3.4.12 AUFS on an idle system with a
> 500 mbit connection and 1 CPU with 4 cores @ 3.7 GHz.
> The first download used 35% of 1 CPU core with a steady download speed
> of 62 MB/sec.
> The second (cached) download used 50% of 1 CPU core with a steady
> download speed of 87 MB/sec.
> I never looked at Squid CPU usage and do not know what is reasonable but
> it feels high.
> 
> With respect to the 100% CPU issue of Jens, one factor is that Squid
> runs in a virtual machine.
> Squid in a virtual machine cannot be compared with a wget test since
> Squid allocates a lot of memory that the host must manage.
> This is a possible explanation for the fact that you see the performance
> going down and up.
> Can you do the same test on the host (i.e. not inside a VM).
> 
> Marcus
> 
> 
> 
> On 07/23/2015 10:39 AM, Jens Offenbach wrote:
>> I have attached strace to Squid and waited until the download rate has
>> decreased to 500 KB/sec.
>> I used "cache_dir aufs /var/cache/squid3 88894 16 256
>> max-size=10737418240".
>> Here is the download link:
>> http://w1.wikisend.com/node-fs/download/6a004a416f65b4cdf7f8eff4ff961199/squid.strace
>>
>> I hope it can help you.
>> *Gesendet:* Donnerstag, 23. Juli 2015 um 13:29 Uhr
>> *Von:* "Marcus Kool" 
>> *An:* "Jens Offenbach" , "Eliezer Croitoru"
>> , "Amos Jeffries" ,
>> squid-users@lists.squid-cache.org
>> *Betreff:* Re: [squid-users] Squid3: 100 % CPU load during object caching
>> I am not sure if it is relevant, maybe it is:
>>
>> I am developing an ICAP daemon and after the ICAP server sends a "100
>> continue"
>> Squid sends the object to the ICAP server in small chunks of varying
>> sizes:
>> 4095, 5813, 1448, 4344, 1448, 1448, 2896, etc.
>> Note that the interval of receiving the chunks is 1/1000th of a second.
>> It seems that Squid forwards the object to the ICAP server every time
>> it receives
>> one or a few TCP packets.
>>
>> I have a suspicion that in the scenario of 100% CPU, large #write
>> calls and low throughput a similar thing is happening:
>> Squid physically stores a small part of the object many times, i.e.
>> every time one or a few TCP packets arrive.
>>
>> Amos, is there a debug setting that can confirm/reject this suspicion?

After a bit more thought and Marcus feedback ; store.cc, mem_node
operations, and fd.cc and comm.cc are probably all worth watching.
"debug_options ALL,9" will get you everything Squid has to offer of course.

But be aware that the debugging itself adds a horribly large amount of
overheads for each line logged. At the highest levels it may noticably
impact the high-speed core routines you are trying to measure by skewing
latency into those with more debugs() statements.

Amos

___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] Squid3: 100 % CPU load during object caching

[Amos Jeffries]

On 24/07/2015 7:49 p.m., Jens Offenbach wrote:
> I have found something out... Hopefully, it helps to reproduce and solve the 
> issue. 
> 
> I got it working with a good download rate, but very high CPU usage on Squid 
> 3.3.8 and Squid 3.5.6. There seems to be problem with large files that get 
> cached on disk in combination with memory caching. When I use these settings, 
> memory usage of Squid grows step-by-step with 100% CPU usage and 500 KB/sec 
> download rate:
> 
> # MEMORY CACHE OPTIONS
> # 
> -
>   maximum_object_size_in_memory 1 GB
>   memory_replacement_policy heap LFUDA
>   cache_mem 4 GB
> 
> # DISK CACHE OPTIONS
> # 
> -
>   maximum_object_size 10 GB
>   cache_replacement_policy heap GDSF
>   cache_dir aufs /var/cache/squid3 25600 16 256
> 
> I decided to turn off memory caching completely and used the following 
> settings:
> 
> # MEMORY CACHE OPTIONS
> # 
> -
>   maximum_object_size_in_memory 0 GB
>   memory_replacement_policy heap LFUDA
>   cache_mem 0 GB
> 
> # DISK CACHE OPTIONS
> # 
> -
>   maximum_object_size 10 GB
>   cache_replacement_policy heap GDSF
>   cache_dir aufs /var/cache/squid3 25600 16 256
> 
> Now, I get stable and high download rates even on a cache miss.
> 

Damn. That gives me ~90% confidence its the mem_node walking as new 4KB
chunks of memory are appended to the memory copy of the object.


I would expect to see a reduced effect in 3.5 that kicks in around
maximum_object_size_in_memory. Since the memory copies are now split
into cache_mem objects, vs transients (disk cache only, or totally
non-cacheable). With the transients getting their unnecessary in-memory
sections pruned away regularly.

Amos
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] Squid3: 100 % CPU load during object caching

[Marcus Kool]

On 07/24/2015 03:25 AM, Jens Offenbach wrote:

I have made a quick test of Squid 3.3.8 on Ubuntu 15.04 and I get the same 
problem: 100 % CPU usage, 500 KB/sec download rate.


Gesendet: Freitag, 24. Juli 2015 um 07:54 Uhr
Von: "Jens Offenbach" 
An: "Marcus Kool" , "Eliezer Croitoru" , 
"Amos Jeffries" , squid-users@lists.squid-cache.org
Betreff: Re: [squid-users] Squid3: 100 % CPU load during object caching
It is not easy for me, but I have tested Squid 3.3.8 from the Ubuntu packaging on a 
"real" physical infrastructure. I get the same results on the physical machine 
(1x Intel(R) Xeon(R) CPU E3-1225 V2 @ 3.20GHz, 32 GB RAM, 1 TB disk) where Squid is 
running: 100 % CPU usage, 500 KB/sec download rate. All machines are idle and we have 1 
GBit ethernet.

The strace log from the physical test scenario can be found here, but I think it does not 
differ from the "virtual" test scenario:
http://wikisend.com/download/293856/squid.strace2

@Marcus:
Have you verified that the file does not fit into memory and gets cached on 
disk? On which OS is Squid running? What are your build options of Squid (squid 
-v)? Is it possible that the issue is not part of 3.4.12? Do we have a 
regression?


I screwed up earlier since the maximum_object_size was too low for the test 
with a 1 GB file and did a new test.

The system has 64 GB memory and for sure the entire file is in the file system 
cache.  The disk system is HW RAID-1 with 1 GB cache.
The OS is Linux 3.10, CentOS 7 latest patches.

New test:
test system: 1 CPU with 4 cores/8 threads @ 3.7 GHz, 64 GB memory, AUFS, 1 Gbit 
pipe, 500 mbit guaranteed

with Squid 3.4.12 :
1st download starts with 90 MB/sec and halfway drops to 30 MB/sec.  My guess is 
that the file system cache got stressed and slowed things down.
2nd cached download with 190 MB/sec sustained and 120% CPU time.

With Squid 3.5.6 :
1st download starts with 90 MB/sec sustained and 80% CPU time.
2nd cached download with 190 MB/sec sustained and 120% CPU time.

As a comparison, I did "dd if=test of=test2 bs=4k" which uses 100% CPU time and 
has a throughput of 1200 MB/sec.
With bs=16k the throughput is 1300 MB/sec and with bs=64k the throughput is 
1400 MB/sec.

relevant parameters :
read_ahead_gap 64 KB
cache_mem 256 MB
maximum_object_size_in_memory 8 MB
maximum_object_size 8000 MB
cache_dir aufs /local/squid34/cache 1 32 256
cache_swap_low 92
cache_swap_high 93
# also ICAP daemon and URL rewriter configured
debug_options ALL,1 93,3 61,9

configure options:
'--prefix=/local/squid35' '--disable-ipv6' '--enable-fd-config' '--with-maxfd=3200' '--enable-async-io=64' '--enable-storeio=aufs' '--with-pthreads' '--enable-removal-policies=lru' 
'--disable-auto-locale' '--enable-default-err-language=English' '--enable-err-languages=Dutch English Portuguese' '--with-openssl' '--enable-ssl' '--enable-ssl-crtd' 
'--enable-cachemgr-hostname=localhost' '--enable-cache-digests' '--enable-follow-x-forwarded-for' '--enable-xmalloc-statistics' '--disable-hostname-checks' '--enable-epoll' '--enable-icap-client' 
'--enable-useragent-log' '--enable-referer-log' '--enable-stacktraces' '--enable-underscores' '--disable-icmp' '--mandir=/usr/local/share' 'CC=gcc' 'CFLAGS=-g -O2 -Wall -march=native' 'CXXFLAGS=-g -O2 
-Wall -march=native' --enable-ltdl-convenience


As you can see the cache_mem is small, If Amos finds it useful, I can do 
another test with a larger cache_mem.

Jens, since all your tests have a drop to 500 KB/sec I think the cause is 
somewhere is the configuration (Squid and/or OS).

Marcus



@Amos, Eliezer
Is someone able to reproduce the disk caching effect?

Regards,
Jens


Gesendet: Donnerstag, 23. Juli 2015 um 20:08 Uhr
Von: "Marcus Kool" 
An: "Jens Offenbach" , "Amos Jeffries" , "Eliezer 
Croitoru" , squid-users@lists.squid-cache.org
Betreff: Re: Aw: Re: [squid-users] Squid3: 100 % CPU load during object caching
The strace output shows this loop:

Squid reads 16K-1 bytes from FD 13 webserver
Squid writes 4 times 4K to FD 17 /var/cache/squid3/00/00/
Squid writes 4 times 4K to FD 12 browser

But this loop does not explain the 100% CPU usage...

Does Squid do a buffer reshuffle when it reads 16K-1 and writes 16K ?

I did the download test with Squid 3.4.12 AUFS on an idle system with a 500 
mbit connection and 1 CPU with 4 cores @ 3.7 GHz.
The first download used 35% of 1 CPU core with a steady download speed of 62 
MB/sec.
The second (cached) download used 50% of 1 CPU core with a steady download 
speed of 87 MB/sec.
I never looked at Squid CPU usage and do not know what is reasonable but it 

Re: [squid-users] Squid3: 100 % CPU load during object caching

[Jens Offenbach]

@Marcus:
I am not sure what exactly causes the problems, but could you please make a 
test with these two different settings:
cache_mem 4 GB
maximum_object_size_in_memory 1 GB

I think you will observe the behavior, that I was confronted with. The bad 
download rates of 500 KB/sec are gone, when I used the following settings:
cache_mem 256 MB
maximum_object_size_in_memory 16 MB

I think Amos has an idea what seems to be the source of the problem:
http://lists.squid-cache.org/pipermail/squid-users/2015-July/004728.html
 
Regards,
Jens


Gesendet: Freitag, 24. Juli 2015 um 14:33 Uhr
Von: "Marcus Kool" 
An: "Jens Offenbach" , squid-users@lists.squid-cache.org
Betreff: Re: [squid-users] Squid3: 100 % CPU load during object caching

On 07/24/2015 03:25 AM, Jens Offenbach wrote:
> I have made a quick test of Squid 3.3.8 on Ubuntu 15.04 and I get the same 
> problem: 100 % CPU usage, 500 KB/sec download rate.
>
>
> Gesendet: Freitag, 24. Juli 2015 um 07:54 Uhr
> Von: "Jens Offenbach" 
> An: "Marcus Kool" , "Eliezer Croitoru" 
> , "Amos Jeffries" , 
> squid-users@lists.squid-cache.org
> Betreff: Re: [squid-users] Squid3: 100 % CPU load during object caching
> It is not easy for me, but I have tested Squid 3.3.8 from the Ubuntu 
> packaging on a "real" physical infrastructure. I get the same results on the 
> physical machine (1x Intel(R) Xeon(R) CPU E3-1225 V2 @ 3.20GHz, 32 GB RAM, 1 
> TB disk) where Squid is running: 100 % CPU usage, 500 KB/sec download rate. 
> All machines are idle and we have 1 GBit ethernet.
>
> The strace log from the physical test scenario can be found here, but I think 
> it does not differ from the "virtual" test scenario:
> http://wikisend.com/download/293856/squid.strace2
>
> @Marcus:
> Have you verified that the file does not fit into memory and gets cached on 
> disk? On which OS is Squid running? What are your build options of Squid 
> (squid -v)? Is it possible that the issue is not part of 3.4.12? Do we have a 
> regression?

I screwed up earlier since the maximum_object_size was too low for the test 
with a 1 GB file and did a new test.

The system has 64 GB memory and for sure the entire file is in the file system 
cache. The disk system is HW RAID-1 with 1 GB cache.
The OS is Linux 3.10, CentOS 7 latest patches.

New test:
test system: 1 CPU with 4 cores/8 threads @ 3.7 GHz, 64 GB memory, AUFS, 1 Gbit 
pipe, 500 mbit guaranteed

with Squid 3.4.12 :
1st download starts with 90 MB/sec and halfway drops to 30 MB/sec. My guess is 
that the file system cache got stressed and slowed things down.
2nd cached download with 190 MB/sec sustained and 120% CPU time.

With Squid 3.5.6 :
1st download starts with 90 MB/sec sustained and 80% CPU time.
2nd cached download with 190 MB/sec sustained and 120% CPU time.

As a comparison, I did "dd if=test of=test2 bs=4k" which uses 100% CPU time and 
has a throughput of 1200 MB/sec.
With bs=16k the throughput is 1300 MB/sec and with bs=64k the throughput is 
1400 MB/sec.

relevant parameters :
read_ahead_gap 64 KB
cache_mem 256 MB
maximum_object_size_in_memory 8 MB
maximum_object_size 8000 MB
cache_dir aufs /local/squid34/cache 1 32 256
cache_swap_low 92
cache_swap_high 93
# also ICAP daemon and URL rewriter configured
debug_options ALL,1 93,3 61,9

configure options:
'--prefix=/local/squid35' '--disable-ipv6' '--enable-fd-config' 
'--with-maxfd=3200' '--enable-async-io=64' '--enable-storeio=aufs' 
'--with-pthreads' '--enable-removal-policies=lru'
'--disable-auto-locale' '--enable-default-err-language=English' 
'--enable-err-languages=Dutch English Portuguese' '--with-openssl' 
'--enable-ssl' '--enable-ssl-crtd'
'--enable-cachemgr-hostname=localhost' '--enable-cache-digests' 
'--enable-follow-x-forwarded-for' '--enable-xmalloc-statistics' 
'--disable-hostname-checks' '--enable-epoll' '--enable-icap-client'
'--enable-useragent-log' '--enable-referer-log' '--enable-stacktraces' 
'--enable-underscores' '--disable-icmp' '--mandir=/usr/local/share' 'CC=gcc' 
'CFLAGS=-g -O2 -Wall -march=native' 'CXXFLAGS=-g -O2
-Wall -march=native' --enable-ltdl-convenience

As you can see the cache_mem is small, If Amos finds it useful, I can do 
another test with a larger cache_mem.

Jens, since all your tests have a drop to 500 KB/sec I think the cause is 
somewhere is the configuration (Squid and/or OS).

Marcus


> @Amos, Eliezer
> Is someone able to reproduce the disk caching effect?
>
> Regards,
> Jens
>
>
> Gesendet: Donnerstag, 23. Juli 2015 um 20:08 Uhr
> Von: "Marc

Re: [squid-users] Squid3: 100 % CPU load during object caching

[Marcus Kool]

On 07/24/2015 01:01 PM, Jens Offenbach wrote:

@Marcus:
I am not sure what exactly causes the problems, but could you please make a 
test with these two different settings:
cache_mem 4 GB
maximum_object_size_in_memory 1 GB


I think this setting for maximum_object_size_in_memory is too high, independent 
of how the performance is.
The tests also show that large objects cached on disk have a good performance.
The perfect place for a large ISO image is the disk cache.

I did the test with squid 3.5.6 and got the same result as you have:
the download starts fast but quickly drops.  Squid uses 100% CPU.
wget displays 14 MB/sec ... 10 MB/sec ... 8 7 6 5 4 3 2 MB/sec and stays there 
for a long time.
At 50% downloaded the speed drops more to 1 MB/sec and at the end of the 
download I got 500 KB/sec *average*.
The second cached download was sustained 190 MB/sec and 120% CPU.

I did a second test with
cache_mem 4 GB
maximum_object_size_in_memory 200 MB

The download speed varied a lot: started with 30 MB/sec and went down and up 
many times between 6 MB/sec and 35 MB/sec.
The final average download speed was 31 MB/sec.  100% CPU.
The second cached download was sustained 190 MB/sec and 120% CPU.

Third test with
cache_mem 4 GB
maximum_object_size_in_memory 8 MB

The download speed started with 70 MB/sec and increased to 87 MB/sec.   100% CPU
The second cached download was sustained 190 MB/sec and 120% CPU.

4th test with
cache_mem 4 GB
maximum_object_size_in_memory 32 MB

The download speed started with 40 MB/sec and increased to 75 MB/sec.   100% 
CPU.
The second cached download was sustained 190 MB/sec and 120% CPU.

So Squid appears to have an issue with higher values of 
maximum_object_size_in_memory, the higher they are, the worse the performance.
For now, I would not go beyond 16 MB.
The question is, what is a reasonable size that you would like to be able to 
use for maximum_object_size_in_memory.
Do you have any particular requirement for a high maximum_object_size_in_memory 
?

Marcus


I think you will observe the behavior, that I was confronted with. The bad 
download rates of 500 KB/sec are gone, when I used the following settings:
cache_mem 256 MB
maximum_object_size_in_memory 16 MB

I think Amos has an idea what seems to be the source of the problem:
http://lists.squid-cache.org/pipermail/squid-users/2015-July/004728.html

Regards,
Jens


Gesendet: Freitag, 24. Juli 2015 um 14:33 Uhr
Von: "Marcus Kool" 
An: "Jens Offenbach" , squid-users@lists.squid-cache.org
Betreff: Re: [squid-users] Squid3: 100 % CPU load during object caching

On 07/24/2015 03:25 AM, Jens Offenbach wrote:

I have made a quick test of Squid 3.3.8 on Ubuntu 15.04 and I get the same 
problem: 100 % CPU usage, 500 KB/sec download rate.


Gesendet: Freitag, 24. Juli 2015 um 07:54 Uhr
Von: "Jens Offenbach" 
An: "Marcus Kool" , "Eliezer Croitoru" , 
"Amos Jeffries" , squid-users@lists.squid-cache.org
Betreff: Re: [squid-users] Squid3: 100 % CPU load during object caching
It is not easy for me, but I have tested Squid 3.3.8 from the Ubuntu packaging on a 
"real" physical infrastructure. I get the same results on the physical machine 
(1x Intel(R) Xeon(R) CPU E3-1225 V2 @ 3.20GHz, 32 GB RAM, 1 TB disk) where Squid is 
running: 100 % CPU usage, 500 KB/sec download rate. All machines are idle and we have 1 
GBit ethernet.

The strace log from the physical test scenario can be found here, but I think it does not 
differ from the "virtual" test scenario:
http://wikisend.com/download/293856/squid.strace2

@Marcus:
Have you verified that the file does not fit into memory and gets cached on 
disk? On which OS is Squid running? What are your build options of Squid (squid 
-v)? Is it possible that the issue is not part of 3.4.12? Do we have a 
regression?


I screwed up earlier since the maximum_object_size was too low for the test 
with a 1 GB file and did a new test.

The system has 64 GB memory and for sure the entire file is in the file system 
cache. The disk system is HW RAID-1 with 1 GB cache.
The OS is Linux 3.10, CentOS 7 latest patches.

New test:
test system: 1 CPU with 4 cores/8 threads @ 3.7 GHz, 64 GB memory, AUFS, 1 Gbit 
pipe, 500 mbit guaranteed

with Squid 3.4.12 :
1st download starts with 90 MB/sec and halfway drops to 30 MB/sec. My guess is 
that the file system cache got stressed and slowed things down.
2nd cached download with 190 MB/sec sustained and 120% CPU time.

With Squid 3.5.6 :
1st download starts with 90 MB/sec sustained and 80% CPU time.
2nd cached download with 190 MB/sec sustained and 120% CPU time.

As a comparison, I did "dd if=test of=test2 bs=4k" which uses 100% CPU time and 
has a throughput of 1200 MB/sec.
With bs=16k the throughput is 1300 MB/sec and with bs=64k the throughput is 
1400 MB/sec.

relevant parameters :
read_ahead_gap 64 KB
cache_mem 256 MB
maximum_object_size_in_memory 8 MB
maximum_object_size 800

Re: [squid-users] Squid3: 100 % CPU load during object caching

[Jens Offenbach]

Thanks a lot for your comprehensive testing. Now we are sure that the issue is 
independent of the underlying OS.

I can easily reduce the maximum_object_size_in_memory. I have some constraints 
from our IT at the moment. I am confronted with a very bad disk IO in our 
BladeCenter, so I am forced to put as many objects in memory as possible in 
order to get adequate performance values. We are also rare with disk space. RAM 
is not an issue right now.

Maybe Amos finds a way to fix the issue. Fortunately, we have a find a 
workaround.

Have a nice weekend!

Regards,
Jens


Gesendet: Freitag, 24. Juli 2015 um 19:01 Uhr
Von: "Marcus Kool" 
An: "Jens Offenbach" , squid-users@lists.squid-cache.org
Betreff: Re: Aw: Re: [squid-users] Squid3: 100 % CPU load during object caching

On 07/24/2015 01:01 PM, Jens Offenbach wrote:
> @Marcus:
> I am not sure what exactly causes the problems, but could you please make a 
> test with these two different settings:
> cache_mem 4 GB
> maximum_object_size_in_memory 1 GB

I think this setting for maximum_object_size_in_memory is too high, independent 
of how the performance is.
The tests also show that large objects cached on disk have a good performance.
The perfect place for a large ISO image is the disk cache.

I did the test with squid 3.5.6 and got the same result as you have:
the download starts fast but quickly drops. Squid uses 100% CPU.
wget displays 14 MB/sec ... 10 MB/sec ... 8 7 6 5 4 3 2 MB/sec and stays there 
for a long time.
At 50% downloaded the speed drops more to 1 MB/sec and at the end of the 
download I got 500 KB/sec *average*.
The second cached download was sustained 190 MB/sec and 120% CPU.

I did a second test with
cache_mem 4 GB
maximum_object_size_in_memory 200 MB

The download speed varied a lot: started with 30 MB/sec and went down and up 
many times between 6 MB/sec and 35 MB/sec.
The final average download speed was 31 MB/sec. 100% CPU.
The second cached download was sustained 190 MB/sec and 120% CPU.

Third test with
cache_mem 4 GB
maximum_object_size_in_memory 8 MB

The download speed started with 70 MB/sec and increased to 87 MB/sec. 100% CPU
The second cached download was sustained 190 MB/sec and 120% CPU.

4th test with
cache_mem 4 GB
maximum_object_size_in_memory 32 MB

The download speed started with 40 MB/sec and increased to 75 MB/sec. 100% CPU.
The second cached download was sustained 190 MB/sec and 120% CPU.

So Squid appears to have an issue with higher values of 
maximum_object_size_in_memory, the higher they are, the worse the performance.
For now, I would not go beyond 16 MB.
The question is, what is a reasonable size that you would like to be able to 
use for maximum_object_size_in_memory.
Do you have any particular requirement for a high maximum_object_size_in_memory 
?

Marcus

> I think you will observe the behavior, that I was confronted with. The bad 
> download rates of 500 KB/sec are gone, when I used the following settings:
> cache_mem 256 MB
> maximum_object_size_in_memory 16 MB
>
> I think Amos has an idea what seems to be the source of the problem:
> http://lists.squid-cache.org/pipermail/squid-users/2015-July/004728.html
>
> Regards,
> Jens
>
>
> Gesendet: Freitag, 24. Juli 2015 um 14:33 Uhr
> Von: "Marcus Kool" 
> An: "Jens Offenbach" , squid-users@lists.squid-cache.org
> Betreff: Re: [squid-users] Squid3: 100 % CPU load during object caching
>
> On 07/24/2015 03:25 AM, Jens Offenbach wrote:
>> I have made a quick test of Squid 3.3.8 on Ubuntu 15.04 and I get the same 
>> problem: 100 % CPU usage, 500 KB/sec download rate.
>>
>>
>> Gesendet: Freitag, 24. Juli 2015 um 07:54 Uhr
>> Von: "Jens Offenbach" 
>> An: "Marcus Kool" , "Eliezer Croitoru" 
>> , "Amos Jeffries" , 
>> squid-users@lists.squid-cache.org
>> Betreff: Re: [squid-users] Squid3: 100 % CPU load during object caching
>> It is not easy for me, but I have tested Squid 3.3.8 from the Ubuntu 
>> packaging on a "real" physical infrastructure. I get the same results on the 
>> physical machine (1x Intel(R) Xeon(R) CPU E3-1225 V2 @ 3.20GHz, 32 GB RAM, 1 
>> TB disk) where Squid is running: 100 % CPU usage, 500 KB/sec download rate. 
>> All machines are idle and we have 1 GBit ethernet.
>>
>> The strace log from the physical test scenario can be found here, but I 
>> think it does not differ from the "virtual" test scenario:
>> http://wikisend.com/download/293856/squid.strace2[http://wikisend.com/download/293856/squid.strace2]
>>
>> @Marcus:
>> Have you verified that the file does not fit into memory and gets cached on 
>> disk? On which OS is Squid running? What are your build options of Squid 
>> (squid -v)? Is it possible that the issue is not part of 

Re: [squid-users] Squid3: 100 % CPU load during object caching

[Marcus Kool]

On 07/25/2015 05:25 AM, Jens Offenbach wrote:

Thanks a lot for your comprehensive testing. Now we are sure that the issue is 
independent of the underlying OS.

I can easily reduce the maximum_object_size_in_memory. I have some constraints 
from our IT at the moment. I am confronted with a very bad disk IO in our 
BladeCenter, so I am forced to put as many objects in memory as possible in 
order to get adequate performance values. We are also rare with disk space. RAM 
is not an issue right now.

Maybe Amos finds a way to fix the issue. Fortunately, we have a find a 
workaround.


If memory is plentyful, just make sure that the OS has a large file system 
cache.
So reduce mem_cahce of Squid a little and tune the OS with
vm.swappiness=10
in /etc/sysctl.conf

Best regards

Marcus



Have a nice weekend!

Regards,
Jens


Gesendet: Freitag, 24. Juli 2015 um 19:01 Uhr
Von: "Marcus Kool" 
An: "Jens Offenbach" , squid-users@lists.squid-cache.org
Betreff: Re: Aw: Re: [squid-users] Squid3: 100 % CPU load during object caching

On 07/24/2015 01:01 PM, Jens Offenbach wrote:

@Marcus:
I am not sure what exactly causes the problems, but could you please make a 
test with these two different settings:
cache_mem 4 GB
maximum_object_size_in_memory 1 GB


I think this setting for maximum_object_size_in_memory is too high, independent 
of how the performance is.
The tests also show that large objects cached on disk have a good performance.
The perfect place for a large ISO image is the disk cache.

I did the test with squid 3.5.6 and got the same result as you have:
the download starts fast but quickly drops. Squid uses 100% CPU.
wget displays 14 MB/sec ... 10 MB/sec ... 8 7 6 5 4 3 2 MB/sec and stays there 
for a long time.
At 50% downloaded the speed drops more to 1 MB/sec and at the end of the 
download I got 500 KB/sec *average*.
The second cached download was sustained 190 MB/sec and 120% CPU.

I did a second test with
cache_mem 4 GB
maximum_object_size_in_memory 200 MB

The download speed varied a lot: started with 30 MB/sec and went down and up 
many times between 6 MB/sec and 35 MB/sec.
The final average download speed was 31 MB/sec. 100% CPU.
The second cached download was sustained 190 MB/sec and 120% CPU.

Third test with
cache_mem 4 GB
maximum_object_size_in_memory 8 MB

The download speed started with 70 MB/sec and increased to 87 MB/sec. 100% CPU
The second cached download was sustained 190 MB/sec and 120% CPU.

4th test with
cache_mem 4 GB
maximum_object_size_in_memory 32 MB

The download speed started with 40 MB/sec and increased to 75 MB/sec. 100% CPU.
The second cached download was sustained 190 MB/sec and 120% CPU.

So Squid appears to have an issue with higher values of 
maximum_object_size_in_memory, the higher they are, the worse the performance.
For now, I would not go beyond 16 MB.
The question is, what is a reasonable size that you would like to be able to 
use for maximum_object_size_in_memory.
Do you have any particular requirement for a high maximum_object_size_in_memory 
?

Marcus


I think you will observe the behavior, that I was confronted with. The bad 
download rates of 500 KB/sec are gone, when I used the following settings:
cache_mem 256 MB
maximum_object_size_in_memory 16 MB

I think Amos has an idea what seems to be the source of the problem:
http://lists.squid-cache.org/pipermail/squid-users/2015-July/004728.html

Regards,
Jens


Gesendet: Freitag, 24. Juli 2015 um 14:33 Uhr
Von: "Marcus Kool" 
An: "Jens Offenbach" , squid-users@lists.squid-cache.org
Betreff: Re: [squid-users] Squid3: 100 % CPU load during object caching

On 07/24/2015 03:25 AM, Jens Offenbach wrote:

I have made a quick test of Squid 3.3.8 on Ubuntu 15.04 and I get the same 
problem: 100 % CPU usage, 500 KB/sec download rate.


Gesendet: Freitag, 24. Juli 2015 um 07:54 Uhr
Von: "Jens Offenbach" 
An: "Marcus Kool" , "Eliezer Croitoru" , 
"Amos Jeffries" , squid-users@lists.squid-cache.org
Betreff: Re: [squid-users] Squid3: 100 % CPU load during object caching
It is not easy for me, but I have tested Squid 3.3.8 from the Ubuntu packaging on a 
"real" physical infrastructure. I get the same results on the physical machine 
(1x Intel(R) Xeon(R) CPU E3-1225 V2 @ 3.20GHz, 32 GB RAM, 1 TB disk) where Squid is 
running: 100 % CPU usage, 500 KB/sec download rate. All machines are idle and we have 1 
GBit ethernet.

The strace log from the physical test scenario can be found here, but I think it does not 
differ from the "virtual" test scenario:
http://wikisend.com/download/293856/squid.strace2[http://wikisend.com/download/293856/squid.strace2]

@Marcus:
Have you verified that the file does not fit into memory and gets cached on 
disk? On which OS is Squid running? What are your build options of Squid (squid 
-v)? Is it possible that the issue is not part of 3.4.12? Do we have a 
regression?


I screwed up earlier since th