Re: [squid-users] Squid cpu usage 100% from few days ago !!

[Alex Rousskov]

On 08/16/2016 11:03 PM, Omid Kosari wrote:
> Even one ip address with less than 5 requests per second can grow squid cpu
> usage up to 30% . And 10 requests per second made 100% cpu usage . While
> there is nothing other than that client goes through squid . The client
> bandwidth is less than 10Kbps .
> 
> Isn't it crazy also ?

It is.

Alex.

___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] Squid cpu usage 100% from few days ago !!

[Omid Kosari]

Aha . We have found that this request belongs to a cheap popular satellite
receiver www.starmax.co . Maybe it has been infected and becomes zombie of a
btnet . Maybe you should buy one device from them 




--
View this message in context: 
http://squid-web-proxy-cache.1019090.n4.nabble.com/Squid-cpu-usage-100-from-few-days-ago-tp4678894p4678978.html
Sent from the Squid - Users mailing list archive at Nabble.com.
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] Squid cpu usage 100% from few days ago !!

[Eliezer Croitoru]

Hey Omid,

I can try to use the PCAP files but I am trying to stick in the upper level of 
operation when testing.
What I mean by that is that I am trying to find real world software which 
encounter an issue when squid is in the middle.
I can write scripts but as long there is something I can reproduce as a user 
and not as a programmer I prefer to stick with it.

Eliezer


Eliezer Croitoru
Linux System Administrator
Mobile: +972-5-28704261
Email: elie...@ngtech.co.il


-Original Message-
From: squid-users [mailto:squid-users-boun...@lists.squid-cache.org] On Behalf 
Of Omid Kosari
Sent: Wednesday, August 17, 2016 4:56 PM
To: squid-users@lists.squid-cache.org
Subject: Re: [squid-users] Squid cpu usage 100% from few days ago !!

Thanks for reply

I have provided a sample wireshark pcap and squid access.log  here 
http://squid-web-proxy-cache.1019090.n4.nabble.com/Malformed-HTTP-on-tproxy-squid-tp4678951p4678952.html

Maybe you can reproduce and resend those requests with the help of something 
like fiddler or any other tool . Also i am volunteer to provide unlimited 
number of such pcap files.





--
View this message in context: 
http://squid-web-proxy-cache.1019090.n4.nabble.com/Squid-cpu-usage-100-from-few-days-ago-tp4678894p4678973.html
Sent from the Squid - Users mailing list archive at Nabble.com.
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] Squid cpu usage 100% from few days ago !!

[Omid Kosari]

Thanks for reply

I have provided a sample wireshark pcap and squid access.log  here
http://squid-web-proxy-cache.1019090.n4.nabble.com/Malformed-HTTP-on-tproxy-squid-tp4678951p4678952.html

Maybe you can reproduce and resend those requests with the help of something
like fiddler or any other tool . Also i am volunteer to provide unlimited
number of such pcap files.





--
View this message in context: 
http://squid-web-proxy-cache.1019090.n4.nabble.com/Squid-cpu-usage-100-from-few-days-ago-tp4678894p4678973.html
Sent from the Squid - Users mailing list archive at Nabble.com.
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] Squid cpu usage 100% from few days ago !!

[Eliezer Croitoru]

Thanks Antony!
It kind of disappeared from my mind\eyes despite to the fact that it's there.

Omid,
You are right about the expectation from a software to be polished.
I am with you on this but naturally port 80 should be used for http in a world 
which everybody obeys the holy RFC's.
There are couple options on how to resolve the issue but non of them are easy 
to put in production.

Amos can say what is to be expected from squid-4.
However I have it on my todo list to test the on_unsupported_protocol 
[http://www.squid-cache.org/Versions/v4/cfgman/on_unsupported_protocol.html] 
with hope it will give you and many others the right solution to weird 
scenarios.
Currently I do not have enough options\sites to test the issue also I am not 
sure about the expected result in a case of websockets.
I have compiled squid-4 but it will take me a bit more time to test over and 
over again.
If you have any suggestion on how to actually test this issue I would be happy 
to test the subject.
Currently my test lab is composed of:
- CentOS\Fedora\Ubuntu Intercept router
- Windows 7\8\10 Client
- Debian\Ubuntu\Fedora\CentOS\OpenSuse client
- Customized tcp service on port 80 which can serve both http and couple other 
protocols

Let me know about how I can verify the issue.

Eliezer


Eliezer Croitoru
Linux System Administrator
Mobile: +972-5-28704261
Email: elie...@ngtech.co.il


-Original Message-
From: squid-users [mailto:squid-users-boun...@lists.squid-cache.org] On Behalf 
Of Antony Stone
Sent: Wednesday, August 17, 2016 12:04 PM
To: squid-users@lists.squid-cache.org
Subject: Re: [squid-users] Squid cpu usage 100% from few days ago !!

On Wednesday 17 August 2016 at 11:01:40, Eliezer Croitoru wrote:

> Hey Omid,
> 
> Just to understand, are you intercepting traffic?

From the original report: "Squid is in tproxy mode with routing"


Antony.

> -Original Message-
> From: squid-users [mailto:squid-users-boun...@lists.squid-cache.org] 
> On Behalf Of Omid Kosari Sent: Wednesday, August 17, 2016 8:04 AM
> To: squid-users@lists.squid-cache.org
> Subject: Re: [squid-users] Squid cpu usage 100% from few days ago !!
> 
> Even one ip address with less than 5 requests per second can grow 
> squid cpu usage up to 30% . And 10 requests per second made 100% cpu 
> usage . While there is nothing other than that client goes through 
> squid . The client bandwidth is less than 10Kbps .
> 
> Isn't it crazy also ?

--
Salad is what food eats.

   Please reply to the list;
 please *don't* CC me.
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] Squid cpu usage 100% from few days ago !!

[Omid Kosari]

Matus UHLAR - fantomas wrote
> are you intercepting traffic for port 80 only?

yes



--
View this message in context: 
http://squid-web-proxy-cache.1019090.n4.nabble.com/Squid-cpu-usage-100-from-few-days-ago-tp4678894p4678968.html
Sent from the Squid - Users mailing list archive at Nabble.com.
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] Squid cpu usage 100% from few days ago !!

[Matus UHLAR - fantomas]

On 16.08.16 22:03, Omid Kosari wrote:

Even one ip address with less than 5 requests per second can grow squid cpu
usage up to 30% . And 10 requests per second made 100% cpu usage . While
there is nothing other than that client goes through squid . The client
bandwidth is less than 10Kbps .

Isn't it crazy also ?


are you intercepting traffic for port 80 only?


--
Matus UHLAR - fantomas, uh...@fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
- Have you got anything without Spam in it?
- Well, there's Spam egg sausage and Spam, that's not got much Spam in it.
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] Squid cpu usage 100% from few days ago !!

[Antony Stone]

On Wednesday 17 August 2016 at 11:01:40, Eliezer Croitoru wrote:

> Hey Omid,
> 
> Just to understand, are you intercepting traffic?

From the original report: "Squid is in tproxy mode with routing"


Antony.

> -Original Message-
> From: squid-users [mailto:squid-users-boun...@lists.squid-cache.org] On
> Behalf Of Omid Kosari Sent: Wednesday, August 17, 2016 8:04 AM
> To: squid-users@lists.squid-cache.org
> Subject: Re: [squid-users] Squid cpu usage 100% from few days ago !!
> 
> Even one ip address with less than 5 requests per second can grow squid cpu
> usage up to 30% . And 10 requests per second made 100% cpu usage . While
> there is nothing other than that client goes through squid . The client
> bandwidth is less than 10Kbps .
> 
> Isn't it crazy also ?

-- 
Salad is what food eats.

   Please reply to the list;
 please *don't* CC me.
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] Squid cpu usage 100% from few days ago !!

[Eliezer Croitoru]

Hey Omid,

Just to understand, are you intercepting traffic?

Eliezer


Eliezer Croitoru
Linux System Administrator
Mobile: +972-5-28704261
Email: elie...@ngtech.co.il


-Original Message-
From: squid-users [mailto:squid-users-boun...@lists.squid-cache.org] On Behalf 
Of Omid Kosari
Sent: Wednesday, August 17, 2016 8:04 AM
To: squid-users@lists.squid-cache.org
Subject: Re: [squid-users] Squid cpu usage 100% from few days ago !!

Even one ip address with less than 5 requests per second can grow squid cpu 
usage up to 30% . And 10 requests per second made 100% cpu usage . While there 
is nothing other than that client goes through squid . The client bandwidth is 
less than 10Kbps .

Isn't it crazy also ?



--
View this message in context: 
http://squid-web-proxy-cache.1019090.n4.nabble.com/Squid-cpu-usage-100-from-few-days-ago-tp4678894p4678961.html
Sent from the Squid - Users mailing list archive at Nabble.com.
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] Squid cpu usage 100% from few days ago !!

[Omid Kosari]

Even one ip address with less than 5 requests per second can grow squid cpu
usage up to 30% . And 10 requests per second made 100% cpu usage . While
there is nothing other than that client goes through squid . The client
bandwidth is less than 10Kbps .

Isn't it crazy also ?



--
View this message in context: 
http://squid-web-proxy-cache.1019090.n4.nabble.com/Squid-cpu-usage-100-from-few-days-ago-tp4678894p4678961.html
Sent from the Squid - Users mailing list archive at Nabble.com.
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] Squid cpu usage 100% from few days ago !!

[Amos Jeffries]

On 17/08/2016 5:06 a.m., Alex Rousskov wrote:
> On 08/15/2016 02:58 AM, Omid Kosari wrote:
> 
>> Maybe our clients are infected and they are zombies .
>>
>> Anyone knows some good ways to defend squid . I mean when squid forwards
>> these requests it becomes crazy .
> 
> If Squid becomes crazy, it is a Squid bug that you should report and
> help fix. Even when Squid deals with zombie clients, it should remain sane.

FYI: Omid is using "crazy" to mean 100% CPU is used under DoS type
conditions. Which is not crazy at all.

Amos

___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] Squid cpu usage 100% from few days ago !!

[Alex Rousskov]

On 08/15/2016 02:58 AM, Omid Kosari wrote:

> Maybe our clients are infected and they are zombies .
> 
> Anyone knows some good ways to defend squid . I mean when squid forwards
> these requests it becomes crazy .

If Squid becomes crazy, it is a Squid bug that you should report and
help fix. Even when Squid deals with zombie clients, it should remain sane.

Alex.

___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] Squid cpu usage 100% from few days ago !!

[Omid Kosari]

One of server's ip addresses that i've found belongs to cloudflare .
Cloudflare does not accept anything other than HTTP on port 80 . So it seems
an attack to some servers .
Maybe our clients are infected and they are zombies .

Anyone knows some good ways to defend squid . I mean when squid forwards
these requests it becomes crazy .

I manage to create some iptables rules on squid box to only accept http
protocol . But i know it will have at least 2 problems .
1. Performance will be degraded
2. Some sites/apps may have problems

Any suggestion ?



--
View this message in context: 
http://squid-web-proxy-cache.1019090.n4.nabble.com/Squid-cpu-usage-100-from-few-days-ago-tp4678894p4678937.html
Sent from the Squid - Users mailing list archive at Nabble.com.
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] Squid cpu usage 100% from few days ago !!

[Yuri Voinov]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256
 


14.08.2016 18:35, Antony Stone пишет:
> On Sunday 14 August 2016 at 14:25:38, Omid Kosari wrote:
>
>> Still could not find the app or url but the 2 server ip addresses are
>> 149.202.92.139 and 173.236.187.17 .
>
> The first does not respond on port 80 for me, and the second simply
tells me
> that it doesn't know which Dreamhost website I'm trying to access.
>
>> TAG_NONE/400 24728 "Strange binary characters here"
>
> I would agree with that description.
>
> My guess is that something is sending non-HTTP data over connections
to port
> 80 on the servers, therefore Squid can't handle it (because it's an HTTP
> proxy).
Bittorrent often use HTTP port, also some IM - like ICQ.
>
> I think you should be able to identify what the target URL / request
being
> made by the client application is, provided you can get at the earliest
> entries in your log files for such a connection.
Identify client application and block it ;) Or bypass proxy for it :)
>
>
>
> Regards,
>
>
> Antony.
>

-BEGIN PGP SIGNATURE-
Version: GnuPG v2
 
iQEcBAEBCAAGBQJXsIHUAAoJENNXIZxhPexGx+kH/RiqhZgiP9KudxTcrpaZJOgm
dPC/5zRoQRt09uQnyGRcaY7xXfAdpSzRU+BI6OIaiOnOZwbo0/rWkaPvpe+FUwHo
D4PSVzzfCd9SyVkuPpIbA6B/gYqpqNZR8FuGkxMBN/860t1yfkXlBjP/fEU5daiL
BfarfyGSl8qc5iA73SgvhK8r+jVJd9wKcT/TdFvlrgxXFD/Z1OPAQs1Gztj00mk6
qVRoj60s8yP0thialLln43Z+kw/Z3H/es6ZFw43HXvT4jeCwZxK1LCXnAwHlPRi/
tzUOTEwAnWoOTOMunF7K5wnKvLdUCY8eWp5LhWDBoZI7TvJ/XnR4wlTZ6E53q3s=
=Qyu3
-END PGP SIGNATURE-



0x613DEC46.asc
Description: application/pgp-keys
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] Squid cpu usage 100% from few days ago !!

[Antony Stone]

On Sunday 14 August 2016 at 14:25:38, Omid Kosari wrote:

> Still could not find the app or url but the 2 server ip addresses are
> 149.202.92.139 and 173.236.187.17 .

The first does not respond on port 80 for me, and the second simply tells me 
that it doesn't know which Dreamhost website I'm trying to access.

> TAG_NONE/400 24728 "Strange binary characters here"

I would agree with that description.

My guess is that something is sending non-HTTP data over connections to port 
80 on the servers, therefore Squid can't handle it (because it's an HTTP 
proxy).

I think you should be able to identify what the target URL / request being 
made by the client application is, provided you can get at the earliest 
entries in your log files for such a connection.


Regards,


Antony.

-- 
"Black holes are where God divided by zero."

 - Steven Wright

   Please reply to the list;
 please *don't* CC me.
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] Squid cpu usage 100% from few days ago !!

[Omid Kosari]

Still could not find the app or url but the 2 server ip addresses are
149.202.92.139 and 173.236.187.17 .

As i said this is a sample random user . The only thing common between these
kind of users are 

TAG_NONE/400 24728 NONE error:invalid-request

or 

TAG_NONE/400 24728 "Strange binary characters here"

in logs



--
View this message in context: 
http://squid-web-proxy-cache.1019090.n4.nabble.com/Squid-cpu-usage-100-from-few-days-ago-tp4678894p4678933.html
Sent from the Squid - Users mailing list archive at Nabble.com.
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] Squid cpu usage 100% from few days ago !!

[Antony Stone]

On Sunday 14 August 2016 at 14:03:44, Omid Kosari wrote:

> I've found that a kind of request makes a loop in squid . Wireshark shows
> infinite loop of
> 
> X-Squid-Error: ERR_INVALID_REQ 0
> 
> and
> 
> X-Squid-Error: ERR_INVALID_URL 0
> 
> which makes high cpu usage.

What is the URL / request which results in this effect?

What application is generating this request?


Antony.

-- 
Douglas was one of those writers who honourably failed to get anywhere with 
'weekending'.  It put a premium on people who could write things that lasted 
thirty seconds, and Douglas was incapable of writing a single sentence that 
lasted less than thirty seconds.

 - Geoffrey Perkins, about Douglas Adams

   Please reply to the list;
 please *don't* CC me.
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] Squid cpu usage 100% from few days ago !!

[Omid Kosari]

My investigation shows even 1 random chosen ip address makes squid cpu usage
about 30% . 
I have chosen that ip address based on users with TAG_NONE/400 errors .

I've found that a kind of request makes a loop in squid . Wireshark shows
infinite loop of

X-Squid-Error: ERR_INVALID_REQ 0

and

X-Squid-Error: ERR_INVALID_URL 0

which makes high cpu usage .

Please find the attachements . The last files edited and personal info
removed from it

squid-access-log.JPG

  
squid-access-log2.JPG

  
squid-problem.squid-problem

  



--
View this message in context: 
http://squid-web-proxy-cache.1019090.n4.nabble.com/Squid-cpu-usage-100-from-few-days-ago-tp4678894p4678931.html
Sent from the Squid - Users mailing list archive at Nabble.com.
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] Squid cpu usage 100% from few days ago !!

[Antony Stone]

On Saturday 13 August 2016 at 20:51:40, Alex Rousskov wrote:

> On 08/13/2016 06:09 AM, Antony Stone wrote:
> > On Saturday 13 August 2016 at 13:40:18, Omid Kosari wrote:
> >> debug_options ALL,1
> > 
> > I would not recommend having debugging turned on for a production server.
> 
> "ALL,1" is the default and recommended verbosity level. The word
> "debugging" is somewhat misleading in this context.

I apologise for making assumptions about one of the options I am clearly not 
sufficiently familiar with.


Antony.

-- 
You can tell that the day just isn't going right when you find yourself using 
the telephone before the toilet.

   Please reply to the list;
 please *don't* CC me.
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] Squid cpu usage 100% from few days ago !!

[Alex Rousskov]

On 08/13/2016 06:09 AM, Antony Stone wrote:
> On Saturday 13 August 2016 at 13:40:18, Omid Kosari wrote:
> 
>> debug_options ALL,1
> 
> I would not recommend having debugging turned on for a production server.

"ALL,1" is the default and recommended verbosity level. The word
"debugging" is somewhat misleading in this context.

Alex.

___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] Squid cpu usage 100% from few days ago !!

[Omid Kosari]

Turned off , still high cpu usage .



--
View this message in context: 
http://squid-web-proxy-cache.1019090.n4.nabble.com/Squid-cpu-usage-100-from-few-days-ago-tp4678894p4678904.html
Sent from the Squid - Users mailing list archive at Nabble.com.
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] Squid cpu usage 100% from few days ago !!

[Antony Stone]

On Saturday 13 August 2016 at 13:40:18, Omid Kosari wrote:

> debug_options ALL,1

I would not recommend having debugging turned on for a production server.

In the first instance, to check whether this is what is causing your problems, 
turn this option off and see whether your CPU load comes back to normal.


Antony.

-- 
The Magic Words are Squeamish Ossifrage.

   Please reply to the list;
 please *don't* CC me.
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] Squid cpu usage 100% from few days ago !!

[Omid Kosari]

debug_options ALL,1



--
View this message in context: 
http://squid-web-proxy-cache.1019090.n4.nabble.com/Squid-cpu-usage-100-from-few-days-ago-tp4678894p4678901.html
Sent from the Squid - Users mailing list archive at Nabble.com.
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] Squid cpu usage 100% from few days ago !!

[Marcus Kool]

It seems that squid is doing a lot of calls to vfprintf.
The first thing that comes to mind is that you have debugging on.
What is the setting for debug_options ?

Marcus

On 08/13/2016 04:18 AM, Omid Kosari wrote:

Hello,

Recently 2 different squid boxes grows from ~40% cpu usage to 100% without
any changes to config/banwidth/number of clients/etc

The problems forced me to bypass squid until the problem found .
Right now even 10% of users can make squid 100% .

Info

Squid is in tproxy mode with routing

Ubuntu Linux 16.04 , 4.4.0-34-generic on x86_64
Squid Cache: Version 3.5.19 from debian repository


samples  %image name   symbol name
1532894  42.8190  libc-2.23.so _IO_strn_overflow
1028537  28.7306  libc-2.23.so _IO_default_xsputn
662802   18.5143  libc-2.23.so vfprintf
77019 2.1514  squid/usr/sbin/squid
28861 0.8062  libc-2.23.so __memset_sse2
26948 0.7528  r8169/r8169
25320 0.7073  libc-2.23.so __memcpy_sse2_unaligned
21712 0.6065  libc-2.23.so __GI___mempcpy
14918 0.4167  libc-2.23.so _int_malloc
8889  0.2483  nf_conntrack /nf_conntrack
8130  0.2271  libc-2.23.so __GI_strchr
6357  0.1776  libc-2.23.so _int_free
4152  0.1160  libc-2.23.so re_search_internal
4043  0.1129  libc-2.23.so strlen
2754  0.0769  libstdc++.so.6.0.21
/usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21
2753  0.0769  libc-2.23.so free
2704  0.0755  ip_tables/ip_tables
2560  0.0715  reiserfs /reiserfs
2332  0.0651  kallsyms ___slab_alloc
2284  0.0638  libc-2.23.so malloc_consolidate
2204  0.0616  libc-2.23.so malloc
2175  0.0608  kallsyms sys_epoll_ctl
2035  0.0568  kallsyms csum_partial_copy_generic
1614  0.0451  libc-2.23.so calloc
1552  0.0434  kallsyms _raw_spin_lock
1208  0.0337  kallsyms memcpy
1203  0.0336  kallsyms nf_iterate
1177  0.0329  kallsyms irq_entries_start
1165  0.0325  kallsyms __fget
1072  0.0299  kallsyms copy_user_generic_string
1037  0.0290  kallsyms __alloc_skb
1002  0.0280  kallsyms tcp_sendmsg
945   0.0264  libc-2.23.so build_upper_buffer
875   0.0244  kallsyms kmem_cache_free
873   0.0244  kallsyms tcp_rack_mark_lost
868   0.0242  nf_nat_ipv4  /nf_nat_ipv4
861   0.0241  kallsyms kfree
837   0.0234  kallsyms __inet_lookup_established
834   0.0233  kallsyms get_partial_node.isra.61
825   0.0230  kallsyms __slab_free
815   0.0228  kallsyms sock_poll
810   0.0226  kallsyms skb_release_data
802   0.0224  nf_conntrack_ipv4/nf_conntrack_ipv4
792   0.0221  kallsyms tcp_transmit_skb
771   0.0215  kallsyms kmem_cache_alloc
719   0.0201  kallsyms fib_table_lookup
704   0.0197  kallsyms _raw_spin_lock_irqsave
701   0.0196  kallsyms tcp_v4_rcv
699   0.0195  libm-2.23.so __ieee754_log_avx
686   0.0192  nf_nat   /nf_nat
684   0.0191  kallsyms tcp_write_xmit
674   0.0188  kallsyms __cmpxchg_double_slab.isra.44
626   0.0175  kallsyms __netif_receive_skb_core
621   0.0173  libnettle.so.6.2
/usr/lib/x86_64-linux-gnu/libnettle.so.6.2
608   0.0170  kallsyms delay_tsc
600   0.0168  kallsyms ksize
595   0.0166  kallsyms tcp_ack
592   0.0165  kallsyms __local_bh_enable_ip




--
View this message in context: 
http://squid-web-proxy-cache.1019090.n4.nabble.com/Squid-cpu-usage-100-from-few-days-ago-tp4678894.html
Sent from the Squid - Users mailing list archive at Nabble.com.
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users


___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] Squid cpu usage 100% from few days ago !!

[Omid Kosari]

The bandwidth was about 120Mbps to each squid box but now even 10Mbps makes
100% cpu usage

With 10% of users
Average HTTP requests per minute since start:   2355.3


16GB of ram and i3-2100 CPU @ 3.10GHz, 4 cores and NO SMP like before .

It seems like an attack to/from our clients to/from internet which makes
squid crazy .

Also the profiling result attached to end of my first post .




--
View this message in context: 
http://squid-web-proxy-cache.1019090.n4.nabble.com/Squid-cpu-usage-100-from-few-days-ago-tp4678894p4678899.html
Sent from the Squid - Users mailing list archive at Nabble.com.
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] Squid cpu usage 100% from few days ago !!

[Antony Stone]

On Saturday 13 August 2016 at 09:18:24, Omid Kosari wrote:

> Hello,
> 
> Recently 2 different squid boxes grows from ~40% cpu usage to 100% without
> any changes to config/banwidth/number of clients/etc

What are your bandwidth and number of clients?

> The problems forced me to bypass squid until the problem found .
> Right now even 10% of users can make squid 100% .

How many users is 10%, and how many accesses per second/minute are they making 
through Squid?

> Info
> 
> Squid is in tproxy mode with routing
> 
> Ubuntu Linux 16.04 , 4.4.0-34-generic on x86_64

How much RAM do you have, how many CPU cores, what speed CPU?

> Squid Cache: Version 3.5.19 from debian repository
> 
> 
> samples  %image name   symbol name
> 1532894  42.8190  libc-2.23.so _IO_strn_overflow
> 1028537  28.7306  libc-2.23.so _IO_default_xsputn
> 662802   18.5143  libc-2.23.so vfprintf
> 77019 2.1514  squid/usr/sbin/squid
> 28861 0.8062  libc-2.23.so __memset_sse2
> 26948 0.7528  r8169/r8169
> 25320 0.7073  libc-2.23.so __memcpy_sse2_unaligned
> 21712 0.6065  libc-2.23.so __GI___mempcpy
> 14918 0.4167  libc-2.23.so _int_malloc
> 8889  0.2483  nf_conntrack /nf_conntrack
> 8130  0.2271  libc-2.23.so __GI_strchr
> 6357  0.1776  libc-2.23.so _int_free
> 4152  0.1160  libc-2.23.so re_search_internal
> 4043  0.1129  libc-2.23.so strlen
> 2754  0.0769  libstdc++.so.6.0.21
> /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21
> 2753  0.0769  libc-2.23.so free
> 2704  0.0755  ip_tables/ip_tables
> 2560  0.0715  reiserfs /reiserfs
> 2332  0.0651  kallsyms ___slab_alloc
> 2284  0.0638  libc-2.23.so malloc_consolidate
> 2204  0.0616  libc-2.23.so malloc
> 2175  0.0608  kallsyms sys_epoll_ctl
> 2035  0.0568  kallsyms csum_partial_copy_generic
> 1614  0.0451  libc-2.23.so calloc
> 1552  0.0434  kallsyms _raw_spin_lock
> 1208  0.0337  kallsyms memcpy
> 1203  0.0336  kallsyms nf_iterate
> 1177  0.0329  kallsyms irq_entries_start
> 1165  0.0325  kallsyms __fget
> 1072  0.0299  kallsyms copy_user_generic_string
> 1037  0.0290  kallsyms __alloc_skb
> 1002  0.0280  kallsyms tcp_sendmsg
> 945   0.0264  libc-2.23.so build_upper_buffer
> 875   0.0244  kallsyms kmem_cache_free
> 873   0.0244  kallsyms tcp_rack_mark_lost
> 868   0.0242  nf_nat_ipv4  /nf_nat_ipv4
> 861   0.0241  kallsyms kfree
> 837   0.0234  kallsyms __inet_lookup_established
> 834   0.0233  kallsyms get_partial_node.isra.61
> 825   0.0230  kallsyms __slab_free
> 815   0.0228  kallsyms sock_poll
> 810   0.0226  kallsyms skb_release_data
> 802   0.0224  nf_conntrack_ipv4/nf_conntrack_ipv4
> 792   0.0221  kallsyms tcp_transmit_skb
> 771   0.0215  kallsyms kmem_cache_alloc
> 719   0.0201  kallsyms fib_table_lookup
> 704   0.0197  kallsyms _raw_spin_lock_irqsave
> 701   0.0196  kallsyms tcp_v4_rcv
> 699   0.0195  libm-2.23.so __ieee754_log_avx
> 686   0.0192  nf_nat   /nf_nat
> 684   0.0191  kallsyms tcp_write_xmit
> 674   0.0188  kallsyms __cmpxchg_double_slab.isra.44
> 626   0.0175  kallsyms __netif_receive_skb_core
> 621   0.0173  libnettle.so.6.2
> /usr/lib/x86_64-linux-gnu/libnettle.so.6.2
> 608   0.0170  kallsyms delay_tsc
> 600   0.0168  kallsyms ksize
> 595   0.0166  kallsyms tcp_ack
> 592   0.0165  kallsyms __local_bh_enable_ip

Regards,


Antony.

-- 
Anyone that's normal doesn't really achieve much.

 - Mark Blair, Australian rocket engineer

   Please reply to the list;
 please *don't* CC me.
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

[squid-users] Squid cpu usage 100% from few days ago !!

[Omid Kosari]

Hello,

Recently 2 different squid boxes grows from ~40% cpu usage to 100% without
any changes to config/banwidth/number of clients/etc

The problems forced me to bypass squid until the problem found . 
Right now even 10% of users can make squid 100% .

Info 

Squid is in tproxy mode with routing

Ubuntu Linux 16.04 , 4.4.0-34-generic on x86_64
Squid Cache: Version 3.5.19 from debian repository


samples  %image name   symbol name
1532894  42.8190  libc-2.23.so _IO_strn_overflow
1028537  28.7306  libc-2.23.so _IO_default_xsputn
662802   18.5143  libc-2.23.so vfprintf
77019 2.1514  squid/usr/sbin/squid
28861 0.8062  libc-2.23.so __memset_sse2
26948 0.7528  r8169/r8169
25320 0.7073  libc-2.23.so __memcpy_sse2_unaligned
21712 0.6065  libc-2.23.so __GI___mempcpy
14918 0.4167  libc-2.23.so _int_malloc
8889  0.2483  nf_conntrack /nf_conntrack
8130  0.2271  libc-2.23.so __GI_strchr
6357  0.1776  libc-2.23.so _int_free
4152  0.1160  libc-2.23.so re_search_internal
4043  0.1129  libc-2.23.so strlen
2754  0.0769  libstdc++.so.6.0.21 
/usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21
2753  0.0769  libc-2.23.so free
2704  0.0755  ip_tables/ip_tables
2560  0.0715  reiserfs /reiserfs
2332  0.0651  kallsyms ___slab_alloc
2284  0.0638  libc-2.23.so malloc_consolidate
2204  0.0616  libc-2.23.so malloc
2175  0.0608  kallsyms sys_epoll_ctl
2035  0.0568  kallsyms csum_partial_copy_generic
1614  0.0451  libc-2.23.so calloc
1552  0.0434  kallsyms _raw_spin_lock
1208  0.0337  kallsyms memcpy
1203  0.0336  kallsyms nf_iterate
1177  0.0329  kallsyms irq_entries_start
1165  0.0325  kallsyms __fget
1072  0.0299  kallsyms copy_user_generic_string
1037  0.0290  kallsyms __alloc_skb
1002  0.0280  kallsyms tcp_sendmsg
945   0.0264  libc-2.23.so build_upper_buffer
875   0.0244  kallsyms kmem_cache_free
873   0.0244  kallsyms tcp_rack_mark_lost
868   0.0242  nf_nat_ipv4  /nf_nat_ipv4
861   0.0241  kallsyms kfree
837   0.0234  kallsyms __inet_lookup_established
834   0.0233  kallsyms get_partial_node.isra.61
825   0.0230  kallsyms __slab_free
815   0.0228  kallsyms sock_poll
810   0.0226  kallsyms skb_release_data
802   0.0224  nf_conntrack_ipv4/nf_conntrack_ipv4
792   0.0221  kallsyms tcp_transmit_skb
771   0.0215  kallsyms kmem_cache_alloc
719   0.0201  kallsyms fib_table_lookup
704   0.0197  kallsyms _raw_spin_lock_irqsave
701   0.0196  kallsyms tcp_v4_rcv
699   0.0195  libm-2.23.so __ieee754_log_avx
686   0.0192  nf_nat   /nf_nat
684   0.0191  kallsyms tcp_write_xmit
674   0.0188  kallsyms __cmpxchg_double_slab.isra.44
626   0.0175  kallsyms __netif_receive_skb_core
621   0.0173  libnettle.so.6.2
/usr/lib/x86_64-linux-gnu/libnettle.so.6.2
608   0.0170  kallsyms delay_tsc
600   0.0168  kallsyms ksize
595   0.0166  kallsyms tcp_ack
592   0.0165  kallsyms __local_bh_enable_ip




--
View this message in context: 
http://squid-web-proxy-cache.1019090.n4.nabble.com/Squid-cpu-usage-100-from-few-days-ago-tp4678894.html
Sent from the Squid - Users mailing list archive at Nabble.com.
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users