Re: [squid-users] Malformed HTTP on tproxy squid

[Alex Rousskov]

On 08/17/2016 10:47 AM, Alex Rousskov wrote:
> On 08/17/2016 10:25 AM, Amos Jeffries wrote:
> 
>> I don't think the delayer approach will work because these are parse
>> error/abort responses that don't go near any ACL system.
> 
> If an error response does not go through http_reply_access, then this is
> a Squid bug IMO.

In my primitive test, an error:invalid-request response does go through
the http_reply_access rules, as expected:

2016/08/17 16:10:11.206| 88,2| client_side_reply.cc(2049)
processReplyAccessResult: The reply for NONE error:invalid-request is
ALLOWED

Alex.

___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] Malformed HTTP on tproxy squid

[Omid Kosari]

Amos Jeffries wrote
> About the only thing you could do to speed it up is locate the error
> page templates (file paths: en/ERR_INVALID_REQ and
> templates/ERR_INVALID_REQ) and remove their contents. Then restart Squid.
> That should remove at least a few of the vprintf() syscalls that your
> earlier trace showed as being a significant source of CPU load.

Fine. This resolved the problem .
Thanks


samples  %image name   symbol name
190728   34.3901  squid/usr/sbin/squid
26003 4.6886  r8169/r8169
22958 4.1396  libc-2.23.so _int_malloc
13812 2.4904  nf_conntrack /nf_conntrack
11146 2.0097  libc-2.23.so re_search_internal
11044 1.9913  libc-2.23.so _int_free
8748  1.5774  libstdc++.so.6.0.21 
/usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21
7240  1.3054  reiserfs /reiserfs
6087  1.0975  libc-2.23.so malloc_consolidate
5850  1.0548  libc-2.23.so malloc
4840  0.8727  libc-2.23.so vfprintf
4468  0.8056  ip_tables/ip_tables
4423  0.7975  libm-2.23.so __ieee754_log_avx
4364  0.7869  libc-2.23.so __memcpy_sse2_unaligned
3935  0.7095  kallsyms sys_epoll_ctl
3929  0.7084  libc-2.23.so free
3829  0.6904  libc-2.23.so build_upper_buffer
3562  0.6423  kallsyms __fget
3413  0.6154  kallsyms copy_user_generic_string
3169  0.5714  libc-2.23.so calloc
2815  0.5076  kallsyms delay_tsc
2767  0.4989  kallsyms csum_partial_copy_generic
2739  0.4939  kallsyms tcp_sendmsg
2454  0.4425  kallsyms memcpy
2192  0.3952  libc-2.23.so _wordcopy_fwd_dest_aligned
2139  0.3857  kallsyms _raw_spin_lock_irqsave
2108  0.3801  kallsyms _raw_spin_lock
2075  0.3741  kallsyms nf_iterate
1916  0.3455  libc-2.23.so __memset_sse2
1900  0.3426  [vdso] (tgid:12101 range:0x7fff9fbca000-0x7fff9fbcbfff)
[vdso] (tgid:12101 range:0x7fff9fbca000-0x7fff9fbcbfff
)
1842  0.3321  libc-2.23.so __strcmp_sse2_unaligned
1794  0.3235  kallsyms sock_poll
1753  0.3161  libc-2.23.so strlen
1702  0.3069  kallsyms entry_SYSCALL_64_after_swapgs
1618  0.2917  kallsyms tcp_poll
1611  0.2905  kallsyms irq_entries_start
1593  0.2872  kallsyms ep_send_events_proc
1567  0.2825  kallsyms ___slab_alloc
1539  0.2775  kallsyms __local_bh_enable_ip
1523  0.2746  nf_conntrack_ipv4/nf_conntrack_ipv4
1467  0.2645  libc-2.23.so re_string_reconstruct
1455  0.2624  kallsyms tcp_transmit_skb
1425  0.2569  nf_nat_ipv4  /nf_nat_ipv4
1366  0.2463  kallsyms _raw_spin_lock_bh
1333  0.2404  kallsyms __alloc_skb
1319  0.2378  kallsyms mutex_spin_on_owner.isra.3
1313  0.2367  kallsyms tcp_recvmsg
1307  0.2357  kallsyms tcp_write_xmit
1279  0.2306  kallsyms __fget_light
1266  0.2283  libc-2.23.so __memmove_sse2
1234  0.2225  libnettle.so.6.2
/usr/lib/x86_64-linux-gnu/libnettle.so.6.2
1202  0.2167  kallsyms __inet_lookup_established
1177  0.2122  kallsyms __lock_text_start
1116  0.2012  kallsyms common_file_perm
1080  0.1947  kallsyms tcp_ack
1075  0.1938  kallsyms tcp_clean_rtx_queue
1046  0.1886  kallsyms tcp_v4_rcv





--
View this message in context: 
http://squid-web-proxy-cache.1019090.n4.nabble.com/Malformed-HTTP-on-tproxy-squid-tp4678951p4679009.html
Sent from the Squid - Users mailing list archive at Nabble.com.
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] Malformed HTTP on tproxy squid

[Alex Rousskov]

On 08/17/2016 10:25 AM, Amos Jeffries wrote:

> I don't think the delayer approach will work because these are parse
> error/abort responses that don't go near any ACL system.

If an error response does not go through http_reply_access, then this is
a Squid bug IMO.

Alex.

___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] Malformed HTTP on tproxy squid

[Amos Jeffries]

On 18/08/2016 4:07 a.m., Alex Rousskov wrote:
> On 08/17/2016 09:02 AM, Amos Jeffries wrote:
> 
>> Your Squid is not even getting far enough to apply security rules to the
>> garbage traffic. It is basically just doing: accept() connection,
>> unmangle the NAT/TPROXY details, read(2) some bytes, try to parse - bam
>> generate and send error page, close the TCP connection and log the event.
> 
> *If* just a few clients doing the above can have a serious effect on
> overall performance of a Squid instance running on decent hardware, then
> we need to fix or optimize something. There is little Squid can do
> against a powerful DDoS, but a few broken clients rarely mimic that.
> 

I'm not convinced that the few req/sec is really that small. It could be
5 proper HTTP req/sec plus some hundreds of attempts to connect with the
non-HTTP transactions.
The latter wont show up in the mg:info report or SNMP req/sec stats (for
HTTP requests/sec), but will only appear in the mgr:utilization report
syscalls.sock.accepts counters and (maybe) the access.log.

Omar: can you clarify how you are identifying the req/sec rates?

> 
>> About the only thing you could do to speed it up is locate the error
>> page templates and remove their contents.
> 
> Also, *if* the clients do not open new connections until their old
> connections are closed, then you may be able to slow them down
> considerably by delaying those error responses. It may be possible to do
> that with an external ACL helper (that delays responses) and
> http_reply_access rules that target those specific error pages.
> 
> 
> Disclaimer: I am not implying that the two conditions marked with "*If*"
> above are true. I have not checked them.

I don't think the delayer approach will work because these are parse
error/abort responses that don't go near any ACL system.

Amos

___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] Malformed HTTP on tproxy squid

[Alex Rousskov]

On 08/17/2016 09:02 AM, Amos Jeffries wrote:

> Your Squid is not even getting far enough to apply security rules to the
> garbage traffic. It is basically just doing: accept() connection,
> unmangle the NAT/TPROXY details, read(2) some bytes, try to parse - bam
> generate and send error page, close the TCP connection and log the event.

*If* just a few clients doing the above can have a serious effect on
overall performance of a Squid instance running on decent hardware, then
we need to fix or optimize something. There is little Squid can do
against a powerful DDoS, but a few broken clients rarely mimic that.


> About the only thing you could do to speed it up is locate the error
> page templates and remove their contents.

Also, *if* the clients do not open new connections until their old
connections are closed, then you may be able to slow them down
considerably by delaying those error responses. It may be possible to do
that with an external ACL helper (that delays responses) and
http_reply_access rules that target those specific error pages.


Disclaimer: I am not implying that the two conditions marked with "*If*"
above are true. I have not checked them.

Alex.

___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] Malformed HTTP on tproxy squid

[Amos Jeffries]

On 17/08/2016 9:26 p.m., Omid Kosari wrote:
> Hi Eliezer,
> 
> 
> Eliezer Croitoru-2 wrote
>> If you know what domain or ip address causes and issue the first thing I
>> can think about is bypassing the malicious traffic to allow other
>> clients\users to reach the Internet.
> 
> Source ip may be 70% of our customers because it is a popular device so it
> is not an option . Destination ip or domains are too much . 
> 
> Unfortunately because the requests are not normal http , so squid log does
> not have the dst url/domain/ip so it is hard job to find them .
> 1- First i should keep looking the squid access.log to find client which has
> such request . 
> 2-Then try to sniff that client from router. 
> 3-Separate normal requests from malformed . 
> 4-Find the destination from malformed requests.
> 5-Put that ip in router acl to exclude from tproxy routing to squid .
> 
> Nobody knows how many times this loop should be repeated because nobody
> knows count of destinations .
> 

Easier way:

 logformat Xips %ts.%03tu %6tr %>a %>la %>ru
 access_log stdio:/var/log/squid/xact.log Xips

Then just grep the xact.log file for the "error:invalid-request" URLs,
and see what the '>la' column IP address is.

If you want to automate it make a logging daemon script.

> 
> Eliezer Croitoru-2 wrote
>> And since squid is also being used as a http ACL enforcement tool
>> malformed requests basically should be dropped and not bypassed
>> automatically.
> 
> So then squid should be able to simply drop them.
> Even it would be fine to have some patterns in iptables or something like
> mod_security for apache etc which introduce by squid gurus to prevent these
> kinds of problems .


Your Squid is not even getting far enough to apply security rules to the
garbage traffic. It is basically just doing: accept() connection,
unmangle the NAT/TPROXY details, read(2) some bytes, try to parse - bam
generate and send error page, close the TCP connection and log the event.


About the only thing you could do to speed it up is locate the error
page templates (file paths: en/ERR_INVALID_REQ and
templates/ERR_INVALID_REQ) and remove their contents. Then restart Squid.
That should remove at least a few of the vprintf() syscalls that your
earlier trace showed as being a significant source of CPU load.

Amos

___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] Malformed HTTP on tproxy squid

[Omid Kosari]

Hi Eliezer,


Eliezer Croitoru-2 wrote
> If you know what domain or ip address causes and issue the first thing I
> can think about is bypassing the malicious traffic to allow other
> clients\users to reach the Internet.

Source ip may be 70% of our customers because it is a popular device so it
is not an option . Destination ip or domains are too much . 

Unfortunately because the requests are not normal http , so squid log does
not have the dst url/domain/ip so it is hard job to find them .
1- First i should keep looking the squid access.log to find client which has
such request . 
2-Then try to sniff that client from router. 
3-Separate normal requests from malformed . 
4-Find the destination from malformed requests.
5-Put that ip in router acl to exclude from tproxy routing to squid .

Nobody knows how many times this loop should be repeated because nobody
knows count of destinations .



Eliezer Croitoru-2 wrote
> And since squid is also being used as a http ACL enforcement tool
> malformed requests basically should be dropped and not bypassed
> automatically.

So then squid should be able to simply drop them.
Even it would be fine to have some patterns in iptables or something like
mod_security for apache etc which introduce by squid gurus to prevent these
kinds of problems .




--
View this message in context: 
http://squid-web-proxy-cache.1019090.n4.nabble.com/Malformed-HTTP-on-tproxy-squid-tp4678951p4678966.html
Sent from the Squid - Users mailing list archive at Nabble.com.
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] Malformed HTTP on tproxy squid

[Eliezer Croitoru]

Hey Omid,

If you know what domain or ip address causes and issue the first thing I can 
think about is bypassing the malicious traffic to allow other clients\users to 
reach the Internet.
Depends on the client and the destination you can choose the right approach.
And since squid is also being used as a http ACL enforcement tool malformed 
requests basically should be dropped and not bypassed automatically.

Eliezer


Eliezer Croitoru
Linux System Administrator
Mobile: +972-5-28704261
Email: elie...@ngtech.co.il


-Original Message-
From: squid-users [mailto:squid-users-boun...@lists.squid-cache.org] On Behalf 
Of Omid Kosari
Sent: Tuesday, August 16, 2016 1:23 PM
To: squid-users@lists.squid-cache.org
Subject: [squid-users] Malformed HTTP on tproxy squid

According to my other post
http://squid-web-proxy-cache.1019090.n4.nabble.com/Squid-cpu-usage-100-from-few-days-ago-td4678894.html


Squid cpu usage becomes 100% when it forwatds some kind of malformed http 
traffic .
Even one ip address with less than 5 requests per second can grow squid cpu 
usage up to 30% 

We have found that this request belongs to a cheap popular satellite receiver 
www.starmax.co . Maybe it has been infected and becomes zombie of a btnet .

Apart from the client type , my question is 

Shouldn't squid have a mechanism to defend this types of problems ? Isn't 
possible for squid to simply ignore malformed http requests ?

Is there any workaround to prevent this problem ?




Squid is in tproxy mode with routing

Ubuntu Linux 16.04 , 4.4.0-34-generic on x86_64 Squid Cache: Version 3.5.19 
from debian repository


samples  %image name   symbol name
1532894  42.8190  libc-2.23.so _IO_strn_overflow
1028537  28.7306  libc-2.23.so _IO_default_xsputn
662802   18.5143  libc-2.23.so vfprintf
77019 2.1514  squid/usr/sbin/squid
28861 0.8062  libc-2.23.so __memset_sse2
26948 0.7528  r8169/r8169
25320 0.7073  libc-2.23.so __memcpy_sse2_unaligned
21712 0.6065  libc-2.23.so __GI___mempcpy
14918 0.4167  libc-2.23.so _int_malloc
8889  0.2483  nf_conntrack /nf_conntrack
8130  0.2271  libc-2.23.so __GI_strchr
6357  0.1776  libc-2.23.so _int_free
4152  0.1160  libc-2.23.so re_search_internal
4043  0.1129  libc-2.23.so strlen
2754  0.0769  libstdc++.so.6.0.21 
/usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21
2753  0.0769  libc-2.23.so free
2704  0.0755  ip_tables/ip_tables
2560  0.0715  reiserfs /reiserfs
2332  0.0651  kallsyms ___slab_alloc
2284  0.0638  libc-2.23.so malloc_consolidate
2204  0.0616  libc-2.23.so malloc
2175  0.0608  kallsyms sys_epoll_ctl
2035  0.0568  kallsyms csum_partial_copy_generic
1614  0.0451  libc-2.23.so calloc
1552  0.0434  kallsyms _raw_spin_lock
1208  0.0337  kallsyms memcpy
1203  0.0336  kallsyms nf_iterate
1177  0.0329  kallsyms irq_entries_start
1165  0.0325  kallsyms __fget
1072  0.0299  kallsyms copy_user_generic_string
1037  0.0290  kallsyms __alloc_skb
1002  0.0280  kallsyms tcp_sendmsg
945   0.0264  libc-2.23.so build_upper_buffer
875   0.0244  kallsyms kmem_cache_free
873   0.0244  kallsyms tcp_rack_mark_lost
868   0.0242  nf_nat_ipv4  /nf_nat_ipv4
861   0.0241  kallsyms kfree
837   0.0234  kallsyms __inet_lookup_established
834   0.0233  kallsyms get_partial_node.isra.61
825   0.0230  kallsyms __slab_free
815   0.0228  kallsyms sock_poll
810   0.0226  kallsyms skb_release_data
802   0.0224  nf_conntrack_ipv4/nf_conntrack_ipv4
792   0.0221  kallsyms tcp_transmit_skb
771   0.0215  kallsyms kmem_cache_alloc
719   0.0201  kallsyms fib_table_lookup
704   0.0197  kallsyms _raw_spin_lock_irqsave
701   0.0196  kallsyms tcp_v4_rcv
699   0.0195  libm-2.23.so __ieee754_log_avx
686   0.0192  nf_nat   /nf_nat
684   0.0191  kallsyms tcp_write_xmit
674   0.0188  kallsyms __cmpxchg_double_slab.isra.44
626   0.0175  kallsyms __netif_receive_skb_core
621   0.0173  libnettle.so.6.2
/usr/lib/x86_64-linux-gnu/libnettle.so.6.2
608   0.0170  kallsyms delay_tsc
600   0.0168  kallsyms ksize
595   0.0166  kallsyms

Re: [squid-users] Malformed HTTP on tproxy squid

[Omid Kosari]

Squid access.log and wireshark PCAP attached
access_(1).log

  
dump2.pcap
  



--
View this message in context: 
http://squid-web-proxy-cache.1019090.n4.nabble.com/Malformed-HTTP-on-tproxy-squid-tp4678951p4678952.html
Sent from the Squid - Users mailing list archive at Nabble.com.
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users