Re: [squid-users] Squid 3.1 and TPROXY 4 Problems
On 5/05/2012 7:58 p.m., Dave Blakey wrote: Hi all, I'm busy working on a tproxy setup with the latest squid on Ubuntu 12.04; tproxy is enabled, squid is compiled with tproxy support etc. The difference with this setup is that traffic is being sent to the host using route-map on a cisco as opposed to WCCP but it seems that should work. Unfortunately it seems there is very little documentation about the latest tproxy+squid3.1 setup method - but this is what I have -- # IP ip -f inet rule add fwmark 1 lookup 100 ip -f inet route add local default dev eth0 table 100 # Sysctl echo 1> /proc/sys/net/ipv4/ip_forward echo 2> /proc/sys/net/ipv4/conf/default/rp_filter echo 2> /proc/sys/net/ipv4/conf/all/rp_filter echo 0> /proc/sys/net/ipv4/conf/eth0/rp_filter # IP Tables iptables -t mangle -N DIVERT iptables -t mangle -A DIVERT -j MARK --set-mark 1 iptables -t mangle -A DIVERT -j ACCEPT iptables -t mangle -A PREROUTING -p tcp -m socket -j DIVERT iptables -t mangle -A PREROUTING -p tcp --dport 80 -j TPROXY --tproxy-mark 0x1/0x1 --on-port 3129 In squid.conf the relevant line for http_port 3129 tproxy is set etc. With this setup I get hits on the iptables rules, and see a request in the access log but it fails to fill it, it also looks very strange -- 1336146295.076 56266 x.x.x.x TCP_MISS/000 0 GET http://www.google.com/url? - DIRECT/www.google.com - 1336146337.969 42875 x.x.x.x TCP_MISS/000 0 GET http://www.google.com/url? - DIRECT/www.google.com - As you can see it's a TCP_MISS/000 and the DIRECT/www.google.com in my experience should have an IP not a hostname? Additionally the sizes seem very weird. The client just hangs. Depends on your squid version, the 3.2+ are IP-only there the older ones display FQDN when its available and log_fqdn is on. Size is zero because upstream was contacted, but things went bad before any bytes were transferred to the client. This is the usual log signature for a forwarding loop. With TPROXY those are a greater risk than with NAT, and harder to track down. You may need to take a very close look at the TCP packets in the different network link places and see what is going on. NP: port number is the only way to identify cleint and server connections apart at the TCP/IP level. Should this setup be working or is there some obvious error? I'm not entirely sure about the rp_filter sysctl. I've had trouble on recent Debian myself with TPROXY hanging. It may be worth experimenting with those a bit. Amos
Re: [squid-users] Squid Restarting
On 4/05/2012 9:59 p.m., Justin Lawler wrote: Hi, We're running squid 3.1.19 - and have seen it restarting from the logs, just after the below error: 2012/04/19 12:12:28| assertion failed: forward.cc:496: "server_fd == fd" 2012/04/19 12:12:59| Starting Squid Cache version 3.1.19 for sparc-sun-solaris2.10... Is this a known issue? any workaround? Seems to be new and a bit strange. Squid opens one connection to the server to fetch content sometime later a connection was closed, but not the one which was opened to begin with. Do you have a core dump or stack trace available to identify what the fd and server_fd values actually were during the crash? It's been in production for 6 weeks now, and have only seen it once, but we need to have an answer for the customer. We're worried it'll be more frequently as traffic goes up. Being the first report over a month after the release, it would seem to be very rare. Amos
Re: [squid-users] Re: external acl code examples
On 6/05/2012 10:55 a.m., Amos Jeffries wrote: On 6/05/2012 8:09 a.m., E.S. Rosenberg wrote: 2012/5/3 Eliezer Croitoru: On 02/05/2012 14:53, E.S. Rosenberg wrote: 2012/5/2 E.S. Rosenberg: Hi, I just thought I'd share the script I have for the squid side, maybe someone finds it useful. I wrote in PHP because I wanted to use prepared statements and am most familiar with PDO. Now my logs have usernames but squid does not allow me to make proxy_auth acls since I have no auth mechanism configured (this particular squid instance is a museum piece - 2.6, soon to be replaced), if this issue also exists in squid 3.1 then how would I control users based on a username returned through an external ACL? Thanks, Eli I stuck the script on my server, that makes an easier read then from inside a mail: http://kotk.nl/verifyIP.phps Hope that helps, Eli i saw your external_acl app and it seems very nice. i wrote another one on ruby that seems almost like that(a mimic for practice). and i was wondering about how do you plan to implement the proxy_auth acls? using AD? some other DB? I am not usre I follow, do you mean how I intend to manage my lists of usernames? In that case I am pushing for the use of LDAP properties, then a script will run every X time, determine whether or not the LDAP database was changed since the last update (based on change timestamps) and generate lists of usernames. Currently we don't have a good way of managing this, I have some sctipts that work based on the location of a user in our organization but that is not always correct. you mentioned something about the network infrastructure\CISCO if i remember right. Yes, the link of IP->username is generated based on the radius logs of the server that provides authentication for the wireless. However as said squid tells me that since I have no auth-mechanism fully setup I can't use proxy_auth lists so I wonder how can I use the username I provided in the external acl in the rest of squid? It is just a label to Squid. Authentication happened outside with no internal state other than teh external_acl_type format key to link it to anything. It can be used in logging with %eo log tag, or passed to other proxies as HTTP auth login with cache_peer login= option. Oops sorry %ue tag, not %eo. Amos
Re: [squid-users] need suggestion for bulk users
On 6/05/2012 9:24 a.m., Muhammad Yousuf Khan wrote: thanks for sharing your experience but what about ISPs who are handling thousands of queries . aren't they using squid? AFAIK my ISP is usring squid and its a biggest ISP in whole country. and some time i am getting the squid error messages. so what would be their strategy ? You have to ask them. Every single network has different usage(s) for Squid. We design it to be a general HTTP proxy, capable of use in any network environment or with any corporate policies. I think its mostly succeeding at meeting that design goal. Amos
Re: [squid-users] Re: external acl code examples
On 6/05/2012 8:09 a.m., E.S. Rosenberg wrote: 2012/5/3 Eliezer Croitoru: On 02/05/2012 14:53, E.S. Rosenberg wrote: 2012/5/2 E.S. Rosenberg: Hi, I just thought I'd share the script I have for the squid side, maybe someone finds it useful. I wrote in PHP because I wanted to use prepared statements and am most familiar with PDO. Now my logs have usernames but squid does not allow me to make proxy_auth acls since I have no auth mechanism configured (this particular squid instance is a museum piece - 2.6, soon to be replaced), if this issue also exists in squid 3.1 then how would I control users based on a username returned through an external ACL? Thanks, Eli I stuck the script on my server, that makes an easier read then from inside a mail: http://kotk.nl/verifyIP.phps Hope that helps, Eli i saw your external_acl app and it seems very nice. i wrote another one on ruby that seems almost like that(a mimic for practice). and i was wondering about how do you plan to implement the proxy_auth acls? using AD? some other DB? I am not usre I follow, do you mean how I intend to manage my lists of usernames? In that case I am pushing for the use of LDAP properties, then a script will run every X time, determine whether or not the LDAP database was changed since the last update (based on change timestamps) and generate lists of usernames. Currently we don't have a good way of managing this, I have some sctipts that work based on the location of a user in our organization but that is not always correct. you mentioned something about the network infrastructure\CISCO if i remember right. Yes, the link of IP->username is generated based on the radius logs of the server that provides authentication for the wireless. However as said squid tells me that since I have no auth-mechanism fully setup I can't use proxy_auth lists so I wonder how can I use the username I provided in the external acl in the rest of squid? It is just a label to Squid. Authentication happened outside with no internal state other than teh external_acl_type format key to link it to anything. It can be used in logging with %eo log tag, or passed to other proxies as HTTP auth login with cache_peer login= option. It does *not* magically insert proxy-auth headers into the request received from the client, or pretend to be such. proxy_auth ACL type tests the proxy-auth headers presented by the client. We do not (yet) have an authentication ACL type that checks those type of credentials. Amos
Re: [squid-users] Strange user name in SQUID log
On 6/05/2012 12:53 a.m., Pavel Bychykhin wrote: Hi! My SQUID version is 3.1.19. Recently I noticed very strange log record (strange user name): 1335604655.033 49 192.168.1.20 TCP_DENIED/407 481 HEAD http://s7.addthis.com/static/r07/sh084.html %ef%bf%af%ef%be%bf%ef%be%90%ef%bf%af%ef%be%be%ef%be%90%ef%bf%af%ef%be%bf%ef%be%90%ef%bf%af%ef%be%be%ef%be%b1%ef%bf%af%ef%be%bf%ef%be%91%ef%bf%af%ef%be%be%ef%be%80%ef%bf%af%ef%be%bf%ef%be%90%ef%bf%af%ef%be%be%ef%be%b0%ef%bf%af%ef%be%bf%ef%be%90%ef%bf%af%ef%be%be%ef%be%ba%ef%bf%af%ef%be%bf%ef%be%90%ef%bf%af%ef%be%be%ef%be%be%ef%bf%af%ef%be%bf%ef%be%90%ef%bf%af%ef%be%be%ef%be%b4%ef%bf%af%ef%be%bf%ef%be%90%ef%bf%af%ef%be%be%ef%be%b0%ef%bf%af%ef%be%bf%ef%be%90%ef%bf%af%ef%be%be%ef%be%b2%ef%bf%af%ef%be%bf%ef%be%91%ef%bf%af%ef%be%be%ef%be%80%ef%bf%af%ef%be%bf%ef%be%90%ef%bf%af%ef%be%be%ef%be%b0 NONE/- text/html All my users have their accounts in plain ASCII. It would not be a big problem (such record occurred only once), but SARG was unable to process this record and does not generate a report. I wonder, is it a correct log record, or it's a bug? This looks like the correct log entry for a mangled (attack?) request. An asian name appears when decoded as Unicode. It was rejected due to incorrect auth credentials by your system. Amos
Re: [squid-users] Connection Reset by Peer (104)
On 5/05/2012 1:53 a.m., Baird, Josh wrote: Hi, Running 2.6.STABLE21-6 (RHEL5) here. I am unable to access http://www.nacuboannualmeeting.org/. The error that is thrown is: The following error was encountered: Read Error The system returned: (104) Connection reset by peer My access.log shows: 04/May/2012:08:49:03 -0500348 172.24.75.138 TCP_MISS/502 1484 GET http://www.nacuboannualmeeting.org/ - DIRECT/64.211.220.113 text/html I have tried the two suggestions in the FAQ: echo 0> /proc/sys/net/ipv4/tcp_ecn echo 0>/proc/sys/net/ipv4/tcp_window_scaling Neither of these solved the problem. Does anyone have any other ideas on how I can solve this? ECN and scaling help with read timeouts. This is apparently the remote end explicit sending TCP close packets. Amos
Re: [squid-users] Re: cache peer communication about HIT/MISS between squid and and non-squid peer
On 2/05/2012 8:44 p.m., x-man wrote: Thanks Amos, what will change if the cache peer is talking ICP or HTCP protocol? Can I use one of this protocols to say if the answer is HIT or MISS, so the main squid who is in intercept mode to the customers, will mark the traffic? ICP and HTCP are use to determine if the sibling typ epeers are useful source for servicing a MISS. That is all. Amos
Re: [squid-users] need suggestion for bulk users
thanks for sharing your experience but what about ISPs who are handling thousands of queries . aren't they using squid? AFAIK my ISP is usring squid and its a biggest ISP in whole country. and some time i am getting the squid error messages. so what would be their strategy ? On Sun, May 6, 2012 at 1:33 AM, E.S. Rosenberg wrote: > 2012/5/5 Muhammad Yousuf Khan : >> right now i am handling 35 users with a squid having 512 MB ram and >> it is on virtual server KVM linux, things are working fine. but for my >> career growth i am looking for a good path to continue with squid. >> like for example if i move to another org. and they have like around >> 300 or 500 users . how come ill manage that. i know 512 configuration >> of squid will not handle that bulk queries. so what is the best >> approach. or do you think this tiny (few GB) 512 MB squid VM gonna >> work with 500 users... however our processor (right now) and >> hardware is strong it is Xeone quade core. 2.6 . >> >> and for me an important question is. how ISP is using Squid what kind >> of infrastructure they have. apart from storage i know ISPs are >> caching youtube and other web contents to lower down their traffic so >> i know they must have good storage system like SAN or NAS but how >> would they cater all the queries where thousands of users are hitting >> just one single box.(may be) >> >> >> >> Thanks, > > Our most heavily used proxy is also our most outdated and oldest > machine at the moment, it is still a hardware machine and not virtual, > it's a dualcore (or dualsocket) Xeon 2.4GHz, has only 1G of RAM, on > average has 500-600 users and handles about 200-300G of traffic per > day. > It's caching performance is less good both due to the much more > diverse nature of the browsing and downloading and the much smaller > cache that it can have since it is only using it's internal harddisks > and not any storage servers. > I think that nicely shows the power of squid. > If you just use it as a device to enable people to get to the Internet > and monitor/block small amounts of traffic even 'weak' machines can > handle a lot. > > HTH, > Eli
Re: [squid-users] Redirector idea using request header
On 05/05/2012 06:31 AM, gslavic wrote: Hi, there, Is there a way to read "Request Header" BEFORE it is writen in access.log ? Example: There are three versions of internet site. One for Simbian phones, one for Android phones, one for regular internet browsers. In request header there is a field: */User-Agent: Mozilla/5.0 (SymbianOS/9.2; U; Series60/3.1 NokiaE71-1/200.21.118; Profile/MIDP-2.0 Configuration/CLDC-1.1 ) AppleWebKit/413 (KHTML, like Gecko) Safari/413/* if the client is using Symbian OS / Nokia E71 phone, and an field ICAP is the perfect solution for this kind of functionality. An ICAP daemon receives all request headers in a REQMOD request and can make decisions (rewrite/redirect) based on those headers. Marcus
Re: [squid-users] need suggestion for bulk users
2012/5/5 Muhammad Yousuf Khan : > right now i am handling 35 users with a squid having 512 MB ram and > it is on virtual server KVM linux, things are working fine. but for my > career growth i am looking for a good path to continue with squid. > like for example if i move to another org. and they have like around > 300 or 500 users . how come ill manage that. i know 512 configuration > of squid will not handle that bulk queries. so what is the best > approach. or do you think this tiny (few GB) 512 MB squid VM gonna > work with 500 users... however our processor (right now) and > hardware is strong it is Xeone quade core. 2.6 . > > and for me an important question is. how ISP is using Squid what kind > of infrastructure they have. apart from storage i know ISPs are > caching youtube and other web contents to lower down their traffic so > i know they must have good storage system like SAN or NAS but how > would they cater all the queries where thousands of users are hitting > just one single box.(may be) > > > > Thanks, Our most heavily used proxy is also our most outdated and oldest machine at the moment, it is still a hardware machine and not virtual, it's a dualcore (or dualsocket) Xeon 2.4GHz, has only 1G of RAM, on average has 500-600 users and handles about 200-300G of traffic per day. It's caching performance is less good both due to the much more diverse nature of the browsing and downloading and the much smaller cache that it can have since it is only using it's internal harddisks and not any storage servers. I think that nicely shows the power of squid. If you just use it as a device to enable people to get to the Internet and monitor/block small amounts of traffic even 'weak' machines can handle a lot. HTH, Eli
Re: [squid-users] Re: external acl code examples
2012/5/3 Eliezer Croitoru : > On 02/05/2012 14:53, E.S. Rosenberg wrote: >> >> 2012/5/2 E.S. Rosenberg: >>> >>> Hi, >>> I just thought I'd share the script I have for the squid side, maybe >>> someone finds it useful. >>> I wrote in PHP because I wanted to use prepared statements and am most >>> familiar with PDO. >>> >>> Now my logs have usernames but squid does not allow me to make >>> proxy_auth acls since I have no auth mechanism configured (this >>> particular squid instance is a museum piece - 2.6, soon to be >>> replaced), if this issue also exists in squid 3.1 then how would I >>> control users based on a username returned through an external ACL? >>> >>> Thanks, >>> Eli >> >> I stuck the script on my server, that makes an easier read then from >> inside a mail: >> http://kotk.nl/verifyIP.phps >> >> Hope that helps, >> Eli >> > i saw your external_acl app and it seems very nice. > i wrote another one on ruby that seems almost like that(a mimic for > practice). > and i was wondering about how do you plan to implement the proxy_auth acls? > using AD? some other DB? I am not usre I follow, do you mean how I intend to manage my lists of usernames? In that case I am pushing for the use of LDAP properties, then a script will run every X time, determine whether or not the LDAP database was changed since the last update (based on change timestamps) and generate lists of usernames. Currently we don't have a good way of managing this, I have some sctipts that work based on the location of a user in our organization but that is not always correct. > you mentioned something about the network infrastructure\CISCO if i remember > right. Yes, the link of IP->username is generated based on the radius logs of the server that provides authentication for the wireless. However as said squid tells me that since I have no auth-mechanism fully setup I can't use proxy_auth lists so I wonder how can I use the username I provided in the external acl in the rest of squid? Thanks, Eli > > Regards, > Eliezer > > > > > >>> 2012/4/10 akadimi: Hi Amos, Could you give me more details on your new session helper as soon as it becomes available. Regards, -- View this message in context: http://squid-web-proxy-cache.1019090.n4.nabble.com/external-acl-code-examples-tp4424505p4546016.html Sent from the Squid - Users mailing list archive at Nabble.com. > > > > -- > Eliezer Croitoru > https://www1.ngtech.co.il > IT consulting for Nonprofit organizations > eliezer ngtech.co.il
Re: [squid-users] anyone knows some info about youtube "range" parameter?
it seems like if a server (apache,nginx) is responding to a "range" request with the full file the player will get it without any problem despite the "range" request. so if you do have a current cache with a lot of files you can still use it. i also got into this nice project: http://code.google.com/p/yt-cache/ it's a Fork of the project http://code.google.com/p/youtube-cache/ and has much more options such as graphs php management menu and some other nice stuff. the only problem is that it from my testing it works good only on debian\ubuntu. i was testing it on gentoo and got some problems running it. it' really a nice project that implements some nice database features. there was another page that was implementing a store_url_rewrite based on the one in squid wiki http://aacable.wordpress.com/2012/01/30/youtube-caching-problem-an-error-occured-please-try-again-later-solved/ i was thinking of adding a url_rewrite (not store_url_rewrite) that will use a database to get statistics done and in a case of a very popular video to cache the whole video instead of chunks and also to add a "last accessed" for the video so the statistics will be relevant. also it seems like if you will add to the uri\url some custom parameters such as found in urls "redirect=1" it wont change anything for yt servers about serving the file that matches the basic parameters. Will update Eliezer -- Eliezer Croitoru https://www1.ngtech.co.il IT consulting for Nonprofit organizations eliezer ngtech.co.il
[squid-users] need suggestion for bulk users
right now i am handling 35 users with a squid having 512 MB ram and it is on virtual server KVM linux, things are working fine. but for my career growth i am looking for a good path to continue with squid. like for example if i move to another org. and they have like around 300 or 500 users . how come ill manage that. i know 512 configuration of squid will not handle that bulk queries. so what is the best approach. or do you think this tiny (few GB) 512 MB squid VM gonna work with 500 users... however our processor (right now) and hardware is strong it is Xeone quade core. 2.6 . and for me an important question is. how ISP is using Squid what kind of infrastructure they have. apart from storage i know ISPs are caching youtube and other web contents to lower down their traffic so i know they must have good storage system like SAN or NAS but how would they cater all the queries where thousands of users are hitting just one single box.(may be) Thanks,
[squid-users] Strange user name in SQUID log
Hi! My SQUID version is 3.1.19. Recently I noticed very strange log record (strange user name): 1335604655.033 49 192.168.1.20 TCP_DENIED/407 481 HEAD http://s7.addthis.com/static/r07/sh084.html %ef%bf%af%ef%be%bf%ef%be%90%ef%bf%af%ef%be%be%ef%be%90%ef%bf%af%ef%be%bf%ef%be%90%ef%bf%af%ef%be%be%ef%be%b1%ef%bf%af%ef%be%bf%ef%be%91%ef%bf%af%ef%be%be%ef%be%80%ef%bf%af%ef%be%bf%ef%be%90%ef%bf%af%ef%be%be%ef%be%b0%ef%bf%af%ef%be%bf%ef%be%90%ef%bf%af%ef%be%be%ef%be%ba%ef%bf%af%ef%be%bf%ef%be%90%ef%bf%af%ef%be%be%ef%be%be%ef%bf%af%ef%be%bf%ef%be%90%ef%bf%af%ef%be%be%ef%be%b4%ef%bf%af%ef%be%bf%ef%be%90%ef%bf%af%ef%be%be%ef%be%b0%ef%bf%af%ef%be%bf%ef%be%90%ef%bf%af%ef%be%be%ef%be%b2%ef%bf%af%ef%be%bf%ef%be%91%ef%bf%af%ef%be%be%ef%be%80%ef%bf%af%ef%be%bf%ef%be%90%ef%bf%af%ef%be%be%ef%be%b0 NONE/- text/html All my users have their accounts in plain ASCII. It would not be a big problem (such record occurred only once), but SARG was unable to process this record and does not generate a report. I wonder, is it a correct log record, or it's a bug? -- Best regards, Pavel
[squid-users] Redirector idea using request header
Hi, there, Is there a way to read "Request Header" BEFORE it is writen in access.log ? Example: There are three versions of internet site. One for Simbian phones, one for Android phones, one for regular internet browsers. In request header there is a field: */User-Agent: Mozilla/5.0 (SymbianOS/9.2; U; Series60/3.1 NokiaE71-1/200.21.118; Profile/MIDP-2.0 Configuration/CLDC-1.1 ) AppleWebKit/413 (KHTML, like Gecko) Safari/413/* if the client is using Symbian OS / Nokia E71 phone, and an field */User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.19 (KHTML, like Gecko) Chrome/18.0.1025.168 Safari/535.19/* if the client is using WIndows 7 x64 / Chrome browser. If (for example) redirector program gets the request header as a input variable (or as a system variable, or has the ability to read it from some log that is written before request is processed by squid) it will have the ability to filter or redirect users to adequate version of the site for the hardware/software combination. Any ideas ? Thanks in advance. Goran -- View this message in context: http://squid-web-proxy-cache.1019090.n4.nabble.com/Redirector-idea-using-request-header-tp4610819.html Sent from the Squid - Users mailing list archive at Nabble.com.
[squid-users] Squid 3.1 and TPROXY 4 Problems
Hi all, I'm busy working on a tproxy setup with the latest squid on Ubuntu 12.04; tproxy is enabled, squid is compiled with tproxy support etc. The difference with this setup is that traffic is being sent to the host using route-map on a cisco as opposed to WCCP but it seems that should work. Unfortunately it seems there is very little documentation about the latest tproxy+squid3.1 setup method - but this is what I have -- # IP ip -f inet rule add fwmark 1 lookup 100 ip -f inet route add local default dev eth0 table 100 # Sysctl echo 1 > /proc/sys/net/ipv4/ip_forward echo 2 > /proc/sys/net/ipv4/conf/default/rp_filter echo 2 > /proc/sys/net/ipv4/conf/all/rp_filter echo 0 > /proc/sys/net/ipv4/conf/eth0/rp_filter # IP Tables iptables -t mangle -N DIVERT iptables -t mangle -A DIVERT -j MARK --set-mark 1 iptables -t mangle -A DIVERT -j ACCEPT iptables -t mangle -A PREROUTING -p tcp -m socket -j DIVERT iptables -t mangle -A PREROUTING -p tcp --dport 80 -j TPROXY --tproxy-mark 0x1/0x1 --on-port 3129 In squid.conf the relevant line for http_port 3129 tproxy is set etc. With this setup I get hits on the iptables rules, and see a request in the access log but it fails to fill it, it also looks very strange -- 1336146295.076 56266 x.x.x.x TCP_MISS/000 0 GET http://www.google.com/url? - DIRECT/www.google.com - 1336146337.969 42875 x.x.x.x TCP_MISS/000 0 GET http://www.google.com/url? - DIRECT/www.google.com - As you can see it's a TCP_MISS/000 and the DIRECT/www.google.com in my experience should have an IP not a hostname? Additionally the sizes seem very weird. The client just hangs. Should this setup be working or is there some obvious error? Thank you in advance Dave
