[squid-users] How to see client IP behind a reverse proxy
I have a web application server that needs to be able to see the client request IP address. But when I add a Squid as a reverse proxy in front of my web server, my application only sees the IP of the reverse proxy. Is there any way to configure Squid to expose client IP's to my web app? Regards Roland Rabben Scala Inc.
[squid-users] Re: Forced caching of some content in acceleration mode
Thanks to adri on the IRC channel I was pointed to header_access and I have got it working. Best regards Michael Boman On 10/4/07, Michael Boman [EMAIL PROTECTED] wrote: Hi list, I am working on a Microsoft SharePoint/IIS setup that doesn't want to provide the correct caching headers. This is a deliberate design feature from the people in Redmond, but it is not what we want to have. Due to the fact images are handled by SharePoint they leave the IIS server with caching set to private, with a max-age of 0. The thing is that these images are static, and I would much rather have them cached by any upstream proxy and browser to limit the amount of traffic goes to the server. My idea is to put a squid proxy in front of the IIS/SharePoint server to re-write the caching header to what I deem to be acceptable (although I know I am breaking the HTTP standard by doing so). This is one of the configuration lines (many like this, one for each file extension I want to force caching for): refresh_pattern .jpg 14400 50% 18000 override-expire override-lastmod reload-into-ims ignore-reload ignore-no-cache ignore-private ignore-auth By right that one should cache all .jpg requests for 4-5 hours, regardless what the originating web server says. However, when I use Fiddler (a pretty cool HTTP debugging proxy) I still see my .jpg image responses as private with a max-age set to 0. I am running 2.6stable16. Could someone please enlighten me what seems to be wrong? Full config can be found at http://michaelboman.org/wiki/index.php?title=Squid Best regards Michael Boman -- IT Security Researcher Developer http://michaelboman.org -- IT Security Researcher Developer http://michaelboman.org
[squid-users] Cache.log parameter descriptions (queue overload)
Hi All,
I am beginner to squid and I have implemented
transparent proxy using squid verion
squid-2.6.STABLE12 in bridge mode in RHEL AS release
4 (Nahant) with 4 GB of RAM and 3 Xeon Processor. The
squid handles 100s of request/minute. The squid has
been working properly but lately I have been getting
following error.
dnsSubmit: queue overload, rejecting www.orkut.com
Even after increasing the dns_children to 400. I am
still getting the queue overload error. After enabling
debug mode using squid -k debug command. I got the
following errors in the logs.
ipcacheParse: parsing: {$fail Temporary network
problem, please retry later}
cbdataValid: 0x35b94900
commConnectDnsHandle: Unknown host: www.orkut.com
There is no load in the server or high latency in the
network and when manually trying to resolve the site
using dig command the site is getting resolved in 2-5
ms.
Is there any bug in the squid version? Also please let
us know where I am find description for parameter like
commCallCloseHandlers,
commSetTimeout,commConnectFree,cbdataFree,cbdataValid
etc so that I can better analyze the problem from my
end. Please adivce on this issue. More detailed log
is given below.
2007/10/04 12:14:19| eventRun: Running
'fwdConnectStart', id 51043
2007/10/04 12:14:19| fwdConnectStart:
http://www.orkut.com/Scrapbook.aspx
2007/10/04 12:14:19| pconnRemoveFD: found FD 1141 at
index 1
2007/10/04 12:14:19| commSetSelect: FD 1141 type 1
2007/10/04 12:14:19| commSetEvents(fd=1141)
2007/10/04 12:14:19| commSetTimeout: FD 1141 timeout
-1
2007/10/04 12:14:19| comm_close: FD 1141
2007/10/04 12:14:19| commCallCloseHandlers: FD 1141
2007/10/04 12:14:19| fd_close FD 1141 www.orkut.com
idle connection
2007/10/04 12:14:19| commSetEvents(fd=1141)
2007/10/04 12:14:19| aclMatchAclList: returning 1
2007/10/04 12:14:19| fwdConnectStart: got addr
221.134.210.253, tos 0
2007/10/04 12:14:19| comm_open: FD 1141 is a new
socket
2007/10/04 12:14:19| fd_open FD 1141
http://www.orkut.com/Scrapbook.aspx
2007/10/04 12:14:19| comm_add_close_handler: FD 1141,
handler=0x807a164, data=0x836f7770
2007/10/04 12:14:19| cbdataLock: 0x836f7770
2007/10/04 12:14:19| commSetTimeout: FD 1141 timeout
30
2007/10/04 12:14:19| commConnectStart: FD 1141,
www.orkut.com:80
2007/10/04 12:14:19| cbdataLock: 0x836f7770
2007/10/04 12:14:19| comm_add_close_handler: FD 1141,
handler=0x80704d3, data=0x35b94900
2007/10/04 12:14:19| cbdataLock: 0x35b94900
2007/10/04 12:14:19| ipcache_nbgethostbyname: Name
'www.orkut.com'.
2007/10/04 12:14:19| ipcache_nbgethostbyname: MISS for
'www.orkut.com'
2007/10/04 12:14:19| cbdataLock: 0x35b94900
2007/10/04 12:14:19| dnsSubmit: queue overload,
rejecting www.orkut.com
2007/10/04 12:14:19| cbdataFree: 0x81c021a0
2007/10/04 12:14:19| cbdataFree: Freeing 0x81c021a0
2007/10/04 12:14:19| ipcacheParse: parsing: {$fail
Temporary network problem, please retry later}
2007/10/04 12:14:19| cbdataValid: 0x35b94900
2007/10/04 12:14:19| commConnectDnsHandle: Unknown
host: www.orkut.com
2007/10/04 12:14:19| comm_remove_close_handler: FD
1141, handler=0x80704d3, data=0x35b94900
2007/10/04 12:14:19| cbdataUnlock: 0x35b94900
2007/10/04 12:14:19| commSetTimeout: FD 1141 timeout
-1
2007/10/04 12:14:19| commConnectFree: FD 1141
2007/10/04 12:14:19| cbdataFree: 0x35b94900
2007/10/04 12:14:19| cbdataFree: 0x35b94900 has 1
locks, not freeing
2007/10/04 12:14:19| cbdataValid: 0x836f7770
2007/10/04 12:14:19| fwdConnectDone: Unknown host:
www.orkut.com
2007/10/04 12:14:19| fwdFail: ERR_DNS_FAIL Service
Unavailable
2007/10/04 12:14:19| cbdataFree: 0x32be2e90
2007/10/04 12:14:19| cbdataFree: Freeing 0x32be2e90
2007/10/04 12:14:19| comm_close: FD 1141
2007/10/04 12:14:19| commCallCloseHandlers: FD 1141
2007/10/04 12:14:19| commCallCloseHandlers:
ch-handler=0x807a164
2007/10/04 12:14:19| cbdataValid: 0x836f7770
2007/10/04 12:14:19| fwdServerClosed: FD 1141
http://www.orkut.com/Scrapbook.aspx
2007/10/04 12:14:19| fwdStateFree: 0x836f7770
2007/10/04 12:14:19| storeLockObject: key
'F044E184E2686400C3F899C13FC5733E' count=4
2007/10/04 12:14:19| creating rep: 0xa60609d0
2007/10/04 12:14:19| init-ing hdr: 0xa6060a10 owner: 2
2007/10/04 12:14:19| 0xa6060a10 lookup for 41
2007/10/04 12:14:19| 0xa6060a10 lookup for 9
2007/10/04 12:14:19| 0xa6060a10 lookup for 41
2007/10/04 12:14:19| 0xa6060a10 lookup for 9
2007/10/04 12:14:19| 0xa6060a10 lookup for 25
2007/10/04 12:14:19| errorConvert: %U --
'http://www.orkut.com/Scrapbook.aspx'
2007/10/04 12:14:19| errorConvert: %U --
'http://www.orkut.com/Scrapbook.aspx'
2007/10/04 12:14:19| errorConvert: %H --
'www.orkut.com'
2007/10/04 12:14:19| errorConvert: %z -- 'Temporary
network problem, please retry later'
2007/10/04 12:14:19| errorConvert: %w -- 'webmaster'
2007/10/04 12:14:19| errorConvert: %w -- 'webmaster'
2007/10/04 12:14:19| errorConvert: %T -- 'Thu, 04 Oct
2007 06:44:19 GMT'
2007/10/04 12:14:19| errorConvert: %h -- 'T-CACHE2'
2007/10/04 12:14:19| errorConvert: %s --
'squid/2.6.STABLE12'
2007/10/04 12:14:19| errorConvert:
Re: [squid-users] How to see client IP behind a reverse proxy
On 04.10.07 10:16, Roland Rabben wrote: I have a web application server that needs to be able to see the client request IP address. But when I add a Squid as a reverse proxy in front of my web server, my application only sees the IP of the reverse proxy. Is there any way to configure Squid to expose client IP's to my web app? use X-Forwarded-For: directive, it may contain a path of clients behing the proxies. However you should noe trust every proxy in the path, only yours. -- Matus UHLAR - fantomas, [EMAIL PROTECTED] ; http://www.fantomas.sk/ Warning: I wish NOT to receive e-mail advertising to this address. Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu. They say when you play that M$ CD backward you can hear satanic messages. That's nothing. If you play it forward it will install Windows.
RE: [squid-users] Squid 2.6-STABLE16 problems accessing HTTPS site with certificate
That's true, it's just a normal HTTP and HTTPS proxy I have managed a workaround by forcing connections to the problem site through a different Squid proxy which works fine I'm just a little confused on what could be causing the issue The fact the same Squid version and binaries is used on both systems but one has a newer version of the OS, makes me think it's OS Rob -Original Message- From: Amos Jeffries [mailto:[EMAIL PROTECTED] Sent: 04 October 2007 02:30 To: Henrik Nordstrom Cc: Amos Jeffries; Robert French; squid-users@squid-cache.org Subject: Re: [squid-users] Squid 2.6-STABLE16 problems accessing HTTPS site with certificate On ons, 2007-10-03 at 15:57 +1300, Amos Jeffries wrote: The visible hostname must match the one inside the certificate or the certificate will be seen as invalid. hostname is set either in the OS configuration /etc/hostname, or overridden in squid.conf with visible_hostname. Hmm.. thought this was a normal proxy setup, not an https:// accelerator / reverse-proxy. Doh!, just got twisted around by some SSL-accel tests here while reading those email. Amos Registered in England Wales under number 4586709 Renesas Technology Europe Ltd Dukes Meadow Millboard Road, Bourne End Buckinghamshire SL8 5FH UK
WG: [squid-users] COSS still under developement?
We use coss on our production proxies with squid-2.6. It has got a really big performance. Switching to aufs wouldn't be an alternative, so we have to wait vor 3.1 Regards Martin Stolle -Ursprüngliche Nachricht- Von: Henrik Nordstrom [mailto:[EMAIL PROTECTED] Gesendet: Donnerstag, 4. Oktober 2007 03:29 An: leongmzlist Cc: Squid Betreff: Re: [squid-users] COSS still under developement? On ons, 2007-10-03 at 17:19 -0700, leongmzlist wrote: I remember reading somewhere coss will be removed from the 3.0 release, just want to make sure it's won't get abandoned since our cache data is really important to us. It's not removed, but downgraded to experimental for 3.0, with updated COSS support scheduled for 3.1 if there is sufficient interest in having the improvements forwardported. There was talk about removing COSS from the 3.0 release as it's known to be unstable and not suitable for production use in squid-3, but it was then decided to not fork 3.0 from the 3.x tree before release and COSS stayed. There is no plans on removing COSS from Squid as such, only the 3.0 release as there has not been any developer resources available to update the Squid-3 COSS implementation and it's far behind Sqiud-2 in that area. Regards Henrik signature.asc Description: PGP signature
Re: [squid-users] Creating multiple delay pools
Norman Noah wrote: hi can anybody help me .. i want to create another delay pool from these 2 class my current config is # delay pool delay_pools 2 #direct bandwitdhfull access to websites delay_class 1 2 delay_parameters 1 -1/-1 -1/-1 delay_access 1 allow fast delay_access 1 deny all # delay #restrict bandwidth for all others delay_class 2 3 delay_parameters 2 -1/-1 3276800/8192000 24576/655360 # 400kbs/1000kb3kbs/80kb delay_access 2 allow all morning delay_access 2 allow all afternoon i want to set another delay pool for another segment for wireless segment can anybody help me ? The usage of all those parameters can be found in http://www.squid-cache.org/Versions/v2/2.6/cfgman/ At the very least you will need to increment the number in delay_pools to 3 (the number of pools now used), and add the new pool with all the tags followed by 3. Amos
Re: [squid-users] Cache.log parameter descriptions (queue overload)
On Thu, Oct 04, 2007, Deepak Das wrote: Hi All, I am beginner to squid and I have implemented transparent proxy using squid verion squid-2.6.STABLE12 in bridge mode in RHEL AS release 4 (Nahant) with 4 GB of RAM and 3 Xeon Processor. The squid handles 100s of request/minute. The squid has been working properly but lately I have been getting following error. dnsSubmit: queue overload, rejecting www.orkut.com This is part of the external DNS helper setup. Is there any reason you're not using the internal DNS code in Squid? Adrian -- - Xenion - http://www.xenion.com.au/ - VPS Hosting - Commercial Squid Support - - $25/pm entry-level bandwidth-capped VPSes available in WA -
[squid-users] Squid 2.6 STABLE 16 also crashing
Hi Amos, I have upgraded to Squid 2.6 STABLE 16 And it is still crashing. See the log extracts just before it crashes: (squid)(death+0x3d)[0x80bb5a9] /lib/i686/libpthread.so.0[0x4005747e] (squid)[0x42028c48] (squid)[0x420c12db] (squid)[0x420bd350] (squid)(regexec+0x65)[0x420c2df5] (squid)(refreshLimits+0x27)[0x80a6d4b] (squid)[0x808e9ef] (squid)[0x808f1a4] (squid)[0x809005e] (squid)[0x807857e] (squid)[0x8078bac] (squid)(comm_select+0xb4)[0x807887c] (squid)(main+0x2a1)[0x809e0f9] (squid)(__libc_start_main+0xa4)[0x420158d4] (squid)(shmat+0x51)[0x8057419] FATAL: Received Segment Violation...dying. (squid)(death+0x3d)[0x80bb5a9] /lib/i686/libpthread.so.0[0x4005747e] (squid)[0x42028c48] (squid)[0x420c12db] (squid)[0x420bd350] (squid)(regexec+0x65)[0x420c2df5] (squid)(refreshLimits+0x27)[0x80a6d4b] (squid)(getMaxAge+0x24)[0x80a76f8] (squid)(httpBuildRequestHeader+0x52e)[0x809094a] (squid)(httpBuildRequestPrefix+0x52)[0x8091096] (squid)[0x80911ee] (squid)[0x80764d9] (squid)[0x807846c] (squid)[0x8078bac] (squid)(comm_select+0xb4)[0x807887c] (squid)(main+0x2a1)[0x809e0f9] (squid)(__libc_start_main+0xa4)[0x420158d4] (squid)(shmat+0x51)[0x8057419] FATAL: Received Segment Violation...dying. 2007/10/04 14:51:23| Not currently OK to rewrite swap log. 2007/10/04 14:51:23| storeDirWriteCleanLogs: Operation aborted. From: Amos Jeffries [EMAIL PROTECTED] To: Ali resting [EMAIL PROTECTED] CC: squid-users@squid-cache.org Subject: Re: [squid-users] Squid 2.5-STABLE14 Crashing Date: Wed, 3 Oct 2007 10:29:08 +1300 (NZDT) Hi, For the last couple of days my squid server keeps crashing and restarting itself. I have looked at the cache.log file and this is what I get. This server has been running fine for the last 2 years: First, check your logs are being rotated properly and haven't taken up all disk space. And that the system has not run out of inodes. Second, upgrade to a currently supported version of squid, 2.6s16+ or 3.0rc1 (squid)[0x80a1afd] /lib/i686/libpthread.so.0[0x4005747e] (squid)[0x42028c48] (squid)[0x420c12db] (squid)[0x420bd350] (squid)(regexec+0x65)[0x420c2df5] (squid)(vfprintf+0x2d36)[0x804d21a] (squid)(vfprintf+0x39a9)[0x804de8d] (squid)(vfprintf+0x3d4b)[0x804e22f] That does not look like cache.log content. if it did come from there its seriously screwed. Amos _ Express yourself instantly with MSN Messenger! Download today it's FREE! http://messenger.msn.click-url.com/go/onm00200471ave/direct/01/
Re: [squid-users] Squid 2.6 STABLE 16 also crashing
Hi Mark, This problem only started in the last couple of days. I had been running squid on the same mchine for years. I have even tried delete the swap directories hoping it would help, but still no joy. Any suggestions, is it hardware related? Regards, Ali From: Marc Kool [EMAIL PROTECTED] To: Ali resting [EMAIL PROTECTED] Subject: Re: [squid-users] Squid 2.6 STABLE 16 also crashing Date: Thu, 4 Oct 2007 16:08:18 +0200 Ali, I see that regexec and pthreads are being used. If you have a glibc before 2.4 then you need to upgrade glibc or configure squid not to use pthreads. -Marcus On Thu, Oct 04, 2007 at 03:14:15PM +0200, Ali resting wrote: Hi Amos, I have upgraded to Squid 2.6 STABLE 16 And it is still crashing. See the log extracts just before it crashes: (squid)(death+0x3d)[0x80bb5a9] /lib/i686/libpthread.so.0[0x4005747e] (squid)[0x42028c48] (squid)[0x420c12db] (squid)[0x420bd350] (squid)(regexec+0x65)[0x420c2df5] (squid)(refreshLimits+0x27)[0x80a6d4b] (squid)[0x808e9ef] (squid)[0x808f1a4] (squid)[0x809005e] (squid)[0x807857e] (squid)[0x8078bac] (squid)(comm_select+0xb4)[0x807887c] (squid)(main+0x2a1)[0x809e0f9] (squid)(__libc_start_main+0xa4)[0x420158d4] (squid)(shmat+0x51)[0x8057419] FATAL: Received Segment Violation...dying. (squid)(death+0x3d)[0x80bb5a9] /lib/i686/libpthread.so.0[0x4005747e] (squid)[0x42028c48] (squid)[0x420c12db] (squid)[0x420bd350] (squid)(regexec+0x65)[0x420c2df5] (squid)(refreshLimits+0x27)[0x80a6d4b] (squid)(getMaxAge+0x24)[0x80a76f8] (squid)(httpBuildRequestHeader+0x52e)[0x809094a] (squid)(httpBuildRequestPrefix+0x52)[0x8091096] (squid)[0x80911ee] (squid)[0x80764d9] (squid)[0x807846c] (squid)[0x8078bac] (squid)(comm_select+0xb4)[0x807887c] (squid)(main+0x2a1)[0x809e0f9] (squid)(__libc_start_main+0xa4)[0x420158d4] (squid)(shmat+0x51)[0x8057419] FATAL: Received Segment Violation...dying. 2007/10/04 14:51:23| Not currently OK to rewrite swap log. 2007/10/04 14:51:23| storeDirWriteCleanLogs: Operation aborted. From: Amos Jeffries [EMAIL PROTECTED] To: Ali resting [EMAIL PROTECTED] CC: squid-users@squid-cache.org Subject: Re: [squid-users] Squid 2.5-STABLE14 Crashing Date: Wed, 3 Oct 2007 10:29:08 +1300 (NZDT) Hi, For the last couple of days my squid server keeps crashing and restarting itself. I have looked at the cache.log file and this is what I get. This server has been running fine for the last 2 years: First, check your logs are being rotated properly and haven't taken up all disk space. And that the system has not run out of inodes. Second, upgrade to a currently supported version of squid, 2.6s16+ or 3.0rc1 (squid)[0x80a1afd] /lib/i686/libpthread.so.0[0x4005747e] (squid)[0x42028c48] (squid)[0x420c12db] (squid)[0x420bd350] (squid)(regexec+0x65)[0x420c2df5] (squid)(vfprintf+0x2d36)[0x804d21a] (squid)(vfprintf+0x39a9)[0x804de8d] (squid)(vfprintf+0x3d4b)[0x804e22f] That does not look like cache.log content. if it did come from there its seriously screwed. Amos _ Express yourself instantly with MSN Messenger! Download today it's FREE! http://messenger.msn.click-url.com/go/onm00200471ave/direct/01/ -- _ Express yourself instantly with MSN Messenger! Download today it's FREE! http://messenger.msn.click-url.com/go/onm00200471ave/direct/01/
[squid-users] Squid 2.6 - access hosts outside LAN through proxy with https://a.b.c.d:8080
How would I define the correct ACL and/or http_access rule to access external hosts, that are to be reached through a https- admin interface that is using port 8080 ? I tried to add 8080 to the list of SSL-Ports like acl SSL_ports 443 563 8080 and thus allow it to CONNECT directly with http_access deny CONNECT !SSL_ports (rest is kept to the recommended defaults) ... but Squid keeps on telling me that the connection is refused (111) Regards, -- Mit freundlichen Grüßen, Philipp Rusch
[squid-users] Squid can't login to Stellent CMS
I've got Squid (squid-2.6.STABLE13-1.RHEL4) running on Red Hat EL 4, pointing to a Solaris based Stellent CMS instance and it's working, but when someone tries to login to the Stellent content contol page, they get the following error page in their browser: _ ERROR The requested URL could not be retrieved While trying to retrieve the URL: http://someserver.com:3128/ecmweb/idcplg?IdcService=LOGINAction=GetTemp latePagePage=HOME_PAGEAuth=Internet The following error was encountered: * Access Denied. Access control configuration prevents your request from being allowed at this time. Please contact your service provider if you feel this is incorrect. Your cache administrator is root. Generated Thu, 04 Oct 2007 15:51:41 GMT by someserver.com (squid/2.6.STABLE13) I have very simple ACLs in place until I get everything working, I've included the other lines so you can see the Ips in place: http_port 192.168.120.225:3128 transparent cache_peer 192.168.210.54 parent 80 0 no-query originserver login=PASS acl all src 0.0.0.0/0.0.0.0 http_access allow all icp_access allow all # note, it failed with the following 3 lines in, and out of the config #acl METHODLIST method post #no_cache deny METHODLIST #strip_query_terms off Meanwhile squid logfiles show the following: == store.log == 1191517193.010 RELEASE -1 2131D03783AB4F076206943690EE6700 403 1191517193 0 1191517193 text/html 1280/1280 GET http://someserver.com:3128/ecmweb/idcplg?IdcService=LOGINAction=GetTemp latePagePage=HOME_PAGEAuth=Internet 1191517193.011 RELEASE -1 A9DF9CDAA1351F6EA2F1C92228F23202 403 1191517193-1 1191517193 text/html 1280/1280 GET http://someserver.com:3128/ecmweb/idcplg?IdcService=LOGINAction=GetTemp latePagePage=HOME_PAGEAuth=Internet == cache.log == 2007/10/04 10:51:41| WARNING: Forwarding loop detected for: Client: 192.168.120.225 http_port: 192.168.120.225:3128 GET http://someserver.com:3128/ecmweb/idcplg?IdcService=LOGINAction=GetTemp latePagePage=HOME_PAGEAuth=Internet HTTP/1.0 Host: someserver.com:3128 User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322; .NET CLR 2.0.50727) Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plai n;q=0.8,image/png,*/*;q=0.5 Accept-Language: en-us,en;q=0.5 Accept-Encoding: gzip,deflate Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7 Referer: http://someserver.com:3128/ecmweb/ Cookie: BCSI-CSAC11A855=2 Authorization: Basic c3lzYWRtaW46c3RlbDJEdmw= X-BlueCoat-Via: C0C9763E7BC80B55 Via: 1.1 someserver:3128 (squid/2.6.STABLE13) X-Forwarded-For: user22.someserver.com Cache-Control: max-age=259200, max-stale Connection: keep-alive So I get a WARNING for a Forwarding loop (that I haven't been able to solve) and then if fails to Authenticate the user. I've been working on this for awhile now, and I'm stumped. What am I missing/doing wrong? Also, funny thing, when a department complained about how slow Stellent was, Stellent told them that most of their clients run a reverse proxy like Squid to speed it up! Since I've used Squid for sometime, I got the task to integrate this. Thanks Phil If you are not the intended recipient of this message (including attachments), or if you have received this message in error, immediately notify us and delete it and any attachments. If you no longer wish to receive e-mail from Edward Jones, please send this request to [EMAIL PROTECTED] You must include the e-mail address that you wish not to receive e-mail communications. For important additional information related to this e-mail, visit www.edwardjones.com/US_email_disclosure
[squid-users] Can Squid grab a list of files from a server and then Prefetch thost files?
Can Squid grab a list of files from a server and then prefetch those specific files? I would like to be able to set up a proxy server (that will run on both windows and Linux) that will grab mp3s and flash files from a server (running a web app) to help avoid a bottleneck. Ideally I could schedule the proxy to server to grab the list of files, then the files specified in that list, at a certain time every day or week. It would then cache those files for use when going to the web app. Can I do this with squid and maybe a script? Thank you, Kyle -- View this message in context: http://www.nabble.com/Can-Squid-grab-a-list-of-files-from-a-server-and-then-Prefetch-thost-files--tf4570960.html#a13046962 Sent from the Squid - Users mailing list archive at Nabble.com.
[squid-users] RE: Store.log filling up
From: Henrik Nordstrom [EMAIL PROTECTED] Date: Tue, 18 Sep 2007 16:57:58 +0200 On mån, 2007-09-17 at 16:30 -0500, [EMAIL PROTECTED] wrote: Could spyware or addware cause the store.log to fill up very quickly? Another tech has had troubles with this in the last couple of days and was asking. He says that they can clear it out and in no time (not sure how long, but under an hour) it is filled up and causing problems. Here is a small post of what was in it. Why does it list all the ? Thanks for any info. 1190033958.390 RELEASE -1 7B1287005AF9902646FDACC9F3EA9C7F ? ? ? ? ?/? ?/? ? ? Looks a bit odd.. the ? is when the information is unknown, but these objects was in memory so the information should have been known I think.. What do access.log say? Regards Henrik He thought he had it figured out, but started getting this problem again so I am sending his other log files. Thanks for any info. -- Scott Mayo System Administrator Bloomfield Schools Squid cache.log 2007/10/04 12:09:23| Starting Squid Cache version 2.4.STABLE7 for i586-mandrake-linux-gnu... 2007/10/04 12:09:23| Process ID 21644 2007/10/04 12:09:23| With 1024 file descriptors available 2007/10/04 12:09:23| DNS Socket created on FD 4 2007/10/04 12:09:23| Adding nameserver 127.0.0.1 from /etc/resolv.conf 2007/10/04 12:09:23| User-Agent logging is disabled. 2007/10/04 12:09:23| Unlinkd pipe opened on FD 9 2007/10/04 12:09:23| Swap maxSize 1024000 KB, estimated 78769 objects 2007/10/04 12:09:23| Target number of buckets: 3938 2007/10/04 12:09:23| Using 8192 Store buckets 2007/10/04 12:09:23| Max Mem size: 16384 KB 2007/10/04 12:09:23| Max Swap size: 1024000 KB 2007/10/04 12:09:23| Store logging disabled 2007/10/04 12:09:23| Rebuilding storage in /var/spool/squid (DIRTY) 2007/10/04 12:09:23| Using Least Load store dir selection 2007/10/04 12:09:23| Set Current Directory to /var/spool/squid 2007/10/04 12:09:23| Loaded Icons. 2007/10/04 12:09:24| Accepting HTTP connections at 0.0.0.0, port 8080, FD 10. 2007/10/04 12:09:24| Accepting HTCP messages on port 4827, FD 12. 2007/10/04 12:09:24| Accepting SNMP messages on port 3401, FD 13. 2007/10/04 12:09:24| WCCP Disabled. 2007/10/04 12:09:24| Ready to serve requests. 2007/10/04 12:09:24| Store rebuilding is 1.1% complete 2007/10/04 12:09:32| diskHandleWrite: FD 7: disk write error: (28) No space left on device FATAL: Write failure -- check your disk space and cache.log Squid Cache (Version 2.4.STABLE7): Terminated abnormally. CPU Usage: 7.880 seconds = 4.090 user + 3.790 sys Maximum Resident Size: 0 KB Page faults with physical i/o: 399 Memory usage for squid via mallinfo(): total space in arena:8550 KB Ordinary blocks: 7896 KB162 blks Small blocks: 0 KB 0 blks Holding blocks: 176 KB 1 blks Free Small blocks: 0 KB Free Ordinary blocks: 654 KB Total in use:8072 KB 94% Total free: 654 KB 8% 2007/10/04 12:09:35| Starting Squid Cache version 2.4.STABLE7 for i586-mandrake-linux-gnu... 2007/10/04 12:09:35| Process ID 21668 2007/10/04 12:09:35| With 1024 file descriptors available 2007/10/04 12:09:35| DNS Socket created on FD 4 2007/10/04 12:09:35| Adding nameserver 127.0.0.1 from /etc/resolv.conf 2007/10/04 12:09:35| User-Agent logging is disabled. 2007/10/04 12:09:35| Unlinkd pipe opened on FD 9 2007/10/04 12:09:35| Swap maxSize 1024000 KB, estimated 78769 objects 2007/10/04 12:09:35| Target number of buckets: 3938 2007/10/04 12:09:35| Using 8192 Store buckets 2007/10/04 12:09:35| Max Mem size: 16384 KB 2007/10/04 12:09:35| Max Swap size: 1024000 KB 2007/10/04 12:09:35| Store logging disabled 2007/10/04 12:09:35| Rebuilding storage in /var/spool/squid (DIRTY) 2007/10/04 12:09:35| Using Least Load store dir selection 2007/10/04 12:09:35| Set Current Directory to /var/spool/squid 2007/10/04 12:09:35| Loaded Icons. 2007/10/04 12:09:35| Accepting HTTP connections at 0.0.0.0, port 8080, FD 10. 2007/10/04 12:09:35| Accepting HTCP messages on port 4827, FD 12. 2007/10/04 12:09:35| Accepting SNMP messages on port 3401, FD 13. 2007/10/04 12:09:35| WCCP Disabled. 2007/10/04 12:09:35| Ready to serve requests. 2007/10/04 12:09:36| Store rebuilding is 1.1% complete 2007/10/04 12:09:43| diskHandleWrite: FD 7: disk write error: (28) No space left on device FATAL: Write failure -- check your disk space and cache.log Squid Cache (Version 2.4.STABLE7): Terminated abnormally. CPU Usage: 7.810 seconds = 3.990 user + 3.820 sys Maximum Resident Size: 0 KB Page faults with physical i/o: 399 Memory usage for squid via mallinfo(): total space in arena:8554 KB Ordinary blocks: 7855 KB172 blks Small blocks: 0 KB 0 blks Holding blocks: 176
Re: [squid-users] Can Squid grab a list of files from a server and then Prefetch thost files?
miles800 wrote: Can Squid grab a list of files from a server and then prefetch those specific files? I would like to be able to set up a proxy server (that will run on both windows and Linux) that will grab mp3s and flash files from a server (running a web app) to help avoid a bottleneck. Ideally I could schedule the proxy to server to grab the list of files, then the files specified in that list, at a certain time every day or week. It would then cache those files for use when going to the web app. Can I do this with squid and maybe a script? Thank you, Kyle I am sure you could script something with wget, saving the files under /dev/null
[squid-users] Re: Can Squid grab a list of files from a server and then Prefetch thost files?
On Thu, 4 Oct 2007 12:17:25 -0700 (PDT) miles800 [EMAIL PROTECTED] wrote: Can Squid grab a list of files from a server and then prefetch those specific files? I would like to be able to set up a proxy server (that will run on both windows and Linux) that will grab mp3s and flash files from a server (running a web app) to help avoid a bottleneck. Ideally I could schedule the proxy to server to grab the list of files, then the files specified in that list, at a certain time every day or week. It would then cache those files for use when going to the web app. Can I do this with squid and maybe a script? It's not really a question of whether squid can do it. You can just schedule any download utility that supports a proxy, such as wget, to do it.
Re: [squid-users] Re: Can Squid grab a list of files from a server and then Prefetch thost files?
Hi, another great program is httrack. solomon. --- RW [EMAIL PROTECTED] wrote: On Thu, 4 Oct 2007 12:17:25 -0700 (PDT) miles800 [EMAIL PROTECTED] wrote: Can Squid grab a list of files from a server and then prefetch those specific files? I would like to be able to set up a proxy server It's not really a question of whether squid can do it. You can just schedule any download utility that supports a proxy, such as wget, to do it.
Re: [squid-users] RE: Store.log filling up
From: Henrik Nordstrom [EMAIL PROTECTED] Date: Tue, 18 Sep 2007 16:57:58 +0200 On mån, 2007-09-17 at 16:30 -0500, [EMAIL PROTECTED] wrote: Could spyware or addware cause the store.log to fill up very quickly? Another tech has had troubles with this in the last couple of days and was asking. He says that they can clear it out and in no time (not sure how long, but under an hour) it is filled up and causing problems. Here is a small post of what was in it. Why does it list all the ? Thanks for any info. 1190033958.390 RELEASE -1 7B1287005AF9902646FDACC9F3EA9C7F ? ? ? ? ?/? ?/? ? ? Looks a bit odd.. the ? is when the information is unknown, but these objects was in memory so the information should have been known I think.. What do access.log say? Regards Henrik He thought he had it figured out, but started getting this problem again so I am sending his other log files. Thanks for any info. -- Scott Mayo System Administrator Bloomfield Schools Squid cache.log 2007/10/04 12:09:23| Starting Squid Cache version 2.4.STABLE7 for i586-mandrake-linux-gnu... Gah! Try using 2.6 Amos
Re: [squid-users] Squid can't login to Stellent CMS
I've got Squid (squid-2.6.STABLE13-1.RHEL4) running on Red Hat EL 4, pointing to a Solaris based Stellent CMS instance and it's working, but when someone tries to login to the Stellent content contol page, they get the following error page in their browser: _ ERROR The requested URL could not be retrieved While trying to retrieve the URL: http://someserver.com:3128/ecmweb/idcplg?IdcService=LOGINAction=GetTemp latePagePage=HOME_PAGEAuth=Internet The following error was encountered: * Access Denied. Access control configuration prevents your request from being allowed at this time. Please contact your service provider if you feel this is incorrect. Your cache administrator is root. Generated Thu, 04 Oct 2007 15:51:41 GMT by someserver.com (squid/2.6.STABLE13) I have very simple ACLs in place until I get everything working, I've included the other lines so you can see the Ips in place: http_port 192.168.120.225:3128 transparent Use accel or vhost instead of transparent. transparent means semi-silently intercepting proxy. accel/vhost means squid pretends to be the origin web-server for some domain(s) but passes any non-cached requests to the real one. http://wiki.squid-cache.org/SquidFaq/ReverseProxy?highlight=%28reverse%29 cache_peer 192.168.210.54 parent 80 0 no-query originserver login=PASS acl all src 0.0.0.0/0.0.0.0 acl all src all is a nicer way of saying that http_access allow all icp_access allow all ICP does not come into this at all (you set peer-ICP-port=0) . You may as well turn it off. # note, it failed with the following 3 lines in, and out of the config #acl METHODLIST method post #no_cache deny METHODLIST These two are the default behaviour under most configs. FYI: forget no_cache, you can safely drop just the no_ bit off the front now and understand what that line does better. Amos
Re: [squid-users] Squid 2.6 - access hosts outside LAN through proxy with https://a.b.c.d:8080
How would I define the correct ACL and/or http_access rule to access external hosts, that are to be reached through a https- admin interface that is using port 8080 ? I tried to add 8080 to the list of SSL-Ports like acl SSL_ports 443 563 8080 and thus allow it to CONNECT directly with http_access deny CONNECT !SSL_ports (rest is kept to the recommended defaults) ... but Squid keeps on telling me that the connection is refused (111) Access Denied is given when squid ACLs block. Connection Refused is a denial by the remote system if its coming from squid. Amos
[squid-users] Re: Can Squid grab a list of files from a server and then Prefetch thost files?
On Thu, 04 Oct 2007 15:33:18 -0400 Jason Taylor [EMAIL PROTECTED] wrote: miles800 wrote: Can Squid grab a list of files from a server and then prefetch those specific files? I would like to be able to set up a proxy server (that will run on both windows and Linux) that will grab mp3s and flash files from a server (running a web app) to help avoid a bottleneck. Ideally I could schedule the proxy to server to grab the list of files, then the files specified in that list, at a certain time every day or week. It would then cache those files for use when going to the web app. Can I do this with squid and maybe a script? Thank you, Kyle I am sure you could script something with wget, saving the files under /dev/null Actually wget has a '--delete-after' option that was implemented for this purpose.
Re: [squid-users] Squid 2.6 STABLE 16 also crashing
On Thu, Oct 04, 2007, Ali resting wrote: Hi Mark, This problem only started in the last couple of days. I had been running squid on the same mchine for years. I have even tried delete the swap directories hoping it would help, but still no joy. Any suggestions, is it hardware related? You need to match that stack trace against Squid function calls. Do you get a coredump? Can you use gdb against the coredump and the Squid binary to get symbols? Is your squid binary install actually lacking debugging symbols? ADrian Regards, Ali From: Marc Kool [EMAIL PROTECTED] To: Ali resting [EMAIL PROTECTED] Subject: Re: [squid-users] Squid 2.6 STABLE 16 also crashing Date: Thu, 4 Oct 2007 16:08:18 +0200 Ali, I see that regexec and pthreads are being used. If you have a glibc before 2.4 then you need to upgrade glibc or configure squid not to use pthreads. -Marcus On Thu, Oct 04, 2007 at 03:14:15PM +0200, Ali resting wrote: Hi Amos, I have upgraded to Squid 2.6 STABLE 16 And it is still crashing. See the log extracts just before it crashes: (squid)(death+0x3d)[0x80bb5a9] /lib/i686/libpthread.so.0[0x4005747e] (squid)[0x42028c48] (squid)[0x420c12db] (squid)[0x420bd350] (squid)(regexec+0x65)[0x420c2df5] (squid)(refreshLimits+0x27)[0x80a6d4b] (squid)[0x808e9ef] (squid)[0x808f1a4] (squid)[0x809005e] (squid)[0x807857e] (squid)[0x8078bac] (squid)(comm_select+0xb4)[0x807887c] (squid)(main+0x2a1)[0x809e0f9] (squid)(__libc_start_main+0xa4)[0x420158d4] (squid)(shmat+0x51)[0x8057419] FATAL: Received Segment Violation...dying. (squid)(death+0x3d)[0x80bb5a9] /lib/i686/libpthread.so.0[0x4005747e] (squid)[0x42028c48] (squid)[0x420c12db] (squid)[0x420bd350] (squid)(regexec+0x65)[0x420c2df5] (squid)(refreshLimits+0x27)[0x80a6d4b] (squid)(getMaxAge+0x24)[0x80a76f8] (squid)(httpBuildRequestHeader+0x52e)[0x809094a] (squid)(httpBuildRequestPrefix+0x52)[0x8091096] (squid)[0x80911ee] (squid)[0x80764d9] (squid)[0x807846c] (squid)[0x8078bac] (squid)(comm_select+0xb4)[0x807887c] (squid)(main+0x2a1)[0x809e0f9] (squid)(__libc_start_main+0xa4)[0x420158d4] (squid)(shmat+0x51)[0x8057419] FATAL: Received Segment Violation...dying. 2007/10/04 14:51:23| Not currently OK to rewrite swap log. 2007/10/04 14:51:23| storeDirWriteCleanLogs: Operation aborted. From: Amos Jeffries [EMAIL PROTECTED] To: Ali resting [EMAIL PROTECTED] CC: squid-users@squid-cache.org Subject: Re: [squid-users] Squid 2.5-STABLE14 Crashing Date: Wed, 3 Oct 2007 10:29:08 +1300 (NZDT) Hi, For the last couple of days my squid server keeps crashing and restarting itself. I have looked at the cache.log file and this is what I get. This server has been running fine for the last 2 years: First, check your logs are being rotated properly and haven't taken up all disk space. And that the system has not run out of inodes. Second, upgrade to a currently supported version of squid, 2.6s16+ or 3.0rc1 (squid)[0x80a1afd] /lib/i686/libpthread.so.0[0x4005747e] (squid)[0x42028c48] (squid)[0x420c12db] (squid)[0x420bd350] (squid)(regexec+0x65)[0x420c2df5] (squid)(vfprintf+0x2d36)[0x804d21a] (squid)(vfprintf+0x39a9)[0x804de8d] (squid)(vfprintf+0x3d4b)[0x804e22f] That does not look like cache.log content. if it did come from there its seriously screwed. Amos _ Express yourself instantly with MSN Messenger! Download today it's FREE! http://messenger.msn.click-url.com/go/onm00200471ave/direct/01/ -- _ Express yourself instantly with MSN Messenger! Download today it's FREE! http://messenger.msn.click-url.com/go/onm00200471ave/direct/01/ -- - Xenion - http://www.xenion.com.au/ - VPS Hosting - Commercial Squid Support - - $25/pm entry-level bandwidth-capped VPSes available in WA -
