RE: [squid-users] Re: Purging cached web objects
I'm looking at writing a web based admin application for squid and have been using the cache manager functions to help retrieve what's stored in the cache. With Henrik's help I've discovered that in memory objects are stored in both the vm_objects objects (objects lists both in memory and on disk) functions, they are listed with their urls, objects stored on disk are also listed, but not with their urls. If the cache manager objects function could be extended to also list the urls it would probably help. You'd be able to make an http request to the proxy for cache_object://localhost/objects and do your own regexing to find the urls you're after and send purge requests for those. It wouldn't be the most efficient way of doing it, but it'd be a start! Joe Joe Tiedeman Support Analyst Higher Education Statistics Agency (HESA) 95 Promenade, Cheltenham, Gloucestershire GL50 1HZ T 01242 211167 F 01242 211122 W www.hesa.ac.uk -Original Message- From: Chris Woodfield [mailto:[EMAIL PROTECTED] Sent: Monday 1 September 2008 16:17 To: Amos Jeffries Cc: RW; squid-users@squid-cache.org Subject: Re: [squid-users] Re: Purging cached web objects squidpurge works, but it's hardly ideal, especially on squids with big disks...in my testing on a box with 3x1TB cache_dirs, it took 15 minutes to run and thrashed the disks pretty hard while it was running, affecting response time for production traffic. The reason for this is that squid stores each object record internally as a hash, not the URL itself, which means that in order to search for regex matches, it's necessary to look at every file in every cache_dir to check against the regex. An easily-searchable URL datastore would help immensely here. As a mad experiment a while back, a former colleague hacked a SQL update into the store and release functions, but it's unlikely anything like that would work well in production without some serious work to guarantee squid/DB data integrity. Some sort of internal b-tree that stores all currently-cached URLs might be a solution...or even an internal sqlite implementation? Has anyone else ever proposed such a solution? -C On Aug 25, 2008, at 11:55 PM, Amos Jeffries wrote: On Sun, 24 Aug 2008 08:59:07 +1200 Amos Jeffries [EMAIL PROTECTED] wrote: Paras Fadte wrote: Hi, Is there any utility for purging cached web objects in squid with wildcard support ? Not that we know of. You presumable know about squidpurge. Has it broken or something? http://www.wa.apana.org.au/~dean/squidpurge/ Ah. no. I didn't. I see the name fly by every now and again, but haven't really noticed it. Amos __ This incoming email was virus scanned for HESA by MessageLabs. __ _ Higher Education Statistics Agency Ltd (HESA) is a company limited by guarantee, registered in England at 95 Promenade Cheltenham GL50 1HZ. Registered No. 2766993. The members are Universities UK and GuildHE. Registered Charity No. 1039709. Certified to ISO 9001 and BS 7799. HESA Services Ltd (HSL) is a wholly owned subsidiary of HESA, registered in England at the same address. Registered No. 3109219. _ This outgoing email was virus scanned for HESA by MessageLabs. _
[squid-users] Cache Manager query (objects/vm_objects)
Hi Guys, I'm currently trying to write an internal web application to manage our proxies (specifically our reverse proxy) and have hit a bit of a stumbling block. It appears that the objects and vm_objects listing from the cache manager only list objects cached in memory and not on disk. i.e. looking in our access logs, urls which appear as tcp_mem_hit show up in both listings, but urls which show up as tcp_hit don't. If I'm correct on this assumption, is there any way of listing in a similar fashion all objects, including those on disk, and if so, a way to distinguish between those in memory and those on disk. If none of the above is true, please feel free to correct me and point me in the right direction! Cheers Joe Joe Tiedeman Support Analyst Higher Education Statistics Agency (HESA) 95 Promenade, Cheltenham, Gloucestershire GL50 1HZ T 01242 211167 F 01242 211122 W www.hesa.ac.uk _ Higher Education Statistics Agency Ltd (HESA) is a company limited by guarantee, registered in England at 95 Promenade Cheltenham GL50 1HZ. Registered No. 2766993. The members are Universities UK and GuildHE. Registered Charity No. 1039709. Certified to ISO 9001 and BS 7799. HESA Services Ltd (HSL) is a wholly owned subsidiary of HESA, registered in England at the same address. Registered No. 3109219. _ This outgoing email was virus scanned for HESA by MessageLabs. _
RE: [squid-users] Cache Manager query (objects/vm_objects)
Having relooked at cache_object://localhost/objects, it appears that there are quite a few objects listed as NOT_IN_MEMORY, I assume that these are cached on disk, but they don't include the URL and method which the IN_MEMORY objects do, is there any way to determine this? Cheers Joe Joe Tiedeman Support Analyst Higher Education Statistics Agency (HESA) 95 Promenade, Cheltenham, Gloucestershire GL50 1HZ T 01242 211167 F 01242 211122 W www.hesa.ac.uk -Original Message- From: Joe Tiedeman [mailto:[EMAIL PROTECTED] Sent: Friday 29 August 2008 15:56 To: squid-users@squid-cache.org Subject: [squid-users] Cache Manager query (objects/vm_objects) Hi Guys, I'm currently trying to write an internal web application to manage our proxies (specifically our reverse proxy) and have hit a bit of a stumbling block. It appears that the objects and vm_objects listing from the cache manager only list objects cached in memory and not on disk. i.e. looking in our access logs, urls which appear as tcp_mem_hit show up in both listings, but urls which show up as tcp_hit don't. If I'm correct on this assumption, is there any way of listing in a similar fashion all objects, including those on disk, and if so, a way to distinguish between those in memory and those on disk. If none of the above is true, please feel free to correct me and point me in the right direction! Cheers Joe Joe Tiedeman Support Analyst Higher Education Statistics Agency (HESA) 95 Promenade, Cheltenham, Gloucestershire GL50 1HZ T 01242 211167 F 01242 211122 W www.hesa.ac.uk _ Higher Education Statistics Agency Ltd (HESA) is a company limited by guarantee, registered in England at 95 Promenade Cheltenham GL50 1HZ. Registered No. 2766993. The members are Universities UK and GuildHE. Registered Charity No. 1039709. Certified to ISO 9001 and BS 7799. HESA Services Ltd (HSL) is a wholly owned subsidiary of HESA, registered in England at the same address. Registered No. 3109219. _ This outgoing email was virus scanned for HESA by MessageLabs. _ __ This incoming email was virus scanned for HESA by MessageLabs. __ _ Higher Education Statistics Agency Ltd (HESA) is a company limited by guarantee, registered in England at 95 Promenade Cheltenham GL50 1HZ. Registered No. 2766993. The members are Universities UK and GuildHE. Registered Charity No. 1039709. Certified to ISO 9001 and BS 7799. HESA Services Ltd (HSL) is a wholly owned subsidiary of HESA, registered in England at the same address. Registered No. 3109219. _ This outgoing email was virus scanned for HESA by MessageLabs. _
RE: [squid-users] caching headers
Hi Henrik, I want to try and achieve this setup so that content is returned to clients as fast as possible, but also trying to make sure that my infrastructure is queried for it every time someone tries to access it, even if it's just for logging purposes. Would the combination of header_access/header_replace be a suitable way to achieve this? Cheers Joe Joe Tiedeman Support Analyst Higher Education Statistics Agency (HESA) 95 Promenade, Cheltenham, Gloucestershire GL50 1HZ T 01242 211167 F 01242 211122 W www.hesa.ac.uk -Original Message- From: Henrik Nordstrom [mailto:[EMAIL PROTECTED] Sent: Monday 18 August 2008 01:13 To: Joe Tiedeman Cc: squid-users@squid-cache.org Subject: Re: [squid-users] caching headers sön 2008-08-17 klockan 21:35 +0100 skrev Joe Tiedeman: I am currently used squid as a caching reverse proxy. I would like to know if it's possible to have the web server return the appropriate caching headers to squid and have squid rewrite/replace the headers to make the client not cache the responses. Why? Most want's to do the opposite... aggressively cache in the clients and less aggressively cache in the reverse proxy.. Regards Henrik __ This incoming email was virus scanned for HESA by MessageLabs. __ _ Higher Education Statistics Agency Ltd (HESA) is a company limited by guarantee, registered in England at 95 Promenade Cheltenham GL50 1HZ. Registered No. 2766993. The members are Universities UK and GuildHE. Registered Charity No. 1039709. Certified to ISO 9001 and BS 7799. HESA Services Ltd (HSL) is a wholly owned subsidiary of HESA, registered in England at the same address. Registered No. 3109219. _ This outgoing email was virus scanned for HESA by MessageLabs. _
RE: [squid-users] POST + NTLM Authentica
Hi Guys, Is there any more information that I could provide to help with the resolution of this issue (bug 2176)? Bill, did you get a chance to test Squid 3.0 to see if the issue is still apparent? Cheers Joe -Original Message- From: Joe Tiedeman Sent: Wednesday 23 July 2008 11:54 To: [EMAIL PROTECTED]; squid-users@squid-cache.org Subject: RE: [squid-users] POST + NTLM Authentica I suddenly remembered that you can enable more debug logging in the cache.log file, I've added debug_options ALL,1 33,2 to squid.conf and this is what now gets output to the cache.log Not sure if this is of any use to any of the developers? 2008/07/23 11:49:02| The request POST http://www.example.ac.uk:80/administrator/index2.php is ALLOWED, because it matched 'www_site' 2008/07/23 11:49:03| clientReadBody: start fd=22 body_size=32002 in.offset=4095 cb=0x8088c4c req=0xa4f3ad0 2008/07/23 11:49:03| clientProcessBody: start fd=22 body_size=32002 in.offset=4095 cb=0x8088c4c req=0xa4f3ad0 2008/07/23 11:49:03| clientProcessBody: end fd=22 size=4095 body_size=27907 in.offset=0 cb=0x8088c4c req=0xa4f3ad0 2008/07/23 11:49:03| clientReadBody: start fd=22 body_size=27907 in.offset=4095 cb=0x8088c4c req=0xa4f3ad0 2008/07/23 11:49:03| clientProcessBody: start fd=22 body_size=27907 in.offset=4095 cb=0x8088c4c req=0xa4f3ad0 2008/07/23 11:49:03| clientProcessBody: end fd=22 size=4095 body_size=23812 in.offset=0 cb=0x8088c4c req=0xa4f3ad0 2008/07/23 11:49:03| clientReadBody: start fd=22 body_size=23812 in.offset=0 cb=0x8088c4c req=0xa4f3ad0 2008/07/23 11:49:03| clientProcessBody: start fd=22 body_size=23812 in.offset=0 cb=0x8088c4c req=0xa4f3ad0 2008/07/23 11:49:03| clientProcessBody: start fd=22 body_size=23812 in.offset=4095 cb=0x8088c4c req=0xa4f3ad0 2008/07/23 11:49:03| clientProcessBody: end fd=22 size=4095 body_size=19717 in.offset=0 cb=0x8088c4c req=0xa4f3ad0 2008/07/23 11:49:03| clientReadBody: start fd=22 body_size=19717 in.offset=4095 cb=0x8088c4c req=0xa4f3ad0 2008/07/23 11:49:03| clientProcessBody: start fd=22 body_size=19717 in.offset=4095 cb=0x8088c4c req=0xa4f3ad0 2008/07/23 11:49:03| clientProcessBody: end fd=22 size=4095 body_size=15622 in.offset=0 cb=0x8088c4c req=0xa4f3ad0 2008/07/23 11:49:03| The reply for POST http://www.example.ac.uk/administrator/index2.php is ALLOWED, because it matched 'all' 2008/07/23 11:49:03| httpAppendBody: Request not yet fully sent POST http://www.example.ac.uk/administrator/index2.php; 2008/07/23 11:49:03| clientProcessBody: start fd=22 body_size=15622 in.offset=4095 cb=0x8068ca0 req=(nil) 2008/07/23 11:49:03| clientProcessBody: start fd=22 body_size=11527 in.offset=0 cb=0x8068ca0 req=(nil) 2008/07/23 11:49:03| clientProcessBody: end fd=22 size=4095 body_size=11527 in.offset=0 cb=0x8068ca0 req=(nil) 2008/07/23 11:49:03| clientProcessBody: start fd=22 body_size=11527 in.offset=4095 cb=0x8068ca0 req=(nil) 2008/07/23 11:49:03| clientProcessBody: start fd=22 body_size=7432 in.offset=0 cb=0x8068ca0 req=(nil) 2008/07/23 11:49:03| clientProcessBody: end fd=22 size=4095 body_size=7432 in.offset=0 cb=0x8068ca0 req=(nil) 2008/07/23 11:49:03| clientProcessBody: start fd=22 body_size=7432 in.offset=4095 cb=0x8068ca0 req=(nil) 2008/07/23 11:49:03| clientProcessBody: start fd=22 body_size=3337 in.offset=0 cb=0x8068ca0 req=(nil) 2008/07/23 11:49:03| clientProcessBody: end fd=22 size=4095 body_size=3337 in.offset=0 cb=0x8068ca0 req=(nil) 2008/07/23 11:49:03| clientProcessBody: start fd=22 body_size=3337 in.offset=3337 cb=0x8068ca0 req=(nil) 2008/07/23 11:49:03| clientProcessBody: end fd=22 size=3337 body_size=0 in.offset=0 cb=0x8068ca0 req=(nil) 2008/07/23 11:49:03| growing request buffer: offset=4095 size=8192 2008/07/23 11:49:03| The request POST http://www.example.ac.uk:80/administrator/index2.php is ALLOWED, because it matched 'www_site' 2008/07/23 11:49:03| clientReadBody: start fd=27 body_size=32002 in.offset=8191 cb=0x8088c4c req=0xa4f3ad0 2008/07/23 11:49:03| clientProcessBody: start fd=27 body_size=32002 in.offset=8191 cb=0x8088c4c req=0xa4f3ad0 2008/07/23 11:49:03| clientProcessBody: end fd=27 size=8191 body_size=23811 in.offset=0 cb=0x8088c4c req=0xa4f3ad0 2008/07/23 11:49:03| clientReadBody: start fd=27 body_size=23811 in.offset=8191 cb=0x8088c4c req=0xa4f3ad0 2008/07/23 11:49:03| clientProcessBody: start fd=27 body_size=23811 in.offset=8191 cb=0x8088c4c req=0xa4f3ad0 2008/07/23 11:49:03| clientProcessBody: end fd=27 size=8191 body_size=15620 in.offset=0 cb=0x8088c4c req=0xa4f3ad0 2008/07/23 11:49:03| clientReadBody: start fd=27 body_size=15620 in.offset=8191 cb=0x8088c4c req=0xa4f3ad0 2008/07/23 11:49:03| clientProcessBody: start fd=27 body_size=15620 in.offset=8191 cb=0x8088c4c req=0xa4f3ad0 2008/07/23 11:49:03| clientProcessBody: end fd=27 size=8191 body_size=7429 in.offset=0 cb=0x8088c4c req=0xa4f3ad0 2008/07/23 11:49:03| clientReadBody: start fd=27 body_size=7429 in.offset=7429 cb=0x8088c4c req=0xa4f3ad0 2008/07/23 11:49:03| clientProcessBody: start fd=27 body_size=7429
RE: [squid-users] POST + NTLM Authentica
Thanks for that information Leonardo, do you think there would be any more info I could provide to help with the issue in 2.6/2.7? Cheers Joe -Original Message- From: Leonardo Rodrigues Magalhães [mailto:[EMAIL PROTECTED] Sent: Monday 18 August 2008 15:36 To: ML squid Cc: Joe Tiedeman Subject: Re: [squid-users] POST + NTLM Authentica Joe Tiedeman escreveu: Hi Guys, Is there any more information that I could provide to help with the resolution of this issue (bug 2176)? Bill, did you get a chance to test Squid 3.0 to see if the issue is still apparent? it was discussed recently on this mailing list the fact that squid 3.0 does NOT support NTLM-authentication pass-trough. So i think with squid 3.0 it wont work at all. You wont even be able to browser NTLM-authentication-enabled sites using squid 3.0. -- Atenciosamente / Sincerily, Leonardo Rodrigues Solutti Tecnologia http://www.solutti.com.br Minha armadilha de SPAM, NÃO mandem email [EMAIL PROTECTED] My SPAMTRAP, do not email it __ This incoming email was virus scanned for HESA by MessageLabs. __ _ Higher Education Statistics Agency Ltd (HESA) is a company limited by guarantee, registered in England at 95 Promenade Cheltenham GL50 1HZ. Registered No. 2766993. The members are Universities UK and GuildHE. Registered Charity No. 1039709. Certified to ISO 9001 and BS 7799. HESA Services Ltd (HSL) is a wholly owned subsidiary of HESA, registered in England at the same address. Registered No. 3109219. _ This outgoing email was virus scanned for HESA by MessageLabs. _
[squid-users] caching headers
Dear All, I am currently used squid as a caching reverse proxy. I would like to know if it's possible to have the web server return the appropriate caching headers to squid and have squid rewrite/replace the headers to make the client not cache the responses. I think I might be able to achieve this with a combination of header_access/header_replace. If anyone could offer any guidance, I would be most grateful. Cheers Joe Joe Tiedeman Support Analyst Higher Education Statistics Agency (HESA) 95 Promenade, Cheltenham, Gloucestershire GL50 1HZ T 01242 211167 F 01242 211122 W www.hesa.ac.uk http://www.hesa.ac.uk/ _ Higher Education Statistics Agency Ltd (HESA) is a company limited by guarantee, registered in England at 95 Promenade Cheltenham GL50 1HZ. Registered No. 2766993. The members are Universities UK and GuildHE. Registered Charity No. 1039709. Certified to ISO 9001 and BS 7799. HESA Services Ltd (HSL) is a wholly owned subsidiary of HESA, registered in England at the same address. Registered No. 3109219. _ This outgoing email was virus scanned for HESA by MessageLabs. _
RE: [squid-users] POST + NTLM Authentica
I suddenly remembered that you can enable more debug logging in the cache.log file, I've added debug_options ALL,1 33,2 to squid.conf and this is what now gets output to the cache.log Not sure if this is of any use to any of the developers? 2008/07/23 11:49:02| The request POST http://www.example.ac.uk:80/administrator/index2.php is ALLOWED, because it matched 'www_site' 2008/07/23 11:49:03| clientReadBody: start fd=22 body_size=32002 in.offset=4095 cb=0x8088c4c req=0xa4f3ad0 2008/07/23 11:49:03| clientProcessBody: start fd=22 body_size=32002 in.offset=4095 cb=0x8088c4c req=0xa4f3ad0 2008/07/23 11:49:03| clientProcessBody: end fd=22 size=4095 body_size=27907 in.offset=0 cb=0x8088c4c req=0xa4f3ad0 2008/07/23 11:49:03| clientReadBody: start fd=22 body_size=27907 in.offset=4095 cb=0x8088c4c req=0xa4f3ad0 2008/07/23 11:49:03| clientProcessBody: start fd=22 body_size=27907 in.offset=4095 cb=0x8088c4c req=0xa4f3ad0 2008/07/23 11:49:03| clientProcessBody: end fd=22 size=4095 body_size=23812 in.offset=0 cb=0x8088c4c req=0xa4f3ad0 2008/07/23 11:49:03| clientReadBody: start fd=22 body_size=23812 in.offset=0 cb=0x8088c4c req=0xa4f3ad0 2008/07/23 11:49:03| clientProcessBody: start fd=22 body_size=23812 in.offset=0 cb=0x8088c4c req=0xa4f3ad0 2008/07/23 11:49:03| clientProcessBody: start fd=22 body_size=23812 in.offset=4095 cb=0x8088c4c req=0xa4f3ad0 2008/07/23 11:49:03| clientProcessBody: end fd=22 size=4095 body_size=19717 in.offset=0 cb=0x8088c4c req=0xa4f3ad0 2008/07/23 11:49:03| clientReadBody: start fd=22 body_size=19717 in.offset=4095 cb=0x8088c4c req=0xa4f3ad0 2008/07/23 11:49:03| clientProcessBody: start fd=22 body_size=19717 in.offset=4095 cb=0x8088c4c req=0xa4f3ad0 2008/07/23 11:49:03| clientProcessBody: end fd=22 size=4095 body_size=15622 in.offset=0 cb=0x8088c4c req=0xa4f3ad0 2008/07/23 11:49:03| The reply for POST http://www.example.ac.uk/administrator/index2.php is ALLOWED, because it matched 'all' 2008/07/23 11:49:03| httpAppendBody: Request not yet fully sent POST http://www.example.ac.uk/administrator/index2.php; 2008/07/23 11:49:03| clientProcessBody: start fd=22 body_size=15622 in.offset=4095 cb=0x8068ca0 req=(nil) 2008/07/23 11:49:03| clientProcessBody: start fd=22 body_size=11527 in.offset=0 cb=0x8068ca0 req=(nil) 2008/07/23 11:49:03| clientProcessBody: end fd=22 size=4095 body_size=11527 in.offset=0 cb=0x8068ca0 req=(nil) 2008/07/23 11:49:03| clientProcessBody: start fd=22 body_size=11527 in.offset=4095 cb=0x8068ca0 req=(nil) 2008/07/23 11:49:03| clientProcessBody: start fd=22 body_size=7432 in.offset=0 cb=0x8068ca0 req=(nil) 2008/07/23 11:49:03| clientProcessBody: end fd=22 size=4095 body_size=7432 in.offset=0 cb=0x8068ca0 req=(nil) 2008/07/23 11:49:03| clientProcessBody: start fd=22 body_size=7432 in.offset=4095 cb=0x8068ca0 req=(nil) 2008/07/23 11:49:03| clientProcessBody: start fd=22 body_size=3337 in.offset=0 cb=0x8068ca0 req=(nil) 2008/07/23 11:49:03| clientProcessBody: end fd=22 size=4095 body_size=3337 in.offset=0 cb=0x8068ca0 req=(nil) 2008/07/23 11:49:03| clientProcessBody: start fd=22 body_size=3337 in.offset=3337 cb=0x8068ca0 req=(nil) 2008/07/23 11:49:03| clientProcessBody: end fd=22 size=3337 body_size=0 in.offset=0 cb=0x8068ca0 req=(nil) 2008/07/23 11:49:03| growing request buffer: offset=4095 size=8192 2008/07/23 11:49:03| The request POST http://www.example.ac.uk:80/administrator/index2.php is ALLOWED, because it matched 'www_site' 2008/07/23 11:49:03| clientReadBody: start fd=27 body_size=32002 in.offset=8191 cb=0x8088c4c req=0xa4f3ad0 2008/07/23 11:49:03| clientProcessBody: start fd=27 body_size=32002 in.offset=8191 cb=0x8088c4c req=0xa4f3ad0 2008/07/23 11:49:03| clientProcessBody: end fd=27 size=8191 body_size=23811 in.offset=0 cb=0x8088c4c req=0xa4f3ad0 2008/07/23 11:49:03| clientReadBody: start fd=27 body_size=23811 in.offset=8191 cb=0x8088c4c req=0xa4f3ad0 2008/07/23 11:49:03| clientProcessBody: start fd=27 body_size=23811 in.offset=8191 cb=0x8088c4c req=0xa4f3ad0 2008/07/23 11:49:03| clientProcessBody: end fd=27 size=8191 body_size=15620 in.offset=0 cb=0x8088c4c req=0xa4f3ad0 2008/07/23 11:49:03| clientReadBody: start fd=27 body_size=15620 in.offset=8191 cb=0x8088c4c req=0xa4f3ad0 2008/07/23 11:49:03| clientProcessBody: start fd=27 body_size=15620 in.offset=8191 cb=0x8088c4c req=0xa4f3ad0 2008/07/23 11:49:03| clientProcessBody: end fd=27 size=8191 body_size=7429 in.offset=0 cb=0x8088c4c req=0xa4f3ad0 2008/07/23 11:49:03| clientReadBody: start fd=27 body_size=7429 in.offset=7429 cb=0x8088c4c req=0xa4f3ad0 2008/07/23 11:49:03| clientProcessBody: start fd=27 body_size=7429 in.offset=7429 cb=0x8088c4c req=0xa4f3ad0 2008/07/23 11:49:03| clientProcessBody: end fd=27 size=7429 body_size=0 in.offset=0 cb=0x8088c4c req=0xa4f3ad0 2008/07/23 11:49:03| clientBuildReplyHeader: Connection oriented auth but server side non-persistent Cheers Joe -Original Message- From: Joe Tiedeman [mailto:[EMAIL PROTECTED] Sent: Thursday 17 July 2008 17:05 To: [EMAIL PROTECTED]; squid
RE: [squid-users] Reverse Proxy, OWA RPCoHTTPS and NTLM authentication passthrough
Amos, I've never been able to get NTLM pass thru to work with squid, I'm guessing because of the double hop issue. Kerberos, on the other hand, works perfectly once you've set up all the service principle names etc and is also much more secure. If you can get Kerberos working between the client and the OWA server directly, you can slot squid in the middle and the clients won't care. Joe Tiedeman Support Analyst Higher Education Statistics Agency (HESA) 95 Promenade, Cheltenham, Gloucestershire GL50 1HZ T 01242 211167 F 01242 211122 W www.hesa.ac.uk -Original Message- From: Amos Jeffries [mailto:[EMAIL PROTECTED] Sent: Thursday 17 July 2008 11:18 To: Abdessamad BARAKAT Cc: squid-users@squid-cache.org Subject: Re: [squid-users] Reverse Proxy, OWA RPCoHTTPS and NTLM authentication passthrough Abdessamad BARAKAT wrote: Hi people, Nobody for give me a feedback about this feature ( ntlm auth pass through) ? You know as much about this as most here. It don't work. I'm no expert myself but I suspect the reason goes something like this: (wild guess) NTLM is a sub-band authentication in background channels directly between the server and client. Now client thinks the reverse-proxy IS the server so is happy to authenticate with it. Squid is possibly able to pass the login details back to exchange, which required NTLM with the client. Client goes, hang on a minute I wasn't talking to you, and kills the auth. Squid does not have the client-stored secret information to setup a fake NTLM sequence to exchange on behalf of the username/pass it knows. As I said, I'm no expert, but it seems to me that is likely what the issue is. If I'm wrong can someone please indicate why such an old and popular item as NTLM re-auth has not been implemented in _any_ version of Squid yet? Amos Thanks Le 14 juil. 08 à 12:39, Abdessamad BARAKAT a écrit : Hi, I need to reverse proxied a OWA 2007 service and I have some problems with NTLM authentication and the RPC connection. Squid offers a SSL service and connect himself to the OWA with a SSL connection The NTLM authentication was made bu the OWA so I need squid to pass the credentials without modified them. Actually I get only 401 error code but when I switch the authentication to Basic authentication on the Outlook anywhere's settings, It's working. I want really to have the NTLM authentication working for don't ask all users to change their settings. The squid is chrooted. I have tried the following versions: - 3.0 STABLE7 - 2.7STABLE3 - 2.6STABLE21 - 2.6STABLE3 My setup (sometime I need to add acl all or logfile_daemon beetween versions, that's all) : CHROOT chroot /usr/local/squid mime_table /etc/mime.conf icon_directory /share/icons error_directory /share/errors/English unlinkd_program /libexec/unlinkd cache_dir ufs /var/cache 100 16 256 cache_store_log /var/logs/store.log access_log /var/logs/access.log squid pid_filename /var/logs/squid.pid logfile_daemon /libexec/logfile-daemon # Define the required extension methods extension_methods RPC_IN_DATA RPC_OUT_DATA # Publish the RPCoHTTP service via SSL https_port 192.168.1.122:8443 cert=/etc/apache2/ssl/webmail.corporate.com.p em defaultsite=webmail.corporate.com cache_peer 172.16.18.13 parent 443 0 no-query originserver login=PASS ssl sslfl ags=DONT_VERIFY_PEER name=exchangeServer acl all src 0.0.0.0/0.0.0.0 acl EXCH dstdomain .corporate.com cache_peer_access exchangeServer allow EXCH cache_peer_access exchangeServer deny all never_direct allow EXCH # Lock down access to just the Exchange Server! http_access allow EXCH http_access deny all miss_access allow EXCH miss_access deny all #no local caching #maximum_object_size 0 KB #minimum_object_size 0 KB #no_cache deny all #access_log /usr/local/squid/var/logs/access.log squid Thanks a lot for any tips or informations . !DSPAM:487b2e138671238159409! -- Please use Squid 2.7.STABLE3 or 3.0.STABLE7 __ This incoming email was virus scanned for HESA by MessageLabs. __ _ Higher Education Statistics Agency Ltd (HESA) is a company limited by guarantee, registered in England at 95 Promenade Cheltenham GL50 1HZ. Registered No. 2766993. The members are Universities UK and GuildHE. Registered Charity No. 1039709. Certified to ISO 9001 and BS 7799. HESA Services Ltd (HSL) is a wholly owned subsidiary of HESA, registered in England at the same address. Registered No. 3109219. _ This outgoing email was virus scanned for HESA by MessageLabs. _
RE: [squid-users] POST + NTLM Authentica
Hi Bill,
Just to keep you up to date, I'm testing with 2.7 Stable 3 and the
behaviour as far as the client goes is identical, however the error in
cache.log has changed to:-
2008/07/17 16:57:22| httpAppendBody: Request not yet fully sent POST
http://www.domain.com/administrator/index2.php;
Doing a quick google brings up this page
http://code.google.com/p/cacheboy/source/browse/branches/CACHEBOY_PRE/sr
c/http.c?r=12951
Line 794 onwards
if (!httpState-flags.request_sent) {
debug(11, 1) (httpAppendBody: Request not yet fully sent \%s
%s\\n,
RequestMethods[orig_request-method].str,
storeUrl(entry));
keep_alive = 0;
}
Hopefully this will help to shed some light on things.
Joe
-Original Message-
From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]
Sent: Thursday 10 July 2008 16:06
To: Joe Tiedeman; squid-users@squid-cache.org
Subject: RE: [squid-users] POST + NTLM Authentica
Joe thanks, that's helpful and appreciated - we'll go 3.0. Fingers
crossed.
-Original Message-
From: [EMAIL PROTECTED]
Sent: 10 July 2008 14:53
To: squid-users@squid-cache.org
Subject: RE: [squid-users] POST + NTLM Authentica
--
--
Bill,
If you want to test upgrading to 3.0 if you don't have a preference
either way, I'll test 2.7 (I'm currently using 2.6 Stable 20) as I can't
move to 3 yet because of some of the reverse proxy limitations.
Here's hoping! :)
Joe
-Original Message-
From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]
Sent: Thursday 10 July 2008 14:35
To: [EMAIL PROTECTED]
Cc: [EMAIL PROTECTED]; squid-users@squid-cache.org
Subject: RE: [squid-users] POST + NTLM Authentica
Thanks. Pressure of work means that it will be a few days before I can
upgrade, but I will let you know when done. What should I consider when
deciding whether to go 3.0 or 2.7?
-Original Message-
From: [EMAIL PROTECTED]
Sent: 10 July 2008 12:44
To: Bill Allison
Cc: [EMAIL PROTECTED]; squid-users@squid-cache.org
Subject: Re: [squid-users] POST + NTLM Authentica
--
--
[EMAIL PROTECTED] wrote:
Hi
I think this may be the same as I reported as bug 2176
(http://www.squid-cache.org/bugs/show_bug.cgi?id=2176) against 2.6
STABLE17. Is it known to be fixed in 2.7 or 3.0?. Apologies for asking
-
I haven't had time yet to go through changes logs for subsequent
versions.
Ah if thats the bug, then unkown. It's still open. Primarily because we
have not ben able to get a usable trace in the current code to find out
whats causing it.
If you upgrade you will either see it disappear; at which point we can
get the others to test 2175 by upgrading themselves. Maybe closing the
issue off.
Or worst case it remains, at which point you are using a version we can
support you while debug tracking the issue down.
Amos
Bill A.
-Original Message-
From: [EMAIL PROTECTED]
Sent: 10 July 2008 05:28
To: [EMAIL PROTECTED]
Cc: squid-users@squid-cache.org
Subject: Re: [squid-users] POST + NTLM Authentica
-
-
--
Luiz Felipe Ferreira wrote:
Guys,
We have a squid-2.5STABLE14 with NTLM authentication.
Please upgrade. 2.5 has been obsolete for several years.
We currently support primarily 3.0stable7, and 2.7stable3 for those
who
can't upgrade to 3.0x.
Either of which is a good upgrade form 2.5.
Amos
--
Please use Squid 2.7.STABLE3 or 3.0.STABLE7
--
This message has been scanned for viruses and dangerous content by
MailScanner, and is believed to be clean.
--
Please use Squid 2.7.STABLE3 or 3.0.STABLE7
--
This message has been scanned for viruses and dangerous content by
MailScanner, and is believed to be clean.
__
This incoming email was virus scanned for HESA by MessageLabs.
__
_
Higher Education Statistics Agency Ltd (HESA) is a company limited by
guarantee, registered in England at 95 Promenade Cheltenham GL50 1HZ.
Registered No. 2766993. The members are Universities UK and GuildHE.
Registered Charity No. 1039709. Certified to ISO 9001 and BS 7799.
HESA Services Ltd (HSL) is a wholly owned subsidiary of HESA, registered
in England at the same address. Registered No. 3109219.
_
This outgoing email was virus scanned for HESA by MessageLabs.
_
--
This message has been scanned for viruses and dangerous content by
MailScanner, and is believed to be clean.
__
This incoming email
RE: [squid-users] POST + NTLM Authentica
Bill, If you want to test upgrading to 3.0 if you don't have a preference either way, I'll test 2.7 (I'm currently using 2.6 Stable 20) as I can't move to 3 yet because of some of the reverse proxy limitations. Here's hoping! :) Joe -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] Sent: Thursday 10 July 2008 14:35 To: [EMAIL PROTECTED] Cc: [EMAIL PROTECTED]; squid-users@squid-cache.org Subject: RE: [squid-users] POST + NTLM Authentica Thanks. Pressure of work means that it will be a few days before I can upgrade, but I will let you know when done. What should I consider when deciding whether to go 3.0 or 2.7? -Original Message- From: [EMAIL PROTECTED] Sent: 10 July 2008 12:44 To: Bill Allison Cc: [EMAIL PROTECTED]; squid-users@squid-cache.org Subject: Re: [squid-users] POST + NTLM Authentica -- -- [EMAIL PROTECTED] wrote: Hi I think this may be the same as I reported as bug 2176 (http://www.squid-cache.org/bugs/show_bug.cgi?id=2176) against 2.6 STABLE17. Is it known to be fixed in 2.7 or 3.0?. Apologies for asking - I haven't had time yet to go through changes logs for subsequent versions. Ah if thats the bug, then unkown. It's still open. Primarily because we have not ben able to get a usable trace in the current code to find out whats causing it. If you upgrade you will either see it disappear; at which point we can get the others to test 2175 by upgrading themselves. Maybe closing the issue off. Or worst case it remains, at which point you are using a version we can support you while debug tracking the issue down. Amos Bill A. -Original Message- From: [EMAIL PROTECTED] Sent: 10 July 2008 05:28 To: [EMAIL PROTECTED] Cc: squid-users@squid-cache.org Subject: Re: [squid-users] POST + NTLM Authentica - - -- Luiz Felipe Ferreira wrote: Guys, We have a squid-2.5STABLE14 with NTLM authentication. Please upgrade. 2.5 has been obsolete for several years. We currently support primarily 3.0stable7, and 2.7stable3 for those who can't upgrade to 3.0x. Either of which is a good upgrade form 2.5. Amos -- Please use Squid 2.7.STABLE3 or 3.0.STABLE7 -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. -- Please use Squid 2.7.STABLE3 or 3.0.STABLE7 -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. __ This incoming email was virus scanned for HESA by MessageLabs. __ _ Higher Education Statistics Agency Ltd (HESA) is a company limited by guarantee, registered in England at 95 Promenade Cheltenham GL50 1HZ. Registered No. 2766993. The members are Universities UK and GuildHE. Registered Charity No. 1039709. Certified to ISO 9001 and BS 7799. HESA Services Ltd (HSL) is a wholly owned subsidiary of HESA, registered in England at the same address. Registered No. 3109219. _ This outgoing email was virus scanned for HESA by MessageLabs. _
RE: [squid-users] httpReadReply: Request not yet fully sentPOSThttp://xxx/yyy.php
Hi Henrik, Just today I came across this bug report submitted in January http://www.squid-cache.org/bugs/show_bug.cgi?id=2176 which appears to match my situation almost exactly. I'm at home at the moment so can't check exactly, but it appears that squid keeps the connection open and the client continues sending the request. Only sometimes (but noticeably often) does the connection appear to be ungracefully closed and the browser errors saying it was reset. I haven't packet sniffed squid - webserver yet, but I can do and post the results if that would help. It only appears to reset the connection on the larger posts, the guy who originally posted the bug report said it's posts over 8kb, I've yet to verify that, but certainly smaller sized posts work without issue. Yeah I'm aware that the MS auth schemes weren't exactly designed to work in harmony with HTTP authentication! Unfortunately needs must!! If you can offer any advice, or let me know what else I can to do help diagnose the issue, I'd be most grateful. Cheers Joe -Original Message- From: Henrik Nordstrom [mailto:[EMAIL PROTECTED] Sent: Sunday 6 July 2008 22:54 To: Joe Tiedeman Cc: squid-users@squid-cache.org Subject: RE: [squid-users] httpReadReply: Request not yet fully sentPOSThttp://xxx/yyy.php; On tor, 2008-07-03 at 15:00 +0100, Joe Tiedeman wrote: It seems to be that IIS is sending the 401 response before squid the client have finished sending the initial request to it, after sniffing the traffic with wireshark on the client, squid is forwarding the 401 response before the client has finished posting the data. The interesting things is what happens after the 401 response. Do Squid close the connection before the client sent all of the request, or is the connection kept open allowing the client to continue sending the request? What about the connection squid-webserver? The microsoft schemes NTLM / Negotiate and Kerberos is a bit at odds with how HTTP authentication works, which causes some quite odd corner cases.. How things are supposed to work in the HTTP way is that the connection is kept open and request data being read, but the client when seeing the 401 should immediately abort the transfer (by closing the connection) and try again with correct credentials. This can not be done in the connection oriented auth schemes and the client must instead transmit the whole request, even when it's known it is now going into the bitbucket.. may not be such a big deal when on a LAN/Intranet, but if over a WAN it can be very annoying.. Regards Henrik _ Higher Education Statistics Agency Ltd (HESA) is a company limited by guarantee, registered in England at 95 Promenade Cheltenham GL50 1HZ. Registered No. 2766993. The members are Universities UK and GuildHE. Registered Charity No. 1039709. Certified to ISO 9001 and BS 7799. HESA Services Ltd (HSL) is a wholly owned subsidiary of HESA, registered in England at the same address. Registered No. 3109219. _ This outgoing email was virus scanned for HESA by MessageLabs. _
RE: [squid-users] httpReadReply: Request not yet fully sent POSThttp://xxx/yyy.php
Hi Guys, I've also begun experiencing this issue with a few sites that we host internally, we have a Mediawiki and a Joomla CMS installation both of which use Windows Integrated Authentication (Kerberos not NTLM) behind a squid reverse proxy. The error seems to show up when doing a large POST such as uploading an image to the wiki or updating a large article in Joomla and is quite often followed by an error in Firefox saying that the connection was reset. It seems to be that IIS is sending the 401 response before squid the client have finished sending the initial request to it, after sniffing the traffic with wireshark on the client, squid is forwarding the 401 response before the client has finished posting the data. I'm really at a loss as to what we can do to either fix or work around the issue. I can't stop using WIA as it's the basis for all our single sign on sites. If there's anything else that anyone suggest I would really appreciate it! If I can help by providing any more information, please let me know Cheers Joe Joe Tiedeman Support Analyst Higher Education Statistics Agency (HESA) 95 Promenade, Cheltenham, Gloucestershire GL50 1HZ T 01242 211167 F 01242 211122 W www.hesa.ac.uk -Original Message- From: Henrik Nordstrom [mailto:[EMAIL PROTECTED] Sent: Wednesday 13 June 2007 22:56 To: Sean Walberg Cc: squid-users@squid-cache.org Subject: Re: [squid-users] httpReadReply: Request not yet fully sent POSThttp://xxx/yyy.php; ons 2007-06-13 klockan 07:45 -0500 skrev Sean Walberg: httpReadReply: Request not yet fully sent POST http://xxx/yyy.php; -xxx varies, yyy.php is usually the same (most of our POSTs are to the same script anyway) Reading up on it a bit tells me that this means that the web server has returned data before squid finished POSTing the form. Yes. This is usually a PMTU problem in forward-cache scenarios though. I wouldn't expect PMTU discovery to be a problem on an Ethernet segment where all devices have the same MTU. No. PMTU is not relevant here at all. How the script behaves is relevant. If the script responds before reading the complete request then the above message will be seen. This may occur if a) The script fails while reading the request or b) The script doesn't really care what the POST data looks like, ignoring it. or c) The web server responded with an error. My initial inclination is to get a packet capture, but these errors are unpredictable so I might be sifting through a lot of data, and I'm not even sure what it would tell me. The most important piece it will tell you is what the response from the script actually looked like when this problem is seen. This will tell you if the problem is the script / web server, or if the problem is related to Squid. Regards Henrik _ Higher Education Statistics Agency Ltd (HESA) is a company limited by guarantee, registered in England at 95 Promenade Cheltenham GL50 1HZ. Registered No. 2766993. The members are Universities UK and GuildHE. Registered Charity No. 1039709. Certified to ISO 9001 and BS 7799. HESA Services Ltd (HSL) is a wholly owned subsidiary of HESA, registered in England at the same address. Registered No. 3109219. _ This outgoing email was virus scanned for HESA by MessageLabs. _
RE: [squid-users] Microsoft OLE DB Provider for SQL Server error '80040e57'
I haven't visited the site without going through our Squid box, but I wonder if they have some kind of db logging on people accessing the site, maybe whatever it's trying to log has issues with the requests that squid is sending? I think it's unlikely to be an issue with Squid itself. Joe -Original Message- From: Kinkie [mailto:[EMAIL PROTECTED] Sent: 03 March 2008 16:54 To: Squid Users Subject: Re: [squid-users] Microsoft OLE DB Provider for SQL Server error '80040e57' On Mon, Mar 3, 2008 at 2:37 PM, Hardie, Ian [EMAIL PROTECTED] wrote: Hi, I'm getting a very strange occurrence when going through our Squid proxy, when I try and access this website via squid: http://www.fleetlineoffers.co.uk/ I get the following error: Microsoft OLE DB Provider for SQL Server error '80040e57' String or binary data would be truncated. /includes/asp_page_logging.asp, line 122 I was convinced it was an error with the website but sure enough, if I bypass squid it opens fine. I've tried adding an always_direct to squid.conf but it hasn't helped. Any ideas? This looks 100% like an application error. Only case where it may be squid-related was if the proxy administrator was playing weird tricks with the error messages or with redirectors. -- /kinkie __ This incoming email was virus scanned for HESA by MessageLabs. __ No virus found in this outgoing message. Checked by AVG Free Edition. Version: 7.5.516 / Virus Database: 269.21.3/1307 - Release Date: 02/03/2008 15:59 _ Higher Education Statistics Agency Ltd (HESA) is a company limited by guarantee, registered in England at 95 Promenade Cheltenham GL50 1HZ. Registered No. 2766993. The members are Universities UK and GuildHE. Registered Charity No. 1039709. Certified to ISO 9001 and BS 7799. HESA Services Ltd (HSL) is a wholly owned subsidiary of HESA, registered in England at the same address. Registered No. 3109219. _ This outgoing email was virus scanned for HESA by MessageLabs. _
[squid-users] Reverse Proxy Redirector Help
Hi All,
I'm using Squid 2.6 Stable 18 on RHEL4 in a reverse proxy configuration
and haven't been able to find any redirectors out there which do exactly
what I would like, so I'm in the process of writing my own very simple
one in php (code below - it's by NO means complete, I'm still in the
really early development stages)
It seems to be working fine so far, just outputting to my logfile
everything which squid passes to it, however every minute (to the
second) each of the spawned redirectors puts this into the cache.log and
I can't for the life of me figure out why!
helperHandleRead: unexpected read from url_rewriter #2, 1 bytes '
It also seems that every now and again squid sends a black request to
the redirectors as they put blank lines into the temporary logfile.
It could be that I'm doing something very stupid (highly likely!) but I
was wondering if anyone could help.
Many Thanks in advance!
Joe
?php
$stdin = fopen('php://stdin', 'r');
while (1) {
#$stdin = fopen('php://stdin', 'r');
$line = trim(fgets($stdin));
$filename = '/tmp/phpredirector.log';
$somecontent = $line.\n;
// Let's make sure the file exists and is writable first.
if (is_writable($filename)) {
// In our example we're opening $filename in append mode.
// The file pointer is at the bottom of the file hence
// that's where $somecontent will go when we fwrite() it.
if (!$handle = fopen($filename, 'a')) {
echo couldn't open the file;
}
// Write $somecontent to our opened file.
if (fwrite($handle, $somecontent) === FALSE) {
echo couldn't write to the file;
}
fclose($handle);
echo PHP_EOL;
} else {
exit;
}
}
?
_
Higher Education Statistics Agency Ltd (HESA) is a company limited by
guarantee, registered in England at 95 Promenade Cheltenham GL50 1HZ.
Registered No. 2766993. The members are Universities UK and GuildHE.
Registered Charity No. 1039709. Certified to ISO 9001 and BS 7799.
HESA Services Ltd (HSL) is a wholly owned subsidiary of HESA,
registered in England at the same address. Registered No. 3109219.
_
This outgoing email was virus scanned for HESA by MessageLabs.
_
