Squid 3.1.0.2 problem with ICAP Posst method data
Hi All,
I am interfacing disturbing problem with squid 3.1 ICAP.
I seems that HTTP POST method does not works properly.
while taking a close look at the sniffer I notice that
post data, though returned unchanged from the ICAP Server,
is not sent to the web server, and results to a problematic behavior,
in the following case there is a login problem.
I am using standard ICAP Configuration as follows:
icap_enable on
icap_preview_enable off
icap_send_client_ip on
icap_send_client_username on
icap_client_username_header X-Authenticated-User
icap_client_username_encode on
icap_service service_req reqmod_precache 0 icap://127.0.0.1:1344/request
adaptation_service_set class_req service_req
adaptation_access class_req allow all
icap_service service_resp respmod_precache 0
icap://127.0.0.1:1345/response
adaptation_service_set class_resp service_resp
adaptation_access class_resp allow all
The ICAP request looks OK:
REQMOD icap://127.0.0.1:1344/request ICAP/1.0
Host: 127.0.0.1:1344
Date: Thu, 12 Feb 2009 12:33:34 GMT
Proxy-Authorization: Basic bW9zaGViOjEyMzQ1Ng==
Encapsulated: req-hdr=0, req-body=567
Allow: 204
X-Client-IP: 10.101.1.241
X-Authenticated-User: bW9zaGVi
POST http://lovin.co.il/reguser/loginForm.do HTTP/1.1
Host: lovin.co.il
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.1.20)
Gecko/20081217 Firefox/2.0.0.20
Accept:
text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plai
n;q=0.8,image/png,*/*;q=0.5
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Referer: http://lovin.co.il/reguser/loginForm.do
Cookie: JSESSIONID=CE9319848513B6B494B8EF992E9C10C7
Content-Type: application/x-www-form-urlencoded
Content-Length: 30
1e
usrName=%D8%91password=11
0
ICAP/1.0 200 OK
ISTAG: PS_CF_ICAP-5.1-46-142
Cache-Control: post-check=0, pre-check=0, false
Pragma: no-cache
X-Response-Info: 4
ICAP-Session-Info:
PElHX1BDQ0lDQVBDbGllbnRTZXNzaW9uQ29udGV4dE1zZyBibG9ja01vZGU9IjQiIHJlZGly
ZWN0VXJsPSIiIHBvbGljeT0iIiBzZXNzaW9uSUQ9IiIgYWN0aXZlQ2F0ZWdvcmllcz0iMTA3
NTg2MzY2NSIgaW5nbm9yZUNhdGVnb3JpZXM9IjAiIHdhcm5DYXRlZ29yaWVzPSIwIiBhdXRo
b3JpemF0aW9uPSIwIi8+Cg==
X-Category: 1
X-Attribute: 1
Encapsulated: req-hdr=0, null-body=536
POST http://lovin.co.il/reguser/loginForm.do HTTP/1.1
Host: lovin.co.il
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.1.20)
Gecko/20081217 Firefox/2.0.0.20
Accept:
text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plai
n;q=0.8,image/png,*/*;q=0.5
Accept-Language: en-us,en;q=0.5
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Referer: http://lovin.co.il/reguser/loginForm.do
Cookie: JSESSIONID=CE9319848513B6B494B8EF992E9C10C7
Content-Type: application/x-www-form-urlencoded
Content-Length: 30
1e
usrName=%D8%91password=11
0
But the HTTP Header is sent to the web server (port 80) without the POST
DATA
POST /reguser/loginForm.do HTTP/1.0
Host: lovin.co.il
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.1.20)
Gecko/20081217 Firefox/2.0.0.20
Accept:
text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plai
n;q=0.8,image/png,*/*;q=0.5
Accept-Language: en-us,en;q=0.5
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Referer: http://lovin.co.il/reguser/loginForm.do
Cookie: JSESSIONID=CE9319848513B6B494B8EF992E9C10C7
Content-Type: application/x-www-form-urlencoded
Content-Length: 30
Via: 1.1 PSSquid (squid/3.1.0.3)
X-Forwarded-For: 10.101.1.241
Cache-Control: max-age=259200
Connection: keep-alive
{HERE SHOULD COME THE MISSING DATA [usrName=%D8%91password=11]}
Is it know issue?
How do I bypass it?
can I configure squid so HTTP POST request will not go through
adaptation process? how?
Thank you,
Moshe Beeri.
PureSight.com
RE: ICAP, bypassing respmod depending reqmod result
Moshe Beeri wrote:
Hi,
I am trying to add extension to ICAP layer in squid 3.0 STABLE
10.
Our service works both in respmod and reqmod, in some cases we
know
right after reqmod that the data should passed directly to the
client.
in those cases we like to be more efficient by not utilizing
respmod
at
all, sending the server's response directly to the client,
bypassing
ICAP server's respmod.
Does any one have an idea how to decided after 'reqmod reply'
whether http server response will be sent to ICAP server's
respmod
or
to
send directly to client?
Any suggestions how to trick the mechanism?
You can add a header to the http request during reqmod processing
eg
X-ICAP-Respmod-Needed: False and play with a squid req_header
acl
to
allow/deny respmod processing:
acl AnXIcapHeader req_header X-ICAP-Respmod-Needed False
icap_access class_respmod deny AnXIcapHeader
Regards,
Christos
Thanks Christos,
Your solution, thought elegant, is not secure since the web server
might
change the parameter and cause unsafe data transfer to client.
can I utilize acl to do the same by adding header to ICAP (reqmod)
response?
No the web server can not change the parameter:
- During the ReqMod ICAP request you are modifying the http request
adding an extra header (the X-ICAP-Respmod-Needed or any other you
want)
- Squid will use the modified (by your ICAP server) requests when
going
to evaluate the AnXIcapHeader acl. The http server can not modify
the
http request.
Hi Christos,
First I would like to thank you for you response it is really
appreciated.
You are right the web server can not change the request,
but it can utilize the data to trick out filter.
We also would like to keep the changes in the scope between
the Squid and the ICAP server and not pollutes the traffic
that goes out to the Internet.
In the worst case scenario we will use your acl solution,
but we prefer to add those capabilities into squid 3.0 source, and to
benefit the community as well.
Regards,
Moshe Beeri.
Hi Christos,
I found the right place to make this efficient change.
ServerStateData::adaptOrFinalizeReply()
{
#if ICAP_CLIENT
if (TheICAPConfig.onoff) {
//start
if(request-bypass_respmod) {
setFinalReply(virginReply());
return;
}
//end
ICAPAccessCheck *icap_access_check =
new ICAPAccessCheck(ICAP::methodRespmod,
ICAP::pointPreCache,
request, virginReply(), icapAclCheckDoneWrapper, this);
icapAccessCheckPending = true;
icap_access_check-check(); // will eventually delete self
return;
}
#endif
setFinalReply(virginReply());
}
Regards,
Moshe Beeri,
Regards,
Christos
In 2.5 I changed the code using fde object associated with
client_fd
to
pass context information from reqmod to respmod, disabling ICAP by
returning a constant instead of icap_writer, but with 3.0 code
improvement this hock has gone or I am missing it, is there any
why
to
do it in 3.0 ?
Regards,
Moshe.
Thanks Allot,
Moshe Beeri.
RE: ICAP, bypassing respmod depending reqmod result
Moshe Beeri wrote: Hi, I am trying to add extension to ICAP layer in squid 3.0 STABLE 10. Our service works both in respmod and reqmod, in some cases we know right after reqmod that the data should passed directly to the client. in those cases we like to be more efficient by not utilizing respmod at all, sending the server's response directly to the client, bypassing ICAP server's respmod. Does any one have an idea how to decided after 'reqmod reply' whether http server response will be sent to ICAP server's respmod or to send directly to client? Any suggestions how to trick the mechanism? You can add a header to the http request during reqmod processing eg X-ICAP-Respmod-Needed: False and play with a squid req_header acl to allow/deny respmod processing: acl AnXIcapHeader req_header X-ICAP-Respmod-Needed False icap_access class_respmod deny AnXIcapHeader Regards, Christos Thanks Christos, Your solution, thought elegant, is not secure since the web server might change the parameter and cause unsafe data transfer to client. can I utilize acl to do the same by adding header to ICAP (reqmod) response? No the web server can not change the parameter: - During the ReqMod ICAP request you are modifying the http request adding an extra header (the X-ICAP-Respmod-Needed or any other you want) - Squid will use the modified (by your ICAP server) requests when going to evaluate the AnXIcapHeader acl. The http server can not modify the http request. Hi Christos, First I would like to thank you for you response it is really appreciated. You are right the web server can not change the request, but it can utilize the data to trick out filter. We also would like to keep the changes in the scope between the Squid and the ICAP server and not pollutes the traffic that goes out to the Internet. In the worst case scenario we will use your acl solution, but we prefer to add those capabilities into squid 3.0 source, and to benefit the community as well. Regards, Moshe Beeri. Regards, Christos In 2.5 I changed the code using fde object associated with client_fd to pass context information from reqmod to respmod, disabling ICAP by returning a constant instead of icap_writer, but with 3.0 code improvement this hock has gone or I am missing it, is there any why to do it in 3.0 ? Regards, Moshe. Thanks Allot, Moshe Beeri.
RE: ICAP, bypassing respmod depending reqmod result
Moshe Beeri wrote: Hi, I am trying to add extension to ICAP layer in squid 3.0 STABLE 10. Our service works both in respmod and reqmod, in some cases we know right after reqmod that the data should passed directly to the client. in those cases we like to be more efficient by not utilizing respmod at all, sending the server's response directly to the client, bypassing ICAP server's respmod. Does any one have an idea how to decided after 'reqmod reply' whether http server response will be sent to ICAP server's respmod or to send directly to client? Any suggestions how to trick the mechanism? You can add a header to the http request during reqmod processing eg X-ICAP-Respmod-Needed: False and play with a squid req_header acl to allow/deny respmod processing: acl AnXIcapHeader req_header X-ICAP-Respmod-Needed False icap_access class_respmod deny AnXIcapHeader Regards, Christos Thanks Christos, Your solution, thought elegant, is not secure since the web server might change the parameter and cause unsafe data transfer to client. can I utilize acl to do the same by adding header to ICAP (reqmod) response? In 2.5 I changed the code using fde object associated with client_fd to pass context information from reqmod to respmod, disabling ICAP by returning a constant instead of icap_writer, but with 3.0 code improvement this hock has gone or I am missing it, is there any why to do it in 3.0 ? Regards, Moshe. Thanks Allot, Moshe Beeri.
ICAP, bypassing respmod depending reqmod result
Hi, I am trying to add extension to ICAP layer in squid 3.0 STABLE 10. Our service works both in respmod and reqmod, in some cases we know right after reqmod that the data should passed directly to the client. in those cases we like to be more efficient by not utilizing respmod at all, sending the server's response directly to the client, bypassing ICAP server's respmod. Does any one have an idea how to decided after 'reqmod reply' whether http server response will be sent to ICAP server's respmod or to send directly to client? Any suggestions how to trick the mechanism? Thanks Allot, Moshe Beeri.
RE: ICAP Parental Control Extension, patch to 2.5 STABLE 10
Hi Adrian, I have trying to do the same to squid 2.6 stable X, but it seems that The ICAP patch is not working fine, configure do not response to --enable-icap-support I tried to fix things up with no success so far :-( I have working out with all the heavy guns and I get unexpected compilation errors. Do you have any idea what I am doing wrong ( [squid_dir]#patch -p1 icap.2_6.patch ) Any way I published my work for squid-icap with squid-2.5.STABLE10 as well as squid-2.6.STABLE5 to the following address http://files.puresight.com/squids/ I will be on [EMAIL PROTECTED] from now on. And I'll be happy assist with squid 3 10x allot, Moshe Beeri. -Original Message- From: Adrian Chadd [mailto:[EMAIL PROTECTED] Sent: Monday, December 04, 2006 2:45 PM To: Moshe Beeri Subject: Re: ICAP Parental Control Extension, patch to 2.5 STABLE 10 On Mon, Dec 04, 2006, Moshe Beeri wrote: Hi Adrian, I will try to grab some time at the next few days to take a look at 2.6 and 3.0. I'll try to apply those changes to 2.6 and 3.0 Do you suggests I'll work on any particular version? Does the code look familiar? 2.6 looks like 2.5; you should look at the Squid-2.6 icap patch and see if you can improve and use that in production for now. Squid-3 should be where you focus ICAP work for the time being. It has ICAP support which a handful of people want to present as a product to customers. I think it'd be good to have more people working on improving Squid-3 and bringing its ICAP implementation up to date. Adrian -- - Xenion - http://www.xenion.com.au/ - Hosting and Commercial Squid Support -
RE: ICAP Parental Control Extension, patch to 2.5 STABLE 10
Hi Puchar, I included the whole ICAP extension, since the original patch couldn't patch To STABLE 10, I thought it might be helpful for some. Do you still think it is too match?? I used the following command to generate the patch: diff -urNp squid-2.5.STABLE10 squid-icap-2.5.STABLE10 PURESIGHT_squid-icap-2.5.STABLE10.patch Moshe. -Original Message- From: Wojciech Puchar [mailto:[EMAIL PROTECTED] Sent: Sunday, December 03, 2006 11:45 PM To: Moshe Beeri Cc: squid-dev@squid-cache.org Subject: Re: ICAP Parental Control Extension, patch to 2.5 STABLE 10 how did you managed to make 4.4MB patch? On Sun, 3 Dec 2006, Moshe Beeri wrote: Hi All, Patch file for squid 2.5 stable 10 is attached. This patch file should be patch using the following command in the squid-2.5-STABLE10 folder using the following command: Patch -p1 PURESIGHT_squid-icap-2.5.STABLE10.patch I included a simple squid.conf file for example of use. I hope the patch will be usefull and that it will be used. I have left the build script I used to compile and test, PSMakefile, you can run it as is, it will configure and make. The following features where added [All changes were made in the separate of squid dev team]: 1. Direct response to client in case the server permits at req_mod. 2. Client context information that will be passed to client at resp_mod. Read the conf file, changes are explained at the bottom of the file. The main changes in squid are: file function name remarks icap_reqmod.c icapReqModReadIcapPart icap_direct_response and icap_server_session_context extracting and handling icap_respmod.cicapRespModStart decides if to go direct, if not lets icap-respmod.icap_server_session_context to handle the context getICAPRespModString writes the context to ICAP resp mod request header. icap_common.c icapStateFree release (if needed) icap_server_session_context http.clook 4 icapRespModStart when calling icapRespModStart it may have -2 in return, they it continues with a normal http flow. structs.h fde store our information regarding each session _IcapStateData store our information for ICAP resp_mod session _IcapConfig stop icap related configuration, should be coordinated with cf.data.pre in case you like to add new. cf.data.pre icap_req_mod_direct_reply - icap_req_mod_direct_reply_resp_info_tag_name- icap_req_mod_direct_reply_values -all our added configuration. icap_session_context_tag_name - For any other Q's do not hesitate to email me. [EMAIL PROTECTED] Moshe Beeri, [EMAIL PROTECTED] Senior Software Engineer, Servers and Networking team leader. PureSight Technologies Ltd, http://www.puresight.com http://www.puresight.com/ 16 Basel st. P.O.B. 4145 Petah Tikva 49130 ISRAEL
RE: Squid-dev subscription
Hi All Squiders, In the following couple of weeks I will post some new features that are needed for parental control, as well as context connectivity from req_mod to resp_mod. But the major change I made is to add the ability to direct response to client in case that classification is already known at req_mod. The changes I made are on 2.5 Stable 10. I hope you will be able to use those features to Squid benefit. Thanks allot, Moshe Beeri. SW Engineer, PureSight Technologies. -Original Message- From: Adrian Chadd [mailto:[EMAIL PROTECTED] Sent: Saturday, October 21, 2006 5:47 PM To: Gernot Tenchio Cc: squid-dev@squid-cache.org Subject: Re: Squid-dev subscription On Fri, Oct 20, 2006, Gernot Tenchio wrote: Hi, My name is Gernot. I'm a developer from Germany and I'm interested in getting the ICAP part of Squid to work. Currently I'm tried to get things working with Squid's 2.6er icap branch and with Squid3 as well. Both versions worked more or less for me, but I think it (c|sh)ould be improved. Welcome! :) More ICAP developers are always welcome. Adrian
context data for squid ICAP patched
Hi all squiders, I found the right way to implement the content filter extention to squid. I will continue the implementation and will send it to our QA Team, I hope the whole squid community will use this extention to squids ICAP capabilities. Best regards, Moshe Beeri. Software Engineer, Server Team. [EMAIL PROTECTED] Petach-Tiqva Bazel 16, Israel. Tel: +972 (3) 928-0400 ext. 429 Fax: +972 (3) 921-7594 Hi Christos, Thank you for your help, but you suggestion is not secure nor best perform, Please read my other remarks below. Now that I read the question again I see it is not clear enough, I will ask again. I would like squid ICAP client to do the logic for couple of reasons, 1. Security - Origin sever might change the replied http header and add the X-MY-SCANNER: Allow it self, and bypass the content filter, In that case I would not be able to prevent kids from viewing un honest pages :-( 2. Performance - Redundant call since I already know that request is allowed. There for I would like to keep in squids session data the classification and upon to the classification prior to response-mod call. For now I have figured out that the best place to set the data between the req-mod and resp-mod if in the fde structure, but since squid saves that object in fd_table (hash?) keyed by ICAP FD there is no continuity with the HTTP FD. I realized that I need to look for the mechanism that changes the next handler (hdl) that switch FD to read from, is the KEY to set up the fde related to the HTTP response, with the classification information. In squid ICAP client implementation there is no connection between the FD sets, ICAP's and HTTP's. Again 10X for reading and good will, I hope there is a short cut out there, If someone has an implementation suggestion or realizes I am missing something please write me. Hi Beeri, Maybe you do not need to modify the squid-ICAP code to support your model. I think that the correct implementation of your problem using squid-ICAP is: 1) An http request come into the squid. Squid sends the reqmod request to the ICAP server and server classifies the request: a) In the case of the BLOCK ICAP server creates a http response saying to the web client that the request blocked b) In the case of UNKNOWN ICAP server does nothing c) In the case of ALLOW ICAP server adds a proprietary http header to the http request for example X-MY-SCANNER: Allow 2) When squid has the http response then sends a respmod request to the ICAP server. The respmod request contains the http response headers AND the http request headers. a) When ICAP server founds the X-MY-SCANNER: Allow header in http request headers it responds with an allow204 response to squid b)The X-MY-SCANNER: Allow is not in the http request headers so the ICAP server takes the http body from squid and check it or modify it or what else. An other solutions is to use only the respmod request because here you have both the http request headers and the http response. The Question: I would like to pass the information that, no call to response mode (call the ICAP Server for the response) is needed. ... I am not sure that I fully understand your question, but I think that this functionality can not included in a general ICAP client of squid. But maybe I am loosing something here. Regards, Christos Background information: I am implementing an extension to squid ICAP client based upon ICAP Patch and squid 2.5 STABLE 10. The squid ICAP client does not support Content Filtering the way we at PureSight.com using it. The ICAP protocol is defined to support also Content Filtering and defines a return value at the request mod stage. I receive the value that can be one of the following: ALLOW, BLOCK, UNKNOWN ..
I would like to subscribe to squid mailing list
Hi, I am Moshe Beeri, and I work for PureSight (www.PureSight.com) Recently I applied the ICAP patch into squid 2.5 stable 10, We at PureSight are using ICAP for content filtering and we are using our own server, as I worked with squid ICAP I realized that the support for content filtering is not yet defined. As I walked trough the code I realized that I need some more information about squid functions and data storage. I would like to pass the classification info from the request mod to the response mod, preventing unnecessary call to request mod in case the classification is known to be ALLOW. We, at PureSight, contribute the development applying our solution to the community. I hope that this solution be best to squid lovely community as well as squid users. Thanks, Moshe Beeri. Software Engineer, Server Team. [EMAIL PROTECTED] Petch-Tiqva Bazel 16, Israel. Tel: +972 (3) 928-0400 ext. 429 Fax: +972 (3) 921-7594
context data for squid ICAP patched
Hi all squids, Background information: I am implementing an extension to squid ICAP client based upon ICAP Patch and squid 2.5 STABLE 10. The squid ICAP client does not support Content Filtering the way we at PureSight.com using it. The ICAP protocol is defined to support also Content Filtering and defines a return value at the request mod stage. I receive the value that can be one of the following: ALLOW, BLOCK, UNKNOWN So far as described by the ICAP protocol (RFC 3507) and implemented by our ICAP server. As for squid client I added the ability to identify classification value, I would like squid to behave according to that value: 1. In case of UNKNOWN the response should go back to the ICAP server to be reclassified. 2. In case of BLOCK the ICAP server is changing the URL to retrieve a blocking information page. 3. In case of ALLOW squid should reply directly to the client with or without using the cache. The Question: I would like to pass the information that, no call to response mode (call the ICAP Server for the response) is needed. I would like your help with the way squid saves session data, I look at the code and I could not find that particular object, is there one? Any suggestions || directions would be blessed. The Polite part: Thanks for reading Participating on that grate project of squid. Your help Moshe Beeri. Software Engineer, Servers Team. [EMAIL PROTECTED] Petch-Tiqva Bazel 16, Israel. Tel: +972 (3) 928-0400 ext. 429 Fax: +972 (3) 921-7594
RE: context data for squid ICAP patched
Hi Christos, Thank you for your help, but you suggestion is not secure nor best perform, Please read my other remarks below. Now that I read the question again I see it is not clear enough, I will ask again. I would like squid ICAP client to do the logic for couple of reasons, 1. Security - Origin sever might change the replied http header and add the X-MY-SCANNER: Allow it self, and bypass the content filter, In that case I would not be able to prevent kids from viewing un honest pages :-( 2. Performance - Redundant call since I already know that request is allowed. There for I would like to keep in squids session data the classification and upon to the classification prior to response-mod call. For now I have figured out that the best place to set the data between the req-mod and resp-mod if in the fde structure, but since squid saves that object in fd_table (hash?) keyed by ICAP FD there is no continuity with the HTTP FD. I realized that I need to look for the mechanism that changes the next handler (hdl) that switch FD to read from, is the KEY to set up the fde related to the HTTP response, with the classification information. In squid ICAP client implementation there is no connection between the FD sets, ICAP's and HTTP's. Again 10X for reading and good will, I hope there is a short cut out there, If someone has an implementation suggestion or realizes I am missing something please write me. Hi Beeri, Maybe you do not need to modify the squid-ICAP code to support your model. I think that the correct implementation of your problem using squid-ICAP is: 1) An http request come into the squid. Squid sends the reqmod request to the ICAP server and server classifies the request: a) In the case of the BLOCK ICAP server creates a http response saying to the web client that the request blocked b) In the case of UNKNOWN ICAP server does nothing c) In the case of ALLOW ICAP server adds a proprietary http header to the http request for example X-MY-SCANNER: Allow 2) When squid has the http response then sends a respmod request to the ICAP server. The respmod request contains the http response headers AND the http request headers. a) When ICAP server founds the X-MY-SCANNER: Allow header in http request headers it responds with an allow204 response to squid b)The X-MY-SCANNER: Allow is not in the http request headers so the ICAP server takes the http body from squid and check it or modify it or what else. An other solutions is to use only the respmod request because here you have both the http request headers and the http response. The Question: I would like to pass the information that, no call to response mode (call the ICAP Server for the response) is needed. ... I am not sure that I fully understand your question, but I think that this functionality can not included in a general ICAP client of squid. But maybe I am loosing something here. Regards, Christos Background information: I am implementing an extension to squid ICAP client based upon ICAP Patch and squid 2.5 STABLE 10. The squid ICAP client does not support Content Filtering the way we at PureSight.com using it. The ICAP protocol is defined to support also Content Filtering and defines a return value at the request mod stage. I receive the value that can be one of the following: ALLOW, BLOCK, UNKNOWN ..
