Re: Squid-ICAP problem (bug?)
Hi all,
Tsantilas Christos wrote :
If you are still looking for the solution, try the following patch in
icap_reqmod.c file. With this patch the icapReqModPassHttpBody function
called if the icap-reqmod.http_entity.buf.size is zero, and at this
phase can handle correctly, the case that the
icap-flags.reqmod_http_entity_eof==1.
Unfortunately it still segfaults.
But, I had time to work on my client's test platform and I spent a whole
day hacking icap_reqmod.c.
The only modification (except adding excessive debug logging) is that one:
in icapReqModReadHttpBody function replacing the test
if (icap-reqmod.http_entity.bytes_read = icap-request-content_length)
by
if (icap-chunk_size 0)
which correct the initial problem ie not reading the 0\r\n marking thee
end of the chunk.
The new problem is that squid correctly reads the end of the chunk but
dies immediatly.
The dying occurs in the icapReqModPassHttpBody function and I spent many
time to find why. In fact I focussed on the call of this function from
the icapReqModReadHttpBody function and I didn't manage to find the
logic of the test and the function parameters (because I don't know well
the squid and icap patch code).
Anyway this I found that the function was called with icap = 0 parameter
(I spent too more time to search a buf=0 or callback=0); so I added a
simple test in the beginning of the function:
static void
icapReqModPassHttpBody(IcapStateData * icap, char *buf, size_t size,
CBCB * callback, void *cbdata)
{
debug(81, 3) (%s:%d: icapReqModPassHttpBody: called\n, __FILE__,
__LINE__);
if (!icap) {
debug(81, 1) (%s:%d: icapReqModPassHttpBody: FD %d called with
icap = (nil) !\n, __FILE__, __LINE__);
return;
}
I added this to prevent segfaulting but without knowing if the return
would lead to a correct behaviour. And it seems that is is ok. We made
some tests and we did not have any more segfaults.
I then tried with gdb (I should have used it before but I am not
familiar with it) to get a backtrace as you asked in a previous mail and
I saw that the icapReqModPassHttpBody call with icap=0 is not called
from the icapReqModReadHttpBody:
Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread -1218496992 (LWP 9508)]
0x08087411 in icapReqModPassHttpBody (icap=0x0, buf=0x0,
size=4294967295, callback=0, cbdata=0x8c95e00) at icap_reqmod.c:933
933 icap_reqmod.c: Aucun fichier ou repertoire de ce type.
in icap_reqmod.c
(gdb) bt
#0 0x08087411 in icapReqModPassHttpBody (icap=0x0, buf=0x0,
size=4294967295, callback=0, cbdata=0x8c95e00) at icap_reqmod.c:933
#1 0x08083905 in requestAbortBody (request=0x8c94578) at HttpRequest.c:215
#2 0x0807a183 in httpStateFree (fd=9, data=0x8cc72a0) at http.c:69
#3 0x08065ccd in commCallCloseHandlers (fd=44) at comm.c:573
#4 0x08065dec in comm_close (fd=44) at comm.c:655
#5 0x0807b32c in httpReadReply (fd=44, data=0x8cc72a0) at http.c:852
#6 0x08067749 in comm_poll (msec=13) at comm_select.c:445
#7 0x0808fc77 in main (argc=4, argv=0xbfff9494) at main.c:754
Here is a sumup of the logs when icap value is tested and squid does not
segfault anymore:
2006/12/01 18:18:19| icapReqModParseHttpRequest: successfully parsed the
HTTP request
2006/12/01 18:18:19| handing request bodies in ICAP REQMOD
2006/12/01 18:18:19| icap_reqmod.c:859: icapReqModReadHttpBody: FD 53 called
2006/12/01 18:18:19| icap_reqmod.c:861: icapReqModReadHttpBody: read
returns 33
2006/12/01 18:18:19| icap_reqmod.c:872: icapReqModReadHttpBody: Before
icapParseChunkedBody bytes_read = 0
2006/12/01 18:18:19| icap_common.c:695: chunk_size=0
2006/12/01 18:18:19| icap_common.c:702: bufOffset=0, len=33
2006/12/01 18:18:19| icap_common.c:705: bufOffset=0, len=33
2006/12/01 18:18:19| icap_common.c:604: icapParseChunkSize:
buf=0x8ccd1b8, len=33
2006/12/01 18:18:19| icap_common.c:607: icapParseChunkSize: start=0
2006/12/01 18:18:19| icapLineLength: returning 4
2006/12/01 18:18:19| icap_common.c:646: icapParseChunkSize: start=0, end=2
2006/12/01 18:18:19| icap_common.c:672: icapParseChunkSize: return
nextStart=4
2006/12/01 18:18:19| icap_common.c:716: got chunksize 27, new offset 4
2006/12/01 18:18:19| icap_common.c:723: X
2006/12/01 18:18:19| icap_common.c:705: bufOffset=31, len=33
2006/12/01 18:18:19| icap_common.c:604: icapParseChunkSize:
buf=0x8ccd1d7, len=2
2006/12/01 18:18:19| icap_common.c:607: icapParseChunkSize: start=0
2006/12/01 18:18:19| icapLineLength: returning 2
2006/12/01 18:18:19| icap_common.c:607: icapParseChunkSize: start=2
2006/12/01 18:18:19| icap_reqmod.c:876: icapReqModReadHttpBody: After
icapParseChunkedBody bytes_read = 27
2006/12/01 18:18:19| icap_reqmod.c:884 icapReqModReadHttpBody: chunk_size=0
Here is the first modification: even if we read the 27 bytes of the body
we continue reading because chunk_size is not -2:
2006/12/01 18:18:19| icap_reqmod.c:892 icapReqModReadHttpBody: call
commSetSelect -
Re: Squid-ICAP problem (bug?)
Tsantilas Christos a e'crit :
Hello Christos,
Maybe I was not clear on my last mail. Sorry.
You were but I forgot to mention that I tried what you adviced.
But in the same function replace the test:
if (icap-reqmod.http_entity.callback
icap-reqmod.http_entity.buf.size) {
icapReqModPassHttpBody(icap,
icap-reqmod.http_entity.callback_buf,
icap-reqmod.http_entity.callback_bufsize,
with:
if (icap-reqmod.http_entity.callback) {
icapReqModPassHttpBody(icap,
icap-reqmod.http_entity.callback_buf,
icap-reqmod.http_entity.callback_bufsize,
Did you try it?
Yes we tried it and this leads to segfault too.
and we exit the icapReqModReadHttpBody function without calling
icapReqModPassHttpBody because http_entity.buf.size is 0.
With the previous change this function must called here.
Did you make this change? Are you still seeing segfaults?
I did not test extensively but it seems that calling
icapReqModPassHttpBody without testing if http_entity.buf.size is 0 will
work in this very specific case (when end chunk is read in an extra
pass) but will lead to segfault in cases which used to work perfectly
before changing this test.
In my tests I am not seeing any problem any more. But maybe I am losing
something
I don't have any magic recipe to reproduce the problem but it occurs
when parsing the request's body in REQMOD and that occurs generally when
doint a POST request (as there is no body in a GET request).
With your modification, we ends with segfaults during GET request !
By reading the log I just noticed that with your modification the
icapReqModPassHttpBody is not called:
This is for the specific request (POST):
2006/11/24 14:15:22| handing request bodies in ICAP REQMOD
2006/11/24 14:15:22| icapReqModReadHttpBody: FD 38 called
2006/11/24 14:15:22| icapReqModReadHttpBody: read returns 33
2006/11/24 14:15:22| icap_common.c:695: chunk_size=0
2006/11/24 14:15:22| icap_common.c:702: bufOffset=0, len=33
2006/11/24 14:15:22| icap_common.c:705: bufOffset=0, len=33
2006/11/24 14:15:22| icapParseChunkSize: buf=0x997fbe0, len=33
2006/11/24 14:15:22| icapParseChunkSize: start=0
2006/11/24 14:15:22| icapLineLength: returning 4
2006/11/24 14:15:22| icapParseChunkSize: start=0, end=2
2006/11/24 14:15:22| icapParseChunkSize: return nextStart=4
2006/11/24 14:15:22| got chunksize 27, new offset 4
2006/11/24 14:15:22| icap_common.c:723: X
2006/11/24 14:15:22| icap_common.c:705: bufOffset=31, len=33
2006/11/24 14:15:22| icapParseChunkSize: buf=0x997fbff, len=2
2006/11/24 14:15:22| icapParseChunkSize: start=0
2006/11/24 14:15:22| icapLineLength: returning 2
2006/11/24 14:15:22| icapParseChunkSize: start=2
2006/11/24 14:15:22| icap_reqmod.c:882 chunk_size=0
2006/11/24 14:15:22| icap_reqmod.c:892 http_entity.callback=(nil)
2006/11/24 14:15:22| icap_reqmod.c:894 http_entity.buf.size=27
2006/11/24 14:15:22| icap_reqmod.c:896 http_entity.callback_bufsize=0
2006/11/24 14:15:22| icapReqModReadHttpBody: FD 38 called
2006/11/24 14:15:22| icapReqModReadHttpBody: read returns 5
2006/11/24 14:15:22| icap_common.c:695: chunk_size=0
2006/11/24 14:15:22| icap_common.c:702: bufOffset=0, len=7
2006/11/24 14:15:22| icap_common.c:705: bufOffset=0, len=7
2006/11/24 14:15:22| icapParseChunkSize: buf=0x997fbe0, len=7
2006/11/24 14:15:22| icapParseChunkSize: start=0
2006/11/24 14:15:22| icapLineLength: returning 2
2006/11/24 14:15:22| icapParseChunkSize: start=2
2006/11/24 14:15:22| icapLineLength: returning 3
2006/11/24 14:15:22| icapParseChunkSize: start=2, end=3
2006/11/24 14:15:22| icapParseChunkSize: return nextStart=5
2006/11/24 14:15:22| got chunksize -2, new offset 5
2006/11/24 14:15:22| zero end chunk reached
2006/11/24 14:15:22| icap_reqmod.c:882 chunk_size=-2
2006/11/24 14:15:22| icap_reqmod.c:892 http_entity.callback=(nil)
2006/11/24 14:15:22| icap_reqmod.c:894 http_entity.buf.size=27
2006/11/24 14:15:22| icap_reqmod.c:896 http_entity.callback_bufsize=0
2006/11/24 14:15:22| icapSendRespMod: Create a new connection to icap
server service_4/icap://xx.xx.xx.xx:1344/wwrespmod/?wwprofile=HTTP_
But what is very strange is:
- respmod seems not to be called;
- the next request which is a GET is handled strangely:
2006/11/24 14:15:34| icapParseEncapsulated: res-hdr=-1, req-hdr=0,
null-body=791, res-body=-1, req-body=-1, opt-body=-1
2006/11/24 14:15:34| icapReqModReadIcapPart: directResponse=0 from ICAP
server service_3/icap://xx.xx.xx.xx:1344/wwreqmod/?wwprofile=HTTP_
2006/11/24 14:15:34| icapReqModReadHttpHdrs:
2006/11/24 14:15:34| expect=791
2006/11/24 14:15:34| so_far=0
2006/11/24 14:15:34| needed=791
2006/11/24 14:15:34| icapReqModReadHttpHdrs: read 791 bytes
2006/11/24 14:15:34| icapReqModReadHttpHdrs: read the entire request headers
2006/11/24 14:15:34| icapReqModParseHttpRequest: URI is
'http://x.xxx/rtn_FR.gif'
2006/11/24 14:15:34| icapReqModParseHttpRequest: Client HTTP version 1.1.
2006/11/24 14:15:34|
Re: ICAP Parental Control Extension, patch to 2.5 STABLE 10
how did you managed to make 4.4MB patch? On Sun, 3 Dec 2006, Moshe Beeri wrote: Hi All, Patch file for squid 2.5 stable 10 is attached. This patch file should be patch using the following command in the squid-2.5-STABLE10 folder using the following command: Patch -p1 PURESIGHT_squid-icap-2.5.STABLE10.patch I included a simple squid.conf file for example of use. I hope the patch will be usefull and that it will be used. I have left the build script I used to compile and test, PSMakefile, you can run it as is, it will configure and make. The following features where added [All changes were made in the separate of squid dev team]: 1. Direct response to client in case the server permits at req_mod. 2. Client context information that will be passed to client at resp_mod. Read the conf file, changes are explained at the bottom of the file. The main changes in squid are: file function name remarks icap_reqmod.c icapReqModReadIcapPart icap_direct_response and icap_server_session_context extracting and handling icap_respmod.cicapRespModStart decides if to go direct, if not lets icap-respmod.icap_server_session_context to handle the context getICAPRespModString writes the context to ICAP resp mod request header. icap_common.c icapStateFree release (if needed) icap_server_session_context http.clook 4 icapRespModStart when calling icapRespModStart it may have -2 in return, they it continues with a normal http flow. structs.h fde store our information regarding each session _IcapStateDatastore our information for ICAP resp_mod session _IcapConfig stop icap related configuration, should be coordinated with cf.data.pre in case you like to add new. cf.data.pre icap_req_mod_direct_reply - icap_req_mod_direct_reply_resp_info_tag_name- icap_req_mod_direct_reply_values -all our added configuration. icap_session_context_tag_name - For any other Q's do not hesitate to email me. [EMAIL PROTECTED] Moshe Beeri, [EMAIL PROTECTED] Senior Software Engineer, Servers and Networking team leader. PureSight Technologies Ltd, http://www.puresight.com http://www.puresight.com/ 16 Basel st. P.O.B. 4145 Petah Tikva 49130 ISRAEL
Re: ICAP Parental Control Extension, patch to 2.5 STABLE 10
Nice work! Is there any particular reason why you didn't work on Squid-2.6? There's already an ICAP patch available for Squid-2.6 which could be improved on. Have you looked at Squid-3 at all? Adrian On Sun, Dec 03, 2006, Moshe Beeri wrote: Hi All, Patch file for squid 2.5 stable 10 is attached. This patch file should be patch using the following command in the squid-2.5-STABLE10 folder using the following command: Patch -p1 PURESIGHT_squid-icap-2.5.STABLE10.patch I included a simple squid.conf file for example of use. I hope the patch will be usefull and that it will be used. I have left the build script I used to compile and test, PSMakefile, you can run it as is, it will configure and make. The following features where added [All changes were made in the separate of squid dev team]: 1. Direct response to client in case the server permits at req_mod. 2. Client context information that will be passed to client at resp_mod. Read the conf file, changes are explained at the bottom of the file. The main changes in squid are: file function name remarks icap_reqmod.c icapReqModReadIcapPart icap_direct_response and icap_server_session_context extracting and handling icap_respmod.cicapRespModStart decides if to go direct, if not lets icap-respmod.icap_server_session_context to handle the context getICAPRespModString writes the context to ICAP resp mod request header. icap_common.c icapStateFree release (if needed) icap_server_session_context http.clook 4 icapRespModStart when calling icapRespModStart it may have -2 in return, they it continues with a normal http flow. structs.h fde store our information regarding each session _IcapStateDatastore our information for ICAP resp_mod session _IcapConfig stop icap related configuration, should be coordinated with cf.data.pre in case you like to add new. cf.data.pre icap_req_mod_direct_reply - icap_req_mod_direct_reply_resp_info_tag_name- icap_req_mod_direct_reply_values -all our added configuration. icap_session_context_tag_name - For any other Q's do not hesitate to email me. [EMAIL PROTECTED] Moshe Beeri, [EMAIL PROTECTED] Senior Software Engineer, Servers and Networking team leader. PureSight Technologies Ltd, http://www.puresight.com http://www.puresight.com/ 16 Basel st. P.O.B. 4145 Petah Tikva 49130 ISRAEL -- - Xenion - http://www.xenion.com.au/ - Hosting and Commercial Squid Support -
Re: Modifications to the ICAP client I would like to see
I Forgot to mention this is against Squid3 HEAD. Sorry, Axel Am 03.12.2006 15:19 Uhr schrieb Axel Westerhold unter [EMAIL PROTECTED]: Hi everyone, attached you'll find 3 modifications against HEAD I would like to have: A.) ICAPServiceRep::TheSessionFailureLimit set through squid.conf B.) ICAPServiceRep delay for a down service set through squid.conf C.) Instead of hardcoding the Header used to transfer the username being able to set the used one through squid.conf I tested the patch the best I could. The defaults are as hardcoded before. A simple web poligraph test did not show any problems. Regards, Axel Westerhold
