Squid-ICAP
Hello, I'm interrested in the ICAP patch for SQUID-Cache and I would like to know if the developpement for squid 3.x will start. What kind of help will be the more needed? Regards, Ghislain Garçon. [EMAIL PROTECTED]
squid ICAP developer
Hello all, I am Tsantilas Christos (chtsanti at users dot sourceforge dot net), I am working with ICAP protocol and I am interested to help the development of ICAP client for squid. Thanks, Christos
squid-icap crash...
This mail posted to c-icap mailing list. I believe that it is related with ICAP's request modification operation. It does not contain enough info but sometimes it is useful to just know the problems. - Christos Mateus Gröess wrote: Hi, Christos Today I was looking in cache.log of Squid ICAP and found another message that was followed by a Squid restart. ... Unfortunally there wasn't messages between the error and the normal Squid startup. 2005/04/04 14:07:00| storeLateRelease: released 0 objects 2005/04/04 17:12:31| assertion failed: client_side.c:3268: "cbdataValid(conn)" 2005/04/04 17:12:42| Starting Squid Cache version 2.5.STABLE9-CVS for i386-slackware-linux-gnu...
Re: Squid-ICAP
On Tue, 13 Sep 2005, Ghislain Garcon wrote: Hello, I'm interrested in the ICAP patch for SQUID-Cache and I would like to know if the developpement for squid 3.x will start. What kind of help will be the more needed? Hi Ghislain, I am working on ICAP for Squid-3. The code is in the sourceforge CVS repository with tag 'squid3-icap'. I don't think it is very useable yet, but in a while perhaps you can run some tests with Squid talking to an ICAP server. Duane W.
Re: Squid-ICAP
Hi, does the ICAP team plan to implement load balancing for v3? If yes, will it only be round-robin or a different protocol?// Thanks. Ghislain. Duane Wessels a écrit : On Tue, 13 Sep 2005, Ghislain Garcon wrote: Hello, I'm interrested in the ICAP patch for SQUID-Cache and I would like to know if the developpement for squid 3.x will start. What kind of help will be the more needed? Hi Ghislain, I am working on ICAP for Squid-3. The code is in the sourceforge CVS repository with tag 'squid3-icap'. I don't think it is very useable yet, but in a while perhaps you can run some tests with Squid talking to an ICAP server. Duane W.
Re: Squid-ICAP
On Thu, 2005-09-15 at 17:10 +0200, Ghislain Garçon wrote: > does the ICAP team plan to implement load balancing for v3? > If yes, will it only be round-robin or a different protocol?// The plan is to support basic ICAP first, without load balancing complications. The ICAP framework being developed should simplify adding load balancing features later. Do you expect to contribute ICAP load-balancing algorithms? Thank you, Alex. > Duane Wessels a écrit : > > On Tue, 13 Sep 2005, Ghislain Garcon wrote: > > > >> Hello, > >> > >> I'm interrested in the ICAP patch for SQUID-Cache and I would like > >> to know if the developpement for squid 3.x will start. What kind of > >> help will be the more needed? > > > > > > Hi Ghislain, > > > > I am working on ICAP for Squid-3. The code is in the sourceforge CVS > > repository > > with tag 'squid3-icap'. > > > > I don't think it is very useable yet, but in a while perhaps you can > > run some tests with Squid talking to an ICAP server. > > > > Duane W. > > > > >
Squid-ICAP 2.6
Hello, I would like to give a try to the 2.6 branch of the ICAP patch and I didn't manage to apply the patch available at: http://devel.squid-cache.org/projects.html#icap I tried on squid-2.6.STABLE9-20070220, squid-2.6.STABLE9 and squid-2.6.STABLE8 without success. Is there anywhere: - an history if the icap patch ? - a changelog - a documentation explaining on which squid version an icap patch apply ? Regards, Christophe.
debugging Squid ICAP interface
Hello, My name is Marcus Kool, author of ufdbGuard - a URL redirector for Squid, and I have started development of an ICAP-based URL filter. As with all new developments, the code of the ICAP server undoubtedly has some bugs that need to be investigated and fixed. I also have seen Squid behaving unexpectedly (2 minutes timeouts where it seems not handle any request from a browser, assertion failure). I have various observations and questions about the Squid ICAP interface and like to discuss these with the persons who wrote or know much about the ICAP client part of Squid. I like to know with whom I can discuss this and which mailing list to use. Thanks, Marcus
Squid-ICAP client developer
Hi, I've been lately coding an ICAP server for my company, and would be interested in following Squid's ICAP client developments, and/or to submit patchs in that area. Best regards, /olivier
Re: squid ICAP developer
On Thu, 3 Mar 2005, Tsantilas Christos wrote: I am Tsantilas Christos (chtsanti at users dot sourceforge dot net), I am working with ICAP protocol and I am interested to help the development of ICAP client for squid. You have now been given access to the devel.squid-cache.org services. While you wait for the permissions to get updated (can take up to 12 hours) please read the documents on how to use the services and rules http://devel.squid-cache.org/CVS.html (tools, routines, howto tips etc) http://devel.squid-cache.org/rules.html (usage guidelines) And please subscribe to the squid-dev mailinglist by sending a message to [EMAIL PROTECTED] Regards Henrik
Re: squid-icap crash...
I will provide a better bug report. I believe I found the problem with generation of coredumps. On Apr 6, 2005 6:07 PM, Tsantilas Christos <[EMAIL PROTECTED]> wrote: > This mail posted to c-icap mailing list. > I believe that it is related with ICAP's request modification operation. > It does not contain enough info but sometimes it is useful > to just know the problems. > - > Christos > > Mateus Gröess wrote: > > >Hi, Christos > > > > Today I was looking in cache.log of Squid ICAP and found > >another message that was followed by a Squid restart. ... > >Unfortunally there wasn't messages between the error and the normal > >Squid startup. > > > >2005/04/04 14:07:00| storeLateRelease: released 0 objects > >2005/04/04 17:12:31| assertion failed: client_side.c:3268: > >"cbdataValid(conn)" > >2005/04/04 17:12:42| Starting Squid Cache version 2.5.STABLE9-CVS for > >i386-slackware-linux-gnu... > > > > > > > >
Re: squid-icap crash...
Mateus send me the backtrace On Apr 5, 2005 10:05 AM, Mateus Gröess wrote: 2005/04/04 14:07:00| storeLateRelease: released 0 objects 2005/04/04 17:12:31| assertion failed: client_side.c:3268: "cbdataValid(conn)" 2005/04/04 17:12:42| Starting Squid Cache version 2.5.STABLE9-CVS for i386-slackware-linux-gnu... Here is the stack trace that I think is the same problem of the assertion above: 2005/04/07 11:22:36| storeLateRelease: released 0 objects 2005/04/07 11:23:38| assertion failed: client_side.c:3268: "cbdataValid(conn)" Program received signal SIGABRT, Aborted. 0x402299f1 in __kill () from /lib/libc.so.6 (gdb) backtrace #0 0x402299f1 in __kill () from /lib/libc.so.6 #1 0x402296d4 in raise (sig=6) at ../sysdeps/posix/raise.c:27 #2 0x4022ae31 in abort () at ../sysdeps/generic/abort.c:88 #3 0x80736c7 in xassert (msg=0x80e15ed "cbdataValid(conn)", file=0x80defdc "client_side.c", line=3268) at debug.c:270 #4 0x806cfbe in clientReadBody (request=0x8502f38, buf=0x84c56d8 "", size=8192, callback=0x809cad4 , cbdata=0x85390d0) at client_side.c:3268 #5 0x809cacc in icapReqModSendBodyChunk (fd=26, bufnotused=0x0, size=627, errflag=0, data=0x85390d0) at icap_reqmod.c:758 #6 0x806ee9a in CommWriteStateCallbackAndFree (fd=26, code=0) at comm.c:99 #7 0x8071380 in commHandleWrite (fd=26, data=0x84f5830) at comm.c:929 #8 0x8072656 in comm_poll (msec=268) at comm_select.c:459 #9 0x80a7aae in main (argc=2, argv=0xbb34) at main.c:748 #10 0x4021a2eb in __libc_start_main (main=0x80a7664 , argc=2, ubp_av=0xbb34, init=0x804a6a4 <_init>, fini=0x80d7a1c <_fini>, rtld_fini=0x4000c130 <_dl_fini>, stack_end=0xbb2c) at ../sysdeps/generic/libc-start.c:129 (gdb) quit The program is running. Exit anyway? (y or n) y
Squid/icap 2.5 CVS
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Hello, I have troubles compiling the latest CVS of icap/2.5, do you have it also ? gcc -DHAVE_CONFIG_H - -DDEFAULT_CONFIG_FILE=\"/usr/local/squid/etc/squid.conf\" -I. -I. - -I../include -I. -I. -I../include -I../include-g -O2 -Wall -c `test - -f dns_internal.c || echo './'`dns_internal.c dns_internal.c:58: error: syntax error before "rfc1035_query" dns_internal.c:58: warning: no semicolon at end of struct or union dns_internal.c:72: error: syntax error before '}' token dns_internal.c: In function `idnsStats': dns_internal.c:345: error: dereferencing pointer to incomplete type dns_internal.c:345: error: dereferencing pointer to incomplete type dns_internal.c:345: error: dereferencing pointer to incomplete type dns_internal.c:346: error: dereferencing pointer to incomplete type dns_internal.c:347: error: dereferencing pointer to incomplete type Thanks in advance, /olivier - -- Olivier Girondel <[EMAIL PROTECTED]> Project Leader Dolphian -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.0 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFCiIRfpqVXaJzJYNIRAoMmAJ0Y6rknFt6mkiIzTycb/xqlsNVeOQCfR4XS pQ2Jh73aKcxDQamxsOvtIEE= =Owx5 -END PGP SIGNATURE-
Squid-ICAP problem (bug?)
Hello, I think I have found a bug in the Squid-ICAP patch and I would like to have your opinion about it. I use a tcpdump strace and verbose log to track a problem which occurs sometimes during a respmod request but is triggered during the reqmod answer analysis I think. We use squid-2.5.STABLE14 and Squid-Icap patch from 2006/06/26. Here is the request sent by Squid to the ICAP server (webwasher): --- begin cut --- REQMOD icap://10.17.30.41:1344/wwreqmod/?wwprofile=HTTP_Sortante ICAP/1.0 Encapsulated: req-hdr=0, req-body=1021 X-Client-IP: 129.182.109.109 X-Authenticated-User: V2lOVDovL2VyaWM= POST http://bubv-u.fraz.bull.fr/ulysse/SelTypF2.jsp HTTP/1.1 Accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, application/x-shockwave-flash, application/vnd.ms-excel, application/vnd.ms-powerpoint, application/msword, */* Referer: http://bubv-u.fraz.bull.fr/ulysse/SelTypF1.jsp Accept-Language: fr Content-Type: application/x-www-form-urlencoded Accept-Encoding: gzip, deflate User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322) Host: bubv-u.fraz.bull.fr Content-Length: 27 Proxy-Connection: Keep-Alive Pragma: no-cache Cookie: REFUTI=27930; LANGUE=FR; NOM=BUL78604FR; STE=BUL; USER=UBUL_BUL78604FR; JSESSIONID=FE83A4E238E9375B7A9C55F6EE76C7EA; WS_LastUid=frank-e; WS_UsrRef=FRANK-E; WS_UsrLng=FR; WS_UsrSid=9743; WS_UsrAut=ClBiRTt7Nz59IjBscjXnrypOQWMAAwd7cnRcW3lZNFd8T3t3JGtgbSxMPXFyc3RUNFhzfjR%2bcjxQVjRcZWEmTGBLLVI6KCM8SHYoTjhSXm97WVtBOnggb3B1KEdhbyZ7ZS1cLywvIDZoLE5lNSxtK09YXEY1QlE1fHBobztxfEsyPSxcTA== Proxy-Authorization: Basic ZXJpYzplcmlj 1b GOTO=&TRI=&ORDER=&COD=&LIB= 0 end cut The request seems ok, the string "GOTO=&TRI=&ORDER=&COD=&LIB=" is 27 characters long as specified in the Content-Lenght header. The request is sent successfully to webwasher: 2006/11/02 15:10:23| icapReqModSendBodyChunk: FD 34 wrote 1208 errflag 0. 2006/11/02 15:10:23| icapReqModBodyHandler: writing chunk size 27 2006/11/02 15:10:23| icapReqModSendBodyChunk: FD 34 wrote 33 errflag 0. 2006/11/02 15:10:23| icapReqModBodyHandler: writing chunk size 0 2006/11/02 15:10:23| icapSendReqModDone: FD 34: size 5: errflag 0. Then comes the response from webwasher. The problem is that the response seems to be split in several packets. Here is the first part: --- begin cut --- ICAP/1.0 200 OK Cache-Control: max-age=3600 Encapsulated: req-hdr=0, req-body=1021 ISTAG: "001-000-000162" X-Attribute-Cacheability: user=V2lOVDovL2VyaWM= X-ICAP-Profile: HTTP_Sortante POST http://bubv-u.fraz.bull.fr/ulysse/SelTypF2.jsp HTTP/1.1 Accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, application/x-shockwave-flash, application/vnd.ms-excel, application/vnd.ms-powerpoint, application/msword, */* Accept-Encoding: gzip, deflate Accept-Language: fr Content-Length: 27 Content-Type: application/x-www-form-urlencoded Cookie: REFUTI=27930; LANGUE=FR; NOM=BUL78604FR; STE=BUL; USER=UBUL_BUL78604FR; JSESSIONID=24FE81492AD05440B0434197D087AC18; WS_LastUid=frank-e; WS_UsrRef=FRANK-E; WS_UsrLng=FR; WS_UsrSid=3168; WS_UsrAut=ClBiRTt7Nz59IjBsckIn7iooyDAAAwd7cnRcW3lZNFd8T3t3JGtgbSxMPXFyc3RUNFhzfjR%2bcjxQVjRcZWEmTGBLLVI6KCM8SHYoTjhSXm97WVtBOnggb3B1KEdhbyZ7ZS1cLywvIDZoLE5lNSxtK09YXEY1QlE1fHBobztxfEsyPSxcTA== Host: bubv-u.fraz.bull.fr Pragma: no-cache Proxy-Authorization: Basic ZXJpYzplcmlj Proxy-Connection: Keep-Alive Referer: http://bubv-u.fraz.bull.fr/ulysse/SelTypF1.jsp User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322) 1B GOTO=&TRI=&ORDER=&COD=&LIB= end cut And the second part: --- begin cut --- 0 end cut The problem is that the end of the chunk (0\r\n\r\n) is not sent (and read) in the same stream that the beginning of the chunk. If we dig in the squid-icap source code we see: 2006/11/02 15:10:23| icapReqModReadHttpBody: FD 34 called 2006/11/02 15:10:23| icapReqModReadHttpBody: read returns 33 -> this is the first part read 2006/11/02 15:10:23| icap_common.c:695: chunk_size=0 2006/11/02 15:10:23| icap_common.c:702: bufOffset=0, len=33 2006/11/02 15:10:23| icap_common.c:705: bufOffset=0, len=33 2006/11/02 15:10:23| icapParseChunkSize: buf=0x8f832e8, len=33 2006/11/02 15:10:23| icapParseChunkSize: start=0 2006/11/02 15:10:23| icapLineLength: returning 4 -> this is the part "1b\r\n" 2006/11/02 15:10:23| icapParseChunkSize: start=0, end=2 -> this is "1b" 2006/11/02 15:10:23| icapParseChunkSize: return nextStart=4 2006/11/02 15:10:23| got chunksize 27, new offset 4 -> 1b <-> 27 bytes, to read at offset 4 (after "1b\r\n") 2006/11/02 15:10:23| icap_common.c:723: X 2006/11/02 15:10:23| icap_common.c:705: bufOffset=31, len=33 2006/11/02 15:10:23| icapParseChunkSize: buf=0x8f83307, len=2 2006/11/02 15:10:23| icapParseChunkSize: start=0 2006/11/02 15:10:23| icapLineLength: returning 2 2006/11/02 15:10:23|
Re: Squid-ICAP 2.6
Bonjour Jeremy, Here is the answer I got, and I managed to patch squid correctly thanks to it: http://www.squid-cache.org/mail-archive/squid-users/200702/0143.html Would you have a corrected patch ? Because I am applying manually and it is a bit long and not obvious to not make a mistake. Christophe.
Re: Squid-ICAP 2.6
Christophe Boyanique a écrit : Hello, I would like to give a try to the 2.6 branch of the ICAP patch and I didn't manage to apply the patch available at: http://devel.squid-cache.org/projects.html#icap I tried on squid-2.6.STABLE9-20070220, squid-2.6.STABLE9 and squid-2.6.STABLE8 without success. Is there anywhere: - an history if the icap patch ? - a changelog - a documentation explaining on which squid version an icap patch apply ? Regards, Christophe. Here is the answer I got, and I managed to patch squid correctly thanks to it: http://www.squid-cache.org/mail-archive/squid-users/200702/0143.html Hope it helps -- Jérémy Lardon Laboratoire DIOM, équipe SATIn - Doctorant 04 77 48 50 34
Re: Squid-ICAP 2.6
Hi Christophe, I think the best is to get the icap branch from cvs: cvs -d :pserver:[EMAIL PROTECTED]:/cvsroot/squid co -r icap-2_6 squid After that run the bootstrap.sh script which included in cvs sources. Do not try to correct rejected patches. The last merge with main squid26 done after STABLE9 released and after some more changes added by squid developers to the main squid26 branch. The current patch does not apply to any stable release but to the latest(I think) squid26 cvs sources. Sorry about that. Regards, Christos Christophe Boyanique wrote: > > Hello, > > I would like to give a try to the 2.6 branch of the ICAP patch and I > didn't manage to apply the patch available at: > > http://devel.squid-cache.org/projects.html#icap > > I tried on squid-2.6.STABLE9-20070220, squid-2.6.STABLE9 and > squid-2.6.STABLE8 without success. > > Is there anywhere: > > - an history if the icap patch ? > - a changelog > - a documentation explaining on which squid version an icap patch apply ? > > Regards, > > Christophe. >
Re: Squid-ICAP 2.6
tis 2007-02-20 klockan 15:05 +0100 skrev Christophe Boyanique: > I would like to give a try to the 2.6 branch of the ICAP patch and I > didn't manage to apply the patch available at: > > http://devel.squid-cache.org/projects.html#icap > > I tried on squid-2.6.STABLE9-20070220, squid-2.6.STABLE9 and > squid-2.6.STABLE8 without success. > > Is there anywhere: > > - an history if the icap patch ? > - a changelog I preliminary history/changelog view of the devel.squid-cache.org projects can be found here: http://www.squid-cache.org/~hno/changesets/squid/ > - a documentation explaining on which squid version an icap patch apply ? From the above: Squid-2 around 2007/01/31. Regards Henrik signature.asc Description: Detta är en digitalt signerad meddelandedel
Re: debugging Squid ICAP interface
tis 2010-10-12 klockan 14:51 -0300 skrev Marcus Kool: > I have various observations and questions about the Squid ICAP interface > and like to discuss these with the persons who wrote or know much about > the ICAP client part of Squid. > I like to know with whom I can discuss this and which mailing list to use. This list is the right place for such discussions. Regards Henrik
Re: debugging Squid ICAP interface
On 10/12/2010 11:51 AM, Marcus Kool wrote: I also have seen Squid behaving unexpectedly (2 minutes timeouts where it seems not handle any request from a browser, assertion failure). If you can reproduce this, please file a bug report with details, especially if this happens to a recent Squid version. I have various observations and questions about the Squid ICAP interface and like to discuss these with the persons who wrote or know much about the ICAP client part of Squid. I like to know with whom I can discuss this and which mailing list to use. As Henrik said, you came to the right place. Cheers, Alex.
Christian Kunst - Squid iCAP Client
Hi, My name is Christian Kunst, and I would like to join the Squid team. I am interested in helping develop the Squid iCAP client. Please let me know what are the next steps that I should take. Thank you and best regards, Christian Kunst.
Fwd: Squid ICAP client problems
-- Forwarded Message --
Subject: FW: Squid ICAP client problems
Date: Mon, 09 Jun 2003 14:35:27 -0400
From: "Rosenbaum, Larry M." <[EMAIL PROTECTED]>
To: [EMAIL PROTECTED]
Could you please forward this to the Squid ICAP developers, or tell
me whom to contact?
Thanks, Larry
-Original Message-
From: Rosenbaum, Larry M.
Sent: Thursday, June 05, 2003 12:57 PM
To: '[EMAIL PROTECTED]'; '[EMAIL PROTECTED]'
Subject: Squid ICAP client problems
Are you still involved with Squid ICAP client development? I have
been trying to use the Squid ICAP patch (v1.2.1 with Squid 2.5) to
do virus scanning and have discovered the following two problems:
1) (icap.c) There is a problem with the way icapRespModReadReply()
tries to read the ICAP header without reading past the header. The
current code does a recv() with the MSG_PEEK flag and a read_sz of
256 bytes, and if the buffer doesn't contain a double CRLF it
doubles the read_sz and tries again. The problem with this approach
is that it takes time for the socket to receive more bytes, and
there is nothing in the loop to ensure that more bytes are read.
For example,
try #1 read_sz = 256, len = 38
try #2 read_sz = 512, len = 38 (i.e. it reads the same 38 bytes
again) try #3 read_sz = 1024, len = 38
...
try #n read_sz = max, len = 38
so it is possible to exit the loop after many iterations without
reading any more bytes than you got on the first try. One possible
fix would be to turn off MSG_PEEK and read repeatedly until you get
to the end of the headers, and then pass the extra bytes to the
function that parses the rest of the message, but I don't know if
the code structure permits this.
The same issue probably applies to icapReqModReadReply().
2) (cache_cf.c) If you don't have a "icap_class" line in the config
file, squid will crash if you do a "-k reconfigure". This is because
the class structure has an element with a null iter->name. Here is
one possible fix:
*** cache_cf.c.030603 Mon Jun 2 13:47:54 2003
--- cache_cf.c Tue Jun 3 12:48:16 2003
***
*** 2291,2297
{
icap_service *iter;
for (iter = Config.icapcfg.service_head; iter; iter =
iter->next) {
! if (! strcmp(name, iter->name)) {
return iter;
}
}
--- 2291,2297
{
icap_service *iter;
for (iter = Config.icapcfg.service_head; iter; iter =
iter->next) {
! if ((iter->name) && (!strcmp(name, iter->name))) {
return iter;
}
}
***
*** 2432,2438
{
icap_class *iter;
for (iter = Config.icapcfg.class_head; iter; iter = iter->next)
{ ! if ((!strcmp(name, iter->name)) && (!iter->hidden)) {
return iter;
}
}
--- 2432,2438
{
icap_class *iter;
for (iter = Config.icapcfg.class_head; iter; iter = iter->next)
{ ! if ((iter->name) && (!strcmp(name, iter->name)) &&
(!iter->hidden)) {
return iter;
}
}
If you are not the correct people to report these bugs to, please
pass this along or tell me who to send it to.
Do you know of any virus scanning ICAP servers that work with the
Squid ICAP client?
Thanks, Larry
---
Re: Squid/icap 2.5 CVS
On Mon, 16 May 2005, olivier wrote: gcc -DHAVE_CONFIG_H - -DDEFAULT_CONFIG_FILE=\"/usr/local/squid/etc/squid.conf\" -I. -I. - -I../include -I. -I. -I../include -I../include-g -O2 -Wall -c `test - -f dns_internal.c || echo './'`dns_internal.c dns_internal.c:58: error: syntax error before "rfc1035_query" This error seems to indicate your include/rfc1035.h file is not up to date. Make sure your whole tree is up to date, not just the src subdirectory. Regards Henrik
Re: Squid/icap 2.5 CVS
On Mon, 16 May 2005, Henrik Nordstrom wrote: On Mon, 16 May 2005, olivier wrote: gcc -DHAVE_CONFIG_H - -DDEFAULT_CONFIG_FILE=\"/usr/local/squid/etc/squid.conf\" -I. -I. - -I../include -I. -I. -I../include -I../include-g -O2 -Wall -c `test - -f dns_internal.c || echo './'`dns_internal.c dns_internal.c:58: error: syntax error before "rfc1035_query" This error seems to indicate your include/rfc1035.h file is not up to date. Make sure your whole tree is up to date, not just the src subdirectory. Just tested compiling the icap-2.5 branch and the above problem did not show up. A number of other problems did however... - strnstr prototype in util.h. needed for compiling strnstr.c - a number module-local functions declared without static - strcasestr is a GNU extension. Prototype only available if _GNU_SOURCE is defined (glibc systems). All required changes committed. Regards Henrik
Re: Squid/icap 2.5 CVS
Henrik Nordstrom wrote: .. All required changes committed. I hope that this means that you are starting developement in icap-client :-) I think icap is a must for squid. Thinks like modifing headers (like those you are talking about with James ) are easier using icap. It is better for someone to put efford on icap developement (one patch) than on a number of different patches At the lastest tests which I done with squid-icap looks that it becomes more stable, but still I do not know about its stability in a production system. Regards, Christos
Re: Squid/icap 2.5 CVS
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Henrik Nordstrom wrote: > > Just tested compiling the icap-2.5 branch and the above problem did not > show up. A number of other problems did however... > > - strnstr prototype in util.h. needed for compiling strnstr.c > > - a number module-local functions declared without static > > - strcasestr is a GNU extension. Prototype only available if > _GNU_SOURCE is defined (glibc systems). > > All required changes committed. Hello, Just updated, works fine, thanks Henrik :) - -- Olivier Girondel <[EMAIL PROTECTED]> Project Leader Dolphian -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.0 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFCibpcpqVXaJzJYNIRAuLNAJwN6/hnE7kqMQCxOSRTNTYmuhCT7gCeLvxG H7OSwRsnarW0LN4qZDK8png= =DmMz -END PGP SIGNATURE-
Re: Squid/icap 2.5 CVS
On Tue, 17 May 2005, Tsantilas Christos wrote: I hope that this means that you are starting developement in icap-client :-) Not in this icap client. But I have another (see below). I think icap is a must for squid. Agreed. Thinks like modifing headers (like those you are talking about with James ) are easier using icap. It is better for someone to put efford on icap developement (one patch) than on a number of different patches I already have a what I consider good ICAP client for Squid (the icap_streaming project). Just missing a customer willing to pay for it to have it published. icap_streaming was initially developed for a bigger potential customer project, but this project unfortunately got cancelled leaving icap_streaming somewhat hanging mid-air but still completed. If there is interest in this please contact [EMAIL PROTECTED] Regards Henrik
Re: Squid/icap 2.5 CVS
Henrik Nordstrom wrote: Not in this icap client. But I have another (see below). I wrote in a previous mail that I am considering the current ICAP client for squid problematic. And I now that it will never be really good (if it will not rewritten from scratch.). I already have a what I consider good ICAP client for Squid (the icap_streaming project). Just missing a customer willing to pay for it to have it published. icap_streaming was initially developed for a bigger potential customer project, but this project unfortunately got cancelled leaving icap_streaming somewhat hanging mid-air but still completed. If there is interest in this please contact [EMAIL PROTECTED] OK, it is good for all squid users to know that already exist a really good ICAP client for squid. :-) Regards, Christos
Volunteering for squid-icap development
Hello, I'm a software engineer working in England. I have a use for squid-icap and I would like to contribute to its development. In particular, I want to extend it so that it can do all 4 vector points: reqmod pre/postcache and respmod pre/postcache. I wrote to Henrik Nordström (the only developer for this who has a mailto: link) and he replied that I should request developer access direct from you. Would it be possible for me to get access to the squid-icap development tree? TIA, Jeff Silver
Re: Squid-ICAP problem (bug?)
ons 2006-11-08 klockan 16:26 +0100 skrev Christophe Boyanique: > We have read enough data (27 bytes) to have all the body (as > content-length is 27), so we set the eof flag. > > if (!icap->flags.reqmod_http_entity_eof) > commSetSelect(fd, COMM_SELECT_READ, icapReqModReadHttpBody, > icap, 0); eof in ICAP is that 0 chunk... so something is not right here in the ICAP patch. Note: As Adrian said most efforts is focused on the Squid-3 release, which also includes ICAP support. Regards Henrik signature.asc Description: Detta är en digitalt signerad meddelandedel
Re: Squid-ICAP problem (bug?)
Henrik Nordstrom a écrit : We have read enough data (27 bytes) to have all the body (as content-length is 27), so we set the eof flag. if (!icap->flags.reqmod_http_entity_eof) commSetSelect(fd, COMM_SELECT_READ, icapReqModReadHttpBody, icap, 0); eof in ICAP is that 0 chunk... so something is not right here in the ICAP patch. The problem is that this 0\r\n arrives after the first read of the 27 first bytes. Note: As Adrian said most efforts is focused on the Squid-3 release, which also includes ICAP support. I understand that but the problem is that we have this bug in production on heavy loaded site (we took several weeks to find it out) and we have to correct it anyway (and I don't think think that targetting Squid3 is possible (I thought Squid3 was considered beta?)). Regards, Christophe.
Re: Squid-ICAP problem (bug?)
Hi, I send this mail from un account which is not subscribed to the mailing list so I am re-sending it. Sory about that. Please to the mailing list moderator to ingore the first mail. - Hi Christophe, First of all squid3 has a better support for icap protocol now. Maybe it is better to use squid3 as icap client. However, yes the problem looks that exists in squid25-icap and squid26-icap. In fle icap_reqmod.c, in function icapReqModReadHttpBody near line 878-880 try this small patch : -if (icap->reqmod.http_entity.bytes_read >= icap->request->content_length) - icap->flags.reqmod_http_entity_eof = 1; + // if (icap->reqmod.http_entity.bytes_read >= icap->request->content_length ) + if (icap->chunk_size < 0 ) + icap->flags.reqmod_http_entity_eof = 1; It must marks the http_entity_eof when it has read the "0\r\n\r\n" chunk. Works for me but I did not test it enough Regards, Christos Christophe Boyanique wrote: Hello, I think I have found a bug in the Squid-ICAP patch and I would like to have your opinion about it. I use a tcpdump strace and verbose log to track a problem which occurs sometimes during a respmod request but is triggered during the reqmod answer analysis I think. We use squid-2.5.STABLE14 and Squid-Icap patch from 2006/06/26. .
Re: Squid-ICAP problem (bug?)
On Thu, Nov 09, 2006, Christophe Boyanique wrote: > >Note: As Adrian said most efforts is focused on the Squid-3 release, > >which also includes ICAP support. > > I understand that but the problem is that we have this bug in production > on heavy loaded site (we took several weeks to find it out) and we have > to correct it anyway (and I don't think think that targetting Squid3 is > possible (I thought Squid3 was considered beta?)). I think it'd be great if the bug were fixed in squid-2.6-icap - but this kinda stuff also needs to go into squid-3. Squid-3 won't be beta for long if I have anything to say about it in a couple of weeks. Adrian
Re: Squid-ICAP problem (bug?)
tor 2006-11-09 klockan 12:39 +0100 skrev Christophe Boyanique: > > eof in ICAP is that 0 chunk... so something is not right here in the > > ICAP patch. > > The problem is that this 0\r\n arrives after the first read of the 27 > first bytes. Which should not be a problem in therms of ICAP. It's in fact quite normal to see this in an ICAP stream. > I understand that but the problem is that we have this bug in production > on heavy loaded site (we took several weeks to find it out) and we have > to correct it anyway (and I don't think think that targetting Squid3 is > possible (I thought Squid3 was considered beta?)). Well.. so is the unofficial ICAP patch for 2.6. Just the bugs are different ;-) Regards Henrik signature.asc Description: Detta är en digitalt signerad meddelandedel
Re: Squid-ICAP problem (bug?)
Hi Christophe, First of all squid3 has a better support for icap protocol now. Maybe it is better to use squid3 as icap client. However, yes the problem exists in squid25-icap and squid26-icap. In fle icap_reqmod.c, in function icapReqModReadHttpBody near line 878-880 try this small patch : -if (icap->reqmod.http_entity.bytes_read >= icap->request->content_length) - icap->flags.reqmod_http_entity_eof = 1; + // if (icap->reqmod.http_entity.bytes_read >= icap->request->content_length ) + if (icap->chunk_size < 0 ) + icap->flags.reqmod_http_entity_eof = 1; It must marks the http_entity_eof when it has read the "0\r\n\r\n" chunk. Works for me but i did not test it enough Regards, Christos Christophe Boyanique wrote: Hello, I think I have found a bug in the Squid-ICAP patch and I would like to have your opinion about it. I use a tcpdump strace and verbose log to track a problem which occurs sometimes during a respmod request but is triggered during the reqmod answer analysis I think. We use squid-2.5.STABLE14 and Squid-Icap patch from 2006/06/26. .
Re: Squid-ICAP problem (bug?)
tor 2006-11-09 klockan 12:37 +0200 skrev Tsantilas Christos: > + if (icap->chunk_size < 0 ) > + icap->flags.reqmod_http_entity_eof = 1; Shouldn't that be <= 0 or maybe even == 0? Regards Henrik signature.asc Description: Detta är en digitalt signerad meddelandedel
Re: Squid-ICAP problem (bug?)
Hi, I think it should be icap->chunk_size == -2. In icapParseChunkSize function in common_icap.c file the icap->chunk_size set to -2 when the 0\r\n\r\n sequence parsed Regards, Christos Henrik Nordstrom wrote: > tor 2006-11-09 klockan 12:37 +0200 skrev Tsantilas Christos: > >> + if (icap->chunk_size < 0 ) >> + icap->flags.reqmod_http_entity_eof = 1; > > Shouldn't that be <= 0 or maybe even == 0? > > Regards > Henrik
Re: Squid-ICAP problem (bug?)
On Wed, 2006-11-08 at 16:26 +0100, Christophe Boyanique wrote: > I think I have found a bug in the Squid-ICAP patch and I would like to > have your opinion about it. ... > The problem is that the end of the chunk (0\r\n\r\n) is not sent (and > read) in the same stream that the beginning of the chunk. FWIW, the problem does not seem to be present in Squid3. The incremental chunked coding parser declares success only when the last-chunk followed by an optional trailer was found. I have not tested this though. Alex.
Re: Squid-ICAP problem (bug?)
Hello,
Thanks for you support.
I tried the patch from Tsantilas:
if (icap->chunk_size < 0)
icap->flags.reqmod_http_entity_eof = 1;
instead of:
if (icap->reqmod.http_entity.bytes_read >= icap->request->content_length)
icap->flags.reqmod_http_entity_eof = 1;
And that corrects the initial problem: remaining data waiting on the
wire is read ("\r\n0\r\n").
But there is a new problem.
After the conditionnal call to commSetSelect there is a conditionnal
call to icapReqModPassHttpBody and this where is the problem:
if (icap->reqmod.http_entity.callback &&
icap->reqmod.http_entity.buf.size) {
icapReqModPassHttpBody(icap,
icap->reqmod.http_entity.callback_buf,
icap->reqmod.http_entity.callback_bufsize,
icap->reqmod.http_entity.callback,
icap->reqmod.http_entity.callback_data);
icap->reqmod.http_entity.callback = NULL;
cbdataUnlock(icap->reqmod.http_entity.callback_data);
}
When calling this function, the test is on
icap->reqmod.http_entity.buf.size but the parameter sent to the function
is cap->reqmod.http_entity.callback_bufsize. Is it normal ?
I added some logs and here is the output:
2006/11/14 13:43:25| icapReqModReadHttpBody: FD 38 called
2006/11/14 13:43:25| icapReqModReadHttpBody: read returns 33
2006/11/14 13:43:25| icap_common.c:695: chunk_size=0
2006/11/14 13:43:25| icap_common.c:702: bufOffset=0, len=33
2006/11/14 13:43:25| icap_common.c:705: bufOffset=0, len=33
2006/11/14 13:43:25| icapParseChunkSize: buf=0x9e362d0, len=33
2006/11/14 13:43:25| icapParseChunkSize: start=0
2006/11/14 13:43:25| icapLineLength: returning 4
2006/11/14 13:43:25| icapParseChunkSize: start=0, end=2
2006/11/14 13:43:25| icapParseChunkSize: return nextStart=4
2006/11/14 13:43:25| got chunksize 27, new offset 4
2006/11/14 13:43:25| icap_common.c:723: X
2006/11/14 13:43:25| icap_common.c:705: bufOffset=31, len=33
2006/11/14 13:43:25| icapParseChunkSize: buf=0x9e362ef, len=2
2006/11/14 13:43:25| icapParseChunkSize: start=0
2006/11/14 13:43:25| icapLineLength: returning 2
2006/11/14 13:43:25| icapParseChunkSize: start=2
2006/11/14 13:43:25| icap_reqmod.c:882 chunk_size=0
2006/11/14 13:43:25| icap_reqmod.c:892 http_entity.callback=(nil)
2006/11/14 13:43:25| icap_reqmod.c:894 http_entity.buf.size=27
2006/11/14 13:43:25| icap_reqmod.c:896 http_entity.callback_bufsize=0
2006/11/14 13:43:25| icapService: type=ICAP_SERVICE_REQMOD_POSTCACHE
2006/11/14 13:43:25| icapService: checking service service_3/id=0
2006/11/14 13:43:25| icapService: checking service service_4/id=0
2006/11/14 13:43:25| icapService: no service found
2006/11/14 13:43:25| icapReqModPassHttpBody: called
2006/11/14 13:43:25| icapReqModPassHttpBody: entity buf size = 27
2006/11/14 13:43:25| icapReqModPassHttpBody: giving 27 bytes to other side
2006/11/14 13:43:25| icapReqModPassHttpBody: entity buf size now = 0
2006/11/14 13:43:25| icapReqModPassHttpBody: called
2006/11/14 13:43:25| icapReqModPassHttpBody: entity buf size = 0
2006/11/14 13:43:25| icapReqModReadHttpBody: FD 38 called
2006/11/14 13:43:25| icapReqModReadHttpBody: read returns 5
2006/11/14 13:43:25| icap_common.c:695: chunk_size=0
2006/11/14 13:43:25| icap_common.c:702: bufOffset=0, len=7
2006/11/14 13:43:25| icap_common.c:705: bufOffset=0, len=7
2006/11/14 13:43:25| icapParseChunkSize: buf=0x9e362d0, len=7
2006/11/14 13:43:25| icapParseChunkSize: start=0
2006/11/14 13:43:25| icapLineLength: returning 2
2006/11/14 13:43:25| icapParseChunkSize: start=2
2006/11/14 13:43:25| icapLineLength: returning 3
2006/11/14 13:43:25| icapParseChunkSize: start=2, end=3
2006/11/14 13:43:25| icapParseChunkSize: return nextStart=5
2006/11/14 13:43:25| got chunksize -2, new offset 5
2006/11/14 13:43:25| zero end chunk reached
2006/11/14 13:43:25| icap_reqmod.c:882 chunk_size=-2
2006/11/14 13:43:25| icap_reqmod.c:892 http_entity.callback=0x807c994
2006/11/14 13:43:25| icap_reqmod.c:894 http_entity.buf.size=0
2006/11/14 13:43:25| icap_reqmod.c:896 http_entity.callback_bufsize=8192
2006/11/14 13:43:25| icapReqModPassHttpBody: called
FATAL: Received Segment Violation...dying.
Perhaps there is a problem with parameters passed to
icapReqModPassHttpBody ?
Christophe.
Re: Squid-ICAP problem (bug?)
Hi,
From your logs what I am seeing is that the
icap->reqmod.http_entity.buf.size==0 so the icapRqModPassHttpBody must
not called, but it is called. It is strange.
Did you touch something else in code? Can you run squid in gdb and send
a backtrace after the crash?
Regards,
Christos
Christophe Boyanique wrote:
>
> ..
>
> After the conditionnal call to commSetSelect there is a conditionnal
> call to icapReqModPassHttpBody and this where is the problem:
>
> if (icap->reqmod.http_entity.callback &&
> icap->reqmod.http_entity.buf.size) {
> icapReqModPassHttpBody(icap,
> icap->reqmod.http_entity.callback_buf,
> icap->reqmod.http_entity.callback_bufsize,
> icap->reqmod.http_entity.callback,
> icap->reqmod.http_entity.callback_data);
> icap->reqmod.http_entity.callback = NULL;
> cbdataUnlock(icap->reqmod.http_entity.callback_data);
>
> }
>
> When calling this function, the test is on
> icap->reqmod.http_entity.buf.size but the parameter sent to the function
> is cap->reqmod.http_entity.callback_bufsize. Is it normal ?
>
> I added some logs and here is the output:\
>
>
> 2006/11/14 13:43:25| zero end chunk reached
> 2006/11/14 13:43:25| icap_reqmod.c:882 chunk_size=-2
> 2006/11/14 13:43:25| icap_reqmod.c:892 http_entity.callback=0x807c994
> 2006/11/14 13:43:25| icap_reqmod.c:894 http_entity.buf.size=0
> 2006/11/14 13:43:25| icap_reqmod.c:896 http_entity.callback_bufsize=8192
> 2006/11/14 13:43:25| icapReqModPassHttpBody: called
> FATAL: Received Segment Violation...dying.
>
> Perhaps there is a problem with parameters passed to
> icapReqModPassHttpBody ?
>
Re: Squid-ICAP problem (bug?)
Hi Christophe,
If you are still looking for the solution, try the following patch in
icap_reqmod.c file. With this patch the icapReqModPassHttpBody function
called if the icap->reqmod.http_entity.buf.size is zero, and at this
phase can handle correctly, the case that the
icap->flags.reqmod_http_entity_eof==1.
@@ -877,7 +876,7 @@
icapParseChunkedBody(icap,
icapReqModMemBufAppend, &icap->reqmod.http_entity.buf);
}
-if (icap->reqmod.http_entity.bytes_read >=
icap->request->content_length)
+if (icap->chunk_size < 0 )
icap->flags.reqmod_http_entity_eof = 1;
if (!icap->flags.reqmod_http_entity_eof)
@@ -889,7 +888,7 @@
icap->reqmod.http_entity.callback);
debug(81, 3) ("%s:%d http_entity.buf.size=%d\n", __FILE__, __LINE__,
icap->reqmod.http_entity.buf.size);
-if (icap->reqmod.http_entity.callback &&
icap->reqmod.http_entity.buf.size) {
+if (icap->reqmod.http_entity.callback /*&&
icap->reqmod.http_entity.buf.size*/) {
icapReqModPassHttpBody(icap,
icap->reqmod.http_entity.callback_buf,
icap->reqmod.http_entity.callback_bufsize,
Christophe Boyanique wrote:
>
> Hello,
>
> Thanks for you support.
>
> I tried the patch from Tsantilas:
>..
> But there is a new problem.
>
> After the conditionnal call to commSetSelect there is a conditionnal
> call to icapReqModPassHttpBody and this where is the problem:
>
> if (icap->reqmod.http_entity.callback &&
> icap->reqmod.http_entity.buf.size) {
At this point try to change the if to:
if (icap->reqmod.http_entity.callback /*&&
icap->reqmod.http_entity.buf.size*/)
to remove the icap->reqmod.http_entity.buf.size from if.
Please, if you try it, tell me if still there is any problem here.
All test I done, worked for me, but I want a second opinion before
commit it to cvs .
> icapReqModPassHttpBody(icap,
> icap->reqmod.http_entity.callback_buf,
> icap->reqmod.http_entity.callback_bufsize,
> icap->reqmod.http_entity.callback,
> icap->reqmod.http_entity.callback_data);
> icap->reqmod.http_entity.callback = NULL;
> cbdataUnlock(icap->reqmod.http_entity.callback_data);
>
> }
>
Re: Squid-ICAP problem (bug?)
Hi all,
Tsantilas Christos wrote :
If you are still looking for the solution, try the following patch in
icap_reqmod.c file. With this patch the icapReqModPassHttpBody function
called if the icap->reqmod.http_entity.buf.size is zero, and at this
phase can handle correctly, the case that the
icap->flags.reqmod_http_entity_eof==1.
Unfortunately it still segfaults.
But, I had time to work on my client's test platform and I spent a whole
day hacking icap_reqmod.c.
The only modification (except adding excessive debug logging) is that one:
in icapReqModReadHttpBody function replacing the test
if (icap->reqmod.http_entity.bytes_read >= icap->request->content_length)
by
if (icap->chunk_size < 0)
which correct the initial problem ie not reading the 0\r\n marking thee
end of the chunk.
The new problem is that squid correctly reads the end of the chunk but
dies immediatly.
The dying occurs in the icapReqModPassHttpBody function and I spent many
time to find why. In fact I focussed on the call of this function from
the icapReqModReadHttpBody function and I didn't manage to find the
logic of the test and the function parameters (because I don't know well
the squid and icap patch code).
Anyway this I found that the function was called with icap = 0 parameter
(I spent too more time to search a buf=0 or callback=0); so I added a
simple test in the beginning of the function:
static void
icapReqModPassHttpBody(IcapStateData * icap, char *buf, size_t size,
CBCB * callback, void *cbdata)
{
debug(81, 3) ("%s:%d: icapReqModPassHttpBody: called\n", __FILE__,
__LINE__);
if (!icap) {
debug(81, 1) ("%s:%d: icapReqModPassHttpBody: FD %d called with
icap = (nil) !\n", __FILE__, __LINE__);
return;
}
I added this to prevent segfaulting but without knowing if the return
would lead to a correct behaviour. And it seems that is is ok. We made
some tests and we did not have any more segfaults.
I then tried with gdb (I should have used it before but I am not
familiar with it) to get a backtrace as you asked in a previous mail and
I saw that the icapReqModPassHttpBody call with icap=0 is not called
from the icapReqModReadHttpBody:
Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread -1218496992 (LWP 9508)]
0x08087411 in icapReqModPassHttpBody (icap=0x0, buf=0x0,
size=4294967295, callback=0, cbdata=0x8c95e00) at icap_reqmod.c:933
933 icap_reqmod.c: Aucun fichier ou repertoire de ce type.
in icap_reqmod.c
(gdb) bt
#0 0x08087411 in icapReqModPassHttpBody (icap=0x0, buf=0x0,
size=4294967295, callback=0, cbdata=0x8c95e00) at icap_reqmod.c:933
#1 0x08083905 in requestAbortBody (request=0x8c94578) at HttpRequest.c:215
#2 0x0807a183 in httpStateFree (fd=9, data=0x8cc72a0) at http.c:69
#3 0x08065ccd in commCallCloseHandlers (fd=44) at comm.c:573
#4 0x08065dec in comm_close (fd=44) at comm.c:655
#5 0x0807b32c in httpReadReply (fd=44, data=0x8cc72a0) at http.c:852
#6 0x08067749 in comm_poll (msec=13) at comm_select.c:445
#7 0x0808fc77 in main (argc=4, argv=0xbfff9494) at main.c:754
Here is a sumup of the logs when icap value is tested and squid does not
segfault anymore:
2006/12/01 18:18:19| icapReqModParseHttpRequest: successfully parsed the
HTTP request
2006/12/01 18:18:19| handing request bodies in ICAP REQMOD
2006/12/01 18:18:19| icap_reqmod.c:859: icapReqModReadHttpBody: FD 53 called
2006/12/01 18:18:19| icap_reqmod.c:861: icapReqModReadHttpBody: read
returns 33
2006/12/01 18:18:19| icap_reqmod.c:872: icapReqModReadHttpBody: Before
icapParseChunkedBody bytes_read = 0
2006/12/01 18:18:19| icap_common.c:695: chunk_size=0
2006/12/01 18:18:19| icap_common.c:702: bufOffset=0, len=33
2006/12/01 18:18:19| icap_common.c:705: bufOffset=0, len=33
2006/12/01 18:18:19| icap_common.c:604: icapParseChunkSize:
buf=0x8ccd1b8, len=33
2006/12/01 18:18:19| icap_common.c:607: icapParseChunkSize: start=0
2006/12/01 18:18:19| icapLineLength: returning 4
2006/12/01 18:18:19| icap_common.c:646: icapParseChunkSize: start=0, end=2
2006/12/01 18:18:19| icap_common.c:672: icapParseChunkSize: return
nextStart=4
2006/12/01 18:18:19| icap_common.c:716: got chunksize 27, new offset 4
2006/12/01 18:18:19| icap_common.c:723: X
2006/12/01 18:18:19| icap_common.c:705: bufOffset=31, len=33
2006/12/01 18:18:19| icap_common.c:604: icapParseChunkSize:
buf=0x8ccd1d7, len=2
2006/12/01 18:18:19| icap_common.c:607: icapParseChunkSize: start=0
2006/12/01 18:18:19| icapLineLength: returning 2
2006/12/01 18:18:19| icap_common.c:607: icapParseChunkSize: start=2
2006/12/01 18:18:19| icap_reqmod.c:876: icapReqModReadHttpBody: After
icapParseChunkedBody bytes_read = 27
2006/12/01 18:18:19| icap_reqmod.c:884 icapReqModReadHttpBody: chunk_size=0
Here is the first modification: even if we read the 27 bytes of the body
we continue reading because chunk_size is not -2:
2006/12/01 18:18:19| icap_reqmod.c:892 icapReqModReadHttpBody: call
commSetSelect -> icapR
Re: Squid-ICAP problem (bug?)
Hi Christophe,
Maybe I was not clear on my last mail. Sorry.
Christophe Boyanique wrote:
>
>
> in icapReqModReadHttpBody function replacing the test
>
> if (icap->reqmod.http_entity.bytes_read >= icap->request->content_length)
>
> by
>
> if (icap->chunk_size < 0)
OK this is needed.
But in the same function replace the test:
if (icap->reqmod.http_entity.callback &&
icap->reqmod.http_entity.buf.size) {
icapReqModPassHttpBody(icap,
icap->reqmod.http_entity.callback_buf,
icap->reqmod.http_entity.callback_bufsize,
with:
if (icap->reqmod.http_entity.callback) {
icapReqModPassHttpBody(icap,
icap->reqmod.http_entity.callback_buf,
icap->reqmod.http_entity.callback_bufsize,
Did you try it?
>
> which correct the initial problem ie not reading the 0\r\n marking thee
> end of the chunk.
>
> The new problem is that squid correctly reads the end of the chunk but
> dies immediatly.
>
I think with the previous change the problem must be solved.
> .
> 2006/12/01 18:18:19| icap_reqmod.c:903 http_entity.callback_bufsize = 8192
> 2006/12/01 18:18:19| icap_reqmod.c:917 icapReqModReadHttpBody: end of
> function
>
> and we exit the icapReqModReadHttpBody function without calling
> icapReqModPassHttpBody because http_entity.buf.size is 0.
>
With the previous change this function must called here.
Did you make this change? Are you still seeing segfaults?
In my tests I am not seeing any problem any more. But maybe I am losing
something
Regards,
Christos
Re: Squid-ICAP problem (bug?)
Tsantilas Christos a e'crit :
Hello Christos,
Maybe I was not clear on my last mail. Sorry.
You were but I forgot to mention that I tried what you adviced.
But in the same function replace the test:
if (icap->reqmod.http_entity.callback &&
icap->reqmod.http_entity.buf.size) {
icapReqModPassHttpBody(icap,
icap->reqmod.http_entity.callback_buf,
icap->reqmod.http_entity.callback_bufsize,
with:
if (icap->reqmod.http_entity.callback) {
icapReqModPassHttpBody(icap,
icap->reqmod.http_entity.callback_buf,
icap->reqmod.http_entity.callback_bufsize,
Did you try it?
Yes we tried it and this leads to segfault too.
and we exit the icapReqModReadHttpBody function without calling
icapReqModPassHttpBody because http_entity.buf.size is 0.
With the previous change this function must called here.
Did you make this change? Are you still seeing segfaults?
I did not test extensively but it seems that calling
icapReqModPassHttpBody without testing if http_entity.buf.size is 0 will
work in this very specific case (when end chunk is read in an extra
pass) but will lead to segfault in cases which used to work perfectly
before changing this test.
In my tests I am not seeing any problem any more. But maybe I am losing
something
I don't have any magic recipe to reproduce the problem but it occurs
when parsing the request's body in REQMOD and that occurs generally when
doint a POST request (as there is no body in a GET request).
With your modification, we ends with segfaults during GET request !
By reading the log I just noticed that with your modification the
icapReqModPassHttpBody is not called:
This is for the specific request (POST):
2006/11/24 14:15:22| handing request bodies in ICAP REQMOD
2006/11/24 14:15:22| icapReqModReadHttpBody: FD 38 called
2006/11/24 14:15:22| icapReqModReadHttpBody: read returns 33
2006/11/24 14:15:22| icap_common.c:695: chunk_size=0
2006/11/24 14:15:22| icap_common.c:702: bufOffset=0, len=33
2006/11/24 14:15:22| icap_common.c:705: bufOffset=0, len=33
2006/11/24 14:15:22| icapParseChunkSize: buf=0x997fbe0, len=33
2006/11/24 14:15:22| icapParseChunkSize: start=0
2006/11/24 14:15:22| icapLineLength: returning 4
2006/11/24 14:15:22| icapParseChunkSize: start=0, end=2
2006/11/24 14:15:22| icapParseChunkSize: return nextStart=4
2006/11/24 14:15:22| got chunksize 27, new offset 4
2006/11/24 14:15:22| icap_common.c:723: X
2006/11/24 14:15:22| icap_common.c:705: bufOffset=31, len=33
2006/11/24 14:15:22| icapParseChunkSize: buf=0x997fbff, len=2
2006/11/24 14:15:22| icapParseChunkSize: start=0
2006/11/24 14:15:22| icapLineLength: returning 2
2006/11/24 14:15:22| icapParseChunkSize: start=2
2006/11/24 14:15:22| icap_reqmod.c:882 chunk_size=0
2006/11/24 14:15:22| icap_reqmod.c:892 http_entity.callback=(nil)
2006/11/24 14:15:22| icap_reqmod.c:894 http_entity.buf.size=27
2006/11/24 14:15:22| icap_reqmod.c:896 http_entity.callback_bufsize=0
2006/11/24 14:15:22| icapReqModReadHttpBody: FD 38 called
2006/11/24 14:15:22| icapReqModReadHttpBody: read returns 5
2006/11/24 14:15:22| icap_common.c:695: chunk_size=0
2006/11/24 14:15:22| icap_common.c:702: bufOffset=0, len=7
2006/11/24 14:15:22| icap_common.c:705: bufOffset=0, len=7
2006/11/24 14:15:22| icapParseChunkSize: buf=0x997fbe0, len=7
2006/11/24 14:15:22| icapParseChunkSize: start=0
2006/11/24 14:15:22| icapLineLength: returning 2
2006/11/24 14:15:22| icapParseChunkSize: start=2
2006/11/24 14:15:22| icapLineLength: returning 3
2006/11/24 14:15:22| icapParseChunkSize: start=2, end=3
2006/11/24 14:15:22| icapParseChunkSize: return nextStart=5
2006/11/24 14:15:22| got chunksize -2, new offset 5
2006/11/24 14:15:22| zero end chunk reached
2006/11/24 14:15:22| icap_reqmod.c:882 chunk_size=-2
2006/11/24 14:15:22| icap_reqmod.c:892 http_entity.callback=(nil)
2006/11/24 14:15:22| icap_reqmod.c:894 http_entity.buf.size=27
2006/11/24 14:15:22| icap_reqmod.c:896 http_entity.callback_bufsize=0
2006/11/24 14:15:22| icapSendRespMod: Create a new connection to icap
server service_4/icap://xx.xx.xx.xx:1344/wwrespmod/?wwprofile=HTTP_
But what is very strange is:
- respmod seems not to be called;
- the next request which is a GET is handled strangely:
2006/11/24 14:15:34| icapParseEncapsulated: res-hdr=-1, req-hdr=0,
null-body=791, res-body=-1, req-body=-1, opt-body=-1
2006/11/24 14:15:34| icapReqModReadIcapPart: directResponse=0 from ICAP
server service_3/icap://xx.xx.xx.xx:1344/wwreqmod/?wwprofile=HTTP_
2006/11/24 14:15:34| icapReqModReadHttpHdrs:
2006/11/24 14:15:34| expect=791
2006/11/24 14:15:34| so_far=0
2006/11/24 14:15:34| needed=791
2006/11/24 14:15:34| icapReqModReadHttpHdrs: read 791 bytes
2006/11/24 14:15:34| icapReqModReadHttpHdrs: read the entire request headers
2006/11/24 14:15:34| icapReqModParseHttpRequest: URI is
'http://x.xxx/rtn_FR.gif'
2006/11/24 14:15:34| icapReqModParseHttpRequest: Client HTTP version 1.1.
2006/11/24 14
Re: Squid-ICAP problem (bug?)
Christophe Boyanique wrote: Yes we tried it and this leads to segfault too. :-( By reading the log I just noticed that with your modification the icapReqModPassHttpBody is not called: This is for the specific request (POST): 2006/11/24 14:15:22| handing request bodies in ICAP REQMOD 2006/11/24 14:15:22| icapReqModReadHttpBody: FD 38 called 2006/11/24 14:15:22| icapReqModReadHttpBody: read returns 33 . 2006/11/24 14:15:22| icap_reqmod.c:882 chunk_size=-2 2006/11/24 14:15:22| icap_reqmod.c:892 http_entity.callback=(nil) 2006/11/24 14:15:22| icap_reqmod.c:894 http_entity.buf.size=27 2006/11/24 14:15:22| icap_reqmod.c:896 http_entity.callback_bufsize=0 2006/11/24 14:15:22| icapSendRespMod: Create a new connection to icap server service_4/icap://xx.xx.xx.xx:1344/wwrespmod/?wwprofile=HTTP_ The strange here is that the http_entity.callback=(nil) In my test cases, this never happens... II don't know why . So why is called icapReqModPassHttpBody if the request does not contain a body ? And why dos it find 27 bytes of data which is exactly the body of the previous request ? This looks very strange, isn't it ? Yes it is .
Re: Christian Kunst - Squid iCAP Client
mån 2003-06-02 klockan 13.54 skrev Christian Kunst: > My name is Christian Kunst, and I would like to join the Squid team. I > am interested in helping develop the Squid iCAP client. Please let me > know what are the next steps that I should take. Hi Cristian, The next steps to become a Squid developer are: * Subscribe to the squid-dev mailinglist * Create a SourceForge account if you do not have one already, and send me your account name. * While your registration is pending, take the time to read the notes on http://devel.squid-cache.org/ Regarding the Squid ICAP client: There is many things to do. There is a mostly working ICAP client implementation for Squid-2.5 in the icap-2_5 branch, but I am working on fixing some issues in REQMOD. There probably also is a few bugs lurking in how it deals with aborted/failed requests etc and this needs testing and fixing. Then there is the major task of writing an ICAP client for Squid-3 with the goal of having ICAP support merged into Squid. The internals of Squid have changed a lot since Squid-2.5 and the approach to hook into the datastreams selected by the Squid-2.5 ICAP client implementation is probably not the most suited for Squid-3. Geetha Manjunath probably have more info on the subject. Regards Henrik
Re: Fwd: Squid ICAP client problems
Hello Henrik, on Mon, 09 Jun 2003, Henrik Nordstrom wrote: > -Original Message- > From: Rosenbaum, Larry M. > Sent: Thursday, June 05, 2003 12:57 PM > To: '[EMAIL PROTECTED]'; '[EMAIL PROTECTED]' > Subject: Squid ICAP client problems [...] > 2) (cache_cf.c) If you don't have a "icap_class" line in the config > file, squid will crash if you do a "-k reconfigure". This is because > the class structure has an element with a null iter->name. Here is > one possible fix: I think the real problem is that some pointers are not reset properly. Here is a patch: diff -u -r1.38.6.11.2.2 cache_cf.c --- cache_cf.c 7 Apr 2003 13:28:50 - 1.38.6.11.2.2 +++ cache_cf.c 10 Jun 2003 09:03:00 - @@ -2145,6 +2145,7 @@ cbdataFree(current_node); current_node = next_node; } + cfg->service_head = NULL; } } @@ -2381,6 +2382,7 @@ memFree(current_node, MEM_ICAP_CLASS); current_node = next_node; } + cfg->class_head = NULL; } } @@ -2553,6 +2555,7 @@ memFree(current_node, MEM_ICAP_ACCESS); current_node = next_node; } + cfg->access_head = NULL; } } Regards, Ralf.
Re: Fwd: Squid ICAP client problems
On Tuesday 10 June 2003 11.37, ralf wrote:
> I think the real problem is that some pointers are not reset
> properly. Here is a patch:
>
> diff -u -r1.38.6.11.2.2 cache_cf.c
> --- cache_cf.c 7 Apr 2003 13:28:50 - 1.38.6.11.2.2
> +++ cache_cf.c 10 Jun 2003 09:03:00 -
> @@ -2145,6 +2145,7 @@
> cbdataFree(current_node);
> current_node = next_node;
> }
> + cfg->service_head = NULL;
> }
> }
Thanks. A slight variation of this patch has been committed to the
icap-2_5 branch. [patch attached]
Regards
HenrikIndex: cache_cf.c
===
RCS file: /cvsroot/squid/squid/src/cache_cf.c,v
retrieving revision 1.38.6.11.2.2
diff -u -w -p -r1.38.6.11.2.2 cache_cf.c
--- cache_cf.c 7 Apr 2003 13:28:50 - 1.38.6.11.2.2
+++ cache_cf.c 10 Jun 2003 12:41:25 -
@@ -2135,16 +2135,12 @@ dump_icap_service_type(StoreEntry *e, co
static void
free_icap_service_type(IcapConfig * cfg)
{
-icap_service *current_node = NULL;
-icap_service *next_node = NULL;
-if (cfg->service_head) {
+while (cfg->service_head) {
+ icap_service *current_node = cfg->service_head;
+ cfg->service_head = current_node->next;
current_node = cfg->service_head;
- while (current_node) {
- next_node = current_node->next;
icap_service_destroy(current_node);
cbdataFree(current_node);
- current_node = next_node;
- }
}
}
@@ -2371,16 +2367,11 @@ dump_icap_class_type(StoreEntry *e, cons
static void
free_icap_class_type(IcapConfig *cfg)
{
-icap_class *current_node = NULL;
-icap_class *next_node = NULL;
-if (cfg->class_head) {
- current_node = cfg->class_head;
- while (current_node) {
- next_node = current_node->next;
+while (cfg->class_head) {
+ icap_class *current_node = cfg->class_head;
+ cfg->class_head = current_node->next;
icap_class_destroy(current_node);
memFree(current_node, MEM_ICAP_CLASS);
- current_node = next_node;
- }
}
}
@@ -2543,16 +2534,11 @@ dump_icap_access_type(StoreEntry *e, con
static void
free_icap_access_type(IcapConfig * cfg)
{
-icap_access *current_node = NULL;
-icap_access *next_node = NULL;
-if (cfg->access_head) {
- current_node = cfg->access_head;
- while (current_node) {
- next_node = current_node->next;
+while (cfg->access_head) {
+ icap_access *current_node = cfg->access_head;
+ cfg->access_head = current_node->next;
icap_access_destroy(current_node);
memFree(current_node, MEM_ICAP_ACCESS);
- current_node = next_node;
- }
}
}
Re: Fwd: Squid ICAP client problems
Thanks for forwarding this, Henrik. Hello Larry, Sorry for this delayed reply.. > -Original Message- > From: Rosenbaum, Larry M. . > > 1) (icap.c) There is a problem with the way icapRespModReadReply() > tries to read the ICAP header without reading past the header. The > current code does a recv() with the MSG_PEEK flag and a read_sz of > 256 bytes, and if the buffer doesn't contain a double CRLF it > doubles the read_sz and tries again. The problem with this approach > is that it takes time for the socket to receive more bytes, and > there is nothing in the loop to ensure that more bytes are read. > For example, > > try #1 read_sz = 256, len = 38 > try #2 read_sz = 512, len = 38 (i.e. it reads the same 38 bytes > again) try #3 read_sz = 1024, len = 38 > ... > try #n read_sz = max, len = 38 > > so it is possible to exit the loop after many iterations without > reading any more bytes than you got on the first try. One possible > fix would be to turn off MSG_PEEK and read repeatedly until you get > to the end of the headers, and then pass the extra bytes to the > function that parses the rest of the message, but I don't know if > the code structure permits this. > > The same issue probably applies to icapReqModReadReply(). > Yes, you are right here - but this case would probably show up only with buggy ICAP servers that do not end ICAP headers with \r\n\r\n . The rest of code catches this error. Actually it signals an HTTP internal error rather than the suggested "did not find a proper ICAP response". We probably should add a separate error code for this. Also, clientReadRequest cannot handle the partial header and so on - to remove MSG_PEEK. Out of curiosity, did you encounter this problem during any test with squid client or thru' a code review? Thanks for analysing. regards Geetha
Re: Fwd: Squid ICAP client problems
mån 2003-06-23 klockan 14.17 skrev Geetha Manjunath: > Yes, you are right here - but this case would probably show up only with > buggy ICAP servers that do not end ICAP headers with \r\n\r\n . The rest > of code catches this error. Actually it signals an HTTP internal error > rather than the suggested "did not find a proper ICAP response". We > probably should add a separate error code for this. I would say it should happens pretty much any ICAP reply where the headers is larger than one IP packet MTU, even if the ICAP server tries it's best to send the headers in one packet. I had fogot about this issue. I'll note this issue so it is not forgotten again. Needs to be fixed to not use MSG_PEEK so it can wait for more data to arrive via commSetSelect(), which involves some magic to get the interactions with the main Squid code correct. Regards Henrik
Re: Fwd: Squid ICAP client problems
Henrik Nordstrom wrote: > > I had fogot about this issue. I'll note this issue so it is not > forgotten again. Needs to be fixed to not use MSG_PEEK so it can wait > for more data to arrive via commSetSelect(), which involves some magic > to get the interactions with the main Squid code correct. Yes, use of commSetSelect is ideal. > > Regards > Henrik
context data for squid ICAP patched
Hi all squids, Background information: I am implementing an extension to squid ICAP client based upon ICAP Patch and squid 2.5 STABLE 10. The squid ICAP client does not support Content Filtering the way we at PureSight.com using it. The ICAP protocol is defined to support also Content Filtering and defines a return value at the request mod stage. I receive the value that can be one of the following: ALLOW, BLOCK, UNKNOWN So far as described by the ICAP protocol (RFC 3507) and implemented by our ICAP server. As for squid client I added the ability to identify classification value, I would like squid to behave according to that value: 1. In case of UNKNOWN the response should go back to the ICAP server to be reclassified. 2. In case of BLOCK the ICAP server is changing the URL to retrieve a blocking information page. 3. In case of ALLOW squid should reply directly to the client with or without using the cache. The Question: I would like to pass the information that, no call to response mode (call the ICAP Server for the response) is needed. I would like your help with the way squid saves session data, I look at the code and I could not find that particular object, is there one? Any suggestions && || directions would be blessed. The Polite part: Thanks for reading && Participating on that grate project of squid. && Your help Moshe Beeri. Software Engineer, Servers Team. [EMAIL PROTECTED] Petch-Tiqva Bazel 16, Israel. Tel: +972 (3) 928-0400 ext. 429 Fax: +972 (3) 921-7594
context data for squid ICAP patched
Hi all squiders, I found the right way to implement the content filter extention to squid. I will continue the implementation and will send it to our QA Team, I hope the whole squid community will use this extention to squids ICAP capabilities. Best regards, Moshe Beeri. Software Engineer, Server Team. [EMAIL PROTECTED] Petach-Tiqva Bazel 16, Israel. Tel: +972 (3) 928-0400 ext. 429 Fax: +972 (3) 921-7594 > > Hi Christos, > > Thank you for your help, but you suggestion is not secure nor > best perform, Please read my other remarks below. > > Now that I read the question again I see it is not clear > enough, I will ask again. > I would like squid ICAP client to do the logic for couple of > reasons, 1. Security - Origin sever might change the replied > http header and add the "X-MY-SCANNER: Allow" it self, > and bypass the content filter, In that case I > would not be able to prevent kids from viewing un honest > pages :-( 2. Performance - Redundant call since I already > know that request is allowed. > > There for I would like to keep in squids session data the > classification and upon to the classification prior to > response-mod call. > > For now I have figured out that the best place to set the > data between the req-mod and resp-mod if in the fde > structure, but since squid saves that object in fd_table > (hash?) keyed by ICAP FD there is no continuity with the HTTP FD. > I realized that I need to look for the mechanism that changes > the next handler (hdl) that switch FD to read from, is the > KEY to set up the fde related to the HTTP response, with the > classification information. > In squid ICAP client implementation there is no connection > between the FD sets, ICAP's and HTTP's. > > > Again 10X for reading and good will, > I hope there is a short cut out there, > If someone has an implementation suggestion or realizes I am > missing something please write me. > > > > > > > Hi Beeri, > > Maybe you do not need to modify the squid-ICAP code to > support your > > model. > > I think that the correct implementation of your problem using > > squid-ICAP > > is: > > > > 1) An http request come into the squid. Squid sends the > reqmod request > > to the ICAP server and server classifies the request: > >a) In the case of the BLOCK ICAP server creates a http response > > saying > > to the web client that the request blocked > >b) In the case of UNKNOWN ICAP server does nothing > >c) In the case of ALLOW ICAP server adds a proprietary > http header > > to the http request for example "X-MY-SCANNER: Allow" > > > > 2) When squid has the http response then sends a respmod request to > > the > >ICAP server. The respmod request contains the http response > >headers AND the http request headers. > > a) When ICAP server founds the "X-MY-SCANNER: Allow" header > > in http request headers it responds with an > allow204 response > > to > > squid > > b)The "X-MY-SCANNER: Allow" is not in the http request headers > >so the ICAP server takes the http body from squid > and check it > > or > >modify it or what else. > > > > > > An other solutions is to use only the respmod request > because here you > > have both the http request headers and the http response. > > > > > The Question: > > > > > > I would like to pass the information that, no call to > response mode > > > (call the ICAP Server for the response) is needed. > > > ... > > > > I am not sure that I fully understand your question, but I > think that > > this functionality can not included in a general ICAP > client of squid. > > But maybe I am loosing something here. > > > > Regards, > >Christos > > > > > Background information: > > > > > > I am implementing an extension to squid ICAP client based > upon ICAP > > > Patch and squid 2.5 STABLE 10. > > > The squid ICAP client does not support Content Filtering > > the way we at > > > PureSight.com using it. > > > The ICAP protocol is defined to support also Content > Filtering and > > > defines a return value at the request mod stage. > > > I receive the value that can be one of the following: > > > ALLOW, > > > BLOCK, > > > UNKNOWN > > > > > > > > > .. > > > > > > > > > > > > >
Re: AW: Christian Kunst - Squid iCAP Client
tis 2003-06-03 klockan 17.12 skrev Christian Kunst: > I sent my request for subscription to the squid-dev mailinglist. To subscribe you need to send a message to [EMAIL PROTECTED] If you have already sent such message please send it again as it seems it may have been lost. See http://www.squid-cache.org/mailing-lists.html#squid-dev for details and policy of the squid-dev list. > I already have a SourceForge account. But it is only a web account, > and I can't access the ssh CVS with it. The account name is: "ckunst". To access CVS your first need to become member of the project you want to access.. You now have CVS access to the Squid project. It is recommended you upload your SSH key to your account for CVS, saving your from having to type the password on each CVS operation. http://sourceforge.net/my/ Then see http://devel.squid-cache.org/ for instructions and rules on how to access the Squid developer CVS tree. If you have any questions post them here on squid-dev and we will try to help you. > From your e-mail, I understood that you are developing the 2.5 branch > of the iCAP client. I am only fixing up some issues seen when reading the patch. The bulk of the ICAP client is developed by Geetha Manjunath with help from Ralf Horstmann. > Do you need some help, maybe in testing and debugging for first? If > yes, then I'll be glad if I can help you! Help is needed in all areas. There is many small things still missing in the Squid-2.5 ICAP client, and a lot of testing to do. I am trying to keep some notes on http://devel.squid-cache.org/icap/ > Regarding the 3.0 branch, I understood that the software architecture > of squid has changed, and it is also not C anymore, but C++, Correct. A lot of the code is still "C" code however, merely translated to C++ syntax, but gradually more and more is refactored into an OO architecture. As a guideline each area of the Squid code extended in Squid-3 (the C++ version of Squid) is refactored into C++ and cleaned up design to get rid of many ugly things which have grown onto the old C code over the years. > which is great for me, because I have much more experience in OOP as > with structured programming ( although I programed in C during my > "Uni" period, back in '92-'97, since then I've programmed mostly in > C++ and Java ). Very Good. Regards Henrik
Squid ICAP client possibly bugs and corrections......
Hi squid developers,
I want to report 1-2 small bugs and corrections at squid
icap client interface that are relative with keeped alive
requests to an icap server.
I am using a (corrected) squid-icap named squid-2.5.STABLE5-icap-6-pre1.
but as I seen this bugs exists and in squid-icap-2.5-200405131634
icap enabled squid.
1)when icapReadReply3 called after a response request with null-body
(e.g when the content of a web page not changed ...)
then enters the block "if (EBIT_TEST(i..., ENTRY_ABORTED)){".
Squid icap-client has request keeped alive connection and
now the connection not closed. This behaviour can cause
a lot of open connection to an icap server until connections expired.
An addition of comm_close(fd) in this block can solve the problem.
Moreover if think there is no need for the connection to be closed
in this phase and can keeped alive. So the entire block :
if (EBIT_TEST(i..., ENTRY_ABORTED)){
...
}
in function icapReadReply3 can removed.
2) In file icap_reqmod.c in function icapSendReqMod
when compute the position of entities for
"Encapsulated:" header squid does not include crlf (2 bytes) in
possition computation.
This fields are the possition in the header not the size of the header.
I propose to
a) add a "crlf" at the end of "icap->request->header"
before Encapsulated header computation
eg "memBufAppend(&mb_hdr, crlf, 2);" at line 613
and
b) remove the "memBufAppend(&mb, crlf, 2);" line 637 that adds the
crlf at the end of
entire buffer.
Regards,
Christos
3 fixes for the squid icap implementation
Hi,
Could someone add the 3 attached patches to the squid icap client?
Here's a description of what they fix...
closebeforesendfix.patch
I noticed that sometimes requests to the site http://www.retail-week.com
would fail to load a certain component and the page wouldn't finish
loading until a timeout occurred. I found that this was due to the
origin server closing the connection without sending any data and squid
wasn't handling this cleanly. This patch frees the icap connection
immediately so that we don't have to wait for the timeout.
http0.9fix.patch
I noticed that requests to the site
http://in.sports.yahoo.com/sportz/homeimages/rightheader_quiz.gif would
fail to load. This was because the origin server does not add any http
headers to the response (http 0.9) and the squid icap code wasn't
handling it well. This patch fixes the problem.
squidgrowthfix.patch
I noticed that if a large file was being downloaded through the squid
icap code then this could cause squid to grow. The growth would occur if
the browser had prompted the user asking them to open or save the file
but the user had not responded - the file would be stored in mem_nodes.
The memory that was used would often not be returned to the system
because of memory fragmentation making the problem even worse. This
patch adds to commSetDefer() calls that will avoid the growth by
deferring reads from both the origin server and the icap server until
the http client is ready to receive the data.
Thanks,
Graeme
This message has been scanned for viruses by BlackSpider MailControl -
www.blackspider.com
diff -urN squid-2.5.STABLE12/src/icap_respmod.c
squid-2.5.STABLE12-closebeforesendfix/src/icap_respmod.c
--- squid-2.5.STABLE12/src/icap_respmod.c 2006-01-05 16:26:37.0
+
+++ squid-2.5.STABLE12-closebeforesendfix/src/icap_respmod.c2006-01-05
16:52:42.0 +
@@ -252,6 +252,16 @@
#endif
if (icap->sc == 0) {
+// http connection has been closed without sending us anything
+if(len == 0 && theEnd == 1) {
+ ErrorState *err;
+ err = errorCon(ERR_INVALID_RESP, HTTP_BAD_GATEWAY);
+ err->request = requestLink(icap->request);
+ errorAppendEntry(icap->respmod.entry, err);
+ comm_close(icap->icap_fd);
+ return;
+}
+
/* No data sent yet. Start with headers */
if ((icap->sc = buildRespModHeader(&mb, icap, buf, len, theEnd)) > 0) {
buf += icap->sc;
diff -urN squid-2.5.STABLE12/src/icap_respmod.c
squid-2.5.STABLE12-http0.9fix/src/icap_respmod.c
--- squid-2.5.STABLE12/src/icap_respmod.c 2006-01-05 16:26:37.0
+
+++ squid-2.5.STABLE12-http0.9fix/src/icap_respmod.c2006-01-05
16:51:08.0 +
@@ -104,6 +104,9 @@
consumed = -1;
o2 = -1;
memBufDefInit(&mb_hdr);
+ httpBuildRequestPrefix(icap->request, icap->request,
+ icap->respmod.entry, &mb_hdr, icap->http_flags);
+ o3 = mb_hdr.size;
} else {
hlen = headersEnd(icap->respmod.req_hdr_copy.buf,
@@ -132,12 +135,12 @@
httpBuildRequestPrefix(icap->request, icap->request,
icap->respmod.entry, &mb_hdr, icap->http_flags);
o2 = mb_hdr.size;
-}
-/* Copy response header - Append to request header mbuffer */
-memBufAppend(&mb_hdr,
- icap->respmod.req_hdr_copy.buf, icap->respmod.req_hdr_copy.size);
-o3 = mb_hdr.size;
+ /* Copy response header - Append to request header mbuffer */
+ memBufAppend(&mb_hdr,
+icap->respmod.req_hdr_copy.buf,
icap->respmod.req_hdr_copy.size);
+ o3 = mb_hdr.size;
+}
service = icap->current_service;
assert(service);
diff -urN squid-2.5.STABLE12/src/http.c
squid-2.5.STABLE12-squidgrowthfix/src/http.c
--- squid-2.5.STABLE12/src/http.c 2006-01-05 16:26:37.0 +
+++ squid-2.5.STABLE12-squidgrowthfix/src/http.c2006-01-05
16:53:30.0 +
@@ -629,6 +629,10 @@
commSetSelect(fd, COMM_SELECT_READ, httpReadReply, httpState, 0);
return;
}
+
+ if(icap->flags.no_content == 1) {
+commSetDefer(fd, fwdCheckDeferRead, icap->respmod.entry);
+ }
}
#endif
diff -urN squid-2.5.STABLE12/src/icap_respmod.c
squid-2.5.STABLE12-squidgrowthfix/src/icap_respmod.c
--- squid-2.5.STABLE12/src/icap_respmod.c 2006-01-05 16:26:37.0
+
+++ squid-2.5.STABLE12-squidgrowthfix/src/icap_respmod.c2006-01-05
16:53:59.0 +
@@ -627,6 +627,7 @@
commSetSelect(fd, COMM_SELECT_READ, icapRespModReadReply, icap, 0);
#if 1
commSetTimeout(fd, Config.Timeout.read, icapReadTimeout, icap);
+ commSetDefer(fd, fwdCheckDeferRead, icap->respmod.entry);
#else
if (icap->flags.wait_for_preview_reply || icap->flags.http_server_eof) {
/*
Re: context data for squid ICAP patched
Hi Beeri, Maybe you do not need to modify the squid-icap code to support your model. I think that the correct implementation of your problem using squid-icap is: 1) An http request come into the squid. Squid sends the reqmod request to the icap server and server clasifies the request: a) In the case of the BLOCK icap server creates a http responce saying to the web client that the request blocked b) In the case of UNKNOWN icap server does nothing c) In the case of ALLOW icap server adds a proprietary http header to the http request for example "X-MY-SCANNER: Allow" 2) When squid has the http responce then sends a respmod request to the icap server. The respmod request contains the http responce headers AND the http request headers. a) When icap server founds the "X-MY-SCANNER: Allow" header in http request headers it responds with an allow204 responce to squid b)The "X-MY-SCANNER: Allow" is not in the http request headers so the icap server takes the http body from squid and check it or modify it or what else. An other solutions is to use only the respmod request becouse here you have both the http request headers and the http responce. > The Question: > > I would like to pass the information that, no call to response mode > (call the ICAP Server for the response) is needed. > ... I am not sure that I fully understand your question, but I think that this functionality can not included in a general ICAP client of squid. But maybe I am loosing something here. Regards, Christos > Background information: > > I am implementing an extension to squid ICAP client based upon ICAP > Patch and squid 2.5 STABLE 10. > The squid ICAP client does not support Content Filtering the way we at > PureSight.com using it. > The ICAP protocol is defined to support also Content Filtering and > defines a return value at the request mod stage. > I receive the value that can be one of the following: > ALLOW, > BLOCK, > UNKNOWN > > > .. >
RE: context data for squid ICAP patched
Hi Christos, Thank you for your help, but you suggestion is not secure nor best perform, Please read my other remarks below. Now that I read the question again I see it is not clear enough, I will ask again. I would like squid ICAP client to do the logic for couple of reasons, 1. Security - Origin sever might change the replied http header and add the "X-MY-SCANNER: Allow" it self, and bypass the content filter, In that case I would not be able to prevent kids from viewing un honest pages :-( 2. Performance - Redundant call since I already know that request is allowed. There for I would like to keep in squids session data the classification and upon to the classification prior to response-mod call. For now I have figured out that the best place to set the data between the req-mod and resp-mod if in the fde structure, but since squid saves that object in fd_table (hash?) keyed by ICAP FD there is no continuity with the HTTP FD. I realized that I need to look for the mechanism that changes the next handler (hdl) that switch FD to read from, is the KEY to set up the fde related to the HTTP response, with the classification information. In squid ICAP client implementation there is no connection between the FD sets, ICAP's and HTTP's. Again 10X for reading and good will, I hope there is a short cut out there, If someone has an implementation suggestion or realizes I am missing something please write me. > > Hi Beeri, > Maybe you do not need to modify the squid-ICAP code to > support your model. > I think that the correct implementation of your problem using > squid-ICAP > is: > > 1) An http request come into the squid. Squid sends the > reqmod request to the ICAP server and server classifies the request: >a) In the case of the BLOCK ICAP server creates a http > response saying > to the web client that the request blocked >b) In the case of UNKNOWN ICAP server does nothing >c) In the case of ALLOW ICAP server adds a proprietary http header > to the http request for example "X-MY-SCANNER: Allow" > > 2) When squid has the http response then sends a respmod > request to the >ICAP server. The respmod request contains the http response >headers AND the http request headers. > a) When ICAP server founds the "X-MY-SCANNER: Allow" header > in http request headers it responds with an allow204 > response to > squid > b)The "X-MY-SCANNER: Allow" is not in the http request headers >so the ICAP server takes the http body from squid and > check it or >modify it or what else. > > > An other solutions is to use only the respmod request because > here you have both the http request headers and the http response. > > > The Question: > > > > I would like to pass the information that, no call to response mode > > (call the ICAP Server for the response) is needed. > > ... > > I am not sure that I fully understand your question, but I > think that this functionality can not included in a general > ICAP client of squid. > But maybe I am loosing something here. > > Regards, >Christos > > > Background information: > > > > I am implementing an extension to squid ICAP client based upon ICAP > > Patch and squid 2.5 STABLE 10. > > The squid ICAP client does not support Content Filtering > the way we at > > PureSight.com using it. > > The ICAP protocol is defined to support also Content Filtering and > > defines a return value at the request mod stage. > > I receive the value that can be one of the following: > > ALLOW, > > BLOCK, > > UNKNOWN > > > > > > .. > > > > > > >
RE: context data for squid ICAP patched
Hi Beeri, > 1. Security - Origin sever might change the replied http header and add > the "X-MY-SCANNER: Allow" it self, > and bypass the content filter, In that case I would not > be able to prevent kids from viewing un honest pages :-( Yes this problem exists. I am just refering it as way to pass info from reqmod to respmod icap services. > 2. Performance - Redundant call since I already know that request is > allowed. Yes the performance is a problem here. Assuming that you have 95% accepted requests and only 5% blocked maybe is not so problem to make your checks only in respmod request. But you know beter than me your requirements. > > There for I would like to keep in squids session data the classification > and upon to the classification prior to response-mod call. > > For now I have figured out that the best place to set the data between > the req-mod and resp-mod if in the > ... I am not sure but I think that the best place to put your classification data is the request_t structure. Regards, Christos
2 fixes for the squid icap implementation
Hi, Could someone add the 2 attached patches to the squid icap client? Here's a description of what they fix... 15minutefix.patch = I noticed that if an icap server replied with a 204 response, the download would continue without being forwarded to icap but it would then abort after 15 minutes. This was because the read timeout on the icap connection wasn't being cancelled. This fix cancels the timeout which avoids the aborted downloads. squidcrashfix.patch === A fix that I submitted some time ago (squidgrowthfix.patch) caused squid to crash under heavy load. This was because the deferred read check I added relied on the store entry object being present but didn't lock it. Under heavy load the store entry was sometimes freed before the defer read check which caused squid to crash. This patch maintains a lock on the store entry object and also has some other code to make sure the store entry object is unlocked and freed when it is no longer required. Thanks, Graeme This message has been scanned for viruses by BlackSpider MailControl - www.blackspider.com 15minutefix.patch Description: 15minutefix.patch squidcrashfix.patch Description: squidcrashfix.patch
Re: 2 fixes for the squid icap implementation
Hi Graeme,
I did not verified the problems but looks that the problems exist.
About the first patch I am proposing a somehow different approach.
When squid-icap takes a 204 response immediately releases the connection
and put it to connections poll so it can be used for other icap requests.
The icap connection's file descriptor is no more related to the
IcapStateData structure and only we have to free this structure when we
have all data from the http server.
The patch is attached if you want to test it.
I must verify it and correct it before apply it to the cvs.
If it causes problems I will proceed with your patch.
About the second patch if there is not any problem I want to test it
more and if it is possible to move most of its code to the icap_*.c
files. This will save as time during merging with main branch..
Regards,
Christos
Graeme Bisset wrote:
> Hi,
>
> Could someone add the 2 attached patches to the squid icap client?
>
> Here's a description of what they fix...
>
> 15minutefix.patch
> =
> I noticed that if an icap server replied with a 204 response, the
> download would continue without being forwarded to icap but it would
> then abort after 15 minutes. This was because the read timeout on the
> icap connection wasn't being cancelled. This fix cancels the timeout
> which avoids the aborted downloads.
>
> squidcrashfix.patch
> ===
> A fix that I submitted some time ago (squidgrowthfix.patch) caused squid
> to crash under heavy load. This was because the deferred read check I
> added relied on the store entry object being present but didn't lock it.
> Under heavy load the store entry was sometimes freed before the defer
> read check which caused squid to crash. This patch maintains a lock on
> the store entry object and also has some other code to make sure the
> store entry object is unlocked and freed when it is no longer required.
>
> Thanks,
>
> Graeme
diff -u -r1.1.2.65 icap_respmod.c
--- icap_respmod.c 25 May 2006 16:04:55 - 1.1.2.65
+++ icap_respmod.c 15 Jun 2006 18:01:52 -
@@ -500,6 +500,8 @@
#if SUPPORT_ICAP_204 || ICAP_PREVIEW
if (204 == status) {
debug(81, 3) ("got 204 status from ICAP server\n");
+ icapRespModKeepAliveOrClose(icap,0);
+
debug(81, 3) ("setting icap->flags.no_content\n");
icap->flags.no_content = 1;
/*
@@ -511,7 +513,8 @@
icap->respmod.resp_copy.buf, icap->respmod.resp_copy.size);
icap->respmod.resp_copy.size = 0;
if (icapReadReply2(icap) < 0)
- comm_close(fd);
+ icapStateFree(-1, icap);
+// comm_close(fd);
/*
* XXX ideally want to clean icap->respmod.resp_copy here
* XXX ideally want to "close" ICAP server connection here
@@ -801,26 +804,29 @@
* transaction.
*/
static void
-icapRespModKeepAliveOrClose(IcapStateData * icap)
+icapRespModKeepAliveOrClose(IcapStateData * icap,int free_icap_state)
{
int fd = icap->icap_fd;
if (fd < 0)
return;
-if (!icap->flags.keep_alive) {
- debug(81, 3) ("%s:%d keep_alive not set, closing\n", __FILE__,
- __LINE__);
- comm_close(fd);
- return;
-}
debug(81, 3) ("%s:%d FD %d looks good, keeping alive\n", __FILE__, __LINE__,
fd);
commSetDefer(fd, NULL, NULL);
commSetTimeout(fd, -1, NULL, NULL);
commSetSelect(fd, COMM_SELECT_READ, NULL, NULL, 0);
comm_remove_close_handler(fd, icapStateFree, icap);
-pconnPush(fd, icap->current_service->hostname, icap->current_service->port);
icap->icap_fd = -1;
-icapStateFree(-1, icap);
+if(free_icap_state)
+ icapStateFree(-1, icap);
+if (!icap->flags.keep_alive) {
+ debug(81, 3) ("%s:%d keep_alive not set, closing\n", __FILE__,
+ __LINE__);
+ comm_close(fd);
+ return;
+}
+else{
+ pconnPush(fd, icap->current_service->hostname, icap->current_service->port);
+}
}
@@ -1042,7 +1048,10 @@
comm_close(fd);
} else if (icapPconnTransferDone(fd, icap)) {
storeComplete(entry);
- icapRespModKeepAliveOrClose(icap);
+ if(icap->flags.no_content)
+ icapStateFree(-1, icap);
+ else
+ icapRespModKeepAliveOrClose(icap,1);
} else if (!icap->flags.no_content) {
/* Wait for EOF condition */
commSetSelect(fd, COMM_SELECT_READ, icapReadReply, icap, 0);
Patch for load-balancing et HA in Squid-ICAP client (fwd)
-- Forwarded message --
Date: Mon, 04 Oct 2004 17:06:44 +0200
From: Stephane DAVY <[EMAIL PROTECTED]>
To: [EMAIL PROTECTED]
Subject: [squid-users] Patch for load-balancing et HA in Squid-ICAP client
Hello,
please find below a message posted on the squid-icapClient ML. Actually,
there is not so much activity on this list even if people are interested
in ICAP stuff in Squid. This message deals with HA and load-balancing,
and testing is needed so any feedback is welcome.
Thanks,
Stéphane
Objet: [squid-icapClient] Patch for load-balancing et HA
Date: Mon, 04 Oct 2004 14:54:34 +0200
Hello all,
here is a patch from Luc Saillard (Alcove company) which implements
load-balancing and HA. You can define a service using different
serveurs, and for each request we take the next server if this one is
reachable:
icap_service service_1 reqmod_precache icap://server1:1344/wwreqmod
icap_service service_1 reqmod_precache icap://server2:1344/wwreqmod
icap_service service_1 reqmod_precache icap://server3:1344/wwreqmod
The patch should be applied against the latest tarball available here:
http://www.squid-cache.org/~wessels/squid-icap-2.5/
Dont' forget to run bootstrap.sh before configure
Feedback is welcome at the following address:
[EMAIL PROTECTED] and of course on this ML
Enjoy!
--
Stephane DAVY <[EMAIL PROTECTED]>--- squid-icap-2.5-200409161544.orig/src/cache_cf.c Wed Aug 4 21:47:58 2004
+++ squid-icap-2.5-200409161544/src/cache_cf.c Tue Sep 28 15:44:03 2004
@@ -2299,13 +2299,27 @@
*/
static void
-icap_service_list_add(icap_service_list ** isl, icap_service * service)
+icap_service_list_add(icap_service_list ** isl, char * service_name)
{
icap_service_list **iter;
icap_service_list *new;
+icap_service *gbl_service;
+int i;
+int max_services;
new = memAllocate(MEM_ICAP_SERVICE_LIST);
-new->service = service;
+/* Found all services with that name, and add to the array */
+max_services = sizeof(new->services)/sizeof(icap_service *);
+gbl_service = Config.icapcfg.service_head;
+i=0;
+while(gbl_service && i < max_services) {
+ if (!strcmp(service_name, gbl_service->name)) {
+ new->services[i++] = gbl_service;
+ break;
+ }
+ gbl_service = gbl_service->next;
+}
+new->nservices = i;
if (*isl) {
iter = isl;
@@ -2400,7 +2414,7 @@
for (iter = c->services; iter; iter = iter->next) {
service = icap_service_lookup(iter->key);
if (service) {
- icap_service_list_add(&isl, service);
+ icap_service_list_add(&isl, iter->key);
} else {
debug(3, 0) ("icap_class_process (line %d): skipping service %s in class %s\n", config_lineno, iter->key, c->name);
}
@@ -2493,7 +2507,9 @@
c->hidden = 1;
wordlistAdd(&c->services, A->service_name);
c->isl = memAllocate(MEM_ICAP_SERVICE_LIST);
- c->isl->service = s;
+ /* FIXME:luc: check what access do */
+ c->isl->services[0] = s;
+ c->isl->nservices = 1;
icap_class_add(c);
A->class = c;
} else {
@@ -2592,7 +2608,9 @@
printf(" %s: \n", c_iter->name);
printf("services = \n");
for (isl_iter = c_iter->isl; isl_iter; isl_iter = isl_iter->next) {
- printf(" %s\n", isl_iter->service->name);
+ int i;
+ for (i = 0; i < isl_iter->nservices; i++)
+ printf(" %s\n", isl_iter->services[i]->name);
}
}
debug(3, 0) ("IcapConfig: access =\n");
--- squid-icap-2.5-200409161544.orig/src/icap_common.c Sat Apr 3 23:12:55 2004
+++ squid-icap-2.5-200409161544/src/icap_common.c Tue Sep 28 15:36:03 2004
@@ -140,6 +140,8 @@
icapService(icap_service_t type, request_t * r)
{
icap_service_list *isl_iter;
+int is_iter;
+
debug(81, 8) ("icapService: type=%s\n", icapServiceToStr(type));
if (NULL == r) {
debug(81, 8) ("icapService: no request_t\n");
@@ -150,10 +152,27 @@
return NULL;
}
for (isl_iter = r->class->isl; isl_iter; isl_iter = isl_iter->next) {
- if (type == isl_iter->service->type) {
- debug(81, 8) ("icapService: found service %s\n", isl_iter->service->name);
- return isl_iter->service;
- }
+/* TODO:luc: Do a round-robin, choose a random value ?
+ * For now, we use a simple round robin with checking is the
+ * icap server is available */
+ is_iter = isl_iter->last_service_used;
+ do
+ {
+ is_iter = (is_iter + 1) % isl_iter->nservices;
+ debug(81, 9) ("icapService: checking service %s/id=%d\n",isl_iter->services[is_iter]->name,is_iter);
+ if (type == isl_iter->services[is_iter]->type)
+ {
+ if (!isl_iter->services[is_iter]->unreachable)
+ {
+ debug(81, 8) ("icapService: found service %s/id=%d\n", isl_iter->services
