I would like to subscribe to squid mailing list

[Moshe Beeri]

Hi,

I am Moshe Beeri, and I work for PureSight (www.PureSight.com)
Recently I applied the ICAP patch into squid 2.5 stable 10,
We at PureSight are using ICAP for content filtering and we are using
our own server, as I worked with squid ICAP I realized that the support
for content filtering is not yet defined.
As I walked trough the code I realized that I need some more information
about squid functions and data storage.
I would like to pass the classification info from the request mod to the
response mod, preventing unnecessary call to request mod in case the
classification is known to be ALLOW.
We, at PureSight, contribute the development applying our solution to
the community.
I hope that this solution be best to squid lovely community as well as
squid users.

Thanks,
Moshe Beeri. 
Software Engineer, Server Team.
[EMAIL PROTECTED]
Petch-Tiqva Bazel 16, Israel.
Tel: +972 (3) 928-0400 ext. 429
Fax: +972 (3) 921-7594

context data for squid ICAP patched

[Moshe Beeri]

Hi all squids,


Background information:

I am implementing an extension to squid ICAP client based upon ICAP
Patch and squid 2.5 STABLE 10.
The squid ICAP client does not support Content Filtering the way we at
PureSight.com using it.
The ICAP protocol is defined to support also Content Filtering and
defines a return value at the request mod stage.
I receive the value that can be one of the following: 
ALLOW, 
BLOCK, 
UNKNOWN


So far as described by the ICAP protocol (RFC 3507) and implemented by
our ICAP server.
As for squid client I added the ability to identify classification
value, I would like squid to behave according to that value:
1. In case of UNKNOWN the response should go back to the ICAP server to
be reclassified.
2. In case of BLOCK the ICAP server is changing the URL to retrieve a
blocking information page.
3. In case of ALLOW squid should reply directly to the client with or
without using the cache.


The Question:

I would like to pass the information that, no call to response mode
(call the ICAP Server for the response) is needed.
I would like your help with the way squid saves session data, I look at
the code and I could not find that particular object, is there one?
Any suggestions && || directions would be blessed.



The Polite part:

Thanks for reading
&&
Participating on that grate project of squid.
&&
Your help




Moshe Beeri. 
Software Engineer, Servers Team.
[EMAIL PROTECTED]
Petch-Tiqva Bazel 16, Israel.
Tel: +972 (3) 928-0400 ext. 429
Fax: +972 (3) 921-7594
 

Re: context data for squid ICAP patched

[Tsantilas Christos]

Hi Beeri,
 Maybe you do not need to modify the squid-icap code to support your model.
I think that the correct implementation of your problem using squid-icap
is:

1) An http request come into the squid. Squid sends the reqmod request to
 the icap server and server clasifies the request:
   a) In the case of the BLOCK icap server creates a http responce saying
  to the web client that the request blocked
   b) In the case of UNKNOWN icap server does nothing
   c) In the case of ALLOW icap server adds a proprietary http header
  to the http request for example "X-MY-SCANNER: Allow"

2) When squid has the http responce then sends a respmod request to the
   icap server. The respmod request contains the http responce
   headers AND the http request headers.
 a) When icap server founds the "X-MY-SCANNER: Allow" header
in http request headers it responds with an allow204 responce to
squid
 b)The "X-MY-SCANNER: Allow" is not in the http request headers
   so the icap server takes the http body from squid and check it or
   modify it or what else.


An other solutions is to use only the respmod request becouse
here you have both the http request headers and the http responce.

> The Question:
>
> I would like to pass the information that, no call to response mode
> (call the ICAP Server for the response) is needed.
> ...

I am not sure that I fully understand your question, but I think that
this functionality can not included in a general ICAP client of squid.
But maybe I am loosing something here.

Regards,
   Christos

> Background information:
>
> I am implementing an extension to squid ICAP client based upon ICAP
> Patch and squid 2.5 STABLE 10.
> The squid ICAP client does not support Content Filtering the way we at
> PureSight.com using it.
> The ICAP protocol is defined to support also Content Filtering and
> defines a return value at the request mod stage.
> I receive the value that can be one of the following:
> ALLOW,
> BLOCK,
> UNKNOWN
>
>
> ..



>

Re: Squid-2.6 update

[Henrik Nordstrom]

Please keep discussions on the mailinglist, thanks.

> Well, then I againg suggest cosmetic changes for C++ compatibility:
> ushort class;  to ushort  _class;  in cache_cf.c
> 
> int class;  to int _class;  in  delay_pools.c
> 
> unsigned char *class;  to unsigned char *_class;  in  structs.h

Submit a patch please.

Regards
Henrik


signature.asc
Description: Detta är en digitalt signerad	meddelandedel

Re: making cache manager presence/registration optional

[Gonzalo Arana]

Hi,

I believe there is a little problem with this.  If store digest is not
enabled.  I get:
store_digest.cc: In function `void
  storeDigestRegisterWithCacheManager(CacheManager&)':
store_digest.cc:135: error: `registerAction' undeclared (first use this
  function)
store_digest.cc:135: error: (Each undeclared identifier is reported only once
  for each function it appears in.)

Attached is my proposal fix for this.

Regards,

On 5/28/06, Robert Collins <[EMAIL PROTECTED]> wrote:

I hadn't heard anything back, so I've committed what I think is a
tasteful implementation of the second option. This has allowed removing
the stub_cache_manager.cc test suite file, as well as making a bunch of
modules' init simpler.

Cheers,
Rob

On Sun, 2006-05-28 at 00:18 +1000, Robert Collins wrote:
> I'd like to make the cachemgrRegister calls in various of our fooInit()
> calls not require dragging in the whole squid to the binary, this is
> part of the blow-out on linked in objects for squid.
>
> Secondly, I'd like to remove the idea of the cachemanager being a global
> object and make it be explicitly passed in when it exists.
>
> We discussed this somewhat on irc.
>
> Some possibilities:
>
> Assuming we have a CacheManager class with 'register' and 'unregister'
> virtual methods, we could:
>
> * add that as a parameter to the Init calls where this is desirable.
> * Have a separate call from Init in modules which asks the module to
> register all its menu items with a supplied CacheManager.
>
> I prefer the second option, as it makes the behaviour of init more
> occupied with 'what is required to use a module' rather than 'do
> everything that /squid/ needs done before running that is related to
> this module.' Henrik is concerned that this will increase the
> maintenance cost of main(), which is possibly true, but I think we can
> address that in future if needed, i.e. by a registry of the modules with
> a few functions like 'init', 'registerWithCacheManger' etc.
>
> Thoughts?
>
> Rob
--
GPG key available at: .


-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.2.2 (GNU/Linux)

iD8DBQBEej3JM4BfeEKYx2ERAoPWAJ4lSy5Rff8itOhm5WfyLcs06CA63ACgluZb
OcntB2Y7OpvtYHxqh/MxQPk=
=XJhV
-END PGP SIGNATURE-






--
Gonzalo A. Arana
Index: store_digest.cc
===
RCS file: /cvsroot/squid/squid3/src/store_digest.cc,v
retrieving revision 1.15
diff -U6 -p -r1.15 store_digest.cc
--- store_digest.cc	29 May 2006 00:50:18 -	1.15
+++ store_digest.cc	29 May 2006 13:26:56 -
@@ -38,15 +38,15 @@
  * TODO: We probably do not track all the cases when
  *   storeDigestNoteStoreReady() must be called; this may prevent
  *   storeDigestRebuild/write schedule to be activated
  */
 
 #include "squid.h"
+#include "CacheManager.h"
 #if USE_CACHE_DIGESTS
 
-#include "CacheManager.h"
 #include "Store.h"
 #include "HttpRequest.h"
 #include "HttpReply.h"
 #include "MemObject.h"
 #include "SquidTime.h"
 #include "StoreSearch.h"

Re: making cache manager presence/registration optional

[Gonzalo Arana]

I've just noticed these other issues:
1) epoll is enabled
2) xprof is not used
3) delay pools are not enabled

Attached is a proposal for this (supersedes my previous patch).

Regards,

On 5/29/06, Gonzalo Arana <[EMAIL PROTECTED]> wrote:

Hi,

I believe there is a little problem with this.  If store digest is not
enabled.  I get:
store_digest.cc: In function `void
   storeDigestRegisterWithCacheManager(CacheManager&)':
store_digest.cc:135: error: `registerAction' undeclared (first use this
   function)
store_digest.cc:135: error: (Each undeclared identifier is reported only once
   for each function it appears in.)

Attached is my proposal fix for this.

Regards,

On 5/28/06, Robert Collins <[EMAIL PROTECTED]> wrote:
> I hadn't heard anything back, so I've committed what I think is a
> tasteful implementation of the second option. This has allowed removing
> the stub_cache_manager.cc test suite file, as well as making a bunch of
> modules' init simpler.
>
> Cheers,
> Rob
>
> On Sun, 2006-05-28 at 00:18 +1000, Robert Collins wrote:
> > I'd like to make the cachemgrRegister calls in various of our fooInit()
> > calls not require dragging in the whole squid to the binary, this is
> > part of the blow-out on linked in objects for squid.
> >
> > Secondly, I'd like to remove the idea of the cachemanager being a global
> > object and make it be explicitly passed in when it exists.
> >
> > We discussed this somewhat on irc.
> >
> > Some possibilities:
> >
> > Assuming we have a CacheManager class with 'register' and 'unregister'
> > virtual methods, we could:
> >
> > * add that as a parameter to the Init calls where this is desirable.
> > * Have a separate call from Init in modules which asks the module to
> > register all its menu items with a supplied CacheManager.
> >
> > I prefer the second option, as it makes the behaviour of init more
> > occupied with 'what is required to use a module' rather than 'do
> > everything that /squid/ needs done before running that is related to
> > this module.' Henrik is concerned that this will increase the
> > maintenance cost of main(), which is possibly true, but I think we can
> > address that in future if needed, i.e. by a registry of the modules with
> > a few functions like 'init', 'registerWithCacheManger' etc.
> >
> > Thoughts?
> >
> > Rob
> --
> GPG key available at: .
>
>
> -BEGIN PGP SIGNATURE-
> Version: GnuPG v1.4.2.2 (GNU/Linux)
>
> iD8DBQBEej3JM4BfeEKYx2ERAoPWAJ4lSy5Rff8itOhm5WfyLcs06CA63ACgluZb
> OcntB2Y7OpvtYHxqh/MxQPk=
> =XJhV
> -END PGP SIGNATURE-
>
>
>


--
Gonzalo A. Arana






--
Gonzalo A. Arana
Index: src/comm_epoll.cc
===
RCS file: /cvsroot/squid/squid3/src/comm_epoll.cc,v
retrieving revision 1.11
diff -U6 -p -r1.11 comm_epoll.cc
--- src/comm_epoll.cc	29 May 2006 00:50:18 -	1.11
+++ src/comm_epoll.cc	29 May 2006 14:05:54 -
@@ -191,20 +191,36 @@ commSetSelect(int fd, unsigned int type,
 }
 
 if (timeout)
 F->timeout = squid_curtime + timeout;
 }
 
+
+static void commIncomingStats(StoreEntry * sentry);
+
+void
+commEPollRegisterWithCacheManager(CacheManager& manager)
+{
+manager.registerAction("comm_select_incoming",
+   "comm_incoming() stats",
+   commIncomingStats, 0, 1);
+}
+static void
+commIncomingStats(StoreEntry * sentry)
+{
+StatCounters *f = &statCounter;
+storeAppendPrintf(sentry, "Total number of epoll(2) loops: %d\n", statCounter.select_loops);
+storeAppendPrintf(sentry, "Histogram of returned filedescriptors\n");
+statHistDump(&f->select_fds_hist, sentry, statHistIntDumper);
+}
+
 /*
+ * comm_select
  * Check all connections for new connections and input data that is to be
  * processed. Also check for connections with data queued and whether we can
  * write it out.
- */
-
-/*
- * comm_select
  *
  * Called to do the new-style IO, courtesy of of squid (like most of this
  * new IO code). This routine handles the stuff we've hidden in
  * comm_setselect and fd_table[] and calls callbacks for IO ready
  * events.
  */
Index: src/main.cc
===
RCS file: /cvsroot/squid/squid3/src/main.cc,v
retrieving revision 1.62
diff -U6 -p -r1.62 main.cc
--- src/main.cc	29 May 2006 00:50:18 -	1.62
+++ src/main.cc	29 May 2006 14:05:55 -
@@ -864,13 +864,15 @@ mainInitialize(void)
 #ifdef USE_SELECT
 
 commSelectRegisterWithCacheManager(manager);
 #endif
 
 clientdbRegisterWithCacheManager(manager);
+#if DELAY_POOLS
 DelayPools::RegisterWithCacheManager(manager);
+#endif
 DiskIOModule::RegisterAllModulesWithCacheManager(manager);
 #if USE_DNSSERVERS
 
 dnsRegisterWithCacheManager(manager);
 #endif
 
@@ -892,13 +894,15 @@ mainInitialize(void)
 storeRegisterWithCacheManager(manager);
 #if DEBUGSTRINGS
 
 StringRegistry::Instanc

RE: context data for squid ICAP patched

[Moshe Beeri]

Hi  Christos,

Thank you for your help, but you suggestion is not secure nor best
perform, Please read my other remarks below.

Now that I read the question again I see it is not clear enough,
I will ask again.
I would like squid ICAP client to do the logic for couple of reasons,
1.  Security - Origin sever might change the replied http header and add
the "X-MY-SCANNER: Allow" it self, 
and bypass the content filter, In that case I would not
be able to prevent kids from viewing un honest pages :-(
2.  Performance - Redundant call since I already know that request is
allowed.

There for I would like to keep in squids session data the classification
and upon to the classification prior to response-mod call.

For now I have figured out that the best place to set the data between
the req-mod and resp-mod if in the
fde structure, but since squid saves that object in fd_table (hash?)
keyed by ICAP FD there is no continuity with the HTTP FD.
I realized that I need to look for the mechanism that changes the next
handler (hdl) that switch FD to read from, is the KEY to set
up the fde related to the HTTP response, with the classification
information.
In squid ICAP client implementation there is no connection between the
FD sets, ICAP's and HTTP's.


Again 10X for reading and good will,
I hope there is a short cut out there,
If someone has an implementation suggestion or realizes I am missing
something please write me.



> 
> Hi Beeri,
>  Maybe you do not need to modify the squid-ICAP code to 
> support your model.
> I think that the correct implementation of your problem using 
> squid-ICAP
> is:
> 
> 1) An http request come into the squid. Squid sends the 
> reqmod request to  the ICAP server and server classifies the request:
>a) In the case of the BLOCK ICAP server creates a http 
> response saying
>   to the web client that the request blocked
>b) In the case of UNKNOWN ICAP server does nothing
>c) In the case of ALLOW ICAP server adds a proprietary http header
>   to the http request for example "X-MY-SCANNER: Allow"
> 
> 2) When squid has the http response then sends a respmod 
> request to the
>ICAP server. The respmod request contains the http response
>headers AND the http request headers.
>  a) When ICAP server founds the "X-MY-SCANNER: Allow" header
> in http request headers it responds with an allow204 
> response to
> squid
>  b)The "X-MY-SCANNER: Allow" is not in the http request headers
>so the ICAP server takes the http body from squid and 
> check it or
>modify it or what else.
> 
> 
> An other solutions is to use only the respmod request because 
> here you have both the http request headers and the http response.
> 
> > The Question:
> >
> > I would like to pass the information that, no call to response mode 
> > (call the ICAP Server for the response) is needed.
> > ...
> 
> I am not sure that I fully understand your question, but I 
> think that this functionality can not included in a general 
> ICAP client of squid.
> But maybe I am loosing something here.
> 
> Regards,
>Christos
> 
> > Background information:
> >
> > I am implementing an extension to squid ICAP client based upon ICAP 
> > Patch and squid 2.5 STABLE 10.
> > The squid ICAP client does not support Content Filtering 
> the way we at 
> > PureSight.com using it.
> > The ICAP protocol is defined to support also Content Filtering and 
> > defines a return value at the request mod stage.
> > I receive the value that can be one of the following:
> > ALLOW,
> > BLOCK,
> > UNKNOWN
> >
> >
> > ..
> 
> 
> 
> >
> 
> 

RE: context data for squid ICAP patched

[Tsantilas Christos]

Hi Beeri,

> 1.  Security - Origin sever might change the replied http header and add
> the "X-MY-SCANNER: Allow" it self,
>   and bypass the content filter, In that case I would not
> be able to prevent kids from viewing un honest pages :-(

Yes this problem exists.
I am just refering it as way to pass info from reqmod to respmod icap
services.
> 2.  Performance - Redundant call since I already know that request is
> allowed.

Yes the performance is a problem here.
Assuming that you have 95% accepted requests and only 5% blocked
maybe is not so problem to make your checks only in respmod request.
But you know beter than me your requirements.

>
> There for I would like to keep in squids session data the classification
> and upon to the classification prior to response-mod call.
>
> For now I have figured out that the best place to set the data between
> the req-mod and resp-mod if in the
> ...

I am not sure but I think that the best place to put your classification
data is the request_t structure.

Regards,
   Christos

Re: making cache manager presence/registration optional

[Robert Collins]

On Mon, 2006-05-29 at 11:13 -0300, Gonzalo Arana wrote:
> I've just noticed these other issues:
> 1) epoll is enabled
> 2) xprof is not used
> 3) delay pools are not enabled
> 
> Attached is a proposal for this (supersedes my previous patch).

Thanks,
applied.

Rob
-- 
GPG key available at: .


signature.asc
Description: This is a digitally signed message part

2.6.PRE1 is out the door

[Henrik Nordstrom]

The 2.6.PRE1 release has been shipped.

  http://www.squid-cache.org/Versions/v2/2.6/

Have a good feeling about it in general, but much testing remains.


Open issues where you can help:

  - WCCPv2 workaround for Linux/IOS when having multiple routers. Should
use one connected socket per router.

  - Some ETag tunings to get rid of "Loop" warnings on odd responses
(Henrik)

  - Release notes still a bit empty. Only covers squid.conf changes, not
why or how to.. (Reuben)

  - diskd still as broken as in 2.5.. (see bugzilla)

  - and a number of other 2.5 issues which have not yet been solved (see
bugzilla)

  - Some benchmarking is needed to see if comm_poll is really worth to
keep, or if comm_select is actually better. The current comm_poll
implementation looks extremely inefficient (same as it always been in
2.x)..

Note: The comm event handling got shuffled around a little in the last
days making life considerably easier for epoll and select, which also
made the inefficiencies in the implementation of comm_poll much more
apparent.

Discussion:

  - Should we fix HTCP to comply with the RFC? The Squid implementation
has several errors in it's implementation of the HTCP protocol messages
making it quite outside of the RFC.. See Bug #1554.

Regards
Henrik


signature.asc
Description: Detta är en digitalt signerad	meddelandedel

Wccp2 config cleanup

[Steven Wilton]

Here's a small patch to wccp2 in squid 2.6.  It fixes a difference between
the code and the documentation in the default squid conf (proto= compared to
protocol=).  It also removes the "ports_defined" config variable, enables
the flag when ports are actually defined and checks to see that this flag is
set before the config is accepted.

Regards

Steven

-- 
No virus found in this outgoing message.
Checked by AVG Free Edition.
Version: 7.1.394 / Virus Database: 268.7.4/351 - Release Date: 29/05/2006
 
  


wccp2.patch
Description: Binary data

Re: 2.6.PRE1 is out the door

[Adrian Chadd]

On Tue, May 30, 2006, Henrik Nordstrom wrote:

>   - Some benchmarking is needed to see if comm_poll is really worth to
> keep, or if comm_select is actually better. The current comm_poll
> implementation looks extremely inefficient (same as it always been in
> 2.x)..

Poll should be better - it'll be much better if/when the comm backon/backoff
stuff is turned on by default and can be used to add/subtract pollfd
entries. I'm sure a little coding can go a long way in improving poll
and select.

I had some paper scribbles for how to do it before the SSL stuff
went in. I also had some checks to make sure delay pools kept working.
I'm now not so sure and won't be until after exams are over.

I'll vote to keep it in there. I'll attempt to tidy the code up after exams
whether or not its done in the 2.6-stable release timeframe.




Adrian

Re: 2.6.PRE1 is out the door

[Jason Titus]

Have you considered using libevent as a way to abstract out the polling 
method?  Seems like it might make porting easier and give you kqueue as 
well...


http://www.monkey.org/~provos/libevent/

Jason

On Mon, 29 May 2006 9:32 pm, Henrik Nordstrom wrote:

The 2.6.PRE1 release has been shipped.

  http://www.squid-cache.org/Versions/v2/2.6/

Have a good feeling about it in general, but much testing remains.


Open issues where you can help:

  - WCCPv2 workaround for Linux/IOS when having multiple routers. 
Should

use one connected socket per router.

  - Some ETag tunings to get rid of "Loop" warnings on odd responses
(Henrik)

  - Release notes still a bit empty. Only covers squid.conf changes, 
not

why or how to.. (Reuben)

  - diskd still as broken as in 2.5.. (see bugzilla)

  - and a number of other 2.5 issues which have not yet been solved 
(see

bugzilla)

  - Some benchmarking is needed to see if comm_poll is really worth to
keep, or if comm_select is actually better. The current comm_poll
implementation looks extremely inefficient (same as it always been in
2.x)..

Note: The comm event handling got shuffled around a little in the last
days making life considerably easier for epoll and select, which also
made the inefficiencies in the implementation of comm_poll much more
apparent.

Discussion:

  - Should we fix HTCP to comply with the RFC? The Squid implementation
has several errors in it's implementation of the HTCP protocol messages
making it quite outside of the RFC.. See Bug #1554.

Regards
Henrik