Re: Long response header problem
Ok, Did some additional checks, It should be --- src/http.cc 2008-04-01 13:54:38.0 +0200 +++ src/http.cc 2008-04-21 16:42:19.0 +0200 @@ -75,7 +75,7 @@ surrogateNoStore = false; fd = fwd-server_fd; readBuf = new MemBuf; -readBuf-init(4096, SQUID_TCP_SO_RCVBUF); +readBuf-init( SQUID_TCP_SO_RCVBUF, SQUID_TCP_SO_RCVBUF); orig_request = HTTPMSGLOCK(fwd-request); if (fwd-servers) Which is btw. the coding used in ICAP where SQUID_TCP_SO_RCVBUF = 16384 Regards, Axel Hi there, I ran, or better a customer ran into a problem today which sounded like this bug. http://www.squid-cache.org/bugs/show_bug.cgi?id=2001 So I applied the attached patch to squid-3.0.STABLE4 and did a quick test. Still the same problem. By cache.log looks like this ... comm_read_try: FD 16, size 4094, retval 2896, errno 0 ... HttpMsg::parse: failed to find end of headers (eof: 0) ... http.cc(1050) needs more at 2896 http.cc(1206) may read up to 1199 bytes from FD 16 ... comm_select(): got FD 16 events=1 monitoring=19 F-read_handler=1 F-write_handler=0 comm_select(): Calling read handler on FD 16 comm_read_try: FD 16, size 1198, retval 1198, errno 0 ... HttpMsg::parse: failed to find end of headers (eof: 0) ... http.cc(1050) needs more at 4094 http.cc(1206) may read up to 1 bytes from FD 16 ... comm_select(): got FD 16 events=1 monitoring=19 F-read_handler=0 F-write_handler=0 comm_select(): no read handler for FD 16 and so on and so on. So I checked the coding in http.cc and changed it as follows. --- src/http.cc 2008-04-01 13:54:38.0 +0200 +++ src/http.cc 2008-04-21 16:42:19.0 +0200 @@ -75,7 +75,7 @@ surrogateNoStore = false; fd = fwd-server_fd; readBuf = new MemBuf; -readBuf-init(4096, SQUID_TCP_SO_RCVBUF); +readBuf-init(16384, SQUID_TCP_SO_RCVBUF); orig_request = HTTPMSGLOCK(fwd-request); if (fwd-servers) Now it works but I am not sure if a.) this is a good solution and b.) a stable one :-). Maybe someone with more knowledge can do a check. Regards, Axel Westerhold DTS Systeme GmbH
Long response header problem
Hi there, I ran, or better a customer ran into a problem today which sounded like this bug. http://www.squid-cache.org/bugs/show_bug.cgi?id=2001 So I applied the attached patch to squid-3.0.STABLE4 and did a quick test. Still the same problem. By cache.log looks like this ... comm_read_try: FD 16, size 4094, retval 2896, errno 0 ... HttpMsg::parse: failed to find end of headers (eof: 0) ... http.cc(1050) needs more at 2896 http.cc(1206) may read up to 1199 bytes from FD 16 ... comm_select(): got FD 16 events=1 monitoring=19 F-read_handler=1 F-write_handler=0 comm_select(): Calling read handler on FD 16 comm_read_try: FD 16, size 1198, retval 1198, errno 0 ... HttpMsg::parse: failed to find end of headers (eof: 0) ... http.cc(1050) needs more at 4094 http.cc(1206) may read up to 1 bytes from FD 16 ... comm_select(): got FD 16 events=1 monitoring=19 F-read_handler=0 F-write_handler=0 comm_select(): no read handler for FD 16 and so on and so on. So I checked the coding in http.cc and changed it as follows. --- src/http.cc 2008-04-01 13:54:38.0 +0200 +++ src/http.cc 2008-04-21 16:42:19.0 +0200 @@ -75,7 +75,7 @@ surrogateNoStore = false; fd = fwd-server_fd; readBuf = new MemBuf; -readBuf-init(4096, SQUID_TCP_SO_RCVBUF); +readBuf-init(16384, SQUID_TCP_SO_RCVBUF); orig_request = HTTPMSGLOCK(fwd-request); if (fwd-servers) Now it works but I am not sure if a.) this is a good solution and b.) a stable one :-). Maybe someone with more knowledge can do a check. Regards, Axel Westerhold DTS Systeme GmbH
Re: Long response header problem
And one more. It might be this patch which solved the issue
--- src/http.cc 2008-04-01 13:54:38.0 +0200
+++ src/http.cc 2008-04-21 19:11:47.0 +0200
@@ -1200,7 +1200,7 @@
void
HttpStateData::maybeReadVirginBody()
{
-int read_sz = replyBodySpace(readBuf-spaceSize());
+int read_sz = replyBodySpace(readBuf-potentialSpaceSize());
debugs(11,9, HERE (flags.do_next_read ? may : wont)
read up to read_sz bytes from FD fd);
spaceSize will only return the size left from initial size. This will result
in read_sz2 and return some lines down in http.cc.
PotentialSpaceSize will return max_capacity - terminatedSize which seems
more logical.
Regards,
Axel Westerhold
DTS Systeme GmbH
Ok,
Did some additional checks,
It should be
--- src/http.cc 2008-04-01 13:54:38.0 +0200
+++ src/http.cc 2008-04-21 16:42:19.0 +0200
@@ -75,7 +75,7 @@
surrogateNoStore = false;
fd = fwd-server_fd;
readBuf = new MemBuf;
-readBuf-init(4096, SQUID_TCP_SO_RCVBUF);
+readBuf-init( SQUID_TCP_SO_RCVBUF, SQUID_TCP_SO_RCVBUF);
orig_request = HTTPMSGLOCK(fwd-request);
if (fwd-servers)
Which is btw. the coding used in ICAP where SQUID_TCP_SO_RCVBUF = 16384
Regards,
Axel
Hi there,
I ran, or better a customer ran into a problem today which sounded like this
bug.
http://www.squid-cache.org/bugs/show_bug.cgi?id=2001
So I applied the attached patch to squid-3.0.STABLE4 and did a quick test.
Still the same problem. By cache.log looks like this
...
comm_read_try: FD 16, size 4094, retval 2896, errno 0
...
HttpMsg::parse: failed to find end of headers (eof: 0)
...
http.cc(1050) needs more at 2896
http.cc(1206) may read up to 1199 bytes from FD 16
...
comm_select(): got FD 16 events=1 monitoring=19 F-read_handler=1
F-write_handler=0
comm_select(): Calling read handler on FD 16
comm_read_try: FD 16, size 1198, retval 1198, errno 0
...
HttpMsg::parse: failed to find end of headers (eof: 0)
...
http.cc(1050) needs more at 4094
http.cc(1206) may read up to 1 bytes from FD 16
...
comm_select(): got FD 16 events=1 monitoring=19 F-read_handler=0
F-write_handler=0
comm_select(): no read handler for FD 16
and so on and so on. So I checked the coding in http.cc and changed it as
follows.
--- src/http.cc 2008-04-01 13:54:38.0 +0200
+++ src/http.cc 2008-04-21 16:42:19.0 +0200
@@ -75,7 +75,7 @@
surrogateNoStore = false;
fd = fwd-server_fd;
readBuf = new MemBuf;
-readBuf-init(4096, SQUID_TCP_SO_RCVBUF);
+readBuf-init(16384, SQUID_TCP_SO_RCVBUF);
orig_request = HTTPMSGLOCK(fwd-request);
if (fwd-servers)
Now it works but I am not sure if a.) this is a good solution and b.) a
stable
one :-).
Maybe someone with more knowledge can do a check.
Regards,
Axel Westerhold
DTS Systeme GmbH
Re: Long response header problem
Hmm.. can't seem to reproduce this.
The proposed change do not fix the problem, just hides it a bit.
See my last mail of three (:-) sorry Not my best day)
The 3.0.STABLE4 code already bumps the read size to 1KB minimum when
headers haven't been successfully parsed yet. See
HttpStateData::maybeReadVirginBody()
Do you have an example URL triggering the problem?
Yes and no. The Url is including a authentication dialog I can't give you
the username and password for. I'll check if I can come up with something
similar.
Are you using ICAP?
ICAP is off for this test.
Any other interesting details about your configuration?
Nothing special. Actually the bug showed up on STABLE1 and I tested with a
STABLE4 without modifications (failed) patched with the longresp patch
(failed).
Regards
Henrik
As said, see my third mail:
SNIP
--- src/http.cc 2008-04-01 13:54:38.0 +0200
+++ src/http.cc 2008-04-21 19:11:47.0 +0200
@@ -1200,7 +1200,7 @@
void
HttpStateData::maybeReadVirginBody()
{
-int read_sz = replyBodySpace(readBuf-spaceSize());
+int read_sz = replyBodySpace(readBuf-potentialSpaceSize());
debugs(11,9, HERE (flags.do_next_read ? may : wont)
read up to read_sz bytes from FD fd);
spaceSize will only return the size left from initial size. This will result
in read_sz2 and return some lines down in http.cc.
PotentialSpaceSize will return max_capacity - terminatedSize which seems
more logical.
---SNIP
Regards,
Axel
Re: [contract gig] Squid cache expert needed!
Hi there, Personally I would use a combination of keepalived (Linux ony), haproxy (all kinds of Unix derivats) and squid. If needed you can run keepalived, haproxy and squid on the same maschine. Keepalived would supply the VIP needed using VRRP, haproxy would do the loadbalancing and cookie detection. For requests without cookie one of the squids would be used otherwise the request would terminate on the app servers directly. There are other ways to do this but this is currently my favorite and tried out more then once. Regards, Axel Am 20.04.2007 2:44 Uhr schrieb Cloude Porteus unter [EMAIL PROTECTED]: Squid developers, We would like to start using Squid-caches to take the load off of our application and http servers. We need someone who has hands-on experience setting up squid-cache clusters in a load balancing environment. We also need a custom cache filter written to check for the existence of a logged-in cookie, so that we only cache and serve pages for non-logged in users. Our proposed setup looks like this: internet | | firewall load balancer | | +--Squid-cache 1 +--Squid-cache 2 | | +--App server 1 +--App server 2 | | +--DB server, etc. I would like the Squid cache to also handle load balancing it's requests to the App servers, but the only documentation I can find is the setup Wikipedia had/has for it's servers. If this is not the case, we could put another load balancer between the Squid App servers, although I'd rather not have the extra server. Squid cache servers used response time measurements to distribute page requests between seven web servers. We have not ordered our servers yet, so I'm also looking for recommended server configurations for our two squid servers, ram, disk space, etc. We will be ordering servers as soon as we have this component of our solution worked out. Please feel free to ask for more details or make suggestions based on this information. Thanks, Cloude
Re: icap in squid3
Hi Alex,
Am 13.02.2007 6:03 Uhr schrieb Alex Rousskov unter
[EMAIL PROTECTED]:
On Mon, 2007-02-12 at 08:38 +0100, Axel Westerhold wrote:
Well, the syntax you are proposing is somewhat limited.
Here are my comments:
1.) cn=%u assumes that the used username equals the assigned CN which is
most of the time wrong. Normally the UID (or in AD the samaccountname) is
used for authentication. This will lead to a failure using cn=%u
2.) The given URI is not flexible enough as it assumes that all user object
will always be located within the same root object. The used syntax provides
fast access because it will avoid search operations but will fail as soon as
the object is located in a sub OU or a totally different tree.
3.) LDAP allows for all kinds of unicode chars we would need to encode
properly. While this is definately possible I wonder if there really should
be another encoding scheme impüplemented into squid.
It seems to me that Jeremy is not proposing any syntax, encoding, or URI
format (except perhaps for some default values). He wants to add ability
to use any URI, with any LDAP (or not LDAP) tags. The patch gives user a
set of supported substitutions. The user can use whatever substitution
codes they need in whatever opaque text filling they need. Please see
below for examples.
My point was that there is no use for a fixed LDAP template due to the way
LDAP is normally build. Let's move on to your proposal about substition and
encoding.
We should be able to agree on that set without much trouble because
adding more substitutions is not a problem. For example, if somebody
needs a username without a domain, there should be a substitution for
it.
If there are more than 5-7 substitutions, we may want to argue whether
single letter %S substitutions are better than easier-to-remember and
harder-to-mistype ${LongNames}. I would probably vote for the latter,
but it is not a big deal.
Alternative encodings should be supported, of course, perhaps as
$encodingName(string-to-be-encoded) substitution, where
string-to-be-encoded may have variable substitutions? Again, there is an
example below.
From what I can see there is following available info (if at all):
From Login:
Username
Prefix
Group
and that's it. Maybe my lack of fantasy let's me miss some of the additions.
One comment on a nice feature I would like to have but still considering for
security reasons:
When an ICAP Server requieres auth for user mapping to rules/policies you
sometimes run into a problem with sources with can't auth or destinations
you do not want to require auth for. While you can use ACL's to get this
done easily on squid sometimes the icap clients won't play ball. As a result
some destinations are not using the icap virus scanner/ content system to
make it work. So, maybe but just as a thought, it would be nice to use ACL's
to automatically assign a username für those services so that they can be
properly matched.
See my further comments below
What I think might work better is as follows:
A.) A user authenticates using a proper DN authenticating against an LDAP
Server. In this case the username will be the DN and can be transmitted.
B.) The user authenticates using a uid (samaccountname). Either this uid is
already usable on it's own an we can transfer it without any changes just
encoding it base64 if requested (which will keep us out of trouble with
UTF-8 or Unicode chars). In case we get this stuff from a windows user
sending us a domain prefix, we should be able to split the username into
domain and username. The hard part will be to find some kind of abstract how
to transfer this.
Encoding aside, can the above two requirements be expressed as a set of
substitutions?
They can but (see below)
What we definately need are the following configuration entries:
A.) Do we need to split the username into parts and if so using which
seperator. ('' = off or '\' or '+' etc.)
Can the separator be up to the admin? Do we need to define it?
Must be configurable so empty string turns off and non-empty turns on and
defines sperator. Samba for instance allows for Seperator modifications.
Also, this gives squid some flexibility.
B.) The X- Header used to transfer the username (bare username without any
instruction on how to use it (X-Authenticated-User, X-User, X-MyUser, X-Blah
etc.)
Agreed. The icap_client_username_header option controls that now, but
please see (C) below.
C.) The X-Header used to transfer the prefix if any.
Should we just support an arbitrary set of user-configurable header
names, with user-configurable values? If we add substitutions support,
then Squid should not really care about the meaning of the header. For
example,
icap_client_add_header X-Username $base64($UserName)
icap_client_add_header X-Prefix bar=$base64($Foo+$Bar)foo=blah
...
I like this from a technical point of view. But I can also see my
Re: icap in squid3
Good morning,
Well, the syntax you are proposing is somewhat limited.
Here are my comments:
1.) cn=%u assumes that the used username equals the assigned CN which is
most of the time wrong. Normally the UID (or in AD the samaccountname) is
used for authentication. This will lead to a failure using cn=%u
2.) The given URI is not flexible enough as it assumes that all user object
will always be located within the same root object. The used syntax provides
fast access because it will avoid search operations but will fail as soon as
the object is located in a sub OU or a totally different tree.
3.) LDAP allows for all kinds of unicode chars we would need to encode
properly. While this is definately possible I wonder if there really should
be another encoding scheme impüplemented into squid.
What I think might work better is as follows:
A.) A user authenticates using a proper DN authenticating against an LDAP
Server. In this case the username will be the DN and can be transmitted.
B.) The user authenticates using a uid (samaccountname). Either this uid is
already usable on it's own an we can transfer it without any changes just
encoding it base64 if requested (which will keep us out of trouble with
UTF-8 or Unicode chars). In case we get this stuff from a windows user
sending us a domain prefix, we should be able to split the username into
domain and username. The hard part will be to find some kind of abstract how
to transfer this.
What we definately need are the following configuration entries:
A.) Do we need to split the username into parts and if so using which
seperator. ('' = off or '\' or '+' etc.)
B.) The X- Header used to transfer the username (bare username without any
instruction on how to use it (X-Authenticated-User, X-User, X-MyUser, X-Blah
etc.)
C.) The X-Header used to transfer the prefix if any.
D.) Something to force base64 Encoding on above headers
This ensures that the ICAP Client get's all the info we might have for the
user authenticating. This works fine if the ICAP Client will only deal with
one squid and it's auth scheme. As soon as we have x squids authenticating
to various sources but only one icap client we need to add some additional
information for the client to find the correct auth source. So we need to
tell the ICAP client the used auth (LDAP,WINNT etc) and where the target
(hostname:port) is. From there the client should use all infos received to
build it's internal request.
How to actually do this I am not sure about. Will think about it a little
more and get back to you.
Have a good day and week.
Axel
Am 09.02.2007 21:55 Uhr schrieb Jeremy Hall unter
[EMAIL PROTECTED]:
Hello,
I can't remember. What was the decided path for what was once the
icap_auth_scheme? I recall there was some concern about my suggestion of
having the ability to use ldap://hostname/cn=%u,dc=%d,dc=name,dc=int
but I don't remember what the outcome was.
_J
Squid 3 HEAD - ICAP Modifications
Hi everyone, Second try this time hopefully complete. This is again patched against Squid 3 HEAD and includes 4 changes I would like to have when working with webwasher/squid systems. A.) ICAPServiceRep::TheSessionFailureLimit set through squid.conf B.) ICAPServiceRep delay for a down service set through squid.conf C.) Instead of hardcoding the Header used to transfer the username being able to set the used one through squid.conf D.) When using X-Authenticated-User in C I need the username to be base64 encoded so I added another option to turn on encoding if needed. Regards Axel Westerhold squid_icap.diff Description: Binary data
Re: Modifications to the ICAP client I would like to see
I Forgot to mention this is against Squid3 HEAD. Sorry, Axel Am 03.12.2006 15:19 Uhr schrieb Axel Westerhold unter [EMAIL PROTECTED]: Hi everyone, attached you'll find 3 modifications against HEAD I would like to have: A.) ICAPServiceRep::TheSessionFailureLimit set through squid.conf B.) ICAPServiceRep delay for a down service set through squid.conf C.) Instead of hardcoding the Header used to transfer the username being able to set the used one through squid.conf I tested the patch the best I could. The defaults are as hardcoded before. A simple web poligraph test did not show any problems. Regards, Axel Westerhold
