RE: squid3 comments
I know this is a minor problem, but I had problems getting the squid3 bootstrap.sh script to run, so I couldn't test the patch. I'm pretty sure it's got something to do with the version of automake and autoconf on my system, but I couldn't find a reference to which versions I needed for squid3. I really think that if squid3 needs a specific version of these programs for the bootstrap.sh script to run, then the script should check for the required version. From what I can tell the current script looks for a range of versions, most of which will not work. Can anyone tell me which versions of automake / autoconf are required for squid3? Steven Okay, I had several bootstrap problems myself due to those apps. Unfortunately it only shows you the versions (both expected and found) _after_ it has located all the needed apps. If you can't see any details about the version it is looking for and the one it found. Then you do not have one of the apps installed. Ah, here we are. Found my notes. (Please note that I am Debian based, the package names for your distro may be different). Working from a clean copy of the branch and getting bootstrap errors. I managed to get it going by installing: autoconf automake libtool to get the bootstrap running properly and displaying all action messages and then libltdl3-dev for a cryptic undefined macro error with no helpful references to say what was missing. This problem only appears to occur in the latest versions of libtool which displays as configure.in:34: error: possibly undefined macro: AC_LTDL_DLLIB If this token and others are legitimate, please use m4_pattern_allow. See the Autoconf documentation. I am currently using: automake 1.10 autoconf 2.61 libtool 1.4.3 The bootstrap says it looks for these: WARNING: Cannot find automake version 1.9 1.7 1.6 1.5 Trying automake (GNU automake) 1.10 WARNING: Cannot find autoconf version 2.59 2.57 2.53 2.52 Trying autoconf (GNU Autoconf) 2.61 Amos
RE: squid3 comments
fre 2007-03-02 klockan 10:02 +0900 skrev Steven Wilton: I know this is a minor problem, but I had problems getting the squid3 bootstrap.sh script to run, so I couldn't test the patch. I'm pretty sure it's got something to do with the version of automake and autoconf on my system, but I couldn't find a reference to which versions I needed for squid3. Anything reasonably recent (released in the last 3 years) should work.. automake autoconf libtool However, many multiversion installations of the tools is a bit confused and aclocal does not find the libtool files.. I don't know how it's supposed to work, but there is a very simple workaround making everything work sanely add the path /usr/local/share/aclocal to the dirlist of the version specific directory /usr/local/share/aclocalversion/dirlist There is a open bug report on this to have it fixed in the aclocal, but who knows when the autotool authors will figure out how this is supposed to work... If you find versions in the bootstrap.sh script not working, please post the version and error seen. Regards Henrik signature.asc Description: Detta är en digitalt signerad meddelandedel
RE: squid3 comments
lör 2007-03-03 klockan 03:29 +1300 skrev [EMAIL PROTECTED]: configure.in:34: error: possibly undefined macro: AC_LTDL_DLLIB If this token and others are legitimate, please use m4_pattern_allow. See the Autoconf documentation. Looks like aclocal not finding the libtool macros (LTDL is the libtool macro signature..).. see previous message... Regards Henrik signature.asc Description: Detta är en digitalt signerad meddelandedel
RE: squid3 comments
fre 2007-03-02 klockan 10:02 +0900 skrev Steven Wilton: I know this is a minor problem, but I had problems getting the squid3 bootstrap.sh script to run, so I couldn't test the patch. I'm pretty sure it's got something to do with the version of automake and autoconf on my system, but I couldn't find a reference to which versions I needed for squid3. Anything reasonably recent (released in the last 3 years) should work.. automake autoconf libtool However, many multiversion installations of the tools is a bit confused and aclocal does not find the libtool files.. I don't know how it's supposed to work, but there is a very simple workaround making everything work sanely add the path /usr/local/share/aclocal to the dirlist of the version specific directory /usr/local/share/aclocalversion/dirlist There is a open bug report on this to have it fixed in the aclocal, but who knows when the autotool authors will figure out how this is supposed to work... If you find versions in the bootstrap.sh script not working, please post the version and error seen. When I try bootstrapping squid3 at the moment, I get the following output: WARNING: Cannot find automake version 1.9 1.7 1.6 1.5 Trying automake (GNU automake) 1.10 WARNING: Cannot find autoconf version 2.59 2.57 2.53 2.52 Trying autoconf (GNU Autoconf) 2.61 automake : autoconfg: libtool : Bootstrapping helpers/basic_auth/Makefile.am:6: required directory helpers/basic_auth/POP3 does not exist configure.in:3297: required file `helpers/basic_auth/POP3/Makefile.in' not found automake failed Autotool bootstrapping failed. You will need to investigate and correct before you can develop on this source tree It looks like removing the references to POP3 from Makefile.am and configure.in fixes the problem. (I did remove all other versions of automake from my system before running this, which could have been my problem before). Am I missing the POP3 directory from my tree, or does a patch need to be committed removing the references to the POP3 directory from these 2 files? Steven
Re: squid3 comments
Hi Steven, [EMAIL PROTECTED] wrote: Bootstrapping helpers/basic_auth/Makefile.am:6: required directory helpers/basic_auth/POP3 does not exist configure.in:3297: required file `helpers/basic_auth/POP3/Makefile.in' not found automake failed Autotool bootstrapping failed. You will need to investigate and correct before you can develop on this source tree Just try cvs update with -d argument which will create and download mising directories: # cvs update -d Reagards, Christos
RE: squid3 comments
lör 2007-03-03 klockan 08:39 +0800 skrev [EMAIL PROTECTED]: helpers/basic_auth/Makefile.am:6: required directory helpers/basic_auth/POP3 does not exist cvs update -d -P Regards Henrik signature.asc Description: Detta är en digitalt signerad meddelandedel
Re: squid3 comments
ons 2007-02-28 klockan 09:19 -0500 skrev Jeremy Hall:
Is squid3 faster or slower than squid2?
From my incomplete tests and experience:
Squid-3 is noticeably slower than Squid-2.6, but not by a huge amount.
Squid-3 is faster than Squid-2.5 in some workloads involving many
concurrent connections, but this only thanks to the epoll/kqueue support
which makes the kernel waste less time looking for ready
filedescriptors. Same for Squid-2.6 compared to 2.5. Neigher Squid-3 or
Squid-2.6 has received any code optimizations compared to 2.5.
Squid-3 source code size is about 30% larger than 2.6 (number of lines
of actual code, after eleminating blanks and {} lines but keeping
comments, ingoring helpers and cppunit).
In raw processing power of the code the ranking is (high request
response rate, but very few concurrent connections)
1. Squid-2.5
2. Squid-2.6
3. Squid-3
In amount of requests/s in larger scale with many concurrent connections
1. Squid-2.6
2. Squid-3
3. Squid-2.5 (if it survives at all)
Squid-2 with the small optimizations done by Adrian ranks top in both..
In terms of known bugs we have
Squid-2.6 (a handful)
Squid-2(about the same as 2.6)
Squid-2.5 (a little more than 2.6, no longer maintained)
[big gap]
Squid-3(quite many...)
Unknown bugs quite likely looks about the same.
Active developers is a bit too few in both code bases.
Personally I find the Squid-3 code base a bit alien in many areas, not
at all easier to follow than Squid-2 and yet sufficient different in
many areas to get quite lost.. Suffering a bit from converting to C++
without first getting a clear view of the underlying code data
structure..
Some parts is much better structured than Squid-2 however.
The biggest negative on Squid-3 is time.. it's been in slow development
for very long, and almost constantly (at least for the last 3+ years)
with core issues preventing serious testing. I am quite scared of how
many bugs will pop up their ugly heads once the code is sufficiently
stable to allow some serious testing.
But hopefully the forward porting of bug fixes from Squid-2 has helped.
At least it won't have most of the bugs fixed in Squid-2 since the two
code bases forked 4.5 years ago.. Many many thanks to Guido which has
done a great job in helping with the forward porting of squid-2 patches.
Without this Squid-3 would surely be a dead end dragging not only it's
own bugs but also many of the Squid-2 bugs from the last 4 years or so..
Regards
Henrik
signature.asc
Description: Detta är en digitalt signerad meddelandedel
Re: squid3 comments
Hi Henrik, At 17.31 01/03/2007, Henrik Nordstrom wrote: But hopefully the forward porting of bug fixes from Squid-2 has helped. At least it won't have most of the bugs fixed in Squid-2 since the two code bases forked 4.5 years ago.. Many many thanks to Guido which has done a great job in helping with the forward porting of squid-2 patches. Without this Squid-3 would surely be a dead end dragging not only it's own bugs but also many of the Squid-2 bugs from the last 4 years or so.. I'm very honored for this, but the results of my work on Squid 3 was very far from my expectations: my C++ knowledge is horribly low, and there was many bugs that I cannot resolve myself for this reason. I have always hoped than some other developer with a more strong C++ knowledge will try to fix them, but this was never happened: all Squid 3 peoples seems to like more the development of new features instead of fixing bugs. As example: I have arranged the TPROXY forward port done by Steven in November, but no one of the Squid 3 supporters seems to have tested it giving any kind of feedback, a little disappointing ... :-( So, sometimes I ask to me why I'm still wasting my time on Squid 3 ?. Regards Guido - Guido Serassio Acme Consulting S.r.l. - Microsoft Certified Partner Via Lucia Savarino, 1 10098 - Rivoli (TO) - ITALY Tel. : +39.011.9530135 Fax. : +39.011.9781115 Email: [EMAIL PROTECTED] WWW: http://www.acmeconsulting.it/
RE: squid3 comments
-Original Message- From: Guido Serassio [mailto:[EMAIL PROTECTED] Sent: Friday, 2 March 2007 5:51 AM To: Henrik Nordstrom; Jeremy Hall Cc: Squid Developers Subject: Re: squid3 comments I have always hoped than some other developer with a more strong C++ knowledge will try to fix them, but this was never happened: all Squid 3 peoples seems to like more the development of new features instead of fixing bugs. As example: I have arranged the TPROXY forward port done by Steven in November, but no one of the Squid 3 supporters seems to have tested it giving any kind of feedback, a little disappointing ... :-( So, sometimes I ask to me why I'm still wasting my time on Squid 3 ?. I know this is a minor problem, but I had problems getting the squid3 bootstrap.sh script to run, so I couldn't test the patch. I'm pretty sure it's got something to do with the version of automake and autoconf on my system, but I couldn't find a reference to which versions I needed for squid3. I really think that if squid3 needs a specific version of these programs for the bootstrap.sh script to run, then the script should check for the required version. From what I can tell the current script looks for a range of versions, most of which will not work. Can anyone tell me which versions of automake / autoconf are required for squid3? Steven -- No virus found in this outgoing message. Checked by AVG Free Edition. Version: 7.5.446 / Virus Database: 268.18.5/707 - Release Date: 1/03/2007 2:43 PM
RE: squid3 comments
-Original Message- From: Guido Serassio [mailto:[EMAIL PROTECTED] Sent: Friday, 2 March 2007 5:51 AM To: Henrik Nordstrom; Jeremy Hall Cc: Squid Developers Subject: Re: squid3 comments I have always hoped than some other developer with a more strong C++ knowledge will try to fix them, but this was never happened: all Squid 3 peoples seems to like more the development of new features instead of fixing bugs. As example: I have arranged the TPROXY forward port done by Steven in November, but no one of the Squid 3 supporters seems to have tested it giving any kind of feedback, a little disappointing ... :-( So, sometimes I ask to me why I'm still wasting my time on Squid 3 ?. I know this is a minor problem, but I had problems getting the squid3 bootstrap.sh script to run, so I couldn't test the patch. I'm pretty sure it's got something to do with the version of automake and autoconf on my system, but I couldn't find a reference to which versions I needed for squid3. I really think that if squid3 needs a specific version of these programs for the bootstrap.sh script to run, then the script should check for the required version. From what I can tell the current script looks for a range of versions, most of which will not work. Can anyone tell me which versions of automake / autoconf are required for squid3? Steven Okay, I had several bootstrap problems myself due to those apps. Unfortunately it only shows you the versions (both expected and found) _after_ it has located all the needed apps. If you can't see any details about the version it is looking for and the one it found. Then you do not have one of the apps installed. I do not presently have immediate access to my squid test machine so I can't give any more details for now. Give it a closer look and bets of luck. Amos
Re: squid3 comments
Hi Adrian, Thanks for your answers. As I can understand you are talking about the squid4 project. If you want an independent opinion, I believe that it is not good idea to start striping the squid2. You will get again the same mistakes done at squid3. For a such project start from ground and then thieve code (and most important knowledge) from squid2 and squid3 (and other projects too :-) ) to make what you wand. Squid has a lot of parts implemented like modules (acl's, authentication, delay pools etc) which can modified to be loadable at runtime. A modular system needs different configuration system, different net-IO etc. You want to support multi-threading but squid2 does not care about it and you have to correct a lot of thinks to support it. What parts can you keep from a striped squid2 version? For such work, I think the best is to start from the base server (com_* stack?), then add the configuration system then the http/ftp protocol then acls as modules, etc... If the base system is quite well designed then the others developers will be able to convert parts they made for squid2 or squid3, or just add new code as modules without having to known the overall squid structure/code... Regards, Christos Adrian Chadd wrote: Now, I don't have icap on my list as a specific thing to support, but: * I'd like to look at whats been done in the icap-2.6 patchset and Squid-3 to plan the next evolution of the client, store and server side codebases to neatly support ICAP as a module, rather than a patch or a compile-time option with lots of #defines everywhere; * But what I'd like to do is support all the modern architecture features well - lots of CPUs - fast or slow; lots of RAM or as little RAM as exists on an embedded board; support the modern disk IO patterns much more efficiently than we do at the moment, etc. This requires some pretty drastic internal changes - threading, at the outside. Maybe multi-process if people can think of a portable way of implementing it. * I'd like to make the code as modular as possible so a lot of things are simply loadable at runtime. Don't need the URL rewriter? Don't load the module, no performance impact. * But to do all of this we need to strip Squid all the way back to its core bits, build fast, flexible code libraries and APIs which support all the stuff we want to do - including, yes, icap. Its just too hard for me to do all of the above with the Squid codebase dragging as much history as it does.
Re: squid3 comments
Is squid3 faster or slower than squid2? _J Alex Rousskov [EMAIL PROTECTED] 02/27/07 5:04 PM On Tue, 2007-02-27 at 13:27 +0200, Tsantilas Christos wrote: In the other hand I need a proxy with an icap client because I spent time (and continue spending) to an icap related project. Squid3 has a good icap client. The first problem someones can see in squid3 is that there are some parts derived from c-code, which are not (fully) converted to real c++ code. The second is a feeling that some parts of code are half-completed. I am thinking that maybe it is good practice for someone to start fixing this things first I agree that many Squid3 parts should be fixed, polished, or thrown away. However, I think that we should focus on making Squid3 stable first, and the performance/polishing work you are discussing should be done for v3.1. There are plenty of users who can use Squid3 that is stable but not very fast. An alternate for me is to try fix the bugs and rewrite some of the icap-related parts of the squid26-icap branch. I don't know This would be a bad idea from my biased point of view. While the code migration to Squid3 was poorly done, we are already at the point where we can make Squid3 work for your purposes instead of going back. Please do not forget that Squid2 has its own problems; it is not like you will be migrating to a great code that just needs a yet another ICAP patch! Alex.
Re: squid3 comments
On Wed, 2007-02-28 at 09:19 -0500, Jeremy Hall wrote: Is squid3 faster or slower than squid2? I will be doing performance tests with Squid3 shortly. Will post. Alex.
Re: squid3 comments
Hi Alex,
Alex Rousskov wrote:
I agree that many Squid3 parts should be fixed, polished, or thrown
away. However, I think that we should focus on making Squid3 stable
first, and the performance/polishing work you are discussing should be
done for v3.1.
I am not talking about big changes. I am talking about small changes
which can be done while reading the code.
As an example of such changes I am attaching the rewritten
parseHttpRequest, prepareTransparentURL and prepareAcceleratedURL
A second example: again In parseHttpRequest we have the HttpParser
struct which we are using it to parse the first line of request. The
HttpRequest::parseFirstLine method (of HttpMsg derived HttpRequest
class) does more or less the same. Maybe they can merged.
A third example: in HttpStateData::processReplyHeader int http.cc file.
I am seeing the line:
const bool parsed = newrep-parse(readBuf, eof, error);
and some lines after I am seeing:
header_bytes_read = headersEnd(readBuf-content(),
readBuf-contentSize());
What the hell parsing we did in previous line if we did not keep the end
of headers? The easier we can do is to make HttpReply::parse to return
the size of headers on success or zero else. Or just keep in an
internall variable of HttpReply class the size of headers and then get
them with a call like newrep-headersSize()
I think such changes are small and can make squid3 to be a little faster
and can simplify in some cases the code, which will help the debuging.
I am not having a lot of time too but I can find some hours here or
there to make such changes, if needed.
Alex Rousskov wrote:
. There are plenty of users who can use Squid3 that is
stable but not very fast.
This is true, in most setups squid is not needed to be (very) fast. But
in the other hand I am seeing that sometimes it is very difficult to
explain to some people that they do not need a server with 2 or more
fast cpus, expensive hardware raids and fast scsi disks, to make a
file-server for only 10-20 clients
Regards,
Christos
int
internalCheckn(const char *urlpath,int size)
{
if(size16)
return 0;
return (0 == strncmp(urlpath, /squid-internal-, 16));
}
char *memstr(char *s,char *pattern,int s_len){
int i,k;
int p_len=strlen(pattern);
for(i=0;i=s_len-p_len;i++){
k=0;
while(kp_len){
if(s[i+k] != pattern[k])
break;
k++;
}
if(k==p_len)
return s+i;
}
return NULL;
}
static void
prepareAcceleratedURL(ConnStateData::Pointer conn, ClientHttpRequest *http,
char *url, int url_size, const char *req_hdr)
{
int vhost = conn-port-vhost;
int vport = conn-port-vport;
int uri_sz = 0, tmpsz = 0;
char *host,*s;
http-flags.accel = 1;
/* BUG: Squid cannot deal with '*' URLs (RFC2616 5.1.2) */
if (url_size = 15 strncasecmp(url, cache_object://, 15) == 0)
return; /* already in good shape */
if (*url != '/') {
if (conn-port-vhost)
return; /* already in good shape */
/* else we need to ignore the host name */
s = NULL;
s = memstr(url, //, u_size);
if(s)
u_size -= ((s-url) + 2);
url = s + 2;
#if SHOULD_REJECT_UNKNOWN_URLS
if (!url)
return parseHttpRequestAbort(conn, error:invalid-request);
#endif
if (url){
s=memchr(url,'/',u_size);
if(s)
u_size -= s-url;
url=s;
}
if (!url){
url = (char *) /;
url_size = 1;
}
}
if (internalCheckn(url,url_size)) {
/* prepend our name port */
s = (char *)xcalloc(url_size+1,1);
memcpy(s,url,url_size);
s[url_size]='\0';
http-uri = xstrdup(internalLocalUri(NULL, s));
xfree(s);
return;
}
if (vhost (host = mime_get_header(req_hdr, Host)) != NULL) {
uri_sz = url_size + 32 + Config.appendDomainLen +
strlen(host);
http-uri = (char *)xcalloc(uri_sz, 1);
tmpsz = snprintf(http-uri, uri_sz, %s://%s,
conn-port-protocol, host);
} else if (conn-port-defaultsite) {
uri_sz = url_size + 32 + Config.appendDomainLen +
strlen(conn-port-defaultsite);
http-uri = (char *)xcalloc(uri_sz, 1);
tmpsz = snprintf(http-uri, uri_sz, %s://%s,
conn-port-protocol, conn-port-defaultsite);
} else if (vport == -1) {
/* Put the local socket IP address as the hostname. */
uri_sz = url_size + 32 + Config.appendDomainLen;
http-uri = (char *)xcalloc(uri_sz, 1);
tmpsz = snprintf(http-uri, uri_sz, %s://%s:%d,
http-getConn()-port-protocol,
inet_ntoa(http-getConn()-me.sin_addr),
ntohs(http-getConn()-me.sin_port));
} else if (vport 0) {
/* Put the local socket IP address as the
Re: squid3 comments
Tsantilas Christos wrote:
As an example of such changes I am attaching the rewritten
parseHttpRequest, prepareTransparentURL and prepareAcceleratedURL
Sorry to all
the code I send in previous mail will not compile. I am sending it
again. Needs some more testing to be sure that it is OK
int
internalCheckn(const char *urlpath,int size)
{
if(size16)
return 0;
return (0 == strncmp(urlpath, /squid-internal-, 16));
}
char *memstr(char *s,const char *pattern,int s_len){
int i,k;
int p_len=strlen(pattern);
for(i=0;i=s_len-p_len;i++){
k=0;
while(kp_len){
if(s[i+k] != pattern[k])
break;
k++;
}
if(k==p_len)
return s+i;
}
return NULL;
}
static void
prepareAcceleratedURL(ConnStateData::Pointer conn, ClientHttpRequest *http,
char *url, int url_size, const char *req_hdr)
{
int vhost = conn-port-vhost;
int vport = conn-port-vport;
int uri_sz = 0, tmpsz = 0;
char *host,*s;
http-flags.accel = 1;
/* BUG: Squid cannot deal with '*' URLs (RFC2616 5.1.2) */
if (url_size = 15 strncasecmp(url, cache_object://, 15) == 0)
return; /* already in good shape */
if (*url != '/') {
if (conn-port-vhost)
return; /* already in good shape */
/* else we need to ignore the host name */
s = NULL;
s = memstr(url, //, url_size);
if(s)
url_size -= ((s-url) + 2);
url = s + 2;
#if SHOULD_REJECT_UNKNOWN_URLS
if (!url)
return parseHttpRequestAbort(conn, error:invalid-request);
#endif
if (url){
s=(char *)memchr(url,'/',url_size);
if(s)
url_size -= s-url;
url=s;
}
if (!url){
url = (char *) /;
url_size = 1;
}
}
if (internalCheckn(url,url_size)) {
/* prepend our name port */
s = (char *)xcalloc(url_size+1,1);
memcpy(s,url,url_size);
s[url_size]='\0';
http-uri = xstrdup(internalLocalUri(NULL, s));
xfree(s);
return;
}
if (vhost (host = mime_get_header(req_hdr, Host)) != NULL) {
uri_sz = url_size + 32 + Config.appendDomainLen +
strlen(host);
http-uri = (char *)xcalloc(uri_sz, 1);
tmpsz = snprintf(http-uri, uri_sz, %s://%s,
conn-port-protocol, host);
} else if (conn-port-defaultsite) {
uri_sz = url_size + 32 + Config.appendDomainLen +
strlen(conn-port-defaultsite);
http-uri = (char *)xcalloc(uri_sz, 1);
tmpsz = snprintf(http-uri, uri_sz, %s://%s,
conn-port-protocol, conn-port-defaultsite);
} else if (vport == -1) {
/* Put the local socket IP address as the hostname. */
uri_sz = url_size + 32 + Config.appendDomainLen;
http-uri = (char *)xcalloc(uri_sz, 1);
tmpsz = snprintf(http-uri, uri_sz, %s://%s:%d,
http-getConn()-port-protocol,
inet_ntoa(http-getConn()-me.sin_addr),
ntohs(http-getConn()-me.sin_port));
} else if (vport 0) {
/* Put the local socket IP address as the hostname, but static port */
uri_sz = url_size + 32 + Config.appendDomainLen;
http-uri = (char *)xcalloc(uri_sz, 1);
tmpsz = snprintf(http-uri, uri_sz, %s://%s:%d,
http-getConn()-port-protocol,
inet_ntoa(http-getConn()-me.sin_addr),
vport);
}
else
return;
/*OK Append the url at the end of uri and we are OK*/
uri_sz -= tmpsz;
url_size = XMIN(uri_sz - 1, url_size);
memcpy(http-uri+tmpsz, url , url_size);
http-uri[tmpsz+url_size] = '\0';
debug(33, 5) (ACCEL VHOST REWRITE: '%s'\n, http-uri);
}
static void
prepareTransparentURL(ConnStateData::Pointer conn, ClientHttpRequest *http,
char *url, int url_size, const char *req_hdr)
{
char *host;
int uri_sz = 0, tmpsz = 0;
http-flags.transparent = 1;
if (*url != '/')
return; /* already in good shape */
/* BUG: Squid cannot deal with '*' URLs (RFC2616 5.1.2) */
if ((host = mime_get_header(req_hdr, Host)) != NULL) {
uri_sz = url_size + 32 + Config.appendDomainLen +
strlen(host);
http-uri = (char *)xcalloc(uri_sz, 1);
tmpsz = snprintf(http-uri, uri_sz, %s://%s,
conn-port-protocol, host);
} else {
/* Put the local socket IP address as the hostname. */
uri_sz = url_size + 32 + Config.appendDomainLen;
http-uri = (char *)xcalloc(uri_sz, 1);
tmpsz = snprintf(http-uri, uri_sz, %s://%s:%d,
http-getConn()-port-protocol,
inet_ntoa(http-getConn()-me.sin_addr),
ntohs(http-getConn()-me.sin_port));
}
uri_sz -= tmpsz;
url_size = XMIN(uri_sz - 1,
Re: squid3 comments
On Wed, 2007-02-28 at 20:48 +0200, Tsantilas Christos wrote: As an example of such changes I am attaching the rewritten parseHttpRequest, prepareTransparentURL and prepareAcceleratedURL A second example: again In parseHttpRequest we have the HttpParser struct which we are using it to parse the first line of request. The HttpRequest::parseFirstLine method (of HttpMsg derived HttpRequest class) does more or less the same. Maybe they can merged. A third example: in HttpStateData::processReplyHeader int http.cc file. I am seeing the line: const bool parsed = newrep-parse(readBuf, eof, error); and some lines after I am seeing: header_bytes_read = headersEnd(readBuf-content(), readBuf-contentSize()); What the hell parsing we did in previous line if we did not keep the end of headers? The easier we can do is to make HttpReply::parse to return the size of headers on success or zero else. Or just keep in an internall variable of HttpReply class the size of headers and then get them with a call like newrep-headersSize() I think such changes are small and can make squid3 to be a little faster and can simplify in some cases the code, which will help the debuging. I am not having a lot of time too but I can find some hours here or there to make such changes, if needed. If you are absolutely, positively, 100% sure that these changes are correct then it is only a question whether they will help us make Squid3 stable faster (by fixing bugs, improving the code, debugging flow, or whatever). If they will, we should implement them. My concern is that in many cases, an innocent-looking parsing function has a couple of well-hidden side effects. Removing a function call or changing calls order introduces subtle bugs. I have been bitten by this many times. I think we should start from stating the goal of these changes in Squid 3.0. If they are for performance improvement, I would suggest waiting until v3.1 or until performance tests indicate that we must improve Squid 3 performance before calling it stable. If the goal is to fix a common-case bug, we should make the necessary changes, of course. If the ultimate goal is different, let's discuss it. Thank you, Alex. P.S. I will try to look at your specific changes soon.
Re: squid3 comments
On Wed, 2007-02-28 at 17:54 -0700, Alex Rousskov wrote: I think we should start from stating the goal of these changes in Squid 3.0. If they are for performance improvement, I would suggest waiting until v3.1 or until performance tests indicate that we must improve Squid 3 performance before calling it stable. If the goal is to fix a common-case bug, we should make the necessary changes, of course. If the ultimate goal is different, let's discuss it. I agree. -Rob -- GPG key available at: http://www.robertcollins.net/keys.txt. signature.asc Description: This is a digitally signed message part
Re: squid3 comments
On Thu, Mar 01, 2007, Robert Collins wrote: On Wed, 2007-02-28 at 17:54 -0700, Alex Rousskov wrote: I think we should start from stating the goal of these changes in Squid 3.0. If they are for performance improvement, I would suggest waiting until v3.1 or until performance tests indicate that we must improve Squid 3 performance before calling it stable. If the goal is to fix a common-case bug, we should make the necessary changes, of course. If the ultimate goal is different, let's discuss it. I agree. So do I. If you want Squid-3 to go ahead in production then please, please, please, concentrate on fixing up all the bugs that are in Bugzilla without huge code restructuring and get the stable release out the door. Adrian
Re: squid3 comments
On Tue, Feb 27, 2007, Tsantilas Christos wrote: Hi all, I was looking in squid3 code last days. I read again Adrian's mails in which he complained about squid3 speed, and cpu usage. Looking in the code there are a number of code-pieces which can improved. An example is the call of headersEnd (Francisco Gimeno note this problem too for squid26) which computed again and again in squid3. Look in HttpStateData::processReplyHeader (http.cc file) method, headerEnd called inside httpMsg::parse call: const bool parsed = newrep-parse(readBuf, eof, error); and then called again some lines after: header_bytes_read = headersEnd(readBuf-content(), readBuf-contentSize()); These days the http headers can easily be 2k, 4k or more (big urls, cookies etc) so such calls are really costs. I also rewrite the parseHttpRequest (client_side.cc file), to not xmalloc space for url parsing and I modified prepareTransparentURL and prepareAcceleratedURL functions too to take an extra argument the url length and to not require url as a null terminated string. OK it was not difficult but I think that the problem is not only writing something faster here. For example the struct HttpParser it is better to be a c++ class (maybe HttpMsg derived?) and some functions be implemented as methods. Do you think that such changes in squid3 code make sense? They make sense but you'd be repeating the work I've done in the squid-2-HEAD branch. Whats really needed is some more thought about the structure of the code. Adding lengths to character buffers is a good thing to save on strlen() calls but they really should be using some referenced buffer setup. Heck, in C++ it should at the -outside- be using String as an intermediate (since you can then later on present an implementation of the String API sitting on top of said reference counted buffer data type..) Want to know what else is silly in the client-side code when you go digging? Here's a snippet just from handling the incoming request: * The order that things are parsed. We call headersEnd() to make sure we've got an entire request, then build clientHttpRequest, then parse the URL to build a request_t, then fill out the headers. This, to be honest, is horrible. * Then we take the URL that we've already delineated the start/end of and we parse it again to delineate the beginning/end of each chunk - we already have method so thats ok, but protocol, user/password, host, port, uripath. We've already parsed it once; we should have recorded what we saw as we did it. * Then we copy all the headers into seperately allocated strings, when the buffer already has the content. We should just reference-count the buffer and build a string type that can reference an extent of a buffer. This has been done more than a thousand times in the last 20 years; its nothing new. What about building the reply, once we've negotiated the myriad of the store manager? * at the moment we copy the HttpReply from the server side (which we used to copy and then parse when we'd already have a parsed copy in MemObject), then go through the headers and extract out all the information we need. Then we walk the list again IIRC and pull out headers that shouldn't be there. Its a pretty big mess. * Once we've got a reply we pack it all up in a membuf (another whole load of copying) and schedule that to write. At least the code used to pack whatever data was left over from header parsing and stuff that into the write too so objects that fit entirely in the 4k copy buffer were written with one write() but still, all that copying is done rather than using writev(). There's also a -lot- of code involved in the data exchange path (store client, appending) between client and server but thats a whole other story. We could spend a few years slowly picking away optimising a few percent here and there or .. we could not. :) Help me sort out the basic store api changes in the storework branch so I can move onto the next area that really needs all the rework - the client side request/reply code. Yes, I'd like to do all of what you've suggested above but I'm going to do it by junking most of the client-side request/reply handling routines and replacing them with stuff written from the ground up to use a sane API. Buffers won't be copied; stuff will be parsed once and parsed quickly; the whole of the URL rewrite/XFF/ Acceleration/ACL lookups/etc will be implemented as a state engine with events rather than callback-driven like it is today. Its also doable in a -lot- less code than is traversed today. I believe the only way we're going to get long-term benefits out of any improvement work is to pick some medium term goals (mostly structured around knowing what we'd like the code to look and picking incremental API and code decisions which take us down that path) and working on them piece by piece. So c'mon, help me out. :) I have a big list of things that we
Re: squid3 comments
Tsantilas Christos [EMAIL PROTECTED] 02/27/07 6:27 AM Hi Adrian, Adrian Chadd wrote: agree . Yes, I'd like to do all of what you've suggested above but I'm going to do it by junking most of the client-side request/reply handling routines and replacing them with stuff written from the ground up to use a sane API. Buffers won't be copied; stuff will be parsed once and parsed quickly; the whole of the URL rewrite/XFF/ Acceleration/ACL lookups/etc will be implemented as a state engine with events rather than callback-driven like it is today. Its also doable in a -lot- less code than is traversed today. I believe the only way we're going to get long-term benefits out of any improvement work is to pick some medium term goals (mostly structured around knowing what we'd like the code to look and picking incremental API and code decisions which take us down that path) and working on them piece by piece. Ok Adrian, I am watching the mailing list and I know what you want to do. I believe too that some parts of squid needs redesign, if the project wants to survive. Squid is an old and huge project. And you must continue your work because you want a fast http proxy, with cleaner code and better api's . In the other hand I need a proxy with an icap client because I spent time (and continue spending) to an icap related project. Squid3 has a good icap client. The first problem someones can see in squid3 is that there are some parts derived from c-code, which are not (fully) converted to real c++ code. The second is a feeling that some parts of code are half-completed. I am thinking that maybe it is good practice for someone to start fixing this things first An alternate for me is to try fix the bugs and rewrite some of the icap-related parts of the squid26-icap branch. I don't know Let me second this. When you start asking questions about squid3 and its stability, you get anything from it's stable to not for prime time and when you ask questions about using it in a production environment, most shy away from that. but I need ICAP support and proposals like this, although valid, scare me a little. (rewriting the client side code from scratch, not putting icap in 2.6) I don't have the time to go digging on 2.6 to make icap work better, so if you have some spare cycles it would be greatly appreciated. _J Regards, Christos
Re: squid3 comments
On Tue, Feb 27, 2007, Tsantilas Christos wrote: Ok Adrian, I am watching the mailing list and I know what you want to do. I believe too that some parts of squid needs redesign, if the project wants to survive. Squid is an old and huge project. And you must continue your work because you want a fast http proxy, with cleaner code and better api's . Yup. Well, I want more than a fast http proxy. I want a faster, simpler, better structured proxy thats flexible enough to be a testbed for all kinds of new stuff. In the other hand I need a proxy with an icap client because I spent time (and continue spending) to an icap related project. Squid3 has a good icap client. The first problem someones can see in squid3 is that there are some parts derived from c-code, which are not (fully) converted to real c++ code. The second is a feeling that some parts of code are half-completed. I am thinking that maybe it is good practice for someone to start fixing this things first An alternate for me is to try fix the bugs and rewrite some of the icap-related parts of the squid26-icap branch. I don't know I'm going to be perfectly honest, and this is just my opinion. I think Squid-3 is ultimately a dead end. I think the people involved got a bit overenthusiastic with applying a whole lot of paradigms which, to be honest, just haven't delivered. So the project stalled and I don't think its going to be going anywhere in a hurry. I tried pushing it along myself for a few months but then I realised Squid-3 was just as complicated as Squid-2, with all of the horribleness of Squid-3 inside and compounded by almost-implemented data types and little understanding of the changes made (for example, the RefCounted stuff is a great idea but its not an immediate dropin replacement for cbdata. A lot of core routines were converted over with some pretty disasterous results.) C++ is a good idea and I think in the long run its the kind of direction the codebase has to take - but we shouldn't have converted the Squid codebase to C++ at the stage it was at. We shouldn't have tried to convert stuff over to C++ without first giving the code a good half dozen passthroughs, simplifying what was there and giving everything a good tidyup. Unfortunately that wasn't done - lots of refactoring was done but nothing was ever really fixed. Now, I don't have icap on my list as a specific thing to support, but: * I'd like to look at whats been done in the icap-2.6 patchset and Squid-3 to plan the next evolution of the client, store and server side codebases to neatly support ICAP as a module, rather than a patch or a compile-time option with lots of #defines everywhere; * But what I'd like to do is support all the modern architecture features well - lots of CPUs - fast or slow; lots of RAM or as little RAM as exists on an embedded board; support the modern disk IO patterns much more efficiently than we do at the moment, etc. This requires some pretty drastic internal changes - threading, at the outside. Maybe multi-process if people can think of a portable way of implementing it. * I'd like to make the code as modular as possible so a lot of things are simply loadable at runtime. Don't need the URL rewriter? Don't load the module, no performance impact. * But to do all of this we need to strip Squid all the way back to its core bits, build fast, flexible code libraries and APIs which support all the stuff we want to do - including, yes, icap. Its just too hard for me to do all of the above with the Squid codebase dragging as much history as it does. Now, these are all my opinions and are definitely not to be taken as the opinions of others or the Squid project in general. (and I should be coding.) Adrian
Re: squid3 comments
On Tue, Feb 27, 2007, Jeremy Hall wrote: Let me second this. When you start asking questions about squid3 and its stability, you get anything from it's stable to not for prime time and when you ask questions about using it in a production environment, most shy away from that. * noone's really stepped up to drag Squid-3 up to production quality. The bugs are relatively well-known and the issues with the codebase show up in bugzilla. * People seem to think we can keep adding functionality without fixing the Squid core. Which is a mess, and in my opinion, needs to be fixed first. but I need ICAP support and proposals like this, although valid, scare me a little. (rewriting the client side code from scratch, not putting icap in 2.6) I don't have the time to go digging on 2.6 to make icap work better, so if you have some spare cycles it would be greatly appreciated. I won't make icap better in squid-2.6. I want to fix the squid core so it can better support stuff like ICAP without needing to be a patch against the current codebase. I thought Squid-3 would bring this but it just hasn't. Every time I try to fix or add something interesting to Squid I keep hitting the same brick wall - the code is overly complicated for what it does. We need to spend time fixing the Squid internals and getting all of that fast, flexible and rock stable so stuff like ICAP can be implemented better. Adrian
Re: squid3 comments
Adrian Chadd [EMAIL PROTECTED] 02/27/07 9:25 AM On Tue, Feb 27, 2007, Jeremy Hall wrote: Let me second this. When you start asking questions about squid3 and its stability, you get anything from it's stable to not for prime time and when you ask questions about using it in a production environment, most shy away from that. * noone's really stepped up to drag Squid-3 up to production quality. The bugs are relatively well-known and the issues with the codebase show up in bugzilla. * People seem to think we can keep adding functionality without fixing the Squid core. Which is a mess, and in my opinion, needs to be fixed first. but I need ICAP support and proposals like this, although valid, scare me a little. (rewriting the client side code from scratch, not putting icap in 2.6) I don't have the time to go digging on 2.6 to make icap work better, so if you have some spare cycles it would be greatly appreciated. I won't make icap better in squid-2.6. I want to fix the squid core so it can better support stuff like ICAP without needing to be a patch against the current codebase. I thought Squid-3 would bring this but it just hasn't. I think we are on the same page for end-term goals, but what would you recommend I do for today? Squid-3 is still a moving target. Every time I try to fix or add something interesting to Squid I keep hitting the same brick wall - the code is overly complicated for what it does. yes I agree with that. We need to spend time fixing the Squid internals and getting all of that fast, flexible and rock stable so stuff like ICAP can be implemented better. So are you saying those of us that need icap need to just wait? _J Adrian
Re: squid3 comments
On Tue, Feb 27, 2007, Jeremy Hall wrote: I think we are on the same page for end-term goals, but what would you recommend I do for today? Squid-3 is still a moving target. So are you saying those of us that need icap need to just wait? There's two parts. One: Alex is working on improvements to the ICAP code in Squid-3 which I hope will act as a kind of reference implementation to use in the future. Help him out any way you can. Two: I think the storework branch is one step along the right path to fixing up the codebase in the long term. The two things I need to sort it out is testing of the branch and some simple(!)ish test suite to implement tests of the client-side to make sure stuff isn't regressed as development continues. So some help dreaming up and coding up some test utilities to act as a test suite of the client side, along with some actual testing. No, its nowhere near ready to even sniff production traffic. :) Adrian
Re: squid3 comments
Adrian Chadd [EMAIL PROTECTED] 02/27/07 10:00 AM On Tue, Feb 27, 2007, Jeremy Hall wrote: I think we are on the same page for end-term goals, but what would you recommend I do for today? Squid-3 is still a moving target. So are you saying those of us that need icap need to just wait? There's two parts. One: Alex is working on improvements to the ICAP code in Squid-3 which I hope will act as a kind of reference implementation to use in the future. Help him out any way you can. but if squid-3 is ultimately dead, why would we want to migrate to it? _J
Re: squid3 comments
On Tue, 2007-02-27 at 13:27 +0200, Tsantilas Christos wrote: In the other hand I need a proxy with an icap client because I spent time (and continue spending) to an icap related project. Squid3 has a good icap client. The first problem someones can see in squid3 is that there are some parts derived from c-code, which are not (fully) converted to real c++ code. The second is a feeling that some parts of code are half-completed. I am thinking that maybe it is good practice for someone to start fixing this things first I agree that many Squid3 parts should be fixed, polished, or thrown away. However, I think that we should focus on making Squid3 stable first, and the performance/polishing work you are discussing should be done for v3.1. There are plenty of users who can use Squid3 that is stable but not very fast. An alternate for me is to try fix the bugs and rewrite some of the icap-related parts of the squid26-icap branch. I don't know This would be a bad idea from my biased point of view. While the code migration to Squid3 was poorly done, we are already at the point where we can make Squid3 work for your purposes instead of going back. Please do not forget that Squid2 has its own problems; it is not like you will be migrating to a great code that just needs a yet another ICAP patch! Alex.
Re: squid3 comments
On Tue, 2007-02-27 at 22:25 +0800, Adrian Chadd wrote: On Tue, Feb 27, 2007, Jeremy Hall wrote: Let me second this. When you start asking questions about squid3 and its stability, you get anything from it's stable to not for prime time and when you ask questions about using it in a production environment, most shy away from that. * noone's really stepped up to drag Squid-3 up to production quality. The bugs are relatively well-known and the issues with the codebase show up in bugzilla. I am working on dragging Squid3 to production quality and fixing bugs that are present in my environment. There are other folks doing that as well. Please do not try to persuade people to help you with your Squid2 projects by attacking Squid3. * People seem to think we can keep adding functionality without fixing the Squid core. Which is a mess, and in my opinion, needs to be fixed first. I agree. Personally, I am against adding new features to Squid 3.0. We need to spend time fixing the Squid internals and getting all of that fast, flexible and rock stable so stuff like ICAP can be implemented better. Agreed. I wish you could work on Squid3 internals instead of Squid2, but it is your call and I respect your choice. Let's just not assume that all work is done or should be done on Squid2. Thank you, Alex. P.S. FWIW, ICAP support is already pretty good in Squid3, regardless of the internals (that are getting better as well).
Re: squid3 comments
On Tue, 2007-02-27 at 23:00 +0800, Adrian Chadd wrote: On Tue, Feb 27, 2007, Jeremy Hall wrote: So are you saying those of us that need icap need to just wait? There's two parts. One: Alex is working on improvements to the ICAP code in Squid-3 which I hope will act as a kind of reference implementation to use in the future. Help him out any way you can. Two: I think the storework branch is one step along the right path to fixing up the codebase in the long term. The two things I need to sort it out is testing of the branch and some simple(!)ish test suite to implement tests of the client-side to make sure stuff isn't regressed as development continues. So some help dreaming up and coding up some test utilities to act as a test suite of the client side, along with some actual testing. No, its nowhere near ready to even sniff production traffic. :) The point of my Squid3/ICAP work is to make Squid3 usable in a content adaptation environment. I do not care much if ICAP code becomes a reference implementation or not. There are many parts. Some work on fixing Squid2 bugs, some work on optimizing Squid2, some work on fixing Squid3 bugs, some work on Windows port, etc. If people want to help optimizing Squid2, it is their choice, of course. Perhaps the storework branch will become a reference implementation to speedup Squid3 when the time comes :-). I would prefer that folks spent more energy on Squid 3.0 to make it stable faster (and only then optimize Squid 3.1), but I am not going to call every project that is not helping me dead. Alex.
Re: squid3 comments
Perhaps the storework branch will become a reference implementation to speedup Squid3 when the time comes :-). I would prefer that folks spent more energy on Squid 3.0 to make it stable faster (and only then optimize Squid 3.1), but I am not going to call every project that is not helping me dead. Thats actually what I hope will happen. The -2 and -3 codebases are still similar enough to allow that kind of code migration. Adrian
Re: squid3 comments
On Tue, Feb 27, 2007, Alex Rousskov wrote: On Tue, 2007-02-27 at 22:25 +0800, Adrian Chadd wrote: On Tue, Feb 27, 2007, Jeremy Hall wrote: Let me second this. When you start asking questions about squid3 and its stability, you get anything from it's stable to not for prime time and when you ask questions about using it in a production environment, most shy away from that. * noone's really stepped up to drag Squid-3 up to production quality. The bugs are relatively well-known and the issues with the codebase show up in bugzilla. I am working on dragging Squid3 to production quality and fixing bugs that are present in my environment. There are other folks doing that as well. Please do not try to persuade people to help you with your Squid2 projects by attacking Squid3. * People seem to think we can keep adding functionality without fixing the Squid core. Which is a mess, and in my opinion, needs to be fixed first. I agree. Personally, I am against adding new features to Squid 3.0. We need to spend time fixing the Squid internals and getting all of that fast, flexible and rock stable so stuff like ICAP can be implemented better. Agreed. I wish you could work on Squid3 internals instead of Squid2, but it is your call and I respect your choice. Let's just not assume that all work is done or should be done on Squid2. Thank you, Alex. P.S. FWIW, ICAP support is already pretty good in Squid3, regardless of the internals (that are getting better as well). Hey, I'm sorry if I came across a little more stabby than usual. I'm just fairly upset with the development process and somewhat lack of progress that we've made over the last few years. I think it'll be easier in the short to medium term to get people to run a modified Squid-2 to drop in-place of their existing Squid-2 than to get Squid-3 run up. Part of the trouble I have is finding testers and doing development on Squid-2 as a base fixes that. Please don't stop hacking on Squid-3 because I don't agree with the current codebase. :) What I hope will happen is the development done in storework will eventually spin off another major release branch and keep interest in the project alive. I then hope someone starts cherry-picking from -2 to -3. I don't want to development on -3 until its stabilised as I'm worried that I'm just making it harder to figure out where the problems lie. At least in Squid-2 I can suck over the changes made as people find and fix bugs in Squid-2.6. Besides, I'm about to start asking the really hard questions which'll somewhat determine where things head from now. Stuff like Threaded or multi-process?, What should our backend storage look like?, Callback or event-driven?, How should we abstract out network and disk IO to make Windows/UNIX porting an easier task?, etc. Thats applicable no matter which path we've chosen. :) Adrian
squid3 comments
Hi all, I was looking in squid3 code last days. I read again Adrian's mails in which he complained about squid3 speed, and cpu usage. Looking in the code there are a number of code-pieces which can improved. An example is the call of headersEnd (Francisco Gimeno note this problem too for squid26) which computed again and again in squid3. Look in HttpStateData::processReplyHeader (http.cc file) method, headerEnd called inside httpMsg::parse call: const bool parsed = newrep-parse(readBuf, eof, error); and then called again some lines after: header_bytes_read = headersEnd(readBuf-content(), readBuf-contentSize()); These days the http headers can easily be 2k, 4k or more (big urls, cookies etc) so such calls are really costs. I also rewrite the parseHttpRequest (client_side.cc file), to not xmalloc space for url parsing and I modified prepareTransparentURL and prepareAcceleratedURL functions too to take an extra argument the url length and to not require url as a null terminated string. OK it was not difficult but I think that the problem is not only writing something faster here. For example the struct HttpParser it is better to be a c++ class (maybe HttpMsg derived?) and some functions be implemented as methods. Do you think that such changes in squid3 code make sense? Regards, Christos
