Re: squid-smp: synchronization issue solutions
On Tue, Nov 17, 2009 at 12:45 PM, Alex Rousskov rouss...@measurement-factory.com wrote: On 11/17/2009 04:09 AM, Sachin Malave wrote: snip I AM THINKING ABOUT HYBRID OF BOTH... Somebody might implement process model, Then we would merge both process and thread models .. together we could have a better squid.. :) What do u think? In my limited squid expierence, cpu usage is hardly a bottleneck. So, why not just use smp for the cpu/disk-intensive parts? The candidates I can think of are: * evaluating regular expressions (url_regex acls). * aufs/diskd (squid already has support for this). Best regards, -- Gonzalo A. Arana
Re: SMP scalability goal
On Thu, Apr 24, 2008 at 1:31 PM, Alex Rousskov [EMAIL PROTECTED] wrote: On Thu, 2008-04-24 at 10:27 -0600, Duane Wessels wrote: On Thu, 24 Apr 2008, Alex Rousskov wrote: I do not think it is realistic to expect nearly linear scale (close to 100% and 200% increase in performance), especially for the first implementation. As you know, disk is usually the bottleneck. So I think your goals and expectations should state whether disk caching is involved or not. Assuming disk caching is involved, maybe even state what storage type and a typical hit ratio. Very good point. I should have said that for this particular question let's consider the pure case of no caching (neither disk nor memory). I believe another thing to consider is what if sysadmin configure squid with a very long url_regex acl (and use it in, say, http_access). Perhaps heavy acls should also be excluded from the metric. Regards, -- Gonzalo A. Arana
Re: bug 1208: internal redirect
On Jan 17, 2008 9:15 PM, Amos Jeffries [EMAIL PROTECTED] wrote:
snip
What I have come up with is what I'd term a mapping rather than a redirect
with the syntax:
'map_domain' fqdn1 fqdn2 [ fqdn2 ...] [deny acl [...]]
- anything that dstdomain matches a fqdn2* gets its dstdomain mapped to
the given fqdn1. fqdn2 can be a wildcard just as in ACL.
This can be achieved with my proposed internal redirect scheme:
acl old_domains dstdomain old.domains.here
acl old_domains dstdomain .fancier.domain
redirect http://your.new.domain/%rp old_domains
'map_url' url1 url2 [url2 ...] [deny acl [...]]
- anything that matches a url2* prefix gets that prefix replaced by url1
- acl can be used to block this behaviour for any request
I see. To fit my needs, I require url1 to interpolate tags like %et ('tag
returned from external acl'). But, with map_url scheme, as we would be
replacing prefixes, it would not make sense to interpolate those tags
They should also be available on suffixes (or in the middle).
I understand the need for map_url scheme. I would rather not add
another url-mapping directive, since it would not be clear which directive
get processed first (something like never_direct, always_direct,
cache_peer_access, and so on).
So, we could adapt redirect directive to a syntax like this:
redirect replace http://the.new.url/to?replace=%ue acl1 !acl2 acl3
redirect prefix http://some.prefix/to/replace
http://prefixes.to/be/replaced http://another.prefix/blah
Having a directive with hibrid syntax is not wise neither I guess, but
this is what comes to my mind to accomodate all suggested needs.
On the other hand, this approach:
1) it is quite clear the order of the directive processing.
2) we have the possibility for further extending the redirect
directive with other second level directives
(we would have 'replace' and 'prefix' for the moment).
Amos, does this satisfy your needs/thoughts?
This plays nicely in with redirection and Adrians latest storage
manipulator. As it can be done at the point of input and replace both with
a single mapping.
I plugged it into the very same stage of external redirectos. Should this be
merged with store_rewrite? If that's the case, it would take me longer.
How close it that to your new version of the patch?
Quite close I think (a memory corruption issue, most likely). Stay tuned.
Regards,
--
Gonzalo A. Arana
bug 1208: internal redirect
To update the patch for this feature, I have some comments regarding hno@'s comment #5: 1. URL escaping is already provided by rfc1738_encode(). I'll follow your advise. 2. I would prefer redoing the core access controls to return other entities than allow/deny to make things like this patch considerably cleaner. There is now quite many acl driven directives selecting various things, and all need special code today.. tcp_outgoing_xxx access_log reply_body_max_size always/never_direct could be joined into one if this was available Should follow this for updating 'internal redirect'? Perhaps that's easier with squid3 async framework, and would involve quite many changes on squid2. Let me know your opinion. Regards, -- Gonzalo A. Arana
Re: tproxy testing
Adrian, On Jan 16, 2008 6:42 AM, Henrik Nordström [EMAIL PROTECTED] wrote: tis 2008-01-15 klockan 18:13 +0900 skrev Adrian Chadd: I'm trying to get tproxy working, and I'm stuck trying to build a kernel with both capabilities -and- the latest tproxy-2 stuff that Squid supports. So far I've got a 2.6.20 vanilla kernel which I've patched tproxy-2 into. That works fine, but there's no capabilities support in the vanilla kernel and thus Squid disables tproxy support. Why not tproxy4? Unless I am mistaken, latest linux kernel tproxy patch does not require the use of capabilities. Have a look at: http://www.squid-cache.org/bugs/show_bug.cgi?id=2129 I'll upload an up to date patch to the bug. Regards, -- Gonzalo A. Arana
bugzilla weirdness
Hi, I may be crazy, but using squid's bugzilla with Firefox 2.0.0.11, I've noticed some random behaviour. Mark an attach as obsolete, go back to the bug in question (by clicking on 'back to bug ' link), and the attach is not striped. Reloading shows it as obsolete. Same thing happends when adding an attachment (it does not get listed until I reload). (this is not an expected behaviur, I believe). I would swear that a patch got attached to another bug (this could me just me). Sometimes, I got listed obsoleted attachments when adding a new one in the 'Check each existing attachment made obsolete by your new attachment.' And sometimes I only get listed non-obsoleted attachments. Some times (and sometimes not), when trying to add an attachment to a bug, I got 'The form is already used' (or similar, I don't recall the exact error message). I belive this was after adding an attachment to another bug. Has anyone else noticed something like this? -- Gonzalo A. Arana
Re: [squid-users] 'include' feature
On Jan 9, 2008 10:04 PM, Adrian Chadd [EMAIL PROTECTED] wrote: On Wed, Jan 09, 2008, Gonzalo Arana wrote: I've filed bug# 2180 with these issues and my proposed fix. Once there is a desition on these, I'll forward this to 2.7. I'm happy for these patches to go in. Forwarding to 2.7 then. Regards, -- Gonzalo A. Arana
Re: [squid-users] 'include' feature
On Jan 10, 2008 9:55 AM, Gonzalo Arana [EMAIL PROTECTED] wrote: On Jan 9, 2008 10:04 PM, Adrian Chadd [EMAIL PROTECTED] wrote: On Wed, Jan 09, 2008, Gonzalo Arana wrote: I've filed bug# 2180 with these issues and my proposed fix. Once there is a desition on these, I'll forward this to 2.7. I'm happy for these patches to go in. Forwarding to 2.7 then. I would swear that I had uploaded the patch for squid2-HEAD before sending my Once there is a desition on these, I'll forward this to 2.7. mail. I've attached (again?) a patch for squid2-HEAD. I've also filed bug #2181 for the squid_2_7 branch, and attached the corresponding patch. Regards, -- Gonzalo A. Arana
Re: [squid-users] 'include' feature
(shifting this thread to squid-dev@) Adrian, Some things on the 'include' feature: * it does not support line wrapping (a line ending with '\\'): include /dev/null \ /dev/null FATAL: Unable to open configuration file: /dev/null \: (2) No such file or directory * it takes the rest of the line as a filename: include /dev/null /dev/null FATAL: Unable to open configuration file: /dev/null /dev/null: (2) No such file or directory * deep recursion triggers a warning, perhaps squid should die in this case? # in /usr/local/squid2-HEAD/etc/squid.conf include /dev/null include /usr/local/squid2-HEAD/etc/squid.conf 2008/01/09 09:39:47| WARNING: can't include /dev/null: includes are nested too deeply (16)! 2008/01/09 09:39:47| WARNING: can't include /usr/local/squid2-HEAD/etc/squid.conf: includes are nested too deeply (16)! and then squid start processing requests. I've filed bug# 2180 with these issues and my proposed fix. Once there is a desition on these, I'll forward this to 2.7. Regards, On Jan 8, 2008 1:12 PM, Adrian Chadd [EMAIL PROTECTED] wrote: On Tue, Jan 08, 2008, Gonzalo Arana wrote: (I do)**3 Will test it tomorow. Porting the changeset to 2.6 and 2.7 will be trivial.. Adrian -- Gonzalo A. Arana
Re: bzr VCS feedback
On Jan 7, 2008 7:15 PM, Robert Collins [EMAIL PROTECTED] wrote: On Mon, 2008-01-07 at 16:50 -0200, Gonzalo Arana wrote: Tsantilas, ... debian (and most linux/BSD distros I guess) come with a usable CVS version. Newer bzr is in backports.org, bsd ports, fedora and so on. Using backports always ended in some dpkg brokeness. If a debian user really wants to user bzr, installing from source I guess is the only long-term option. bzr by default, installed in /usr; while /usr/local would be a better choice I suppose. I guess this is a non-stopper anyway. * bzr 1.0 seems to use a lot of memory for checkout at least: $ ps axuw | grep bzr garana 25130 8.9 15.7 87256 80276 pts/16 S+ 10:46 1:06 /usr/bin/python /usr/bin/bzr branch http://www.squid-cache.org/bzr/squid3/trunk /home/garana/src/squid--bzr/trunk $ ps axuw | grep cvs garana 28048 16.4 0.3 4228 1860 pts/18 S+ 16:11 0:02 cvs co squid CVS used only 4K, while bzr used more than 80M (it devastated my workstation). They are doing completely different things. CVS is streaming the final texts from the remote server and is not processing any of the history data. bzr is cloning the deep history of the repository. 80M is more I understand the cause of this memory usage, but that does not make the memory usage less inconvenient. than I would have expected; I'd call that a bug. It won't ever be '4K' though because python takes more memory than that. I believe a reasonable memory usage would be accepted, but 80M to fetch the history data makes me think that bzr may not be as tested as people may think. Again, that's just my opinion. If squid-3.1 is shifted to bzr, I'll manage. $ ps axuw | grep bzr garana 27912 7.1 2.9 17872 15096 pts/16 S+ 16:03 0:02 /usr/bin/python /usr/bin/bzr bind http://www.squid-cache.org/bzr/squid3/trunk Since I am not a core squid developer, I know that I don't have a vote on this, but I have to say that I am quite disappointed with bzr resource usage. Do you mind me asking how much RAM your workstation has? Not at all! My office workstation has 512M. I wouln't like to have to shutdown firefox, apache, mysql, gkrellm, gaim and so on to do a 'bzr branch'. I don't need to do that when I do 'cvs checkout'. Regards, -- Gonzalo A. Arana
Re: bzr VCS feedback
On Jan 7, 2008 8:09 PM, Tsantilas Christos [EMAIL PROTECTED] wrote: Gonzalo Arana wrote: CVS used only 4K, while bzr used more than 80M (it devastated my workstation). It was not so bad But in the other hand 80M are not huge amount in these days .. Remember that some of us work at companies with small budgets :(. Does this overhead in getting a working copy is shadowed by the benefits of shifting to bzr? Since I am not a core squid developer, I know that I don't have a vote on this, but I have to say that I am quite disappointed with bzr resource usage. Did you use the local bzr copy to create local branches, get diffs, etc? These thinks looks very useful . No, sory, I just stopped after bzr branch finished. As soon as I have some spare time, I'll read bzr documentation a little further and give it a try at home (with more RAM). Regards, -- Gonzalo A. Arana
Re: bzr VCS feedback
Tsantilas, Thank you very much for the pointer. I was able to 'branch' squid3-bzr. About bzr, here are two things that I've noticed: * debian etch (stable) has an old version of bzr (0.11-1.1), which does not recognize squid-bzr repository: $ bzr branch http://www.squid-cache.org/bzr/squid3/trunk bzr: ERROR: Unknown branch format: 'Bazaar Branch Format 6 (bzr 0.15)\n' debian (and most linux/BSD distros I guess) come with a usable CVS version. * bzr 1.0 seems to use a lot of memory for checkout at least: $ ps axuw | grep bzr garana 25130 8.9 15.7 87256 80276 pts/16 S+ 10:46 1:06 /usr/bin/python /usr/bin/bzr branch http://www.squid-cache.org/bzr/squid3/trunk /home/garana/src/squid--bzr/trunk $ ps axuw | grep cvs garana 28048 16.4 0.3 4228 1860 pts/18 S+ 16:11 0:02 cvs co squid CVS used only 4K, while bzr used more than 80M (it devastated my workstation). $ ps axuw | grep bzr garana 27912 7.1 2.9 17872 15096 pts/16 S+ 16:03 0:02 /usr/bin/python /usr/bin/bzr bind http://www.squid-cache.org/bzr/squid3/trunk Since I am not a core squid developer, I know that I don't have a vote on this, but I have to say that I am quite disappointed with bzr resource usage. Regards, On Jan 3, 2008 4:27 PM, Tsantilas Christos [EMAIL PROTECTED] wrote: Hi Gonzalo, I am using the following repository: http://www.squid-cache.org/bzr/squid3/trunk Robert has a wiki page about squid bzr repository : http://wiki.squid-cache.org/Squid3VCS Regards, Christos Gonzalo Arana wrote: Robert, Robert Collins wrote: has anyone tried the bzr copy of the squid3 repository ? Any feedback/questions/concerns? $ wget http://www.squid-cache.org/~robertc/bzr/cvsps/squid3/bzr/squid3/branches/HEAD/ ... 15:55:29 ERROR 404: Not Found. Your bzr repository has gone away? Regards, -Rob -- Gonzalo A. Arana
Re: bzr VCS feedback
Robert, Robert Collins wrote: has anyone tried the bzr copy of the squid3 repository ? Any feedback/questions/concerns? $ wget http://www.squid-cache.org/~robertc/bzr/cvsps/squid3/bzr/squid3/branches/HEAD/ ... 15:55:29 ERROR 404: Not Found. Your bzr repository has gone away? Regards, -Rob -- Gonzalo A. Arana
zeroing buffers
Hi, I've been profiling squid2.6S16, and reached a similar conclusion to Adrian's: zeroing structures use about 2% of CPU usage, when squid as a hole is using about 30%. I did developed an optimization a little different than what Adrian did: instead of leaving large buffers untouched, I added an initialization function to every pool: o the default is simple to zero it (for most structures, the behaviour is unmodified). o pools of large buffers: the buffers per-se get '\0' assigned as its first character (just for sanity). o structures with large char[] members get their non char[] members zeroed via a single memset call, and other members get their first char assigned to '\0'. I believe this approach: o does not suffer from the fear of not knowing what would be the side effects of leaving some structures/buffers uninitialized, o we get large structures/buffers always initialized (this is particularily CPU saving in heavily used structures with large structures, like request_t, as well as for medium/large string memory pools). o there are CPU savings (according to my profilings, need verification, we save about 3% of CPU usage while squid is using about 30%). There is a preliminar patch implementing this in http://www.squid-cache.org/bugs/show_bug.cgi?id=2137 (which applies to squid-2.6.STABLE16). How does this sound to you? Regards, -- Gonzalo A. Arana
Re: string - refcounted string - buffer referencing
On Dec 9, 2007 2:57 AM, Adrian Chadd [EMAIL PROTECTED] wrote: This should explain why I'd like to get Squid-2.7 tagged and out the door so I can continue development. I'm well into changing Squid-2's string handling to take advantage of reference counting and it works - there's plenty of stringDup() calls in the main path now (especially when cloning the reply headers during client replies) but the allocator CPU savings have been eaten by the requirement for a seperate buffer header structure to be reference counted. snip Finally, once all the above is done and stable, the http header parsing routines can be modified to create buffer references rather than creating new strings - this should then give noticable CPU and memory footprint gains. In the meantime, seems that squid2 may be optimized a little, simply by avoiding expensive memset calls from memPoolFree. I've filed a bug for this enhancement: http://www.squid-cache.org/bugs/show_bug.cgi?id=2137 My test need verification, as my squid box was not a dedicated on (actually I run it on my desktop PC). I've tested only under Linux. Regards, -- Gonzalo A. Arana
Re: Style of commit messages
On 4/16/07, Henrik Nordstrom [EMAIL PROTECTED] wrote:
...
The first two lines (Author:, and the summary) is picked up by the
changeset tools.
This applies to both CVS repositories, both the devel.squid-cache.org
aka SourceForge, and the main repository.
http://www.squid-cache.org/Versions/v2/HEAD/changesets/
http://www.squid-cache.org/Versions/v3/3.0/changesets/
http://devel.squid-cache.org/changesets/
Just a cosmetic improvement to changesets table:
[style type=text/css]
td { white-space: nowrap }
[/style]
(angular brackets replaced by square brackets)
This makes changesets html tables more readable.
Regards,
--
Gonzalo A. Arana
Re: unsigned/64bit counters?
On 12/27/06, Henrik Nordstrom [EMAIL PROTECTED] wrote: ons 2006-12-27 klockan 16:20 -0300 skrev Gonzalo Arana: I've come across select_loop counter wrap (currently it is an 'int' variable). which for most things isn't a problem.. counters do wrap. It's a problem when gauges wraps, but not really a problem when counters wrap unless something reads the counter as an absolute value since startup and not a counter.. Having negative values for a counter does not seem right to me. Shouldn't counters be at least unsigned? Most isn't.. Does anyone disagree with having 64 bit unsigned counters? The drawbacks I see are: 1) 3rd party utility that parse cache manager information must support 64 bit unsigned values 2) and that snmp mibs should be changed accordingly. What would the real use benefits be of changing the MIB? Consistency between SNMP and cache manager. I'm fine with making the counter a long internally if that helps, but I just don't see the point of making it a 64-bit SNMP counter. It was just a thought anyway. Regards, -- Gonzalo A. Arana
unsigned/64bit counters?
Hi, I've come across select_loop counter wrap (currently it is an 'int' variable). Shouldn't counters be at least unsigned? Does anyone disagree with having 64 bit unsigned counters? The drawbacks I see are: 1) 3rd party utility that parse cache manager information must support 64 bit unsigned values 2) and that snmp mibs should be changed accordingly. Regards, -- Gonzalo A. Arana
Re: more profiling
On 9/19/06, Adrian Chadd [EMAIL PROTECTED] wrote: On Tue, Sep 19, 2006, Andres Kroonmaa wrote: ... Since then quite alot of changes have happened, so I'd suggest to look at the gprof stats to decide what funcs to probe with hires prof and add them. Yeah, I'm thinking that too. Is hires profiling *that* heavy? I've used it in my production squids (while I've used squid3) and the overhead was neglible. There is a comment in profiling.h claiming that rdtsc (for x86 arch) stalls CPU pipes. That's not what Intel documentation says (page 213 -numbered as 4-209- of the Intel Architecture Software Developer Manual, volume 2b, Instruction Reference N-Z). So, it should be harmless to profile as much code as possible, am I right? Also review the probes already there - you'd want to make sure a probe isn't left running at any function exit point - this would lead to accounting to a probe sections of code incorrectly. This could be automatically done by the compiler, if the profile probe was contained in an object. The object will get automatically destroyed (and therefore the profiling probe will stop) when the function exits. There's something fishy with best case timers. They shouldn't be zero, ever. Ditto worst case - they *can* get high due to task switches, but your worst cases look way too high, on P4 2.6G there should be 2.6G ticks per second. Your worst case looks like probes have been running for 8.9secs straight, seems unlikely. So there seems to be a need to get hires profiling uptodate with current squid code base. I did notice that but I don't know enough about the code to go digging. On x86 architecture, timestamp counter may not run at external clock rate. On different processor versions the meaning of a tick in this clock may vary. From Intel Architecture * Volume 3b (chap 18.9): For Pentium 4 processors, Intel Xeon Intel Core Solo and Intel Core Duo ...: the timestamp counter increments at a constant rate. That rate may be set by the maximum core-clock to bus-clock ratio of the processor or may be set by the frequency at which the processor is booted. The specific processor configuration determines the behaviour. The only important issue is that TS counter runs at a contant rate, but it is somewhat unknown (could be measured anyway). That said, the traces look much nicer now. There's definitely something weird going on with the nested traces though. I just don't have the time to go through the profiling code. Its definitely nicer to use than gprof but it'd be nice to keep counts of call graphs. Thats all I really use gprof for these days. We could build something like gprof call graph (with some limitations). Adding this shouln't be *that* difficult, right? Is there interest in improving the profiling code this way? (i.e.: somewhat automated probe collection adding call graph support). Regards, -- Gonzalo A. Arana
Re: more profiling
On 9/19/06, Adrian Chadd [EMAIL PROTECTED] wrote: On Tue, Sep 19, 2006, Gonzalo Arana wrote: There is a comment in profiling.h claiming that rdtsc (for x86 arch) stalls CPU pipes. That's not what Intel documentation says (page 213 -numbered as 4-209- of the Intel Architecture Software Developer Manual, volume 2b, Instruction Reference N-Z). So, it should be harmless to profile as much code as possible, am I right? Thats what I'm thinking! Things like perfsuite seem to do a pretty good job of it without requiring re-compilation as well. That seems promising. This could be automatically done by the compiler, if the profile probe was contained in an object. The object will get automatically destroyed (and therefore the profiling probe will stop) when the function exits. Cute! It'd still be a good idea to explicitly state beginning/end where appropriate. What might be nice is a i was deallocated at the end of the function rather than being deallocated explicitly counter so things could be noted? I don't understand the so things could be noted meaning :(, sory. We could build something like gprof call graph (with some limitations). Adding this shouln't be *that* difficult, right? Is there interest in improving the profiling code this way? (i.e.: somewhat automated probe collection adding call graph support). It'd be a pretty interesting experiment. gprof seems good enough to obtain call graph information (and call graph information only) and I'd rather we put our efforts towards fixing what we can find and porting over the remaining stuff from 2.6 into 3. We really need to concentrate on fixing up -3 rather than adding shinier things. Yet :) Agreed, getting a stable squid3 is a priority. It would be good to the goals of having a squid3 release to get better profiling information. But if we can trust gprof's call graph, then this profiling code improvement is not needed right now. I'm going to continue doing microbenchmarks to tax certain parts of Squid (request parsing, reply parsing, connection creation/teardown, storage memory management, small/large object proxying/caching, probably should do some range request tests as well) to find the really crinkly points and iron them out before the -3 release. Bout the only really crinkly point I see atm is the zero-sized reply stuff. I have a sneaking sense that the forwarder code is still slightly broken. Nothing the squid-guru-team cannot solve I hope :). Regards, -- Gonzalo A. Arana
Re: more profiling
On 9/19/06, Adrian Chadd [EMAIL PROTECTED] wrote: On Tue, Sep 19, 2006, Gonzalo Arana wrote: Bout the only really crinkly point I see atm is the zero-sized reply stuff. I have a sneaking sense that the forwarder code is still slightly broken. Nothing the squid-guru-team cannot solve I hope :). Hey, want to join? :) Tempting invitation :) but for the moment I'll be commited to another projects (I believe for 2 months or so). But once I finish on-going projects, I'd love to join. I manage about 20 squid servers for a hundred thousand ADSL dial-up clients. As soon as I return to squid development, I can provide a realworld testbench for squid3. Regards, -- Gonzalo A. Arana
Re: gzip+squid3 code
Hi, Inded, I've used this code in production for a while, but I had to finally shift back to squid2 due to squid3 being seriously broken (issue a query in squid's bugzilla looking for my e-mail address in the CC list or as a reporter). If there is still interest in this, I could continue my testings, but squid3 needs to get rid of some serious bug before I can give this a serious torture in my production servers. You may want to check this: http://webs.sinectis.com.ar/garana/patches/squid/1-add-gz.patch.dpatch.CVS Unless I am mistaken, that is the latest patch I've used for content compression. I will dig into this during the weekend and grab the latest patch. Regards, On 9/1/06, Joe Cooper [EMAIL PROTECTED] wrote: Henrik Nordstrom wrote: sön 2006-08-27 klockan 17:33 -0500 skrev Joe Cooper: Hey guys, Jon Kay here in Austin wrote it under contract to Swell (US work-for-hire laws apply, and the contract stipulates it explicitly) so I own it, and Ganzalo Arana made a few bugfixes. I'm happy to have it merged into mainline Squid, and I had given Ganzalo permission to do so some time ago, but I guess it never happened and I've been too busy to follow up. The first steps have now been taken and the code is up on devel.squid-cache.org for maintenance and review. Excellent. I'll just add that the code did work exactly as it was supposed to, in my testing at the time, but there remained at least one serious memory leak (possibly others) that led to it being unusable. I believe all crashes that I experienced running the code were attributable to this leak. Whether there are other issues that never had a chance to exhibit themselves is unknown to me. Ganzalo was running it, I believe, in production, so he may actually have fixes for the issues that I saw. I haven't been in contact with him for some time, however, so I don't know the state of things in his neck of the woods. As you may know, I'm out of the Squid business for the foreseeable, so I won't be doing anything else with the code. But I do hope it sees some real world use. -- Gonzalo A. Arana
Re: Tproxy patch
Cool, this fixes what some people have been telling me about FD limits under Squid-2.6 being stuck at 1024. Good find! If the limit comes from select(2) limits, people should consider shifting to epoll or kevent/kqueue, since they scale much better. Using select(2) with too many FDs is not really wise. Regards, -- Gonzalo A. Arana
Re: Occasional Zero Sized Reply error with 2.6
I've noticed this on squid3 as well. While trying to hunt this, I've
found this in
HttpStateData::readReply:
...
} else if (flag == COMM_OK len == 0 !flags.headers_parsed) {
fwd-fail(errorCon(ERR_ZERO_SIZE_OBJECT, HTTP_BAD_GATEWAY));
eof = 1;
flags.do_next_read = 0;
comm_close(fd);
} else if (flag == COMM_OK len == 0) {
/* Connection closed; retrieval done. */
eof = 1;
if (!flags.headers_parsed)
/*
* When we called processReplyHeader() before, we
* didn't find the end of headers, but now we are
* definately at EOF, so we want to process the reply
* headers.
*/
processReplyHeader();
else
Some thoughts:
0) The 'if (!flags.headers_parsed)' is dead code. Any ideas what
was the original idea of this?
1) A 'zero size object' is a good reply in this case? if server
closed the connection before we even got reply headers, this could
mean that server actually closed connection before it got our last
request.
2) In fact, sniffing with ethereal I've 'verified' this. Not a
foolproff check, but I have about 220ms of RTT to osnews.com, and you
can see that between packets 88 89 of attached pcap file there are
about 50ms. This means that: server closed connection, my squid3
sends the request, my squid3 receives the FIN packet for the very same
TCP connection.
3) How to reproduce: As Guido pointed out, this can be reproduced
(with some effort) while navigating through osnews.com. Server-side
connection keepalive timeout timing makes it difficult to reproduce.
4) How to react if server closes connection before we even get reply
headers? I propose to retry the same request (if method is GET, no
authentication data present in request, and possible many other
conditions).
I believe it would be great if we could check last TCP ACK value from
server-side when read(2) returns 0. This way, we could know if server
actually got the request and only retry if it didn't. Is there any
way to read that? tcp(7) manpage does not show anything useful :( (at
least on Linux).
Regards,
--
Gonzalo A. Arana
zero_size_object.pcap
Description: Binary data
Re: making cache manager presence/registration optional
Hi, I believe there is a little problem with this. If store digest is not enabled. I get: store_digest.cc: In function `void storeDigestRegisterWithCacheManager(CacheManager)': store_digest.cc:135: error: `registerAction' undeclared (first use this function) store_digest.cc:135: error: (Each undeclared identifier is reported only once for each function it appears in.) Attached is my proposal fix for this. Regards, On 5/28/06, Robert Collins [EMAIL PROTECTED] wrote: I hadn't heard anything back, so I've committed what I think is a tasteful implementation of the second option. This has allowed removing the stub_cache_manager.cc test suite file, as well as making a bunch of modules' init simpler. Cheers, Rob On Sun, 2006-05-28 at 00:18 +1000, Robert Collins wrote: I'd like to make the cachemgrRegister calls in various of our fooInit() calls not require dragging in the whole squid to the binary, this is part of the blow-out on linked in objects for squid. Secondly, I'd like to remove the idea of the cachemanager being a global object and make it be explicitly passed in when it exists. We discussed this somewhat on irc. Some possibilities: Assuming we have a CacheManager class with 'register' and 'unregister' virtual methods, we could: * add that as a parameter to the Init calls where this is desirable. * Have a separate call from Init in modules which asks the module to register all its menu items with a supplied CacheManager. I prefer the second option, as it makes the behaviour of init more occupied with 'what is required to use a module' rather than 'do everything that /squid/ needs done before running that is related to this module.' Henrik is concerned that this will increase the maintenance cost of main(), which is possibly true, but I think we can address that in future if needed, i.e. by a registry of the modules with a few functions like 'init', 'registerWithCacheManger' etc. Thoughts? Rob -- GPG key available at: http://www.robertcollins.net/keys.txt. -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.2.2 (GNU/Linux) iD8DBQBEej3JM4BfeEKYx2ERAoPWAJ4lSy5Rff8itOhm5WfyLcs06CA63ACgluZb OcntB2Y7OpvtYHxqh/MxQPk= =XJhV -END PGP SIGNATURE- -- Gonzalo A. Arana Index: store_digest.cc === RCS file: /cvsroot/squid/squid3/src/store_digest.cc,v retrieving revision 1.15 diff -U6 -p -r1.15 store_digest.cc --- store_digest.cc 29 May 2006 00:50:18 - 1.15 +++ store_digest.cc 29 May 2006 13:26:56 - @@ -38,15 +38,15 @@ * TODO: We probably do not track all the cases when * storeDigestNoteStoreReady() must be called; this may prevent * storeDigestRebuild/write schedule to be activated */ #include squid.h +#include CacheManager.h #if USE_CACHE_DIGESTS -#include CacheManager.h #include Store.h #include HttpRequest.h #include HttpReply.h #include MemObject.h #include SquidTime.h #include StoreSearch.h
Re: making cache manager presence/registration optional
I've just noticed these other issues:
1) epoll is enabled
2) xprof is not used
3) delay pools are not enabled
Attached is a proposal for this (supersedes my previous patch).
Regards,
On 5/29/06, Gonzalo Arana [EMAIL PROTECTED] wrote:
Hi,
I believe there is a little problem with this. If store digest is not
enabled. I get:
store_digest.cc: In function `void
storeDigestRegisterWithCacheManager(CacheManager)':
store_digest.cc:135: error: `registerAction' undeclared (first use this
function)
store_digest.cc:135: error: (Each undeclared identifier is reported only once
for each function it appears in.)
Attached is my proposal fix for this.
Regards,
On 5/28/06, Robert Collins [EMAIL PROTECTED] wrote:
I hadn't heard anything back, so I've committed what I think is a
tasteful implementation of the second option. This has allowed removing
the stub_cache_manager.cc test suite file, as well as making a bunch of
modules' init simpler.
Cheers,
Rob
On Sun, 2006-05-28 at 00:18 +1000, Robert Collins wrote:
I'd like to make the cachemgrRegister calls in various of our fooInit()
calls not require dragging in the whole squid to the binary, this is
part of the blow-out on linked in objects for squid.
Secondly, I'd like to remove the idea of the cachemanager being a global
object and make it be explicitly passed in when it exists.
We discussed this somewhat on irc.
Some possibilities:
Assuming we have a CacheManager class with 'register' and 'unregister'
virtual methods, we could:
* add that as a parameter to the Init calls where this is desirable.
* Have a separate call from Init in modules which asks the module to
register all its menu items with a supplied CacheManager.
I prefer the second option, as it makes the behaviour of init more
occupied with 'what is required to use a module' rather than 'do
everything that /squid/ needs done before running that is related to
this module.' Henrik is concerned that this will increase the
maintenance cost of main(), which is possibly true, but I think we can
address that in future if needed, i.e. by a registry of the modules with
a few functions like 'init', 'registerWithCacheManger' etc.
Thoughts?
Rob
--
GPG key available at: http://www.robertcollins.net/keys.txt.
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.2.2 (GNU/Linux)
iD8DBQBEej3JM4BfeEKYx2ERAoPWAJ4lSy5Rff8itOhm5WfyLcs06CA63ACgluZb
OcntB2Y7OpvtYHxqh/MxQPk=
=XJhV
-END PGP SIGNATURE-
--
Gonzalo A. Arana
--
Gonzalo A. Arana
Index: src/comm_epoll.cc
===
RCS file: /cvsroot/squid/squid3/src/comm_epoll.cc,v
retrieving revision 1.11
diff -U6 -p -r1.11 comm_epoll.cc
--- src/comm_epoll.cc 29 May 2006 00:50:18 - 1.11
+++ src/comm_epoll.cc 29 May 2006 14:05:54 -
@@ -191,20 +191,36 @@ commSetSelect(int fd, unsigned int type,
}
if (timeout)
F-timeout = squid_curtime + timeout;
}
+
+static void commIncomingStats(StoreEntry * sentry);
+
+void
+commEPollRegisterWithCacheManager(CacheManager manager)
+{
+manager.registerAction(comm_select_incoming,
+ comm_incoming() stats,
+ commIncomingStats, 0, 1);
+}
+static void
+commIncomingStats(StoreEntry * sentry)
+{
+StatCounters *f = statCounter;
+storeAppendPrintf(sentry, Total number of epoll(2) loops: %d\n, statCounter.select_loops);
+storeAppendPrintf(sentry, Histogram of returned filedescriptors\n);
+statHistDump(f-select_fds_hist, sentry, statHistIntDumper);
+}
+
/*
+ * comm_select
* Check all connections for new connections and input data that is to be
* processed. Also check for connections with data queued and whether we can
* write it out.
- */
-
-/*
- * comm_select
*
* Called to do the new-style IO, courtesy of of squid (like most of this
* new IO code). This routine handles the stuff we've hidden in
* comm_setselect and fd_table[] and calls callbacks for IO ready
* events.
*/
Index: src/main.cc
===
RCS file: /cvsroot/squid/squid3/src/main.cc,v
retrieving revision 1.62
diff -U6 -p -r1.62 main.cc
--- src/main.cc 29 May 2006 00:50:18 - 1.62
+++ src/main.cc 29 May 2006 14:05:55 -
@@ -864,13 +864,15 @@ mainInitialize(void)
#ifdef USE_SELECT
commSelectRegisterWithCacheManager(manager);
#endif
clientdbRegisterWithCacheManager(manager);
+#if DELAY_POOLS
DelayPools::RegisterWithCacheManager(manager);
+#endif
DiskIOModule::RegisterAllModulesWithCacheManager(manager);
#if USE_DNSSERVERS
dnsRegisterWithCacheManager(manager);
#endif
@@ -892,13 +894,15 @@ mainInitialize(void)
storeRegisterWithCacheManager(manager);
#if DEBUGSTRINGS
StringRegistry::Instance().registerWithCacheManager(manager);
#endif
+#if USE_XPROF_STATS
Re: external acl cache level
Just to summ up: Letting the helper do random combining/reordering leads into highly-ineffitient lookup algorithm, and apparently it is not needed. Cache level could structured in a 'path'-alike, or disjoint. Disjoint solves reordering issue in an efficient manner. Having each token a member of a cache level is discarded as an option. I would like to have a somehwat explicit cache level. Alternatives to this is: a) expand %| to some string: discarded. b) helper tells squid about format specification cache levels upon startup. c) squid notifies helper at startup about format cache levels. If there is any inconsistency, helper can notify admin. I prefer b, but c sounds good as well. -- Gonzalo A. Arana
Re: Squid-2.6 update
Either set up a new branch from squid-2.6, or submit it via squid-dev.
The latter is perhaps most suitable as it's a small feature ready for
immediate merge.
Attached is the patch for:
1) external acl grace patch for squid 2.6, leaving 'level' for near (?) future.
2) adds 'srcport', 'myaddr', and 'myport' acl tags.
But bringing external-acl-fuzzy up to 2.6 is also a good idea for the
level discussion below.
I've noticed the 'level' concept in squid2's branch
external-acl-fuzzy, which is not present in squid3. I would like to
add support for this to squid3, unless someone object.
The cache level concept unfortunately did not work out quite the way I
had hoped for and is why it has not been merged. It solves some
problems, but many still unsolved. The problem needs a little more study
before deciding on what should be done.
The main problem with the existing cache level approach is how to handle
the acl arguments.
And I guess that a reply of level=N should delete lower levels
(level=n where n N) from cache. So, for each reply, many cached
entries would be deleted ('N-1' to be precise).
Perhaps the solution is to simply allow format tags in the acl arguments
as well, allowing the data to be freely reordered making the cache level
concept fit much better, or a new format tag which expands into the
arguments.
Reordering and combining perhaps? Allowing combining would raise the
number of cached entries invalidations from N-2 to (2**N)-2 (I am not
counting current reply as an invalidation).
Thinking about it a bit further I start to like the idea of introducing
a new format tag for the acl arguments, with a default of adding the
arguments added at the end of the request if this tag is not used in the
external_acl_type specification.
The format tag should expand to some string we are sure is not present
in any other tags, which is something difficult to assure since we
have %{Header:member} tag. Adding 'level' support for external acl
cache implies the request/reply pair need some higher level structure
(say XML, or HTTP-alike), unless I am missing something.
Regards,
--
Gonzalo A. Arana
Index: src/cf.data.pre
===
RCS file: /cvsroot/squid/squid/src/cf.data.pre,v
retrieving revision 1.105
diff -u -r1.105 cf.data.pre
--- src/cf.data.pre 22 May 2006 01:28:25 - 1.105
+++ src/cf.data.pre 22 May 2006 14:12:28 -
@@ -2014,6 +2014,9 @@
concurrency=n concurrency level per process. Use 0 for simple helpers
who can only process a single request at a time.
cache=n result cache size, 0 is unbounded (default)
+ grace= Percentage remaining of TTL where a refresh of a
+ cached entry should be initiated without needing to
+ wait for a new reply. (default 0 for no grace period)
protocol=3.0 Use URL-escaped strings instead of quoting
FORMAT specifications
@@ -2021,10 +2024,13 @@
%LOGIN Authenticated user login name
%IDENT Ident user name
%SRC Client IP
+ %SRCPORT Client source port
%DST Requested host
%PROTO Requested protocol
%PORT Requested port
%METHOD Request method
+ %MYADDR Squid interface address
+ %MYPORT Squid http_port number
%PATH Requested URL-path (including query-string if any)
%USER_CERT SSL User certificate in PEM format
%USER_CERTCHAIN SSL User certificate chain in PEM format
Index: src/client_side.c
===
RCS file: /cvsroot/squid/squid/src/client_side.c,v
retrieving revision 1.95
diff -u -r1.95 client_side.c
--- src/client_side.c 22 May 2006 01:28:25 - 1.95
+++ src/client_side.c 22 May 2006 14:12:30 -
@@ -444,6 +444,7 @@
new_request-client_addr = old_request-client_addr;
new_request-my_addr = old_request-my_addr;
new_request-my_port = old_request-my_port;
+ new_request-client_port = old_request-client_port;
new_request-flags = old_request-flags;
new_request-flags.redirected = 1;
if (old_request-auth_user_request) {
@@ -3499,6 +3500,7 @@
request-client_addr = conn-peer.sin_addr;
request-my_addr = conn-me.sin_addr;
request-my_port = ntohs(conn-me.sin_port);
+ request-client_port = ntohs(conn-peer.sin_port);
request-http_ver = http-http_ver;
if (!urlCheckRequest(request) ||
httpHeaderHas(request-header, HDR_TRANSFER_ENCODING)) {
Index: src/external_acl.c
===
RCS file: /cvsroot/squid/squid/src/external_acl.c,v
retrieving revision 1.17
diff -u -r1.17 external_acl.c
--- src/external_acl.c 22 May 2006 01:28:30 - 1.17
+++ src/external_acl.c 22 May 2006 14:12:31 -
@@ -55,6 +55,7 @@
static char *makeExternalAclKey(aclCheck_t * ch, external_acl_data * acl_data);
static void external_acl_cache_delete(external_acl * def, external_acl_entry * entry);
static int external_acl_entry_expired(external_acl * def, external_acl_entry * entry);
+static int
Re: external acl cache level
On 5/22/06, Henrik Nordstrom [EMAIL PROTECTED] wrote:
mån 2006-05-22 klockan 11:46 -0300 skrev Gonzalo Arana:
Reordering and combining perhaps? Allowing combining would raise the
number of cached entries invalidations from N-2 to (2**N)-2 (I am not
counting current reply as an invalidation).
The big problem is the lookup, which we want to keep quick..
Invalidation is not strictly needed, depending on the lookup order. As
long as the lookup gives less detail higher priority there is no
conflict (only unneeded entries in the extacl cache).
Ah, I see now. As long as the helper squid follow lower level
number, higher priority policy, there is no need for cache
invalidation.
Just for the record: if a helper replies with level=N means that
there is no reponse cached for any level for this key (otherwise,
cached response would have been used and the question to the helper
would have not been asked.).
To be able to make sane lookup structures it is very beneficial if the
data can be structured in a path like structure. This worked out quite
okay except that there is acl types where the acl arguments (the data in
the acl statement) is more important than some request details
(external_acl_type format tags)...
I may be wrong, but reordering is needed in those cases, which is why
I proposed 'combining' key components: letting the helper specify
which request-tokens may be used for caching this response.
The format tag should expand to some string we are sure is not present
in any other tags, which is something difficult to assure since we
have %{Header:member} tag. Adding 'level' support for external acl
cache implies the request/reply pair need some higher level structure
(say XML, or HTTP-alike), unless I am missing something.
I am not sure I see the problem you refer to. Can you eloberate a bit on
what kind of problem you see?
Sure! Here is an example:
external_acl_type useless %{Host} %| /some/helper some-argument
acl yet_another_useless external useless %{Cookie} %| %{MYADDR}
We could just demand that /some/helper should be aware of request
levels (this is something you pointed out below). Sooner or later
this will lead to confusions.
Options:
1) To expand '%|' to some string that we know it won't be present in
any other tags. I fear no matter which string we choose for '%|'
expansion; that string could be present in (for instance) Cookie
request header.
2) As you proposed:
Another approach would be to mark the arguments per their key detail level.
Unless I misunderstand this, you are proposing that each request could
look something like this (I know that there are cleaner ways to do it,
this is just an example):
1=localhost 1=blah1 2=user_xxx 3=1.1.1.1
where each integer represent the key level.
With this approach, key-component level is assigned by squid
configuration, and is not per-request (which perhaps is what is
wanted).
3) We could let external helper to decide key-component level by using
something like XMLRPC or we could come up with our own protocol based
in, say, HTTP.
This encoding/protocol/structure (whatever this should be called)
should add support for something like HTTP's Vary: in the response,
the helper should indicate which components of the request were taken
into consideration for building the reply.
Of course, this 'custom-made-protocol' aproach is needed only if
key-component level is to be assigned per-request, which is what I've
been chasing so far.
Draft patch attached. This patch adds %DATA expanding into the acl
arguments (and %ACL expanding into the acl name for completeness).
Let me see If I follow correctly: with %DATA you can switch the order
of the arguments to external_acl, right? So you can make acl
arguments have higher priorities than external_acl formats.
Problem: %DATA have a slight problem with whitespace characters if the
helper is to handle arguments with whitespace AND multiple arguments in
the same acl type.. as currently written they both looks the same in the
%DATA expansion.. (a space character, appropriately encoded per the
helper protocol).
we seem to fall into some higher level structure is needed again.
Mainly because the external helper is needed to tell squid which
arguments have been used (combining approach).
Which reminds me.. external acl helper protocol should be switched by
detault to the 3.0 format for 2.6. The shell escaped format used in
2.5 was a bit of mistake.. (looks pleasing to humans, but is quite ugly
to computers)
I guess url-escaped arguments are much easier to decode than 'shell
escaped' ones.
The level adds structure to the requests by allowing it to be
structured in a path like manner when needed by introducing the level
separators in the request format.
%DST %| %PATH
Problems:
The helper is assumed to know the key levels defined in
external_acl_type. These are not reflected in the request data. Not sure
this actually is a problem, but results may be odd if the admin
configures
Re: Squid-2.6 update
On 5/17/06, Henrik Nordstrom [EMAIL PROTECTED] wrote: ons 2006-05-17 klockan 20:53 -0300 skrev Gonzalo Arana: I am particularly interested in epoll, connection pinning external acl grace parameter. If more hands are needed for any of them (coding, testing, what-ever), please let me know. You are more than welcome to get your hands dirty in splitting and isolating patches to make them suitable for merge. epoll is being worked on by Adrian Steven, but the other two is ready to be merged once they have been separated out cleanly.. OK, I'll leave that to them. The connection pinning is found in Adrians cacheboy tree. The external acl grace parameter is found in the external acl fuzzy branch, or Squid-3. I have the external acl grace patch ready for 2.6. Should I: 1) merge external-acl-fuzzy branch at sf.net's cvs, 2) apply it commit to squid2 HEAD at sf.net, 3) or report it to bugzilla? I've noticed the 'level' concept in squid2's branch external-acl-fuzzy, which is not present in squid3. I would like to add support for this to squid3, unless someone object. Regards, -- Gonzalo A. Arana
Re: Squid-2.6 update
On 5/17/06, Henrik Nordstrom [EMAIL PROTECTED] wrote: The SSL support has now been merged, and the queue of things to merge is now quite small. Remaining things: - epoll - Cygwin Windows service - external acl password= from 3.0 - external acl grace parameter from external_acl_fuzzy / 3.0 - Connection pinning - ETag (?) - New improved COSS I am particularly interested in epoll, connection pinning external acl grace parameter. If more hands are needed for any of them (coding, testing, what-ever), please let me know. Regards, -- Gonzalo A. Arana
Re: SourceForge CVS online again, almost.
On 5/14/06, Henrik Nordstrom [EMAIL PROTECTED] wrote: ... There also seem to be some automake issues at the moment.. make distclean is failing for me leaving quite a bit of stuff around.. Seems that it has been true since Dec 2005: http://www.squid-cache.org/mail-archive/squid-dev/200512/.html Regards, -- Gonzalo A. Arana
Re: Squid-3.0.PRE4 release plan
On 5/9/06, Doug Dixon [EMAIL PROTECTED] wrote: Hopefully this finalises our list of PRE4 bugs ... As if by magic, our list is still nine bugs long, but they're probably now the right ones to concentrate on. Isn't bug 1125 a duplicate of 624? Regards, -- Gonzalo A. Arana
Re: Squid-3.0.PRE4 release plan
On 5/6/06, Doug Dixon [EMAIL PROTECTED] wrote: ... If we have a manpower problem, but otherwise a good rate of progress, it may be worth trying to attract others we know may be interested in helping. I can help testing for bugfixes reproducing bugs. I'm adding my self to CC list of bugs that I've came across. Regards, -- Gonzalo A. Arana
proposal: add external_acl flags: max_requests min_alive
I am using external_acl extensively in my production servers, and I think it would be nice to add external_acl these flags: o max_requests: after this number or requests, the helper will be shutdown and replaced by a new one. This is to help leaky helpers. o min_alive: to avoid short-lived helpers, they should stay alive at least for this amount of seconds. This should help against fork-query-kill per request behaviour if request rate is drastically increased. As a side note, only one helper would be replaced at a time, to avoid replacing all of them at the same time. defaults would be: max_requests = no limit min_alive = 1 Does this make sense? Regards, -- Gonzalo A. Arana
Re: proposal: add external_acl flags: max_requests min_alive
..snip.. max_requests o min_alive: to avoid short-lived helpers, they should stay alive at least for this amount of seconds. This should help against fork-query-kill per request behaviour if request rate is drastically increased. Not sure about this one. If you have problems with a helper leaking a lot of memory requiring it to be frequently restarted then the helper should be fixed. Having safeguards in Squid to prevent Squid from restarting the helper frequently if the same configuration says it needs to be restarted after only a few requests is confusing I think. There very rarely is time related issues with the helpers. Nearly always the only relation to garbage/leaks in the helper is the number of requests processed. So even if the helper functions so poorly that it needs to be restarted after only a few requests you'd probably still want to do this even if the request rate is suddenly higher than you planned for. I proposed min_alive to limit helper recycle rate, not to increase it. max_requests imposes a relation between request rate and helper recycle rate. If request rate increases drastically (not a difficult thing to do), helpers may start recycling too fast. That's why it's measured in seconds, not in requests. Perhaps 'min_age' is a better name. And obviously the best action is to get the helper fixed... agreed :D. I am no having troubles with my helpers, but process rotation is a good policy (at some extent). helper rotation would help not only as a dirty way of coping with resource leakage, but as well as a way of using shorter idle timeouts for database conections. Regards, -- Gonzalo A. Arana
Re: proposal: add external_acl flags: max_requests min_alive
On 4/26/06, Henrik Nordstrom [EMAIL PROTECTED] wrote: ons 2006-04-26 klockan 11:13 -0300 skrev Gonzalo Arana: I proposed min_alive to limit helper recycle rate, not to increase it. I know, and I question the need to have this lower limit on the helper restarts. max_requests imposes a relation between request rate and helper recycle rate. If request rate increases drastically (not a difficult thing to do), helpers may start recycling too fast. Only if max_requests is set relatively low compared to how buggy the helper is. It should only be set low if there is serious problems, in which case you actually want to have it restarted even if the request rate is high. I see your point now (max_requests should be big enough). agreed :D. I am no having troubles with my helpers, but process rotation is a good policy (at some extent). As long as the helpers behave reasonably the daily rotation done as part of the log rotation process should be sufficient. Indeed. helper rotation would help not only as a dirty way of coping with resource leakage, but as well as a way of using shorter idle timeouts for database conections. idle timeouts is best managed in the helpers imho. And if this was the Right. gole there would be a need for a restart_interval option rather than min_alive, placing an upper limit on how long the helper is kept running. I have no particular issue with my own helpers, it just came to my mind that helper rotation should bring benefits and no troubles. I understand now that helper rotation rate should be slow anyway, so it would not bring concerns. Regards, -- Gonzalo A. Arana
Re: proposal: make config parsing non-blocking.
On 4/26/06, Robert Collins [EMAIL PROTECTED] wrote: On Wed, 2006-04-26 at 07:23 -0300, Gonzalo Arana wrote: Hi, perhaps we should discuss this on list ? (You took it offlist, I just followed suit). Absolutely! My mistake, sory (I'm used to have 'Reply-To' headers set in mails I receive from mailing lists). On 4/25/06, Robert Collins [EMAIL PROTECTED] wrote: On Tue, 2006-04-25 at 09:40 -0300, Gonzalo Arana wrote: I may be wrong, but the only signifficant difference (in wall clock time of blocking squid) between checking configuration and applying it are the external helpers (external_acl, auth helpers, piped loggers). Getting configuration data from the network could be a nice thing on the other hand, but external_acl provide the means for doing something similar. Rather than providing a 'slow migration' for every configuration directive, making 'slow migration' for 'expensive' directives (like external_acl, auth helpers, etc.) would have the same result but with less work, right? Nope. Any non blocking task can only be used by other non blocking tasks, all the way out the main event loop. Put another way, if you imagine a call stack: main() event_loop() FUNCTION() if FUNCTION is a non blocking task, such as doEvents(), or comm_select(), then it can call either blocking calls (for a performance penalty), or other non blocking calls (which is ideal). if FUNCTION is a blocking task like squidReconfigure(), then it can only call other blocking calls. So the *first* blocking call means that all other calls down from there need to be blocking. We cannot use any non blocking calls in the reconfiguration process while the configuration is a blocking event. While I understand your point, I don't understand why does this contradicts what I suggested. My proposal was something like this: 1) reconfigure is requested 2) parse configuration apply non-slow-migration directives. 3) open new passive socket (if port is changed). 4) assign new callbacks to that port. 5) mark all helpers as 'die create a new one when idle' (support for this flag has to be added, I think). This way, applying new configuration is a blocking procedure, but should be much faster than waiting for all external helpers to warm up. This is a way to achieve what you suggested, am I right? Not really. For instance: one common configuration of squid is to have the ACL's configured outside of the squid configuration, and reading these can take some time. right. Anyhow, what I thought you were saying was to have some slow sections within the parsing, which is where my statements did contradict you - because the parsing then has to be slow (as opposed to grouping the actions to be taken after parsing as fast/slow). I see, and I agree with you. It would be just great to have squid reconfiguration without closing passive sockets, but any deep rework on squid3 will delay it's stable release. My counterproposal was just to suggest a way of achieving this with minimum changes to squid3. Regards, -- Gonzalo A. Arana
Re: proposal: make config parsing non-blocking.
I may be wrong, but the only signifficant difference (in wall clock time of blocking squid) between checking configuration and applying it are the external helpers (external_acl, auth helpers, piped loggers). Getting configuration data from the network could be a nice thing on the other hand, but external_acl provide the means for doing something similar. Rather than providing a 'slow migration' for every configuration directive, making 'slow migration' for 'expensive' directives (like external_acl, auth helpers, etc.) would have the same result but with less work, right? Regards, On 4/25/06, Robert Collins [EMAIL PROTECTED] wrote: This is just a long term thought: should we make config parsing non blocking. This would allow reconfiguration to on a running system with less disruption than it has now, but... would require that all the configuration data be much more dynamic than it is now - for instance the closure of listening ports, conversion of a port that is currently listening as http to listen as https etc. The nice thing about this is that we could configure up a new configuration that is not 'active', make sure its all koscher and so forth, then let it slide into place by a pivot-like process - change the callbacks for all the listening ports, and update any other global variables - but not removing the prior config - and then let the old config free as references to it die. This isn't very tied into async/sync behaviour, but making it explicitly async would allow for parsing very big configs, or geting configuration data from over the network etc. Rob -- GPG key available at: http://www.robertcollins.net/keys.txt. -- Gonzalo A. Arana
Re: so what is involved in calling squid-3.0 'stable'?
I guess the short answer to the subject is 'a lot of time from the core team' :(. ..snip.. So, what is required. How can we engage the community in making squid-3 stable ? There seems to be non-trivial interest in making it happen, but whats the actual benchmark ? This reminds me the apache's 1.3 - 2.X transition: apache 2 is much more flexible and powerfull, but a lot of people still preffer 1.3. Mainly because apache 2.X does not add any signifficant feature from the sysadmin/apache user perception, and only a small fraction of apache users' actually need those new architecture features. I guess something similar is happening with squid3: current squid2 users do not realize about the good new features of squid3 (client streams = [icap, content compression], external_acl with it's negative_ttl, acl based tcp_outgoing_address, and so on), or perhaps they just don't need them. I'll start using it again and pushing forward with bug reports if there's someone there to work on them...last time I tried squid-3 I was seeing some odd While it's easy to say 'someone has to fix the bugs I find', I guess that's the squid3-users feeling. If a sysadmin is going to spend time testing squid3, he/she expects to have some feedback on the issues he/she finds. The same happends for newcomers to the squid3-dev community: as squid3 is extremely complex, they expect some 'tutor' to guide them; which again falls back to 'squid core team time' :(. I've been using squid3 in my production proxys for about 2 years, and it's been working great. I've allways commited every single bug I've came across to bugzilla, and posted a patch every time I could write one. Of course, I do not have enough knowledge of squid internals to commit the patches by my self, so, basically 'someone has to review commit the patches I write' :D. Fixing bugs for squid3 is not easy at all due to: 1) naming conventions (I agree with Nick Lewycky at some level). 2) missing documentation: doxygen is my best friend on understanding squid3 architecture. But sometimes is not enough :(, and reading hundreds of lines of code across tens of files is the only way I find and it takes a lot of time due to missing documentation. 3) abuse of some fancy C++ features complicates the whole matter, and sometimes they only make squid3 compile time even longer (over 40 minutes in an old CPU). There are definately people doing things around the source - I think harnessing the energy is the issue. I only have a small amount of time, and I'll probably be using it on toolchain support to make it easier for others to fix bugs - because thats something effective I can do in the timeframes I have available. That's cool. Changing the subject a little, there have been many new people introduce themselves on this list maybe with good intentions of working on squid, who seem to vanish as fast as they arrive. I wonder if they've simply (a) never intended to contribute in the first place, (b) done some work privately but never released it or (c) taken a good look at the code, and run away fast deciding it was all too hard ;-) Squid3 has a stair-shaped learning curve, so I believe it's (c). From a development perspective, I think it'd be of value to know why are there not more people developing squid. It seems to be just a hardcore few. Developers will come if squid3 is promoted, say by enumerating it's features in www.squid-cache.org. Hope this helps, -- Gonzalo A. Arana
Re: Where can I get the class diagram of squid 3.0 ?
doxygen may be helpful to build a class diagram. Regards, -- Gonzalo A. Arana
Re: Squid content compression
On 11/26/05, Henrik Nordstrom [EMAIL PROTECTED] wrote: On Fri, 25 Nov 2005, Gonzalo Arana wrote: Seems that I don't have CVS access to SF's CVS squid repository (I get Permission denied, even with the right password). What is your SF account name? My account name is garana. Regards Henrik -- Gonzalo A. Arana
Re: Squid content compression
Henrik, Seems that I don't have CVS access to SF's CVS squid repository (I get Permission denied, even with the right password). Regards, On 11/24/05, Henrik Nordstrom [EMAIL PROTECTED] wrote: On Wed, 23 Nov 2005, Gonzalo Arana wrote: I've been using content compression (content encoding) for a while and seems to work just fine. How should I provide the source code? A cvs.sf.net branch? Preferably so. Fro details see Routine for adding a new branch url:http://devel.squid-cache.org/CVS.html Adjust steps 1 2 for what your developments are based on.. Regards Henrik -- Gonzalo A. Arana
internal redirector support
I would like to add internal redirection support for squid. There is a patch (see: http://www.squid-cache.org/bugs/show_bug.cgi?id=1208). Henrik suggested in that the core access controls should be reworked. Any particular thoughts/wishes about this? Regards, -- Gonzalo A. Arana
Squid content compression
I've been using content compression (content encoding) for a while and seems to work just fine. How should I provide the source code? A cvs.sf.net branch? Regards, -- Gonzalo A. Arana
lib/cppunit-1.10.0/bootstrap.sh missing in squid-HEAD.snapshot
Hi, The file lib/cppunit-1.10.0/bootstrap.sh is missing in squid3-HEAD snapshot (downloaded from http://www.squid-cache.org/Versions/v3/HEAD/squid-HEAD.snapshot.tar.bz2), but is present in cvs repository (cvs.squid-cache.org:/squid). This causes an error when trying to run bootstrap.sh. There are other differences as well, but only this one seems signifficant (at least for me). Regards, -- Gonzalo A. Arana
Subscription
Hi, I was previously subscribed as [EMAIL PROTECTED], but I would like to shift my subscription to [EMAIL PROTECTED] ([EMAIL PROTECTED] is getting full quite often). Best regards, -- Gonzalo A. Arana
Re: squid benchmarking results - squid-3.0 + epoll()
On Thu, Dec 09, 2004 at 10:45:58PM +0530, Muthukumar wrote: hai gonzalo arana, Thanks again for detailed reply. Could you send the results? It shouln't be too hard to tell if squid3 can be optimized some way to enhance the number of requests per second. cpu usage is mostly user-mode, so --enable-cpu-profiling/-pg results should be meaningful. Attachment contains cpu_profiling for 160 req / sec of squid-3.0 with epoll(). From these results, it seems that you have reached the maximum request rate possible (bounded by CPU usage). Hard to tell actually, since there is too much PROF_UNACCOUNTED :(. Is there any tool available to detect NETWORK SATURATION / failure detection ? Difficult to answer since there are a number of possible limits in network bandwidth, such as: 1) PCI controller (server motherboards have a much better PCI controller than workstatsion motherboards), 2) If your network adapter driver uses DMA or interrupt based communication, and if it supports 'queueing' packets, or reading multiple packets per interrupt 3) Switching capability of your ehternet switch. It is good to check the histogram of packet sizes (at least in ethernet networks), and iptraf tool gives a nice and quick snapshot of this. If there are too many small packets, your server will have too many context switches, and that may starve your CPU. This is just a thought. In linux, I prefer to use iptables. ifconfig provides some numbers, but some processing is needed. I was curious about throughput on epoll vs poll/select. Never mind. Through put for epoll() - 180 rep / sec poll() - 165 rep / sec I have tuned sysctl -w net.ipv4.tcp_syncookies=1, ulimit -HSn there. But I am getting same error reply as, Connection.cc:485: error: 1/1 (s110) Connection timed out This may be caused by some resource exaustion (probably file descritors). You may use cache_mgr to find out what's the problem here (dmesg may give some useful tips as well). I have tuned parameters list on polyclt, polyserver and squid server as, echo 1 /proc/sys/net/ipv4/tcp_timestamps In production servers, I preffer to disable tcp timestamps. There seems to be some kind of incompatibility between linux implementation of tcp timestamps and some other OS's. snip... thanks Muthukumar Regards, G.
Re: squid benchmarking results - squid-3.0 + epoll()
Hi Muthukumar, On Thu, Dec 09, 2004 at 11:49:35AM +0530, Muthukumar wrote: hai gonzalo arana, thanks for detailed reply. Looks like you are running into a CPU bottleneck. Perhaps you may want to add --enable-cpu-profiling to configure, and check cpu usage data with cache manager, or compile and link with -pg and check results with gprof. I have done configuration with --enable-cpu-profiling and monitoring with cachemgr.cgi script. Could you send the results? It shouln't be too hard to tell if squid3 can be optimized some way to enhance the number of requests per second. Also, verify whether CPU usage is in kernel-mode or user-mode with vmstat during the test (sar gives this information as well). vmstat result when squid is running at peak load ( 180 req / sec ), procs ---memory-- ---swap-- -io --system-- cpu r b swpd free buff cache si sobibo in cs us sy id wa 1 0 0 119024 34116 6125200 0 110 1103579 69 31 0 0 1 0 0 99464 34144 6125600 0 148 13033 26 65 35 0 0 cpu usage is mostly user-mode, so --enable-cpu-profiling/-pg results should be meaningful. Looks like you are running with (the same?) CPU bottleneck. [snip] I did not get this. How to measure throughput (in bps)? In linux, I prefer to use iptables. ifconfig provides some numbers, but some processing is needed. I was curious about throughput on epoll vs poll/select. Never mind. 008.75| Connection.cc:485: error: 1/1 (s110) Connection timed out 008.75| error: raw write after connect failed after these req / sec configuration setting. Try enabling SYN cookies, and running squid with ulimit -HSn 131072. I have tuned sysctl -w net.ipv4.tcp_syncookies=1, ulimit -HSn there. But I am getting same error reply as, Connection.cc:485: error: 1/1 (s110) Connection timed out This may be caused by some resource exaustion (probably file descritors). You may use cache_mgr to find out what's the problem here (dmesg may give some useful tips as well). can we get more req / sec satisfacation for 512 MB RAM, 927.753 MHz cpu (p) ?? Hard to tell. With cache manager CPU usage information, I could try to give you an answer. Any way, you are getting time outs when connecting to squid, so there might be another problem which is limiting your squid response rate (such as fd exaustion). I am using /dev/null file system for benchmarking? Sounds like the right choice for testing/benchmarking squid's network code. Do you have benchmarking any results for squid? No, I'm sorry. I've just limited my self to use squid3 with epoll, and CPU savings is great in real life situations, where the number of idle connections per squid main loop is large (due to network delays and user's click interval). I assume your test scenario have a really low delay between client(s) and squid, and between squid and web server(s). In this context, I guess epoll, or kevent/kqueue can only give you a little improvement in the number of concurrent connections. regards muthukumar Regards, Gonzalo
Re: squid benchmarking results - squid-3.0 + epoll()
Hi Muthukumar, On Wed, Dec 08, 2004 at 04:49:40PM +0530, Muthukumar wrote: Hello Development Team, (I'm not a core squid developer). We had a benchmark and got results for the hardware setup, model name : Pentium III (Coppermine) cpu MHz : 927.753 RAM size : 512 I like to have your review on this. can we get more req / sec satisfaction on this setup? --- squid 3.0 without epoll(): Squid Cache: Version 3.0-PRE3 configure options: '--prefix=/usr/local/squid3pre' '--with-aufs-threads=32' '--with-descriptors=32768' '--with-pthreads' '--enable-storeio=null,ufs,aufs' '--enable-debug-cbdata' cache_mem 200MB cache_dir null /dev/null top output: PID USER PR NI VIRT RES SHRS %CPU %MEMTIME+ COMMAND 6428 squid 25 0 336m 322m 3276 R 99.9 63.8 3:05.54 squid Results: req.rate:167.50 rep.rate:167.17 Looks like you are running into a CPU bottleneck. Perhaps you may want to add --enable-cpu-profiling to configure, and check cpu usage data with cache manager, or compile and link with -pg and check results with gprof. Also, verify whether CPU usage is in kernel-mode or user-mode with vmstat during the test (sar gives this information as well). squid 3.0 with epoll(): Squid Cache: Version 3.0-PRE3 configure options: '--prefix=/home/muthu/squidepoll' '--enable-epoll' '--with-aufs-threads=32' '--with-descriptors=32768' '--with-pthreads' '--enable-storeio=null,ufs,aufs' '--disable-poll' '--disable-select' '--disable-kqueue' '--disable-optimizations' '--enable-debug-cbdata' cache_mem 200MB cache_dir null /dev/null top output: PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND 8358 squid 16 0 425m 407m 3428 R81.2 90.6 1:46.13 squid Results: req.rate:182.35 rep.rate:180.20 I want to have your analysis on this. I am getting errors, Looks like you are running with (the same?) CPU bottleneck. epoll's advantage is that CPU usage does not grow with the number of idle TCP connections. If the number of concurrent connections is large, and there are no idle connections, epoll should only give a small increase in throughput (no cpu is used for traversing the list of fds). Is, by any chance, throughput (in bps) slightly larger with epoll? 008.75| Connection.cc:485: error: 1/1 (s110) Connection timed out 008.75| error: raw write after connect failed after these req / sec configuration setting. Try enabling SYN cookies, and running squid with ulimit -HSn 131072. Regards, Muthukumar. Hope this helps, Gonzalo
squid-3.0-20031218: can't get core dump
Hi,
I've run into a couple of 'Assertion failed' and 'FATAL: Segmantation
violation..' messages, but I couln't get a core dump.
Besides ulimit -HSc unlimited, enough disk space and directory
permissions for user nobody in coredump_dir, am I missing something?
I don't even get a core if I do kill -ABRT `cat /var/run/squid.pid`, but
squid dies.
I get a core dump with this source:
cat x.c EOF
#include assert.h
int main() { assert(0); }
EOF
Thanks in advance,
--
Gonzalo Arana [EMAIL PROTECTED]
UOL-Sinectis S.A.
squid-3.0-PRE3-20031112 icp denied bug?
Hi, Brief version: squid 3 dies when an icp query is denied. Long version: If I configure squid 3.0-PRE3-20031112 with icp_access deny all (which I think is the default), and receives an ICP query from a squid 2.5STABLE3, squid 3 dies with (debug_options ALL,9): icpHandleUdp: FD 11: received 107 bytes from REMOTE-IP. init-ing hdr: 0x84ca038 owner: 1 aclCheckFast: list: 0x82da8c0 ACLChecklist::preCheck: 0xb8f8 checking 'icp_access deny all' ACLList::matches: checking all aclMatchAcl: checking 'acl all src 0.0.0.0/0.0.0.0' aclMatchIp: 'REMOTE-IP' found aclmatchAclList: 0xb8f8 returning true (AND list satisfied) ACLChecklist::markFinished: 0xb8f8 checklist processing finished hdr-owner = 1 cleaning hdr: 0x84ca038 owner: 1 hdr-owner = 1 cleaning hdr: 0x84ca038 owner: 1 ACLChecklist::~ACLChecklist: destroyed 0xb8f8 icpDenyAccess: Access Denied for REMOTE-IP by all. icpUdpSend: FD 11 sending ICP_DENIED, 103 bytes to REMOTE-IP:3130 assertion failed: HttpHeader.cc:374: hdr-owner hoNone hdr-owner = hoReply hdr-owner is 0 (zero), so hdr-owner hoNone is violated. Is that assert ok? Should it be hdr-owner = hoNone? Thank you very much in advance, -- Gonzalo Arana [EMAIL PROTECTED] UOL-Sinectis S.A.
Where to do Transfer-Encoding?
Hi, It's me again. Is clientReplyContext::sendMoreData a good place to do actual encodings of data ([de]chunk/deflate)? If so, may (must?) I encode next(), and replace it with encoded data? May I replace that node with more than one node? May I traverse (and encode) all *ourNode list nodes in a single call to sendMoreData? Thank you very much in advance, -- Gonzalo Arana [EMAIL PROTECTED] UOL-Sinectis S.A.
TE(gzip) on squid 3.0
Hi, It's me again. Is clientReplyContext::pushStreamData() a good place to encode data (deflate/chunk/dechunk)? May I use a buffer other than 'source' parameter to pushStreamData? Must I? I am 'forwarding' TE patch to squid 3.0, but I found my self into trouble when deciding where to do the acutal encoding. Thank you very much in advance, -- Gonzalo Arana [EMAIL PROTECTED] UOL-Sinectis S.A.
Re: squid-3.0-PRE3-20031008 w epoll bug?
Hi, On Mon, 2003-10-13 at 09:37, Robert Collins wrote: On Sat, 2003-10-11 at 06:30, Gonzalo Arana wrote: Hi, (I'm back to squid-gzip task now). I come up to this situation: squid 3.0-PRE3-20031008 with epoll kernel 2.4.21 patched with http://www.xmailserver.org/linux-patches/epoll-lt-2.4.21-0.18.diff When a client requests a very long object (such as a video), squid uses 100% of CPU. It was caused because epoll_wait returned immediately reporting that connection to client can be written without blocking. So squid was continously calling to epoll_wait, which in turn returned immediately. This is something I was about to look into. Thank you. I'm glad I can help Two things: 1) why the change to COMM_TIMEOUT as the return value? That seems unrelated to the issue. It is true that it does not fix the problem, but I think it is more appropiate to return COMM_TIMEOUT if no file descriptor had any activity after timeout specified (comm_poll.cc returns COMM_TIMEOUT in comm_select if this happends -unless I am wrong-). 2) Perhaps we should set the epoll to edge triggered, which IIRC was the default in the early epoll API (and is not now ?) mmm... that would require (I think) to read/write repeatedly until EAGAIN is returned (non-blocking i/o). I will submit my suggested patch to bugzilla (Reuben Farrelly had reported this problem). Hope it helps, Cheers, Rob -- Gonzalo Arana [EMAIL PROTECTED] UOL-Sinectis S.A.
squid-3.0-PRE3-20031008 w epoll bug?
Hi,
(I'm back to squid-gzip task now).
I come up to this situation:
squid 3.0-PRE3-20031008 with epoll
kernel 2.4.21 patched with
http://www.xmailserver.org/linux-patches/epoll-lt-2.4.21-0.18.diff
When a client requests a very long object (such as a video), squid uses
100% of CPU.
It was caused because epoll_wait returned immediately reporting that
connection to client can be written without blocking.
So squid was continously calling to epoll_wait, which in turn returned
immediately.
Here's my attempt to solve this (first attach).
Also, I provide attachments generated with:
debug_options ALL,1 5,9
If it isn't a reasonable/logical way to solve this problem, please let
me know (and explain to me how it should be fixed, so I can rewrite the
fix).
Hope it helps awaiting for feedback,
--
Gonzalo Arana [EMAIL PROTECTED]
UOL-Sinectis S.A.
Fixes (correctly?) a 100% CPU usage on large/streaming content delivered
when using epoll.
Kernel 2.4.21 patched with
http://www.xmailserver.org/linux-patches/epoll-lt-2.4.21-0.18.diff
diff -bur squid-3.0/src/comm_epoll.cc squid-3.0-fixed/src/comm_epoll.cc
--- squid-3.0/src/comm_epoll.cc Sun Aug 3 18:38:15 2003
+++ squid-3.0-fixed/src/comm_epoll.cc Fri Oct 10 16:37:35 2003
@@ -239,7 +239,7 @@
getCurrentTime();
if (num == 0)
-return COMM_OK;/* No error.. */
+return COMM_TIMEOUT; /* No error.. */
for (i = 0, cevents = pevents; i num; i++, cevents++) {
fd = cevents-data.fd;
@@ -247,11 +247,14 @@
debug(5, DEBUG_EPOLL ? 0 : 8) (comm_select(): got fd=%d events=%d
F-read_handler=%p F-write_handler=%p\n,
fd,cevents-events,F-read_handler,F-write_handler);
+ //TODO: add EPOLLPRI??
if(cevents-events (EPOLLIN|EPOLLHUP|EPOLLERR)) {
if((hdl = F-read_handler) != NULL) {
debug(5, DEBUG_EPOLL ? 0 : 8) (comm_select(): Calling read handler
on fd=%d\n,fd);
F-read_handler = NULL;
hdl(fd, F-read_data);
+} else { // if POLLIN but no handler, remove interest
+commSetSelect(fd, COMM_SELECT_READ, NULL, NULL, 0);
}
}
@@ -260,6 +263,8 @@
debug(5, DEBUG_EPOLL ? 0 : 8) (comm_select(): Calling write handler
on fd=%d\n,fd);
F-write_handler = NULL;
hdl(fd, F-write_data);
+} else {
+commSetSelect(fd, COMM_SELECT_WRITE, NULL, NULL, 0);
}
}
}
Situation:
fd type
9 to client
14 to web server
read more data from web server
2003/10/10 16:08:55.588| comm_poll: 1+0 FDs ready
2003/10/10 16:08:55.588| comm_poll: FD 14 ready for reading
2003/10/10 16:08:55.588| comm_read_try: fd 14, size 87379, retval 1620, errno 0
2003/10/10 16:08:55.588| comm_calliocallback: 0x84f7b3c
2003/10/10 16:08:55.588| comm_iocallbackpending: 0x84f7b3c
2003/10/10 16:08:55.588| comm_calliocallback: 0x84f7b3c
2003/10/10 16:08:55.588| commSetTimeout: FD 14 timeout 900
want to write to FD 9
2003/10/10 16:08:55.589| commSetSelect: FD 9 type 2
2003/10/10 16:08:55.589| comm_read, queueing read for FD 14
and read from fd 14
2003/10/10 16:08:55.589| commSetSelect: FD 14 type 1
2003/10/10 16:08:55.589| comm_iocallbackpending: (nil)
2003/10/10 16:08:55.589| comm_poll: 1+0 FDs ready
2003/10/10 16:08:55.589| comm_poll: FD 9 ready for writing
2003/10/10 16:08:55.589| comm_write_try: fd 9: tried to write 1620 bytes, retval 1620,
errno 0
ok, data has been written to client
2003/10/10 16:08:55.589| comm_calliocallback: 0x852e9d0
2003/10/10 16:08:55.589| comm_iocallbackpending: 0x852e9d0
2003/10/10 16:08:55.589| comm_calliocallback: 0x852e9d0
2003/10/10 16:08:55.589| comm_iocallbackpending: (nil)
2003/10/10 16:08:55.658| comm_poll: 1+0 FDs ready
squid can read more data from web server.
2003/10/10 16:08:55.658| comm_poll: FD 14 ready for reading
2003/10/10 16:08:55.658| comm_read_try: fd 14, size 87379, retval 665, errno 0
2003/10/10 16:08:55.658| comm_calliocallback: 0x84f7b3c
2003/10/10 16:08:55.658| comm_iocallbackpending: 0x84f7b3c
2003/10/10 16:08:55.658| comm_calliocallback: 0x84f7b3c
2003/10/10 16:08:55.658| commSetTimeout: FD 14 timeout 900
squid want to to write to client (since it has read more data), and read from
server
2003/10/10 16:08:55.659| commSetSelect: FD 9 type 2
2003/10/10 16:08:55.659| comm_read, queueing read for FD 14
2003/10/10 16:08:55.659| commSetSelect: FD 14 type 1
2003/10/10 16:08:55.659| comm_iocallbackpending: (nil)
2003/10/10 16:08:55.659| comm_poll: 1+0 FDs ready
writing to client.
2003/10/10 16:08:55.659| comm_poll: FD 9 ready for writing
2003/10/10 16:08:55.659| comm_write_try: fd 9: tried to write 665 bytes, retval 665,
errno 0
2003/10/10 16:08:55.659| comm_calliocallback: 0x852e9d0
2003/10/10 16:08:55.659| comm_iocallbackpending: 0x852e9d0
2003/10/10 16:08:55.659| comm_calliocallback: 0x852e9d0
2003/10
CE(gzip) for squid-2.5STABLE3
Hi,
I wrote a mail a couple of days ago asking for guidelines about adding
Content-Encoding: gzip support for squid.
I have been working, but now I'm stucked.
I modified clientSendMoreData this way (see attached patch for more
detailed info):
if (http-out.offset == 0) {
check if log mime headers
rep = clientBuildReply(http, buf, size);
if (rep) {
body too large check
if (must_compress) {
compress 'size' bytes located at 'buf', producing 'zbytes'
of compressed bytes in zipped_buff.
size = zbytes - rep-hdr_sz;
memcpy(buf + rep-hdr_sz, zipped_buff, zbytes);
}
...
} else if (!http-request-range) {
if (must_compress) {
compress 'size' bytes located at 'buf' to zipped_buf
body_size += zbytes - pbytes;
size = zbytes;
memcpy(buf, zipped_buf, zbytes);
}
http-out.offset += body_size;
comm_write(fd, buf, size, clientWriteBodyComplete, http, NULL);
/* NULL because clientWriteBodyComplete frees it */
return;
}
...
And the call sequence from cache_log is this (omitting some entries):
clientProcessRequest: GET $URL
clientProcessRequest: TCP_MISS for $URL
clientProcessMiss: 'GET URL'
storeClientCopy: $HASH_KEY, seen 0, want 0, size 4096
storeClientCopy2: $HASH_KEY
storeClientCopy3: Waiting for more
storeClientCopy2: $HASH_KEY
storeClientCopy3: Copying from memory (1368 bytes, hi=1368, lo=0)
reply got from server is 1368 bytes = 133 headers + 1335 body
clientSendMoreData: $URL, 1368 bytes
clientSendMoreData: FD 9 '$URL', out.offset=0
clientBuildReplyHeader: can't keep-alive, unknown body size
clientSendMoreData: (no data sent yet) (http-out.offset == 0)
gzip_data: Got 1235 bytes, out avail 4086 bytes
have 1235 plain bytes and 4k zipped buffer to write output
gzip_data writes 10 bytes (gzip header) to outbuffer.
clientSendMoreData: Appending 10 bytes after 133 bytes of headers
clientSendMoreData: packed reply: 207 bytes
reply sent to client has: 207 bytes header + 10 bytes content
more content may come later
clientSendMoreData: queueing mb(217 bytes) and clientWriteComplete
clientWriteComplete: FD 9, sz 217, err 0, off 143, len -1
storeClientCopy: $HASH_KEY, seen 143, want 143, size 4096, cb !NULL,
cbdata !NULL
storeClientCopy2: $HASH_KEY
storeClientCopy3: Copying from memory (1225 bytes, hi=1368, lo=0)
ok, here comes my problem:
1235 bytes have been 'eaten' by last call to clientSendMoreData-
gzip_data, but storeClientCopy3 thinks it has only 'consumed' 10
bytes.
Should I alter http-entry-mem_obj-inmem_hi??
I guess storeClientCopy3 thinks that 10 bytes has been 'consumed'
because http-out.offset has been incremented by 10, rather than 1335
(original body size so far).
How should I fix this? I mean, clientSendMoreData is called with data is
has already processed.
Thank you very much in advance,
--
Gonzalo Arana [EMAIL PROTECTED]
UOL-Sinectis S.A.
--- squid-2.5.STABLE3/src/client_side.c Sat May 24 08:08:41 2003
+++ squid-2.5.STABLE3-visolve_tcp_rtsignal-gzip/src/client_side.c Wed Aug 20
15:31:24 2003
@@ -1395,57 +1396,87 @@
getMyHostname(), ntohs(Config.Sockaddr.http-s.sin_port));
#endif
if (httpReplyBodySize(request-method, rep) 0) {
debug(33, 3) (clientBuildReplyHeader: can't keep-alive, unknown body size\n);
request-flags.proxy_keepalive = 0;
}
/* Signal keep-alive if needed */
httpHeaderPutStr(hdr,
http-flags.accel ? HDR_CONNECTION : HDR_PROXY_CONNECTION,
request-flags.proxy_keepalive ? keep-alive : close);
#if ADD_X_REQUEST_URI
/*
* Knowing the URI of the request is useful when debugging persistent
* connections in a client; we cannot guarantee the order of http headers,
* but X-Request-URI is likely to be the very last header to ease use from a
* debugger [hdr-entries.count-1].
*/
httpHeaderPutStr(hdr, HDR_X_REQUEST_URI,
http-entry-mem_obj-url ? http-entry-mem_obj-url : http-uri);
#endif
+
+#if USE_CEGZIP
+/* If no ranges involved, and
+ * client accepts gzipped data, and
+ * content isn't alreadly 'encoded' (compressed, or something else)
+ */
+if (!request-range
+(httpHeaderGetAcceptEncoding(http-request-header) ENCODING_GZIP)
+ !httpHeaderGetContentEncoding(rep-header)) {
+int cl = 9; /* //TODO: write CompressionLevel(); */
+
+/* if client accepts gzipped data
+* and acls are matched, do compress.
+*/
+httpHeaderPutStr(hdr, HDR_CONTENT_ENCODING, gzip);
+assert(http-conn);
+debug(33, 3)(Setting compression %d level on fd %d\n, cl, http-conn-fd);
+http-compress.level = cl;
+http-compress.offset = rep-hdr_sz; /* //TODO: set to header size */
+http-compress.gzdata = xmalloc(sizeof(z_stream)); /* //TODO: use mem pools
instead */
+/* //TODO: where should I free gzdata */
+ deflateInit2(http-compress.gzdata, http-compress.level
squid adding gzip support
Hi, I am interested in adding support for Content-Encoding: gzip to squid. I had done some patching, and I managed to get squid compressed response, but I get corrupted data. I would like to ask some questions about squid code in order to solve that problem. Sample question: size of buffer has to change, since it has compressed data, but (in client_sice.c: clientSendMoreData()) entry-mem_obj-inmem_hi has the size too. How should I modify this? May I alter this lvalue directly? Cheers, -- Gonzalo Arana [EMAIL PROTECTED] UOL-Sinectis S.A.
Re: squid adding gzip support
Hi,
Thanks Henrik and Robert for all of your suggestions (particularly hints
of where to patch squid-2.5, and about rfc2616, 215 warning).
New issues about this:
1) When to compress:
a) Client must have sent Accept-Encoding: and gzip has to be listed
in supported encodings ('compress' could be implemented with not
too much effort).
b) Server response should not be Transfer-Encoding: chunked. If it
is, it should be de-chunked, right?
c) Content-type of answer is text/html or text/plain (this could be
moved to gzip_access, I think).
d) Answer from server is not already compressed :-).
d) Server response does not contain Content-Range header (in case of
a miss). I simply won't compress partial data. This is due to
the fact that Ranges header usually is used on 'resume download'
operation, and I want to speed up interactive navigation.
e) gzip_access (new directive) is ok
I will add this in order to enable compression only on dial-up
clients (slow clients).
Is this ok?
2) Where to compress:
I will compress on comm_write.
Thank you very much for your wise adivce.
I have to check where to start compressing, since comm_write only
gets data to be written (other wise, headers will be compressed to).
Would it be OK to add something like this to struct _fde? (I mean,
is it the right place?).
struct {
unsigned int compress_level:4; /* compression level: 1-9 */
unsigned int compress_offset;
/* how many bytes have to skip before to compress */
z_stream* zstream; /* zlib internal state */
} compress;
3) Which squid to patch:
Unless squid 3 is going to be STABLE next week, I have to work on
squid 2.5, since I have a deadline of 1 month to accomplish this :-(
Of course, Once this is working, I will start working on a patch
for squid3.
4) Answer modification:
a) Should I alter Vary header (by adding Accept-Encoding)?
b) ETag: May I build it from md5sum of compressed content?
Thank you very much in advance,
--
Gonzalo Arana [EMAIL PROTECTED]
UOL-Sinectis S.A.
