Re: [Server-devel] Squid tuning recommendations for OLPC School Server tuning...
2008/9/23 Martin Langhoff [EMAIL PROTECTED]: Any way we can kludge our way around it for the time being? Does squid take any signal that gets it to shed its index? It'd be pretty trivial to write a few cachemgr hooks to implement that kind of behaviour. 'flush memory cache', 'flush disk cache entirely', etc. The trouble is that the index is -required- at the moment for the disk cache. if you flush the index you flush the disk cache entirely. There's no hard limit for squid and squid (any version) handles memory allocation failures very very poorly (read: crashes.) Is it relatively sane to run it with a tight rlimit and restart it often? Or just monitor it and restart it? It probably won't like that very much if you decide to also use disk caching. You can limit the amount of cache_mem which limits the memory cache size; you could probably modify the squid codebase to start purging objects at a certain object count rather than based on the disk+memory storage size. That wouldn't be difficult. Any chance of having patches that do this? I could probably do that in a week or so once I've finished my upcoming travel. Someone could try beating me to it.. The big problem: you won't get Squid down to 24meg of RAM with the current tuning parameters. Well, I couldn't; and I'm playing around Hmmm... with Squid on OLPC-like hardware (SBC with 500mhz geode, 256/512mb RAM.) Its something which will require quite a bit of development to slim some of the internals down to scale better with restricted memory footprints. Its on my personal TODO list (as it mostly is in line with a bunch of performance work I'm slowly working towards) but as the bulk of that is happening in my spare time, I do not have a fixed timeframe at the moment. Thanks for that -- at whatever pace, progress is progress. I'll stay tuned. I'm not on squid-devel, but generally interested in any news on this track; I'll be thankful if you CC me or rope me into relevant threads. Ok. Is there interest within the squid dev team in moving towards a memory allocation model that is more tunable and/or relies more on the abilities of modern kernels to do memory mgmt? Or an alternative approach to handle scalability (both down to small devices and up to huge kit) more dynamically and predictably? You'll generally find the squid dev team happy to move in whatever directions make sense. The problem isn't direction as so much as the coding to make it happen. Making Squid operate well in small memory footprints turns out to be quite relevant to higher performance and scalability; the problem is in the doing. I'm hoping to start work on some stuff to reduce the memory footprint in my squid-2 branch (cacheboy) once the current round of IPv6 preparation is completed and stable. The developers working on Squid-3 are talking about similar stuff. Adrian
Re: [Server-devel] Squid tuning recommendations for OLPC School Server tuning...
On tis, 2008-09-23 at 14:57 +0800, Adrian Chadd wrote: You can limit the amount of cache_mem which limits the memory cache size; you could probably modify the squid codebase to start purging objects at a certain object count rather than based on the disk+memory storage size. That wouldn't be difficult. Any chance of having patches that do this? I could probably do that in a week or so once I've finished my upcoming travel. Someone could try beating me to it.. The relevant code locations for implementing this if you want to take a stab at it yourself is the maintain function in each cache_dir type (src/fs/*/store_dir_...) Should be trivial to add a cache_dir parameter specifyung the max number of files in this cache_dir, and use this in the maintenance function. Regards Henrik signature.asc Description: This is a digitally signed message part
Re: [MERGE] Address more comments from Amos.
Amos Jeffries has voted approve. Status is now: Semi-approved For details, see: http://bundlebuggy.aaronbentley.com/project/squid/request/%3C200809222346.m8MNkYjC059268%40harfy.jeamland.net%3E Project: Squid
Re: [RFC] 3.1 branching
Lets set a date: 29 September? Well, I'll call for a partial hold right now. Please only commit bug fixes, minor cleanups, and features already on the roadmap for 3.1. The rest can be queued for commit post branch. 5 days to go and these bits left * connection pinning I thinking it's been outstanding long enough we should get it into 3.1, even if we have to hold PRE1 a week after the branch to stabilize it (or test it in PRE1 ?). Awaiting merge. (1-2 days). * kerberos helper Markus has supplied another set of code, I assume its been tested externally and just needs a drop-in. Awaiting re-submission with some alterations. Non-blocker. * eCAP Awaiting merge submission. * source layout Started. Non-blocker. * source formatting scripts committed for use already, I was planning on starting the re-format and testbed run today here. Now waiting on completion of other feature code. Other misc stuff which qualifies but need some work if people have time: Awaiting a verify: http://www.squid-cache.org/bugs/show_bug.cgi?id=2433 Awaiting a verify test and debug: http://www.squid-cache.org/bugs/show_bug.cgi?id=2393 Awaiting a developer: * testbed level-02-maximus.opts opposite of minimal. Everything enabled, leave configure to handle clashes properly etc. Amos -- Please use Squid 2.7.STABLE4 or 3.0.STABLE9
[Success!] [MERGE] Added Comm close handler for the FTP data channel
This change has been merged. Previous status: Semi-approved For details, see: http://bundlebuggy.aaronbentley.com/project/squid/request/%3C1222064389.6395.868.camel%40pail%3E Project: Squid
Re: [RFC] 3.1 branching
On Wed, 2008-09-24 at 02:16 +1200, Amos Jeffries wrote: Lets set a date: 29 September? Well, I'll call for a partial hold right now. Please only commit bug fixes, minor cleanups, and features already on the roadmap for 3.1. The rest can be queued for commit post branch. 5 days to go and these bits left * connection pinning I thinking it's been outstanding long enough we should get it into 3.1, even if we have to hold PRE1 a week after the branch to stabilize it (or test it in PRE1 ?). Awaiting merge. (1-2 days). Do we go from trunk to PRE or from trunk to DEVEL? I do not think trunk code is stable enough to be called PRE at this point, so I was expecting DEVEL. Personally, I would just call it 3.1.0 and categorize it as development on the web site until we can move it into the stable category (after a few 3.1.x releases). * kerberos helper Markus has supplied another set of code, I assume its been tested externally and just needs a drop-in. Awaiting re-submission with some alterations. Non-blocker. * eCAP Awaiting merge submission. * source layout Started. Non-blocker. * source formatting scripts committed for use already, I was planning on starting the re-format and testbed run today here. Now waiting on completion of other feature code. Other misc stuff which qualifies but need some work if people have time: Awaiting a verify: http://www.squid-cache.org/bugs/show_bug.cgi?id=2433 Awaiting a verify test and debug: http://www.squid-cache.org/bugs/show_bug.cgi?id=2393 Awaiting a developer: * testbed level-02-maximus.opts opposite of minimal. Everything enabled, leave configure to handle clashes properly etc. Other important problems: http://www.squid-cache.org/bugs/show_bug.cgi?id=2459 http://www.squid-cache.org/bugs/show_bug.cgi?id=2460 I have asked somebody to work on these for the next 10 days, but I do not know if they will be able to fix them. I can work on them after eCAP. Also, it would be nice to review and commit TCP_RESET propagation patch: http://www.squid-cache.org/bugs/show_bug.cgi?id=2042 Cheers, Alex.
Re: [Fwd: Bundle Buggy: Voting error]
Talked with Aaron about this, and most likely the merge request did not have a proper public branch location and as result Bundlebuggy failed to select the proper project for the merge request. voting and approval is per merge request, and is governed by the project assigned to the merge... Regards Henrik tis 2008-09-23 klockan 00:35 +1200 skrev Amos Jeffries: I'm getting these with every bundlebuggy command sent via squid-dev, same as Alex mentioned. Amos e-postmeddelande-bilaga (Bundle Buggy: Voting error.eml) Vidarebefordrat meddelande Från: Bundle Buggy [EMAIL PROTECTED] Till: Amos Jeffries [EMAIL PROTECTED] Ämne: Bundle Buggy: Voting error Datum: Mon, 22 Sep 2008 08:09:55 -0400 (EDT)
[Success!] [MERGE] Address more comments from Amos.
This change has been merged. Previous status: Semi-approved For details, see: http://bundlebuggy.aaronbentley.com/project/squid/request/%3C200809222346.m8MNkYjC059268%40harfy.jeamland.net%3E Project: Squid
Re: /bzr/squid3/trunk/ r9220: Added Comm close handler for the data channel of FtpStateData
revno: 9220 committer: Alex Rousskov [EMAIL PROTECTED] branch nick: trunk timestamp: Tue 2008-09-23 08:49:50 -0600 message: Added Comm close handler for the data channel of FtpStateData transaction in preparation for officially dropping connect callbacks for closing descriptors. The data channel can be opened and closed a few times and the descriptor must be kept in sync with the close handler. I factored out the open/closing code into a simple FtpChannel class. That class is now used for both FTP control and data channels. The changes resolve one XXX discussion regarding FTP not having a close handler for the data channel. On the other hand, adding a second close handler attached to the same transaction is not a trivial change as the side-effects of Squid cleanup code are often illusive. For example, I suspect that FTP cleanup code does not close or even check the control channel. I added a DBG_IMPORTANT statement to test whether the control channel remains open. Or should that be an assert()? I think that only one out of the two callbacks can be dialed because the close handler executed first will invalidate the transaction object. FTP data channel can open close any time. It's close handler needs to only handle the fd, with no implications on the other FTP state. The FTP general close handler should close the control channel with a QUIT\0 or ABOR\0 (per RFC), and then close the data channel (discarding anything in-transit on ABOR, reading to end of current in buffer and closing on QUIT). So the shutdown procedure for FTP should be calling the control channel close handler and letting it handle the data channel cleanup. However, this would not play nicely on shutdown right now. Just on regular connection closes. FTP still needs a re-work cleanup at some later date which can do this sequence checking and fixup. Also to get rid of many global functions and do translations of generated pages properly from templates. Amos
Re: [RFC] 3.1 branching
On Wed, 2008-09-24 at 02:16 +1200, Amos Jeffries wrote: Lets set a date: 29 September? Well, I'll call for a partial hold right now. Please only commit bug fixes, minor cleanups, and features already on the roadmap for 3.1. The rest can be queued for commit post branch. 5 days to go and these bits left * connection pinning I thinking it's been outstanding long enough we should get it into 3.1, even if we have to hold PRE1 a week after the branch to stabilize it (or test it in PRE1 ?). Awaiting merge. (1-2 days). Do we go from trunk to PRE or from trunk to DEVEL? I do not think trunk code is stable enough to be called PRE at this point, so I was expecting DEVEL. Personally, I would just call it 3.1.0 and categorize it as development on the web site until we can move it into the stable category (after a few 3.1.x releases). i ws * kerberos helper Markus has supplied another set of code, I assume its been tested externally and just needs a drop-in. Awaiting re-submission with some alterations. Non-blocker. * eCAP Awaiting merge submission. * source layout Started. Non-blocker. * source formatting scripts committed for use already, I was planning on starting the re-format and testbed run today here. Now waiting on completion of other feature code. Other misc stuff which qualifies but need some work if people have time: Awaiting a verify: http://www.squid-cache.org/bugs/show_bug.cgi?id=2433 Awaiting a verify test and debug: http://www.squid-cache.org/bugs/show_bug.cgi?id=2393 Awaiting a developer: * testbed level-02-maximus.opts opposite of minimal. Everything enabled, leave configure to handle clashes properly etc. Other important problems: http://www.squid-cache.org/bugs/show_bug.cgi?id=2459 http://www.squid-cache.org/bugs/show_bug.cgi?id=2460 I have asked somebody to work on these for the next 10 days, but I do not know if they will be able to fix them. I can work on them after eCAP. Also, it would be nice to review and commit TCP_RESET propagation patch: http://www.squid-cache.org/bugs/show_bug.cgi?id=2042 Cheers, Alex.
Re: [Server-devel] Squid tuning recommendations for OLPC School Server tuning...
On Wed, Sep 24, 2008 at 12:08 AM, Henrik Nordstrom [EMAIL PROTECTED] wrote: On tis, 2008-09-23 at 14:57 +0800, Adrian Chadd wrote: I could probably do that in a week or so once I've finished my upcoming travel. Someone could try beating me to it.. The relevant code locations for implementing this if you want to take a stab at it yourself is the maintain function in each cache_dir type (src/fs/*/store_dir_...) Should be trivial to add a cache_dir parameter specifyung the max number of files in this cache_dir, and use this in the maintenance function. Good hint, thanks! If we did have such a control, what is the wired memory that squid will use for each entry? In an email earlier I wrote... - Each index entry takes between 56 bytes and 88 bytes, plus additional, unspecificed overhead. Is 1KB per entry a reasonable conservative estimate? - Discussions about compressing or hashing the URL in the index are recurrent - is the uncompressed URL there? That means up to 4KB per index entry? the notes I read about the index structure were rather old... m -- [EMAIL PROTECTED] [EMAIL PROTECTED] -- School Server Architect - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff
Re: [RFC] 3.1 branching
On Wed, 2008-09-24 at 02:16 +1200, Amos Jeffries wrote: Lets set a date: 29 September? Well, I'll call for a partial hold right now. Please only commit bug fixes, minor cleanups, and features already on the roadmap for 3.1. The rest can be queued for commit post branch. 5 days to go and these bits left * connection pinning I thinking it's been outstanding long enough we should get it into 3.1, even if we have to hold PRE1 a week after the branch to stabilize it (or test it in PRE1 ?). Awaiting merge. (1-2 days). Do we go from trunk to PRE or from trunk to DEVEL? I do not think trunk code is stable enough to be called PRE at this point, so I was expecting DEVEL. Personally, I would just call it 3.1.0 and categorize it as development on the web site until we can move it into the stable category (after a few 3.1.x releases). Before conn-pinning came up as an option I was thinking PRE. But it and the other things destabilize enough to warrant a DEVEL period at least. I'll decide after branching whether to release DEVEL, immediately or wait. Concentration on the branch blockers then we can split and put the new stuff and enhancements in trunk away from the hardening 3.1.x. * kerberos helper Markus has supplied another set of code, I assume its been tested externally and just needs a drop-in. Awaiting re-submission with some alterations. Non-blocker. * eCAP Awaiting merge submission. * source layout Started. Non-blocker. * source formatting scripts committed for use already, I was planning on starting the re-format and testbed run today here. Now waiting on completion of other feature code. Other misc stuff which qualifies but need some work if people have time: Awaiting a verify: http://www.squid-cache.org/bugs/show_bug.cgi?id=2433 Awaiting a verify test and debug: http://www.squid-cache.org/bugs/show_bug.cgi?id=2393 Awaiting a developer: * testbed level-02-maximus.opts opposite of minimal. Everything enabled, leave configure to handle clashes properly etc. Doing right now. Other important problems: http://www.squid-cache.org/bugs/show_bug.cgi?id=2459 http://www.squid-cache.org/bugs/show_bug.cgi?id=2460 I have asked somebody to work on these for the next 10 days, but I do not know if they will be able to fix them. I can work on them after eCAP. If they need a mentor I can probably assist in auditing 2459. Also, it would be nice to review and commit TCP_RESET propagation patch: http://www.squid-cache.org/bugs/show_bug.cgi?id=2042 Well, this is what the branch is for, so we can commit these 'nice' things to trunk for 3.2 from 30Sept on and start their testing as well without affecting the stability of 3.1. The non-major but important bugs can be fixed during DEVEL and PRE cycles. Branching is about features not bugs. Amos
Re: /bzr/squid3/trunk/ r9220: Added Comm close handler for the data channel of FtpStateData
On Wed, 2008-09-24 at 13:37 +1200, Amos Jeffries wrote: revno: 9220 committer: Alex Rousskov [EMAIL PROTECTED] branch nick: trunk timestamp: Tue 2008-09-23 08:49:50 -0600 message: Added Comm close handler for the data channel of FtpStateData transaction in preparation for officially dropping connect callbacks for closing descriptors. The data channel can be opened and closed a few times and the descriptor must be kept in sync with the close handler. I factored out the open/closing code into a simple FtpChannel class. That class is now used for both FTP control and data channels. The changes resolve one XXX discussion regarding FTP not having a close handler for the data channel. On the other hand, adding a second close handler attached to the same transaction is not a trivial change as the side-effects of Squid cleanup code are often illusive. For example, I suspect that FTP cleanup code does not close or even check the control channel. I added a DBG_IMPORTANT statement to test whether the control channel remains open. Or should that be an assert()? I think that only one out of the two callbacks can be dialed because the close handler executed first will invalidate the transaction object. FTP data channel can open close any time. Agreed. It's close handler needs to only handle the fd, with no implications on the other FTP state. Yes if the close handler purpose is to handle planned closing of the data channel. Not so sure if the handler purpose is to handle unexpected data channel closures only. Before the above change, there were no close handler for the data channel at all, so we can say that the old code did not want to cleanup during planned data channel closing or that the cleanup code was called before comm_close. Currently, the close handler for the data channel should only deal with unexpected closures. When the closure is unexpected, the current code will abort the entire FTP transaction. For planned closures, we remove the handler before closing so it is not involved, just like before. It is possible that a different overall design would be better, but I tried not to disturb the exiting code when addressing a specific issue (lack of a closing handler for comm connect). The FTP general close handler should close the control channel with a QUIT\0 or ABOR\0 (per RFC), and then close the data channel (discarding anything in-transit on ABOR, reading to end of current in buffer and closing on QUIT). I am not sure I follow. The close handler is called when the descriptor is closing so the handler cannot send anything on that channel. This is a low-level comm close handler, not some high-level quit nicely routine. So the shutdown procedure for FTP should be calling the control channel close handler and letting it handle the data channel cleanup. Yes, probably, but we may be talking about different handlers here. My commit message talks about close handlers and FTP cleanup code. Those are related but not the same. Transaction cleanup code is called when transaction ends or is aborted. The close handlers (if any) are called when a channel descriptor is closed. My commit comment mentions that the cleanup code does not seem to check whether the control channel is closed, which seems odd to me, but I could easily overlook something. Normally, the cleanup code should close all still-open descriptors owned by the transaction. FWIW, I tried not to change the normal shutdown procedure in FTP. If correct, my changes should affect unexpected data channel closures only. However, this would not play nicely on shutdown right now. Just on regular connection closes. FTP still needs a re-work cleanup at some later date which can do this sequence checking and fixup. Also to get rid of many global functions and do translations of generated pages properly from templates. I agree that we need to clean it up. IMO, it is not as bad as the client side though, so we may want to concentrate on that first if we cannot do it in parallel. Thank you, Alex.
Re: [Server-devel] Squid tuning recommendations for OLPC School Server tuning...
2008/9/24 Martin Langhoff [EMAIL PROTECTED]: Good hint, thanks! If we did have such a control, what is the wired memory that squid will use for each entry? In an email earlier I wrote... sizeof(StoreEntry) per index entry, basically. - Each index entry takes between 56 bytes and 88 bytes, plus additional, unspecificed overhead. Is 1KB per entry a reasonable conservative estimate? 1kb per entry is pretty conservative. The per-object overhead includes the StoreEntry, the couple of structures for the memory/disk replacement policies, plus the MD5 URL for the index hash, whatever other stuff hangs off MemObject for in-memory objects. You'll find that the RAM requirements grow a bit more for things like in-memory cache objects as the full reply headers stay in memory, and are copied whenever anyone wants to request it. - Discussions about compressing or hashing the URL in the index are recurrent - is the uncompressed URL there? That means up to 4KB per index entry? The uncompressed URL and headers are in memory during: * request/reply handling * in-memory object; (objects with MemObject's allocated); on-disk entries just have the MD5 URL hash per StoreEntry. HTH, Oh, and I'll be in the US from October for a few months; I can always do a side-trip out to see you guys if there's enough interest. Adrian
Re: [RFC] 3.1 branching
On Wed, 2008-09-24 at 14:34 +1200, Amos Jeffries wrote: Do we go from trunk to PRE or from trunk to DEVEL? I do not think trunk code is stable enough to be called PRE at this point, so I was expecting DEVEL. Personally, I would just call it 3.1.0 and categorize it as development on the web site until we can move it into the stable category (after a few 3.1.x releases). Before conn-pinning came up as an option I was thinking PRE. But it and the other things destabilize enough to warrant a DEVEL period at least. I'll decide after branching whether to release DEVEL, immediately or wait. Please email RFCs before making the final decision about the labels and releases. Until we get to a stable stage, the labeling and timing of the releases should be discussed as it can benefit from what others have seen in their tests. For example, IMO, the trunk was/is not PRE-ready even after the recent fixes and before conn-pinning. In general, I think we should virtually always do at least one development release before a PRE. Jumping from trunk to PRE is much more likely to discredit the branch than a development-to-PRE delay. PRE, to me, means we think it is stable, what do you think?. A development release, to me, means we are done adding features, please help us with testing and polishing. And yes, I know that the definitions at http://www.squid-cache.org/Versions/ are somewhat different. IIRC, I failed to convince others that mine are better :-) Concentration on the branch blockers then we can split and put the new stuff and enhancements in trunk away from the hardening 3.1.x. Other important problems: http://www.squid-cache.org/bugs/show_bug.cgi?id=2459 http://www.squid-cache.org/bugs/show_bug.cgi?id=2460 I have asked somebody to work on these for the next 10 days, but I do not know if they will be able to fix them. I can work on them after eCAP. If they need a mentor I can probably assist in auditing 2459. Thank you, I will keep that in mind! Also, it would be nice to review and commit TCP_RESET propagation patch: http://www.squid-cache.org/bugs/show_bug.cgi?id=2042 Well, this is what the branch is for, so we can commit these 'nice' things to trunk for 3.2 from 30Sept on and start their testing as well without affecting the stability of 3.1. I meant that it would be nice to include RESET propagation in v3.1 not v3.2 :-). Not a big deal though. The non-major but important bugs can be fixed during DEVEL and PRE cycles. Branching is about features not bugs. Agreed, except I do not think we should have any known important bugs when doing the first PRE (if we do PRE at all). Thank you, Alex.
