Re: [Server-devel] Squid tuning recommendations for OLPC School Server tuning...
On Wed, Sep 24, 2008 at 12:08 AM, Henrik Nordstrom [EMAIL PROTECTED] wrote: On tis, 2008-09-23 at 14:57 +0800, Adrian Chadd wrote: I could probably do that in a week or so once I've finished my upcoming travel. Someone could try beating me to it.. The relevant code locations for implementing this if you want to take a stab at it yourself is the maintain function in each cache_dir type (src/fs/*/store_dir_...) Should be trivial to add a cache_dir parameter specifyung the max number of files in this cache_dir, and use this in the maintenance function. Good hint, thanks! If we did have such a control, what is the wired memory that squid will use for each entry? In an email earlier I wrote... - Each index entry takes between 56 bytes and 88 bytes, plus additional, unspecificed overhead. Is 1KB per entry a reasonable conservative estimate? - Discussions about compressing or hashing the URL in the index are recurrent - is the uncompressed URL there? That means up to 4KB per index entry? the notes I read about the index structure were rather old... m -- [EMAIL PROTECTED] [EMAIL PROTECTED] -- School Server Architect - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff
Squid tuning recommendations for OLPC School Server tuning...
Hi! I am working on the School Server (aka XS: a Fedora 9 spin, tailored to run on fairly limited hw), I'm preparing the configuration settings for it. It's a somewhat new area for me -- I've setup Squid before on mid-range hardware... but this is... different. So I'm interested in understanding more aobut the variables affecting memory footprint and how I can set a _hard limit_ on the wired memory that squid allocates. In brief: - The workload is relatively light - 3K clients is the upper bound. - The XS will (in some locations) be hooked to *very* unreliable power... uncontrolled shutdowns are the norm. Is this ever a problem with Squid? - After a bad shutdown, graceful recovery is the most important aspect. If a few cached items are lost, we can cope... - The XS hardware runs many services (mostly webbased), so Squid gets only a limited slice of memory. To make matters worse, I *really* don't want the core working set (Squid, Pg, Apache/PHP) to get paged out. So I am interested in pegging the max memory Squid will take to itself. - The XS hw is varied. In small schools it may have 256MB RAM (likely to be running on XO hardware + usb-connected ext hard-drive). Medium-to-large schools will have the recommended 1GB RAM and a cheap SATA disk. A few very large schools will be graced with more RAM (2 or 4GB). ... so RAM allocation for Squid will prob range between 24MB at the lower-end and 96MB at the 1GB recommended RAM. My main question is: how would you tune Squid 3 so that - it does not allocate directly more than 24MB / 96MB? (Assume that the linux kernel will be smart about mmapped stuff, and aggressive about caching -- I am talking about the memory Squid will claim to itself). - still gives us good thoughput? :-) So far Google has turned up very little info, and it seems to be rather old. What I've found can be summarised as follows: - The index is malloc'd, so the number of entries in the index will be the dominant concern WRT memory footprint. - Each index entry takes between 56 bytes and 88 bytes, plus additional, unspecificed overhead. Is 1KB per entry a reasonable conservative estimate? - Discussions about compressing or hashing the URL in the index are recurrent - is the uncompressed URL there? That means up to 4KB per index entry? - The index does nto seem to be mmappable or otherwise We can rely on the (modern) linux kernel doing a fantastic job at caching disk IO and shedding those cached entries when under memory pressure, so I am likely to set Squid's own cache to something really small. Everything I read points to the index being my main concern - is there a way to limit (a) the total memory the index is allowed to take or (b) the number of index entries allowed? Does the above make sense in general? Or am I barking up the wrong tree? cheers, martin -- [EMAIL PROTECTED] [EMAIL PROTECTED] -- School Server Architect - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff
Re: Squid tuning recommendations for OLPC School Server tuning...
On Tue, Sep 23, 2008 at 11:24 AM, Martin Langhoff [EMAIL PROTECTED] wrote: Does the above make sense in general? Or am I barking up the wrong tree? The current squid config file is here - I have not done any tuning on it at all: http://dev.laptop.org/git?p=projects/xs-config;a=blob;f=altfiles/etc/squid/squid-xs.conf.in;h=959e720db514325a557a3bce171dc1eed98bca2b;hb=378f568f9c8a4d2c7f707ab5de5528008cc053d0 cheers, martin -- [EMAIL PROTECTED] [EMAIL PROTECTED] -- School Server Architect - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff
Re: [Server-devel] Squid tuning recommendations for OLPC School Server tuning...
On Tue, Sep 23, 2008 at 3:09 PM, Adrian Chadd [EMAIL PROTECTED] wrote: I've looked into this a bit (and have a couple of OLPC laptops to do testing with) and .. well, its going to take a bit of effort to make squid fit. Any way we can kludge our way around it for the time being? Does squid take any signal that gets it to shed its index? There's no hard limit for squid and squid (any version) handles memory allocation failures very very poorly (read: crashes.) Is it relatively sane to run it with a tight rlimit and restart it often? Or just monitor it and restart it? You can limit the amount of cache_mem which limits the memory cache size; you could probably modify the squid codebase to start purging objects at a certain object count rather than based on the disk+memory storage size. That wouldn't be difficult. Any chance of having patches that do this? The big problem: you won't get Squid down to 24meg of RAM with the current tuning parameters. Well, I couldn't; and I'm playing around Hmmm... with Squid on OLPC-like hardware (SBC with 500mhz geode, 256/512mb RAM.) Its something which will require quite a bit of development to slim some of the internals down to scale better with restricted memory footprints. Its on my personal TODO list (as it mostly is in line with a bunch of performance work I'm slowly working towards) but as the bulk of that is happening in my spare time, I do not have a fixed timeframe at the moment. Thanks for that -- at whatever pace, progress is progress. I'll stay tuned. I'm not on squid-devel, but generally interested in any news on this track; I'll be thankful if you CC me or rope me into relevant threads. Is there interest within the squid dev team in moving towards a memory allocation model that is more tunable and/or relies more on the abilities of modern kernels to do memory mgmt? Or an alternative approach to handle scalability (both down to small devices and up to huge kit) more dynamically and predictably? cheers, m -- [EMAIL PROTECTED] [EMAIL PROTECTED] -- School Server Architect - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff
Re: [Server-devel] Squid tuning recommendations for OLPC School Server tuning...
On Tue, Sep 23, 2008 at 4:12 PM, Mark Nottingham [EMAIL PROTECTED] wrote: Overall, what do you want to use Squid for here; caching, access control..? Caching and plugins such as squidgard (does that qualify as access control?) If you want caching, realise that you're not going to see much benefit from such a resource-limited box, and indeed it may be more of a bottleneck than is worthwhile. well, we are very constrained RAM wise, but we have a reasonable hard drive quota *and* a horrible internet connection. Picture 200 kids behind a dsl line, 50 kids behind a satellite link or 3G modem. Granted, youtube isn't going to work but well-behaved cacheable content (in http terms) can work well with a good proxy. cheers, martin -- [EMAIL PROTECTED] [EMAIL PROTECTED] -- School Server Architect - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff
