Re: [squid-users] caching behavior during COSS rebuild
Chris Woodfield wrote: Just tested this - -F appears to work for aufs rebuilds but not COSS rebuilds. To reproduce: - Config squid with an aufs and a coss store, like so: cache_dir aufs /usr/squidcache 5000 16 256 min-size=50 cache_dir coss /usr/squidcache/coss1.dat 500 block-size=4096 max-size=50 membufs=100 cache_swap_log /usr/squidcache/%s.swap - Start squid, send it a bunch of queries (I use a script that grabs random URLs from recent access.log files) - Stop squid, delete /usr/squidcache/usr.squidcache.swap - Start sending squid requests (again, the random URL script above) - Restart squid with -F What I'm seeing when I do the above is that as soon as the AUFS stores finish rebuilding, squid starts answering queries - but the COSS isn't rebuilt yet, and until that completes, all objects 500K are cache misses (SO_FAIL in store.log). Will get a bugzilla entry in place for this. Aha thank you. Methinks this has more to do with the global flag indicating store rebuild in progress is being unset by the first store type to finish. :( Squid needs a set of per-store flags and a function to test whether any stores are still pending (temporary until the per-store rework is done). That would solve several other rebuild related bugs at the same time. Amos -C On Apr 22, 2009, at 10:00 AM, Chris Woodfield wrote: ...and sure enough, it's right there in -h output... cache$ /usr/local/squid/sbin/squid -h ... -FDon't serve any requests until store is rebuilt. ... /me goes to write I will RTFM Before Posting To squid-users 100 times on the whiteboard... :) -C On Apr 22, 2009, at 9:56 AM, Adrian Chadd wrote: Well, I killed the swaplog writing entirely in Lusca - the COSS rebuild code doesn't read from it (it was broken for various reasons revolving mostly around code bitrot IIRC.) There's a flag you can pass Squid to not handle requests until the store is rebuilt - its the -F flag. I'm fixing the store rebuild times in Lusca-HEAD at the moment and this includes writing some new COSS rebuild-from-index, rebuild-from-log and rebuild-from-rawdevice tools. Adrian On Wed, Apr 22, 2009, Chris Woodfield wrote: On Apr 22, 2009, at 4:56 AM, Amos Jeffries wrote: Chris Woodfield wrote: So I'm running with COSS under 2.7STABLE6, we've noticed (as I can see others have, teh Googles tell me so) that the COSS rebuild a. happens every time squid is restarted, and b. takes quite a while if the COSS stripes are large. However, I've noticed that while the stripes are being rebuilt, squid still listens for and handles requests - it just SO_FAILs on every object that would normally get saved to a COSS stripe. So much for that hit ratio. SO - the questions are: 1. Is there *any* way to prevent the COSS rebuild if squid is terminated normally? The indexes are stored in swap.state. Check that it is being done properly by your Squid. This could be the issue - when I exit squid, I notice that my $coss_file.dat and $coss_file.dat.last-clean files all have zero size. Any idea why this might be happening? The relevant section of our squid.conf reads as follows: cache_dir aufs /usr/squidcache.0/cache/ 75 16 256 min-size=100 cache_dir coss /usr/squidcache.0/cache/coss1.dat 3 block-size=4096 max-size=100 membufs=100 cache_dir coss /usr/squidcache.0/cache/coss2.dat 3 block-size=4096 max-size=100 membufs=100 cache_dir coss /usr/squidcache.0/cache/coss3.dat 3 block-size=4096 max-size=100 membufs=100 cache_swap_log /usr/squidcache.0/cache/%s Thanks, -C 2. Is there a way to prevent squid from handling requests until the COSS stripe is fully rebuilt (this is obviously not good if you don't have redundant squids, but that's not a problem for us) ? I believe its possible. If its not a local failure to find swap.state for the COSS dir then it will be a code fix. Unfortunately we developers are no longer actively working on Squid-2 without a paid support contract. Also Adrian our storage expert who would be the best to ask has retired from active alterations. Amos -- Please be using Current Stable Squid 2.7.STABLE6 or 3.0.STABLE14 Current Beta Squid 3.1.0.7 -- - Xenion - http://www.xenion.com.au/ - VPS Hosting - Commercial Squid Support - - $25/pm entry-level VPSes w/ capped bandwidth charges available in WA - -- Please be using Current Stable Squid 2.7.STABLE6 or 3.0.STABLE14 Current Beta Squid 3.1.0.7
Re: [squid-users] caching behavior during COSS rebuild
Chris Woodfield wrote: So I'm running with COSS under 2.7STABLE6, we've noticed (as I can see others have, teh Googles tell me so) that the COSS rebuild a. happens every time squid is restarted, and b. takes quite a while if the COSS stripes are large. However, I've noticed that while the stripes are being rebuilt, squid still listens for and handles requests - it just SO_FAILs on every object that would normally get saved to a COSS stripe. So much for that hit ratio. SO - the questions are: 1. Is there *any* way to prevent the COSS rebuild if squid is terminated normally? The indexes are stored in swap.state. Check that it is being done properly by your Squid. 2. Is there a way to prevent squid from handling requests until the COSS stripe is fully rebuilt (this is obviously not good if you don't have redundant squids, but that's not a problem for us) ? I believe its possible. If its not a local failure to find swap.state for the COSS dir then it will be a code fix. Unfortunately we developers are no longer actively working on Squid-2 without a paid support contract. Also Adrian our storage expert who would be the best to ask has retired from active alterations. Amos -- Please be using Current Stable Squid 2.7.STABLE6 or 3.0.STABLE14 Current Beta Squid 3.1.0.7
Re: [squid-users] caching behavior during COSS rebuild
On Apr 22, 2009, at 4:56 AM, Amos Jeffries wrote: Chris Woodfield wrote: So I'm running with COSS under 2.7STABLE6, we've noticed (as I can see others have, teh Googles tell me so) that the COSS rebuild a. happens every time squid is restarted, and b. takes quite a while if the COSS stripes are large. However, I've noticed that while the stripes are being rebuilt, squid still listens for and handles requests - it just SO_FAILs on every object that would normally get saved to a COSS stripe. So much for that hit ratio. SO - the questions are: 1. Is there *any* way to prevent the COSS rebuild if squid is terminated normally? The indexes are stored in swap.state. Check that it is being done properly by your Squid. This could be the issue - when I exit squid, I notice that my $coss_file.dat and $coss_file.dat.last-clean files all have zero size. Any idea why this might be happening? The relevant section of our squid.conf reads as follows: cache_dir aufs /usr/squidcache.0/cache/ 75 16 256 min-size=100 cache_dir coss /usr/squidcache.0/cache/coss1.dat 3 block-size=4096 max-size=100 membufs=100 cache_dir coss /usr/squidcache.0/cache/coss2.dat 3 block-size=4096 max-size=100 membufs=100 cache_dir coss /usr/squidcache.0/cache/coss3.dat 3 block-size=4096 max-size=100 membufs=100 cache_swap_log /usr/squidcache.0/cache/%s Thanks, -C 2. Is there a way to prevent squid from handling requests until the COSS stripe is fully rebuilt (this is obviously not good if you don't have redundant squids, but that's not a problem for us) ? I believe its possible. If its not a local failure to find swap.state for the COSS dir then it will be a code fix. Unfortunately we developers are no longer actively working on Squid-2 without a paid support contract. Also Adrian our storage expert who would be the best to ask has retired from active alterations. Amos -- Please be using Current Stable Squid 2.7.STABLE6 or 3.0.STABLE14 Current Beta Squid 3.1.0.7
Re: [squid-users] caching behavior during COSS rebuild
...and sure enough, it's right there in -h output... cache$ /usr/local/squid/sbin/squid -h ... -FDon't serve any requests until store is rebuilt. ... /me goes to write I will RTFM Before Posting To squid-users 100 times on the whiteboard... :) -C On Apr 22, 2009, at 9:56 AM, Adrian Chadd wrote: Well, I killed the swaplog writing entirely in Lusca - the COSS rebuild code doesn't read from it (it was broken for various reasons revolving mostly around code bitrot IIRC.) There's a flag you can pass Squid to not handle requests until the store is rebuilt - its the -F flag. I'm fixing the store rebuild times in Lusca-HEAD at the moment and this includes writing some new COSS rebuild-from-index, rebuild-from- log and rebuild-from-rawdevice tools. Adrian On Wed, Apr 22, 2009, Chris Woodfield wrote: On Apr 22, 2009, at 4:56 AM, Amos Jeffries wrote: Chris Woodfield wrote: So I'm running with COSS under 2.7STABLE6, we've noticed (as I can see others have, teh Googles tell me so) that the COSS rebuild a. happens every time squid is restarted, and b. takes quite a while if the COSS stripes are large. However, I've noticed that while the stripes are being rebuilt, squid still listens for and handles requests - it just SO_FAILs on every object that would normally get saved to a COSS stripe. So much for that hit ratio. SO - the questions are: 1. Is there *any* way to prevent the COSS rebuild if squid is terminated normally? The indexes are stored in swap.state. Check that it is being done properly by your Squid. This could be the issue - when I exit squid, I notice that my $coss_file.dat and $coss_file.dat.last-clean files all have zero size. Any idea why this might be happening? The relevant section of our squid.conf reads as follows: cache_dir aufs /usr/squidcache.0/cache/ 75 16 256 min- size=100 cache_dir coss /usr/squidcache.0/cache/coss1.dat 3 block- size=4096 max-size=100 membufs=100 cache_dir coss /usr/squidcache.0/cache/coss2.dat 3 block- size=4096 max-size=100 membufs=100 cache_dir coss /usr/squidcache.0/cache/coss3.dat 3 block- size=4096 max-size=100 membufs=100 cache_swap_log /usr/squidcache.0/cache/%s Thanks, -C 2. Is there a way to prevent squid from handling requests until the COSS stripe is fully rebuilt (this is obviously not good if you don't have redundant squids, but that's not a problem for us) ? I believe its possible. If its not a local failure to find swap.state for the COSS dir then it will be a code fix. Unfortunately we developers are no longer actively working on Squid-2 without a paid support contract. Also Adrian our storage expert who would be the best to ask has retired from active alterations. Amos -- Please be using Current Stable Squid 2.7.STABLE6 or 3.0.STABLE14 Current Beta Squid 3.1.0.7 -- - Xenion - http://www.xenion.com.au/ - VPS Hosting - Commercial Squid Support - - $25/pm entry-level VPSes w/ capped bandwidth charges available in WA -
Re: [squid-users] caching behavior during COSS rebuild
Just tested this - -F appears to work for aufs rebuilds but not COSS rebuilds. To reproduce: - Config squid with an aufs and a coss store, like so: cache_dir aufs /usr/squidcache 5000 16 256 min-size=50 cache_dir coss /usr/squidcache/coss1.dat 500 block-size=4096 max- size=50 membufs=100 cache_swap_log /usr/squidcache/%s.swap - Start squid, send it a bunch of queries (I use a script that grabs random URLs from recent access.log files) - Stop squid, delete /usr/squidcache/usr.squidcache.swap - Start sending squid requests (again, the random URL script above) - Restart squid with -F What I'm seeing when I do the above is that as soon as the AUFS stores finish rebuilding, squid starts answering queries - but the COSS isn't rebuilt yet, and until that completes, all objects 500K are cache misses (SO_FAIL in store.log). Will get a bugzilla entry in place for this. -C On Apr 22, 2009, at 10:00 AM, Chris Woodfield wrote: ...and sure enough, it's right there in -h output... cache$ /usr/local/squid/sbin/squid -h ... -FDon't serve any requests until store is rebuilt. ... /me goes to write I will RTFM Before Posting To squid-users 100 times on the whiteboard... :) -C On Apr 22, 2009, at 9:56 AM, Adrian Chadd wrote: Well, I killed the swaplog writing entirely in Lusca - the COSS rebuild code doesn't read from it (it was broken for various reasons revolving mostly around code bitrot IIRC.) There's a flag you can pass Squid to not handle requests until the store is rebuilt - its the -F flag. I'm fixing the store rebuild times in Lusca-HEAD at the moment and this includes writing some new COSS rebuild-from-index, rebuild- from-log and rebuild-from-rawdevice tools. Adrian On Wed, Apr 22, 2009, Chris Woodfield wrote: On Apr 22, 2009, at 4:56 AM, Amos Jeffries wrote: Chris Woodfield wrote: So I'm running with COSS under 2.7STABLE6, we've noticed (as I can see others have, teh Googles tell me so) that the COSS rebuild a. happens every time squid is restarted, and b. takes quite a while if the COSS stripes are large. However, I've noticed that while the stripes are being rebuilt, squid still listens for and handles requests - it just SO_FAILs on every object that would normally get saved to a COSS stripe. So much for that hit ratio. SO - the questions are: 1. Is there *any* way to prevent the COSS rebuild if squid is terminated normally? The indexes are stored in swap.state. Check that it is being done properly by your Squid. This could be the issue - when I exit squid, I notice that my $coss_file.dat and $coss_file.dat.last-clean files all have zero size. Any idea why this might be happening? The relevant section of our squid.conf reads as follows: cache_dir aufs /usr/squidcache.0/cache/ 75 16 256 min- size=100 cache_dir coss /usr/squidcache.0/cache/coss1.dat 3 block- size=4096 max-size=100 membufs=100 cache_dir coss /usr/squidcache.0/cache/coss2.dat 3 block- size=4096 max-size=100 membufs=100 cache_dir coss /usr/squidcache.0/cache/coss3.dat 3 block- size=4096 max-size=100 membufs=100 cache_swap_log /usr/squidcache.0/cache/%s Thanks, -C 2. Is there a way to prevent squid from handling requests until the COSS stripe is fully rebuilt (this is obviously not good if you don't have redundant squids, but that's not a problem for us) ? I believe its possible. If its not a local failure to find swap.state for the COSS dir then it will be a code fix. Unfortunately we developers are no longer actively working on Squid-2 without a paid support contract. Also Adrian our storage expert who would be the best to ask has retired from active alterations. Amos -- Please be using Current Stable Squid 2.7.STABLE6 or 3.0.STABLE14 Current Beta Squid 3.1.0.7 -- - Xenion - http://www.xenion.com.au/ - VPS Hosting - Commercial Squid Support - - $25/pm entry-level VPSes w/ capped bandwidth charges available in WA -
[squid-users] caching behavior during COSS rebuild
So I'm running with COSS under 2.7STABLE6, we've noticed (as I can see others have, teh Googles tell me so) that the COSS rebuild a. happens every time squid is restarted, and b. takes quite a while if the COSS stripes are large. However, I've noticed that while the stripes are being rebuilt, squid still listens for and handles requests - it just SO_FAILs on every object that would normally get saved to a COSS stripe. So much for that hit ratio. SO - the questions are: 1. Is there *any* way to prevent the COSS rebuild if squid is terminated normally? 2. Is there a way to prevent squid from handling requests until the COSS stripe is fully rebuilt (this is obviously not good if you don't have redundant squids, but that's not a problem for us) ? Thanks, -C
