Re: squid-2.5 and coss
On Wed, Aug 27, 2003, Duane Wessels wrote: Duane, would you be willing to forward port your squid-2.5 COSS work to squid-3? I'm currently evaluating Squid-3 for a cache here and I'd like to test both epoll and COSS out. Done. I've committed the code to squid-3. It compiles, but I haven't tested it. Thanks, Rob, for the SwapDir options tutorial :-) Excellent. I'll test it now. Adrian
Re: squid-2.5 and coss
On Thu, Aug 21, 2003, Robert Collins wrote: On Thu, 2003-08-21 at 05:45, Duane Wessels wrote: FYI, my coss changes have not been commited to squid-3 yet because - I was going to wait until the time when it is okay to commit non-bugs to the tree. And on this note - anything that is ok to checkinto the Stable 2.5 tree is ok to commit into the 3.0 tree. It is feature-frozen, but by definition anything going into 2.5 is a bugfix :}. Yeah - Henrik pointed out that COSS, being experimental, is fine to muck around with in squid-3 as long as we don't need to touch anything else in the tree. Here's a fun suggestion from our friend Robert - how about creating a _directory_ instead of a file for the COSS cachedir? Then we can place the store logfile in there, the coss storefile in there _and_ any other metadata. It'd make life a whole lot easier and mean you won't get bitten by the requirement to specify swaplog paths.. Comments? adrian
Re: squid-2.5 and coss
Here's a fun suggestion from our friend Robert - how about creating a _directory_ instead of a file for the COSS cachedir? Then we can place the store logfile in there, the coss storefile in there _and_ any other metadata. It'd make life a whole lot easier and mean you won't get bitten by the requirement to specify swaplog paths.. The only reason not to do that is if we want to eventually support using raw devices/partitions for COSS storage. I think it was one of the original plans. However, to do that, the current code would need some enhancements to make sure that I/Os occur on 512-byte boundaries and are always a multiple of 512 in size. That might get around some 2GB file-size limits in some cases too. Performance-wise, I suspect it doesn't matter. Using the filesystem seems to give darn good throughput already. DW
Re: squid-2.5 and coss
Duane Wessels [EMAIL PROTECTED] writes: Here's a fun suggestion from our friend Robert - how about creating a _directory_ instead of a file for the COSS cachedir? Then we can place the store logfile in there, the coss storefile in there _and_ any other metadata. It'd make life a whole lot easier and mean you won't get bitten by the requirement to specify swaplog paths.. The only reason not to do that is if we want to eventually support using raw devices/partitions for COSS storage. I think it was one of the original plans. However, to do that, the current code would need some enhancements to make sure that I/Os occur on 512-byte boundaries and are always a multiple of 512 in size. That might get around some 2GB file-size limits in some cases too. Performance-wise, I suspect it doesn't matter. Using the filesystem seems to give darn good throughput already. In fact, it's a common trend in current RDBMs: they're starting to favour storage areas on fileystems rather than on raw devices. Linux has better support for filesystem-based raw operations (O_DIRECT) than to raw devices proper. BTW: O_DIRECT would be a good approach to our FS I/O work but: - we need to be able to keep an object hot after reading it - the OS won't do it for us anymore - it only supports reading and writing on page boundaries, typically 4kb (we'd need to be able to pad) I think it will be an interesting little project after Robert has completed his I/O refactoring work. -- kinkie (kinkie-squid [at] kinkie [dot] it) Random fortune, unrelated to the message: The first duty of a revolutionary is to get away with it. -- Abbie Hoffman
Re: squid-2.5 and coss
On Wednesday 27 August 2003 09.51, Kinkie wrote: BTW: O_DIRECT would be a good approach to our FS I/O work but: - we need to be able to keep an object hot after reading it - the OS won't do it for us anymore This is already on the todo, at least last time we discussed how Squid cache should be designed. The hot object cache should be separate from the transit objects, and include swapin objects. - it only supports reading and writing on page boundaries, typically 4kb (we'd need to be able to pad) Which should not be much of a problem I think.. Regards Henrik
Re: squid-2.5 and coss
On Wed, Aug 27, 2003, Henrik Nordstrom wrote: On Wednesday 27 August 2003 09.51, Kinkie wrote: BTW: O_DIRECT would be a good approach to our FS I/O work but: - we need to be able to keep an object hot after reading it - the OS won't do it for us anymore This is already on the todo, at least last time we discussed how Squid cache should be designed. The hot object cache should be separate from the transit objects, and include swapin objects. - it only supports reading and writing on page boundaries, typically 4kb (we'd need to be able to pad) Which should not be much of a problem I think.. Well, it'd be _wonderful_ to have the storage interface reworked a little so instead of working with fixed-size buffers we pass them a store mem chain and say do as much of this as you can and let me know... Then, for example, COSS reads would become instantly more efficient. adrian
Re: squid-2.5 and coss
On Wednesday 27 August 2003 10.41, Adrian Chadd wrote: Well, it'd be _wonderful_ to have the storage interface reworked a little so instead of working with fixed-size buffers we pass them a store mem chain and say do as much of this as you can and let me know... Then, for example, COSS reads would become instantly more efficient. I am still of the opinion that data should be passed around in refcounted buffers within Squid, created at the data source. I.e. network code creates the buffers for data received from network, store I/O creates the buffers for data received from disk, hot object cache gives the buffers of objects in the hot object cache.. (already created). This also allows for store I/O implementations to buffer as much data as it needs on writes without relying on the transit object and without any copying of data. Regards Henrik
Re: squid-2.5 and coss
On Wed, 2003-08-27 at 14:31, Duane Wessels wrote: Here's a fun suggestion from our friend Robert - how about creating a _directory_ instead of a file for the COSS cachedir? Then we can place the store logfile in there, the coss storefile in there _and_ any other metadata. It'd make life a whole lot easier and mean you won't get bitten by the requirement to specify swaplog paths.. The only reason not to do that is if we want to eventually support using raw devices/partitions for COSS storage. I think it was one of the original plans. Well, the specific file could be specified via a coss specific option. i.e. 'coss-file=', with the swapdir path used for the default location + the logs. Rob -- GPG key available at: http://members.aardvark.net.au/lifeless/keys.txt. signature.asc Description: This is a digitally signed message part
Re: squid-2.5 and coss
On Wed, 2003-08-27 at 18:41, Adrian Chadd wrote: Well, it'd be _wonderful_ to have the storage interface reworked a little so instead of working with fixed-size buffers we pass them a store mem chain and say do as much of this as you can and let me know... Then, for example, COSS reads would become instantly more efficient. Well, we don't have a 'chain' of store mem buffers anymore. It's now a random access no-contiguous splay. That forms the basis for range combining from upstream + store, and other such funky things. To separate out into hot cache and transit objects, refcounting is needed, along with a separation between in-creation and created buffers. Cheers, Rob -- GPG key available at: http://members.aardvark.net.au/lifeless/keys.txt. signature.asc Description: This is a digitally signed message part
Re: squid-2.5 and coss
Well, theoretically, I should be able to easily forwardport my COSS changes to 3.0 once I know it works. Believe me, once its running I'm going to be looking to move to 3.0. kqueue and epoll are my next thing to sell. :) Adrian, FYI, my coss changes have not been commited to squid-3 yet because - I was going to wait until the time when it is okay to commit non-bugs to the tree. - I haven't figured out the new magic OO way of doing cache_dir options yet. DW
Re: squid-2.5 and coss
On Thu, 2003-08-21 at 05:45, Duane Wessels wrote: FYI, my coss changes have not been commited to squid-3 yet because - I was going to wait until the time when it is okay to commit non-bugs to the tree. And on this note - anything that is ok to checkinto the Stable 2.5 tree is ok to commit into the 3.0 tree. It is feature-frozen, but by definition anything going into 2.5 is a bugfix :}. Cheers, Rob -- GPG key available at: http://members.aardvark.net.au/lifeless/keys.txt. signature.asc Description: This is a digitally signed message part
Re: squid-2.5 and coss
On Thu, 2003-08-14 at 15:54, Henrik Nordstrom wrote: On Thursday 14 August 2003 07.45, Adrian Chadd wrote: Well, theoretically, I should be able to easily forwardport my COSS changes to 3.0 once I know it works. Believe me, once its running I'm going to be looking to move to 3.0. kqueue and epoll are my next thing to sell. :) Not forwardport the changes to Squid-3 is plain stupid as it locks you in to Squid-2.5. It would be very undesireable from a release management perspective if COSS in Squid-2.5 is newer than COSS in Squid-3.0, and neither me or Robert will be very happy if this happens. Let me add: It will be a -lot- easier for you to: Get 3.0 stable on those uni caches. Get kqueue / epoll stable on those uni caches. Then get COSS stable on 3.0, than to: get COSS stable. get 3.0 stable get coss with 3.0 stable get coss with 3.0 stable with epoll/kqueue. But hey - it's your dime. Cheers, Rob -- GPG key available at: http://members.aardvark.net.au/lifeless/keys.txt. signature.asc Description: This is a digitally signed message part
Re: squid-2.5 and coss
On Wed, Aug 13, 2003, Adrian Chadd wrote: On Thu, Aug 14, 2003, Henrik Nordstrom wrote: It would be very undesireable from a release management perspective if COSS in Squid-2.5 is newer than COSS in Squid-3.0, and neither me or Robert will be very happy if this happens. Let me try to make it work first. heh. Actually, I've just hit a snag I should have remembered from a while ago. The objects don't actually have any size information stored in the metadata. Which makes sense at the moment, the store layer opens and outputs to files _before_ their size is absolutely known. There's no way to do a dirty rebuild unless: * we store the object size in the metadata, if its known (which it will be for COSS. heh.) * we try to add the list of object sizes at the front of the stripe. Now, the former might be neater but the latter could work out. The only problem is if there's not enough space to store the sizes of each of the objects. For example: a 1 megabyte COSS stripe will hold, on average, 85 objects (at 12k an object). 340 bytes will be needed to store 4-byte sizes for those 85 objects. I personally am doing all my tests with a 64k blocksize. Works a treat. We can always throw a warning if we can't pack all the object sizes into a block. However, I may need to store a byte of flags (especially if the object has been deleted) _AND_ I'm not entirely sure the current squid index system will be COSS-friendly (read: I may have to populate a stack of object sizes when the stripe is being filled and then handle the situation where an object is purged in the active stripe.) (Yes, another possibility is to actually store the metadata at the end of the stripe - if we start tracking how many objects are being stored in the stripe I can calculate how much space we need each time through storeCossAllocate() to store the objects AND all their state/offsets...) The second object is a bit hackish _BUT_ its just another verification technique. I also think it'd make the dirty rebuilds quite fast - especially if implementing using asyncio. Another issue - stale objects. We can't jump in and delete an object from a COSS stripe once its written to disk - the only time this happens is when it comes time to delete said stripe and all the objects (which translates to deleting from the tail of the LRU). If, at a later date, someone PURGEs an object it won't be deleted from the COSS store until its stripe is up for re-allocation. Obviously this will only occur during a dirty rebuild but its a real problem. (Hm, thinking about it, how about keeping a stack of deleted storeentry's and write it out as part of writing a stripe to disk? Then I'd know which entry to delete..) Comments? I'd like to start hacking on this tonight. adrian
Re: squid-2.5 and coss
On Thu, Aug 14, 2003, Henrik Nordstrom wrote: It would be very undesireable from a release management perspective if COSS in Squid-2.5 is newer than COSS in Squid-3.0, and neither me or Robert will be very happy if this happens. Let me try to make it work first. heh. Adrian
Re: squid-2.5 and coss
On Thu, Aug 14, 2003, Robert Collins wrote: Let me add: It will be a -lot- easier for you to: Get 3.0 stable on those uni caches. Get kqueue / epoll stable on those uni caches. Then get COSS stable on 3.0, than to: get COSS stable. get 3.0 stable get coss with 3.0 stable get coss with 3.0 stable with epoll/kqueue. Heh. I'm going to try out squid-3.0 soon and see how it fares. I'd _like_ to run it straight off, but then I'm not as familiar with the 3.0 internals and I kind of need to get something upgraded and stable soon. Once its upgraded and stable I'll be able to do some further work.. Adrian
Re: squid-2.5 and coss
On Thu, 2003-08-14 at 16:31, Adrian Chadd wrote: There's no way to do a dirty rebuild unless: Easy answer: a dirty coss rebuild is simply to purge the stripes that are dirty. Why make the common case difficult to deal with a corner case? Long term, we can give coss some real power, once we finish the async key lookups begun in 3.0, as coss then won't need any in-core index - and the other things you discuss can be done as part of the coss disk index lookup logic. Cheers, Rob -- GPG key available at: http://members.aardvark.net.au/lifeless/keys.txt. signature.asc Description: This is a digitally signed message part
Re: squid-2.5 and coss
On Thursday 14 August 2003 07.14, Adrian Chadd wrote: I'd like to commit the assertions and reservations to 2.5. I've run them through many many tests and it seems to be just as stable as the existing COSS code. OK for me. COSS is an experimental feature and not under code freeze even for Squid-2.5. Just remember to update the patches page after commit. If you need to touch any files outside of src/fs/coss/ then stricter rules apply. But my back of the head says why is this development not done in Squid-3? We need to start moving development to Squid-3 soon or Squid-3.0 will never receive any good amount of testing. If there is customers involved who require Squid-2.5 then sure, 2.5 it is. If not I see no reason why making anything in 2.5 beyond fixing bugs to non-experimental features. Regards Henrik
Re: squid-2.5 and coss
On Thu, Aug 14, 2003, Henrik Nordstrom wrote: OK for me. COSS is an experimental feature and not under code freeze even for Squid-2.5. Just remember to update the patches page after commit. I'll do that once I've finished my rebuild code. If you need to touch any files outside of src/fs/coss/ then stricter rules apply. I won't be. Pwomise. But my back of the head says why is this development not done in Squid-3? We need to start moving development to Squid-3 soon or Squid-3.0 will never receive any good amount of testing. Because I want to deploy this as an easily reversable change on a university cache or two. If there is customers involved who require Squid-2.5 then sure, 2.5 it is. If not I see no reason why making anything in 2.5 beyond fixing bugs to non-experimental features. Well, theoretically, I should be able to easily forwardport my COSS changes to 3.0 once I know it works. Believe me, once its running I'm going to be looking to move to 3.0. kqueue and epoll are my next thing to sell. :) adrian
squid-2.5 and coss
hi all, I've recently been working a little more on COSS. Thanks to Duane and his blocksize work its actually reasonably useful now. I'm having issues tuning it (since it does waste quite a bit of space, eep!) _but_ the performance increases make up for it. I've got some initial patches which include a bunch of sanity asserts() and reserving a (blocksize) block near the beginning of each stripe. I'm planning on storing rebuild-relevant information (checksum, timestamp, etc) in there. I'd like to commit the assertions and reservations to 2.5. I've run them through many many tests and it seems to be just as stable as the existing COSS code. Any comments? Adrian
Re: squid-2.5 and coss
On Thu, Aug 14, 2003, Robert Collins wrote: There are multiple stripes per coss 'dir' right? So, perhaps you write a tag records to the swaplog for the coss dir after the normal records, listing the status of the stripes? Thats what I've been saying, but there's a few gotchas which I've covered. I think, however, that having COSS work well enough for normal web caches in 2.5 would be a big and useful thing. Hmm, I'm not so sure of that. In 2.5 COSS is marked as experimental - and I'd hesitate to be putting substantial changes into a stable release. Folk that are tracking stable don't expect non-bugfix commits... because they are often in change controlled environments. Thats a good point, but then, they can just apply the patches from the website. If I'm not _completely_ sure I get it right, I'm not going to commit it. Adrian
Re: squid-2.5 and coss
On Thu, Aug 14, 2003, Robert Collins wrote: On Thu, 2003-08-14 at 16:31, Adrian Chadd wrote: There's no way to do a dirty rebuild unless: Easy answer: a dirty coss rebuild is simply to purge the stripes that are dirty. Well, at the moment, there's _no_ dirty rebuild. :) Why make the common case difficult to deal with a corner case? Because, right now, there's no rebuild and there's no actual way to know whether a stripe is valid or not. Then, there's no way to know what the object sizes are and if they're deleted after being written to disk there's no way to actually _know_ they've been deleted. If they've been replaced and the replacement is in a future stripe the object will jus tbe replaced on rebuild, - squid will do that for us. But, if someone PURGE's an object or series of objects and then does a dirty shutdown the objects will appear again unless we _store_ the deletes for that storefs. Now, I don't know whether that really matters much - this is just a cache, and manual PURGE's won't happen often - but still... Long term, we can give coss some real power, once we finish the async key lookups begun in 3.0, as coss then won't need any in-core index - and the other things you discuss can be done as part of the coss disk index lookup logic. Oh, when we get the async key lookups finished and a few other changes occur I'm sure some real magic can happen. I think, however, that having COSS work well enough for normal web caches in 2.5 would be a big and useful thing. Adrian
