On Wednesday 06 May 2009 22:37:17 Juiceman wrote: > On Wed, May 6, 2009 at 1:09 PM, Matthew Toseland > <toad at amphibian.dyndns.org> wrote: > > On Wednesday 06 May 2009 17:34:41 Juiceman wrote: > >> On Wed, May 6, 2009 at 11:29 AM, Matthew Toseland > >> <toad at amphibian.dyndns.org> wrote: > >> > IMHO the number of disk reads is simply a function of how much memory > > there > >> > is, how much can be used to cache the node.db4o by the operating system. > > We > >> > can do more caching at the JVM level, this will reduce CPU usage but not > >> > actual IOs hitting the disk, and will cost more heap memory. > >> > > >> > But we can reduce the number of disk writes further for downloads: > >> > > >> > Turn off scheduling by retry count. Schedule solely by priority then > >> > round-robin over the different request clients and requests then randomly > >> > over the individual SendableGet's: as we do now, but without the retry > > count > >> > layer. > >> > > >> > Only update the retry count for an individual block if MaxRetries != -1. > >> > > >> > Keep the cooldown queue entirely in RAM. Two structures so that we can > > select > >> > requests quickly: > >> > > >> > 1) A structure tracking all the keys fetched in the last half hour. For > > each > >> > key, the 3 times at which it was last fetched. When selecting requests, we > >> > would not send any request for a key in this list - at least not for > >> > persistent requests. > >> > > >> > 2) A structure tracking all the BaseSendableGet's sent in the last half > > hour. > >> > For each, record the start and finish times for the last 3 times it was > >> > fetched. When selecting requests, we can shortcut having to ask the first > >> > structure for each block by excluding a whole BaseSendableGet through this > >> > structure. > >> > > >> > Memory cost? Well, we do actually keep the first structure already, in the > >> > FailureTable ... some changes would be needed perhaps. The second > > structure > >> > would be new, and somewhat problematic, as it would keep these objects in > >> > memory even if they are deactivated. We could introduce a unique > > identifier > >> > or even use db4o's object IDs to avoid this (probably the best solution). > > A > >> > simple hashset of the objects themselves probably would not work well for > >> > activation reasons. > >> > > >> > Network performance cost? Well, what would be the impact of not scheduling > > by > >> > retry count any more? I dunno, any opinions? > >> > > >> > CPU cost? We might spend more time trying to find requests to run ... this > >> > might even result in more disk reads ... ? But on the whole it ought to be > >> > positive ... > >> > > >> > Code changes? Well, we could keep SplitFileFetcherSubSegment, and be > > backward > >> > compatible, it'd be a bit awkward though... eventually we'd want a new > > class > >> > merging SFFS and SFFSS... > >> > >> RAM is cheap nowadays and if we make this an option then low memory > >> nodes can still trade disk io for RAM. > > > > No, it cannot be optional. These sorts of architectural changes are big enough > > that they are either part of the code or not. > > > >> I give Freenet 512MB and don't > >> miss it. ?The disk io, however is unacceptable and causes me to only > >> run Freenet when I am not using my PC since db4o was merged... > > > > And I do not yet understand why. As I asked you on the support list, are you > > doing any *up*loads, or is it all downloads? > > > > No uploads.
And you have plenty of RAM ... not sure what's going on then. This severe disk I/O is continual? There are phases when it might be expected e.g. when starting a big download... Please get some more information by doing the following (this will increase logging a bit, but hopefully not dramatically): Set the log level details to freenet.support.PrioritizedSerialExecutor:MINOR Let the node run for an hour or so. Send me your statistics page and the contents of your last log (maybe narrow it down by grep'ing for PrioritizedSerialExecutor, hopefully there shouldn't be any keys or anything on the output). Am I correct in assuming you are using Windows XP 32-bit? -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 835 bytes Desc: This is a digitally signed message part. URL: <https://emu.freenetproject.org/pipermail/devl/attachments/20090507/3e2c8a47/attachment.pgp>
