On Wednesday 13 February 2008 18:41, Jerome Flesch wrote: > > After spending some time debating how to implement a client API > > for ULPRs, I propose the following: > > > > Every key queued by the client layer (from FCP or other sources), will be > > tried at most N times every M minutes. N is configurable, but M is not > > (well it could be configurable per node but not per request). I suggest a > > default N of 3 and M of 30. At the moment, if a key is queued at > > MaxRetries=-1, and no other keys are queued, it will be retried forever > > every few seconds. > > > > This eliminates the need to have any other FCP API for ULPRs: all a polling > > client writer needs to do is request the key with MaxRetries=-1, and > > Freenet will request it every so often, unless the client disconnects. > > Obviously we would need a way to kill a non-persistent request as well, but > > that's not difficult. > > > > The basic problem here is that ULPRs may find the key at a time when the > > client is not actually requesting it i.e. between explicit polling. We need > > the node to know that the client wants the key. > > > > Do we need an explicit subscribe-without-requesting message, or would the > > above be sufficient? Even with the above, if a client asked explicitly for > > an update, the client could cancel the request and start a new one. Also, a > > subscription without a request is of very limited value: no node will know > > we want it unless we request it. > > > Yes, I think an explicit subscribe-without-requesting message would be useful. > Because I can't imagine myself starting requests on all the indexes or on all > the trust lists of the WoT I'm implementing.
Why not? You will have to request them at some point. > At the moment, I will use SubscribeUSK (with DontPoll=true), but I think a > more general message (able to work on KSK etc using ULPRs) would be nice. Any such subscription would have to be a full request, with a flag to not initiate any requests. It would have the memory overhead of a full request, need to be cancelable, and have more or less the same FCP API. The reason for this is simply that it must work for any key, and most keys require multiple low level blocks to be fetched. So the first thing I thought of was to have a request which if it can't find the data in N retries, it goes dormant - passively waiting for the key. However, ULPRs are unreliable, and the client would have to renew it regularly; it's a waste of memory and CPU. So wouldn't it be better to simply automatically retry every 30 minutes (subject to whatever else is queued) ? The other really neat thing about it is we would no longer continually request the same key over and over if it's the only thing on the queue: we'd throttle it at source, in the client layer, rather than having to repeatedly block it with RecentlyFailed errors, saving CPU, memory and bandwidth. > > Also, I would like to know, how many ULPRs can be started without making > troubles do you think ? ~100 ? I ask that because an index tree can contain > many hundred of indexes if the user just add all the indexes [s]he can find > (I have a Thaw with more than 600 indexes). A lot more than that IMHO. Even before ULPRs, all keys which a client wants were registered in a global hashmap, to enable "back-door coalescing" i.e. if one client fetches a block, or if an external node fetches or inserts a block, we check whether any of our clients want it. This applies to every key involved in a request, so potentially a very large number of keys. And it doesn't take up much memory either. So subscription, in the passive sense, is reasonably cheap, it costs the same as having a running request, and if it's for one SSK pointing to one CHK, this is cheap. ULPRs are however a lot more than passive local subscription: they are remote subscription. We do a request, and every node along the chain remembers who it's requested the key from and who has requested it from them, for a certain amount of time. The datastructure implementing this is limited to 10,000 keys on each node (this will take up some RAM). We could expand that onto disk in future, but probably not for a considerable period, perhaps only after we have true passive requests in 0.8. ULPRs are unreliable - after an hour, we forget who asked for the data; we don't reroute if we lose a connection or another node becomes the optimal target etc. So it makes sense imho to rerequest roughly every half hour, if that is possible within load/priority/etc constraints. > > > Please read my other mail for details on ULPRs and failure tables. > > > > Internal implementation details: > > > > The reason M is fixed is that it makes the data structures very easy and > > low overhead. If M is variable, we need a full tree, and it needs to be far > > more memory efficient than java.util.TreeMap. But if we have a fixed M, we > > can just use a sorted (circular) array, since we only ever add to the end, > > remove from the beginning, and very rarely remove from the middle (in which > > case we can remember the time to look for in another structure and do a > > binary search to find the item to delete). > > > > The ULPRs code remembers which nodes have recently requested a specific > > key; the length of time it remembers it for is currently 1 hour, but this > > can be negotiated. Longer equals faster propagation at the cost of > > obviously slightly weaker security. This is why I suggest a 30 minute > > cooldown. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available URL: <https://emu.freenetproject.org/pipermail/devl/attachments/20080213/7d17da11/attachment.pgp>
