[20:16] <toad_> linyos: can you very briefly summarize the reason for not just limiting it to a low number of streams and caching them in RAM? [20:17] <linyos> yes, that should not be done because it limits you to a low number of streams. [20:18] <toad_> but also, we will want to implement TUKs and passive requests at some point [20:18] <linyos> users will probably be very enthusiastic about keeping tons of streams open.
On Thu, Oct 20, 2005 at 08:17:04PM +0100, Matthew Toseland wrote: > On Thu, Oct 20, 2005 at 07:56:56PM +0100, Matthew Toseland wrote: > > On Thu, Oct 20, 2005 at 07:46:46PM +0100, Matthew Toseland wrote: > > > On Thu, Oct 20, 2005 at 06:54:42PM +0100, Matthew Toseland wrote: > > > > Suppose: > > > > - We have a separate store for SSKs. They have 1kB of payload, plus the > > > > headers. > > > > - SSKs are versioned, with a timestamp. I.e. they become TUKs. Requests > > > > can optionally propagate far enough to fetch all values. There can be > > > > any number of SSKs with the same key and different version numbers / > > > > datestamps. Requests can ask for either a single specific version, the > > > > most recent version, or everything since a given revision. Only the > > > > data blocks which are actually returned are promoted in the cache. > > > > - To subscribe to a stream, I send a request out for the key I want, > > > > with the minimum acceptable version number/datestamp, plus three > > > > flags: > > > > - Passive request - If the data is not found, keep a tag on each node > > > > on the chain, and if it ever passes through any of the nodes on the > > > > chain, pass it back to me. > > > > - TUK - Return the most recent version of the SSK. > > > > - Subscribe - Requires the other two. Means the tag persists. > > > > Obviously it will be deleted if the originator rejects the data, or > > > > unsubscribes. > > > > > > > > Now, how would this actually work? > > > > > > > > Each node we are connected to can have up to 1000 subscriptions. Each > > > > subscription is for a key - this can be an SSK or a CHK - and if it is > > > > an SSK, it can have the "persist" flag. If it does not have this flag, > > > > or this flag is false, then when the data is found, the subscription is > > > > removed. If we no do not have enough subscription quota left for a given > > > > subscription, for that node, or for ourselves with another node, we > > > > reject the request attempting to add the subscription with a > > > > RejectOverload. Nodes can unsubscribe. Each subscription can have an > > > > expiry date (actually, perhaps we should impose one). > > > > > > > > Passive requests can only be created by actual requests with the passive > > > > request flag. The request will be routed as normal, until HTL runs out > > > > (or it finds the data if the TUK flag is not enabled). > > > > > > > > Now, how do we make this scale?: > > > > 1. If the TUK flag is off, and the request reaches a node which has the > > > > data, we return it, and the request ends. No passive requests are > > > > installed. > > > > 2. If any request for a TUK, even if not passive, reaches a node which > > > > already has a valid subscription at a higher or equal HTL for the key, > > > > then the request ends at that point, and returns the data that node has. > > > > If the passive-request flag is enabled, then passive request tags are > > > > added up to that node, and that node adds one for the node connecting to > > > > it if necessary. > > > > 3. What to do about looping and coalescing? One option is to not > > > > coalesce at all with running requests, and either forward the request > > > > as-is or reject it... more in my next email... > > > > > > As far as looping and coalescing goes: We can coalesce the requests > > > exactly as proposed for regular requests. So the request will not get > > > stuck by eating its own tail. As long as the network isn't totally > > > degenerate this shouldn't be a big problem. Also, if we do hit our own > > > tail, we could have a "lightweight subscribe" which isn't refcounted, > > > but passes packets both ways to strengthen the network. > > > > > > As far as resubscription and renewals go, there are two options really: > > > 1. Try to get the network to maintain the subscriptions itself; > > > automatic resubscription etc. > > > 2. Make the client resubscribe itself every so often. We would have to > > > let some of their requests through... > > > 3. Don't bother. Switch streams from time to time at the client level. > > > The problem with this is that we will still have to deal with situations > > > such as the parent of a node being disconnected. > > > > Which we can do by simply telling our clients/dependants to resubscribe! > > So the choice is this: either > a) When something breaks, everyone has to resubscribe, or > b) When something breaks, we tell all dependant nodes to let our > requests through, and resubscribe ourself. > > Either way there has to be a concept of "tree" - the node which we > expect to get messages from (the node we routed to). We don't have to > adhere strictly to this in passing data packets onwards, but it does > have to exist. Therefore, we will have a concept of "root", and of "root > location". So a certain amount of what has been planned has to be kept. > A lot of it, in fact, but the data storage is different, and we can have > very many streams. > > So: > > If we lose our parent: > - Simple way: We forcibly unsubscribe all our dependants. Their clients > will resubscribe. > - Hard way: We send a SubscribeRestarted to all our dependants. Then we > send a request out, similar to the one we have previously received, > seeking the key. It has a key-must-be-better-than property, as do > regular requests (they use it in combination with HTL); this is unset, > or rather it is set far away from the target. > -- > Matthew J Toseland - toad at amphibian.dyndns.org > Freenet Project Official Codemonkey - http://freenetproject.org/ > ICTHUS - Nothing is impossible. Our Boss says so. > _______________________________________________ > Tech mailing list > Tech at freenetproject.org > http://emu.freenetproject.org/cgi-bin/mailman/listinfo/tech -- Matthew J Toseland - toad at amphibian.dyndns.org Freenet Project Official Codemonkey - http://freenetproject.org/ ICTHUS - Nothing is impossible. Our Boss says so. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: Digital signature URL: <https://emu.freenetproject.org/pipermail/tech/attachments/20051020/4cc562ca/attachment.pgp>
