On Thursday 10 April 2008 21:28, you wrote: > > On Apr 8, 2008, at 2:00 PM, Matthew Toseland wrote: > > Most of the short-term traffic is Frost spam, which is just SSKs, > > because it's > > cheaper to insert SSKs pointing to random CHKs that already exist... > > > > If it wasn't for the spam, it would be equal numbers of SSKs and CHKs, > > assuming a typical message can't fit in 1kB (or has a MIME type). > > > > I don't think short term traffic is that big a part of the total > > though. Stats > > would be welcome of course. Also any sort of preference system will > > be abused > > by clients to prioritise their data over everyone else's, and by > > attackers as > > an extra bit of information to identify what the data is. > > True, and I don't think that natural decay of data is the issue here. > > >> Is this more to do with natural decay, network churn (blocks actually > >> being removed from the network), or nodes just changing location? > >> > >> Maybe inserts need to be stored on more nodes' stores (than present > >> 2-3), meaning that the caches (20) do not suffice? > > > > Well, most keys are in fact fetched from the cache, according to the > > statistics. > > Also true, of course, but if data is being fetched from the cache, > that does not really imply that the routing algorithim/network-state > 'found' the data, but simply that the data is likely 'popular'. > > I guess the confusion I am having comes from this... that my 9GB store > is not even full yet. So... even if the average data store is ~1gb, > this would be a rollover time in *months* of total-inactivity in order > for a key to drop out of the store, and then if the local case is > indicative of the global case, data should be stored very well on the > network (for a long time).
Not for the cache though. The store has a long rollover time because we're very choosy... hence the problem is with routing... > > This would imply what you said earlier... storage of data is not a > problem, finding it is. So my question is, "Why can't freenet find the > data?" > (1) It puts it in a different place than it would look for it, Not true - at the time. > (2) the node(s) it stores the data on leave the network, Probably quite common. We can however address this to some degree through detecting low uptime nodes and not relying on them. This is a relatively easy change which will go in in the next few days. > (3) the node(s) it stores the data on move locations within network, This is a problem. We now random reset 4x less frequently, which may help, but if there is long term drift then it won't help *much*. > (4) the keyspace all around the location changes (i.e. catastrophic > merging), > (5) data drops from the store while it is served from the cache (high > level race) We always check the store first afaik. > > Judging by 'store writes' it looks like requests outweigh inserts by > 1700:1 on my node! Has removing the 'nearestLoc' logic effected the > number of nodes that store the inserts? I don't see why it would have. We go to an average of 10 + 8 + 4 = 22 nodes now, maybe we did go to more before but this should still be plenty on average, and even reasonable in worst case (10). > I wonder how to safely get the > insert data onto more nodes... would it be feasible to store every > insert that comes across the node? We already do random reinsert (1 in 200 successful requests), and splitfile healing. Much more block level redundancy likely would reduce the effectiveness of the store, and might have privacy issues. We have a factor of two redundancy at the FEC layer as well, maybe 3 here is enough with some basic precautions? One other idea (from Oskar) for improving the reachability of data: Bloom filters! For our peers, and perhaps for some number of hops. Not for the cache (at least not until premix routing), but for the store only. Is this sufficiently safe? The store doesn't store our requests at all, but it does store some portion of our inserts... so I suppose we'll have to wait for premix routing after all? -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available URL: <https://emu.freenetproject.org/pipermail/devl/attachments/20080411/8ec94601/attachment.pgp>
