On Tuesday 30 Aug 2011 18:27:03 Ian Clarke wrote: > On Tue, Aug 30, 2011 at 11:49 AM, Robert Hailey > <robert at freenetproject.org>wrote: > > > On 2011/08/29 (Aug), at 12:58 PM, Ian Clarke wrote: > > > > The problem is that we come up with solutions that are too complicated to > > analyze or fix when they don't work.... > > > > The cause is complexity, which just grows and grows as we try to fix > > problems we don't understand by layering on more complexity. > > > > > > And what is the cause of that? Is the problem really one of *behavior*? > > emotions? workflow? organization? > > A combination, but at its core I think its a failure to recognize that most > working systems are built on principles that are simple enough to understand > that they are either "obviously" correct, or can easily be proven to be > correct. For example, AIMD, which is TCP's load management system, is > simple enough that you can describe the basic idea in a sentence. It is > also fairly obvious that it will work, or at least its easy to simulate it > and convince yourself that it will work.
TCP as a whole is not simple. > > In contrast, Freenet's current load management system is a combination of > mechanisms that are perhaps individually of comparable complexity to AIMD > (in fact, AIMD is part of it although used in quite a different context to > how it is used in TCP), but together they are far more complex, and interact > in ways that are hard to predict. > > For example, in a conversation with Matthew a few days ago, he realized that > the "fair sharing" mechanism which tries to allocate resources fairly among > "downstream" nodes, was probably generating a bunch of rejected messages > when it transitions from one mode of operation to another, with all kinds of > negative consequences. Who knows how many other unintended interactions > there are? TCP has never had major architectural bugs. Except for all the ones it has had. For instance, it took many years to figure out that increasing the window size when you're not using it fully is a bad idea. > > > Matthew has presented some very real problems which he is trying to work > > around (with much frustration, I'm sure). I think he needs more leverage > > My criticism is that Matthew's proposals for "fixing" the problem follow > exactly the same pattern of all the previous "fixes" that turned out to have > either no effect, or a negative effect. It starts out with a bunch of > hand-wavey hypotheses about what the problem might be, which is followed by > a bunch of fixes for what might be the problem. There is no evidence that > these problems are real, I think a big part of the reason is that the > existing system is too complicated to debug. There is ample evidence that these problems are real. For instance, there is the classic result that triple inserting is far, far, far more effective in terms of data persistence than single inserting - FOR THE SAME KEY! Given that inserts are always routed the same way, the only possible explanation is that the insert is being rejected by many of the nodes on the first attempt, and inserting it 3 times means either it is routed better, or it is accepted by the right nodes - either way, spurious rejections are the problem. > > I know it sounds like I'm beating up on Matthew here, and that isn't my > intention, he is following a precedent set by me and others that have long > since left the project. Having (I hope) learned some tough lessons, and > recognized the folly in our past approach, I'm now hoping its not too late > to rescue the project from an ignominious death. I think we're talking > about weeks of work here, not months, and frankly I don't think we've got a > choice if the project is to survive. Freenet is no use to anyone if it > doesn't work, regardless of how cool our goals are, or how clever our > algorithms and heuristics. I think we are talking about months. We'll be arguing about it for weeks, it'll take weeks to write good simulations and a lot more weeks to tweak them until they actually work, and then it'll take months to implement a new system. For instance, NLM was originally intended to be relatively simple. But it turned out that not only was it more complex in practice than in theory, it also touched on a lot of other areas (mostly related to predicting load on our peers). The same might be true of any new mechanism, although admittedly many of the bugs are gone now. I have explained how NLM could be tweaked to be workable elsewhere. I agree that AIMD's aren't ideal but anything else would be much more work, so can be postponed for now. And ArneBab has a basic mathematical model which explains why NLM had such problems, and why it should be fixable. > > > If you read my suggestion below, we can discuss how it would allow: > > > > With an investment of developer time, we could separate the current freenet > > code into three interfaced sections (link-layer, routing-layer, > > user/client-interface-layer). > > > > If we then were to modify the outer layers to accept two routing-layers > > (e.g. client requests round-robin between the two but thereafter stay in > > that network) we could have "two networks in one" a stable-net (for the > > nay-sayers, a disaster/fallback, and as a control for measurement), and a > > development-net where experimentation could take place. > > > > Drawing the interface lines on theory (rather than present code-state) > > would be critical [e.g. load-balancing should be in the middle layer, imo]. > > The goal being, reliable communication with near-guaranteed/methodical > > improvement > > > While I'm sure there is much room for improvement in the way the code is > architected, and the separation of concerns - I don't think refactoring is > the answer. We need to refactor the fundamental way that we solve problems > like load balancing. We need more data. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 198 bytes Desc: This is a digitally signed message part. URL: <https://emu.freenetproject.org/pipermail/devl/attachments/20110831/45e8f56d/attachment.pgp>
