On Tue, Aug 30, 2011 at 11:49 AM, Robert Hailey <robert at freenetproject.org>wrote:
> On 2011/08/29 (Aug), at 12:58 PM, Ian Clarke wrote: > > The problem is that we come up with solutions that are too complicated to > analyze or fix when they don't work.... > > The cause is complexity, which just grows and grows as we try to fix > problems we don't understand by layering on more complexity. > > > And what is the cause of that? Is the problem really one of *behavior*? > emotions? workflow? organization? > A combination, but at its core I think its a failure to recognize that most working systems are built on principles that are simple enough to understand that they are either "obviously" correct, or can easily be proven to be correct. For example, AIMD, which is TCP's load management system, is simple enough that you can describe the basic idea in a sentence. It is also fairly obvious that it will work, or at least its easy to simulate it and convince yourself that it will work. In contrast, Freenet's current load management system is a combination of mechanisms that are perhaps individually of comparable complexity to AIMD (in fact, AIMD is part of it although used in quite a different context to how it is used in TCP), but together they are far more complex, and interact in ways that are hard to predict. For example, in a conversation with Matthew a few days ago, he realized that the "fair sharing" mechanism which tries to allocate resources fairly among "downstream" nodes, was probably generating a bunch of rejected messages when it transitions from one mode of operation to another, with all kinds of negative consequences. Who knows how many other unintended interactions there are? > Matthew has presented some very real problems which he is trying to work > around (with much frustration, I'm sure). I think he needs more leverage > My criticism is that Matthew's proposals for "fixing" the problem follow exactly the same pattern of all the previous "fixes" that turned out to have either no effect, or a negative effect. It starts out with a bunch of hand-wavey hypotheses about what the problem might be, which is followed by a bunch of fixes for what might be the problem. There is no evidence that these problems are real, I think a big part of the reason is that the existing system is too complicated to debug. I know it sounds like I'm beating up on Matthew here, and that isn't my intention, he is following a precedent set by me and others that have long since left the project. Having (I hope) learned some tough lessons, and recognized the folly in our past approach, I'm now hoping its not too late to rescue the project from an ignominious death. I think we're talking about weeks of work here, not months, and frankly I don't think we've got a choice if the project is to survive. Freenet is no use to anyone if it doesn't work, regardless of how cool our goals are, or how clever our algorithms and heuristics. > If you read my suggestion below, we can discuss how it would allow: > > With an investment of developer time, we could separate the current freenet > code into three interfaced sections (link-layer, routing-layer, > user/client-interface-layer). > > If we then were to modify the outer layers to accept two routing-layers > (e.g. client requests round-robin between the two but thereafter stay in > that network) we could have "two networks in one" a stable-net (for the > nay-sayers, a disaster/fallback, and as a control for measurement), and a > development-net where experimentation could take place. > > Drawing the interface lines on theory (rather than present code-state) > would be critical [e.g. load-balancing should be in the middle layer, imo]. > The goal being, reliable communication with near-guaranteed/methodical > improvement > > While I'm sure there is much room for improvement in the way the code is architected, and the separation of concerns - I don't think refactoring is the answer. We need to refactor the fundamental way that we solve problems like load balancing. Ian. -- Ian Clarke Founder, The Freenet Project Email: ian at freenetproject.org -------------- next part -------------- An HTML attachment was scrubbed... URL: <https://emu.freenetproject.org/pipermail/devl/attachments/20110830/42369abb/attachment.html>
