On Sun, Sep 18, 2005 at 04:12:19PM +0200, Guido Winkelmann wrote: > Hi, > > I have a proposal to make with regard to the development of 0.7: > > Let's remove everything that's not strictly essential to the operation of the > node from the code of fred 0.7 and write an external daemon to support these > functions, that communicates with the node via FCP (and/or something similar > for Freenet streams). Let's call that daemon fad, "Freenet accompanying > daemon" for now. > If 0.7 should be done this way, the mentioned external daemon should, of > course, come bundled with the standard Freenet package. A stripped down fred > on its own would be of no use to most, just like a TCP/IP stack on its own - > or i2p...
And it would have to run in the same JVM, for performance and user friendliness reasons (an extra several tens of megs for a second VM is *BAD*). Why is it beneficial? > > Things that should be thrown out that are present in current versions (5xxx) > include: > > - The Web interface (definitely - but it might be wise to leave a stripped > down version for strictly node-maintainance related purposes.) If you're going to chuck it out, then chuck it out. We do have FCP, after all. > - The Distribution Servlet (definitely) > - Everything related to splitfile handling (maybe, more talk on that further > down) Yuck. That would mean that we have to have two separate FCP protocols - high level FCP and low level FCP. IMHO we must not require every single client author to implement binary metadata, compressed metadata, hierarchical metadata, FEC codecs, and everything else required; we must have a "high level FCP" including reasonably transparent splitfile support, and it should probably include download direct to disk, download when the client is not connected, and queueing, as well, for reasons related to starting splitfile downloads from fproxy. > - The command line client (I don't think there's a good reason to include > that > in the normal freenet.jar) It's tiny, and it's a useful debugging tool. There is no compelling reason to include it in the jar except to save 10kB or whatever it is. Which is insufficient reason given Fred's memory footprint! > and maybe other stuff I don't know about. > > The advantages to this approach are: > > - The node source code becomes smaller and easier to maintain. It does? How so? We'd have to maintain two separate source modules! > - The operation of the node becomes more reliable. (There is less stuff in > there that might cause it to crash.) The node must be cleanly separated from the UI code, including e.g. splitfile assembly. This happens via interfaces, not via running it in a separate VM, and not via having it in a separate CVS module. > - Resource consumption of the node will be lower and much more predictable. Only if it runs without the extra daemon. Which it won't for the majority. The memory overhead for running two JVMs instead of one will be significant, unless the JVM automatically coalesces, in which case there is no point anyway. > - The code for enduser related functionality becomes a lot more accessible > for > new programmers. (Patches to fad are pretty much guaranteed not to interfere > with the node's core operations and, thus, are more easily accepted.) Nobody (in the OSS world) codes java. And of those who do, they don't use Fred's existing interfaces. This is a matter of documentation and communication and stubbornness, not a matter of what is in what jar. > - The external daemon can be written in a language other than Java (Whether > that's an advantage may depend on your POV. IMO, it's a big one.) Not if it is bundled with Fred it can't. The high level daemon must be maintained by the project itself, meaning it has to be in java. And, without intending to start a language flamewar here, you do know that java can be compiled, right? > - Third party implementations of the node itself can be done more easily. > (The > programmers won't have to rewrite all of the most basic user interface stuff > from scratch) (I think this is a pretty important issue in long run.) Why does it matter? As long as there are clear internal interfaces it is quite possible to plug in an alternative node implementation. And given the SUBSTANTIAL effort involved in cloning either the node or the high level code, this is not a big deal. > - We can draw a very clear line between what's enduser stuff and what isn't. > (And the node should not be enduser stuff. Install it, configure resource > restrictions, start it, stop it should be the sum of the cases where the > (opennet-) enduser has to interface with it.) Configure resource restrictions? How exactly? Via command line? Via telnet to port 8481? By definition all user friendly code must be left out of the node! > - Fred can finally become a lean and mean unbloated routing machine. Which is totally meaningless if it requires a bloated user daemon for it to do anything useful. > > Disadvantages: > > - The project needs to make extra sure that "fad" really is in a usable state > before it can release 0.7 > > About splitfile handling inside or outside the node: > > In my opinion, there are two sensible ways one could go with regard to > splitfiles: > > - Make the node completely ignorant about splitfiles, having it all done on > the client side. This has the advantage that the node becomes simpler and > slicker and the disadvantage that clients become more complex. In the worst > case, we may end up with several incompatible splitfile handling algorithms > floating around. (This might be countered by releasing easy to use libraries > for splitfile handling early on.) We had this debate months ago, it's a shame you didn't speak out then. In any case: We decided to go with "put as much as possible into the node", for the following reasons: 1. The core of the node can still be simple. There are such things as interfaces. 2. We have to implement splitfile handling *ANYWAY* for Fproxy. It is important to me and to ian that we not have a large amount of code in Fred that is only used by Fproxy. 3. It is more important that third party apps have a simple interface to work with, and therefore be easy to build, than that Freenet itself is simple. 4. Standardization, integration, etc. > Also, splitfile handling is a very resource intensive operation. People might > want to delegate it to a different computer. Not all splitfile algorithms are equal. The onion FEC algorithm can use up to 24MB of RAM for decoding a large file. Other, un-segmented (but somewhat selective) algorithms are often leaner, in terms of the amount of data that must be in RAM at once. > (At least on my current node, splitfile decoding/encoding is the most common > cause for OutOfMemory errors.) > > - Handle splitfiles completely inside the node, completely abstracting it > away > from clients. In this case, programmers of third party tools won't even have > to be aware of the fact that fred internally only handles blocks of 32 KiB. > FCPv2 can be simpler, client apps can be a lot simpler. That is the current intention for the reasons given above. Clients should not have to be aware of the 32kB limit. They should not have to be aware of binary metadata, or the fact that a splitfile manifest may not fit into a single block. It should be possible to write a simple but functional client in a small amount of code in any language. So that third party devs can get on with writing, in whatever obscure language they choose to use, good GUI tools that people want to use. > > The current situation, where node and client cooperate over FCP on the > splitfile handling is (still imho) the worst one can do. Agreed. There have been many complaints about the FCP FEC API. > > Thus far for my proposal. Comments would be appreciated. > > Guido -- Matthew J Toseland - toad at amphibian.dyndns.org Freenet Project Official Codemonkey - http://freenetproject.org/ ICTHUS - Nothing is impossible. Our Boss says so. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: Digital signature URL: <https://emu.freenetproject.org/pipermail/tech/attachments/20050919/27e3aab8/attachment.pgp>
