On Tue, Nov 22, 2005 at 02:33:07PM +0100, Roman Bednarek wrote: > > On Mon, 7 Nov 2005, Matthew Toseland wrote: > > >On Sun, Nov 06, 2005 at 10:33:25AM +0100, Roman Bednarek wrote: > >> > >> Hello. > >> Probably it is too late for changes and you probably already discussed > >>it, but I would like to say few words about the design. > >> Building monolithic solutions - that is doing everything in one place - > >>is often not the best solution. The better approach may be some smaller > >>cooperating layers. For example it may look like this: > > > >Everyone uses layers; this is perfectly normal. > > > >> Layer 0 - the freenet core/server - it manages datastore and network > >>connections. The api to it can be extremaly simple: Data getData(key) and > >>Key putData(data), where size of data is limited by the desing(currently > >>1M), the metadata can also be handled here or on the next layer. > > > >Each layer can use the layers below, but will generally depend most on > >the layer immediately below: > > > >Layer 0 - transport plugins (very simple, interface depends on class) > >Layer 1 - transport classes, inc. packet retransmission, crypto > >Layer 2 - messages (taken from dijjer) > >Layer 3 - node locations, swapping node locations > >Layer 4 - requests, inserts, datastore > >Layer 5 - trivial interface to above - get/put CHK's > > This is the internal structure I was referring to what is visible and > usable to the user. So I see that my proposition basically goes down to > including these two calls (get/put CHK) into FCP2.
Absolutely not. Clients do not need to know about hierarchical, compressed, binary splitfile zip manifest metadata! And we certainly don't want 100 different client implementations each with its own metadata format. > > > > >> Layer 1 - client services - splitfile handling, download queue, > >>containers, calculate keys and similar utilities. It may be run on the > >>same computer, as part of the node(started on the same javaVM, as layer 0) > >>or it may run elswere. It also may be available as a C library (such as > >>fcptools) > > > >Layer 6 - metadata handling, splitfiles, updatable keys, redirects, etc. > >Will also include the download manager and so on eventually. > > > >The stuff I'm building now. > > > >The interface to layer 7 is either FCPv2, or an internal interface > >equivalent to FCPv2 (to allow running fproxy in the same JVM). > > > >This layer will be part of the node. 0.7 metadata is binary, and quite > >complex (it can itself be split!), and I do not want clients to have to > >deal with it. FCPv2 will therefore not (normally) expose it to clients; > >FCPv2 is fairly high level. > > It is good that clients will not have to deal with all the complex > details, but it is really bad that they cannot do it. I never liked > situations when something is possible inside the library, but is hidden > from the user who cannot use it. Why? There is simply no reason for a client to want to deal with binary, often split, metadata. > > > > >There is no reason for layer 7 clients to access anything below layer 6. > >If you want to reimplement layer 6 in C, or run it on another machine, > >that is quite possible, however it would mean providing a low-level FCP > >as well as a high-level FCP, which I consider to be ugly and > >unnecessary. > > > As I written before I think that hiding something is unnecessary. Why? > > >Note that you can have a fixed darknet link between the node on your > >fast workstation which gets switched off regularly, and the node on your > >slow firewall which is on 24/7. In which case you could run the fetches > >on the workstation (at the cost of them not running while it is switched > >off), even though the firewall node is on the open-net. > > In the meantime my setup has changed, I use my small firewall machine > for downloads too, using solution based on fcp tools. As a C > program/library it has very low memory/CPU usage and can run on PII > without problems. Yeah, rewriting Freenet in C will solve all our problems. Especially the ones which don't originate from language difficulties. The heaviest operation is probably FEC decoding (or worse, encoding), which is implemented in native code *anyway*. And if you don't use FEC encoding, you're a fool. > > > > >> Layer 2 - applications - FPProxy, frost, fuqid. They usually talk to > >>the server through layer 1, but some more advanced(like fuqid or maybe > >>frost) may choose to talk directly to the core server at layer 0. > > > >Layer 7 in my model. FUQID, Frost and FIW should talk to layer 6 > >exclusively, via the exposed APIs, which will include a download manager > >backend. The main advantages are: > >- We can take control of download policy, and optimize it however we > > want. The code is maintained. > > It is not necessary a good thing, maybe a download queue in a client > will perform better? and client can optimize it in its own way. Why would it perform better? It is of course possible to queue on the client level even with the planned FCPv2 functionality, but if the client starts too many simultaneous requests, or too frequent, then they will be automatically queued (for reasons of network stability and possibly security). > > >- Clients can be very simple, because they don't need to worry about > > e.g. multi-layer splitfiles. > >- We only need one HTML anonymity filter implementation, and we can > > include I18N (charset detection on insert) regardless of the client. How many insertion tools will automatically detect charsets and put that in the metadata, as of today? Zero afaik. > > > So the bottom line is that I think clients should be able to work with > lower level of the node. Today clients are working that way and are > doing good job with it, there is no need to throw away that code and rely > only on the node. That's your problem. And I dispute that they are doing a good job of it. And 0.7's metadata is necessarily significantly different from 0.5's, for the simple reason that 0.7 is different. In 0.5, keys could be any size from 1kB to 1MB. In 0.7, they are all 32kB. Keys can be transparently compressed, splitfiles will often take up more than one block, and so on. -- Matthew J Toseland - toad at amphibian.dyndns.org Freenet Project Official Codemonkey - http://freenetproject.org/ ICTHUS - Nothing is impossible. Our Boss says so. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: Digital signature URL: <https://emu.freenetproject.org/pipermail/tech/attachments/20051123/b40edcf0/attachment.pgp>
