(Top-posting because previous post is too long to reply to in-line.) ## refactoring
Refactoring the code helps to make it more understandable for other coders. One of the reasons why I don't work more on freenet myself is because it's extremely difficult to make any changes that are more than a simple bug fix. When I was writing code for the config subsystem to separate out different files into different directories, this was very frustrating and took up much more time than I expected. NB: I'm not implying you should be doing all the work toad, please don't think this. And please don't think that my opinions are automatically invalid / less worthy just because I haven't committed code for ages. I just want to express my view of What I Would Do If I Were God. People can do what they want but it doesn't mean it's a good idea. Granted, I consider "code quality" independently of any issues such as funding, but focusing too much on the latter leads to bad software. If there's pressure like this, ignore it, find something else in the meantime until it goes away. ## freenet Events API freenet does not provide a great API for running callbacks and scheduled tasks. Converting everything to use java.util.concurrent and/or com.google.common.util.concurrent would help this a lot. Of course, Library can and currently does implement this itself, but it's fairly sloppy, and other plugins would benefit if such a framework were provided. Some common tasks that plugins would like to do, which really should be provided by the underlying application: - run tasks in the background - run tasks according to a particular schedule - cancel running tasks - handle dependencies between tasks, so that e.g. if A depends on B and I cancel B, A is automatically cancelled. - group tasks into related categories so one group doesn't affect the other. (e.g. tasks for the group "Library/b-tree-write/index-<URL>" won't starve the resources of group "WoT/background-trust-calculation") ## Library algorithm You keep talking about specific things like "only fetching a single block in the common case". Nobody else knows what this means, even I don't (off the top of my head) because I haven't looked at the code in several years. Having a proper specification that separates different concerns out into layers (e.g. logical structure; on-freenet structure; data format; network performance) helps greatly to make the algorithm understandable to other people, and yourself if you don't work on it for ages. If, instead of saying "only fetch a single block" you say "the on-freenet structure format has persistence issues, and here's a potential solution", it's easier for other people to understand what you mean. ## config management Code for read/writing the config in embedded inside the classes they control, which clutters up the code and makes it confusing. Config code should be separated from the actual application. It would also be nice if this was exposed to plugins. Being a daemon is why the current config system is NOT SUFFICIENT. I need to be able to lock certain config settings, such as where to keep the datastore / run-time files themselves, to conform to the FHS. It's best that such settings are kept in a separate read-only file. The current system only reads one file, freenet.ini. Updating its own binaries is not a problem if freenet knows where the binaries are. The current installer puts them in a rigid place, however this is incompatible with FHS. X On 15/03/12 13:03, Matthew Toseland wrote: > On Wednesday 14 Mar 2012 11:49:33 Ximin Luo wrote: >> Library has some architectural problems. I don't think it's a great use of >> time >> to try to improve the existing code directly. >> >> - We didn't create a proper specification for how the data structure should >> work over freenet >> - I made a mistake in trying to shoe-horn a stub remote data structure into >> the >> Java collections framework. >> - To support loading from remote, I wrote my own event handling framework, >> which was not a very clean design as I wasn't familiar with >> java.util.concurrent at the time. >> - I did write an asynchronous merge algorithm (SkelBTreeMap.update) which >> currently does the majority of the write work, but it's quite complex and >> doesn't integrate well with the rest of the code. Toad also made some >> adjustments on top of it for performance, which makes understanding it even >> more complex as it distracts from the "pure" form of the algorithm. >> >> What is needed to get Library working well is to: >> >> 1. specify the data structure, and associated algorithms, properly. I have >> notes, I can help with this >> 2. use an event handling framework for Java (something similar to Twisted for >> python, but also has the ability to use threads if necessary) >> 3. use this to implement the algorithm in a concise way > > Some functional stuff for Library (related to on-network format), that may or > may not fit in depending on how much is being thrown out and rewritten: > - B+tree (no data in the nodes so reduced depth), not B-tree > - Do we need any on-network changes to limit memory usage? Maybe consider how > to deal with term positions... > - Some optimisations were proposed that might allow for only fetching a > single block in the common case (we'd fetch 2 others for redundancy as well); > this would probably require a binary format and might run into metadata size > issues? > See the bug tracker anyway. > > https://bugs.freenetproject.org/view.php?id=4066 > And categories: > library > b-tree-index > > Also, there is another search plugin, it doesn't scale but has a very compact > format and is very fast. Worth having a look and/or reaching out to its > anonymous author? I dunno what the current status is. I never got around to > making it an official plugin but the rumour was that it worked well. It's a > shame not to bring in an anonymous contributor who goes to the trouble of > writing a working plugin... >> >> freenet itself should use something like this as well - atm it uses its own >> custom solution which is a bit messy and not well-separated from the rest of >> the code. > > For what exactly? I'm not sure I follow? > > One thing we MUST MUST MUST be able to do is *WRITE* the config file. > Obviously this is optional, and on some unixy setups with a competent admin > we can run without the ability to write the config file; but for 99.9% of > users the ability to configure Freenet in a GUI way is useful, and from time > to time it is helpful to be able to update config settings from e.g. a > useralert. > >> It also uses many of its own crypto and concurrent utilities, which >> bloats the code for no reason. > > True. I have no problem with replacing them with the standard stuff provided > the impact on performance on the common case of no hardware acceleration is > positive or negligible. (The API is a bit messy). Also, are we sure the JVM > export keylength BS is finished, over and gone? > > One major complication is quite a lot of Freenet uses 256/256 AES which is > sort of nonstandard, and probably not going to be hardware accelerated. > Replacing it would probably involve changing some link-level formats e.g. so > we could use CBC. Likewise PCFB is fairly widely used, which is also somewhat > nonstandard, although with 256/256 it is identical to CFB, which is more > common. > > Let me know what you need, but I'm not going to do all the work for you. We > have plenty of back compatibility related stuff in place - autonegotiation of > connection type, version numbers on keys etc. >> >> In other words, both Library and freenet (probably freenet first) needs some >> major refactoring work, before major "features" can progress. > > No, I disagree. Refactoring can happen at any time, if a new major feature > depends on it then it should be expedited. However I'm happy to accept your > view re Library; re Freenet, I'm happy for other people to do refactorings > they see as important, but I don't have time (or funding) for anything large > myself. >> >> Most of this requires that freenet-ext be available as separate modules, >> which >> needs an overhaul of the update-over-freenet system as that currently assumse >> freenet-ext comes in a single JAR. > > Yes, this is important. The tricky part here is it would affect *everything*: > The installer, the manual update scripts, the updater ... Let me know what > you need. >> >> Oh, and another thing, the config framework is horrible and we really should >> use an existing external solution. > > Possibly. It works; what are the main problems as you see it? I'm certainly > open to ripping out custom code and replacing it with standard libraries; > even third party libraries if they're not going to massively bloat the > download, although they do *increase* maintenance costs... > >> This would make the freenet installation >> portable to different systems (e.g. to conform to FHS), rather than being >> constrained to the current rigid set of directories. I did make this part >> customisable but it's still untidy and has some fiddly non-fatal problems. > > Well, Freenet is a daemon. You can't treat it as a utility that has each > user's data in their home and the program files in /usr. At the same time, > unless it is packaged, it needs to be able to update its own binaries. Both > of these conflict with the Standard Unix Way Of Doing Things. >> >> Dependency diagram: >> >> [ overhaul freenet update system ] > > This is the most urgent/important thing: Split up freenet-ext.jar and make > the installers, manual update scripts and internal auto-update code able to > deal with this. Most of this is fairly straightforward but it requires a > variety of skills and has to be deployed all at once ... Beyond that we've > discussed other possibilities e.g. using a single script to deploy downloaded > jars from both freenet and the manual emergency update from the web ... > >> |-> [ use external crypto library ] > > This is a good thing, but there are complications as mentioned. > >> |-> [ use external config management library ] > > I am yet to be convinced that this is a big gain. > >> | |-> [ make installation more portable / robust ] > > I would be interested in specific examples. Be careful not to be too > unix-centric. > >> |-> [ use external event-handling library (both async and concurrent) ] > > I'm not sure exactly what is involved here, apart from making Ticker and > Executor use the standard API's. (While not losing the ability to use > PrioRunnable's and NativeThread's to tweak the thread priorities). > > If you're talking about client.async, well, that's a whole other email (which > I'm not gonna write right now) ... > >> | |-> [ update the Library plugin as described ] >> >> This is the ideal non-hacky correct way of doing things, but I gather we >> don't >> have that much development time on the whole. > > That depends on you, and other volunteers :) "Toad helps and does things > Ian/FPI think are important", not "Toad does everything". >> >> X >> >> On 14/03/12 05:06, Juiceman wrote: >>> Hi, I think you wrote or worked on the "Library" plugin for Freenet? >>> I posted a job at >>> http://gun.io/open/45/update-freenet-plugin-library-to-snakeyaml-110 >>> for someone to update the code, figured you might be interested? The >>> bounty is only $25 USD right now, I wish I could afford more.... >>> >>> "The Seeker" took a stab at it but couldn't get it to work. This is >>> what he said to me on Sone: >>> >>> The Seeker >>> @Juiceman I tried doing this, but just making it compile and run isn't >>> nearly enough :-/ When I try to do a search, I get this 3-layer >>> stacktrace... trying to follow wtf is going on makes my brain hurt. I >>> give up. plugins.Library.util.exec.TaskAbortException: Could not >>> construct index from data -- >>> plugins.Library.index.ProtoIndexSerialiser.pull(ProtoIndexSerialiser.java:123) >>> -- plugins.Library.Library.getIndex(Library.java:677) -- >>> plugins.Library.Library.getIndex(Library.java:640) -- >>> plugins.Library.Library.getIndex(Library.java:616) -- >>> plugins.Library.search.Search.splitQuery(Search.java:245) -- >>> plugins.Library.search.Search.startSearch(Search.java:111) -- >>> plugins.Library.search.Search.splitQuery(Search.java:347) -- >>> plugins.Library.search.Search.startSearch(Search.java:111) -- >>> plugins.Library.search.Search.startSearch(Search.java:117) -- >>> plugins.Library.ui.MainPage.processPostRequest(MainPage.java:192) -- >>> plugins.Library.ui.MainPageToadlet.handleMethodPOST(MainPageToadlet.java:96) >>> -- sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) -- >>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) >>> -- >>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) >>> -- java.lang.reflect.Method.invoke(Method.java:597) -- >>> freenet.clients.http.ToadletContextImpl.handle(ToadletContextImpl.java:564) >>> -- >>> freenet.clients.http.SimpleToadletServer$SocketHandler.run(SimpleToadletServer.java:1102) >>> -- freenet.support.PooledExecutor$MyThread.realRun(PooledExecutor.java:233) >>> -- freenet.support.io.NativeThread.run(NativeThread.java:130) >>> plugins.Library.io.DataFormatException: TreeMapTranslator: reverse >>> translation failed. Try supplying a non-null key-translator. -- >>> plugins.Library.util.SkeletonTreeMap$TreeMapTranslator.rev(SkeletonTreeMap.java:330) >>> -- >>> plugins.Library.index.ProtoIndexComponentSerialiser$TreeMapTranslator.rev(ProtoIndexComponentSerialiser.java:324) >>> -- >>> plugins.Library.index.ProtoIndexComponentSerialiser$TreeMapTranslator.rev(ProtoIndexComponentSerialiser.java:310) >>> -- >>> plugins.Library.util.SkeletonBTreeMap$NodeTranslator.rev(SkeletonBTreeMap.java:1519) >>> -- >>> plugins.Library.util.SkeletonBTreeMap$TreeTranslator.rev(SkeletonBTreeMap.java:1582) >>> -- >>> plugins.Library.util.SkeletonBTreeMap$TreeTranslator.rev(SkeletonBTreeMap.java:1551) >>> -- >>> plugins.Library.index.ProtoIndexSerialiser$IndexTranslator.rev(ProtoIndexSerialiser.java:209) >>> -- >>> plugins.Library.index.ProtoIndexSerialiser$IndexTranslator.rev(ProtoIndexSerialiser.java:134) >>> -- >>> plugins.Library.index.ProtoIndexSerialiser.pull(ProtoIndexSerialiser.java:121) >>> -- plugins.Library.Library.getIndex(Library.java:677) -- >>> plugins.Library.Library.getIndex(Library.java:640) -- >>> plugins.Library.Library.getIndex(Library.java:616) -- >>> plugins.Library.search.Search.splitQuery(Search.java:245) -- >>> plugins.Library.search.Search.startSearch(Search.java:111) -- >>> plugins.Library.search.Search.splitQuery(Search.java:347) -- >>> plugins.Library.search.Search.startSearch(Search.java:111) -- >>> plugins.Library.search.Search.startSearch(Search.java:117) -- >>> plugins.Library.ui.MainPage.processPostRequest(MainPage.java:192) -- >>> plugins.Library.ui.MainPageToadlet.handleMethodPOST(MainPageToadlet.java:96) >>> -- sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) -- >>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) >>> -- >>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) >>> -- java.lang.reflect.Method.invoke(Method.java:597) -- >>> freenet.clients.http.ToadletContextImpl.handle(ToadletContextImpl.java:564) >>> -- >>> freenet.clients.http.SimpleToadletServer$SocketHandler.run(SimpleToadletServer.java:1102) >>> -- freenet.support.PooledExecutor$MyThread.realRun(PooledExecutor.java:233) >>> -- freenet.support.io.NativeThread.run(NativeThread.java:130) >>> java.lang.ClassCastException: java.lang.String cannot be cast to >>> java.lang.Double -- java.lang.Double.compareTo(Double.java:32) -- >>> java.util.TreeMap.getEntry(TreeMap.java:328) -- >>> java.util.TreeMap.get(TreeMap.java:255) -- >>> plugins.Library.util.SkeletonTreeMap.putGhost(SkeletonTreeMap.java:95) >>> -- >>> plugins.Library.util.SkeletonTreeMap$TreeMapTranslator.rev(SkeletonTreeMap.java:327) >>> -- >>> plugins.Library.index.ProtoIndexComponentSerialiser$TreeMapTranslator.rev(ProtoIndexComponentSerialiser.java:324) >>> -- >>> plugins.Library.index.ProtoIndexComponentSerialiser$TreeMapTranslator.rev(ProtoIndexComponentSerialiser.java:310) >>> -- >>> plugins.Library.util.SkeletonBTreeMap$NodeTranslator.rev(SkeletonBTreeMap.java:1519) >>> -- >>> plugins.Library.util.SkeletonBTreeMap$TreeTranslator.rev(SkeletonBTreeMap.java:1582) >>> -- >>> plugins.Library.util.SkeletonBTreeMap$TreeTranslator.rev(SkeletonBTreeMap.java:1551) >>> -- >>> plugins.Library.index.ProtoIndexSerialiser$IndexTranslator.rev(ProtoIndexSerialiser.java:209) >>> -- >>> plugins.Library.index.ProtoIndexSerialiser$IndexTranslator.rev(ProtoIndexSerialiser.java:134) >>> -- >>> plugins.Library.index.ProtoIndexSerialiser.pull(ProtoIndexSerialiser.java:121) >>> -- plugins.Library.Library.getIndex(Library.java:677) -- >>> plugins.Library.Library.getIndex(Library.java:640) -- >>> plugins.Library.Library.getIndex(Library.java:616) -- >>> plugins.Library.search.Search.splitQuery(Search.java:245) -- >>> plugins.Library.search.Search.startSearch(Search.java:111) -- >>> plugins.Library.search.Search.splitQuery(Search.java:347) -- >>> plugins.Library.search.Search.startSearch(Search.java:111) -- >>> plugins.Library.search.Search.startSearch(Search.java:117) -- >>> plugins.Library.ui.MainPage.processPostRequest(MainPage.java:192) -- >>> plugins.Library.ui.MainPageToadlet.handleMethodPOST(MainPageToadlet.java:96) >>> -- sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) -- >>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) >>> -- >>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) >>> -- java.lang.reflect.Method.invoke(Method.java:597) -- >>> freenet.clients.http.ToadletContextImpl.handle(ToadletContextImpl.java:564) >>> -- >>> freenet.clients.http.SimpleToadletServer$SocketHandler.run(SimpleToadletServer.java:1102) >>> -- freenet.support.PooledExecutor$MyThread.realRun(PooledExecutor.java:233) >>> -- freenet.support.io.NativeThread.run(NativeThread.java:130) >>> >>> What do you think? >>> >>> -- >>> I may disagree with what you have to say, but I shall defend, to the >>> death, your right to say it. - Voltaire >>> Those who would give up Liberty, to purchase temporary Safety, deserve >>> neither Liberty nor Safety. - Ben Franklin >> >> >> -- >> GPG: 4096R/5FBBDBCE >> https://github.com/infinity0 >> https://bitbucket.org/infinity0 >> https://launchpad.net/~infinity0 >> >> -- GPG: 4096R/5FBBDBCE https://github.com/infinity0 https://bitbucket.org/infinity0 https://launchpad.net/~infinity0
signature.asc
Description: OpenPGP digital signature
_______________________________________________ Devl mailing list Devl@freenetproject.org http://freenetproject.org/cgi-bin/mailman/listinfo/devl
