[freenet-dev] Gun.IO and Freenet
Library has some architectural problems. I don't think it's a great use of time to try to improve the existing code directly. - We didn't create a proper specification for how the data structure should work over freenet - I made a mistake in trying to shoe-horn a stub remote data structure into the Java collections framework. - To support loading from remote, I wrote my own event handling framework, which was not a very clean design as I wasn't familiar with java.util.concurrent at the time. - I did write an asynchronous merge algorithm (SkelBTreeMap.update) which currently does the majority of the write work, but it's quite complex and doesn't integrate well with the rest of the code. Toad also made some adjustments on top of it for performance, which makes understanding it even more complex as it distracts from the "pure" form of the algorithm. What is needed to get Library working well is to: 1. specify the data structure, and associated algorithms, properly. I have notes, I can help with this 2. use an event handling framework for Java (something similar to Twisted for python, but also has the ability to use threads if necessary) 3. use this to implement the algorithm in a concise way freenet itself should use something like this as well - atm it uses its own custom solution which is a bit messy and not well-separated from the rest of the code. It also uses many of its own crypto and concurrent utilities, which bloats the code for no reason. In other words, both Library and freenet (probably freenet first) needs some major refactoring work, before major "features" can progress. Most of this requires that freenet-ext be available as separate modules, which needs an overhaul of the update-over-freenet system as that currently assumse freenet-ext comes in a single JAR. Oh, and another thing, the config framework is horrible and we really should use an existing external solution. This would make the freenet installation portable to different systems (e.g. to conform to FHS), rather than being constrained to the current rigid set of directories. I did make this part customisable but it's still untidy and has some fiddly non-fatal problems. Dependency diagram: [ overhaul freenet update system ] |-> [ use external crypto library ] |-> [ use external config management library ] ||-> [ make installation more portable / robust ] |-> [ use external event-handling library (both async and concurrent) ] ||-> [ update the Library plugin as described ] This is the ideal non-hacky correct way of doing things, but I gather we don't have that much development time on the whole. X On 14/03/12 05:06, Juiceman wrote: > Hi, I think you wrote or worked on the "Library" plugin for Freenet? > I posted a job at > http://gun.io/open/45/update-freenet-plugin-library-to-snakeyaml-110 > for someone to update the code, figured you might be interested? The > bounty is only $25 USD right now, I wish I could afford more > > "The Seeker" took a stab at it but couldn't get it to work. This is > what he said to me on Sone: > > The Seeker > @Juiceman I tried doing this, but just making it compile and run isn't > nearly enough :-/ When I try to do a search, I get this 3-layer > stacktrace... trying to follow wtf is going on makes my brain hurt. I > give up. plugins.Library.util.exec.TaskAbortException: Could not > construct index from data -- > plugins.Library.index.ProtoIndexSerialiser.pull(ProtoIndexSerialiser.java:123) > -- plugins.Library.Library.getIndex(Library.java:677) -- > plugins.Library.Library.getIndex(Library.java:640) -- > plugins.Library.Library.getIndex(Library.java:616) -- > plugins.Library.search.Search.splitQuery(Search.java:245) -- > plugins.Library.search.Search.startSearch(Search.java:111) -- > plugins.Library.search.Search.splitQuery(Search.java:347) -- > plugins.Library.search.Search.startSearch(Search.java:111) -- > plugins.Library.search.Search.startSearch(Search.java:117) -- > plugins.Library.ui.MainPage.processPostRequest(MainPage.java:192) -- > plugins.Library.ui.MainPageToadlet.handleMethodPOST(MainPageToadlet.java:96) > -- sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) -- > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > -- > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > -- java.lang.reflect.Method.invoke(Method.java:597) -- > freenet.clients.http.ToadletContextImpl.handle(ToadletContextImpl.java:564) > -- > freenet.clients.http.SimpleToadletServer$SocketHandler.run(SimpleToadletServer.java:1102) > -- freenet.support.PooledExecutor$MyThread.realRun(PooledExecutor.java:233) > -- freenet.support.io.NativeThread.run(NativeThread.java:130) > plugins.Library.io.DataFormatException: TreeMapTranslator: reverse > translation failed. Try supplying a non-null key-translator. -- > plugins.Library.util.SkeletonTreeMap$TreeMapTranslator.rev(SkeletonTreeMap.java:330) > -- > plugins.Library.index.ProtoInd
Re: [freenet-dev] Gun.IO and Freenet
Library has some architectural problems. I don't think it's a great use of time to try to improve the existing code directly. - We didn't create a proper specification for how the data structure should work over freenet - I made a mistake in trying to shoe-horn a stub remote data structure into the Java collections framework. - To support loading from remote, I wrote my own event handling framework, which was not a very clean design as I wasn't familiar with java.util.concurrent at the time. - I did write an asynchronous merge algorithm (SkelBTreeMap.update) which currently does the majority of the write work, but it's quite complex and doesn't integrate well with the rest of the code. Toad also made some adjustments on top of it for performance, which makes understanding it even more complex as it distracts from the "pure" form of the algorithm. What is needed to get Library working well is to: 1. specify the data structure, and associated algorithms, properly. I have notes, I can help with this 2. use an event handling framework for Java (something similar to Twisted for python, but also has the ability to use threads if necessary) 3. use this to implement the algorithm in a concise way freenet itself should use something like this as well - atm it uses its own custom solution which is a bit messy and not well-separated from the rest of the code. It also uses many of its own crypto and concurrent utilities, which bloats the code for no reason. In other words, both Library and freenet (probably freenet first) needs some major refactoring work, before major "features" can progress. Most of this requires that freenet-ext be available as separate modules, which needs an overhaul of the update-over-freenet system as that currently assumse freenet-ext comes in a single JAR. Oh, and another thing, the config framework is horrible and we really should use an existing external solution. This would make the freenet installation portable to different systems (e.g. to conform to FHS), rather than being constrained to the current rigid set of directories. I did make this part customisable but it's still untidy and has some fiddly non-fatal problems. Dependency diagram: [ overhaul freenet update system ] |-> [ use external crypto library ] |-> [ use external config management library ] ||-> [ make installation more portable / robust ] |-> [ use external event-handling library (both async and concurrent) ] ||-> [ update the Library plugin as described ] This is the ideal non-hacky correct way of doing things, but I gather we don't have that much development time on the whole. X On 14/03/12 05:06, Juiceman wrote: > Hi, I think you wrote or worked on the "Library" plugin for Freenet? > I posted a job at > http://gun.io/open/45/update-freenet-plugin-library-to-snakeyaml-110 > for someone to update the code, figured you might be interested? The > bounty is only $25 USD right now, I wish I could afford more > > "The Seeker" took a stab at it but couldn't get it to work. This is > what he said to me on Sone: > > The Seeker > @Juiceman I tried doing this, but just making it compile and run isn't > nearly enough :-/ When I try to do a search, I get this 3-layer > stacktrace... trying to follow wtf is going on makes my brain hurt. I > give up. plugins.Library.util.exec.TaskAbortException: Could not > construct index from data -- > plugins.Library.index.ProtoIndexSerialiser.pull(ProtoIndexSerialiser.java:123) > -- plugins.Library.Library.getIndex(Library.java:677) -- > plugins.Library.Library.getIndex(Library.java:640) -- > plugins.Library.Library.getIndex(Library.java:616) -- > plugins.Library.search.Search.splitQuery(Search.java:245) -- > plugins.Library.search.Search.startSearch(Search.java:111) -- > plugins.Library.search.Search.splitQuery(Search.java:347) -- > plugins.Library.search.Search.startSearch(Search.java:111) -- > plugins.Library.search.Search.startSearch(Search.java:117) -- > plugins.Library.ui.MainPage.processPostRequest(MainPage.java:192) -- > plugins.Library.ui.MainPageToadlet.handleMethodPOST(MainPageToadlet.java:96) > -- sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) -- > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > -- > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > -- java.lang.reflect.Method.invoke(Method.java:597) -- > freenet.clients.http.ToadletContextImpl.handle(ToadletContextImpl.java:564) > -- > freenet.clients.http.SimpleToadletServer$SocketHandler.run(SimpleToadletServer.java:1102) > -- freenet.support.PooledExecutor$MyThread.realRun(PooledExecutor.java:233) > -- freenet.support.io.NativeThread.run(NativeThread.java:130) > plugins.Library.io.DataFormatException: TreeMapTranslator: reverse > translation failed. Try supplying a non-null key-translator. -- > plugins.Library.util.SkeletonTreeMap$TreeMapTranslator.rev(SkeletonTreeMap.java:330) > -- > plugins.Library.index.ProtoInd
