On 20/05/13 22:36, leuchtkaefer wrote: > > > Hi infinity0, > > My proposal to GSoC13 is highly related to your code (Library). > > First, do you have any extra documentation on the code that you think it > could be useful for me to understand the most important parts, such the > SkeletonBTreeMap? >
Hello, I did Library for GSoC 2009 and back then I was inexperienced with building and engineering large software codebases (such as freenet and its plugin ecosystem). There are many aspects of Library that I would do differently today if I was re-doing that project. A large part of Library focuses on serialisation of massively-large(1) data structures, implemented *on top of* freenet's decentralised(2) storage. (1) and (2) together is what makes the problem hard.[1] For my GSoC 2009 project, I tried to solve this problem by implementing a load-on-demand local data structure (SkeletonBTreeMap) that represents the *overall* data structure (a B-tree) as it exists on freenet storage. By contrast, massively scalable distributed systems such as Bigtable, and even the underlying freenet DHT storage system, never expose the *overall* data structure to the clients of those systems - instead they allow piece-by-piece access, e.g. by row, or by key, and the client never sees the top-level data structure. For Library's B-Tree, this is not feasible, because (due to the design of freenet) we cannot offload computation (i.e. data structure book-keeping) onto other nodes.[2] It was also not feasible to use freenet's decentralised storage more directly, because it has certain properties (such as LRU cache) that are not acceptable for a search index. So. That was an overview of the abstract algorithmic issues surrounding the design of Library. Please let me know if any part of what I just said is not understandable. Every sentence makes an important theoretical point. If you do not fully understand *any part*, ask me to clarify, otherwise I fear that you may repeat the same mistakes that I did. This is not exactly a problem since GSoC is partly about learning - but it would be sub-optimal for the project's progress. I'll hold off on answering the rest of your questions to give you a chance to digest my previous answers. Understanding those will make it easier for you to understand my answers to the next section - and you may even be able to figure those answers out for yourself without me explaining it explicitly. Also, if you give me some context on what you're trying to achieve, I can give more specific advice. Let me know how you get along, and good luck! Ximin [1] We are lucky that we don't have to further worry about security because the underlying freenet storage allows us to restrict access to one single user. [2] Perhaps one day, a system that supports fully homomorphic encryption will allow this to happen. > Second, I have some questions: > > 1. You disabled the "boolean internal_entries" inside the > classSkeletonBTreeMap and use option 2. I don't understand what do you mean > about a dummy serialiser that copies task.data to task.meta. What contains > task.data? > > 2. What means deflate/inflate the node? > > 3. What is a GhostNode? I understood is a not desirable structure used to > contain some metadata or sth related with the serializer and needs to be > removed. > > If you can elaborate more about Library, besides the documentation already > published in the wiki, it will be of great help. > > Thanks in advance, > > leuchtkaefer > > > > _______________________________________________ > Devl mailing list > [email protected] > https://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl -- GPG: 4096R/5FBBDBCE https://github.com/infinity0 https://bitbucket.org/infinity0 https://launchpad.net/~infinity0
signature.asc
Description: OpenPGP digital signature
_______________________________________________ Devl mailing list [email protected] https://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl
