Major issues for 1.0 as I see them. I've tried to limit this to fundamental technical challenges and essentials for users: - Data persistence. This is being worked on, IMHO we can probably improve it significantly, and probably without drastic changes. - Load management. We have proposals which are very close to implementible. The objective is to make nodes with a few fast darknet connections use them to the full, while not being overly reliant on ubernodes, not giving information away about who originates requests, and limiting misrouting. Consider different load management for small stuff we want quickly vs bulk downloads. - Darknet: The Pitch Black attack: We need to fix this. Oskar has a good idea how but has not yet simulated a fix. - Passive requests. These are essential for WoT-style chat etc to work well. They will also greatly reduce the polling load caused by requesting files that are not immediately findable. As explained in another mail, I think we could actually implement this relatively easily. Native publish/subscribe based on rendezvous at a key is not necessary for 1.0 IMHO. Most applications which would use it can use passive requests albeit a little less efficiently. - The mobile attacker source tracing attack: We have a partial fix for inserts, but not for reinserts, and not for requests. There are at least 5 ways to fix this, and most of them actually improve performance. We should implement one of them. - *Consider* bloom filter sharing (possibly only on darknet), better bursting support and long-term requests. The first two are solutions, the last is functionality but it's closely related to all these things, and it's important for darknet, low uptime and rare files. - Minimise fred's resource usage. In particular in disk I/O due to the datastore and the persistent downloads queue. - Efficient transport layer. Essentially this is zidel's SoC. And it paves the way for transport plugins, which are not IMHO required for 1.0. - Reconsider the CHK block size. A larger block size would make a lot of performance issues go away, on several layers. However, it might make Freenet work less well on really slow links. It probably wouldn't be a big problem for freesites, especially big containerised ones. But it might be an issue for chat. Some of the proposals to changes to data return etc (e.g. bursting) might reduce the issues with different timeouts. An indirect hash table implementation for the store might allow us to implement this back compatibly by allowing much easier resizing, and would probably improve performance. Or some variant of cuckoo hashing perhaps. - Perfect Windows installer. Consider packages for various linux distros. Consider better Mac packaging. - Good, fast, usable search: New spider is the beginning, there is quite a bit more work to do on bugs, stop-words, metadata-in-parent-node, etc. - Sufficient user-level functionality in the base install: WoT, Freetalk, FlogHelper, possibly Fritter, something like Freemail. Fully integrated, so you can comment on a blog, and send private messages to people chatting on Freetalk. Probably a searchable spam-proof distributed filesharing system of some kind, also based on WoT. - Better big freesite insertion, in particular containerisation of big sites into multiple containers. Essentially this means debugging saces' work. - Consider building a proper plugins API. Sandboxing is IMHO optional as it's very hard to get right in any case. - Filters for most popular MIME types, and detailed explanations of what may be an issue for the rest. - Make absolutely sure that no incriminating data is stored plaintext, and auto-backup the database (and if necessary the download list). - Consider scaling issues with seednodes. - Greatly improved user interface!
Now, what are the inter-related structural challenges, design decisions that affect several of these issues? Most of the above are isolated from each other and can just be implemented. But there are some big overlaps: - The mobile attacker source tracing attack. All fixes for this are fairly radical. - Load management. This is related to the radical changes related to the first. - Better bursting support. Closely related. - Passive requests and long-term requests. Again these are closely related to this whole nexus. As I see it there are 5 possible solutions for mobile attacker source tracing: - Tunnels. Possibly combined with selective reinserts, and possibly used only for predictable blocks. High performance cost: Likely we would use tunnels for predictable blocks on insert only, make most blocks not predictable, and not for requests. Does not therefore really solve the reinsert problem, unless reinserts are selective i.e. small. And all tunnel schemes have their own various vulnerabilities. Might be necessary for frequent small inserts (e.g. chat) even with some of the below however... - Set a time and only route to nodes connected before that point. Does not work for big stuff. Similar caveats to tunnels but not as big a performance cost. - Instant route to the endpoint then return data gradually, possibly rerouting and buffering if necessary, possibly with offline support, trickling back data after the node reconnects. Requires capacity-based load management, which may be tricky on opennet, but at least we can quantify it. My preferred option. Both fast and secure, but rerouting return path may be difficult. On the other hand for an initial implementation we might be able to avoid that. - Advance a hop only when the originator has told us to. Later hops don't need to consider time, which should minimise data return rerouting costs. However it does require the originator is online - or some other node it has nominated to act on its behalf (e.g. if we make the metadata decryptable with a key which is told to that node). Hence doesn't solve uptime-related attacks. - Instant route of the whole bundle and then activate individual routes to return the data. Possibly with some sort of rerouting. Similar uptime issues to previous option. Most of these options interact with passive requests. The third one in particular interacts with load management. The third one is also closely related to better bursting support. IMHO the correct solution is: - Instant route to the endpoint then return data gradually: Option 3. - Capacity based load management. - Darknet nodes with high trust levels can do big single-route bursts and fetch a file all at once; opennet users will have to make multiple such bursts and as such reduce their security, but in a quantifiable way. - Optional integration with passive requests. - Optional low-uptime/trickle-over-restart support, particularly on darknet. However, a shortcut might be: - Partial reinsert support. - Set a time and only route to nodes connected before that time. Only works with small stuff, so no protection for big downloads, or predictable big inserts; but we avoid predictable big inserts. - No low-uptime support. - Relatively poor burst support as load management would be exactly as it is now. Partial reinsert support we will need anyway. Time-based routing restriction would be relatively easy to implement. So we could have protection for both original inserters and small partial reinserts at very low cost. However, the real solution solves a much wider range of issues. In any case we will eventually need tunnels for top blocks and chat. However tunnels are likely to be so slow that we do not want to use them for anything else! I therefore propose the following tentative roadmap: 0.8.0: - Lots of bug fixes. - Lots of minor usability improvements. - Freetalk, WoT and FlogHelper. - Fixed wininstaller. - All easy improvements for data persistence. - Easy full reinsert. - Make absolutely sure nothing is stored plaintext. - Implement auto-backup of database. 0.8.5: - Partial reinsert support. - Time-based protection. - Filters for many popular MIME types. - Significantly reduced disk I/O. 0.9.0: - Freemail. - Distributed file-oriented search. - Very fast site search. - Fix the Pitch Black attack. - Efficient transport layer. - Passive requests. - Filters for most popular MIME types. (Obviously we are likely to have some restrictions e.g. an H.264 filter is unlikely due to lack of access to docs, although we may have Theora/VP8, and PDF will be an incomplete but usable subset) 0.10.0: - Capacity based load management. - Instant route to endpoint then return the data gradually. - Very simple tunnels implementation for chat, top blocks, etc. - Consider transport plugins as very popular on uservoice, but most likely postpone for post-1.0. 1.0: - LOTS OF DEBUGGING! - Consider plugins API. - Consider seednodes scaling issues. - Lots of work on the UI. - More filters if possible. Post-1.0: - Transport plugins. - Offline support in both passive requests and returning data. - Plugin sandboxing. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 197 bytes Desc: This is a digitally signed message part. URL: <https://emu.freenetproject.org/pipermail/devl/attachments/20100709/37ceb962/attachment.pgp>
