Hello, This is an update on the first stage of my project. I have been working out the details regarding modifications in the core to simplify the implementation of the transport plugins. This refactoring stage will involve some minor modifications and some major ones. These are based on the discussions on #freenet and #freenet-chat. However I would like to avoid doing changes that would bring about major differences in the way things function at present.
I have planned to finish refactoring by May 21 (official coding begins for GSoC). I intend to do that in 3 stages. I have also simultaneously discussed the entire implementation of transport plugins here. I will review the feedback and this report, to evaluate what constitutes the refactoring. After those changes are made I can work on implementing the needed core level changes, which mostly involve design for streams. 1) Cryptography and Node This will involve mainly the classes NodeCrypto and Node. I intend to make NodeCrypto function purely for keys and not do any network related task. Presently it also handles binding to the port, selection of ports for Darknet mode and Opennet mode. I am considering moving these portions to TransportManager and PortManager classes. TransportManager - A registry of registered transports with other information. Initially I ll dump the refactor changes of UDP into it. Later on, once UDP is ready as a plugin, along with TCP, it ll be a generic class for any transport. There will be two objects of this - one each for opennet and darknet modes. NodeCrypto e.g. queries TransportManager to find all the transports' addresses (as strings) to put in the noderef. Now UDPSocketHandler is equivalent to a UDP transport plugin, only it's currently part of the core of freenet. So for the moment TransportManager can communicate with UDPSocketHandler. That I believe will be an intermediate step before actually converting UDPSokcetHandler to a plugin. PortManager - A registry of all active ports used by freenet - 1) List of ports on fproxy page 2) List of ports for UPnP Must contain - 1)Port number 2)Socket(TCP/UDP) 3)Forwarded 4)Description 5)Transport reference However PortManager is second priority, and will not be addressed if time does not permit. Also a major refactoring change would be regarding the fact that objects no longer take a socket or a port for granted. All queries must go through the TransportManager. I would need help at certain places to figure out how this can achieved. For e.g. NodeCrypto/Node/some other classes assume that an unique darknet port exists. toad suggested we use a debugging identifier here(some default number, etc). That makes it simpler for me too. 2) Messages and Packets These part have been discussed on #freenet(and #freenet-chat) and here are some issues and conclusions- a. The existing architecture supports only packet based transports. The first step would be to draw a clear distinction between packet-based(connectionless sockets) and stream-based(connection-based sockets). Thus we will need separate classes for streams(with a base class for streams and packets), that will handle Message-in-flight and Packets-in-flight, tracking them, allowing for transports to be switched midway. This will be handled differently in streams. For Packets the existing classes can be used, however with modifications to address the following. b. For streams we need two features - To handle messages as a whole entity, and acknowledge once an entire message has been received; and To support partial messages, if for e.g. we have just switched from UDP to TCP or we are sending a really large message on TCP. c. NewPacketFormat currently 1) takes a byte[] from UdpSocketHandler and decodes it 2) creates and sends a packet. Now the idea is to create NewStreamFormat that handles streams, and also provide a pair of streams StreamConnectionHandler for the plugin to write/read. NewPacketFormat handles the list of message-in-flight. A new object must be used for this purpose while NewPacketFormat performs lesser tasks. Both NewPacketFormat and NewStreamFormat can communicate with it. Question: MessageWrapper tracks a message-in-flight. But is it really necessary to create a base class for MessageWrapper and create StreamMessageWrapper too. Or would it be simple to make necessary modifications(if there are any) to MessageWrapper. Please explain the purpose of doing so, as I have very little understanding, as of now, of the object Message. Also how it will be treated differently for streams against packets. d. Another issue is regarding FNPPacketMangler. This is an important aspect of freenet, as it deals with the handshake between two nodes, and the exchange of cryptographic keys. So the issue being whether per NodeCrypto we have an object or per transport we have an object, of the FNPPacketMangler type. And assuming these are packet based transports. Question: How can the equivalent of FNPPacketMangler be created for stream based connections. I don't think this was discussed at all or in detail on the IRC. Some suggestions here would be helpful. My knowledge is that the JFK protocol works for packet based transports, specifically UDP. How would this function in case of streams? e. Another thing that needs clarification is regarding who decides when to send the packets. Since stenography is all about the timing and the format of the packet, the plugin must have control. However that would make some other transport plugins difficult to write. toad suggested the following. To quote him- "PacketSender constantly loops over peers to send packets to them. We'll probably have two separate mechanisms for packet-based - one where we are just wrapping packets with different headers, so the node decides when to send a packet, and one where the plugin decides when to send a packet and demands data to fill it from the node. The latter is "constant bitrate", or really accurate fakery; it's a long term project, as is how to fill up the packets when there's nothing to send." This sounds acceptable, however when I get there I will discuss how this should be implemented. But I believe that it will not affect the initial stages of the project. Question: How does the same work for streams? Firstly the transport must have its own flow control mechanism to decide when to send. Also there must be a way to let the node know this. I am guessing NewStreamFormat handle this? It must also create StreamConnectionHandler for the plugin to read/write. Must also do encryption using existing BlockCiphers? It must convert the encrypted data into a stream? I suppose I can draw parallels to NewPacketFormat and do it. 3) Plugins, streams, transports I will need to create a base class for transports - TransportPlugin. This is designed as per the developer's requirements. However the major idea is that it would open sockets, and listen for connections. For a packet-based plugin as mentioned it is called by PacketSender to send data. If it wishes to for stenographic reasons, can send dummy data by itself. Also it will handle all necessary parsing and encoding for whichever protocol it is trying to mimic. It creates its own threads to copy data from freenet and send it. For a stream-based plugin, it runs its own loop of getting/writing data from/to StreamConnectionHandler(node-side), encoding/decoding, and sending/receiving to its socket. It must have its own flow control mechanism. To be discussed- One major point I haven't discussed is how multiple transports work. For e.g. a node might be able to communicate using several transports, while its peer might not be able to do so. It had been the case that for datagrams one needed a single port that sent messages to all its clients. But with multiple transports, the node needs a record of active transports for each peer. Apart from this we need to discuss how the messages are distributed between transports. This also means some transports might be slow, or the user prefers one of the transports. Although I wish to address it within the timeline of GSoC, I am not sure how it will turn out. But for the moment my idea is to be able to choose the needed transport, although the framework to use them simultaneously or switch from one to another will be supported. If I find that the logic to implement selection of transports is easy, then I can implement it. But I assume that needs some usage analysis as well. Based on that we can comment. My objective is to atleast get a functional freenet that will atleast figure one way to connect to a peer, and this can be different for different peers. I need your opinions on this. But atleast some modification of PeerNode to keep track of which transport works is a must. This is a haphazard report, that has questions, issues as well as conclusions. Based on feedback and suggestions, for the next two weeks I wish to work on refactoring changes to support the above. Also thanks to toad for making it very simple for me, and the fact that over a month, all his design implementation ideas mentioned on email and irc are consistent and complementary to each other. I will work on a separate branch and I am not sure how it follows from now on, as I submit code. The present idea is to create a branch for every task, to be completed every week. Comments and suggestions are a must! Regards, Chetan Hosmani
