On Sunday 30 August 2009 12:52:31 Jonas Bengtsson wrote: > This is description of my work on node to node file transfers. Most of > the work where made a couple of weeks ago. The last few weeks I have > been very busy but now I have got some time hour to put it together. > > I have chosen an architecture where the file transfers are stored in > the request queue. The advantage of this is twofold. It makes it easy > to send and receive files over FCP. It is also makes it easy to make > the transfers appear on the uploads and downloads pages, which is where > the users expect to find them. > > To achieve this two new subclasses of ClientRequest has been created, > ClientSend and ClientReceive. A ClientSend is created when a > user offers a file to a friend. When it is created a a N2N file > offer message is sent to the friend. This results in a UserAlert being > registered at the friend. If the friend accepts the offer a > ClientReceive is created and a N2N file offer accept is sent. If > the friend rejects the offer a N2N file offer rejected is sent and > nothing is created in the request queue. > > Previously two Maps where used i DarknetPeerNode to keep track of > sent and received file offers. The Map with sent file offers have been > removed. The entries in it have been replaced with ClientSend entries in > the request queue which are persistent. The Map with received offers has > been kept since a ClientReceive is only created after an offer has > been accepted. This Map is recreated at startup by reading file offers > from the extra peer data directory. > > Since ClientSender uses a BulkTransmitter and a PartiallyReceivedBulk > it needs references to the MessageCore, a DarknetPeerNode and to the > node to node ByteCounter. These references are made transient and are > not stored in the database. Instead entries for the byte counter and > the message core have been added to the ClientContext. A reference to > the PeerMananger has also been added to the ClientContext. The > ClientSender stores the node identity and used the PeerMananger to look > up the DarknetPeerNode. ClientReceiver works in a similar way. > > ClientSend finds out about new progress by ClientSender by listening on > ClientEvents. ClientSender generates a SplitFileProgressEvents when the > transfer of a block have succeeded. For N2N file transfers there is no > concept of split files, but using the existing event makes it easy to > generate SimpleProgress-messages which can be sent to FCP-clients. The > following conventions have been followed to apply a > SplitFileProgressEvent to direct file transfers > > - Required always equals Total > - FinalizedTotal always equals true. > - Failed and FatallyFailed equals zero. > > ClientReceive and ClientReceiver communicates in the same way. > > To create a ClientSend which sends a file offer the ClientSend > FCP-message can be used > > ClientSend > Filename=name of file > ClientToken=arbitrary string > Global=true of false > Verbosity=0 or 1 > Identifier=arbitrary string > NodeIdentifier=node identifier > EndMessage > > To find file offers a user can use the WatchFeeds FCP-message > WatchFeeds > EndMessage > If there is a file being offered this returns a ReceivedFileOfferFeed > which includes such information as uid, filename, mimetype, node > identifier and size. > > To create ClientReceive which sends file offer accepted the > ClientReceive FCP-message can be used. > > ClientReceive > Uid=long from RecievedFileOfferFeed > Global=true of false > ClientToken=arbitrary string > Verbosity=0 or 1 > Identifier=arbitrary string > NodeIdentifier=node identifier > EndMessage > > In repy to ClientSend and ClientReceive messages a PersistentSend > and a PersistentReceive is returned respectively. > > At the moment N2N file transfers always use forever persistence. Does > it make sense to allow them to have reboot and connection persistence > as well? It is also only possible to send and receive files to the > hard drive of the node. > > This is work in progress. It will take some time before it > works properly. At the moment it can serve as an evaluation of one > possible architecture. > > The major problems that I haven't solved are related to threading. File > transfers works under ideal conditions, if both users are online when > the file offer is accepted. There should be some way to automatically > check if a file transfer can be started. At the sender the condition > could be that the friend is online and that the file offer has been > accepted. At the receiver the condition only needs to be that friend > is online. How should this be achieved? Should a RequestScheduler be > used or is there a more simple way?. > > There should also be a way to resume failed transfers. Since the data > is sent and received sequentially and every block is acknowledged the > only thing needed to restart a transfer is the number of blocks > sent/received and this is stored in the ClientSender and > ClientReceiver. For this purpose I have created a constructor in > PartiallyReceivedBulk that takes an integer and sets the n first bits > in the BitArray to 1. > > The way the BulkTransmitter informs the ClientSender also needs to be > fixed. At the moment the BulkTransmitter is given a DBJobRunner > when it is created and it creates a new DBJob every time a block has > been successfully sent. This doesn't only lead to extremely ugly code > but also to very poor performance. The same problem also exists with the > BulkReceiver and ClientReceiver. > > The exception handling is also poor, some exceptions are not handled, > others are handled at the wrong place. Maybe the new exceptions > SendException and ReceiveException should be created to mimic > InsertException and FetchException? > > The code is available at > http://github.com/ljb/fred-staging > > I will be very busy the next two weeks and won't be doing any coding. > Later however I might have a few hours over every week that I > could use to work on persistent file transfers. > Okay, so at present: - Until the destination user accepts the transfer, it is not persisted. - After that, it is persisted. - Transfers are shown on the downloads/uploads pages??? - If a peer goes offline, we try the transfer again from the beginning when they reconnect??? - If we restart, we try the transfer again from the beginning after restart???
Is there any good reason not to persist offers we have made that have not yet been persisted, and offers we have received that the user has not yet accepted? -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 835 bytes Desc: This is a digitally signed message part. URL: <https://emu.freenetproject.org/pipermail/devl/attachments/20090831/91cb76e6/attachment.pgp>
