YAY! ----- Forwarded message from Stefan Majewski <[EMAIL PROTECTED]> -----
Date: Wed, 4 Jun 2003 20:34:43 +0200 (CEST)
From: Stefan Majewski <[EMAIL PROTECTED]>
X-X-Sender: [EMAIL PROTECTED]
To: Greg Wooledge <[EMAIL PROTECTED]>
Subject: Re: [freenet-dev] Silence after Pending message?
X-Spam-Status: No, hits=-25.4 required=5.0
tests=IN_REP_TO,QUOTED_EMAIL_TEXT,REFERENCES,REPLY_WITH_QUOTES,
USER_AGENT_PINE
version=2.53
X-Spam-Level:
X-Spam-Checker-Version: SpamAssassin 2.53 (1.174.2.15-2003-03-30-exp)
On Tue, 3 Jun 2003, Greg Wooledge wrote:
> malcolm handley ([EMAIL PROTECTED]) wrote:
>
> > Where is the FCP documentation these days?
>
> The last time I asked this (a couple weeks ago), the answer was that
> it had vanished in the last round of web site problems, so the only
> definitive resource was the source code.
Hi Greg,
I found those two on my disk. Is this what you were talking about? I think
especially the FreenetClientProtocol.txt is not exactly up to date, but if
it is lost it might be better than nothing.
Unfortunately I don't remember right now the emailaddress I am known to
the list, so I am directing this directly to you. If you like, it would be
nice if you could hand it to the persons who need the docs.
Cheers and hth,
Stefan
----- End forwarded message -----
--
Greg Wooledge | "Truth belongs to everybody."
[EMAIL PROTECTED] | - The Red Hot Chili Peppers
http://wooledge.org/~greg/ |
Abstract
The FreenetClientProtocol (FCP) is designed to abstract the basics of
Freenet so that client developers do not have to track the main
Freenet protocol. FCP should be the bare bones of Freenet - metadata
handling is not included in FCP though an extension to FCP may come
about at a later date to avoid writing metadata handling libraries in
many languages.
Note
This protocol is never meant to go across a network - only via the
loopback. Nodes should not accept FCP connections from hosts other
than localhost by default.
Basics
By default FCP is port 8481, but any client that uses FCP should leave
this configurable, because this may be changed in the node's
configuration file or by some future FCP revision.
FCP follows the FNP setup for session and presentation.
In the following, numbers are always hex-encoded and fields in
square-brackets are optional.
FCP allows one transaction per connection, after which the connection
is torn down. At the beginning of each connection, the client must
send these 4 bytes:
00 00 00 02
These are the 2-byte session identifier and the 2-byte presentation
identifier. In the future, different identifiers may be used to allow
alternate syntaxes or encrypted FCP connections from remote hosts, for
example.
After sending the session and presentation identifiers, the client
sends a message to initiate the transaction, then waits for one or
more messages from the node until the transaction is complete.
Messages are a series of lines terminated by LF or CRLF, in this form:
Header
[Field1=Value1]
.
.
[FieldN=ValueN]
EndMessage
Message summary
This is the complete set of client to node messages, with the possible
node to client responses (only the headers are listed).
* ClientHello
+ NodeHello
* ClientGet
+ URIError
+ Restarted
+ DataNotFound
+ RouteNotFound
+ DataFound
o DataChunk
* ClientPut
+ URIError
+ Restarted
+ RouteNotFound
+ KeyCollision
+ Pending
+ Success
* GenerateCHK
+ Success
* GenerateSVKPair
+ Success
* ClientDelete
+ Success
Additionally, the node may respond to any client message with a
FormatError, meaning the command was not understood, and the node may
responsd at any time with a Failed, indicating a fault in the node
itself:
(Node -> Client)
FormatError
[Reason=<descriptive string>]
EndMessage
(Node -> Client)
Failed
[Reason=<descriptive string>]
EndMessage
Failed and FormatError will not be discussed in the remainder of this
document. Clients should be prepared to handle a Failed at any time,
and a FormatError as teh response to any client message. Either of
these messages terminates the transaction and the connection.
Handshaking
This is totally optional for the client. Note that this counts as a
transation and thus the connection is torn down afterwards.
(Client -> Node)
ClientHello
EndMessage
In response the node sends the following message:
(Node -> Client)
NodeHello
Protocol=<number: protocol version number. Currently 1>
Node=<string: freeform: Description of the nodes>
EndMessage
Requesting
(Client -> Node)
ClientGet
URI=<string: fully specified URI, such as freenet:[EMAIL PROTECTED]>
HopsToLive=<number: hops to live>
EndMessage
The client is now in the waiting state. The node may return one of the
following messages:
* URIError: Invalid Freenet URL. The transaction is terminated.
* Restarted: The client should continue waiting.
* DataNotFound: The transaction is terminated due to not being able
to find data.
* RouteNotFound: The transaction is terminated due to not being able
to find a route.
Otherwise a DataFound message is returned:
(Node -> Client)
DataFound
DataLength=<number: number of bytes of metadata + data>
[MetadataLength=<number: default = 0, number of bytes of metadata>
EndMessage
After a DataFound message the data itself is sent in chunks:
(Node -> Client)
DataChunk
Length=<number: number of bytes in trailing field>
Data
<@Length bytes of data>
At any time when the full payload of data has not been sent a
Restarted message may be sent. This means that the data to verify and
the transfer will be restarted. The client should return to the
waiting state, and if a DataFound is then received, the data transfer
will start over from the beginning. Otherwise, when the final
DataChunk is received, the transaction is complete and the connection
dies.
Inserting
(Client->Node)
ClientPut
HopsToLive=<number: hops to live>
URI=<string: fully specified URI, such as freenet:[EMAIL PROTECTED]>
DataLength=<number: number of bytes of metadata + data>
[MetadataLength=<number: default = 0, number of bytes of metadata>]
Data
<@DataLength number of bytes>
If the client is inserting a CHK, the URI may be abbreviated as just
[EMAIL PROTECTED] In this case, the node will calculate the CHK. The node must get
all of the trailing field before it can start the insert into Freenet.
The node may reply with one of the following messages:
* URIError: Invalid Freenet URL. The transaction is terminated.
* Restarted: The client should continue waiting.
* RouteNotFound: The transaction is terminated due to not being able
to find a route.
* KeyCollision: The transaction is terminated due to a document with
the same key already existing in Freenet. This message contains a
URI field with the Freenet URI of the document.
* SizeError: The transaction is terminated due to the data being too
large for the key type; all non-CHK keys have a limit of 32 kB of
data.
During an insertion, multiple Pending messages may be returned. These
messages signal that the data is being successfully inserted, but
insertion is not complete, and the node has not received a StoreData
message yet:
(Node -> Client)
Pending
URI=<string: fully specified URI, such as freenet:[EMAIL PROTECTED]>
[PublicKey=<string: public key>]
[PrivateKey=<string: private key>]
EndMessage
When the node receives a StoreData message (and thus insertion is
complete), a Success message is returned with the Freenet URI of the
new document and possibly a private/public keypair, if the inserted
document was an SVK. See the section on key generation about this.
(Node -> Client)
Success
URI=<string: fully specified URI, such as freenet:[EMAIL PROTECTED]>
[PublicKey=<string: public key>]
[PrivateKey=<string: private key>]
EndMessage
Key generation
These messages allow a client to generate keys. This does not affect
Freenet at all - the calculations are carried out at the node.
Key generation requests are done via a GenerateKey message. Either a
CHK or an SVK keypair can be generated:
(Client -> Node)
GenerateCHK
DataLength=<number: number of bytes of data + metadata>
[MetadataLength=<nubmer: defaul = 0, number of bytes of metadata>]
Data
<@DataLength number of bytes>
The node calculates the CHK as it would do if inserting, but instead
returns it. This completes the transaction:
(Node -> Client)
Success
URI=<string: fully specified URI, such as freenet:[EMAIL PROTECTED]>
EndMessage
The format for generating SVKs is very similar but generates a pair of
keys (public and private) which are independent of any data. This is
generally used for setting up SSKs:
(Client -> Node)
GenerateSVKPair
EndMessage
The node generates a key pair and returns:
(Node -> Client)
Success
PublicKey=<string: public Freenet key>
PrivateKey=<string: private Freenet key>
EndMessage
The PublicKey and PrivateKey are returned as Freenet-base64 encoded
strings. These can be used to construct URIs for requesting or
inserting SSKs:
(insert) freenet:SSK@<PrivateKey>/<name>
(request) freenet:SSK@<PublicKey>/<name>
_________________________________________________________________
------------------------------------------------------------
This document describes the FCP FEC commands as currently
implemented in fred.
------------------------------------------------------------
FCP FEC Proposal rev. 1.1
[EMAIL PROTECTED] 20030125
I. INTRODUCTION:
This proposal presents a set of new FCP commands that can be used to
encode and decode files using forward error correction (FEC).
FEC is a way of encoding packetized data files with extra error
recovery information which can be used to reconstruct lost packets.
In this document I will refer to the packets containing data as "data
blocks" and those containing error recovery information as "check
blocks".
One of the objectives of this design is to separate FEC encoding and
decoding from inserting and retrieving the data and check blocks
to/from Freenet. By separating encoding and decoding from insertion
and retrieval I sidestep the problem of having to hold FCP connections
open while waiting for large amounts of data to be fetched from /
inserted into Freenet.
For a given maximum block size, some FEC algorithms can only
practically handle files up to a certain maximum size. The design
uses segmentation to handle this case. Large files are divided into
smaller segments and FEC is only done on a per segment basis. This
compromise provides a least limited redundancy for large files.
II. Assumptions
This proposal doesn't specify any particular FEC algorithm.
However the following assumptions are implicit in the design:
A. For a given segment with k data blocks and n - k check blocks, it
must be possible to decode all k data blocks from any k of n data or
check blocks.
B. Encoder and decoder implementations must be completely specified by
an implementation name and a file length. No other parameters can be
required to instantiate the encoder or decoder.
C. Within a segment all data blocks must be the same size and all
check blocks must be the same size. The check block and data block
sizes are not required to be the same however. Smaller trailing
blocks must be zero padded to the required length.
D. The encoder may ask for extra trailing data blocks. These extra
blocks must contain zeros.
II. Proposed FCP FEC commands
convention: All numbers are hexadecimal
convention: Blocks are numbered from 0 to n - 1 on a
per segment basis.
Data blocks are indices 0 to k - 1.
Check blocks are indices k to n - 1.
A. Helper messages, SegmentHeader and BlockMap
A SegmentHeader message contains all the information necessary to FEC
encode or decode a segment of a file. SegmentHeaders may contain FEC
implementation specific fields. They are guaranteed to contain the
documented fields given in the example SegmentHeader
message below:
SegmentHeader
FECAlgorithm=OnionFEC_a_1_2 // The FEC implementation name
FileLength=170000 // Total file length
Offset=0 // Offset from the start of the file
BlockCount=6 // Number of data blocks
BlockSize=40000 // Data block size
CheckBlockCount=3 // Number of check blocks
CheckBlockSize=3 // Check block size
Segments=1 // Total number of segments
SegmentNum=0 // Index of the current segment
BlocksRequired=6 // Blocks required to decode this segment
EndMessage
Client code should not rely on any undocumented fields.
BlockMap messages are used to list the CHKs of the data and check
blocks for a segment.
Here's an example:
BlockMap
Block.0=freenet:[EMAIL PROTECTED],jGonMeCCz1GCHde5bc1t~w
Block.1=freenet:[EMAIL PROTECTED],5cxWki4YzWyKP0s3g9~Vow
Block.2=freenet:[EMAIL PROTECTED],Il2ztTbQImZvVlsnuDq-8Q
Block.3=freenet:[EMAIL PROTECTED],2D5~Mm~MjAfup3edGXy6Eg
Block.4=freenet:[EMAIL PROTECTED],J7HpLvPscLyW3Sc6Nq2S5g
Check.0=freenet:[EMAIL PROTECTED],4ZX2inJ7gg0EectTxPYRSg
Check.1=freenet:[EMAIL PROTECTED],xjJCPsCxpnw9lyNI2VBRGA
EndMessage
B. FECSegmentFile
The FECSegmentFile message is used to generate the segment headers
necessary to encode a file of a given length with a specified FEC
algorithm.
FECSegmentFile
AlgoName=OnionFEC_a_1_2
FileLength=ABC123
EndMessage
If this command is successful one or more SegmentHeader messages are sent in order
of ascending SegmentNumber.
The client can detect when the last segment has been sent by checking the SegmentNumber
and Segments field of each received SegmentHeader.
On failure a Failed message is sent.
C. FECEncodeSegment
The FECEncodeSegment message is used to create check blocks for a
segment of a file. The RequestedList field contains a comma delimited
list of the requested check blocks. If the list is empty or omitted completely
all the check blocks are sent.
The SegmentHeader for the requested segment must sent as data in the
trailing field of the FECEncodeSegment message, preceding the raw
segment data to encode.
FECEncodeSegment
[RequestedList=0,A,F]
DataLength=<SegmentHeader length> + <segment length>
Data
< SegmentHeader >
< raw data >
If the encode request is successful, the server sends a BlocksEncoded
confirmation message, followed by DataChunk messages for the encoded
blocks. Check blocks are sent in order of ascending index.
e.g:
BlocksEncoded
BlockCount=3
BlockSize=40000
EndMessage
DataChunk
...
(3 * 40000 = 0xC0000 worth of DataChunk messages)
...
If the requests fails a Failed message is sent.
Note:
The total segment size
(SegmentHeader.BlockSize * SegmentHeader.BlockCount) can exceed the
length of the data present in the last segment. In this case partial
blocks should be zero padded and extra zero filled blocks should
be sent if requested.
D. FECDecodeSegment
The FECDecodeSegment message is used to decode missing data blocks for
a segment of a file. The RequestedList field contains a comma
delimited list of the requested blocks. Similarly, BlockList and
CheckList contain the indices of the data blocks and check blocks that
are being sent to decode from. All index lists must be in ascending
order.
Note that RequestedList may contain check block indices as well
as data block indices. If check indices are present the FCP server
will encode the requested check blocks and send them after the decoded
data blocks. This feature was added to support healing re-insertion.
The SegmentHeader for the segment must sent as data in the trailing
field of the FECDecodeSegment message preceding the blocks to
decode from.
FECDecodeSegment
BlockList=0,2,3,5,6
CheckList=9,c
RequestedList=1,4
DataLength=<SegmentHeader length> + <total length of data and check blocks>
Data
< SegmentHeader >
< raw data blocks in order of index >
< raw check blocks in order of index >
If the decode request is successful, the server sends a BlocksDecoded
confirmation message, followed by DataChunk messages for the decoded
blocks. The decoded data blocks are sent in order of ascending index.
e.g:
BlocksDecoded
BlockCount=2
BlockSize=40000
DataLength
EndMessage
DataChunk
...
(2 * 40000 = 0x80000 worth of DataChunk messages)
...
If the requests fails a Failed message is sent.
E. FECSegmentSplitFile
The FECSegmentSplitFile command generates a list of SegmentHeaders and
BlockMaps from SplitFile metadata. The SplitFile metadata should
be sent as data in its trailing field.
e.g.:
FECSegmentSplitFile
DataLength=<SplitFile metadata length>
Data
<SplitFile metadata>
If successful the FCP server sends back one or more pairs of
SegmentHeader and BlockMap messages in order of ascending segment number.
The client can tell how many pairs are coming by inspecting the Segments
field in the first SegmentHeader.
On failure a Failed message is sent.
F. FECMakeMetadata
The FECMakeMetadata command creates metadata for a SplitFile from
a list of SegmentHeader BlockMap pairs sent as data in it's trailing field.
The list must be in order of ascending segment number.
The Checksum field is optional. If provided, it should contain an
SHA-1 hash of the entire file. The hash can be generated using the
GenerateSHA1 FCP command, though in most cases it will be faster to
use a local library if one is available.
Clients are strongly encouraged to set the Checksum value.
FECMakeMetadata
Description=file
CheckSum=c8766a76f81727b2b7c131e9ad6a53c8aa820a1e
MimeType=text/plain
DataLength=<total length of all SegmentHeaders and BlockMaps>
Data
<SegmentHeader, BlockMap pairs>
III. Usage cases
Here's a brief description of how these messages are used to
encode and decode files.
A. Encoding
The client code sends a FECSegmentFile with the file's length and
saves the returned SegmentHeaders.
For each segment, it calls FECEncodeSegment with the segment's data
blocks and saves the returned check blocks.
It then inserts the data and check blocks into Freenet using the
existing ClientPut command, and saves the resulting CHK URIs.
After the data blocks and check blocks for all segments have been
inserted, the client code sends a FECMakeMetadata command with the
SegmentHeaders and BlockMaps listing the inserted blocks. It saves
the SplitFile metadata and inserts it into Freenet.
The client may optionally specify the SHA-1 checksum for the file
in the FECMakeMetadata command. In this case, the Checksum value is
included in the InfoPart of the generated metadata.
Done.
A. Decoding
The client code sends a FECSegmentSplitFile with the SplitFile metadata
and saves the SegmentHeader,BlockMap pairs.
For each segment, it uses the CHK lists int the BlockMap to
download the minimum number of data and check blocks
as given by SegmentHeader.BlocksRequired. It then calls FECDecodeSegment to
decode the missing data blocks.
Finally, it concatenates the data blocks together, possibly ignoring trailing
padding blocks.
If a checksum value for the file was specified in an InfoPart in the
metadata somewhere along the redirect chain, that value can be used to
check the integrity of the reconstructed file by calculating its SHA-1
hash.
Done.
III. Changes to SplitFile metadata format.
0) Deprecate the BlockSize field, since check blocks are not necessarily the
same size as data blocks and blocks may be different sizes across segments.
[None of my code depends on this, but it has not been removed from the code.]
1) Add an AlgoName field. This is the name for the decoder and encoder implementation,
that can be used to decode or re-encode the file. This replaces decoder.name and
decoder.encoder
in the previous implementation.
[This has been done.]
2) Remove the decoder FieldSet.
[freenet.client.metadata.SplitFile detects that this field is
present so that fproxy / SplitFileRequestServlet can
warn about unsupported obsolete encoders.]
* SplitFile.Graph is currently not being used and is not implemented.
IV. GenerateSHA1
This FCP command isn't part of the FEC commands but it's useful
for FEC client authors.
GenerateSHA1
DataLength=abc123
Data
<data>
On success a Success message is sent, with the hash in its
SHA1 field.
On failure a Failed message is sent.
------------------------------------------------------------
CHANGE HISTORY
------------------------------------------------------------
20030125
- optional checksum in FECMakeMetadata
- optional re-encoding of check blocks for healing in FECDecodeSegment
- brief example of GenerateSHA1.
pgp00000.pgp
Description: PGP signature
