I'm assuming that the goals of an optimized transport for Avro RPC are
something like the following:
* Framing should be efficient, easy to implement.
* Streaming of large values, both as part of a request and as a response is
very important.
* Being able to have multiple concurrent requests in flight, while also
being able to have ordering guarantees where desired is necessary.
* It should be easy to implement this in Java, C, Python, Ruby, etc.
* Security is or will be important. This security can include authorization
as well as privacy concerns.
I'd like to see something based largely upon RFC 3080, with some
simplifications and extensions:
http://www.faqs.org/rfcs/rfc3080.html
What does this get us?
* This system has mechanisms in place for streaming both a single large
message and breaking a single reply up into multiple answers, allowing for
pretty flexible streaming. (You can even mix these by having an answer that
gets chunked itself.)
* Concurrency is achieved by having multiple channels. Each channel
executes messages in order, so you have a good mechanism for sending
multiple things at once as well as maintaining ordering guarantees as
necessary.
* Reporting errors is very clear as it is a separate response type.
* It has already been specified pretty clearly and we'd just be evolving
that to something that more closely matches our needs.
* It specifies sufficient data that you could implement this over
transports other than TCP, such as UDP.
Changes, rough list:
* Use Avro-encoding for most things, so the encoding of a message would
become an Avro struct.
* Lose profiles in the sense that they're used in that specification since
we're just exchanging Avro RPCs.
* Do length prefixing rather than in the header, so that it is very
amenable to binary I/O at high volumes.
* No XML stuff, just existing things like the Avro handshake, wrapped up in
messages.
* For now, don't worry about things like flow control as expressed in RFC
3081, mapping of 3080 to TCP.
* Think about adding something for true one-way messages, but an empty
reply frame is probably sufficient, since that still allows reporting errors
if needed (or desired).
* May well need some extensions for a more flexible security model.
* Use Avro RPC stuff to encode the channel management commands on channel 0
rather than XML.
RFC 3117 (http://www.faqs.org/rfcs/rfc3117.html) goes into some of the
philosophy and thinking behind the design of RFC 3080. Both are short and
easy reading.
- Bruce
