On Apr 9, 2010, at 11:56 AM, Bo Shi wrote: > On Fri, Apr 9, 2010 at 2:35 AM, Bruce Mitchener > <bruce.mitche...@gmail.com> wrote: >> Doug, >> >> I'm happy to hear that you like this approach! >> >> Allocation of channels seems to be something specific to an application. In >> my app, I'd have a channel for the streaming data that is constantly >> arriving and a channel for making requests on and getting back answers >> immediately. Others could have a channel per object or whatever. > > One ubiquitous protocol that shares many of the same requirements and > properties (in particular multiplexed channels over a transport) is > SSH. Their channel mechanism may provide additional inspiration: > [http://tools.ietf.org/html/rfc4254#section-5]. One interesting bit > is that SSH doesn't have an explicit channel for control commands, > instead they create additional message types for control messages that > aren't associated with any channel. It's only a minor distinction > though. >
One flaw in SSH is that the bandwidth over a WAN is often pathetic. Because it has multiple channels, it implements its own flow control and receive windows. The effective bandwidth is the window size divided by the RTT. Many implementations have a hard-coded 64KB receive buffer -- over a connection with a 30ms RTT this is peak data throughput of 2MB/sec. The latest implementation versions of SSH patch the issue by having an automatically growing buffer, but the in memory buffer size required for high throughput transfer over higher latency links is large. http://www.docstoc.com/docs/18191581/High-Performance-Networking-with-the-SSH-Protocol/ http://www.psc.edu/networking/projects/tcptune/ http://www.psc.edu/networking/projects/hpn-ssh/ This arises out of flow control to make sure that channels do not interfere with each other too much. HTTP does not have this problem -- it just uses TCP's flow control and its only multi-channel-like feature -- http pipelining, doesn't bother with flow control. Whatever Avro does for a custom socket based transport should avoid this. This sort of use case will likely be common (distributed copy of HDFS across a WAN for example). > > If, as you suggest above, we enforce a 1-1 mapping of channel and avro > protocol, wouldn't that eliminate the need for a handshake on > subsequent requests on the same channel? The handshake process could > be part of the open-channel negotiation. I'm still wrapping my head > around the routing use-case; not sure if this meets the requirements > there though. I also have not wrapped my head around routing/proxy use cases. From a somewhat ignorant perspective on them -- I'd rather have a solid point-to-point protocol that just works, is simple, and can meet the vast majority of use cases with high performance than one that happens to be capable of sophisticated routing but has a lot of other limitations or is a lot harder to implement and debug. >>> >>
