[
https://issues.apache.org/jira/browse/AVRO-406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12834983#action_12834983
]
Todd Lipcon commented on AVRO-406:
----------------------------------
We just had a bit of a discussion about this. In discussion, we determined that
the primary difficulty is going to be flow control when multiplexing several
streamed RPCs over the same TCP pipe.
For example, if there are two block writes streaming into one Hadoop DN, going
to different disks, we need a way of applying backpressure on one write without
backpressure on the other. This implies that we either (a) must implement our
own flow control mechanism for streamed calls, or (b) cannot multiplex multiple
streamed calls on the same TCP stream, thus allowing normal TCP backpressure to
work here.
Unfortunately, option A above is a pain, since we'd end up reimplementing
sliding window. Option B is a pain because we're throwing away other benefits
of keeping a low number of sockets open (eg avoiding resetting TCP slow start
for each block read).
> Support streaming RPC calls
> ---------------------------
>
> Key: AVRO-406
> URL: https://issues.apache.org/jira/browse/AVRO-406
> Project: Avro
> Issue Type: New Feature
> Components: java, spec
> Reporter: Todd Lipcon
>
> Avro nicely supports chunking of container types into multiple frames. We
> need to expose this to RPC layer to facilitate use cases like the Hadoop
> Datanode where a single "RPC" can yield far more data than should be buffered
> in memory.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
