[
https://issues.apache.org/jira/browse/AVRO-406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12830358#action_12830358
]
Doug Cutting commented on AVRO-406:
-----------------------------------
The existing API to minimize copies is Decoder#readBytes(ByteBuffer). When
writing data, if a value is larger than 4k or so, it is written as a separate
frame. If, when reading, one calls Decoder#readBytes(ByteBuffer) at a frame
boundary, it should read directly from the socket or file handle without
intermediate buffering.
For the datanode case, I had imagined that one would make RPC requests to read
a chunk at a time, and each would be returned with its checksum. To avoid
round-trip delays, the RPC client would need to be able to submit requests
without blocking. It could then submit multiple requests, then read responses
until the number of outstanding requests fell below some threshold, then submit
more requests. If the transport can return responses out of order then the
client would need to re-order them. Alternately, if a transport like HTTP is
used, multiple requests can be sent over a connection and their responses are
guaranteed to be returned in order.
> Support streaming RPC calls
> ---------------------------
>
> Key: AVRO-406
> URL: https://issues.apache.org/jira/browse/AVRO-406
> Project: Avro
> Issue Type: New Feature
> Components: java, spec
> Reporter: Todd Lipcon
>
> Avro nicely supports chunking of container types into multiple frames. We
> need to expose this to RPC layer to facilitate use cases like the Hadoop
> Datanode where a single "RPC" can yield far more data than should be buffered
> in memory.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
