[
https://issues.apache.org/jira/browse/AVRO-406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12831614#action_12831614
]
Todd Lipcon commented on AVRO-406:
----------------------------------
bq. As I think about it more, I believe the Iterator<Iterator<Foo>> approach
has merit. Avro's runtime supports the notion of efficient skipping
Skipping is one thing, but there's no way to rewind a socket. For example, what
do we do about the following user code:
{code}
void myStreamedRpc(Iterable<Iterable<Foo>> myArg) {
Iterator outerIter = myArg.iterator();
Iterator<Foo> firstIter = outerIter.next().iterator();
Iterator<Foo> secondIter = outerIter.next().iterator();
Foo a = secondIter.next();
Foo b = firstIter.next();
}
{code}
This code tries to read the data off the stream in the opposite order from
which they arrive. When assigning 'a' we could certainly skip all of
firstIter's data, but then we'd be screwed when it comes time to assign 'b'
since we can't skip back. We could buffer all of firstIter as soon as we access
secondIter, but then we're not being very transparent at all to end users. I
don't like the idea that users of this API would have to really understand its
workings under the hood or else risk potentially unbounded memory buffering.
Keeping the API restricted to "you may have one streamed input and one streamed
output" has its downsides in loss of generality, but at least it is very
transparent to implementors, and slight deviations in processing order can't
cause huge swings in performance.
> Support streaming RPC calls
> ---------------------------
>
> Key: AVRO-406
> URL: https://issues.apache.org/jira/browse/AVRO-406
> Project: Avro
> Issue Type: New Feature
> Components: java, spec
> Reporter: Todd Lipcon
>
> Avro nicely supports chunking of container types into multiple frames. We
> need to expose this to RPC layer to facilitate use cases like the Hadoop
> Datanode where a single "RPC" can yield far more data than should be buffered
> in memory.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
