On Tue, 1 Dec 2009, Doug Cutting wrote:
Arrays and maps can optionally encode their length in bytes. If the
item count is a negative number, then -count is the actual count and
immediately following the count is the size in bytes. Java's
BlockingBinaryEncoder implements this. All decoders must implement it.
http://hadoop.apache.org/avro/docs/current/api/java/org/apache/avro/io/BlockingBinaryEncoder.html
It does not emit a size for every array and map, but only for arrays and
maps whose contained size exceeds a threshold, so the overhead of adding
the size is limited. It also splits arrays and maps that are larger
than can be buffered as a whole into a sequence of blocks.
Interesting, thanks for the reference. Sounds like pretty much what I was
looking for.
--
Steinar Knutsen <[email protected]>
Yahoo! Technologies Norway AS
