[
https://issues.apache.org/jira/browse/AVRO-380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12806025#action_12806025
]
Philip Zeyliger commented on AVRO-380:
--------------------------------------
bq. setSyncInterval()
If you set syncInterval to Integer.MAX_VALUE, I bet things will break, because
we allocate a ByteArrayOutputStream of 2*Integer.MAX_VALUE, which is going to
overflow.
If there's a goal of supporting individual values larger than 2GB (I don't know
that such a goal is useful), then the byte counts need to be a long. I'm fine
with this not being a goal, though.
bq. DataFileStream: in.read(block);
I think it's your responsibility here to check that block.length bytes were
actually read. I used LenghtLimitedInputStream for that. You could certainly
also read into the buffer until you've read the number of bytes you expect to.
I think this also introduces an extra copy for the null codec, but, as you say,
there are more things that could be done for performance here.
bq. There is also either a flaw in the Deflate Codec documentation, or the
Deflate Codec code.
Hmm. D'oh, I think you're right. I think we should do RFC-1951 (for
"deflate"; if we called it "zlib"...). I think this just means changing two
'false''s to 'true''s. Let's do it in this patch.
Hm. I think you're right, which is a bit embarassing
> Avro Container File format change: add block size to block descriptor
> ----------------------------------------------------------------------
>
> Key: AVRO-380
> URL: https://issues.apache.org/jira/browse/AVRO-380
> Project: Avro
> Issue Type: Improvement
> Components: doc, java, spec
> Affects Versions: 1.3.0
> Reporter: Scott Carey
> Fix For: 1.3.0
>
> Attachments: AVRO-380.patch
>
>
> The new file format in AVRO-160 limits a few use cases that I have found to
> be important.
> A block currently contains a count of the number of records, the block data,
> and a sync marker.
> This change would add the block size, in bytes, along side the number of
> records.
> This allows efficient access to a block's data without the need to decode the
> data into individual Datums, which is useful for various use cases.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.