[ 
https://issues.apache.org/jira/browse/AVRO-380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12806025#action_12806025
 ] 

Philip Zeyliger commented on AVRO-380:
--------------------------------------

bq. setSyncInterval()

If you set syncInterval to Integer.MAX_VALUE, I bet things will break, because 
we allocate a ByteArrayOutputStream of 2*Integer.MAX_VALUE, which is going to 
overflow.

If there's a goal of supporting individual values larger than 2GB (I don't know 
that such a goal is useful), then the byte counts need to be a long.  I'm fine 
with this not being a goal, though.

bq. DataFileStream: in.read(block);

I think it's your responsibility here to check that block.length bytes were 
actually read.  I used LenghtLimitedInputStream for that.  You could certainly 
also read into the buffer until you've read the number of bytes you expect to.  
I think this also introduces an extra copy for the null codec, but, as you say, 
there are more things that could be done for performance here.

bq. There is also either a flaw in the Deflate Codec documentation, or the 
Deflate Codec code.

Hmm.  D'oh, I think you're right.  I think we should do RFC-1951 (for 
"deflate"; if we called it "zlib"...).  I think this just means changing two 
'false''s to 'true''s.  Let's do it in this patch.
Hm.  I think you're right, which is a bit embarassing

> Avro Container File format change:  add block size to block descriptor
> ----------------------------------------------------------------------
>
>                 Key: AVRO-380
>                 URL: https://issues.apache.org/jira/browse/AVRO-380
>             Project: Avro
>          Issue Type: Improvement
>          Components: doc, java, spec
>    Affects Versions: 1.3.0
>            Reporter: Scott Carey
>             Fix For: 1.3.0
>
>         Attachments: AVRO-380.patch
>
>
> The new file format in AVRO-160 limits a few use cases that I have found to 
> be important.
> A block currently contains a count of the number of records, the block data, 
> and a sync marker.  
> This change would add the block size, in bytes, along side the number of 
> records.   
> This allows efficient access to a block's data without the need to decode the 
> data into individual Datums, which is useful for various use cases.  

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Reply via email to