[ 
https://issues.apache.org/jira/browse/AVRO-380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Scott Carey updated AVRO-380:
-----------------------------

    Status: Patch Available  (was: Open)

This patch adds the block size to the block format.  

Each block then has:
blockCount -- the number of records in the block
blockSize -- the size in bytes of the block data, after compression
bytes -- blockSize bytes of data encoded by the file's codec
sync -- 16 byte sync marker

The file will fail to read a block if the block size or count does not match 
the data, or if a sync marker is wrong.

This worked best by changing the Codec API.  The current deflate and null 
codecs are not ideal for performance, but this patch focuses on the 
functionality.  I'll open a performance enhancement ticket later.
The Codec API and the file reader/writer will likely also change later due to 
performance enhancements on the BinaryDecoder and BinaryEncoder that I plan to 
work on.  These enhancements are all interdependent.

There is also either a flaw in the Deflate Codec documentation, or the Deflate 
Codec code.  The document claims that we are using RFC-1951 (unwrapped deflate 
compression), but the code is writing RFC-1950 (deflate wrapped in a "ZLIB" 
container).
I am on the fence as to which we should use.  The right answer is probably 
'whatever is easiest on all platforms'.  
If we decide on which to do I can add the fix to this patch, or we can defer to 
another ticket.


> Avro Container File format change:  add block size to block descriptor
> ----------------------------------------------------------------------
>
>                 Key: AVRO-380
>                 URL: https://issues.apache.org/jira/browse/AVRO-380
>             Project: Avro
>          Issue Type: Improvement
>          Components: doc, java, spec
>    Affects Versions: 1.3.0
>            Reporter: Scott Carey
>             Fix For: 1.3.0
>
>         Attachments: AVRO-380.patch
>
>
> The new file format in AVRO-160 limits a few use cases that I have found to 
> be important.
> A block currently contains a count of the number of records, the block data, 
> and a sync marker.  
> This change would add the block size, in bytes, along side the number of 
> records.   
> This allows efficient access to a block's data without the need to decode the 
> data into individual Datums, which is useful for various use cases.  

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Reply via email to