Support for blocking arrays/maps in python
------------------------------------------

                 Key: AVRO-61
                 URL: https://issues.apache.org/jira/browse/AVRO-61
             Project: Avro
          Issue Type: Bug
          Components: python
            Reporter: Thiruvalluvan M. G.


We have introduced a blocking version of writer in Java that enables readers to 
efficiently skip large arrays and maps. The avro format encoded arrays and maps 
by encoding the number of elements by the the elements themselves. Zero element 
count indicates that the array/map has ended. The change we introduced is that 
if the element count is negative, it is followed by the byte-count of the 
encoded elements that follow. The reader, on seeing a negative element count 
should flip the sign to get the actual number of elements. In addition, if it 
is interested in supporting fast skip, it should use the byte count to skip the 
elements en-bloc instead of decoding them individually. If it does not want to 
support fast skip, it has to just read the byte count and ignore its value.

The changes are already made in Java's ValueReader to support this. Similar 
changes need to done in Python as well.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Reply via email to