Well, as luck would have it, 1 minute after sending this JIRA came back to life.
AVRO-392.

On Feb 3, 2010, at 2:46 AM, Scott Carey wrote:

> I have a patch well under way for overhauling BinaryDecoder performance.  
> I'll put the partial patch up in the morning (new BinaryDecoder and test 
> code), since it doesn't look like JIRA will be up for a while.  It has said 
> it will be back in '30 minutes' for the last 3 hours.  
> 
> I would like to get these changes in for 1.3 because there are some API 
> semantic changes that would be better to change now than later.  If we get 
> too locked in we may never be able to get the performance on the low level 
> decoding/encoding to its peak capability.
> 
> 
> Here is the would-be description of the ticket in case anyone wants to 
> comment before I can get it into JIRA:
> 
> ------------------------------------------------
> BinaryDecoder has room for significant performance improvement.  
> [AVRO-327|https://issues.apache.org/jira/browse/AVRO-327] has some 
> preliminary work here, but in order to satisfy some use cases there is much 
> more work to do.
> 
> I am opening a new ticket because the scope of the changes needed to do this 
> the right way are larger.
> 
> I have done a large bulk of a new implementation that abstracts a 
> 'ByteSource' from the BinaryDecoder.  Currently BinaryDecoder is tightly 
> coupled to InputStream.  The ByteSource can wrap an InputStream, FileChannel, 
> or byte[] in this version, but could be extended to support other channel 
> types, sockets, etc.  This abstraction allows the BinaryDecoder to buffer 
> data from various sources while supporting interleaved access to the 
> underlying data and greater flexibility going forward.
> The performance of this abstraction has been heavily tuned so that maximum 
> performance can be achieved even for slower ByteSource implementations.
> 
> For readers that must interleave reads on a stream with the decoder, this 
> includes a
> {code}
> public InputStream inputStream();
> {code}
> method on the decoder that can serve interleaved reads.  
> 
> Additionally it will be necessary to have a constructor on BinaryDecoder that 
> allows two BinaryDecoders to share a stream (and buffer).
> 
> 
> Performance results on this new version is better than previous prototypes:
> 
> *current trunk BinaryDecoder*
> {noformat}
> ReadInt: 983 ms, 30.497877855999185 million entries/sec
> ReadLongSmall: 1058 ms, 28.336666040111496 million entries/sec
> ReadLong: 1518 ms, 19.75179889508437 million entries/sec
> ReadFloat: 657 ms, 45.61031157924184 million entries/sec
> ReadDouble: 761 ms, 39.387756709704355 million entries/sec
> ReadBoolean: 331 ms, 90.4268145647456 million entries/sec
> RepeaterTest: 7718 ms, 3.886725782038378 million entries/sec
> NestedRecordTest: 1884 ms, 15.91964611687992 million entries/sec
> ResolverTest: 8296 ms, 3.616055866616717 million entries/sec
> MigrationTest: 21216 ms, 1.4139999570144013 million entries/sec
> {noformat}
> 
> *buffering BinaryDecoder*
> {noformat}
> ReadInt: 187 ms, 160.22131904871262 million entries/sec
> ReadLongSmall: 372 ms, 80.4863521975457 million entries/sec
> ReadLong: 613 ms, 48.882385721129246 million entries/sec
> ReadFloat: 253 ms, 118.16606270679061 million entries/sec
> ReadDouble: 275 ms, 108.94314257389068 million entries/sec
> ReadBoolean: 222 ms, 134.85327963176064 million entries/sec
> RepeaterTest: 3335 ms, 8.993007936329503 million entries/sec
> NestedRecordTest: 1152 ms, 26.0256943004597 million entries/sec
> ResolverTest: 4213 ms, 7.120659335077578 million entries/sec
> MigrationTest: 15310 ms, 1.9594884898992941 million entries/sec
> {noformat}
> 
> Performance is 2x to 5x the throughput of trunk on most tests.  
> 
> 
> On Feb 2, 2010, at 2:20 PM, Doug Cutting wrote:
> 
>> I'd like to roll an Avro 1.3 release candidate soon.
>> 
>> The biggest thing we're missing is interop testing for various 
>> implementations.  Ruby and Python both have data file implementations, 
>> so this should not be hard to add to these.  This requires two scripts, 
>> one to write a data file to build/test/interop/data/$lang.avro that uses 
>> the schema in share/test/schemas/interop.avsc.  The second then just 
>> needs to read all of the data files in build/test/interop/data/.  These 
>> scripts should both be invoked by the top-level build.sh.
>> 
>> RPC interop tests would be nice to have too, but I'm not sure we should 
>> hold the release for that.
>> 
>> Do folks agree that these are the right priorities for the 1.3 release? 
>> If so, do we have volunteers who can get data file interop tests 
>> implemented soon?  Ideally we might roll a release candidate late this week.
>> 
>> Thanks,
>> 
>> Doug
> 

Reply via email to