Question about the state of the AVRO C# (csharp) implementation

2014-03-13 Thread Longton, Nigel
Hi,
Are there plans to make C# feature equivalent? The major differences seem to be:

1.   C# doesn't  support the Avro IDL schema definition

2.   C# doesn't support the JSON serialization codec

3.   We had a case where we wanted to deserialize a json array which we can 
do in Java with Schema.createArray. The equivalent construct in C# was private 
and so not externally accessible.

We're making extensive use of Apache avro to communicate between Java and C# 
applications using JSON serialization rather than binary and in some cases we 
are creating schemas to match non-avro generated json from 'legacy' 
applications. I copied the Java json codec to C# and have got it working (I 
wont claim its fully tested though). More than happy to post this somewhere if  
its useful.
The other tweak we made was to change how 'default' values were serialized. We 
created a new writer classes that if the field value equaled the default it 
didn't serialize it, and a new reader class that set missing fields to the 
schema default if defined (missing non-defaulted fields are still an error). 
The advantage is that the JSON created is much cleaner and can be readily 
consumed by a jackson or any other json framework. (Currently these aren't 
ported to C# due to lack of time).

Are any of the above already scheduled to be added. Some of them might be 
contra to the avro approach though.

Thanks
Nigel



This message contains confidential information and is intended only for the 
individual named. If you are not the named addressee, you should not 
disseminate, distribute, alter or copy this e-mail. Please notify the sender 
immediately by e-mail if you have received this e-mail by mistake and delete 
this e-mail from your system. E-mail transmissions cannot be guaranteed to be 
secure or without error as information could be intercepted, corrupted, lost, 
destroyed, arrive late or incomplete, or contain viruses. The sender, 
therefore, does not accept liability for any errors or omissions in the 
contents of this message which arise during or as a result of e-mail 
transmission. If verification is required, please request a hard-copy version. 
This message is provided for information purposes and should not be construed 
as a solicitation or offer to buy or sell any securities or related financial 
instruments in any jurisdiction.  Securities are offered in the U.S. through 
PIMCO Investments LLC, distributor and a company of PIMCO LLC.


Re: Question about the state of the AVRO C# (csharp) implementation

2014-03-13 Thread Doug Cutting
On Thu, Mar 13, 2014 at 7:34 AM, Longton, Nigel nigel.long...@pimco.com wrote:
 Are there plans to make C# feature equivalent?

As a volunteer-based open-source project, we don't have a long-term
plan.  Rather we, as a group, consider contributions as they arrive
and generally accept those that aren't counter to other's needs.

The major differences seem to be:

 1.   C# doesn’t  support the Avro IDL schema definition

The IDL compiler does not need to be re-implemented per language.  You
can use the existing IDL compiler to convert IDL to protocols 
schemas, then generate C# from those.  A single implementation of the
IDL compiler is easier to maintain.

 2.   C# doesn’t support the JSON serialization codec

 3.   We had a case where we wanted to deserialize a json array which we
 can do in Java with Schema.createArray. The equivalent construct in C# was
 private and so not externally accessible.

These would make great improvements to Avro C#.  Please open issues in Jira.

 We’re making extensive use of Apache avro to communicate between Java and C#
 applications using JSON serialization rather than binary and in some cases
 we are creating schemas to match non-avro generated json from ‘legacy’
 applications. I copied the Java json codec to C# and have got it working (I
 wont claim its fully tested though). More than happy to post this somewhere
 if  its useful.

This sounds useful.  Please attach it as a patch in Jira.

https://cwiki.apache.org/confluence/display/AVRO/How+To+Contribute

 The other tweak we made was to change how ‘default’ values were serialized.
 We created a new writer classes that if the field value equaled the default
 it didn’t serialize it, and a new reader class that set missing fields to
 the schema default if defined (missing non-defaulted fields are still an
 error). The advantage is that the JSON created is much cleaner and can be
 readily consumed by a jackson or any other json framework. (Currently these
 aren’t ported to C# due to lack of time).

As you note, this is a non-standard mode of operation for Avro, but,
if you find it useful, others might too.  So long as it doesn't alter
Avro's standard behavior we should consider adding it to Avro as an
optional capability.

Cheers,

Doug