Re: C# POCO serializer/deserializer

[Brian Lachniet]

Hey Patrick,

This sounds very useful! I'd love to see this introduced to the C# library.

On Thu, May 2, 2019 at 5:47 PM Ivan Greene 
wrote:

> Patrick,
>
> This sounds as though it would be the C# equivalent of Java's ReflectData,
> which has been part of the Avro API for several years now, so it’s likely
> there would be interest. Your best bet is to open a Jira describing the
> planned feature and open a pull request on Github to start the discussion.
> The contributing page on the wiki is a bit out of date and still recommends
> submitting patches on Jira but I believe that Github is now the official
> repository.
>
> > On May 1, 2019, at 4:31 PM, Patrick Farry 
> wrote:
> >
> > Hi All,
> >
> > I have written code that implements Avro serialization for POCO classes
> - i.e. classes that are not generated by Avro codegen and do not implement
> ISpecificRecord. The idea was to make it work as much like JSON.net as
> possible.
> >
> > The serializer inherits from SpecificDefaultWriter and the deserializer
> from SpecificDefaultReader.
> >
> > Avro fields are mapped to C# properties either by matching the field
> name and property name or by using an attribute to specify the field
> sequence number.
> >
> > Is this something that would be of interest to the Avro project? My
> company has approved committing the code and I’d be available and happy to
> maintain this code and work on other parts of the C# implementation.
> >
> > Regards.
> >
> >
>
>

-- 

[image: 51b630b05e01a6d5134ccfd520f547c4.png]

Brian Lachniet

Software Engineer

E: blachn...@gmail.com | blachniet.com 

 

Re: C# POCO serializer/deserializer

[Ivan Greene]

Patrick,

This sounds as though it would be the C# equivalent of Java's ReflectData, 
which has been part of the Avro API for several years now, so it’s likely there 
would be interest. Your best bet is to open a Jira describing the planned 
feature and open a pull request on Github to start the discussion. The 
contributing page on the wiki is a bit out of date and still recommends 
submitting patches on Jira but I believe that Github is now the official 
repository.

> On May 1, 2019, at 4:31 PM, Patrick Farry  wrote:
> 
> Hi All,
> 
> I have written code that implements Avro serialization for POCO classes - 
> i.e. classes that are not generated by Avro codegen and do not implement 
> ISpecificRecord. The idea was to make it work as much like JSON.net as 
> possible.
> 
> The serializer inherits from SpecificDefaultWriter and the deserializer from 
> SpecificDefaultReader.
> 
> Avro fields are mapped to C# properties either by matching the field name and 
> property name or by using an attribute to specify the field sequence number. 
> 
> Is this something that would be of interest to the Avro project? My company 
> has approved committing the code and I’d be available and happy to maintain 
> this code and work on other parts of the C# implementation.
> 
> Regards. 
> 
> 

Re: [DISCUSS] Support additional timestamp semantic

[Zoltan Ivanfi]

Hi,

I also vote for the 3rd option (two new logical types:
‘local-timestamp-millis’ and ‘local-timestamp-micros’).

Could you please create a JIRA for this task and send a link to it to
this e-mail thread for everyone interested in the topic?

Thanks,

Zoltan

On Tue, Apr 23, 2019 at 1:49 PM Ryan Skraba  wrote:
>
> Hello!  I read the document with interest.  Very well-written and clean --
> I feel better equipped to explain the importance of the different flavours
> of date/time after reading it.
>
> I didn't go through the POC code in detail, but I did go through a bunch of
> our code to check how the proposed implementations would affect us (to
> provide a single, anecdotal data point).  We currently use Avro to
> represent hierachical data internally as it passes through a transformation
> pipeline running on a cluster.  We mostly rely on generic data.  The input
> or output might already be in Avro (file or binary message format), but it
> isn't necessary.  We do the schema inference and conversion on non-Avro
> when required.
>
> For us, it looks like both option#2 and option#3 should be more-or-less
> safe.  If we don't recognize a logical type, we'll just fall back on the
> underlying Avro type, and even propagate the unknown logical type down the
> pipeline if we can.
>
> Specifically, the bold proposal (option#2) for a new, unified logical type
> would mostly work without code modification on our part.  There's one or
> two places where we'd lose some helpful features where the semantic
> date/time type is taken into account, until we did the necessary rewrites.
> It wouldn't be a difficult task for us to bump to an Avro version that uses
> the new, unified logical type.
>
> Of course, the problem occurs when we're writing out data in Avro ... and
> the user has a next stage that doesn't understand the change.  Even if I
> appreciate the elegance of having a unified date/type logical type, it
> really seems like the more conservative third option (multiplying the
> number of logical types) is preferable.  Even if Avro ends up with a dozen
> logical types to describe the different flavours of date/time, this can
> eventually be unified in the language-specific API tools without breaking
> the schema specification.
>
> TL;DR: I read it, I appreciated it, I agree with your conclusions.
>
> Thanks again for the thorough and articulate work!  Ryan
>
>
>
> On Wed, Apr 17, 2019 at 9:44 AM Nandor Kollar 
> wrote:
>
> > Hi all,
> >
> > There is an ongoing effort to harmonize timestamp types for various popular
> > SQL engines for Hadoop (see details here
> > <
> > https://docs.google.com/document/d/1E-7miCh4qK6Mg54b-Dh5VOyhGX8V4xdMXKIHJL36a9U/edit#
> > >).
> > As part of this effort, on disk file formats should be able to support all
> > of these semantics. Avro timestamp logical type supports only one semantic:
> > UTC normalized. I put together a simple design doc an two POCs which
> > introduce additional local date/time semantics into Avro. Here is the
> > design doc:
> >
> > https://docs.google.com/document/d/1rLmb4-6G8LHBwHUU2P_8gE1o3lvMV0gSitnmiXXmlWY/edit?usp=sharing
> >
> > What are the thoughts on this? Please have a look at the POCs, and feel
> > free to comment the design doc!
> >
> > Thanks,
> > Nandor
> >

Re: [VOTE] Release Apache Avro 1.9.0 RC3

[Ismaël Mejía]

I am referring to the 'SNAPSHOT' suffix in the names (and content) of
the dist link.
https://dist.apache.org/repos/dist/dev/avro/avro-1.9.0-rc3/

It seems the github link is ok, but I assume that the 'defacto
version' for validation is the dist.apache.org one.
The binary staged artifacts look ok (the poms don't have the SNAPSHOT
suffix) so they should  ok to test.
https://repository.apache.org/content/groups/staging/org/apache/avro/avro/1.9.0/
So maybe it is just a question of updating the files in dist.

Extra comment for the relase info 'avro-core' is now !MB smaller due
to the removal of the extra dependencies.


On Wed, May 1, 2019 at 10:39 PM Driesprong, Fokko  wrote:
>
> Thank you Ismaël for looking into it and for the additional interesting
> info.
>
> Are you referring to the SNAPSHOT in the tests? I can remove these:
> root@83c927afb89b:/avro# grep -R "SNAPSHOT" .
> ./lang/js/node_modules/uglify-js/tools/domprops.json:
> "ORDERED_NODE_SNAPSHOT_TYPE",
> ./lang/js/node_modules/uglify-js/tools/domprops.json:
> "UNORDERED_NODE_SNAPSHOT_TYPE",
> ./lang/java/archetypes/avro-service-archetype/src/test/integration/projects/basic/archetype.properties:version=0.1-SNAPSHOT
> ./lang/java/maven-plugin/src/test/resources/unit/idl/pom-jsr310.xml:
> 1.9.0-SNAPSHOT
> ./lang/java/maven-plugin/src/test/resources/unit/idl/pom-joda.xml:
> 1.9.0-SNAPSHOT
> ./lang/java/maven-plugin/src/test/resources/unit/protocol/pom-jsr310.xml:
>   1.9.0-SNAPSHOT
> ./lang/java/maven-plugin/src/test/resources/unit/protocol/pom-joda.xml:
> 1.9.0-SNAPSHOT
> ./lang/java/maven-plugin/src/test/resources/unit/schema/pom-jsr310.xml:
> 1.9.0-SNAPSHOT
> ./lang/java/maven-plugin/src/test/resources/unit/schema/pom-joda.xml:
> 1.9.0-SNAPSHOT
> ./lang/ruby/Rakefile:VERSION =
> File.open('../../share/VERSION.txt').read.sub('-SNAPSHOT', '.pre1').chomp
> ./lang/ruby/.gem/gems/json_pure-2.1.0/data/prototype.js:  null,
> XPathResult.ORDERED_NODE_SNAPSHOT_TYPE, null);
> ./doc/examples/java-example/pom.xml:  1.0-SNAPSHOT
>
> These have slipped the `mvn versions:set`.
>
> Cheers, Fokko
>
> Op di 30 apr. 2019 om 17:24 schreef Ismaël Mejía :
>
> > I did a quick review on the JIRA issues included and extracted some
> > extra interesting info worth of addint to the release notes. Up to you
> > to choose which matter or not to be added.
> >
> > * Remove Jackson classes from public API
> > * Avro is built by default with Java 8
> > * Avro is compiled and tested too with Java 11 to guarantee compatibility
> > * Avro MapReduce is also compiled and tested with Hadoop 3
> > * Avro is now leaner, multiple dependencies were removed: guava,
> > paranamer, commons-codec and commons-logging
> > * Introduce JMH Performance Testing Framework
> > * Add Snappy support for C++ DataFile
> >
> > On Tue, Apr 30, 2019 at 4:50 PM Ismaël Mejía  wrote:
> > >
> > > Sorry, playing the killjoy again.
> > >
> > > It seems the files (and more critical the poms) still have the -SNAPSHOT
> > suffix.
> > > Also the comment of Dan Kulp about the extra directory structure in
> > > the main file  build/avro-1.9.0 directory would be a nice extra thing
> > > to fix.
> > >
> > > On Tue, Apr 30, 2019 at 12:54 PM Driesprong, Fokko 
> > wrote:
> > > >
> > > > Hi everyone,
> > > >
> > > > Since the last release of Apache Avro 1.8.2 on May 31, 2017. Two years
> > > > later,
> > > > I'm thrilled to propose the following RC to be released as official
> > Apache
> > > > Avro 1.9.0 release.
> > > >
> > > > The commit id is 24ff48c32d8d13433a1e31e485ef2af187d1eb62
> > > > * This corresponds to the tag: release-1.9.0-rc3
> > > > * https://github.com/apache/avro/releases/tag/release-1.9.0-rc3/
> > > >
> > > > The release tarball, signature, and checksums are here:
> > > > * https://dist.apache.org/repos/dist/dev/avro/avro-1.9.0-rc3/
> > > >
> > > > You can find the KEYS file here:
> > > > * https://dist.apache.org/repos/dist/dev/avro/KEYS
> > > >
> > > > Binary artifacts for Java are staged in Nexus here:
> > > > *
> > > >
> > https://repository.apache.org/content/groups/staging/org/apache/avro/avro/1.9.0/
> > > >
> > > > This release includes 272 Jira issues:
> > > > https://issues.apache.org/jira/projects/AVRO/versions/1294
> > > > * Deprecate Joda-Time in favor of Java8 JSR310 and setting it as
> > default
> > > > * Remove support for Hadoop 1.x
> > > > * Move from Jackson 1.x to 2.9
> > > > * Add ZStandard Codec
> > > > * Lots of updates on the dependencies to fix CVE's
> > > > * and many, many more!
> > > >
> > > > Since RC1, two commits have been added:
> > > > * https://jira.apache.org/jira/browse/AVRO-2381
> > > > * https://jira.apache.org/jira/browse/AVRO-2383
> > > >
> > > > Since RC2 the SHA1/MD5 checksums have been replaced with SHA512
> > > >
> > > > Please download, verify, and test. This vote will remain open for at
> > least
> > > > 72 hours. Given sufficient votes, I would like to close it on or about
> > > > midnight
> > > > on Saturday, 4th of May 2019.
> > > >
> > > > [ ] +1