Yes, sorry for polluting this thread, will fork it.
On Thu, May 16, 2019 at 1:45 PM Sean Busbey <bus...@cloudera.com.invalid> wrote: > > The matter of upgrading downstream projects should get its own thread > on dev@avro rather than occurring on this VOTE. > > On Thu, May 16, 2019 at 5:14 AM Ismaël Mejía <ieme...@gmail.com> wrote: > > > > Argh this looks like problematic if Spark somehow wants to align the > > deps, hope it won't be mandatory. > > > > I filled some more tickets, so the list of tickets so far: > > > > BEAM-7328 Update Avro to version 1.9.0 in Java SDK > > HIVE-21737 Upgrade Avro to version 1.9.0 > > FLINK-12532 Upgrade Avro to version 1.9.0 > > PARQUET-1576 Upgrade to Avro 1.9.0 > > SPARK-27733 Upgrade to Avro 1.9.x > > > > On Thu, May 16, 2019 at 10:48 AM Driesprong, Fokko <fo...@driesprong.frl> > > wrote: > > > > > > Thanks Nandor, > > > > > > Upgrading Hive 2.x (Hadoop 2) is going to be painful since this is still > > > on > > > Java 1.7. It is still relying on Avro 1.7.7. This is an issue because this > > > is also blocking upgrading Parquet, this implies that Hive 2.x does not > > > support reading lz4 and zstd compression. > > > > > > From Hive 3.x the Java version is 1.8 and we could upgrade Avro. The > > > current version is 1.8.2, so that should be doable. > > > > > > Cheers, Fokko > > > > > > Op wo 15 mei 2019 om 23:51 schreef Nandor Kollar > > > <nkol...@cloudera.com.invalid>: > > > > > > > If Spark is upgraded, then I'm afraid first Hive should be upgraded to > > > > 1.9.0. When I did a quick release verification, I remember I saw several > > > > failures due to Hive using deprecated and removed Avro methods (related > > > > to > > > > Jackson removal from public API). > > > > > > > > On Wed, May 15, 2019 at 11:04 PM Ismaël Mejía <ieme...@gmail.com> wrote: > > > > > > > > > Getting a new release of Avro feels amazing, thanks a lot Fokko for > > > > > all your wokr to get this out with the release. > > > > > > > > > > An interesting next step is downstreaming. I saw Fokko created a > > > > > ticket for Parquet, is anyone working on the Spark one? It will be > > > > > really nice to get this into Spark 3. Anything I can do to help there > > > > > or that we can sync work? > > > > > I filled https://issues.apache.org/jira/browse/SPARK-27733 for the > > > > moment. > > > > > > > > > > I am tackling the upgrade on Beam (the other project I work mostly on > > > > too. > > > > > https://issues.apache.org/jira/browse/BEAM-7328 > > > > > > > > > > Any other project worth the sync/work? > > > > > > > > > > > > > > > On Wed, May 15, 2019 at 8:54 PM Driesprong, Fokko > > > > > <fo...@driesprong.frl> > > > > > wrote: > > > > > > > > > > > > Hi Jacob, > > > > > > > > > > > > This looks off, I'll dive into it. I think it got released for some > > > > > reason. > > > > > > While releasing it today, I also noticed that the artifact wasn't > > > > staged > > > > > in > > > > > > Nexus anymore. However, RC4 is the same as the release. > > > > > > > > > > > > Cheers, Fokko > > > > > > > > > > > > Op wo 15 mei 2019 om 20:27 schreef Jacob Tolar > > > > > > <jto...@verizonmedia.com.invalid>: > > > > > > > > > > > > > Great news! We have been looking forward to the 1.9 release. > > > > > > > > > > > > > > I see 1.9.0 published on Maven Central. It appears to have been > > > > > published > > > > > > > 5/8 > > > > > > > (the same day as RC4 was announced but ~1week before vote passed). > > > > > > > > > > > > > > Is this the release build (presumably the same as RC4) or an > > > > > > > earlier > > > > > RC? > > > > > > > > > > > > > > https://repo1.maven.org/maven2/org/apache/avro/avro/1.9.0/ > > > > > > > > > > > > > > Would it be helpful to have maven include the git sha somewhere in > > > > > > > MANIFEST.MF? > > > > > > > > > > > > > > jacob > > > > > > > > > > > > > > On Tue, May 14, 2019 at 1:04 PM Driesprong, Fokko > > > > <fo...@driesprong.frl > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > Thank you all, > > > > > > > > > > > > > > > > I'm happy to announce, the vote for Avro 1.9.0 has passed. > > > > > > > > > > > > > > > > Binding: > > > > > > > > +1 Daniel Kulp > > > > > > > > +1 Sean Busbey > > > > > > > > +1 Fokko Driesprong > > > > > > > > > > > > > > > > Non-binding: > > > > > > > > +1 Lunjie Jin > > > > > > > > +1 Michael A. Smith > > > > > > > > +1 Ismaël Mejía > > > > > > > > +1 Brian Lachniet > > > > > > > > > > > > > > > > I would like to thank the community from participating in the > > > > > release. > > > > > > > Both > > > > > > > > in providing patches for Apache Avro and giving the RC's a try. > > > > > > > > > > > > > > > > Yesterday I did some additional testing against Apache Spark > > > > > (similar to > > > > > > > > Nandor :-) and Apache Flink and did not find any blocking > > > > > > > > issues. > > > > > There > > > > > > > are > > > > > > > > some incompatibilities, but most of them are; revoking Jackson > > > > > > > > from > > > > > the > > > > > > > > public API and deprecating Joda. > > > > > > > > > > > > > > > > I'll publish the artifacts to the different repositories soon. > > > > > > > > On > > > > > this, I > > > > > > > > consider myself a polyglot programmer, but my knowledge is > > > > > > > > limited > > > > on > > > > > > > .Net, > > > > > > > > Perl, and Javascript, so I might ask some help from the > > > > > > > > community > > > > > here as > > > > > > > > well. > > > > > > > > > > > > > > > > Thanks all, and keep up the good work. > > > > > > > > > > > > > > > > Cheers, Fokko Driesprong > > > > > > > > > > > > > > > > Op di 14 mei 2019 om 14:48 schreef Nandor Kollar > > > > > > > > <nkol...@cloudera.com.invalid>: > > > > > > > > > > > > > > > > > +1 (non-binding) > > > > > > > > > > > > > > > > > > * verified signatures and checksums > > > > > > > > > * ran unit tests, passed > > > > > > > > > * did a quick sanity test against Spark master: upgraded Spark > > > > Avro > > > > > > > > > dependency to this release candidate, ran spark-avro tests, > > > > > > > > > all > > > > > passed > > > > > > > > (for > > > > > > > > > the record, two minor Maven changes are required due to > > > > > > > > > removal > > > > of > > > > > > > > Hadoop 1 > > > > > > > > > support and making org.tukaani.xz provided dependency - looks > > > > like > > > > > > > Spark > > > > > > > > > tests against each codec). spark-hive tests failed, because > > > > > deprecated > > > > > > > > > methods were removed in Avro (related to Jackson removal from > > > > > public > > > > > > > API, > > > > > > > > > and Hive uses them), but I think it is expected, upgrading to > > > > > > > > > a > > > > > major > > > > > > > > > release could have breaking changes. > > > > > > > > > > > > > > > > > > Nandor > > > > > > > > > > > > > > > > > > On Sun, May 12, 2019 at 7:11 AM Sean Busbey > > > > > > > > > <bus...@apache.org> > > > > > wrote: > > > > > > > > > > > > > > > > > > > +1 (binding) > > > > > > > > > > > > > > > > > > > > -- Good: > > > > > > > > > > * signatures > > > > > > > > > > * checksums > > > > > > > > > > * tag/ref lines up with src artifact (except for one file > > > > > mentioned > > > > > > > > > below) > > > > > > > > > > * LICENSE/NOTICE spot check > > > > > > > > > > * apache rat says the license files for all the artifacts > > > > > > > > > > are > > > > > fine > > > > > > > > > (except > > > > > > > > > > it couldn't understand the gem; using the gem cli confirms > > > > > > > > > > it's > > > > > also > > > > > > > > > > correct) > > > > > > > > > > > > > > > > > > > > -- We Can Be Better Later: > > > > > > > > > > * filed AVRO-2395 because the "java" convenience binary > > > > > > > > > > part of > > > > > the > > > > > > > > dist > > > > > > > > > > section is redundant > > > > > > > > > > * in the future please post the specific staged maven > > > > repository > > > > > > > > instead > > > > > > > > > > of somewhere within the staged repository group. that'll > > > > > > > > > > help > > > > us > > > > > > > avoid > > > > > > > > > > possible conflicts should someone forget to cancel a staged > > > > > > > > > > RC > > > > or > > > > > > > > > > accidentally stage an additional repository after the vote > > > > > > > > > > is > > > > > called. > > > > > > > > > FWIW > > > > > > > > > > as far as I can tell from the Nexus UI, this is the staged > > > > > > > > > > repo > > > > > for > > > > > > > > this > > > > > > > > > > VOTE (and it's what I verified): > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://repository.apache.org/content/repositories/orgapacheavro-1020/ > > > > > > > > > > > > > > > > > > > > * When unpacking the source tarball I got this warning on > > > > > > > > > > OSX. > > > > > > > > > > > > > > > > > > > > > Busbey-MBA:1.9.0-RC4 busbey$ tar -C src_untar -xzf > > > > > > > > > > > > > > > > > > > > > > > dist.apache.org/repos/dist/dev/avro/avro-1.9.0-rc4/avro-src-1.9.0.tar.gz > > > > > > > > > > > tar: copyfile unpack > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > (avro-src-1.9.0/lang/java/mapred/src/test/resources/org/apache/avro/mapreduce/mapreduce-test-input.avro/SUCCESS.crc) > > > > > > > > > > failed: No such file or directory > > > > > > > > > > > > > > > > > > > > I don't think it should be a blocker because it's shown up > > > > > > > > > > in > > > > > prior > > > > > > > RCs > > > > > > > > > > since 2012 and AFAICT things are fine despite it, save a > > > > > > > > > > unit > > > > > test. > > > > > > > > > > > > > > > > > > > > * An upgrade guide will help a bunch of folks given the time > > > > > since > > > > > > > last > > > > > > > > > > release and this being a major version. > > > > > > > > > > > > > > > > > > > > For example, someone asked about how incompatible things > > > > > > > > > > are. > > > > So > > > > > I > > > > > > > ran > > > > > > > > > the > > > > > > > > > > Java API Compliance Checker on the java libraries, since > > > > > > > > > > it's > > > > > > > > > (relatively) > > > > > > > > > > easy. After filtering out the "avro.shaded" package and > > > > > excluding the > > > > > > > > > > hadoop1 specific jars from 1.8.2: > > > > > > > > > > > > > > > > > > > > > Busbey-MBA:1.9.0-RC4 busbey$ japi-compliance-checker -l > > > > "apache > > > > > > > avro" > > > > > > > > > > -d1 avro-1.8.2-jacc.xml -d2 avro-1.9.0-jacc.xml > > > > > > > > > > -skip-packages > > > > > > > > > > skip-packages.txt > > > > > > > > > > > Preparing, please wait ... > > > > > > > > > > > Using Java 1.8.0_161 > > > > > > > > > > > Reading classes 1.8.2 ... > > > > > > > > > > > WARNING: skipping "internal" packages > > > > > > > > > > > NOTE: use --keep-internal option to check them > > > > > > > > > > > Reading classes 1.9.0-rc4 ... > > > > > > > > > > > WARNING: skipping "internal" packages > > > > > > > > > > > Comparing classes ... > > > > > > > > > > > Creating compatibility report ... > > > > > > > > > > > Binary compatibility: 94.6% > > > > > > > > > > > Source compatibility: 93.6% > > > > > > > > > > > Total binary compatibility problems: 155, warnings: 26 > > > > > > > > > > > Total source compatibility problems: 160, warnings: 4 > > > > > > > > > > > Report: compat_reports/apache > > > > > > > > > avro/1.8.2_to_1.9.0-rc4/compat_report.html > > > > > > > > > > > > > > > > > > > > The report is here: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > http://people.apache.org/~busbey/avro/1.9.0-RC4/1.8.2_to_1.9.0RC4_compat_report.html > > > > > > > > > > > > > > > > > > > > This is a major version, so I think breaking things is fine. > > > > But > > > > > this > > > > > > > > > > report is still pretty long and so I'm guessing downstream > > > > could > > > > > use > > > > > > > > some > > > > > > > > > > help. Maybe the current summary list covers all of the > > > > > > > > > > breakage > > > > > > > > listed; I > > > > > > > > > > didn't try to compare them. > > > > > > > > > > > > > > > > > > > > On 2019/05/08 19:45:15, "Driesprong, Fokko" > > > > <fo...@driesprong.frl > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > Hi everyone, > > > > > > > > > > > > > > > > > > > > > > Since the last release of Apache Avro 1.8.2 on May 31, > > > > > > > > > > > 2017. > > > > > Two > > > > > > > > years > > > > > > > > > > > later, > > > > > > > > > > > I'm thrilled to propose the following RC to be released as > > > > > official > > > > > > > > > > Apache > > > > > > > > > > > Avro 1.9.0 release. > > > > > > > > > > > > > > > > > > > > > > The commit id is 3c76495e9524ef322726d03d7ee406be89e8fde0 > > > > > > > > > > > * This corresponds to the tag: release-1.9.0-rc4 > > > > > > > > > > > * > > > > > https://github.com/apache/avro/releases/tag/release-1.9.0-rc4 > > > > > > > > > > > > > > > > > > > > > > The release tarball, signature, and checksums are here: > > > > > > > > > > > * > > > > https://dist.apache.org/repos/dist/dev/avro/avro-1.9.0-rc4/ > > > > > > > > > > > > > > > > > > > > > > You can find the KEYS file here: > > > > > > > > > > > * https://dist.apache.org/repos/dist/dev/avro/KEYS > > > > > > > > > > > > > > > > > > > > > > Binary artifacts for Java are staged in Nexus here: > > > > > > > > > > > * > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://repository.apache.org/content/groups/staging/org/apache/avro/avro/1.9.0/ > > > > > > > > > > > > > > > > > > > > > > This release includes 272 Jira issues: > > > > > > > > > > > > > > > https://issues.apache.org/jira/projects/AVRO/versions/12333394 > > > > > > > > > > > * Deprecate Joda-Time in favor of Java8 JSR310 and > > > > > > > > > > > setting it > > > > > as > > > > > > > > > default > > > > > > > > > > > * Remove support for Hadoop 1.x > > > > > > > > > > > * Move from Jackson 1.x to 2.9 > > > > > > > > > > > * Add ZStandard Codec > > > > > > > > > > > * Lots of updates on the dependencies to fix CVE's > > > > > > > > > > > * Remove Jackson classes from public API > > > > > > > > > > > * Apache Avro is built by default with Java 8 > > > > > > > > > > > * Apache Avro is compiled and tested with Java 11 to > > > > guarantee > > > > > > > > > > compatibility > > > > > > > > > > > * Apache Avro MapReduce is compiled and tested with > > > > > > > > > > > Hadoop 3 > > > > > > > > > > > * Apache Avro is now leaner, multiple dependencies were > > > > > removed: > > > > > > > > guava, > > > > > > > > > > > paranamer, commons-codec, and commons-logging > > > > > > > > > > > * Introduce JMH Performance Testing Framework > > > > > > > > > > > * Add Snappy support for C++ DataFile > > > > > > > > > > > * and many, many more! > > > > > > > > > > > > > > > > > > > > > > Since RC1, two commits have been added: > > > > > > > > > > > * https://jira.apache.org/jira/browse/AVRO-2381 > > > > > > > > > > > * https://jira.apache.org/jira/browse/AVRO-2383 > > > > > > > > > > > > > > > > > > > > > > Since RC2: The SHA1/MD5 checksums have been replaced with > > > > > SHA512 > > > > > > > > > > > > > > > > > > > > > > Since RC3: > > > > > > > > > > > * Regression failure, the customEncode methods are public > > > > > again. > > > > > > > > > > > * The release tarball does not contain snapshot anymore > > > > > > > > > > > > > > > > > > > > > > Please download, verify, and test. This vote will remain > > > > > > > > > > > open > > > > > for > > > > > > > at > > > > > > > > > > least > > > > > > > > > > > 72 hours. Given sufficient votes, I would like to close > > > > > > > > > > > it on > > > > > or > > > > > > > > about > > > > > > > > > > > midnight > > > > > > > > > > > on Saturday, 11th of May 2019. > > > > > > > > > > > > > > > > > > > > > > [ ] +1 Release this as Apache Avro 1.9.0 > > > > > > > > > > > [ ] +0 > > > > > > > > > > > [ ] -1 Do not release this because... > > > > > > > > > > > > > > > > > > > > > > Consider this a +1 (non-binding) from my side: > > > > > > > > > > > * Compiled the new version of Parquet against the Divolte > > > > > collector > > > > > > > > and > > > > > > > > > > > Apache Parquet > > > > > > > > > > > > > > > > > > > > > > Cheers, Fokko Driesprong > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > busbey
