Michael, the problem you're hitting is that Parquet's dependency moved to Avro 1.8.0 from 1.7.7 and added a method that is missing when you use 1.7.7.
It looks like Spark is still using Avro 1.7.7. I think updating that to 1.8.x should fix the problem. rb On Tue, Jan 24, 2017 at 5:35 AM, Michael Heuer <heue...@gmail.com> wrote: > Per comment > https://github.com/bigdatagenomics/adam/pull/1360#issuecomment-274681650 > > and Jenkins failure > https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/ > 1757/HADOOP_VERSION=2.6.0,SCALAVER=2.10,SPARK_VERSION=1.5.2,label=centos/ > > when bumping our build to 1.8.2-rc1 unit tests succeed but we encounter > NoSuchMethodError errors at runtime in spark-submit. Our build is a mess > of transitive dependencies and exclusions, so I haven't yet been able to > identify why bumping the parquet version would cause this problem. > > I wanted to mention this here in case anyone else runs into it. > > michael > > > ---------- Forwarded message ---------- >> From: Julien Le Dem <jul...@dremio.com> >> To: d...@parquet.apache.org, dev@spark.apache.org >> Cc: >> Date: Mon, 23 Jan 2017 11:43:40 -0800 >> Subject: Re: [VOTE] Release Apache Parquet 1.8.2 RC1 >> Hi Spark dev, >> Here is the voting thread for parquet 1.8.2 release. >> Cheng or someone else we would appreciate you verify it as well and reply >> to the thread. >> >> On Mon, Jan 23, 2017 at 11:40 AM, Julien Le Dem <jul...@dremio.com> >> wrote: >> >>> +1 >>> Followed: https://cwiki.apache.org/confluence/display/PARQUET/How+To+V >>> erify+A+Release >>> checked sums, ran the build and tests. >>> We would appreciate someone from the Spark project (Cheng?) to verify >>> the release as well. >>> CC'ing spark >>> >>> >>> On Mon, Jan 23, 2017 at 10:15 AM, Ryan Blue <rb...@netflix.com.invalid> >>> wrote: >>> >>>> +1 >>>> >>>> On Mon, Jan 23, 2017 at 10:15 AM, Daniel Weeks >>>> <dwe...@netflix.com.invalid> >>>> wrote: >>>> >>>> > +1 checked sums, built, tested >>>> > >>>> > On Mon, Jan 23, 2017 at 9:58 AM, Ryan Blue <rb...@netflix.com.invalid >>>> > >>>> > wrote: >>>> > >>>> > > Gabor, that md5 matches what I get. Are you sure you used the right >>>> file? >>>> > > It isn’t the same format that md5sum produces, but if you check the >>>> > octets >>>> > > the hash matches.. >>>> > > >>>> > > [blue@work Downloads]$ md5sum apache-parquet-1.8.2.tar.gz >>>> > > b3743995bee616118c28f324598684ba apache-parquet-1.8.2.tar.gz >>>> > > >>>> > > rb >>>> > > >>>> > > >>>> > > On Thu, Jan 19, 2017 at 8:06 AM, Gabor Szadovszky < >>>> > > gabor.szadovs...@cloudera.com> wrote: >>>> > > >>>> > > > Hi Ryan, >>>> > > > >>>> > > > I’ve downloaded the tar and checked the signature and the >>>> checksums. >>>> > SHA >>>> > > > and ASC are fine. MD5 is not and the content does not seem to be a >>>> > common >>>> > > > MD5 either: >>>> > > > apache-parquet-1.8.2.tar.gz: B3 74 39 95 BE E6 16 11 8C 28 F3 24 >>>> 59 86 >>>> > > 84 >>>> > > > BA >>>> > > > >>>> > > > The artifacts on Nexus are good with all the related signatures >>>> and >>>> > > > checksums. The source zip properly contains the files from the >>>> repo >>>> > with >>>> > > > the tag apache-parquet-1.8.2. >>>> > > > >>>> > > > Regards, >>>> > > > Gabor >>>> > > > >>>> > > > > On 19 Jan 2017, at 04:09, Ryan Blue <b...@apache.org> wrote: >>>> > > > > >>>> > > > > Hi everyone, >>>> > > > > >>>> > > > > I propose the following RC to be released as official Apache >>>> Parquet >>>> > > > 1.8.2 >>>> > > > > release. >>>> > > > > >>>> > > > > The commit id is c6522788629e590a53eb79874b95f6c3ff11f16c >>>> > > > > * This corresponds to the tag: apache-parquet-1.8.2 >>>> > > > > * https://github.com/apache/parquet-mr/tree/c6522788 >>>> > > > > * >>>> > > > > https://git-wip-us.apache.org/repos/asf/projects/repo?p= >>>> > > > parquet-mr.git&a=commit&h=c6522788 >>>> > > > > >>>> > > > > The release tarball, signature, and checksums are here: >>>> > > > > * https://dist.apache.org/repos/dist/dev/parquet/apache- >>>> > > > parquet-1.8.2-rc1 >>>> > > > > >>>> > > > > You can find the KEYS file here: >>>> > > > > * https://dist.apache.org/repos/dist/dev/parquet/KEYS >>>> > > > > >>>> > > > > Binary artifacts are staged in Nexus here: >>>> > > > > * >>>> > > > > https://repository.apache.org/content/groups/staging/org/ >>>> > > > apache/parquet/parquet/1.8.2/ >>>> > > > > >>>> > > > > This is a patch release with backports from the master branch. >>>> For a >>>> > > > > detailed summary, see the spreadsheet here: >>>> > > > > >>>> > > > > * >>>> > > > > https://docs.google.com/spreadsheets/d/1NAuY3c77Egs6REu- >>>> > > > UVkQqPswpVYVgZTTnY3bM0SPVRs/edit#gid=0 >>>> > > > > >>>> > > > > Please download, verify, and test. >>>> > > > > >>>> > > > > Please vote by the end of Monday, 18 January. >>>> > > > > >>>> > > > > [ ] +1 Release this as Apache Parquet 1.8.2 >>>> > > > > [ ] +0 >>>> > > > > [ ] -1 Do not release this because... >>>> > > > > >>>> > > > > >>>> > > > > >>>> > > > > -- >>>> > > > > Ryan Blue >>>> > > > >>>> > > > >>>> > > >>>> > > >>>> > > -- >>>> > > Ryan Blue >>>> > > Software Engineer >>>> > > Netflix >>>> > > >>>> > >>>> >>>> >>>> >>>> -- >>>> Ryan Blue >>>> Software Engineer >>>> Netflix >>>> >>> >>> >>> >>> -- >>> Julien >>> >> >> >> >> -- >> Julien >> >> > -- Ryan Blue Software Engineer Netflix
