+1 (non-binding) Bumping our build to 2.3.2 rc6 and Avro to 1.8.2 and Parquet to 1.8.3 works for us, running on version 2.3.2 rc6 and older Spark versions.
https://github.com/bigdatagenomics/adam/pull/2055 michael On Thu, Sep 20, 2018 at 7:09 PM, Ryan Blue <rb...@netflix.com.invalid> wrote: > Changing my vote to +1 with this fixed. > > Here's what was going on -- and thanks to Owen O'Malley for debugging: > > The problem was that Iceberg contained a fix for a JVM bug for timestamps > before the unix epoch where the timestamp was off by 1s. Owen moved this > code into ORC as well and using the new version of Spark pulled in the > newer version of ORC. That meant that the values were "fixed" twice and > were wrong. > > Updating the Iceberg code to rely on the fix in the version of ORC that > Spark includes fixes the problem. > > On Thu, Sep 20, 2018 at 2:38 PM Dongjoon Hyun <dongjoon.h...@gmail.com> > wrote: > >> Hi, Ryan. >> >> Could you share the result on 2.3.1 since this is 2.3.2 RC? That would be >> helpful to narrow down the scope. >> >> Bests, >> Dongjoon. >> >> On Thu, Sep 20, 2018 at 11:56 Ryan Blue <rb...@netflix.com.invalid> >> wrote: >> >>> -0 >>> >>> My DataSourceV2 implementation for Iceberg is failing ORC tests when I >>> run with the 2.3.2 RC that pass when I run with 2.3.0. I'm tracking down >>> the cause and will report back, but I'm -0 on the release because there may >>> be a behavior change. >>> >>> On Thu, Sep 20, 2018 at 10:37 AM Denny Lee <denny.g....@gmail.com> >>> wrote: >>> >>>> +1 >>>> >>>> On Thu, Sep 20, 2018 at 9:55 AM Xiao Li <gatorsm...@gmail.com> wrote: >>>> >>>>> +1 >>>>> >>>>> >>>>> John Zhuge <john.zh...@gmail.com> 于2018年9月19日周三 下午1:17写道： >>>>> >>>>>> +1 (non-binding) >>>>>> >>>>>> Built on Ubuntu 16.04 with Maven flags: -Phadoop-2.7 -Pmesos -Pyarn >>>>>> -Phive-thriftserver -Psparkr -Pkinesis-asl -Phadoop-provided >>>>>> >>>>>> java version "1.8.0_181" >>>>>> Java(TM) SE Runtime Environment (build 1.8.0_181-b13) >>>>>> Java HotSpot(TM) 64-Bit Server VM (build 25.181-b13, mixed mode) >>>>>> >>>>>> >>>>>> On Wed, Sep 19, 2018 at 2:31 AM Takeshi Yamamuro < >>>>>> linguin....@gmail.com> wrote: >>>>>> >>>>>>> +1 >>>>>>> >>>>>>> I also checked `-Pyarn -Phadoop-2.7 -Pkinesis-asl -Phive >>>>>>> -Phive-thriftserve` on the openjdk below/macOSv10.12.6 >>>>>>> >>>>>>> $ java -version >>>>>>> java version "1.8.0_181" >>>>>>> Java(TM) SE Runtime Environment (build 1.8.0_181-b13) >>>>>>> Java HotSpot(TM) 64-Bit Server VM (build 25.181-b13, mixed mode) >>>>>>> >>>>>>> >>>>>>> On Wed, Sep 19, 2018 at 10:45 AM Dongjoon Hyun < >>>>>>> dongjoon.h...@gmail.com> wrote: >>>>>>> >>>>>>>> +1. >>>>>>>> >>>>>>>> I tested with `-Pyarn -Phadoop-2.7 -Pkinesis-asl -Phive >>>>>>>> -Phive-thriftserve` on OpenJDK(1.8.0_181)/CentOS 7.5. >>>>>>>> >>>>>>>> I hit the following test case failure once during testing, but it's >>>>>>>> not persistent. >>>>>>>> >>>>>>>> KafkaContinuousSourceSuite >>>>>>>> ... >>>>>>>> subscribing topic by name from earliest offsets >>>>>>>> (failOnDataLoss: false) *** FAILED *** >>>>>>>> >>>>>>>> Thank you, Saisai. >>>>>>>> >>>>>>>> Bests, >>>>>>>> Dongjoon. >>>>>>>> >>>>>>>> On Mon, Sep 17, 2018 at 6:48 PM Saisai Shao <sai.sai.s...@gmail.com> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> +1 from my own side. >>>>>>>>> >>>>>>>>> Thanks >>>>>>>>> Saisai >>>>>>>>> >>>>>>>>> Wenchen Fan <cloud0...@gmail.com> 于2018年9月18日周二 上午9:34写道： >>>>>>>>> >>>>>>>>>> +1. All the blocker issues are all resolved in 2.3.2 AFAIK. >>>>>>>>>> >>>>>>>>>> On Tue, Sep 18, 2018 at 9:23 AM Sean Owen <sro...@apache.org> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> +1 . Licenses and sigs check out as in previous 2.3.x releases. A >>>>>>>>>>> build from source with most profiles passed for me. >>>>>>>>>>> On Mon, Sep 17, 2018 at 8:17 AM Saisai Shao < >>>>>>>>>>> sai.sai.s...@gmail.com> wrote: >>>>>>>>>>> > >>>>>>>>>>> > Please vote on releasing the following candidate as Apache >>>>>>>>>>> Spark version 2.3.2. >>>>>>>>>>> > >>>>>>>>>>> > The vote is open until September 21 PST and passes if a >>>>>>>>>>> majority +1 PMC votes are cast, with a minimum of 3 +1 votes. >>>>>>>>>>> > >>>>>>>>>>> > [ ] +1 Release this package as Apache Spark 2.3.2 >>>>>>>>>>> > [ ] -1 Do not release this package because ... >>>>>>>>>>> > >>>>>>>>>>> > To learn more about Apache Spark, please see >>>>>>>>>>> http://spark.apache.org/ >>>>>>>>>>> > >>>>>>>>>>> > The tag to be voted on is v2.3.2-rc6 (commit >>>>>>>>>>> 02b510728c31b70e6035ad541bfcdc2b59dcd79a): >>>>>>>>>>> > https://github.com/apache/spark/tree/v2.3.2-rc6 >>>>>>>>>>> > >>>>>>>>>>> > The release files, including signatures, digests, etc. can be >>>>>>>>>>> found at: >>>>>>>>>>> > https://dist.apache.org/repos/dist/dev/spark/v2.3.2-rc6-bin/ >>>>>>>>>>> > >>>>>>>>>>> > Signatures used for Spark RCs can be found in this file: >>>>>>>>>>> > https://dist.apache.org/repos/dist/dev/spark/KEYS >>>>>>>>>>> > >>>>>>>>>>> > The staging repository for this release can be found at: >>>>>>>>>>> > https://repository.apache.org/content/repositories/ >>>>>>>>>>> orgapachespark-1286/ >>>>>>>>>>> > >>>>>>>>>>> > The documentation corresponding to this release can be found >>>>>>>>>>> at: >>>>>>>>>>> > https://dist.apache.org/repos/dist/dev/spark/v2.3.2-rc6-docs/ >>>>>>>>>>> > >>>>>>>>>>> > The list of bug fixes going into 2.3.2 can be found at the >>>>>>>>>>> following URL: >>>>>>>>>>> > https://issues.apache.org/jira/projects/SPARK/versions/ >>>>>>>>>>> 12343289 >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > FAQ >>>>>>>>>>> > >>>>>>>>>>> > ========================= >>>>>>>>>>> > How can I help test this release? >>>>>>>>>>> > ========================= >>>>>>>>>>> > >>>>>>>>>>> > If you are a Spark user, you can help us test this release by >>>>>>>>>>> taking >>>>>>>>>>> > an existing Spark workload and running on this release >>>>>>>>>>> candidate, then >>>>>>>>>>> > reporting any regressions. >>>>>>>>>>> > >>>>>>>>>>> > If you're working in PySpark you can set up a virtual env and >>>>>>>>>>> install >>>>>>>>>>> > the current RC and see if anything important breaks, in the >>>>>>>>>>> Java/Scala >>>>>>>>>>> > you can add the staging repository to your projects resolvers >>>>>>>>>>> and test >>>>>>>>>>> > with the RC (make sure to clean up the artifact cache >>>>>>>>>>> before/after so >>>>>>>>>>> > you don't end up building with a out of date RC going forward). >>>>>>>>>>> > >>>>>>>>>>> > =========================================== >>>>>>>>>>> > What should happen to JIRA tickets still targeting 2.3.2? >>>>>>>>>>> > =========================================== >>>>>>>>>>> > >>>>>>>>>>> > The current list of open tickets targeted at 2.3.2 can be >>>>>>>>>>> found at: >>>>>>>>>>> > https://issues.apache.org/jira/projects/SPARK and search for >>>>>>>>>>> "Target Version/s" = 2.3.2 >>>>>>>>>>> > >>>>>>>>>>> > Committers should look at those and triage. Extremely >>>>>>>>>>> important bug >>>>>>>>>>> > fixes, documentation, and API tweaks that impact compatibility >>>>>>>>>>> should >>>>>>>>>>> > be worked on immediately. Everything else please retarget to an >>>>>>>>>>> > appropriate release. >>>>>>>>>>> > >>>>>>>>>>> > ================== >>>>>>>>>>> > But my bug isn't fixed? >>>>>>>>>>> > ================== >>>>>>>>>>> > >>>>>>>>>>> > In order to make timely releases, we will typically not hold >>>>>>>>>>> the >>>>>>>>>>> > release unless the bug in question is a regression from the >>>>>>>>>>> previous >>>>>>>>>>> > release. That being said, if there is something which is a >>>>>>>>>>> regression >>>>>>>>>>> > that has not been correctly targeted please ping me or a >>>>>>>>>>> committer to >>>>>>>>>>> > help target the issue. >>>>>>>>>>> >>>>>>>>>>> ------------------------------------------------------------ >>>>>>>>>>> --------- >>>>>>>>>>> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org >>>>>>>>>>> >>>>>>>>>>> >>>>>>> >>>>>>> -- >>>>>>> --- >>>>>>> Takeshi Yamamuro >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> John >>>>>> >>>>> >>> >>> -- >>> Ryan Blue >>> Software Engineer >>> Netflix >>> >> > > -- > Ryan Blue > Software Engineer > Netflix >
