Changing my vote to +1 with this fixed. Here's what was going on -- and thanks to Owen O'Malley for debugging:
The problem was that Iceberg contained a fix for a JVM bug for timestamps before the unix epoch where the timestamp was off by 1s. Owen moved this code into ORC as well and using the new version of Spark pulled in the newer version of ORC. That meant that the values were "fixed" twice and were wrong. Updating the Iceberg code to rely on the fix in the version of ORC that Spark includes fixes the problem. On Thu, Sep 20, 2018 at 2:38 PM Dongjoon Hyun <dongjoon.h...@gmail.com> wrote: > Hi, Ryan. > > Could you share the result on 2.3.1 since this is 2.3.2 RC? That would be > helpful to narrow down the scope. > > Bests, > Dongjoon. > > On Thu, Sep 20, 2018 at 11:56 Ryan Blue <rb...@netflix.com.invalid> wrote: > >> -0 >> >> My DataSourceV2 implementation for Iceberg is failing ORC tests when I >> run with the 2.3.2 RC that pass when I run with 2.3.0. I'm tracking down >> the cause and will report back, but I'm -0 on the release because there may >> be a behavior change. >> >> On Thu, Sep 20, 2018 at 10:37 AM Denny Lee <denny.g....@gmail.com> wrote: >> >>> +1 >>> >>> On Thu, Sep 20, 2018 at 9:55 AM Xiao Li <gatorsm...@gmail.com> wrote: >>> >>>> +1 >>>> >>>> >>>> John Zhuge <john.zh...@gmail.com> 于2018年9月19日周三 下午1:17写道： >>>> >>>>> +1 (non-binding) >>>>> >>>>> Built on Ubuntu 16.04 with Maven flags: -Phadoop-2.7 -Pmesos -Pyarn >>>>> -Phive-thriftserver -Psparkr -Pkinesis-asl -Phadoop-provided >>>>> >>>>> java version "1.8.0_181" >>>>> Java(TM) SE Runtime Environment (build 1.8.0_181-b13) >>>>> Java HotSpot(TM) 64-Bit Server VM (build 25.181-b13, mixed mode) >>>>> >>>>> >>>>> On Wed, Sep 19, 2018 at 2:31 AM Takeshi Yamamuro < >>>>> linguin....@gmail.com> wrote: >>>>> >>>>>> +1 >>>>>> >>>>>> I also checked `-Pyarn -Phadoop-2.7 -Pkinesis-asl -Phive >>>>>> -Phive-thriftserve` on the openjdk below/macOSv10.12.6 >>>>>> >>>>>> $ java -version >>>>>> java version "1.8.0_181" >>>>>> Java(TM) SE Runtime Environment (build 1.8.0_181-b13) >>>>>> Java HotSpot(TM) 64-Bit Server VM (build 25.181-b13, mixed mode) >>>>>> >>>>>> >>>>>> On Wed, Sep 19, 2018 at 10:45 AM Dongjoon Hyun < >>>>>> dongjoon.h...@gmail.com> wrote: >>>>>> >>>>>>> +1. >>>>>>> >>>>>>> I tested with `-Pyarn -Phadoop-2.7 -Pkinesis-asl -Phive >>>>>>> -Phive-thriftserve` on OpenJDK(1.8.0_181)/CentOS 7.5. >>>>>>> >>>>>>> I hit the following test case failure once during testing, but it's >>>>>>> not persistent. >>>>>>> >>>>>>> KafkaContinuousSourceSuite >>>>>>> ... >>>>>>> subscribing topic by name from earliest offsets (failOnDataLoss: >>>>>>> false) *** FAILED *** >>>>>>> >>>>>>> Thank you, Saisai. >>>>>>> >>>>>>> Bests, >>>>>>> Dongjoon. >>>>>>> >>>>>>> On Mon, Sep 17, 2018 at 6:48 PM Saisai Shao <sai.sai.s...@gmail.com> >>>>>>> wrote: >>>>>>> >>>>>>>> +1 from my own side. >>>>>>>> >>>>>>>> Thanks >>>>>>>> Saisai >>>>>>>> >>>>>>>> Wenchen Fan <cloud0...@gmail.com> 于2018年9月18日周二 上午9:34写道： >>>>>>>> >>>>>>>>> +1. All the blocker issues are all resolved in 2.3.2 AFAIK. >>>>>>>>> >>>>>>>>> On Tue, Sep 18, 2018 at 9:23 AM Sean Owen <sro...@apache.org> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> +1 . Licenses and sigs check out as in previous 2.3.x releases. A >>>>>>>>>> build from source with most profiles passed for me. >>>>>>>>>> On Mon, Sep 17, 2018 at 8:17 AM Saisai Shao < >>>>>>>>>> sai.sai.s...@gmail.com> wrote: >>>>>>>>>> > >>>>>>>>>> > Please vote on releasing the following candidate as Apache >>>>>>>>>> Spark version 2.3.2. >>>>>>>>>> > >>>>>>>>>> > The vote is open until September 21 PST and passes if a >>>>>>>>>> majority +1 PMC votes are cast, with a minimum of 3 +1 votes. >>>>>>>>>> > >>>>>>>>>> > [ ] +1 Release this package as Apache Spark 2.3.2 >>>>>>>>>> > [ ] -1 Do not release this package because ... >>>>>>>>>> > >>>>>>>>>> > To learn more about Apache Spark, please see >>>>>>>>>> http://spark.apache.org/ >>>>>>>>>> > >>>>>>>>>> > The tag to be voted on is v2.3.2-rc6 (commit >>>>>>>>>> 02b510728c31b70e6035ad541bfcdc2b59dcd79a): >>>>>>>>>> > https://github.com/apache/spark/tree/v2.3.2-rc6 >>>>>>>>>> > >>>>>>>>>> > The release files, including signatures, digests, etc. can be >>>>>>>>>> found at: >>>>>>>>>> > https://dist.apache.org/repos/dist/dev/spark/v2.3.2-rc6-bin/ >>>>>>>>>> > >>>>>>>>>> > Signatures used for Spark RCs can be found in this file: >>>>>>>>>> > https://dist.apache.org/repos/dist/dev/spark/KEYS >>>>>>>>>> > >>>>>>>>>> > The staging repository for this release can be found at: >>>>>>>>>> > >>>>>>>>>> https://repository.apache.org/content/repositories/orgapachespark-1286/ >>>>>>>>>> > >>>>>>>>>> > The documentation corresponding to this release can be found at: >>>>>>>>>> > https://dist.apache.org/repos/dist/dev/spark/v2.3.2-rc6-docs/ >>>>>>>>>> > >>>>>>>>>> > The list of bug fixes going into 2.3.2 can be found at the >>>>>>>>>> following URL: >>>>>>>>>> > https://issues.apache.org/jira/projects/SPARK/versions/12343289 >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > FAQ >>>>>>>>>> > >>>>>>>>>> > ========================= >>>>>>>>>> > How can I help test this release? >>>>>>>>>> > ========================= >>>>>>>>>> > >>>>>>>>>> > If you are a Spark user, you can help us test this release by >>>>>>>>>> taking >>>>>>>>>> > an existing Spark workload and running on this release >>>>>>>>>> candidate, then >>>>>>>>>> > reporting any regressions. >>>>>>>>>> > >>>>>>>>>> > If you're working in PySpark you can set up a virtual env and >>>>>>>>>> install >>>>>>>>>> > the current RC and see if anything important breaks, in the >>>>>>>>>> Java/Scala >>>>>>>>>> > you can add the staging repository to your projects resolvers >>>>>>>>>> and test >>>>>>>>>> > with the RC (make sure to clean up the artifact cache >>>>>>>>>> before/after so >>>>>>>>>> > you don't end up building with a out of date RC going forward). >>>>>>>>>> > >>>>>>>>>> > =========================================== >>>>>>>>>> > What should happen to JIRA tickets still targeting 2.3.2? >>>>>>>>>> > =========================================== >>>>>>>>>> > >>>>>>>>>> > The current list of open tickets targeted at 2.3.2 can be found >>>>>>>>>> at: >>>>>>>>>> > https://issues.apache.org/jira/projects/SPARK and search for >>>>>>>>>> "Target Version/s" = 2.3.2 >>>>>>>>>> > >>>>>>>>>> > Committers should look at those and triage. Extremely important >>>>>>>>>> bug >>>>>>>>>> > fixes, documentation, and API tweaks that impact compatibility >>>>>>>>>> should >>>>>>>>>> > be worked on immediately. Everything else please retarget to an >>>>>>>>>> > appropriate release. >>>>>>>>>> > >>>>>>>>>> > ================== >>>>>>>>>> > But my bug isn't fixed? >>>>>>>>>> > ================== >>>>>>>>>> > >>>>>>>>>> > In order to make timely releases, we will typically not hold the >>>>>>>>>> > release unless the bug in question is a regression from the >>>>>>>>>> previous >>>>>>>>>> > release. That being said, if there is something which is a >>>>>>>>>> regression >>>>>>>>>> > that has not been correctly targeted please ping me or a >>>>>>>>>> committer to >>>>>>>>>> > help target the issue. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> --------------------------------------------------------------------- >>>>>>>>>> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org >>>>>>>>>> >>>>>>>>>> >>>>>> >>>>>> -- >>>>>> --- >>>>>> Takeshi Yamamuro >>>>>> >>>>> >>>>> >>>>> -- >>>>> John >>>>> >>>> >> >> -- >> Ryan Blue >> Software Engineer >> Netflix >> > -- Ryan Blue Software Engineer Netflix
