[
https://issues.apache.org/jira/browse/SPARK-24950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16560249#comment-16560249
]
Sean Owen commented on SPARK-24950:
-----------------------------------
It's pretty clear this is down to differences in how time zones are defined, as
they change over time and the JDK incorporates updated versions of the standard
definitions in each release.
It looks like the difference between _171 and _181 is the difference between
2018c and 2018e in this table:
http://www.oracle.com/technetwork/java/javase/tzdata-versions-138805.html
Nothing obviously relevant from Oracle's release notes. But I found this in the
notes for 2018d:
[http://mm.icann.org/pipermail/tz-announce/2018-March/000049.html]
"Enderbury and Kiritimati skipped New Year's Eve 1994, not New Year's Day 1995.
(Thanks to Kerry Shetline.)"
So the answer is probably that the test has to be updated to reflect the fix to
the timezone definition.
Of course, if the test changes, it also starts failing on older Java 8
versions! probably not worth it.
I'd suggest we resolve it by commenting this out with a note. There's no
evidence this is a problem in Spark itself.
> scala DateTimeUtilsSuite daysToMillis and millisToDays fails w/java 8 181-b13
> -----------------------------------------------------------------------------
>
> Key: SPARK-24950
> URL: https://issues.apache.org/jira/browse/SPARK-24950
> Project: Spark
> Issue Type: Bug
> Components: Build, Tests
> Affects Versions: 2.4.0
> Reporter: shane knapp
> Priority: Major
>
> during my travails to port the spark builds to run on ubuntu 16.04LTS, i have
> encountered a strange and apparently java version-specific failure on *one*
> specific unit test.
> the failure is here:
> [https://amplab.cs.berkeley.edu/jenkins/job/spark-master-test-sbt-hadoop-2.6-ubuntu-test/868/testReport/junit/org.apache.spark.sql.catalyst.util/DateTimeUtilsSuite/daysToMillis_and_millisToDays/]
> the java version on this worker is:
> sknapp@ubuntu-testing:~$ java -version
> java version "1.8.0_181"
> Java(TM) SE Runtime Environment (build 1.8.0_181-b13)
> Java HotSpot(TM) 64-Bit Server VM (build 25.181-b13, mixed mode)
> however, when i run this exact build on the other ubuntu workers, it passes.
> they systems are set up (for the most part) identically except for the java
> version:
> sknapp@amp-jenkins-staging-worker-02:~$ java -version
> java version "1.8.0_171"
> Java(TM) SE Runtime Environment (build 1.8.0_171-b11)
> Java HotSpot(TM) 64-Bit Server VM (build 25.171-b11, mixed mode)
> there are some minor kernel and other package differences on these ubuntu
> workers, but nothing that (in my opinion) would affect this test. i am
> willing to help investigate this, however.
> the test also passes on the centos 6.9 workers, which have the following java
> version installed:
> [sknapp@amp-jenkins-worker-05 ~]$ java -version
> java version "1.8.0_60"
> Java(TM) SE Runtime Environment (build 1.8.0_60-b27)
> Java HotSpot(TM) 64-Bit Server VM (build 25.60-b23, mixed mode)my guess is
> that either:
> sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeUtils.scala
> or
> sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/util/DateTimeUtilsSuite.scala
> is doing something wrong. i am not a scala expert by any means, so i'd
> really like some help in trying to un-block the project to port the builds to
> ubuntu.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
