[jira] [Updated] (SPARK-40851) TimestampFormatter behavior changed when using the latest Java 8/11/17

2022-10-24 Thread Sean R. Owen (Jira) 


 [ 
https://issues.apache.org/jira/browse/SPARK-40851?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean R. Owen updated SPARK-40851:
-
Fix Version/s: 3.3.2
   (was: 3.3.1)

> TimestampFormatter behavior changed when using the latest Java 8/11/17
> --
>
> Key: SPARK-40851
> URL: https://issues.apache.org/jira/browse/SPARK-40851
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 3.4.0
>Reporter: Yang Jie
>Assignee: Yang Jie
>Priority: Blocker
> Fix For: 3.4.0, 3.2.3, 3.3.2
>
>
> {code:java}
> [info] *** 12 TESTS FAILED ***
> [error] Failed: Total 6746, Failed 12, Errors 0, Passed 6734, Ignored 5
> [error] Failed tests:
> [error]   org.apache.spark.sql.catalyst.expressions.CastWithAnsiOffSuite
> [error]   org.apache.spark.sql.catalyst.util.TimestampFormatterSuite
> [error]   org.apache.spark.sql.catalyst.expressions.CastWithAnsiOnSuite
> [error]   org.apache.spark.sql.catalyst.util.RebaseDateTimeSuite
> [error]   org.apache.spark.sql.catalyst.expressions.TryCastSuite {code}
> We can reproduce this issue using Java 8u352/11.0.17/17.0.5,  the test errors 
> are similar to the following:
> run
> {code:java}
> build/sbt clean "catalyst/testOnly *CastWithAnsiOffSuite" {code}
> with 8u352:
> {code:java}
> [info] - SPARK-35711: cast timestamp without time zone to timestamp with 
> local time zone *** FAILED *** (190 milliseconds)
> [info]   Incorrect evaluation (codegen off): cast(0001-01-01 00:00:00 as 
> timestamp), actual: -6213561782000, expected: -621355968 
> (ExpressionEvalHelper.scala:209)
> [info]   org.scalatest.exceptions.TestFailedException:
> [info]   at 
> org.scalatest.Assertions.newAssertionFailedException(Assertions.scala:472)
> [info]   at 
> org.scalatest.Assertions.newAssertionFailedException$(Assertions.scala:471)
> [info]   at 
> org.scalatest.funsuite.AnyFunSuite.newAssertionFailedException(AnyFunSuite.scala:1564)
> [info]   at org.scalatest.Assertions.fail(Assertions.scala:933)
> [info]   at org.scalatest.Assertions.fail$(Assertions.scala:929)
> [info]   at org.scalatest.funsuite.AnyFunSuite.fail(AnyFunSuite.scala:1564)
> [info]   at 
> org.apache.spark.sql.catalyst.expressions.ExpressionEvalHelper.checkEvaluationWithoutCodegen(ExpressionEvalHelper.scala:209)
> [info]   at 
> org.apache.spark.sql.catalyst.expressions.ExpressionEvalHelper.checkEvaluationWithoutCodegen$(ExpressionEvalHelper.scala:199)
> [info]   at 
> org.apache.spark.sql.catalyst.expressions.CastSuiteBase.checkEvaluationWithoutCodegen(CastSuiteBase.scala:49)
> [info]   at 
> org.apache.spark.sql.catalyst.expressions.ExpressionEvalHelper.checkEvaluation(ExpressionEvalHelper.scala:87)
> [info]   at 
> org.apache.spark.sql.catalyst.expressions.ExpressionEvalHelper.checkEvaluation$(ExpressionEvalHelper.scala:82)
> [info]   at 
> org.apache.spark.sql.catalyst.expressions.CastSuiteBase.checkEvaluation(CastSuiteBase.scala:49)
> [info]   at 
> org.apache.spark.sql.catalyst.expressions.CastSuiteBase.$anonfun$new$198(CastSuiteBase.scala:893)
> [info]   at 
> org.apache.spark.sql.catalyst.expressions.CastSuiteBase.$anonfun$new$198$adapted(CastSuiteBase.scala:890)
> [info]   at scala.collection.immutable.List.foreach(List.scala:431)
> [info]   at 
> org.apache.spark.sql.catalyst.expressions.CastSuiteBase.$anonfun$new$197(CastSuiteBase.scala:890)
> [info]   at 
> scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
> [info]   at 
> org.apache.spark.sql.catalyst.util.DateTimeTestUtils$.withDefaultTimeZone(DateTimeTestUtils.scala:61)
> [info]   at 
> org.apache.spark.sql.catalyst.expressions.CastSuiteBase.$anonfun$new$196(CastSuiteBase.scala:890)
> [info]   at 
> org.apache.spark.sql.catalyst.expressions.CastSuiteBase.$anonfun$new$196$adapted(CastSuiteBase.scala:888)
> [info]   at scala.collection.immutable.List.foreach(List.scala:431)
> [info]   at 
> org.apache.spark.sql.catalyst.expressions.CastSuiteBase.$anonfun$new$195(CastSuiteBase.scala:888)
> [info]   at 
> scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
> [info]   at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
> [info]   at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
> [info]   at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
> [info]   at org.scalatest.Transformer.apply(Transformer.scala:22)
> [info]   at org.scalatest.Transformer.apply(Transformer.scala:20)
> [info]   at 
> org.scalatest.funsuite.AnyFunSuiteLike$$anon$1.apply(AnyFunSuiteLike.scala:226)
> [info]   at 
> org.apache.spark.SparkFunSuite.withFixture(SparkFunSuite.scala:207)
> [info]   at 
> org.scalatest.funsuite.AnyFunSuiteLike.invokeWithFixture$1(AnyFunSuiteLike.scala:224)
> [info]   at 
>

[jira] [Updated] (SPARK-40851) TimestampFormatter behavior changed when using the latest Java 8/11/17

2022-10-24 Thread Hyukjin Kwon (Jira) 


 [ 
https://issues.apache.org/jira/browse/SPARK-40851?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hyukjin Kwon updated SPARK-40851:
-
Fix Version/s: 3.3.1
   3.2.3

> TimestampFormatter behavior changed when using the latest Java 8/11/17
> --
>
> Key: SPARK-40851
> URL: https://issues.apache.org/jira/browse/SPARK-40851
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 3.4.0
>Reporter: Yang Jie
>Assignee: Yang Jie
>Priority: Blocker
> Fix For: 3.4.0, 3.3.1, 3.2.3
>
>
> {code:java}
> [info] *** 12 TESTS FAILED ***
> [error] Failed: Total 6746, Failed 12, Errors 0, Passed 6734, Ignored 5
> [error] Failed tests:
> [error]   org.apache.spark.sql.catalyst.expressions.CastWithAnsiOffSuite
> [error]   org.apache.spark.sql.catalyst.util.TimestampFormatterSuite
> [error]   org.apache.spark.sql.catalyst.expressions.CastWithAnsiOnSuite
> [error]   org.apache.spark.sql.catalyst.util.RebaseDateTimeSuite
> [error]   org.apache.spark.sql.catalyst.expressions.TryCastSuite {code}
> We can reproduce this issue using Java 8u352/11.0.17/17.0.5,  the test errors 
> are similar to the following:
> run
> {code:java}
> build/sbt clean "catalyst/testOnly *CastWithAnsiOffSuite" {code}
> with 8u352:
> {code:java}
> [info] - SPARK-35711: cast timestamp without time zone to timestamp with 
> local time zone *** FAILED *** (190 milliseconds)
> [info]   Incorrect evaluation (codegen off): cast(0001-01-01 00:00:00 as 
> timestamp), actual: -6213561782000, expected: -621355968 
> (ExpressionEvalHelper.scala:209)
> [info]   org.scalatest.exceptions.TestFailedException:
> [info]   at 
> org.scalatest.Assertions.newAssertionFailedException(Assertions.scala:472)
> [info]   at 
> org.scalatest.Assertions.newAssertionFailedException$(Assertions.scala:471)
> [info]   at 
> org.scalatest.funsuite.AnyFunSuite.newAssertionFailedException(AnyFunSuite.scala:1564)
> [info]   at org.scalatest.Assertions.fail(Assertions.scala:933)
> [info]   at org.scalatest.Assertions.fail$(Assertions.scala:929)
> [info]   at org.scalatest.funsuite.AnyFunSuite.fail(AnyFunSuite.scala:1564)
> [info]   at 
> org.apache.spark.sql.catalyst.expressions.ExpressionEvalHelper.checkEvaluationWithoutCodegen(ExpressionEvalHelper.scala:209)
> [info]   at 
> org.apache.spark.sql.catalyst.expressions.ExpressionEvalHelper.checkEvaluationWithoutCodegen$(ExpressionEvalHelper.scala:199)
> [info]   at 
> org.apache.spark.sql.catalyst.expressions.CastSuiteBase.checkEvaluationWithoutCodegen(CastSuiteBase.scala:49)
> [info]   at 
> org.apache.spark.sql.catalyst.expressions.ExpressionEvalHelper.checkEvaluation(ExpressionEvalHelper.scala:87)
> [info]   at 
> org.apache.spark.sql.catalyst.expressions.ExpressionEvalHelper.checkEvaluation$(ExpressionEvalHelper.scala:82)
> [info]   at 
> org.apache.spark.sql.catalyst.expressions.CastSuiteBase.checkEvaluation(CastSuiteBase.scala:49)
> [info]   at 
> org.apache.spark.sql.catalyst.expressions.CastSuiteBase.$anonfun$new$198(CastSuiteBase.scala:893)
> [info]   at 
> org.apache.spark.sql.catalyst.expressions.CastSuiteBase.$anonfun$new$198$adapted(CastSuiteBase.scala:890)
> [info]   at scala.collection.immutable.List.foreach(List.scala:431)
> [info]   at 
> org.apache.spark.sql.catalyst.expressions.CastSuiteBase.$anonfun$new$197(CastSuiteBase.scala:890)
> [info]   at 
> scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
> [info]   at 
> org.apache.spark.sql.catalyst.util.DateTimeTestUtils$.withDefaultTimeZone(DateTimeTestUtils.scala:61)
> [info]   at 
> org.apache.spark.sql.catalyst.expressions.CastSuiteBase.$anonfun$new$196(CastSuiteBase.scala:890)
> [info]   at 
> org.apache.spark.sql.catalyst.expressions.CastSuiteBase.$anonfun$new$196$adapted(CastSuiteBase.scala:888)
> [info]   at scala.collection.immutable.List.foreach(List.scala:431)
> [info]   at 
> org.apache.spark.sql.catalyst.expressions.CastSuiteBase.$anonfun$new$195(CastSuiteBase.scala:888)
> [info]   at 
> scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
> [info]   at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
> [info]   at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
> [info]   at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
> [info]   at org.scalatest.Transformer.apply(Transformer.scala:22)
> [info]   at org.scalatest.Transformer.apply(Transformer.scala:20)
> [info]   at 
> org.scalatest.funsuite.AnyFunSuiteLike$$anon$1.apply(AnyFunSuiteLike.scala:226)
> [info]   at 
> org.apache.spark.SparkFunSuite.withFixture(SparkFunSuite.scala:207)
> [info]   at 
> org.scalatest.funsuite.AnyFunSuiteLike.invokeWithFixture$1(AnyFunSuiteLike.scala:224)
> [info]   at 
>

[jira] [Updated] (SPARK-40851) TimestampFormatter behavior changed when using the latest Java 8/11/17

2022-10-19 Thread Yang Jie (Jira) 


 [ 
https://issues.apache.org/jira/browse/SPARK-40851?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yang Jie updated SPARK-40851:
-
Summary: TimestampFormatter behavior changed when using the latest Java 
8/11/17  (was: TimestampFormatter behavior changed when using the latest Java)

> TimestampFormatter behavior changed when using the latest Java 8/11/17
> --
>
> Key: SPARK-40851
> URL: https://issues.apache.org/jira/browse/SPARK-40851
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 3.4.0
>Reporter: Yang Jie
>Priority: Blocker
>
> {code:java}
> [info] *** 12 TESTS FAILED ***
> [error] Failed: Total 6746, Failed 12, Errors 0, Passed 6734, Ignored 5
> [error] Failed tests:
> [error]   org.apache.spark.sql.catalyst.expressions.CastWithAnsiOffSuite
> [error]   org.apache.spark.sql.catalyst.util.TimestampFormatterSuite
> [error]   org.apache.spark.sql.catalyst.expressions.CastWithAnsiOnSuite
> [error]   org.apache.spark.sql.catalyst.util.RebaseDateTimeSuite
> [error]   org.apache.spark.sql.catalyst.expressions.TryCastSuite {code}
> We can reproduce this issue using Java 8u352/11.0.17/17.0.5,  the test errors 
> are similar to the following:
> run
> {code:java}
> build/sbt clean "catalyst/testOnly *CastWithAnsiOffSuite" {code}
> with 8u352:
> {code:java}
> [info] - SPARK-35711: cast timestamp without time zone to timestamp with 
> local time zone *** FAILED *** (190 milliseconds)
> [info]   Incorrect evaluation (codegen off): cast(0001-01-01 00:00:00 as 
> timestamp), actual: -6213561782000, expected: -621355968 
> (ExpressionEvalHelper.scala:209)
> [info]   org.scalatest.exceptions.TestFailedException:
> [info]   at 
> org.scalatest.Assertions.newAssertionFailedException(Assertions.scala:472)
> [info]   at 
> org.scalatest.Assertions.newAssertionFailedException$(Assertions.scala:471)
> [info]   at 
> org.scalatest.funsuite.AnyFunSuite.newAssertionFailedException(AnyFunSuite.scala:1564)
> [info]   at org.scalatest.Assertions.fail(Assertions.scala:933)
> [info]   at org.scalatest.Assertions.fail$(Assertions.scala:929)
> [info]   at org.scalatest.funsuite.AnyFunSuite.fail(AnyFunSuite.scala:1564)
> [info]   at 
> org.apache.spark.sql.catalyst.expressions.ExpressionEvalHelper.checkEvaluationWithoutCodegen(ExpressionEvalHelper.scala:209)
> [info]   at 
> org.apache.spark.sql.catalyst.expressions.ExpressionEvalHelper.checkEvaluationWithoutCodegen$(ExpressionEvalHelper.scala:199)
> [info]   at 
> org.apache.spark.sql.catalyst.expressions.CastSuiteBase.checkEvaluationWithoutCodegen(CastSuiteBase.scala:49)
> [info]   at 
> org.apache.spark.sql.catalyst.expressions.ExpressionEvalHelper.checkEvaluation(ExpressionEvalHelper.scala:87)
> [info]   at 
> org.apache.spark.sql.catalyst.expressions.ExpressionEvalHelper.checkEvaluation$(ExpressionEvalHelper.scala:82)
> [info]   at 
> org.apache.spark.sql.catalyst.expressions.CastSuiteBase.checkEvaluation(CastSuiteBase.scala:49)
> [info]   at 
> org.apache.spark.sql.catalyst.expressions.CastSuiteBase.$anonfun$new$198(CastSuiteBase.scala:893)
> [info]   at 
> org.apache.spark.sql.catalyst.expressions.CastSuiteBase.$anonfun$new$198$adapted(CastSuiteBase.scala:890)
> [info]   at scala.collection.immutable.List.foreach(List.scala:431)
> [info]   at 
> org.apache.spark.sql.catalyst.expressions.CastSuiteBase.$anonfun$new$197(CastSuiteBase.scala:890)
> [info]   at 
> scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
> [info]   at 
> org.apache.spark.sql.catalyst.util.DateTimeTestUtils$.withDefaultTimeZone(DateTimeTestUtils.scala:61)
> [info]   at 
> org.apache.spark.sql.catalyst.expressions.CastSuiteBase.$anonfun$new$196(CastSuiteBase.scala:890)
> [info]   at 
> org.apache.spark.sql.catalyst.expressions.CastSuiteBase.$anonfun$new$196$adapted(CastSuiteBase.scala:888)
> [info]   at scala.collection.immutable.List.foreach(List.scala:431)
> [info]   at 
> org.apache.spark.sql.catalyst.expressions.CastSuiteBase.$anonfun$new$195(CastSuiteBase.scala:888)
> [info]   at 
> scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
> [info]   at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
> [info]   at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
> [info]   at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
> [info]   at org.scalatest.Transformer.apply(Transformer.scala:22)
> [info]   at org.scalatest.Transformer.apply(Transformer.scala:20)
> [info]   at 
> org.scalatest.funsuite.AnyFunSuiteLike$$anon$1.apply(AnyFunSuiteLike.scala:226)
> [info]   at 
> org.apache.spark.SparkFunSuite.withFixture(SparkFunSuite.scala:207)
> [info]   at 
> org.scalatest.funsuite.AnyFunSuiteLike.invokeWithFixture$1(AnyFunSuiteLike.scala:224)
> [info]   at 
>