This is an automated email from the ASF dual-hosted git repository.
wenchen pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 50bdc9b [SPARK-27423][SQL][FOLLOWUP] Minor polishes to Cast codegen
templates for Date <-> Timestamp
50bdc9b is described below
commit 50bdc9befad078c933876c640a393c9cb4e78838
Author: Kris Mok <kris....@databricks.com>
AuthorDate: Thu Apr 18 14:27:33 2019 +0800
[SPARK-27423][SQL][FOLLOWUP] Minor polishes to Cast codegen templates for
Date <-> Timestamp
## What changes were proposed in this pull request?
https://github.com/apache/spark/pull/24332 introduced an unnecessary
`import` statement and two slight issues in the codegen templates in `Cast` for
`Date` <-> `Timestamp`.
This PR removes the unused import statement and fixes the slight codegen
issue.
The issue in those two codegen templates is this pattern:
```scala
val zid = JavaCode.global(
ctx.addReferenceObj("zoneId", zoneId, "java.time.ZoneId"),
zoneId.getClass)
```
`zoneId` can refer to an instance of a non-public class, e.g.
`java.time.ZoneRegion`, and while this code correctly puts in the 3rd argument
to `ctx.addReferenceObj()`, it's still passing `zoneId.getClass` to
`JavaCode.global()` which is not desirable, but doesn't cause any immediate
bugs in this particular case, because `zid` is used in an expression
immediately afterwards.
If this `zid` ever needs to spill to any explicitly typed variables, e.g. a
local variable, and if the spill handling uses the `javaType` on this
`GlobalVariable`, it'd generate code that looks like:
```java
java.time.ZoneRegion value1 = ((java.time.ZoneId) references[2] /* literal
*/);
```
which would then be a real bug:
- a non-accessible type `java.time.ZoneRegion` is referenced in the
generated code, and
- `ZoneId` -> `ZoneRegion` requires an explicit downcast.
## How was this patch tested?
Existing tests. This PR does not change behavior, and the original PR won't
cause any real behavior bug to begin with.
Closes #24392 from rednaxelafx/spark-27423-followup.
Authored-by: Kris Mok <kris....@databricks.com>
Signed-off-by: Wenchen Fan <wenc...@databricks.com>
---
.../apache/spark/sql/catalyst/expressions/Cast.scala | 17 +++++++++++------
1 file changed, 11 insertions(+), 6 deletions(-)
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala
index f7bc8b9..3de9f5a 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala
@@ -18,7 +18,7 @@
package org.apache.spark.sql.catalyst.expressions
import java.math.{BigDecimal => JavaBigDecimal}
-import java.time.{LocalDate, LocalDateTime, LocalTime}
+import java.time.ZoneId
import java.util.concurrent.TimeUnit._
import org.apache.spark.SparkException
@@ -936,9 +936,10 @@ case class Cast(child: Expression, dataType: DataType,
timeZoneId: Option[String
}
"""
case TimestampType =>
+ val zoneIdClass = classOf[ZoneId]
val zid = JavaCode.global(
- ctx.addReferenceObj("zoneId", zoneId, "java.time.ZoneId"),
- zoneId.getClass)
+ ctx.addReferenceObj("zoneId", zoneId, zoneIdClass.getName),
+ zoneIdClass)
(c, evPrim, evNull) =>
code"""$evPrim =
org.apache.spark.sql.catalyst.util.DateTimeUtils.microsToEpochDays($c, $zid);"""
@@ -1028,7 +1029,10 @@ case class Cast(child: Expression, dataType: DataType,
timeZoneId: Option[String
from: DataType,
ctx: CodegenContext): CastFunction = from match {
case StringType =>
- val zid = ctx.addReferenceObj("zoneId", zoneId, "java.time.ZoneId")
+ val zoneIdClass = classOf[ZoneId]
+ val zid = JavaCode.global(
+ ctx.addReferenceObj("zoneId", zoneId, zoneIdClass.getName),
+ zoneIdClass)
val longOpt = ctx.freshVariable("longOpt", classOf[Option[Long]])
(c, evPrim, evNull) =>
code"""
@@ -1045,9 +1049,10 @@ case class Cast(child: Expression, dataType: DataType,
timeZoneId: Option[String
case _: IntegralType =>
(c, evPrim, evNull) => code"$evPrim = ${longToTimeStampCode(c)};"
case DateType =>
+ val zoneIdClass = classOf[ZoneId]
val zid = JavaCode.global(
- ctx.addReferenceObj("zoneId", zoneId, "java.time.ZoneId"),
- zoneId.getClass)
+ ctx.addReferenceObj("zoneId", zoneId, zoneIdClass.getName),
+ zoneIdClass)
(c, evPrim, evNull) =>
code"""$evPrim =
org.apache.spark.sql.catalyst.util.DateTimeUtils.epochDaysToMicros($c, $zid);"""
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
