[ https://issues.apache.org/jira/browse/SPARK-31887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17141246#comment-17141246 ]
Hyukjin Kwon commented on SPARK-31887: -------------------------------------- Very likely there won't be 2.5 and 2.6. So we should be good here. > Date casting to string is giving wrong value > -------------------------------------------- > > Key: SPARK-31887 > URL: https://issues.apache.org/jira/browse/SPARK-31887 > Project: Spark > Issue Type: Bug > Components: Spark Core > Affects Versions: 2.4.5 > Environment: The spark is running on cluster mode with Mesos. > > Mesos agents are dockerised running on Ubuntu 18. > > Timezone setting of docker instance: UTC > Timezone of server hosting docker: America/New_York > Timezone of driver machine: America/New_York > Reporter: Amit Gupta > Priority: Major > > The code converts the string to date and then write it in csv. > {code:java} > val x = Seq(("2020-02-19", "2020-02-19 05:11:00")).toDF("a", > "b").select('a.cast("date"), 'b.cast("timestamp")) > x.show() > +----------+-------------------+ > | a| b| > +----------+-------------------+ > |2020-02-19|2020-02-19 05:11:00| > +----------+-------------------+ > x.write.mode("overwrite").option("header", true).csv("/tmp/test1.csv") > {code} > > The date written in CSV file is different: > {code:java} > > snakebite cat "/tmp/test1.csv/*.csv" > a,b > 2020-02-18,2020-02-19T05:11:00.000Z{code} > -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org