[ https://issues.apache.org/jira/browse/SPARK-31887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17140127#comment-17140127 ]
Amit Gupta edited comment on SPARK-31887 at 6/21/20, 7:34 AM: -------------------------------------------------------------- This issue is fixed in spark 3.0. Spark -2.5- 2.4.5 and -2.6- 2.4.6 are affected by this. was (Author: amit24jan): This issue is fixed in spark 3.0. Spark 2.5 and 2.6 are affected by this. > Date casting to string is giving wrong value > -------------------------------------------- > > Key: SPARK-31887 > URL: https://issues.apache.org/jira/browse/SPARK-31887 > Project: Spark > Issue Type: Bug > Components: Spark Core > Affects Versions: 2.4.5 > Environment: The spark is running on cluster mode with Mesos. > > Mesos agents are dockerised running on Ubuntu 18. > > Timezone setting of docker instance: UTC > Timezone of server hosting docker: America/New_York > Timezone of driver machine: America/New_York > Reporter: Amit Gupta > Priority: Major > > The code converts the string to date and then write it in csv. > {code:java} > val x = Seq(("2020-02-19", "2020-02-19 05:11:00")).toDF("a", > "b").select('a.cast("date"), 'b.cast("timestamp")) > x.show() > +----------+-------------------+ > | a| b| > +----------+-------------------+ > |2020-02-19|2020-02-19 05:11:00| > +----------+-------------------+ > x.write.mode("overwrite").option("header", true).csv("/tmp/test1.csv") > {code} > > The date written in CSV file is different: > {code:java} > > snakebite cat "/tmp/test1.csv/*.csv" > a,b > 2020-02-18,2020-02-19T05:11:00.000Z{code} > -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org