[ https://issues.apache.org/jira/browse/SPARK-20152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Navya Krishnappa updated SPARK-20152: ------------------------------------- Description: When reading the below mentioned time value by specifying the "timestampFormat": "MM-dd-yyyy'T'HH:mm:ss.SSSZZ", time zone is ignored. Source File: TimeColumn 03-21-2017T03:30:02Z Source code: Dataset dataset = getSqlContext().read() .option(DAWBConstant.PARSER_LIB, "commons") .option(INFER_SCHEMA, "true") .option(DAWBConstant.DELIMITER, ",") .option(DAWBConstant.QUOTE, "\"") .option(DAWBConstant.ESCAPE, "\\") .option("timestampFormat" , "MM-dd-yyyy'T'HH:mm:ss.SSSZZ") .option(DAWBConstant.MODE, Mode.PERMISSIVE) .csv(sourceFile); Result: TimeColumn [ StringType] and value is "03-21-2017T03:30:02Z", but expected result is TimeCoumn should be of "TimestampType" and should consider time zone for manipulation was: When reading the below mentioned time value by specifying the "timestampFormat": "MM-dd-yyyy'T'HH:mm:ss.SSSZZ", time zone is ignored. Source File: TimeColumn 03-21-2017T03:30:02Z Source code1: Dataset dataset = getSqlContext().read() .option(DAWBConstant.PARSER_LIB, "commons") .option(INFER_SCHEMA, "true") .option(DAWBConstant.DELIMITER, ",") .option(DAWBConstant.QUOTE, "\"") .option(DAWBConstant.ESCAPE, "\\") .option("timestampFormat" , "MM-dd-yyyy'T'HH:mm:ss.SSSZZ") .option(DAWBConstant.MODE, Mode.PERMISSIVE) .csv(sourceFile); Result: TimeColumn [ StringType] and value is "03-21-2017T03:30:02Z", but expected result is TimeCoumn should be of "TimestampType" and should consider time zone for manipulation Source code2: Dataset dataset = getSqlContext().read() .option(DAWBConstant.PARSER_LIB, "commons") .option(INFER_SCHEMA, "true") .option(DAWBConstant.DELIMITER, ",") .option(DAWBConstant.QUOTE, "\"") .option(DAWBConstant.ESCAPE, "\\") .option("timestampFormat" , "MM-dd-yyyy'T'HH:mm:ss") .option(DAWBConstant.MODE, Mode.PERMISSIVE) .csv(sourceFile); Result: TimeColumn [ TimestampType] and value is "2017-04-22 03:30:02.0", but expected result is TimeCoumn should consider time zone for manipulation > Time zone is not respected while parsing csv for timeStampFormat > "MM-dd-yyyy'T'HH:mm:ss.SSSZZ" > ---------------------------------------------------------------------------------------------- > > Key: SPARK-20152 > URL: https://issues.apache.org/jira/browse/SPARK-20152 > Project: Spark > Issue Type: Bug > Components: SQL > Affects Versions: 2.1.0 > Reporter: Navya Krishnappa > > When reading the below mentioned time value by specifying the > "timestampFormat": "MM-dd-yyyy'T'HH:mm:ss.SSSZZ", time zone is ignored. > Source File: > TimeColumn > 03-21-2017T03:30:02Z > Source code: > Dataset dataset = getSqlContext().read() > .option(DAWBConstant.PARSER_LIB, "commons") > .option(INFER_SCHEMA, "true") > .option(DAWBConstant.DELIMITER, ",") > .option(DAWBConstant.QUOTE, "\"") > .option(DAWBConstant.ESCAPE, "\\") > .option("timestampFormat" , "MM-dd-yyyy'T'HH:mm:ss.SSSZZ") > .option(DAWBConstant.MODE, Mode.PERMISSIVE) > .csv(sourceFile); > Result: TimeColumn [ StringType] and value is "03-21-2017T03:30:02Z", but > expected result is TimeCoumn should be of "TimestampType" and should > consider time zone for manipulation -- This message was sent by Atlassian JIRA (v6.3.15#6346) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org