[
https://issues.apache.org/jira/browse/SPARK-34679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18046436#comment-18046436
]
Raghav Jindal commented on SPARK-34679:
---------------------------------------
[~P7hB] / [~gurwls223] / [~schilukoori] Can I pick this up is still unresolved?
> inferTimestamp option is missing from the list of options in
> DataFrameReader.json.
> ----------------------------------------------------------------------------------
>
> Key: SPARK-34679
> URL: https://issues.apache.org/jira/browse/SPARK-34679
> Project: Spark
> Issue Type: Documentation
> Components: Documentation
> Affects Versions: 3.0.1, 3.0.2, 3.1.0, 3.1.1
> Reporter: Prashanth Babu
> Priority: Minor
> Labels: documentation, easyfix, newbie, starter
> Original Estimate: 0.5h
> Remaining Estimate: 0.5h
>
> {color:#e01e5a}inferTimestamp{color} option is missing in the list of options
> in {color:#e01e5a}DataFrameReader.json{color} method in the API docs missing
> from the [Scaladocs
> here|[DataFrameReader.json|https://github.com/apache/spark/blob/v3.1.1/sql/core/src/main/scala/org/apache/spark/sql/DataFrameReader.scala#L432-L520]].
> Simiarly in the [Pyspark
> docs|[pyspark.sql.DataFrameReader.json|https://spark.apache.org/docs/latest/api/python/reference/api/pyspark.sql.DataFrameReader.json.html?highlight=json#pyspark.sql.DataFrameReader.json]]
> as well.
> However we have this blurb in the [migration guide|[Spark 3.0 to 3.0.1
> migration
> guide|https://spark.apache.org/docs/latest/sql-migration-guide.html#upgrading-from-spark-sql-30-to-301]]
> * In Spark 3.0, JSON datasource and JSON function {{schema_of_json}} infer
> TimestampType from string values if they match to the pattern defined by the
> JSON option {{timestampFormat}}. Since version 3.0.1, the timestamp type
> inference is disabled by default. Set the JSON option {{inferTimestamp}} to
> {{true}} to enable such type inference.
> We should add this in the documentation as well as there is a possibility
> that the Data Engineers might not be aware of this option.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
