SaurabhChawla100 edited a comment on pull request #32558:
URL: https://github.com/apache/spark/pull/32558#issuecomment-851428101
> I think we should also add the parameter at `csv` and `json` at
`DataFrameWriter`, `DataFrameReader`, `DataStreamReader`, `DataStreamWriter`
(and also `readwriter.py` and `streaming.py`). @itholic can you help review
this one please?
Not able to understand what paramater is needed to add in the csv and json
at at `DataFrameWriter`, `DataFrameReader`, `DataStreamReader`,
`DataStreamWriter` (and also `readwriter.py` and `streaming.py`).
There is already `private var extraOptions =
CaseInsensitiveMap[String](Map.empty)` and this code inserts the options that
are added to it in
```
def option(key: String, value: String): DataFrameReader = {
this.extraOptions = this.extraOptions + (key -> value)
this
}
```
So this is working on giving in the option.
```
scala> spark.read.option("inferSchema", "true").option("inferDateType",
"true").option("dateFormat", "yyyy-MM-dd").json(Seq("""{"a": {"b": 1, "c":
"2021-02-26"}}""").toDS()).schema
res18: org.apache.spark.sql.types.StructType =
StructType(StructField(a,StructType(StructField(b,LongType,true),
StructField(c,DateType,true)),true))
```
in Pyspark
```
spark.read.option("inferSchema", "true").option("inferDateType",
"true").option("dateFormat", "yyyy-MM-dd").json("/testDir/test1.json").schema
StructType(List(StructField(a,StructType(List(StructField(b,LongType,true),StructField(c,DateType,true))),true)))
```
Please do let me know if my understanding is not correct here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
