Note:
In the code (org.apache.spark.sql.parquet.DefaultSource) I've found this:
val relation = if (doInsertion) {
// This is a hack. We always set
nullable/containsNull/valueContainsNull to true
// for the schema of a parquet data.
val df =
sqlContext.createDataFrame(
Hi,
I have serious problems with saving DataFrame as parquet file.
I read the data from the parquet file like this:
val df = sparkSqlCtx.parquetFile(inputFile.toString)
and print the schema (you can see both fields are required)
root
|-- time: long (nullable = false)
|-- time_ymdhms: long
