[ https://issues.apache.org/jira/browse/SPARK-10588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14744157#comment-14744157 ]
Yin Huai commented on SPARK-10588: ---------------------------------- Right, that is better. > Saving a DataFrame containing only nulls to JSON doesn't work > ------------------------------------------------------------- > > Key: SPARK-10588 > URL: https://issues.apache.org/jira/browse/SPARK-10588 > Project: Spark > Issue Type: Bug > Components: SQL > Affects Versions: 1.5.0 > Reporter: Cheng Lian > Priority: Minor > > Snippets to reproduce this issue: > {noformat} > val path = "file:///tmp/spark/null" > // A single row containing a single null double, saving to JSON, wrong > sqlContext. > range(1).selectExpr("CAST(NULL AS DOUBLE) AS c0"). > write.mode("overwrite").json(path) > sqlContext.read.json(path).show() > ++ > || > ++ > || > ++ > // Two rows each containing a single null double, saving to JSON, wrong > sqlContext. > range(2).selectExpr("CAST(NULL AS DOUBLE) AS c0"). > write.mode("overwrite").json(path) > sqlContext.read.json(path).show() > ++ > || > ++ > || > || > ++ > // A single row containing two null doubles, saving to JSON, wrong > sqlContext. > range(1).selectExpr("CAST(NULL AS DOUBLE) AS c0", "CAST(NULL AS DOUBLE) AS > c1"). > write.mode("overwrite").json(path) > sqlContext.read.json(path).show() > ++ > || > ++ > || > ++ > // A single row containing a single null double, saving to Parquet, OK > sqlContext. > range(1).selectExpr("CAST(NULL AS DOUBLE) AS c0"). > write.mode("overwrite").parquet(path) > sqlContext.read.parquet(path).show() > +----+ > | d| > +----+ > |null| > +----+ > // Two rows, one containing a single null double, one containing non-null > double, saving to JSON, OK > sqlContext. > range(2).selectExpr("IF(id % 2 = 0, CAST(NULL AS DOUBLE), id) AS c0"). > write.mode("overwrite").json(path) > sqlContext.read.json(path).show() > +----+ > | d| > +----+ > |null| > | 1.0| > +----+ > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org