[GitHub] spark issue #20666: [SPARK-23448][SQL] Clarify JSON and CSV parser behavior ...

[fuqiliang]

Github user fuqiliang commented on the issue:

    https://github.com/apache/spark/pull/20666
  
    for specify, the json file (Sanity4.json) is
    `{"a":"a1","int":1,"other":4.4}
    {"a":"a2","int":"","other":""}`
    
    code ：
    
    > val config = new SparkConf().setMaster("local[5]").setAppName("test")
    > val sc = SparkContext.getOrCreate(config)
    > val sql = new SQLContext(sc)
    > 
    >  val file_path = 
this.getClass.getClassLoader.getResource("Sanity4.json").getFile
    >  val df = sql.read.schema(null).json(file_path)
    >  df.show(30)
    
    
    
    then in spark 1.6, result is
    +---+----+-----+
    |  a| int|other|
    +---+----+-----+
    | a1|   1|  4.4|
    | a2|null| null|
    +---+----+-----+
    
    root
     |-- a: string (nullable = true)
     |-- int: long (nullable = true)
     |-- other: double (nullable = true)
    
    but in spark 2.2, result is
    +----+----+-----+
    |   a| int|other|
    +----+----+-----+
    |  a1|   1|  4.4|
    |null|null| null|
    +----+----+-----+
    
    root
     |-- a: string (nullable = true)
     |-- int: long (nullable = true)
     |-- other: double (nullable = true)
    



---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org