[jira] [Updated] (SPARK-29575) from_json can produce nulls for fields which are marked as non-nullable

2019-10-28 Thread Hyukjin Kwon (Jira) 


 [ 
https://issues.apache.org/jira/browse/SPARK-29575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hyukjin Kwon updated SPARK-29575:
-
Component/s: SQL

> from_json can produce nulls for fields which are marked as non-nullable
> ---
>
> Key: SPARK-29575
> URL: https://issues.apache.org/jira/browse/SPARK-29575
> Project: Spark
>  Issue Type: Bug
>  Components: PySpark, SQL
>Affects Versions: 2.4.4
>Reporter: Victor Lopez
>Priority: Major
>
> I believe this issue was resolved elsewhere 
> (https://issues.apache.org/jira/browse/SPARK-23173), though for Pyspark this 
> bug seems to still be there.
> The issue appears when using {{from_json}} to parse a column in a Spark 
> dataframe. It seems like {{from_json}} ignores whether the schema provided 
> has any {{nullable:False}} property.
> {code:java}
> schema = T.StructType().add(T.StructField('id', T.LongType(), 
> nullable=False)).add(T.StructField('name', T.StringType(), nullable=False))
> data = [{'user': str({'name': 'joe', 'id':1})}, {'user': str({'name': 
> 'jane'})}]
> df = spark.read.json(sc.parallelize(data))
> df.withColumn("details", F.from_json("user", 
> schema)).select("details.*").show()
> {code}
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Updated] (SPARK-29575) from_json can produce nulls for fields which are marked as non-nullable

2019-10-23 Thread Victor Lopez (Jira) 


 [ 
https://issues.apache.org/jira/browse/SPARK-29575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Victor Lopez updated SPARK-29575:
-
Description: 
I believe this issue was resolved elsewhere 
(https://issues.apache.org/jira/browse/SPARK-23173), though for Pyspark this 
bug seems to still be there.

The issue appears when using {{from_json}} to parse a column in a Spark 
dataframe. It seems like {{from_json}} ignores whether the schema provided has 
any {{nullable:False}} property.
{code:java}
schema = T.StructType().add(T.StructField('id', T.LongType(), 
nullable=False)).add(T.StructField('name', T.StringType(), nullable=False))
data = [{'user': str({'name': 'joe', 'id':1})}, {'user': str({'name': 'jane'})}]
df = spark.read.json(sc.parallelize(data))
df.withColumn("details", F.from_json("user", schema)).select("details.*").show()
{code}
 

  was:
I believe this issue was resolved elsewhere 
(https://issues.apache.org/jira/browse/SPARK-23173), though for Pyspark this 
bug seems to still be there.

{{The issue appears when using from_json to parse a column in a Spark 
dataframe. It seems like from_json ignores whether the schema provided has any 
nullable:False property.}}
{code:java}
schema = T.StructType().add(T.StructField('id', T.LongType(), 
nullable=False)).add(T.StructField('name', T.StringType(), nullable=False))
data = [{'user': str({'name': 'joe', 'id':1})}, {'user': str({'name': 'jane'})}]
df = spark.read.json(sc.parallelize(data))
df.withColumn("details", F.from_json("user", schema)).select("details.*").show()
{code}
 


> from_json can produce nulls for fields which are marked as non-nullable
> ---
>
> Key: SPARK-29575
> URL: https://issues.apache.org/jira/browse/SPARK-29575
> Project: Spark
>  Issue Type: Bug
>  Components: PySpark
>Affects Versions: 2.4.4
>Reporter: Victor Lopez
>Priority: Major
>
> I believe this issue was resolved elsewhere 
> (https://issues.apache.org/jira/browse/SPARK-23173), though for Pyspark this 
> bug seems to still be there.
> The issue appears when using {{from_json}} to parse a column in a Spark 
> dataframe. It seems like {{from_json}} ignores whether the schema provided 
> has any {{nullable:False}} property.
> {code:java}
> schema = T.StructType().add(T.StructField('id', T.LongType(), 
> nullable=False)).add(T.StructField('name', T.StringType(), nullable=False))
> data = [{'user': str({'name': 'joe', 'id':1})}, {'user': str({'name': 
> 'jane'})}]
> df = spark.read.json(sc.parallelize(data))
> df.withColumn("details", F.from_json("user", 
> schema)).select("details.*").show()
> {code}
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Updated] (SPARK-29575) from_json can produce nulls for fields which are marked as non-nullable

2019-10-23 Thread Victor Lopez (Jira) 


 [ 
https://issues.apache.org/jira/browse/SPARK-29575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Victor Lopez updated SPARK-29575:
-
Description: 
I believe this issue was resolved elsewhere 
(https://issues.apache.org/jira/browse/SPARK-23173), though for Pyspark this 
bug seems to still be there.

{{The issue appears when using `from_json` to parse a column in a Spark 
dataframe. It seems like `from_json` ignores whether the schema provided has 
any `nullable:False` property.}}
{code:java}
schema = T.StructType().add(T.StructField('id', T.LongType(), 
nullable=False)).add(T.StructField('name', T.StringType(), nullable=False))
data = [{'user': str({'name': 'joe', 'id':1})}, {'user': str({'name': 'jane'})}]
df = spark.read.json(sc.parallelize(data))
df.withColumn("details", F.from_json("user", schema)).select("details.*").show()
{code}
 

  was:
I believe this issue was resolved elsewhere 
(https://issues.apache.org/jira/browse/SPARK-23173), though for Pyspark this 
bug seems to still be there.

The issue appears when using `from_json` to parse a column in a Spark 
dataframe. It seems like `from_json` ignores whether the schema provided has 
any `nullable:False` property.
{code:java}
schema = T.StructType().add(T.StructField('id', T.LongType(), 
nullable=False)).add(T.StructField('name', T.StringType(), nullable=False))
data = [{'user': str({'name': 'joe', 'id':1})}, {'user': str({'name': 'jane'})}]
df = spark.read.json(sc.parallelize(data))
df.withColumn("details", F.from_json("user", schema)).select("details.*").show()
{code}
 


> from_json can produce nulls for fields which are marked as non-nullable
> ---
>
> Key: SPARK-29575
> URL: https://issues.apache.org/jira/browse/SPARK-29575
> Project: Spark
>  Issue Type: Bug
>  Components: PySpark
>Affects Versions: 2.4.4
>Reporter: Victor Lopez
>Priority: Major
>
> I believe this issue was resolved elsewhere 
> (https://issues.apache.org/jira/browse/SPARK-23173), though for Pyspark this 
> bug seems to still be there.
> {{The issue appears when using `from_json` to parse a column in a Spark 
> dataframe. It seems like `from_json` ignores whether the schema provided has 
> any `nullable:False` property.}}
> {code:java}
> schema = T.StructType().add(T.StructField('id', T.LongType(), 
> nullable=False)).add(T.StructField('name', T.StringType(), nullable=False))
> data = [{'user': str({'name': 'joe', 'id':1})}, {'user': str({'name': 
> 'jane'})}]
> df = spark.read.json(sc.parallelize(data))
> df.withColumn("details", F.from_json("user", 
> schema)).select("details.*").show()
> {code}
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Updated] (SPARK-29575) from_json can produce nulls for fields which are marked as non-nullable

2019-10-23 Thread Victor Lopez (Jira) 


 [ 
https://issues.apache.org/jira/browse/SPARK-29575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Victor Lopez updated SPARK-29575:
-
Description: 
I believe this issue was resolved elsewhere 
(https://issues.apache.org/jira/browse/SPARK-23173), though for Pyspark this 
bug seems to still be there.

{{The issue appears when using from_json to parse a column in a Spark 
dataframe. It seems like from_json ignores whether the schema provided has any 
nullable:False property.}}
{code:java}
schema = T.StructType().add(T.StructField('id', T.LongType(), 
nullable=False)).add(T.StructField('name', T.StringType(), nullable=False))
data = [{'user': str({'name': 'joe', 'id':1})}, {'user': str({'name': 'jane'})}]
df = spark.read.json(sc.parallelize(data))
df.withColumn("details", F.from_json("user", schema)).select("details.*").show()
{code}
 

  was:
I believe this issue was resolved elsewhere 
(https://issues.apache.org/jira/browse/SPARK-23173), though for Pyspark this 
bug seems to still be there.

{{The issue appears when using `from_json` to parse a column in a Spark 
dataframe. It seems like `from_json` ignores whether the schema provided has 
any `nullable:False` property.}}
{code:java}
schema = T.StructType().add(T.StructField('id', T.LongType(), 
nullable=False)).add(T.StructField('name', T.StringType(), nullable=False))
data = [{'user': str({'name': 'joe', 'id':1})}, {'user': str({'name': 'jane'})}]
df = spark.read.json(sc.parallelize(data))
df.withColumn("details", F.from_json("user", schema)).select("details.*").show()
{code}
 


> from_json can produce nulls for fields which are marked as non-nullable
> ---
>
> Key: SPARK-29575
> URL: https://issues.apache.org/jira/browse/SPARK-29575
> Project: Spark
>  Issue Type: Bug
>  Components: PySpark
>Affects Versions: 2.4.4
>Reporter: Victor Lopez
>Priority: Major
>
> I believe this issue was resolved elsewhere 
> (https://issues.apache.org/jira/browse/SPARK-23173), though for Pyspark this 
> bug seems to still be there.
> {{The issue appears when using from_json to parse a column in a Spark 
> dataframe. It seems like from_json ignores whether the schema provided has 
> any nullable:False property.}}
> {code:java}
> schema = T.StructType().add(T.StructField('id', T.LongType(), 
> nullable=False)).add(T.StructField('name', T.StringType(), nullable=False))
> data = [{'user': str({'name': 'joe', 'id':1})}, {'user': str({'name': 
> 'jane'})}]
> df = spark.read.json(sc.parallelize(data))
> df.withColumn("details", F.from_json("user", 
> schema)).select("details.*").show()
> {code}
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Updated] (SPARK-29575) from_json can produce nulls for fields which are marked as non-nullable

2019-10-23 Thread Victor Lopez (Jira) 


 [ 
https://issues.apache.org/jira/browse/SPARK-29575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Victor Lopez updated SPARK-29575:
-
Description: 
I believe this issue was resolved elsewhere 
(https://issues.apache.org/jira/browse/SPARK-23173), though for Pyspark this 
bug seems to still be there.

The issue appears when using `from_json` to parse a column in a Spark 
dataframe. It seems like `from_json` ignores whether the schema provided has 
any `nullable:False` property.
{code:java}
schema = T.StructType().add(T.StructField('id', T.LongType(), 
nullable=False)).add(T.StructField('name', T.StringType(), nullable=False))
data = [{'user': str({'name': 'joe', 'id':1})}, {'user': str({'name': 'jane'})}]
df = spark.read.json(sc.parallelize(data))
df.withColumn("details", F.from_json("user", schema)).select("details.*").show()
{code}
 

  was:
I believe this issue was resolved elsewhere 
(https://issues.apache.org/jira/browse/SPARK-23173), though for Pyspark this 
bug seems to still be there.

The issue appears when using `from_json` to parse a column in a Spark 
dataframe. It seems like `from_json` ignores whether the schema provided has 
any `nullable:False` property.

  

 
{code:java}
schema = T.StructType().add(T.StructField('id', T.LongType(), 
nullable=False)).add(T.StructField('name', T.StringType(), nullable=False))
data = [{'user': str({'name': 'joe', 'id':1})}, {'user': str({'name': 'jane'})}]
df = spark.read.json(sc.parallelize(data))
df.withColumn("details", F.from_json("user", schema)).select("details.*").show()
{code}
 


> from_json can produce nulls for fields which are marked as non-nullable
> ---
>
> Key: SPARK-29575
> URL: https://issues.apache.org/jira/browse/SPARK-29575
> Project: Spark
>  Issue Type: Bug
>  Components: PySpark
>Affects Versions: 2.4.4
>Reporter: Victor Lopez
>Priority: Major
>
> I believe this issue was resolved elsewhere 
> (https://issues.apache.org/jira/browse/SPARK-23173), though for Pyspark this 
> bug seems to still be there.
> The issue appears when using `from_json` to parse a column in a Spark 
> dataframe. It seems like `from_json` ignores whether the schema provided has 
> any `nullable:False` property.
> {code:java}
> schema = T.StructType().add(T.StructField('id', T.LongType(), 
> nullable=False)).add(T.StructField('name', T.StringType(), nullable=False))
> data = [{'user': str({'name': 'joe', 'id':1})}, {'user': str({'name': 
> 'jane'})}]
> df = spark.read.json(sc.parallelize(data))
> df.withColumn("details", F.from_json("user", 
> schema)).select("details.*").show()
> {code}
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org