[ https://issues.apache.org/jira/browse/SPARK-29610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hyukjin Kwon resolved SPARK-29610. ---------------------------------- Resolution: Duplicate > Keys with Null values are discarded when using to_json function > --------------------------------------------------------------- > > Key: SPARK-29610 > URL: https://issues.apache.org/jira/browse/SPARK-29610 > Project: Spark > Issue Type: Bug > Components: Build > Affects Versions: 2.4.4 > Reporter: Jonathan > Priority: Major > > When calling to_json on a Struct if a key has Null as a value then the key is > thrown away. > {code:java} > import pyspark > import pyspark.sql.functions as F > l = [("a", "foo"), ("b", None)] > df = spark.createDataFrame(l, ["id", "data"]) > ( > df.select(F.struct("*").alias("payload")) > .withColumn("payload", > F.to_json(F.col("payload")) > ).select("payload") > .show() > ){code} > Produces the following output: > {noformat} > +--------------------+ > | payload| > +--------------------+ > |{"id":"a","data":...| > | {"id":"b"}| > +--------------------+{noformat} > The `data` key in the second row has just been silently deleted. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org