[ https://issues.apache.org/jira/browse/SPARK-9101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Apache Spark reassigned SPARK-9101: ----------------------------------- Assignee: Apache Spark > Can't use null in selectExpr > ---------------------------- > > Key: SPARK-9101 > URL: https://issues.apache.org/jira/browse/SPARK-9101 > Project: Spark > Issue Type: Bug > Components: PySpark > Affects Versions: 1.4.0, 1.4.1 > Reporter: Mateusz Buśkiewicz > Assignee: Apache Spark > > In 1.3.1 this worked: > {code:python} > df = sqlContext.createDataFrame([[1]], schema=['col']) > df.selectExpr('null as newCol').collect() > {code} > In 1.4.0 it fails with the following stacktrace: > {code} > Traceback (most recent call last): > File "<input>", line 1, in <module> > File > "/opt/boxen/homebrew/opt/apache-spark/libexec/python/pyspark/sql/dataframe.py", > line 316, in collect > cls = _create_cls(self.schema) > File > "/opt/boxen/homebrew/opt/apache-spark/libexec/python/pyspark/sql/dataframe.py", > line 229, in schema > self._schema = _parse_datatype_json_string(self._jdf.schema().json()) > File > "/opt/boxen/homebrew/opt/apache-spark/libexec/python/pyspark/sql/types.py", > line 519, in _parse_datatype_json_string > return _parse_datatype_json_value(json.loads(json_string)) > File > "/opt/boxen/homebrew/opt/apache-spark/libexec/python/pyspark/sql/types.py", > line 539, in _parse_datatype_json_value > return _all_complex_types[tpe].fromJson(json_value) > File > "/opt/boxen/homebrew/opt/apache-spark/libexec/python/pyspark/sql/types.py", > line 386, in fromJson > return StructType([StructField.fromJson(f) for f in json["fields"]]) > File > "/opt/boxen/homebrew/opt/apache-spark/libexec/python/pyspark/sql/types.py", > line 347, in fromJson > _parse_datatype_json_value(json["type"]), > File > "/opt/boxen/homebrew/opt/apache-spark/libexec/python/pyspark/sql/types.py", > line 535, in _parse_datatype_json_value > raise ValueError("Could not parse datatype: %s" % json_value) > ValueError: Could not parse datatype: null > {code} > https://github.com/apache/spark/blob/v1.4.0/python/pyspark/sql/types.py#L461 > The cause:_atomic_types doesn't contain NullType -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org