[ https://issues.apache.org/jira/browse/SPARK-7506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15280606#comment-15280606 ]
Nicholas Chammas commented on SPARK-7506: ----------------------------------------- [~davies] - Would you be interested in a PR that adds new methods called {{toDict()}} and {{fromDict()}}, and made {{json()}} and {{fromJson()}} aliases of those methods? It would preserve compatibility, while correcting the naming problem here. > pyspark.sql.types.StructType.fromJson() is incorrectly named > ------------------------------------------------------------ > > Key: SPARK-7506 > URL: https://issues.apache.org/jira/browse/SPARK-7506 > Project: Spark > Issue Type: Sub-task > Components: PySpark, SQL > Affects Versions: 1.3.0, 1.3.1 > Reporter: Nicholas Chammas > Priority: Minor > > {code} > >>> json_rdd = sqlContext.jsonRDD(sc.parallelize(['{"name": "Nick"}'])) > >>> json_rdd.schema > StructType(List(StructField(name,StringType,true))) > >>> type(json_rdd.schema) > <class 'pyspark.sql.types.StructType'> > >>> json_rdd.schema.json() > '{"fields":[{"metadata":{},"name":"name","nullable":true,"type":"string"}],"type":"struct"}' > >>> pyspark.sql.types.StructType.fromJson(json_rdd.schema.json()) > Traceback (most recent call last): > File "<stdin>", line 1, in <module> > File > "/Applications/apache-spark/spark-1.3.1-bin-hadoop2.4/python/pyspark/sql/types.py", > line 346, in fromJson > return StructType([StructField.fromJson(f) for f in json["fields"]]) > TypeError: string indices must be integers, not str > >>> import json > >>> pyspark.sql.types.StructType.fromJson(json.loads(json_rdd.schema.json())) > StructType(List(StructField(name,StringType,true))) > >>> > {code} > So {{fromJson()}} doesn't actually expect JSON, which is a string. It expects > a dictionary. > This method should probably be renamed. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org