inferSchema() will work better than jsonRDD() in your case, >>> from pyspark.sql import Row >>> srdd = sqlContext.inferSchema(rdd.map(lambda x: Row(**x))) >>> srdd.first() Row( field1=5, field2='string', field3={'a'=1, 'c'=2})
On Wed, Dec 3, 2014 at 12:11 AM, sahanbull <sa...@skimlinks.com> wrote: > Hi Guys, > > I am trying to use SparkSQL to convert an RDD to SchemaRDD so that I can > save it in parquet format. > > A record in my RDD has the following format: > > RDD1 > { > field1:5, > field2: 'string', > field3: {'a':1, 'c':2} > } > > I am using field3 to represent a "sparse vector" and it can have keys: > 'a','b' or 'c' and values any int value > > The current approach I am using is : > schemaRDD1 = sqc.jsonRDD(RDD1.map(lambda x: simplejson.dumps(x))) > > But when I do this, the dictionary in field 3 also gets converted to a > SparkSQL Row. This converts field3 to be a dense data structure where it > holds value None for every key that is not present in the field 3 for each > record. > > When I do > > test = RDD1.map(lambda x: simplejson.dumps(x)) > test.first() > > my output is: {"field1": 5, "field2":"string", "field3" :{"a":1,"c":2}} > > But then when I do > schemaRDD = sqc.jsonRDD(test) > schemaRDD.first() > > my output is : Row( field1=5, field2='string', field3 = Row(a=1,b=None,c=2)) > > in realty, I have 1000s of probable keys in field 3 and only 2 to 3 of them > occur per record. So When tic converts to a Row, it generates thousands of > None fields per record. > Is there anyways for me to store "field3" as a dictionary instead of > converting it into a Row in the schemaRDD?? > > > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/Using-sparkSQL-to-convert-a-collection-of-python-dictionary-of-dictionaries-to-schma-RDD-tp20228.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > --------------------------------------------------------------------- > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org