[ https://issues.apache.org/jira/browse/SPARK-27037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16782940#comment-16782940 ]
Hyukjin Kwon commented on SPARK-27037: -------------------------------------- Use {{asDict(recursive=True)}} {code} >>> df.head().asDict(recursive=True) {'tuple_map': {Row(_1=u'hello', _2=True): {'_2': 111, '_1': 1234}}} {code} > Pyspark Row .asDict() cannot handle MapType with a Struct as the key or value > ----------------------------------------------------------------------------- > > Key: SPARK-27037 > URL: https://issues.apache.org/jira/browse/SPARK-27037 > Project: Spark > Issue Type: Bug > Components: PySpark > Affects Versions: 2.4.0 > Reporter: Tanjin Panna > Priority: Major > > When we have a tuple as the key or value in a {{MapType}} and call the > .{{asDict()}}, we still have a {{Row}} in the output for the key and value: > {code:java} > >>> from pyspark.sql import Row > >>>> df = spark.createDataFrame( > [ > Row(tuple_map={('hello', True): (1234, 111)}), > Row(tuple_map={('there', False): (5678, 343)}) > ] > ) > >>> df.schema > StructType(List(StructField(tuple_map,MapType(StructType(List(StructField(_1,StringType,true),StructField(_2,BooleanType,true))),StructType(List(StructField(_1,LongType,true),StructField(_2,LongType,true))),true),true))) > >>> df.show(truncate=False) > +-------------------------------+ > |tuple_map | > +-------------------------------+ > |[[hello, true] -> [1234, 111]] | > |[[there, false] -> [5678, 343]]| > +-------------------------------+ > >>> df.head().asDict() > {'tuple_map': {Row(_1=u'hello', _2=True): Row(_1=1234, _2=111)}} > {code} > -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org