[ https://issues.apache.org/jira/browse/ARROW-3850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Bryan Cutler resolved ARROW-3850. --------------------------------- Resolution: Fixed MapType has been added to PySpark, I think we can close this now. > [Python] Support MapType and StructType for enhanced PySpark integration > ------------------------------------------------------------------------ > > Key: ARROW-3850 > URL: https://issues.apache.org/jira/browse/ARROW-3850 > Project: Apache Arrow > Issue Type: New Feature > Components: Python > Affects Versions: 0.11.1 > Reporter: Florian Wilhelm > Priority: Major > Fix For: 3.0.0 > > > It would be great to support MapType and (nested) StructType in Arrow so that > PySpark can make use of it. > > Quite often as in my use-case in Hive table cells are also complex types > saved. Currently it's not possible to user the new > {{[pandas_udf|https://spark.apache.org/docs/latest/api/python/pyspark.sql.html?highlight=explode#pyspark.sql.functions.pandas_udf]}} > decorator which internally uses Arrow to generate a UDF for columns with > complex types. -- This message was sent by Atlassian Jira (v8.3.4#803005)