[ https://issues.apache.org/jira/browse/SPARK-11281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15007272#comment-15007272 ]
Apache Spark commented on SPARK-11281: -------------------------------------- User 'zero323' has created a pull request for this issue: https://github.com/apache/spark/pull/9743 > Issue with creating and collecting DataFrame using environments > ---------------------------------------------------------------- > > Key: SPARK-11281 > URL: https://issues.apache.org/jira/browse/SPARK-11281 > Project: Spark > Issue Type: Bug > Components: SparkR > Affects Versions: 1.6.0 > Environment: R 3.2.2, Spark build from master > 487d409e71767c76399217a07af8de1bb0da7aa8 > Reporter: Maciej Szymkiewicz > Fix For: 1.6.0 > > > It is not possible to to access Map field created from an environment. > Assuming local data frame is created as follows: > {code} > ldf <- data.frame(row.names=1:2) > ldf$x <- c(as.environment(list(a=1, b=2)), as.environment(list(c=3))) > str(ldf) > ## 'data.frame': 2 obs. of 1 variable: > ## $ x:List of 2 > ## ..$ :<environment: 0x35c94d8> > ## ..$ :<environment: 0x35c7ac0> > get("a", ldf$x[[1]]) > ## [1] 1 > get("c", ldf$x[[2]]) > ## [1] 3 > {code} > It is possible to create a Spark data frame: > {code} > sdf <- createDataFrame(sqlContext, ldf) > printSchema(sdf) > ## root > ## |-- x: array (nullable = true) > ## | |-- element: map (containsNull = true) > ## | | |-- key: string > ## | | |-- value: double (valueContainsNull = true) > {code} > but it throws: > {code} > java.lang.IllegalArgumentException: Invalid array type e > {code} > on collect / head. > Problem seems to be specific to environments and cannot be reproduced when > Map comes for example from Cassandra table. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org