[ https://issues.apache.org/jira/browse/SPARK-21691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16124532#comment-16124532 ]
Anton Okolnychyi commented on SPARK-21691: ------------------------------------------ The issue is related to `Project \[\*\]` but not to `Limit`. You will always have this exception if you have a non-top level `Project \[\*\]` in your logical plan. For instance, the following query will also produce the same exception: {noformat} spark.sql("select * from (select * from (values 0, 1)) as v").queryExecution.logical.canonicalized {noformat} In the failing example from the ticket description, the non-canonicalized logical plan looks like: {noformat} 'GlobalLimit 1 +- 'LocalLimit 1 +- 'Project [*] +- 'SubqueryAlias __auto_generated_subquery_name +- 'UnresolvedInlineTable [col1], [List(0), List(1)] {noformat} Once Spark tries to canonicalize it and processes `LocalLimit 1`, it will get all attributes by calling `children.flatMap(_.output)`, which triggers the problem. `Project#output` will try to convert its project list to attributes, which will fail for `UnresolvedStar` with the aforementioned exception. I see that `UnresolvedRelation` and `UnresolvedInlineTable` return Nil as output. Therefore, one option to fix this problem is to return `Nil` as output from `Project` if it is unresolved. {noformat} override def output: Seq[Attribute] = if (resolved) projectList.map(_.toAttribute) else Nil {noformat} I can fix it once we agree on a solution. > Accessing canonicalized plan for query with limit throws exception > ------------------------------------------------------------------ > > Key: SPARK-21691 > URL: https://issues.apache.org/jira/browse/SPARK-21691 > Project: Spark > Issue Type: Bug > Components: SQL > Affects Versions: 2.2.0 > Reporter: Bjoern Toldbod > > Accessing the logical, canonicalized plan fails for queries with limits. > The following demonstrates the issue: > {code:java} > val session = SparkSession.builder.master("local").getOrCreate() > // This works > session.sql("select * from (values 0, > 1)").queryExecution.logical.canonicalized > // This fails > session.sql("select * from (values 0, 1) limit > 1").queryExecution.logical.canonicalized > {code} > The message in the thrown exception is somewhat confusing (or at least not > directly related to the limit): > "Invalid call to toAttribute on unresolved object, tree: *" -- This message was sent by Atlassian JIRA (v6.4.14#64029) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org