[
https://issues.apache.org/jira/browse/SPARK-51692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18030048#comment-18030048
]
Zizon commented on SPARK-51692:
-------------------------------
I encounter the same issue.
Any one looking into it?
> can not find field for sql with partition prune and cte
> -------------------------------------------------------
>
> Key: SPARK-51692
> URL: https://issues.apache.org/jira/browse/SPARK-51692
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 3.5.5, 4.0.0
> Environment: spark3.5
> Reporter: lifulong
> Priority: Major
> Labels: pull-request-available
>
> java.lang.IllegalStateException: Couldn't find client_id#49 in
> [p_date#3543,client_id#3428,user_classify#3504]
> at
> org.apache.spark.sql.catalyst.expressions.BindReferences$$anonfun$bindReference$1.applyOrElse(BoundAttribute.scala:80)
> at
> org.apache.spark.sql.catalyst.expressions.BindReferences$$anonfun$bindReference$1.applyOrElse(BoundAttribute.scala:73)
> at
> org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformDownWithPruning$1(TreeNode.scala:461)
> at
> org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(origin.scala:76)
> at
> org.apache.spark.sql.catalyst.trees.TreeNode.transformDownWithPruning(TreeNode.scala:461)
> at
> org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:437)
> at
> org.apache.spark.sql.catalyst.trees.TreeNode.transform(TreeNode.scala:405)
> at
> org.apache.spark.sql.catalyst.expressions.BindReferences$.bindReference(BoundAttribute.scala:73)
> at
> org.apache.spark.sql.catalyst.expressions.BindReferences$.$anonfun$bindReferences$1(BoundAttribute.scala:94)
> at scala.collection.immutable.List.map(List.scala:293)
> at
> org.apache.spark.sql.catalyst.expressions.BindReferences$.bindReferences(BoundAttribute.scala:94)
> at
> org.apache.spark.sql.execution.adaptive.PlanAdaptiveDynamicPruningFilters$$anonfun$apply$2.applyOrElse(PlanAdaptiveDynamicPruningFilters.scala:45)
> at
> org.apache.spark.sql.execution.adaptive.PlanAdaptiveDynamicPruningFilters$$anonfun$apply$2.applyOrElse(PlanAdaptiveDynamicPruningFilters.scala:40)
> at
> org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformDownWithPruning$1(TreeNode.scala:461)
>
> Rule PartitionPruning will add DynamicPruningSubquery Filter for partition
> prune, DynamicPruningSubquery Node contains prune joinKeys and
> filteringPlan, that info will finally used for gen
> SubqueryAdaptiveBroadcastExec node
>
> Rule ReplaceCTERefWithRepartition will gen new filed ref when replace CTE
> ref, but do not replace buildKeys in SubqueryAdaptiveBroadcastExec node which
> contains in CTE.
>
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
