JingsongLi commented on code in PR #10510:
URL: https://github.com/apache/paimon/pull/10510#discussion_r4253525326
##########
paimon-spark/paimon-spark-common/src/main/scala/org/apache/spark/sql/catalyst/parser/extensions/RewritePaimonFunctionCommands.scala:
##########
@@ -123,28 +125,91 @@ case class RewritePaimonFunctionCommands(spark:
SparkSession) extends Rule[Logic
u,
alias =>
transformPaimonV1Function(alias).asInstanceOf[SubqueryAlias])
case l: LogicalPlan =>
-
l.transformExpressionsWithPruning(_.containsAnyPattern(UNRESOLVED_FUNCTION)) {
+
l.transformExpressionsWithPruning(_.containsAnyPattern(UNRESOLVED_FUNCTION,
GENERATOR)) {
Review Comment:
[P2] Visit expression subquery plans when rewriting generators
This traversal does not enter `SubqueryExpression.plan`, so the new handler
misses ordinary scalar/EXISTS queries containing a REST UDTF in LATERAL VIEW.
After `CREATE FUNCTION review_udtf AS
'org.apache.hadoop.hive.ql.udf.generic.GenericUDTFPosExplode'`, I reproduced
`ROUTINE_NOT_FOUND` with:
```sql
SELECT (SELECT count(*)
FROM VALUES (array(10,20)) AS t(items)
LATERAL VIEW review_udtf(items) u AS position,item) AS n;
```
The equivalent `posexplode` query returns 2, and an EXISTS version fails for
the same reason. Spark's recursive subquery analyzer does not rerun this parser
rewrite; the remaining native `UnresolvedGenerator` goes to SessionCatalog
rather than Paimon's private registry. This leaves the PR's new LATERAL VIEW
support incomplete for count/existence query composition. Recurse through
expression subquery plans and replace them with their rewritten plans (while
retaining CTE handling), and add scalar/EXISTS integration cases.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
