HyukjinKwon commented on PR #59310:
URL: https://github.com/apache/spark/pull/59310#issuecomment-6162582941
Sure, here's the short version. I agree with the goal. When no
`WithRelations` attaches the plan id,
`CONNECT_INVALID_PLAN.SUBQUERY_PLAN_NOT_ATTACHED` is the right error, and that
part is done.
The remaining ask covers one case: the plan was attached, but the server
analyzed the expression before substituting it. `transformWithRelations`
transforms the root first and substitutes the attached plans only afterwards.
`transformStatSampleBy` (classic `sampleBy` runs `df.filter`) and
`transformAsOfJoin` (`joinAsOf` analyzes the join) analyze eagerly during that
root transform. For example, the Python client sends
`df.sampleBy(F.when(F.col("id").isin(sub_df), "a").otherwise("b"), {"a": 0.5})`
with `sub_df` attached in `WithRelations`, yet the new case reports that the
plan is not attached. That is a server-side ordering bug, not a client error,
so `INTERNAL_ERROR` is still the accurate classification there.
One way to handle it: in `transformWithRelations`, collect the reference
plan ids before transforming the root. If the root transform throws
`SUBQUERY_PLAN_NOT_ATTACHED` for one of those ids, rethrow it as an internal
error. Also reword the `CheckAnalysis` comment so it doesn't claim that a
surviving plan id always means the client didn't attach it, and add a planner
test for each outcome. This is non-blocking.
<!-- SPARK_DEV_REVIEW_REPLY
{"feedback_id":"conversation:6161562743","thread_id":"conversation:6161562743","verdict_sha256":"bf7427dcb583ee516224913fb052db034f38cff965df3e1e146698670ba916ad"}
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
