[
https://issues.apache.org/jira/browse/SPARK-58423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nikolina Vraneš updated SPARK-58423:
------------------------------------
Description:
A clause like ORDER BY / WHERE / HAVING / QUALIFY / DISTRIBUTE BY can reference
a column that is not in its operator's output. The single-pass resolver
resolves such a column from hidden output and appends it to the Project /
Aggregate / Window below, in
ResolvesNameByHiddenOutput.expandOperatorsOutputList, so the clause can
reference it.
That append never checked whether the child actually produces the column. When
the child drops it, the operator ends up referencing an attribute its child
does not output. This is an invalid query that the fixed-point analyzer rejects
with a clean MISSING_ATTRIBUTES error. Single-pass instead hit a failed assert
in the phase-2 ResolutionValidator and reported INTERNAL_ERROR.
The single-pass missingInput check in Resolver.validateOperatorResolution runs
once per operator when that operator is resolved. The operator that becomes
invalid here is a Project / Aggregate / Window further down the tree, which is
checked (and passes) when it is resolved; the later append that makes it
invalid does not re-run the check.
The fix runs the same missing-input check right after expandOperatorsOutputList
builds the operator, raising MISSING_ATTRIBUTES. A shared
QueryCompilationErrors.missingAttributesError builds the error for both the
single-pass call sites and the fixed-point CheckAnalysis. This only affects
invalid queries that both analyzers already reject; no valid query changes
result.
> Raise MISSING_ATTRIBUTES for missing input after single-pass hidden-output
> insertion
> ------------------------------------------------------------------------------------
>
> Key: SPARK-58423
> URL: https://issues.apache.org/jira/browse/SPARK-58423
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 5.0.0
> Reporter: Nikolina Vraneš
> Priority: Major
>
> A clause like ORDER BY / WHERE / HAVING / QUALIFY / DISTRIBUTE BY can
> reference a column that is not in its operator's output. The single-pass
> resolver resolves such a column from hidden output and appends it to the
> Project / Aggregate / Window below, in
> ResolvesNameByHiddenOutput.expandOperatorsOutputList, so the clause can
> reference it.
> That append never checked whether the child actually produces the column.
> When the child drops it, the operator ends up referencing an attribute its
> child does not output. This is an invalid query that the fixed-point analyzer
> rejects with a clean MISSING_ATTRIBUTES error. Single-pass instead hit a
> failed assert in the phase-2 ResolutionValidator and reported INTERNAL_ERROR.
> The single-pass missingInput check in Resolver.validateOperatorResolution
> runs once per operator when that operator is resolved. The operator that
> becomes invalid here is a Project / Aggregate / Window further down the tree,
> which is checked (and passes) when it is resolved; the later append that
> makes it invalid does not re-run the check.
> The fix runs the same missing-input check right after
> expandOperatorsOutputList builds the operator, raising MISSING_ATTRIBUTES. A
> shared QueryCompilationErrors.missingAttributesError builds the error for
> both the single-pass call sites and the fixed-point CheckAnalysis. This only
> affects invalid queries that both analyzers already reject; no valid query
> changes result.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
