[
https://issues.apache.org/jira/browse/SPARK-58522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-58522:
-----------------------------------
Labels: pull-request-available (was: )
> Accept a single tuple of columns in Window and TableArg partitionBy/orderBy
> ---------------------------------------------------------------------------
>
> Key: SPARK-58522
> URL: https://issues.apache.org/jira/browse/SPARK-58522
> Project: Spark
> Issue Type: Improvement
> Components: PySpark
> Affects Versions: 4.3.0
> Reporter: Haotian Sun
> Priority: Minor
> Labels: pull-request-available
>
> Window.partitionBy/orderBy (on both Window and WindowSpec) and
> TableArg.partitionBy/orderBy accept their columns as varargs, and also allow
> a single sequence to be passed in place of the varargs (e.g.
> Window.partitionBy(["a", "b"])). Today that single-sequence form is unwrapped
> only when it is a list: the runtime check is isinstance(cols[0], list), so
> passing a tuple such as Window.partitionBy(("a", "b")) is not unwrapped and
> is instead treated as a single, invalid column argument.
> This is inconsistent with the other column-collecting varargs methods on
> DataFrame (select, groupBy, rollup, cube), which already accept any non-str
> Sequence and declare it with a two-overload signature, and with
> describe/selectExpr, which were widened the same way in SPARK-58500.
> The annotations are also inaccurate today, in opposite directions.
> Window.partitionBy/orderBy are annotated *cols: Union[ColumnOrName,
> Sequence[ColumnOrName]], so a tuple type-checks even though the runtime does
> not unwrap it. TableArg.partitionBy/orderBy are annotated *cols:
> ColumnOrName, which admits no sequence at all, even though the runtime
> unwraps a single list and the docstrings document "str, Column, or list";
> because the declared element type has no sequence member, the list-handling
> branch is dead code to the type checker.
> This change widens the runtime unwrap check to accept any non-str Sequence,
> widens the implementation signatures to Union[ColumnOrName,
> Sequence[ColumnOrName]] across the base, classic, and connect layers, and
> adds the standard two-overload public signature (a spread of columns, or a
> single sequence of columns) which neither Window nor TableArg had. It also
> removes the # type: ignore[assignment] comments on the single-sequence unwrap
> in the _to_cols and _to_java_cols helpers, by assigning tuple(cols[0]) rather
> than reusing the tuple-typed variable to hold a list.
> Follow-up to SPARK-58488 and SPARK-58500: like describe/selectExpr, these
> methods previously accepted a list only, so this adds tuple/sequence support
> (a small, backward-compatible behavior change).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
