Haotian Sun created SPARK-58522:
-----------------------------------
Summary: Accept a single tuple of columns in Window and TableArg
partitionBy/orderBy
Key: SPARK-58522
URL: https://issues.apache.org/jira/browse/SPARK-58522
Project: Spark
Issue Type: Improvement
Components: PySpark
Affects Versions: 4.3.0
Reporter: Haotian Sun
Window.partitionBy/orderBy (on both Window and WindowSpec) and
TableArg.partitionBy/orderBy accept their columns as varargs, and also allow a
single sequence to be passed in place of the varargs (e.g.
Window.partitionBy(["a", "b"])). Today that single-sequence form is unwrapped
only when it is a list: the runtime check is isinstance(cols[0], list), so
passing a tuple such as Window.partitionBy(("a", "b")) is not unwrapped and is
instead treated as a single, invalid column argument.
This is inconsistent with the other column-collecting varargs methods on
DataFrame (select, groupBy, rollup, cube), which already accept any non-str
Sequence and declare it with a two-overload signature, and with
describe/selectExpr, which were widened the same way in SPARK-58500.
The annotations are also inaccurate today, in opposite directions.
Window.partitionBy/orderBy are annotated *cols: Union[ColumnOrName,
Sequence[ColumnOrName]], so a tuple type-checks even though the runtime does
not unwrap it. TableArg.partitionBy/orderBy are annotated *cols: ColumnOrName,
which admits no sequence at all, even though the runtime unwraps a single list
and the docstrings document "str, Column, or list"; because the declared
element type has no sequence member, the list-handling branch is dead code to
the type checker.
This change widens the runtime unwrap check to accept any non-str Sequence,
widens the implementation signatures to Union[ColumnOrName,
Sequence[ColumnOrName]] across the base, classic, and connect layers, and adds
the standard two-overload public signature (a spread of columns, or a single
sequence of columns) which neither Window nor TableArg had. It also removes the
# type: ignore[assignment] comments on the single-sequence unwrap in the
_to_cols and _to_java_cols helpers, by assigning tuple(cols[0]) rather than
reusing the tuple-typed variable to hold a list.
Follow-up to SPARK-58488 and SPARK-58500: like describe/selectExpr, these
methods previously accepted a list only, so this adds tuple/sequence support (a
small, backward-compatible behavior change).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
