Spenserrrr opened a new pull request, #57702:
URL: https://github.com/apache/spark/pull/57702
### What changes were proposed in this pull request?
`DataFrame.describe` and `DataFrame.selectExpr` accept their column names /
SQL expressions as varargs, and also allow a single sequence to be passed in
place of the varargs (e.g. `df.describe(["a", "b"])`). Today that
single-sequence form is unwrapped only when it is a `list` — the runtime check
is `isinstance(x[0], list)` — so passing a tuple such as `df.describe(("a",
"b"))` is not unwrapped and is instead treated as a single, invalid column
argument.
This PR:
- Widens the `describe` and `selectExpr` type annotations from `Union[str,
List[str]]` to `Union[str, Sequence[str]]` (overloads + implementation) across
the base (`sql/dataframe.py`), classic (`sql/classic/dataframe.py`), and
connect (`sql/connect/dataframe.py`) layers.
- Changes the runtime unwrap check from `isinstance(x[0], list)` to `not
isinstance(x[0], str) and isinstance(x[0], Sequence)`, and unwraps with `x =
tuple(x[0])`, so a single tuple is unpacked the same way a list is.
- Normalizes the overload shape: `describe` had no overloads in any layer
(added the 2-overload shape `*x: str` / `__x: Sequence[str]`); `selectExpr` had
a `*expr: List[str]` overload in base + classic (fixed to a single-arg `__expr:
Sequence[str]`) and none in connect (added). Drops the now-unnecessary `# type:
ignore[assignment]` on the unwrap.
This is a follow-up to the SPARK-58488 cleanup (#57693). Unlike the writer
methods (`partitionBy`/`clusterBy`/`bucketBy`/`sortBy`), which already accepted
a list or a tuple, `describe`/`selectExpr` accepted a `list` only, so this adds
tuple/sequence support.
### Why are the changes needed?
The runtime already intends to accept "a name or a sequence of names", but
the check and the annotation only admitted `list`. Passing a tuple silently did
the wrong thing (treated the whole tuple as one column/expression) rather than
being unpacked. Accepting any non-str `Sequence` makes the behavior consistent
with the writer methods and with the general varargs-or-sequence convention in
the DataFrame API, and makes the type annotation honest about what is accepted.
### Does this PR introduce _any_ user-facing change?
Yes. Previously `df.describe(("a", "b"))` / `df.selectExpr(("e1", "e2"))` (a
single tuple) was not unpacked and did not behave like the list form; now a
single tuple of column names / expressions is accepted and unpacked just like a
list. This is a backward-compatible addition — the existing str-varargs and
single-list forms are unchanged.
### How was this patch tested?
- Added tuple-form cases to the existing connect-vs-classic parity tests
`test_connect_stat.py::test_describe` and
`test_connect_basic.py::test_select_expr` (these assert the same call on the
classic and connect sessions produce equal results, so they cover both layers).
Both pass.
- Full-scope `mypy` over `python/pyspark` is clean.
- `ruff format --check` and `ruff check` pass on the changed files.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 4.8)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
