Haotian Sun created SPARK-58500:
-----------------------------------
Summary: Accept a single tuple of column names in
DataFrame.describe and selectExpr
Key: SPARK-58500
URL: https://issues.apache.org/jira/browse/SPARK-58500
Project: Spark
Issue Type: Improvement
Components: PySpark
Affects Versions: 4.3.0
Reporter: Haotian Sun
DataFrame.describe and DataFrame.selectExpr accept their column names / SQL
expressions as varargs, and also allow a single sequence to be passed in place
of the varargs (e.g. df.describe(["a", "b"])). Today that single-sequence form
is unwrapped only when it is a list: the runtime check is isinstance(x[0],
list), so passing a tuple such as df.describe(("a", "b")) is not unwrapped and
is instead treated as a single, invalid column argument.
This is inconsistent with the writer methods
(partitionBy/clusterBy/bucketBy/sortBy), which already accept either a list or
a tuple, and with the general expectation that these APIs take "a name or a
sequence of names".
This change widens the describe and selectExpr type annotations from Union[str,
List[str]] to Union[str, Sequence[str]] (overloads and implementation) across
the base, classic, and connect layers, and changes the runtime unwrap check to
accept any non-str Sequence, so a single tuple is unpacked the same way a list
is. It also normalizes the overload shape (describe had no overloads; connect
selectExpr had none).
Follow-up to SPARK-58488: unlike the writer methods, describe/selectExpr
previously accepted a list only, so this adds tuple/sequence support (a small,
backward-compatible behavior change).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
