[
https://issues.apache.org/jira/browse/SPARK-58500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-58500:
-----------------------------------
Labels: pull-request-available (was: )
> Accept a single tuple of column names in DataFrame.describe and selectExpr
> --------------------------------------------------------------------------
>
> Key: SPARK-58500
> URL: https://issues.apache.org/jira/browse/SPARK-58500
> Project: Spark
> Issue Type: Improvement
> Components: PySpark
> Affects Versions: 4.3.0
> Reporter: Haotian Sun
> Priority: Minor
> Labels: pull-request-available
>
> DataFrame.describe and DataFrame.selectExpr accept their column names / SQL
> expressions as varargs, and also allow a single sequence to be passed in
> place of the varargs (e.g. df.describe(["a", "b"])). Today that
> single-sequence form is unwrapped only when it is a list: the runtime check
> is isinstance(x[0], list), so passing a tuple such as df.describe(("a", "b"))
> is not unwrapped and is instead treated as a single, invalid column argument.
> This is inconsistent with the writer methods
> (partitionBy/clusterBy/bucketBy/sortBy), which already accept either a list
> or a tuple, and with the general expectation that these APIs take "a name or
> a sequence of names".
> This change widens the describe and selectExpr type annotations from
> Union[str, List[str]] to Union[str, Sequence[str]] (overloads and
> implementation) across the base, classic, and connect layers, and changes the
> runtime unwrap check to accept any non-str Sequence, so a single tuple is
> unpacked the same way a list is. It also normalizes the overload shape
> (describe had no overloads; connect selectExpr had none).
> Follow-up to SPARK-58488: unlike the writer methods, describe/selectExpr
> previously accepted a list only, so this adds tuple/sequence support (a
> small, backward-compatible behavior change).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
