[
https://issues.apache.org/jira/browse/SPARK-58488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haotian Sun updated SPARK-58488:
--------------------------------
Summary: Fix type annotations for DataFrameWriter and DataStreamWriter
partitionBy/clusterBy/bucketBy/sortBy (was: Fix type annotations for varargs
methods that also accept a single sequence)
> Fix type annotations for DataFrameWriter and DataStreamWriter
> partitionBy/clusterBy/bucketBy/sortBy
> ---------------------------------------------------------------------------------------------------
>
> Key: SPARK-58488
> URL: https://issues.apache.org/jira/browse/SPARK-58488
> Project: Spark
> Issue Type: Improvement
> Components: PySpark
> Affects Versions: 4.3.0
> Reporter: Haotian Sun
> Priority: Minor
> Labels: pull-request-available
>
> The column-name varargs methods on DataFrameWriter (partitionBy, clusterBy,
> bucketBy, sortBy) and DataStreamWriter (partitionBy, clusterBy), in both
> classic and Spark Connect, accept either multiple column names as varargs or
> a single sequence of column names, unwrapping the sequence at runtime. Their
> type annotations did not describe this accurately and relied on # type:
> ignore comments:
> - partitionBy/clusterBy were annotated List[str] but the runtime already
> accepted a tuple; the streaming implementations were declared *cols: str with
> a # type: ignore[misc] suppressing an overload mismatch.
> - bucketBy/sortBy used a local TupleOrListOfString alias and reused a
> variable across a type change, requiring # type: ignore[assignment].
> - DataFrameNaFunctions.replace had the same overload issue SPARK-56731 fixed
> for DataFrame.replace (subset positionally accepted after skipping value),
> suppressed with # type: ignore[misc].
> This widens the annotations to Sequence[str] (matching the approach in
> SPARK-55967), makes the runtime checks consistent, and removes the
> unnecessary # type: ignore comments. Widening to accept any sequence is
> backward compatible.
> This is one of a few related PRs cleaning up the "varargs that also accept a
> single sequence" typing pattern across PySpark.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
