[
https://issues.apache.org/jira/browse/SPARK-58488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haotian Sun updated SPARK-58488:
--------------------------------
Description:
The column-name varargs methods on DataFrameWriter (partitionBy, clusterBy,
bucketBy, sortBy) and DataStreamWriter (partitionBy, clusterBy), in both
classic and Spark Connect, accept either multiple column names as varargs or a
single sequence of column names, unwrapping the sequence at runtime. Their type
annotations did not describe this accurately and relied on # type: ignore
comments:
- partitionBy/clusterBy were annotated List[str] but the runtime already
accepted a tuple; the streaming implementations were declared *cols: str with a
# type: ignore[misc] suppressing an overload mismatch.
- bucketBy/sortBy used a local TupleOrListOfString alias and reused a variable
across a type change, requiring # type: ignore[assignment].
- DataFrameNaFunctions.replace had the same overload issue SPARK-56731 fixed
for DataFrame.replace (subset positionally accepted after skipping value),
suppressed with # type: ignore[misc].
This widens the annotations to Sequence[str] (matching the approach in
SPARK-55967), makes the runtime checks consistent, and removes the unnecessary
# type: ignore comments. Widening to accept any sequence is backward compatible.
This is one of a few related PRs cleaning up the "varargs that also accept a
single sequence" typing pattern across PySpark.
was:
Many PySpark methods are declared as varargs (`*cols`) but also accept a single
sequence of the same elements passed as one argument, unwrapping it at runtime
with a check like `if len(cols) == 1 and isinstance(cols[0], (list, tuple)):
cols = cols[0]`.
Across the codebase these methods have type annotations that do not accurately
describe what they accept, and rely on `# type: ignore[assignment]` / `# type:
ignore[misc]` to suppress the resulting mismatches. Two recurring problems:
1. The annotation is narrower than the runtime contract â e.g. declared
`List[str]` while the code also accepts a `tuple` (any `Sequence`).
2. The implementation signature does not conform to its `@overload`
declarations, suppressed with `# type: ignore[misc]`, which leaves the
single-sequence call form effectively unchecked.
This is an umbrella for correcting these annotations across PySpark so they
honestly describe the accepted inputs and the suppression comments can be
removed, following the approach in SPARK-55967 (which unified and corrected the
column-conversion annotations for the connect DataFrame). Affected areas
include the DataFrame read/write and streaming writers
(`partitionBy`/`clusterBy`), DataFrame methods (`describe`, `selectExpr`,
`hint`, `withColumns`), the column-building functions (`struct`, `array`,
`map`, `map_concat`), and vector constructors in ML/MLlib linalg. The work will
be split into focused PRs per coherent area.
> Fix type annotations for varargs methods that also accept a single sequence
> ---------------------------------------------------------------------------
>
> Key: SPARK-58488
> URL: https://issues.apache.org/jira/browse/SPARK-58488
> Project: Spark
> Issue Type: Improvement
> Components: PySpark
> Affects Versions: 4.3.0
> Reporter: Haotian Sun
> Priority: Minor
>
> The column-name varargs methods on DataFrameWriter (partitionBy, clusterBy,
> bucketBy, sortBy) and DataStreamWriter (partitionBy, clusterBy), in both
> classic and Spark Connect, accept either multiple column names as varargs or
> a single sequence of column names, unwrapping the sequence at runtime. Their
> type annotations did not describe this accurately and relied on # type:
> ignore comments:
> - partitionBy/clusterBy were annotated List[str] but the runtime already
> accepted a tuple; the streaming implementations were declared *cols: str with
> a # type: ignore[misc] suppressing an overload mismatch.
> - bucketBy/sortBy used a local TupleOrListOfString alias and reused a
> variable across a type change, requiring # type: ignore[assignment].
> - DataFrameNaFunctions.replace had the same overload issue SPARK-56731 fixed
> for DataFrame.replace (subset positionally accepted after skipping value),
> suppressed with # type: ignore[misc].
> This widens the annotations to Sequence[str] (matching the approach in
> SPARK-55967), makes the runtime checks consistent, and removes the
> unnecessary # type: ignore comments. Widening to accept any sequence is
> backward compatible.
> This is one of a few related PRs cleaning up the "varargs that also accept a
> single sequence" typing pattern across PySpark.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
