Haotian Sun created SPARK-58488:
-----------------------------------
Summary: Fix type annotations for varargs methods that also accept
a single sequence
Key: SPARK-58488
URL: https://issues.apache.org/jira/browse/SPARK-58488
Project: Spark
Issue Type: Improvement
Components: PySpark
Affects Versions: 4.3.0
Reporter: Haotian Sun
Many PySpark methods are declared as varargs (`*cols`) but also accept a single
sequence of the same elements passed as one argument, unwrapping it at runtime
with a check like `if len(cols) == 1 and isinstance(cols[0], (list, tuple)):
cols = cols[0]`.
Across the codebase these methods have type annotations that do not accurately
describe what they accept, and rely on `# type: ignore[assignment]` / `# type:
ignore[misc]` to suppress the resulting mismatches. Two recurring problems:
1. The annotation is narrower than the runtime contract â e.g. declared
`List[str]` while the code also accepts a `tuple` (any `Sequence`).
2. The implementation signature does not conform to its `@overload`
declarations, suppressed with `# type: ignore[misc]`, which leaves the
single-sequence call form effectively unchecked.
This is an umbrella for correcting these annotations across PySpark so they
honestly describe the accepted inputs and the suppression comments can be
removed, following the approach in SPARK-55967 (which unified and corrected the
column-conversion annotations for the connect DataFrame). Affected areas
include the DataFrame read/write and streaming writers
(`partitionBy`/`clusterBy`), DataFrame methods (`describe`, `selectExpr`,
`hint`, `withColumns`), the column-building functions (`struct`, `array`,
`map`, `map_concat`), and vector constructors in ML/MLlib linalg. The work will
be split into focused PRs per coherent area.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
