Haotian Sun created SPARK-58410:
-----------------------------------
Summary: Remove unnecessary type: ignore comments in the DataFrame
read/write API
Key: SPARK-58410
URL: https://issues.apache.org/jira/browse/SPARK-58410
Project: Spark
Issue Type: Improvement
Components: PySpark
Affects Versions: 4.3.0
Reporter: Haotian Sun
Several `# type: ignore` comments in the PySpark DataFrame read/write API
(DataFrameReader/DataFrameWriter and their streaming and Spark Connect
variants) sit on top of fixable annotation gaps rather than genuine type
deviations.
This change removes 7 such comments across the read/write modules:
1. OptionUtils._set_opts (classic and Connect): the mixin calls
self.option(...) without declaring that its host provides it, requiring a `#
type: ignore[attr-defined]`. Declaring a small `SupportsOption` Protocol and
annotating `self` documents the real contract and removes the ignores. (The
sibling `self.schema(...)` call is intentionally left with its ignore: `schema`
is a reader-only method but `_set_opts` is shared with writers, so requiring it
on `self` would break writer call sites.)
2. DataFrameReader.load (Connect): a `paths` local was assigned from the wide
`Optional[Union[str, List[str]]]` parameter and then reused across an
isinstance narrowing, so mypy kept the wide first-assignment type and needed a
`# type: ignore[arg-type]`. Declaring `paths: Optional[List[str]]` up front and
normalizing the input removes the ignore and makes the None/str/list cases
explicit.
3. DataStreamWriter.partitionBy / clusterBy (classic and Connect): the overload
implementation signatures were `*cols: str`, which contradicts the `List[str]`
overload and required `# type: ignore[misc]`; it also left the list-handling
branch as dead code to the type checker. Widening the implementation signatures
to `*cols: Union[str, List[str]]` (matching the non-streaming DataFrameWriter)
makes them consistent with their overloads, and normalizing into a correctly
typed `Sequence[str]` local avoids introducing any new ignore.
No behavioral change; verified with mypy over the full `python/pyspark` scope.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
