[
https://issues.apache.org/jira/browse/SPARK-58410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tian Gao resolved SPARK-58410.
------------------------------
Fix Version/s: 4.4.0
Resolution: Fixed
Issue resolved by pull request 57611
[https://github.com/apache/spark/pull/57611]
> Remove unnecessary type: ignore comments in the DataFrame read/write API
> ------------------------------------------------------------------------
>
> Key: SPARK-58410
> URL: https://issues.apache.org/jira/browse/SPARK-58410
> Project: Spark
> Issue Type: Improvement
> Components: PySpark
> Affects Versions: 4.3.0
> Reporter: Haotian Sun
> Assignee: Haotian Sun
> Priority: Minor
> Labels: pull-request-available
> Fix For: 4.4.0
>
>
> Several `# type: ignore` comments in the PySpark DataFrame read/write API
> (DataFrameReader/DataFrameWriter and their streaming and Spark Connect
> variants) sit on top of fixable annotation gaps rather than genuine type
> deviations.
> This change removes 7 such comments across the read/write modules:
> 1. OptionUtils._set_opts (classic and Connect): the mixin calls
> self.option(...) without declaring that its host provides it, requiring a `#
> type: ignore[attr-defined]`. Declaring a small `SupportsOption` Protocol and
> annotating `self` documents the real contract and removes the ignores. (The
> sibling `self.schema(...)` call is intentionally left with its ignore:
> `schema` is a reader-only method but `_set_opts` is shared with writers, so
> requiring it on `self` would break writer call sites.)
> 2. DataFrameReader.load (Connect): a `paths` local was assigned from the wide
> `Optional[Union[str, List[str]]]` parameter and then reused across an
> isinstance narrowing, so mypy kept the wide first-assignment type and needed
> a `# type: ignore[arg-type]`. Declaring `paths: Optional[List[str]]` up front
> and normalizing the input removes the ignore and makes the None/str/list
> cases explicit.
> 3. DataStreamWriter.partitionBy / clusterBy (classic and Connect): the
> overload implementation signatures were `*cols: str`, which contradicts the
> `List[str]` overload and required `# type: ignore[misc]`; it also left the
> list-handling branch as dead code to the type checker. Widening the
> implementation signatures to `*cols: Union[str, List[str]]` (matching the
> non-streaming DataFrameWriter) makes them consistent with their overloads,
> and normalizing into a correctly typed `Sequence[str]` local avoids
> introducing any new ignore.
> No behavioral change; verified with mypy over the full `python/pyspark` scope.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
