[
https://issues.apache.org/jira/browse/SPARK-62018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon resolved SPARK-62018.
----------------------------------
Fix Version/s: connect-rust-4.4.0
Resolution: Fixed
Issue resolved by pull request 861
[https://github.com/apache/spark-connect-rust/pull/861]
> Keep in the type stubs the docstrings that PySpark's objects have at run
> time, so that editors show PySpark's documentation as with pyspark-client
> --------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-62018
> URL: https://issues.apache.org/jira/browse/SPARK-62018
> Project: Spark
> Issue Type: Improvement
> Components: Connect
> Affects Versions: 5.0.0
> Reporter: Hyukjin Kwon
> Assignee: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.4.0
>
>
> The drop-in for pyspark-client 4.3.0 ships type stubs (.pyi) of the modules
> whose classes are native or whose names are made at import time
> (scripts/gen_type_stubs.py, SPARK-61854 and SPARK-62011), which type checkers
> and editors read instead of the modules. The stubs dropped every docstring,
> so an editor showed the signature of a function or method of these modules
> without its documentation.
> Repro: a hover of pyright 1.1.414's language server over these names, of
> `from pyspark.sql import functions as F; from pyspark.sql.dataframe import
> DataFrame; from pyspark.sql.column import Column`:
> - `F.col`: pyspark-client shows 416 characters, the signature and PySpark's
> docstring ("Returns a :class:`~pyspark.sql.Column` based on the given column
> name. ..."); the drop-in showed 38, the signature only.
> - `DataFrame.select`: 1097 characters with pyspark-client, the signatures
> only with the drop-in.
> - `Column.cast`: 828 characters with pyspark-client, the signature only with
> the drop-in.
> - The classes of pyspark.ml.connect: no docstring.
> Expected: the docstring of each object as PySpark has it at run time
> (inspect.getdoc of PySpark's object), which is not always the docstring of
> its source:
> - pyspark.since appends `.. versionadded::` (python/pyspark/__init__.py,
> since);
> - the modules of PySpark Connect copy the docstrings of the classic ones
> (`col.__doc__ = pysparkfuncs.col.__doc__`,
> python/pyspark/sql/connect/functions/builtin.py; `__doc__ =
> PySparkTableValuedFunction.__doc__` in a class body, connect/tvf.py);
> - a method of no docstring has that of its base (inspect.getdoc).
> Also, scripts/gen_type_stubs.py takes about 41 s of CPU to generate the
> stubs, mostly in ast.get_source_segment, which splits the whole source of
> PySpark's module again for each default value of a parameter that it renders.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]