[ 
https://issues.apache.org/jira/browse/SPARK-62011?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hyukjin Kwon resolved SPARK-62011.
----------------------------------
    Fix Version/s: connect-rust-4.4.0
       Resolution: Fixed

Issue resolved by pull request 857
[https://github.com/apache/spark-connect-rust/pull/857]

> Make type checkers see the messages of pyspark.sql.connect.proto, the Avro 
> and Protobuf functions of Spark Connect, the classes of pyspark.ml.connect 
> and PySpark's classic SQLContext, as with pyspark-client
> --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-62011
>                 URL: https://issues.apache.org/jira/browse/SPARK-62011
>             Project: Spark
>          Issue Type: Bug
>          Components: Connect
>    Affects Versions: 5.0.0
>            Reporter: Hyukjin Kwon
>            Assignee: Hyukjin Kwon
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: connect-rust-4.4.0
>
>
> The drop-in for pyspark-client 4.3.0 ships stubs (.pyi) of the modules whose 
> classes are native or whose names are made at import time 
> (scripts/gen_type_stubs.py), so that mypy and pyright type an application as 
> with pyspark-client. Several modules had no stub, or a stub that missed a 
> name, and SQLContext was the drop-in's own class of fewer methods.
> Problems, each with mypy 2.4.1 and pyright 1.1.414 against pyspark-client 
> 4.3.0 and the drop-in:
> 1. pyspark.sql.connect.proto. PySpark's package star-imports the messages of 
> its generated modules (python/pyspark/sql/connect/proto/__init__.py). The 
> drop-in's installs its native messages in globals(), so type checkers saw 
> none:
>      import pyspark.sql.connect.proto as pb2
>      reveal_type(pb2.Plan(root=pb2.Relation()))
>    pyspark-client: mypy `Revealed type is 
> "pyspark.sql.connect.proto.base_pb2.Plan"`, pyright `Plan`.
>    Drop-in: mypy `Module has no attribute "Plan"` and `Any`; pyright `"Plan" 
> is not a known attribute of module "pyspark.sql.connect.proto"`. Every 
> annotation of a message in PySpark's modules (SparkConnectClient.to_table's 
> plan, MergeIntoWriter._matched_actions, ...) was Any.
> 2. pyspark.sql.connect.avro.functions and 
> pyspark.sql.connect.protobuf.functions copy their functions into globals(), 
> with no stub: `from pyspark.sql.connect.avro.functions import from_avro` is 
> an error (`Module has no attribute "from_avro"`, pyright `unknown import 
> symbol`), where pyspark-client types `from_avro(data: ColumnOrName, 
> jsonFormatSchema: str, options: Optional[Dict[str, str]] = None) -> Column`.
> 3. pyspark.ml.connect. The drop-in's modules of its native ML classes 
> (classification, clustering, evaluation, fpm, pipeline, recommendation, 
> regression, tuning, the package and the native stages of the feature module) 
> had no stub, so their classes were Any for mypy, pyright reported `"Pipeline" 
> is not exported from module "pyspark.ml.connect.pipeline"`, and `from 
> pyspark.ml.connect import VectorAssembler` was an error. At run time each is 
> a subclass of PySpark's class of its name of pyspark.ml 
> (pyspark.ml._native_classes.publish).
> 4. SQLContext. pyspark-client ships PySpark's classic 
> python/pyspark/sql/context.py, whose SQLContext has all its methods (sql, 
> table, read, ...) and whose getOrCreate() returns the SQLContext of 
> pyspark.sql.connect.context (context.py:146-200). The drop-in's had 
> getOrCreate() only: `hasattr(SQLContext, "sql")` was False (True with 
> pyspark-client), and for `ctx = SQLContext.getOrCreate()` mypy reported 
> `"SQLContext" has no attribute "sql"` of ctx.sql(...), where pyspark-client 
> types `SQLContext` and `DataFrame`.
> 5. The classic names of pyspark. PySpark's pyspark/__init__.py imports 
> SparkContext, RDD, RDDBarrier, SparkFiles, Broadcast, StatusTracker, 
> SparkJobInfo, SparkStageInfo and SparkExecutorInfo of pyspark.core when not 
> is_remote_only() (python/pyspark/__init__.py:52-63, 118-127), which type 
> checkers do not evaluate: with pyspark-client mypy takes `from pyspark import 
> SparkContext` as Any. The drop-in's module had none: mypy reported `Module 
> "pyspark" has no attribute "SparkContext"`.
> 6. The stub of pyspark.sql.connect.table_arg left out TableArg.to_plan, as 
> PySpark Connect's plumbing, after the drop-in had the method: 
> `ta.to_plan(client)` was an error (`"TableArg" has no attribute "to_plan"`), 
> typed `(session: SparkConnectClient) -> Expression` with pyspark-client.
> 7. The stub test checked only the module-level names of stubbed modules: a 
> class member that a stub missed (6) and a name of DROPPED that the drop-in 
> had passed.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to