[ 
https://issues.apache.org/jira/browse/SPARK-61848?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated SPARK-61848:
-----------------------------------
    Labels: pull-request-available  (was: )

> Vendor the modules of pyspark-client 4.3.0 that the drop-in lacks: PySpark's 
> RDD helpers, tools, server-side workers and gRPC client modules
> --------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-61848
>                 URL: https://issues.apache.org/jira/browse/SPARK-61848
>             Project: Spark
>          Issue Type: Improvement
>          Components: Connect
>    Affects Versions: connect-rust-4.4.0
>            Reporter: Hyukjin Kwon
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: connect-rust-4.4.0
>
>
> The pyspark-client 4.3.0 package ships 22 modules that the Python drop-in 
> lacks; importing each raises ModuleNotFoundError here, while it works on 
> pyspark-client:
> - PySpark's RDD helpers: pyspark.join, rddsampler, resultiterable, shuffle, 
> statcounter;
> - its tools: pyspark.errors_doc_gen, install, threaddump, 
> pyspark.sql.transpile, pyspark.sql.connect.local_server;
> - the modules that the server runs in Python: pyspark.worker, pyspark.daemon, 
> pyspark.worker_message, pyspark.sql.streaming.python_streaming_source_runner, 
> pyspark.sql.connect.streaming.worker.foreach_batch_worker and 
> listener_worker, pyspark.sql.streaming.proto.StateMessage_pb2;
> - the modules of PySpark's gRPC client: pyspark.sql.connect.client.retries, 
> artifact, reattach, and pyspark.sql.connect.proto.base_pb2_grpc, which import 
> grpcio.
> Vendor them verbatim from Apache Spark v4.3.0, under the drift gate, so that 
> they import as in pyspark-client (the gRPC ones where grpcio is installed).



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to