Hyukjin Kwon created SPARK-61848:
------------------------------------

             Summary: Vendor the modules of pyspark-client 4.3.0 that the 
drop-in lacks: PySpark's RDD helpers, tools, server-side workers and gRPC 
client modules
                 Key: SPARK-61848
                 URL: https://issues.apache.org/jira/browse/SPARK-61848
             Project: Spark
          Issue Type: Improvement
          Components: Connect
    Affects Versions: connect-rust-4.4.0
            Reporter: Hyukjin Kwon


The pyspark-client 4.3.0 package ships 22 modules that the Python drop-in 
lacks; importing each raises ModuleNotFoundError here, while it works on 
pyspark-client:
- PySpark's RDD helpers: pyspark.join, rddsampler, resultiterable, shuffle, 
statcounter;
- its tools: pyspark.errors_doc_gen, install, threaddump, 
pyspark.sql.transpile, pyspark.sql.connect.local_server;
- the modules that the server runs in Python: pyspark.worker, pyspark.daemon, 
pyspark.worker_message, pyspark.sql.streaming.python_streaming_source_runner, 
pyspark.sql.connect.streaming.worker.foreach_batch_worker and listener_worker, 
pyspark.sql.streaming.proto.StateMessage_pb2;
- the modules of PySpark's gRPC client: pyspark.sql.connect.client.retries, 
artifact, reattach, and pyspark.sql.connect.proto.base_pb2_grpc, which import 
grpcio.

Vendor them verbatim from Apache Spark v4.3.0, under the drift gate, so that 
they import as in pyspark-client (the gRPC ones where grpcio is installed).




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to