Jungtaek Lim created SPARK-62045:
------------------------------------

             Summary: Driver-side Python workers of a streaming query started 
through StreamingPythonRunner can't see the session's artifacts
                 Key: SPARK-62045
                 URL: https://issues.apache.org/jira/browse/SPARK-62045
             Project: Spark
          Issue Type: Bug
          Components: Structured Streaming, PySpark, Connect
    Affects Versions: 4.0.5, 4.1.4, 4.2.1, 4.3.0, 4.4.0, 5.0.0
            Reporter: Jungtaek Lim


With artifact isolation on (the default in Spark Connect), the Python workers 
that {{StreamingPythonRunner}} starts on the driver get no session folder: the 
runner sets no {{SPARK_JOB_ARTIFACT_UUID}} and sends no files root 
({{StreamingPythonRunner.scala}}), and the modules it starts read the function 
without setting up Spark files ({{transform_with_state_driver_worker.py}}, 
{{foreach_batch_worker.py}}). So these workers can't import Python files or 
open files that the session added with {{addArtifacts}}:

* the pre-init worker of {{transformWithStateInPandas}} / 
{{transformWithState}}, which the stream thread starts while it plans the first 
batch of every run;
* the Spark Connect {{foreachBatch}} worker.

Reproduction with Spark Connect (local mode):

{code:python}
# A processor class defined in a file added to the session.
spark.addArtifacts("/tmp/mods/my_processor.py", pyfile=True)   # defines class 
Proc(StatefulProcessor)
import my_processor   # also on the client's sys.path
q = (spark.readStream.schema("id STRING").json(input_dir)
     .groupBy("id")
     .transformWithStateInPandas(statefulProcessor=my_processor.Proc(),
                                 outputStructType="id STRING", 
outputMode="Update", timeMode="None")
     
.writeStream.format("noop").outputMode("update").trigger(availableNow=True).start())
q.awaitTermination()
# ModuleNotFoundError: No module named 'my_processor' (when the first batch is 
planned)

# A foreachBatch function that imports a file added to the session.
spark.addArtifacts("/tmp/mods/my_module.py", pyfile=True)
def fb(df, batch_id):
    import my_module
q = spark.readStream.schema("id 
STRING").json(input_dir).writeStream.foreachBatch(fb) \
    .trigger(availableNow=True).start()
q.awaitTermination()
# ModuleNotFoundError: No module named 'my_module'
{code}

{{SparkFiles.get}} in either worker fails too, because no files root is set 
there.

A possible fix: let {{StreamingPythonRunner}} take an optional job artifact 
UUID. With one, it sets {{SPARK_JOB_ARTIFACT_UUID}} and sends the session 
folder and the function's Python includes in its init message, and the workers 
apply them with {{worker_util.setup_spark_files}} before they unpickle the 
function. Every module the runner starts, including the listener worker, must 
read the extended init message.

Found in the review of SPARK-61959 (https://github.com/apache/spark/pull/60657).




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to