Jungtaek Lim created SPARK-62045:
------------------------------------
Summary: Driver-side Python workers of a streaming query started
through StreamingPythonRunner can't see the session's artifacts
Key: SPARK-62045
URL: https://issues.apache.org/jira/browse/SPARK-62045
Project: Spark
Issue Type: Bug
Components: Structured Streaming, PySpark, Connect
Affects Versions: 4.0.5, 4.1.4, 4.2.1, 4.3.0, 4.4.0, 5.0.0
Reporter: Jungtaek Lim
With artifact isolation on (the default in Spark Connect), the Python workers
that {{StreamingPythonRunner}} starts on the driver get no session folder: the
runner sets no {{SPARK_JOB_ARTIFACT_UUID}} and sends no files root
({{StreamingPythonRunner.scala}}), and the modules it starts read the function
without setting up Spark files ({{transform_with_state_driver_worker.py}},
{{foreach_batch_worker.py}}). So these workers can't import Python files or
open files that the session added with {{addArtifacts}}:
* the pre-init worker of {{transformWithStateInPandas}} /
{{transformWithState}}, which the stream thread starts while it plans the first
batch of every run;
* the Spark Connect {{foreachBatch}} worker.
Reproduction with Spark Connect (local mode):
{code:python}
# A processor class defined in a file added to the session.
spark.addArtifacts("/tmp/mods/my_processor.py", pyfile=True) # defines class
Proc(StatefulProcessor)
import my_processor # also on the client's sys.path
q = (spark.readStream.schema("id STRING").json(input_dir)
.groupBy("id")
.transformWithStateInPandas(statefulProcessor=my_processor.Proc(),
outputStructType="id STRING",
outputMode="Update", timeMode="None")
.writeStream.format("noop").outputMode("update").trigger(availableNow=True).start())
q.awaitTermination()
# ModuleNotFoundError: No module named 'my_processor' (when the first batch is
planned)
# A foreachBatch function that imports a file added to the session.
spark.addArtifacts("/tmp/mods/my_module.py", pyfile=True)
def fb(df, batch_id):
import my_module
q = spark.readStream.schema("id
STRING").json(input_dir).writeStream.foreachBatch(fb) \
.trigger(availableNow=True).start()
q.awaitTermination()
# ModuleNotFoundError: No module named 'my_module'
{code}
{{SparkFiles.get}} in either worker fails too, because no files root is set
there.
A possible fix: let {{StreamingPythonRunner}} take an optional job artifact
UUID. With one, it sets {{SPARK_JOB_ARTIFACT_UUID}} and sends the session
folder and the function's Python includes in its init message, and the workers
apply them with {{worker_util.setup_spark_files}} before they unpickle the
function. Every module the runner starts, including the listener worker, must
read the extended init message.
Found in the review of SPARK-61959 (https://github.com/apache/spark/pull/60657).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]