Hyukjin Kwon created SPARK-62021:
------------------------------------
Summary: Make the relation of df._plan.plan(client) hold the
relation of the DataFrame, built at the call, so that it is sent after the
DataFrame is gone, as PySpark's
Key: SPARK-62021
URL: https://issues.apache.org/jira/browse/SPARK-62021
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: 5.0.0
Reporter: Hyukjin Kwon
Problem
df._plan.plan(client) gives a placeholder of the relation of the DataFrame: an
extension whose
value is a marker. The core replaced it, when a plan, a command or an analysis
that holds it was
sent, with the relation of the DataFrame of the marker, which it looked up in a
registry of weak
references (plan::placeholder_target, crates/spark-connect/src/plan.rs). So a
placeholder whose
DataFrame was gone failed every request with "a placeholder of a DataFrame that
no longer
exists":
- the inline client.to_table(pb2.Plan(root=spark.sql(q)._plan.plan(client)),
{}), whose DataFrame
is freed before the call;
- a relation that the program keeps after its DataFrame is freed;
- a plan parsed again of its bytes.
PySpark's LogicalPlan.plan(session) builds the relation at the call
(python/pyspark/sql/connect/
plan.py), so the message stands on its own, whatever happens to the DataFrame.
As the drop-in built the relation at the send and not at the call, the call did
not do what
PySpark's build does either:
- the PicklingError of a UDF that does not pickle was raised by the request,
not by plan(client);
- the warning of a window of no partitions came from the request;
- a UDF whose state changed after plan(client) was sent with the state of the
request.
Repro (master 50594b2e, against a Spark 4.3.0 server)
import gc, threading
import pyspark.sql.connect.proto as pb2
from pyspark.sql import functions as F
client = spark.client
client.to_table(pb2.Plan(root=spark.sql("SELECT 1 AS a")._plan.plan(client)),
{})[0].to_pydict()
rel = spark.range(3).filter("id > 0")._plan.plan(client); gc.collect()
client.to_table(pb2.Plan(root=rel), {})[0].to_pydict()
client.to_table(pb2.Plan.FromString(pb2.Plan(root=rel).SerializeToString()),
{})[0].to_pydict()
lock = threading.Lock()
spark.range(1).select(F.udf(lambda x: lock is not None,
"boolean")("id"))._plan.plan(client)
state = ["at the call"]
rel = spark.range(1).select(F.udf(lambda x: state[0],
"string")("id").alias("s"))._plan.plan(client)
state[0] = "after the call"
client.to_table(pb2.Plan(root=rel), {})[0].to_pydict()
Actual (drop-in at master 50594b2e)
- each of the first three raises PySparkValueError: a placeholder of a
DataFrame that no longer
exists;
- plan(client) of the unpicklable UDF returns, and the request raises the
PicklingError;
- {'s': ['after the call']}.
Expected (pyspark-client 4.3.0)
- {'a': [1]}, {'id': [1, 2]} and {'id': [1, 2]};
- plan(client) raises PicklingError: Could not serialize object: TypeError:
cannot pickle
'_thread.lock' object, and no request is sent;
- {'s': ['at the call']}.
Fix
- The placeholder of df._plan.plan(client) holds the relation, built at the
call, with its
pickles, warnings and errors, and encoded in the value of its extension: the
marker, the levels
and the encoding of the relation. It is one level, so Python's protobuf never
parses the
relation of a deep DataFrame.
- The core replaces the placeholder with the relation that it holds. The
registry by marker keeps
weak entries only, for the Observations of the DataFrame of a placeholder.
- A placeholder of a marker alone stays for the plans that the core builds over
a DataFrame and
resolves while the DataFrame is alive, such as the binding's ML transforms of
a DataFrame of
another session.
- The docs say this and state the reasons of what stays different:
- a placeholder in an expression's extension, or one that a plugin changes;
- the plan ID of DataFrame(plan, session);
- the retry policies of can_retry and to_state;
- get_tags.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]