Hyukjin Kwon created SPARK-62021:
------------------------------------

             Summary: Make the relation of df._plan.plan(client) hold the 
relation of the DataFrame, built at the call, so that it is sent after the 
DataFrame is gone, as PySpark's
                 Key: SPARK-62021
                 URL: https://issues.apache.org/jira/browse/SPARK-62021
             Project: Spark
          Issue Type: Bug
          Components: Connect
    Affects Versions: 5.0.0
            Reporter: Hyukjin Kwon


Problem
df._plan.plan(client) gives a placeholder of the relation of the DataFrame: an 
extension whose
value is a marker. The core replaced it, when a plan, a command or an analysis 
that holds it was
sent, with the relation of the DataFrame of the marker, which it looked up in a 
registry of weak
references (plan::placeholder_target, crates/spark-connect/src/plan.rs). So a 
placeholder whose
DataFrame was gone failed every request with "a placeholder of a DataFrame that 
no longer
exists":
- the inline client.to_table(pb2.Plan(root=spark.sql(q)._plan.plan(client)), 
{}), whose DataFrame
  is freed before the call;
- a relation that the program keeps after its DataFrame is freed;
- a plan parsed again of its bytes.
PySpark's LogicalPlan.plan(session) builds the relation at the call 
(python/pyspark/sql/connect/
plan.py), so the message stands on its own, whatever happens to the DataFrame.

As the drop-in built the relation at the send and not at the call, the call did 
not do what
PySpark's build does either:
- the PicklingError of a UDF that does not pickle was raised by the request, 
not by plan(client);
- the warning of a window of no partitions came from the request;
- a UDF whose state changed after plan(client) was sent with the state of the 
request.

Repro (master 50594b2e, against a Spark 4.3.0 server)
  import gc, threading
  import pyspark.sql.connect.proto as pb2
  from pyspark.sql import functions as F
  client = spark.client
  client.to_table(pb2.Plan(root=spark.sql("SELECT 1 AS a")._plan.plan(client)), 
{})[0].to_pydict()
  rel = spark.range(3).filter("id > 0")._plan.plan(client); gc.collect()
  client.to_table(pb2.Plan(root=rel), {})[0].to_pydict()
  client.to_table(pb2.Plan.FromString(pb2.Plan(root=rel).SerializeToString()), 
{})[0].to_pydict()
  lock = threading.Lock()
  spark.range(1).select(F.udf(lambda x: lock is not None, 
"boolean")("id"))._plan.plan(client)
  state = ["at the call"]
  rel = spark.range(1).select(F.udf(lambda x: state[0], 
"string")("id").alias("s"))._plan.plan(client)
  state[0] = "after the call"
  client.to_table(pb2.Plan(root=rel), {})[0].to_pydict()

Actual (drop-in at master 50594b2e)
  - each of the first three raises PySparkValueError: a placeholder of a 
DataFrame that no longer
    exists;
  - plan(client) of the unpicklable UDF returns, and the request raises the 
PicklingError;
  - {'s': ['after the call']}.

Expected (pyspark-client 4.3.0)
  - {'a': [1]}, {'id': [1, 2]} and {'id': [1, 2]};
  - plan(client) raises PicklingError: Could not serialize object: TypeError: 
cannot pickle
    '_thread.lock' object, and no request is sent;
  - {'s': ['at the call']}.

Fix
- The placeholder of df._plan.plan(client) holds the relation, built at the 
call, with its
  pickles, warnings and errors, and encoded in the value of its extension: the 
marker, the levels
  and the encoding of the relation. It is one level, so Python's protobuf never 
parses the
  relation of a deep DataFrame.
- The core replaces the placeholder with the relation that it holds. The 
registry by marker keeps
  weak entries only, for the Observations of the DataFrame of a placeholder.
- A placeholder of a marker alone stays for the plans that the core builds over 
a DataFrame and
  resolves while the DataFrame is alive, such as the binding's ML transforms of 
a DataFrame of
  another session.
- The docs say this and state the reasons of what stays different:
  - a placeholder in an expression's extension, or one that a plugin changes;
  - the plan ID of DataFrame(plan, session);
  - the retry policies of can_retry and to_state;
  - get_tags.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to