Hyukjin Kwon created SPARK-62064:
------------------------------------

             Summary: Send the relations of DataFrames that the extensions of 
expressions hold, and with the changes a plan makes to them, and resolve the 
messages of extensions in a time of their length, as PySpark sends its plans
                 Key: SPARK-62064
                 URL: https://issues.apache.org/jira/browse/SPARK-62064
             Project: Spark
          Issue Type: Bug
          Components: Connect
    Affects Versions: 5.0.0
            Reporter: Hyukjin Kwon


Problem
df._plan.plan(session) gives a placeholder that holds the relation of the 
DataFrame, which the core
replaces with that relation where a plan built in Python holds it 
(plan::resolve_relation,
resolve_command). Three gaps were left:

1. The extensions of expressions. The core did not look into the expressions of 
a plan built in
   Python, so a placeholder in the message of a plugin's expression 
(Expression.extension) was sent
   as a placeholder, where PySpark's plan sends the relation.
2. A placeholder that a plan changes. PySpark's relation of plan(session) is a 
message that a plan
   or a plugin may change, such as relation.common.plan_id = ...:
   - as a relation of a plan, the core replaced the placeholder with its 
relation and dropped the
     change, so a column of the changed plan ID failed with 
CANNOT_RESOLVE_DATAFRAME_COLUMN;
   - in the message of an extension, the core found a placeholder by its exact 
encoding, which the
     change alters, and sent it as a placeholder.
3. The time of the messages of extensions. The core read the message of an 
extension of a relation
   or a command level by level, and at each level searched all of its bytes for 
the type URL of a
   placeholder and copied them again. A crafted message that nests many 
messages took a time of its
   length times its depth (#785): in a release build, 0.29 s for 8,000 levels 
and 20.9 s for
   64,000; in a debug build, 41 to 69 s for 30,000 levels. PySpark sends the 
bytes as they are.

Repro (master 80046f2f, against a Spark 4.3.0 server)
  import pyspark.sql.connect.proto as pb2
  client = spark.client
  plan = pb2.Plan()
  plan.root.project.input.CopyFrom(spark.range(3)._plan.plan(client))
  plan.root.project.input.common.plan_id = 424242
  column = plan.root.project.expressions.add().unresolved_attribute
  column.unparsed_identifier = "id"; column.plan_id = 424242
  client.to_table(plan, {})[0].to_pydict()

Actual (drop-in)
  AnalysisException: [CANNOT_RESOLVE_DATAFRAME_COLUMN] Cannot resolve dataframe 
column "id".

Expected (pyspark-client 4.3.0)
  {'id': [0, 1, 2]}

For an expression extension, ExamplePluginRelation(input=df._plan.plan(client)) 
packed into a
project's expression is received with the input as the placeholder, where 
PySpark's carries the
relation. (Checked with the recording mock server; a server has no handler of 
such an expression.)

Fix
- Of each relation built in Python of a kind that holds expressions (a 
projection, a filter, a
  join, an aggregation, ...; an exhaustive match of the relation types), and of 
each command built
  in Python but an extension or an ML command, which are resolved as before, 
the fields other than
  the inputs are encoded, and read as messages for a placeholder only when that 
encoding holds the
  type URL of one (plan::resolve_own_fields). The relations of the 
placeholders, which the core
  builds, are not encoded, and nor are the kinds that hold no expressions: 
reads, limits, set
  operations, local data, ranges, catalog operations, ...
- A placeholder whose common is set, as a relation and in the message of an 
extension, is
  replaced by its relation with that common merged, as protobuf's MergeFrom 
does.
- plan::resolve_encoded finds the offsets of the type URL once, reads a field 
only when its value
  holds one, and writes the resolved encoding once, of the lengths computed 
first.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to