Hyukjin Kwon created SPARK-61367:
------------------------------------

             Summary: Match PySpark and the server in the Rust API's pivot 
literals, year-month intervals, range repartitions, sample_by strata, DDL, 
Arrow batch checks, release_session, listener panics and handler threads
                 Key: SPARK-61367
                 URL: https://issues.apache.org/jira/browse/SPARK-61367
             Project: Spark
          Issue Type: Bug
          Components: Connect
    Affects Versions: connect-rust-4.3.0
            Reporter: Hyukjin Kwon


Low-priority findings of the s16 review of the Rust API (apache-spark-connect), 
each checked against official PySpark 4.3.0, the server or the Scala client:
- pivot sent a Float value as a double literal, so its column was named 
"1.100000023841858" where the pivot without values (and the Scala client) names 
it "1.1"; Byte and Short were sent as ints; and its NOT_EXPECTED_TYPE message 
named only bool, float, int and str.
- Value had no year-month interval: collect gave Value::String("1-0"), so a 
pivot on collected values named the column "1-0", not "INTERVAL '1-0' YEAR TO 
MONTH".
- repartition_by_range with no columns was split round-robin by the server; 
PySpark raises CANNOT_BE_EMPTY (and VALUE_NOT_POSITIVE for a count that is not 
positive). sample_by turned a stratum that is not a literal into an unset one; 
PySpark raises DISALLOWED_TYPE_FOR_CONTAINER.
- DataType::from_ddl rejected whitespace before "<" (array <int>, STRUCT 
<a:int>), which the server accepts.
- PySpark checks the start_offset and the row count of each Arrow batch of a 
result; ours did not, and grew the joined buffer of chunked batches without 
reserving it.
- SparkSession::release_session after stop() or close() still sent 
ReleaseSession; PySpark raises NO_ACTIVE_SESSION.
- A native streaming listener's panic on an asynchronous event failed the whole 
subscription and cleared every listener; PySpark's post_to_all isolates each 
listener.
- In a current_thread Tokio runtime, the progress handlers' update calls ran on 
a helper thread; PySpark calls them on the thread that iterates the responses.
- With several invalid UTF-8 values in one struct or map column, ours reported 
the first row by row; PySpark reports field by field, keys before values.
- A Raw relation (a plan of pyspark.ml) counted as one level for the depth 
bound whatever its nesting, and cloning one at the bound overflowed a small 
stack; LogicalPlan::to_proto has no bounded variant.
- Docs: the E0509 of moving fields out of a LogicalPlan (it implements Drop), 
and that is_local() is a client-side guess.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to