Hyukjin Kwon created SPARK-61876:
------------------------------------
Summary: Make the catalog and ML golden tests compare with goldens
of PySpark 4.3.0, and send PySpark's requests from the Rust API's
create_external_table, variant_set and tuple_union functions
Key: SPARK-61876
URL: https://issues.apache.org/jira/browse/SPARK-61876
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.4.0
Reporter: Hyukjin Kwon
As readwriter_golden.rs did before SPARK-61853, three golden tests claim parity
with PySpark but read no golden: catalog_golden.rs refers to
tests/golden/catalog.jsonl, ml_golden.rs to ml.jsonl and ml_extra_golden.rs to
ml_extra.jsonl, none of which exists, so they pass without comparing anything.
Their capture scripts, and capture_golden.py and capture_window_golden.py, read
FieldDescriptor.label, which protobuf 7 removed, and capture_catalog_golden.py
captured no case. functions_golden.rs has no case for 7 of its goldens.
With real goldens, the Rust API differs from PySpark 4.3.0 in:
- Catalog::create_external_table sends a CreateExternalTable, where PySpark's
createExternalTable calls createTable, so it sends a CreateTable whose
description option is the table's description;
- variant_set and try_variant_set of three arguments send no create_if_missing,
where PySpark sends lit(True);
- tuple_union_agg_double, tuple_union_agg_integer, tuple_union_double,
tuple_union_integer, tuple_union_theta_double and tuple_union_theta_integer
send no lgNomEntries and mode, where PySpark sends lit(12) and lit("sum").
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]