Hyukjin Kwon created SPARK-61417:
------------------------------------
Summary: Fix review findings: Spark's ML column defaults, the
multiclass metric, the depth of a cross-session transform, empty observe,
done-call errors over batch errors, and unencodable option strings
Key: SPARK-61417
URL: https://issues.apache.org/jira/browse/SPARK-61417
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
Follow-ups of the s17 review of SPARK-61362/61363/61367/61385/61386, each
checked against PySpark 4.3.0 or Spark v4.3.0:
- The input/output column getters of StandardScaler, MaxAbsScaler,
StringIndexer and VectorAssembler returned made-up defaults ("features",
"scaled_features", "maxAbs_scaled", "indexed", "assembled"), which SPARK-61386
documented as Spark's defaults. Spark's HasInputCol has no default and
HasOutputCol defaults to <uid>__output, the column that the server writes, so
select(col(scaler.output_col())) failed with UNRESOLVED_COLUMN. input_col now
returns Option<&str> and output_col the set value or <uid>__output.
VectorAssembler::set_input_cols and Pipeline::set_stages take impl
IntoIterator<Item = impl Into<String>>, as AGENTS.md asks of string lists.
- pyspark.ml.connect's MulticlassClassificationEvaluator() evaluated the JVM
default f1, where PySpark's constructor sets metricName "accuracy", so a
no-argument evaluator gave another number. The evaluators' signatures now are
PySpark's, with their defaults.
- The drop-in's transform of a DataFrame of another session put the input
relation inline, so its levels counted as one: levels added over it were not
bounded, and a deep input could overflow a small stack when the relation was
cloned. It now passes the DataFrame's placeholder, which the core resolves
counting its levels, as the plans of pyspark.ml do.
- DataFrame::observe_with with no expressions failed at the action with the
server's INTERNAL_ERROR; PySpark's observe raises CANNOT_BE_EMPTY at the call.
- When the decoding of a batch (or of the schema) failed, collect,
collect_record_batches, the local iterator and catalog operations returned that
error and made the done call of the progress handlers on drop, losing its
error; PySpark's Progress.__exit__ makes it in the with of the execution, and
its error replaces the decoding error.
- Reader, writer, streaming and session option keys and values, and fill
values, converted a str that UTF-8 cannot encode (a lone surrogate) lossily
(U+FFFD), so that a write could go to another path, or raised a misleading
TypeError; PySpark's protobuf string fields raise ValueError.
- Docs: the empty agg of the Rust API, PipelineModel's public fields, and the
year-month interval columns that create_dataframe takes.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]