[
https://issues.apache.org/jira/browse/SPARK-61417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon updated SPARK-61417:
---------------------------------
Fix Version/s: connect-rust-4.3.0
> Fix review findings: Spark's ML column defaults, the multiclass metric, the
> depth of a cross-session transform, empty observe, done-call errors over
> batch errors, and unencodable option strings
> -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61417
> URL: https://issues.apache.org/jira/browse/SPARK-61417
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Priority: Major
> Fix For: connect-rust-4.3.0
>
>
> Follow-ups of the s17 review of SPARK-61362/61363/61367/61385/61386, each
> checked against PySpark 4.3.0 or Spark v4.3.0:
> - The input/output column getters of StandardScaler, MaxAbsScaler,
> StringIndexer and VectorAssembler returned made-up defaults ("features",
> "scaled_features", "maxAbs_scaled", "indexed", "assembled"), which
> SPARK-61386 documented as Spark's defaults. Spark's HasInputCol has no
> default and HasOutputCol defaults to <uid>__output, the column that the
> server writes, so select(col(scaler.output_col())) failed with
> UNRESOLVED_COLUMN. input_col now returns Option<&str> and output_col the set
> value or <uid>__output. VectorAssembler::set_input_cols and
> Pipeline::set_stages take impl IntoIterator<Item = impl Into<String>>, as
> AGENTS.md asks of string lists.
> - pyspark.ml.connect's MulticlassClassificationEvaluator() evaluated the JVM
> default f1, where PySpark's constructor sets metricName "accuracy", so a
> no-argument evaluator gave another number. The evaluators' signatures now are
> PySpark's, with their defaults.
> - The drop-in's transform of a DataFrame of another session put the input
> relation inline, so its levels counted as one: levels added over it were not
> bounded, and a deep input could overflow a small stack when the relation was
> cloned. It now passes the DataFrame's placeholder, which the core resolves
> counting its levels, as the plans of pyspark.ml do.
> - DataFrame::observe_with with no expressions failed at the action with the
> server's INTERNAL_ERROR; PySpark's observe raises CANNOT_BE_EMPTY at the call.
> - When the decoding of a batch (or of the schema) failed, collect,
> collect_record_batches, the local iterator and catalog operations returned
> that error and made the done call of the progress handlers on drop, losing
> its error; PySpark's Progress.__exit__ makes it in the with of the execution,
> and its error replaces the decoding error.
> - Reader, writer, streaming and session option keys and values, and fill
> values, converted a str that UTF-8 cannot encode (a lone surrogate) lossily
> (U+FFFD), so that a write could go to another path, or raised a misleading
> TypeError; PySpark's protobuf string fields raise ValueError.
> - Docs: the empty agg of the Rust API, PipelineModel's public fields, and the
> year-month interval columns that create_dataframe takes.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]