Hyukjin Kwon created SPARK-61417:
------------------------------------

             Summary: Fix review findings: Spark's ML column defaults, the 
multiclass metric, the depth of a cross-session transform, empty observe, 
done-call errors over batch errors, and unencodable option strings
                 Key: SPARK-61417
                 URL: https://issues.apache.org/jira/browse/SPARK-61417
             Project: Spark
          Issue Type: Bug
          Components: Connect
    Affects Versions: connect-rust-4.3.0
            Reporter: Hyukjin Kwon


Follow-ups of the s17 review of SPARK-61362/61363/61367/61385/61386, each 
checked against PySpark 4.3.0 or Spark v4.3.0:
- The input/output column getters of StandardScaler, MaxAbsScaler, 
StringIndexer and VectorAssembler returned made-up defaults ("features", 
"scaled_features", "maxAbs_scaled", "indexed", "assembled"), which SPARK-61386 
documented as Spark's defaults. Spark's HasInputCol has no default and 
HasOutputCol defaults to <uid>__output, the column that the server writes, so 
select(col(scaler.output_col())) failed with UNRESOLVED_COLUMN. input_col now 
returns Option<&str> and output_col the set value or <uid>__output. 
VectorAssembler::set_input_cols and Pipeline::set_stages take impl 
IntoIterator<Item = impl Into<String>>, as AGENTS.md asks of string lists.
- pyspark.ml.connect's MulticlassClassificationEvaluator() evaluated the JVM 
default f1, where PySpark's constructor sets metricName "accuracy", so a 
no-argument evaluator gave another number. The evaluators' signatures now are 
PySpark's, with their defaults.
- The drop-in's transform of a DataFrame of another session put the input 
relation inline, so its levels counted as one: levels added over it were not 
bounded, and a deep input could overflow a small stack when the relation was 
cloned. It now passes the DataFrame's placeholder, which the core resolves 
counting its levels, as the plans of pyspark.ml do.
- DataFrame::observe_with with no expressions failed at the action with the 
server's INTERNAL_ERROR; PySpark's observe raises CANNOT_BE_EMPTY at the call.
- When the decoding of a batch (or of the schema) failed, collect, 
collect_record_batches, the local iterator and catalog operations returned that 
error and made the done call of the progress handlers on drop, losing its 
error; PySpark's Progress.__exit__ makes it in the with of the execution, and 
its error replaces the decoding error.
- Reader, writer, streaming and session option keys and values, and fill 
values, converted a str that UTF-8 cannot encode (a lone surrogate) lossily 
(U+FFFD), so that a write could go to another path, or raised a misleading 
TypeError; PySpark's protobuf string fields raise ValueError.
- Docs: the empty agg of the Rust API, PipelineModel's public fields, and the 
year-month interval columns that create_dataframe takes.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to