[
https://issues.apache.org/jira/browse/SPARK-20099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16086422#comment-16086422
]
Joseph K. Bradley commented on SPARK-20099:
-------------------------------------------
[~holdenk] [~yanboliang] [~yuhaoyan] [~mlnick] CCing a few people since
[~WeichenXu123] is interested in working on this. Do you think it's reasonable
to add PipelineStage to Python in order to add transformSchema?
Pro: early schema failure detection in Python
Con: duplication of schema checking logic in Python
* I don't see a good way to do schema checking in Python for Pipelines without
this duplication. The only way would be to convert Pipelines to Scala
equivalents before executing them; i.e., the Pipeline implementation would be
in Scala only. The problem is that we need Pipelines implemented in Python as
well in order to support Python-only implementations of Transformers and
Estimators (for custom use cases).
A reasonable way to do this in a series of PRs would be to:
* Add PipelineStage abstraction, with abstract transformSchema method
* For each Transformer/Estimator/Model in Python, change it to inherit from
PipelineStage
* Finally, change Pipeline and PipelineModel to call transformSchema on their
sequences of stages
> Add transformSchema to pyspark.ml
> ---------------------------------
>
> Key: SPARK-20099
> URL: https://issues.apache.org/jira/browse/SPARK-20099
> Project: Spark
> Issue Type: Improvement
> Components: ML, PySpark
> Affects Versions: 2.1.0
> Reporter: Joseph K. Bradley
>
> Python's ML API currently lacks the PipelineStage abstraction. This
> abstraction's main purpose is to provide transformSchema() for checking for
> early failures in a Pipeline.
> As mentioned in https://github.com/apache/spark/pull/17218 it would also be
> useful in Python for checking Params in Python wrapper for Scala
> implementations; in these, transformSchema would involve passing Params in
> Python to Scala, which would then be able to validate the Param values. This
> could prevent late failures from bad Param settings in Pipeline execution,
> while still allowing us to check Param values on only the Scala side.
> This issue is for adding transformSchema() to pyspark.ml. If it's
> reasonable, we could create a PipelineStage abstraction. But it'd probably
> be fine to add transformSchema() directly to Transformer and Estimator,
> rather than creating PipelineStage.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
