[
https://issues.apache.org/jira/browse/SPARK-11106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15424689#comment-15424689
]
Jeff Levy edited comment on SPARK-11106 at 8/17/16 3:26 PM:
------------------------------------------------------------
The objections to implementing RFormula in pySpark that Max raises strike me as
trivial - not because the issues with assumptions aren't real, but because
users of other data analysis tools _are already entirely familiar with working
this way._ As was suggested, leave the option of doing each step explicitly in
place for users who need it, but leaving it as the _only_ way to do what
generally takes one or two clean lines in every other comparable environment
(Python Statsmodels, R, SparkR, Stata, SAS) strikes me as a huge barrier to
getting people to use pySpark.
I was quite disappointed to see this wasn't already in the Spark 2.0 release,
especially with the focus of ML on dataframes.
Edit: if this isn't a change that can be quickly or easily done, one temporary
work-around might be to add references to the RFormula transformer to the
places where it can be used in the pyspark.ml documentation, e.g. with the
LinearRegression and GLM help topics.
was (Author: jlevy):
The objections to implementing RFormula in pySpark that Max raises strike me as
trivial - not because the issues with assumptions aren't real, but because
users of other data analysis tools _are already entirely familiar with working
this way._ As was suggested, leave the option of doing each step explicitly in
place for users who need it, but leaving it as the _only_ way to do what
generally takes one or two clean lines in every other comparable environment
(Python Statsmodels, R, SparkR, Stata, SAS) strikes me as a huge barrier to
getting people to use pySpark.
I was quite disappointed to see this wasn't already in the Spark 2.0 release,
especially with the focus of ML on dataframes.
> Should ML Models contains single models or Pipelines?
> -----------------------------------------------------
>
> Key: SPARK-11106
> URL: https://issues.apache.org/jira/browse/SPARK-11106
> Project: Spark
> Issue Type: Sub-task
> Components: ML
> Reporter: Joseph K. Bradley
> Priority: Critical
>
> This JIRA is for discussing whether an ML Estimators should do feature
> processing.
> h2. Issue
> Currently, almost all ML Estimators require strict input types. E.g.,
> DecisionTreeClassifier requires that the label column be Double type and have
> metadata indicating the number of classes.
> This requires users to know how to preprocess data.
> h2. Ideal workflow
> A user should be able to pass any reasonable data to a Transformer or
> Estimator and have it "do the right thing."
> E.g.:
> * If DecisionTreeClassifier is given a String column for labels, it should
> know to index the Strings.
> * See [SPARK-10513] for a similar issue with OneHotEncoder.
> h2. Possible solutions
> There are a few solutions I have thought of. Please comment with feedback or
> alternative ideas!
> h3. Leave as is
> Pro: The current setup is good in that it forces the user to be very aware of
> what they are doing. Feature transformations will not happen silently.
> Con: The user has to write boilerplate code for transformations. The API is
> not what some users would expect; e.g., coming from R, a user might expect
> some automatic transformations.
> h3. All Transformers can contain PipelineModels
> We could allow all Transformers and Models to contain arbitrary
> PipelineModels. E.g., if a DecisionTreeClassifier were given a String label
> column, it might return a Model which contains a simple fitted PipelineModel
> containing StringIndexer + DecisionTreeClassificationModel.
> The API could present this to the user, or it could be hidden from the user.
> Ideally, it would be hidden from the beginner user, but accessible for
> experts.
> The main problem is that we might have to break APIs. E.g., OneHotEncoder
> may need to do indexing if given a String input column. This means it should
> no longer be a Transformer; it should be an Estimator.
> h3. All Estimators should use RFormula
> The best option I have thought of is to make RFormula be the primary method
> for automatic feature transformation. We could start adding an RFormula
> Param to all Estimators, and it could handle most of these feature
> transformation issues.
> We could maintain old APIs:
> * If a user sets the input column names, then those can be used in the
> traditional (no automatic transformation) way.
> * If a user sets the RFormula Param, then it can be used instead. (This
> should probably take precedence over the old API.)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
