[
https://issues.apache.org/jira/browse/SPARK-8418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14961739#comment-14961739
]
Joseph K. Bradley commented on SPARK-8418:
------------------------------------------
{quote}I vote for strategy 2 of Nick Buroojy proposed. But I think we don't
need to reimplement all transformers to support a multi-value implementation
because of some feature transformers not needed.{quote}
* This sounds like a good way to start. I'd prefer just doing strategy 2 (not
1) since it's a bit deceptive to provide the multi-value API if it is not
optimized underneath. +1 for only adding support where needed.
Starting with StringIndexer and OneHotEncoder sounds good to me.
{quote}I don't think RFormula is the best way to resolve this issue because it
still use the pipeline chained transformers one by one to encode multiple
columns which is low performance.{quote}
* That's currently true, but it could be optimized. Ideally, it would call
these multi-value implementations when available---and would convert to a
single Vector as soon as possible in the transformations to be efficient.
* I guess RFormula is really a separate discussion, so I won't discuss it here
more.
@yanboliang I'm fine if we skip a design doc for this task. It seems pretty
straightforward given the discussion above.
> Add single- and multi-value support to ML Transformers
> ------------------------------------------------------
>
> Key: SPARK-8418
> URL: https://issues.apache.org/jira/browse/SPARK-8418
> Project: Spark
> Issue Type: Sub-task
> Components: ML
> Reporter: Joseph K. Bradley
>
> It would be convenient if all feature transformers supported transforming
> columns of single values and multiple values, specifically:
> * one column with one value (e.g., type {{Double}})
> * one column with multiple values (e.g., {{Array[Double]}} or {{Vector}})
> We could go as far as supporting multiple columns, but that may not be
> necessary since VectorAssembler could be used to handle that.
> Estimators under {{ml.feature}} should also support this.
> This will likely require a short design doc to describe:
> * how input and output columns will be specified
> * schema validation
> * code sharing to reduce duplication
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
