[
https://issues.apache.org/jira/browse/SPARK-21005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Apache Spark reassigned SPARK-21005:
------------------------------------
Assignee: (was: Apache Spark)
> VectorIndexerModel does not prepare output column field correctly
> -----------------------------------------------------------------
>
> Key: SPARK-21005
> URL: https://issues.apache.org/jira/browse/SPARK-21005
> Project: Spark
> Issue Type: Bug
> Components: MLlib
> Affects Versions: 2.1.1
> Reporter: Chen Lin
>
> From my understanding through reading the documentation, VectorIndexer
> decides which features should be categorical based on the number of distinct
> values, where features with at most maxCategories are declared categorical.
> Meanwhile, those features which exceed maxCategories are declared continuous.
> Currently, VectorIndexerModel works all right with a dataset which has empty
> schema. However, when VectorIndexerModel is transforming on a dataset with
> `ML_ATTR` metadata, it may not output the expected result. For example, a
> feature with nominal attribute which has distinct values exceeding
> maxCategorie will not be treated as a continuous feature as we expected but
> still a categorical feature. Thus, it may cause all the tree-based algorithms
> (like Decision Tree, Random Forest, GBDT, etc.) throw errors as "DecisionTree
> requires maxBins (= $maxPossibleBins) to be at least as large as the number
> of values in each categorical feature, but categorical feature $maxCategory
> has $maxCategoriesPerFeature values. Considering remove this and other
> categorical features with a large number of values, or add more training
> examples.".
> Correct me if my understanding is wrong.
> I will submit a PR soon to solve this issue.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
