[ https://issues.apache.org/jira/browse/SPARK-21005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Apache Spark reassigned SPARK-21005: ------------------------------------ Assignee: (was: Apache Spark) > VectorIndexerModel does not prepare output column field correctly > ----------------------------------------------------------------- > > Key: SPARK-21005 > URL: https://issues.apache.org/jira/browse/SPARK-21005 > Project: Spark > Issue Type: Bug > Components: MLlib > Affects Versions: 2.1.1 > Reporter: Chen Lin > > From my understanding through reading the documentation, VectorIndexer > decides which features should be categorical based on the number of distinct > values, where features with at most maxCategories are declared categorical. > Meanwhile, those features which exceed maxCategories are declared continuous. > Currently, VectorIndexerModel works all right with a dataset which has empty > schema. However, when VectorIndexerModel is transforming on a dataset with > `ML_ATTR` metadata, it may not output the expected result. For example, a > feature with nominal attribute which has distinct values exceeding > maxCategorie will not be treated as a continuous feature as we expected but > still a categorical feature. Thus, it may cause all the tree-based algorithms > (like Decision Tree, Random Forest, GBDT, etc.) throw errors as "DecisionTree > requires maxBins (= $maxPossibleBins) to be at least as large as the number > of values in each categorical feature, but categorical feature $maxCategory > has $maxCategoriesPerFeature values. Considering remove this and other > categorical features with a large number of values, or add more training > examples.". > Correct me if my understanding is wrong. > I will submit a PR soon to solve this issue. -- This message was sent by Atlassian JIRA (v6.3.15#6346) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org