[
https://issues.apache.org/jira/browse/MADLIB-1173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16267857#comment-16267857
]
Frank McQuillan commented on MADLIB-1173:
-----------------------------------------
RF does not fail for predict on large feature vectors because the input check
is not done in the same way as DT. We can leave RF as is for now - it just
means that the error message will not be as clear as it could be.
> DT predict fails with large feature vector arrays
> -------------------------------------------------
>
> Key: MADLIB-1173
> URL: https://issues.apache.org/jira/browse/MADLIB-1173
> Project: Apache MADlib
> Issue Type: Bug
> Components: Module: Decision Tree
> Reporter: Rahul Iyer
> Fix For: v1.13
>
>
> Since decision trees can now take in arrays, it’s possible to train a model
> with more than 1600 (table column limit) features. For example, we can
> assemble a feature_array field with 2000 elements, and pass that into
> tree_train as the independent variable. The model trains successfully and
> produces the expected model and model_summary tables.
>
> But then when trying to use that model table to make predictions using
> tree_predict, it gets the following error:
>
> {code}
> ERROR: plpy.Error: Decision tree error: Missing columns in predict data
> table (tbl_test_1_data_final) that were used during training (plpython.c:4656)
> CONTEXT: Traceback (most recent call last):
> PL/Python function "tree_predict", line 19, in <module>
> return decision_tree.tree_predict(**globals())
> PL/Python function "tree_predict", line 1752, in tree_predict
> PL/Python function "tree_predict", line 75, in _assert
> PL/Python function "tree_predict"
> {code}
> We think what’s happening (this is only a guess) is that the tree_train
> function correctly makes use of the passed features, even if they number in
> excess of 1600, but tree_predict is still for some reason limiting the
> feature count to 1600. So when it tries to run the predictions, it sees that
> there are 2000 expected features according to the model table, but
> tree_predict has limited the new_data_table to only 1600 features. It then
> sees a discrepancy between the 2000 expected by the model and the 1600 that
> it has perceived on the new_data_table, and gets the above error.
>
> We’ve observed the error only happens when over 1600 features are used. If,
> for example, we trained a model with 900 features, it would able to predict
> successfully. If we train a model with 2000 features, it gets the above
> error.
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
