[
https://issues.apache.org/jira/browse/MADLIB-1318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16887459#comment-16887459
]
Nikhil commented on MADLIB-1318:
--------------------------------
w/ [~okislal]
We were able to reproduce this on our local mac as well as a multi host centos
cluster with the same query/arguments. The error is caused by the internal dt
state increasing over 1 GB. We printed the size of the state for each depth
```
madlib=#
select madlib.tree_train(
'training_data', -- training data table
'out1', -- model output table
'id', -- id column
'response', -- dependent var
'a,b,c,d,e', -- list of features
NULL::text, -- features to exclude
'gini', -- split criteria
NULL::text, -- grouping
NULL::text, -- weights
23, -- max depth of tree
3, -- min split
1, -- num bucket
10, -- num splits
'cp=0.001', -- pruning parameters
);
INFO: tree state for depth 0 in MB 0.000698089599609
INFO: tree state for depth 1 in MB 0.00148963928223
INFO: tree state for depth 2 in MB 0.00306415557861
INFO: tree state for depth 3 in MB 0.00616645812988
INFO: tree state for depth 4 in MB 0.0123672485352
INFO: tree state for depth 5 in MB 0.0247278213501
INFO: tree state for depth 6 in MB 0.0493907928467
INFO: tree state for depth 7 in MB 0.099142074585
INFO: tree state for depth 8 in MB 0.199924468994
INFO: tree state for depth 9 in MB 0.405049324036
INFO: tree state for depth 10 in MB 0.823385238647
INFO: tree state for depth 11 in MB 1.67316532135
INFO: tree state for depth 12 in MB 3.39407730103
INFO: tree state for depth 13 in MB 6.86264705658
INFO: tree state for depth 14 in MB 13.8314762115
INFO: tree state for depth 15 in MB 27.8028259277
INFO: tree state for depth 16 in MB 55.7793083191
INFO: tree state for depth 17 in MB 111.762926102
INFO: tree state for depth 18 in MB 223.752031326
INFO: tree state for depth 19 in MB 447.745757103
**INFO: tree state for depth 20 in MB 895.742589951**
ERROR: plpy.Error: unrecognized error in PLy_spi_execute_fetch_result
(plpython.c:5038)
CONTEXT: Traceback (most recent call last):
PL/Python function "tree_train", line 25, in <module>
null_handling_params, verbose_mode)
PL/Python function "tree_train", line 529, in tree_train
PL/Python function "tree_train", line 296, in _get_tree_states
PL/Python function "tree_train", line 2293, in _tree_train_using_bins
PL/Python function "tree_train", line 1226, in _one_step
PL/Python function "tree_train"
```
Since this is a postgres limitation, we can't do much without overhauling the
dt code.
> DT error when tree depth increases past a certain limit
> -------------------------------------------------------
>
> Key: MADLIB-1318
> URL: https://issues.apache.org/jira/browse/MADLIB-1318
> Project: Apache MADlib
> Issue Type: Bug
> Components: Module: Decision Tree
> Reporter: Frank McQuillan
> Priority: Major
> Fix For: v1.17
>
>
> drop table if exists out1;
> drop table if exists out1_summary;
> select madlib.tree_train(
> 'training_data', -- training data table
> 'out1', -- model output table
> 'id', -- id column
> 'response', -- dependent var
> '...', -- list of features
> NULL::text, -- features to exclude
> 'gini', -- split criteria
> NULL::text, -- grouping
> NULL::text, -- weights
> 20, -- max depth of tree
> 3, -- min split
> 1, -- num bucket
> 100, -- num splits
> 'cp=0.000001', -- pruning parameters
> 'max_surrogates=5' -- by default, this is 0, all rows containing any NULL
> predictor values will not be used in modeling or prediction
> );
> {code}
> ERROR: plpy.Error: unrecognized error in PLy_spi_execute_fetch_result
> (plpython.c:5038)
> CONTEXT: Traceback (most recent call last):
> PL/Python function "tree_train", line 25, in <module>
> null_handling_params, verbose_mode)
> PL/Python function "tree_train", line 529, in tree_train
> PL/Python function "tree_train", line 296, in _get_tree_states
> PL/Python function "tree_train", line 2293, in _tree_train_using_bins
> PL/Python function "tree_train", line 1226, in _one_step
> PL/Python function "tree_train"
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
