[
https://issues.apache.org/jira/browse/MADLIB-1301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Frank McQuillan updated MADLIB-1301:
------------------------------------
Description:
When correlation and covariance are run with large number of groups (100's),
can run out of memory. Increasing statement_mem helps, but this JIRA is to
investigate and improve memory usage with large numbers of groups.
Sample findings on correlation for 300K input data set:
#groups | statement mem 186M | statement mem 200M | statement mem 500M
statement mem 1000M
6 | Success | Success | Success | -
127 | Success | Success | - | -
930 | Fail | Fail | Success | -
1213 | Fail | Fail | Success | -
4852 | Fail | Fail | Fail | Fail
was:When correlation and covariance are run with large number of groups
(100's), can run out of memory. This JIRA is to investigate and improve memory
usage with large numbers of groups.
> Improve correlation and covariance memory usage with large number of groups
> ---------------------------------------------------------------------------
>
> Key: MADLIB-1301
> URL: https://issues.apache.org/jira/browse/MADLIB-1301
> Project: Apache MADlib
> Issue Type: Improvement
> Components: Module: Descriptive Statistics
> Reporter: Frank McQuillan
> Priority: Major
> Fix For: v2.0
>
>
> When correlation and covariance are run with large number of groups (100's),
> can run out of memory. Increasing statement_mem helps, but this JIRA is to
> investigate and improve memory usage with large numbers of groups.
> Sample findings on correlation for 300K input data set:
> #groups | statement mem 186M | statement mem 200M | statement mem 500M
> statement mem 1000M
> 6 | Success | Success | Success | -
> 127 | Success | Success | - | -
> 930 | Fail | Fail | Success | -
> 1213 | Fail | Fail | Success | -
> 4852 | Fail | Fail | Fail | Fail
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
