[
https://issues.apache.org/jira/browse/MADLIB-1301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Frank McQuillan updated MADLIB-1301:
------------------------------------
Description:
When correlation and covariance are run with large number of groups (100's),
can run out of memory. Increasing statement_mem helps, but this JIRA is to
investigate and improve memory usage with large numbers of groups.
Sample findings on correlation for 300K input data set:
|| #groups || statement mem 186M || statement mem 200M || statement mem 500M ||
statement mem 1000M ||
6 | Success | Success | Success | - |
| 127 | Success | Success | - | - |
| 930 | Fail | Fail | Success | - |
| 1213 | Fail | Fail | Success | - |
| 4852 | Fail | Fail | Fail | Fail |
||Heading 1||Heading 2||
|Col A1|Col A2|
was:
When correlation and covariance are run with large number of groups (100's),
can run out of memory. Increasing statement_mem helps, but this JIRA is to
investigate and improve memory usage with large numbers of groups.
Sample findings on correlation for 300K input data set:
#groups | statement mem 186M | statement mem 200M | statement mem 500M
statement mem 1000M
6 | Success | Success | Success | -
127 | Success | Success | - | -
930 | Fail | Fail | Success | -
1213 | Fail | Fail | Success | -
4852 | Fail | Fail | Fail | Fail
||Heading 1||Heading 2||
|Col A1|Col A2|
> Improve correlation and covariance memory usage with large number of groups
> ---------------------------------------------------------------------------
>
> Key: MADLIB-1301
> URL: https://issues.apache.org/jira/browse/MADLIB-1301
> Project: Apache MADlib
> Issue Type: Improvement
> Components: Module: Descriptive Statistics
> Reporter: Frank McQuillan
> Priority: Major
> Fix For: v2.0
>
>
> When correlation and covariance are run with large number of groups (100's),
> can run out of memory. Increasing statement_mem helps, but this JIRA is to
> investigate and improve memory usage with large numbers of groups.
> Sample findings on correlation for 300K input data set:
> || #groups || statement mem 186M || statement mem 200M || statement mem 500M
> || statement mem 1000M ||
> 6 | Success | Success | Success | - |
> | 127 | Success | Success | - | - |
> | 930 | Fail | Fail | Success | - |
> | 1213 | Fail | Fail | Success | - |
> | 4852 | Fail | Fail | Fail | Fail |
> ||Heading 1||Heading 2||
> |Col A1|Col A2|
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
