[
https://issues.apache.org/jira/browse/MADLIB-1301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16882510#comment-16882510
]
Nandish Jayaram commented on MADLIB-1301:
-----------------------------------------
w/ [~okislal]
It is not a bug, but while deconstructing the correlation matrix to create the
output table, a big {{UNION ALL}} query is created, with one sub-query for each
distinct grouping value. This was causing the reported failures. The fix is to
run multiple queries, with each query processing the deconstruction of the
correlation matrix for a limited number of groups (we have defaulted the value
to 10). This value can be parameterized by the user with a newly introduced
optional parameter named {{n_groups_per_run}}.
> Improve correlation and covariance memory usage with large number of groups
> ---------------------------------------------------------------------------
>
> Key: MADLIB-1301
> URL: https://issues.apache.org/jira/browse/MADLIB-1301
> Project: Apache MADlib
> Issue Type: Improvement
> Components: Module: Descriptive Statistics
> Reporter: Frank McQuillan
> Assignee: Nandish Jayaram
> Priority: Major
> Fix For: v1.17
>
>
> When correlation and covariance are run with large number of groups (100's),
> can run out of memory. Increasing statement_mem helps, but this JIRA is to
> investigate and improve memory usage with large numbers of groups.
> Sample findings on correlation for 300K input data set:
> || #groups || statement mem 186M || statement mem 200M || statement mem 500M
> || statement mem 1000M ||
> | 6 | Success | Success | Success | - |
> | 127 | Success | Success | - | - |
> | 930 | Fail | Fail | Success | - |
> | 1213 | Fail | Fail | Success | - |
> | 4852 | Fail | Fail | Fail | Fail |
> The question is:
> * is this a bug?
> * or is it a limitation of current grouping design?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
