[
https://issues.apache.org/jira/browse/MADLIB-1509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17569392#comment-17569392
]
Orhan Kislal commented on MADLIB-1509:
--------------------------------------
Hi,
The first execution of serialize_nd_weights is for the initial weights. If you
don't have any, it will go very fast as you noted. The actual serialization has
to happen on each iteration, this piece of the code is very straightforward:
{code:java}
flattened_weights = [w.flatten() for w in model_weights]
flattened_weights = np.concatenate(flattened_weights)
return np.float32(flattened_weights).tostring() {code}
If you have a different suggestion, I would be happy to look into it.
MADlib doesn't set memory limits (except for GPU operations). You might want to
look at PostgreSQL/Greenplum documentation to ensure the memory is actually
available to MADlib.
Thanks
> Memory Shortage when serializing the model
> ------------------------------------------
>
> Key: MADLIB-1509
> URL: https://issues.apache.org/jira/browse/MADLIB-1509
> Project: Apache MADlib
> Issue Type: Bug
> Components: Deep Learning
> Reporter: Xinyi Zhang
> Priority: Major
> Fix For: v1.19.0
>
>
> When I train a model whose size is about 400MB, the execution time for
> serialize_nd_weights can be very slow. Specifically, the first-time execution
> of serialize_nd_weights lasts about {color:#FF0000}2 seconds{color}, which is
> reasonable. However, afterward, its execution time becomes about
> {color:#FF0000}190 seconds{color}.
> I think the memory shortage causes the long execution time for
> serialize_nd_weights, since the time can be reduced with a more
> memory-efficient implementation for serializing (from 190 seconds to 70
> seconds).
> However, the instance I run Madlib has 290G memory available. I think that
> Madlib might set the wrong memory limit for its task. Is there any way to
> configure the memory limit for Madlib tasks?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
