[
https://issues.apache.org/jira/browse/SYSTEMML-1561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15987815#comment-15987815
]
Mike Dusenberry commented on SYSTEMML-1561:
-------------------------------------------
Thanks, [~mboehm7]. For this scenario, I did have the recompile debug logging
enabled, and that showed that the sizes remained unknown due to the sub-graph
not being evaluated and replaced, and Spark jobs were indeed launched. Here's
some more information: These equations for {{Hout}} and {{Wout}} are based on
scalar values passed into the {{conv2d::forward}} & {{max_pool2d::forward}}
functions, and the scalars are known at regular compile time, either directly
as constants (i.e. hyperparams), or indirectly from an initial set of constants
(i.e. image sizes as the images move through the model are based on a chain of
size equations starting with initial image size constants). Especially with
the IPA enhancement in [PR 468 |
https://github.com/apache/incubator-systemml/pull/468], debugging shows that
the scalar leaf nodes for these sub-dags are all replaced with literals during
regular compilation, so we should be able to evaluate it as part of the
constant folding during regular compilation.
Here is an example of the of the {{Hout}} and {{Wout}} equations from
[{{conv2d::forward}} |
https://github.com/apache/incubator-systemml/blob/master/scripts/nn/layers/conv2d_builtin.dml#L69]:
{code}
forward = function(matrix[double] X, matrix[double] W, matrix[double] b,
int C, int Hin, int Win, int Hf, int Wf,
int strideh, int stridew, int padh, int padw)
return (matrix[double] out, int Hout, int Wout) {
...
Hout = as.integer(floor((Hin + 2*padh - Hf)/strideh + 1))
Wout = as.integer(floor((Win + 2*padw - Wf)/stridew + 1))
...
{code}
Just to explain this a bit, {{padh}}, {{padw}}, {{Hf}}, {{Wf}}, {{stridew}},
and {{strideh}} are generally all defined as constants, and {{Hin}} & {{Win}}
(height & width of the input "images") will be constants for the initial layer
(i.e. the initial images passed in will have constants defined for height and
width), and subsequent layers will depend on {{Hout}} and {{Wout}} from
previous layers. Therefore, any given {{Hout}} or {{Wout}} is based either
directly or indirectly on constants, and can be determined, at least
mathematically, from the initial script.
> Improve constant folding during compilation
> -------------------------------------------
>
> Key: SYSTEMML-1561
> URL: https://issues.apache.org/jira/browse/SYSTEMML-1561
> Project: SystemML
> Issue Type: Improvement
> Reporter: Mike Dusenberry
> Attachments: scenario1_plan.txt, scenario1.py, scenario2_plan.txt,
> scenario2.py
>
>
> In our `nn` library, our convolution and pooling layers have to pass around
> the spatial dimensions (height and width) of the images that are stretched
> out into rows of the input/output matrices. These output dimensions are
> computed within the forward functions of the above layers as small scalar
> equations. From a mathematical standpoint, these sizes can be determined at
> compile time, and it is nice to have these size equations in DML (v.s. hiding
> them inside the engine within built-in functions). However, we do not
> currently evaluate these expressions during compilation, and thus we are left
> with unknown sizes even during recompilation. This naturally leads to max
> memory estimates and thus often leads to unnecessary distributed runtime ops
> rather than simple CP ones.
> I have two related scenarios for which this is a problem. They both involve
> the {{Houtc1}} & {{Woutc1}} values that are returned from a
> `conv2d::forward(...)` function. These represent the spatial dimensions of
> the volume with each of the rows of the output {{outc1}} of the function, and
> the third dimension is {{F1}}. Thus, {{outc1}} has a number of columns equal
> to {{F1*Houtc1*Wouc1}}.
> In the first scenario ({{scenario1.py}}), a random matrix {{doutc1}} is
> created that should have the same dimensions as {{outc1}}. For the columns,
> if I use {{cols=ncol(outc1)}} in this rand statement, the size will be
> propagated and CP ops will be compiled and run. I I instead use
> {{cols=F1*Houtc1*Woutc1}}, the size will forever be unknown, even during
> recompilation, and thus Spark ops will be compiled and run. I have included
> the recompile hops plan ({{scenario1_plan.txt}}).
> In the second scenario ({{scenario2.py}}), a {{max_pool2d::forward(...)}}
> function is inserted after the {{conv2d::forward(...)}} function that
> requires the {{Houtc1}} and {{Woutc1}} variables to be supplied as arguments.
> Since those latter variables are not executed during compilation time, the
> max pooling sizes remain unknown, even during recompilation, and thus Spark
> ops will be compiled and run. I have included the recompile hops plan
> ({{scenario2_plan.txt}}).
> We should either improve or fix our constant folding rewrites so that these
> scenarios are fixed, as they are necessary for performant deep learning
> applications. Note too that this issue will be present in other non-deep
> learning scenarios as well.
> Mailing list thread:
> https://www.mail-archive.com/dev@systemml.incubator.apache.org/msg01657.html
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
