that's a good catch - thanks Felix. It would be great if you could
modify rewriteSetExecutionStategy and
rewriteSetFusedDataPartitioningExecution in OptimizerConstrained to
handle the respective Spark execution types. Thanks.
Regards,
Matthias
On 11/22/2016 7:54 PM, fschue...@posteo.de wrote:
The constrained optimizer doesn't seem to know about a REMOTE_SPARK
execution mode and either sets CP or REMOTE_MR. I can open a jira for
that and provide a fix.
Felix
Am 22.11.2016 02:07 schrieb Matthias Boehm:
yes, this came up several times - initially we only supported opt=NONE
where users had to specify all other parameters. Meanwhile, there is a
so-called "constrained optimizer" that does the same as the rule-based
optimizer but respects any given parameters. Please try something like
this:
parfor (i in 1:10, opt=CONSTRAINED, par=10, mode=REMOTE_SPARK) {
// some code here
}
Regards,
Matthias
On 11/22/2016 12:33 AM, fschue...@posteo.de wrote:
While debugging some ParFor code it became clear that the parameters for
parfor can be easily overwritten by the optimizer.
One example is when I write:
```
parfor (i in 1:10, par=10, mode=REMOTE_SPARK) {
// some code here
}
```
Depending on the data size and cluster resources, the optimizer
(OptimizerRuleBased.java, line 844) will recognize that the work can be
done locally and overwrite it to local execution. This might be valid
and definitely works (in my case) but kind of contradicts what I want
SystemML to do.
I wonder if we should disable this optimization in case a concrete
execution mode is given and go with the mode that is provided.
Felix
