Baunsgaard commented on pull request #1328:
URL: https://github.com/apache/systemds/pull/1328#issuecomment-872334806
Okay i have been looking at this, and there is a bug.
The bug for the paramserv cannot be called inside a builtin function since
it in turn try parse functions for update and gradients.
If we change the line initializing the parameterserver to:
model_trained = paramserv(model=model, features=X, labels=y,
upd="./scripts/builtin/ffTrain.dml::gradients",
agg="./scripts/builtin/ffTrain.dml::aggregation",
epochs=epochs, hyperparams=params, mode=mode, utype=utype, freq=freq,
batchsize=batch_size, k=workers,
checkpointing="NONE")
Then it is syntactically correct, but in turn our passer encounter a bug,
where it cannot build the ffTrain builtin.
```
[ERROR] Failures:
[ERROR]
BuiltinFFNeuralNetworkTest.testClassificationFit:91->AutomatedTestBase.runTest:1333->AutomatedTestBase.runTest:1337->AutomatedTestBase.runTest:1341->AutomatedTestBase.runTest:1446
failed to run script:
./src/test/scripts/functions/builtin/ffNeuralNetwork.dml
Standard Out:
StackTrace:
LEVEL : 0
Exception : class org.apache.sysds.api.DMLException
Message : failed add function to graph fcall _main
0 >
org.apache.sysds.hops.ipa.FunctionCallGraph.addFunctionOpToGraph(FunctionCallGraph.java:507)
0 >
org.apache.sysds.hops.ipa.FunctionCallGraph.rConstructFunctionCallGraph(FunctionCallGraph.java:424)
0 >
org.apache.sysds.hops.ipa.FunctionCallGraph.constructFunctionCallGraph(FunctionCallGraph.java:360)
0 >
org.apache.sysds.hops.ipa.FunctionCallGraph.<init>(FunctionCallGraph.java:94)
0 >
org.apache.sysds.hops.ipa.InterProceduralAnalysis.<init>(InterProceduralAnalysis.java:133)
0 >
org.apache.sysds.parser.DMLTranslator.rewriteHopsDAG(DMLTranslator.java:285)
0 > org.apache.sysds.api.DMLScript.execute(DMLScript.java:400)
0 > org.apache.sysds.api.DMLScript.executeScript(DMLScript.java:269)
0 >
org.apache.sysds.test.AutomatedTestBase.main(AutomatedTestBase.java:1464)
0 >
org.apache.sysds.test.AutomatedTestBase.runTest(AutomatedTestBase.java:1420)
0 >
org.apache.sysds.test.AutomatedTestBase.runTest(AutomatedTestBase.java:1341)
0 >
org.apache.sysds.test.AutomatedTestBase.runTest(AutomatedTestBase.java:1337)
0 >
org.apache.sysds.test.AutomatedTestBase.runTest(AutomatedTestBase.java:1333)
0 >
org.apache.sysds.test.functions.builtin.BuiltinFFNeuralNetworkTest.testClassificationFit(BuiltinFFNeuralNetworkTest.java:91)
0 > sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
0 >
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
0 >
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
0 > java.lang.reflect.Method.invoke(Method.java:498)
> ... Stopping Stack Trace at JUnit
LEVEL : 1
Exception : class org.apache.sysds.api.DMLException
Message : failed add function to graph fcall .builtinNS::m_ffTrain
1 >
org.apache.sysds.hops.ipa.FunctionCallGraph.addFunctionOpToGraph(FunctionCallGraph.java:507)
1 >
org.apache.sysds.hops.ipa.FunctionCallGraph.rConstructFunctionCallGraph(FunctionCallGraph.java:451)
1 >
org.apache.sysds.hops.ipa.FunctionCallGraph.rConstructFunctionCallGraph(FunctionCallGraph.java:442)
1 >
org.apache.sysds.hops.ipa.FunctionCallGraph.rConstructFunctionCallGraph(FunctionCallGraph.java:428)
1 >
org.apache.sysds.hops.ipa.FunctionCallGraph.rConstructFunctionCallGraph(FunctionCallGraph.java:399)
1 >
org.apache.sysds.hops.ipa.FunctionCallGraph.addFunctionOpToGraph(FunctionCallGraph.java:487)
1 >
org.apache.sysds.hops.ipa.FunctionCallGraph.rConstructFunctionCallGraph(FunctionCallGraph.java:424)
1 >
org.apache.sysds.hops.ipa.FunctionCallGraph.constructFunctionCallGraph(FunctionCallGraph.java:360)
1 >
org.apache.sysds.hops.ipa.FunctionCallGraph.<init>(FunctionCallGraph.java:94)
1 >
org.apache.sysds.hops.ipa.InterProceduralAnalysis.<init>(InterProceduralAnalysis.java:133)
1 >
org.apache.sysds.parser.DMLTranslator.rewriteHopsDAG(DMLTranslator.java:285)
1 > org.apache.sysds.api.DMLScript.execute(DMLScript.java:400)
1 > org.apache.sysds.api.DMLScript.executeScript(DMLScript.java:269)
1 >
org.apache.sysds.test.AutomatedTestBase.main(AutomatedTestBase.java:1464)
1 >
org.apache.sysds.test.AutomatedTestBase.runTest(AutomatedTestBase.java:1420)
1 >
org.apache.sysds.test.AutomatedTestBase.runTest(AutomatedTestBase.java:1341)
1 >
org.apache.sysds.test.AutomatedTestBase.runTest(AutomatedTestBase.java:1337)
1 >
org.apache.sysds.test.AutomatedTestBase.runTest(AutomatedTestBase.java:1333)
1 >
org.apache.sysds.test.functions.builtin.BuiltinFFNeuralNetworkTest.testClassificationFit(BuiltinFFNeuralNetworkTest.java:91)
1 > sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
1 >
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
1 >
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
1 > java.lang.reflect.Method.invoke(Method.java:498)
> ... Stopping Stack Trace at JUnit
LEVEL : 2
Exception : class java.lang.NullPointerException
Message : null
2 >
org.apache.sysds.hops.ipa.FunctionCallGraph.addFunctionOpToGraph(FunctionCallGraph.java:485)
2 >
org.apache.sysds.hops.ipa.FunctionCallGraph.rConstructFunctionCallGraph(FunctionCallGraph.java:451)
2 >
org.apache.sysds.hops.ipa.FunctionCallGraph.rConstructFunctionCallGraph(FunctionCallGraph.java:442)
2 >
org.apache.sysds.hops.ipa.FunctionCallGraph.rConstructFunctionCallGraph(FunctionCallGraph.java:428)
2 >
org.apache.sysds.hops.ipa.FunctionCallGraph.rConstructFunctionCallGraph(FunctionCallGraph.java:399)
2 >
org.apache.sysds.hops.ipa.FunctionCallGraph.addFunctionOpToGraph(FunctionCallGraph.java:487)
2 >
org.apache.sysds.hops.ipa.FunctionCallGraph.rConstructFunctionCallGraph(FunctionCallGraph.java:424)
2 >
org.apache.sysds.hops.ipa.FunctionCallGraph.constructFunctionCallGraph(FunctionCallGraph.java:360)
2 >
org.apache.sysds.hops.ipa.FunctionCallGraph.<init>(FunctionCallGraph.java:94)
2 >
org.apache.sysds.hops.ipa.InterProceduralAnalysis.<init>(InterProceduralAnalysis.java:133)
2 >
org.apache.sysds.parser.DMLTranslator.rewriteHopsDAG(DMLTranslator.java:285)
2 > org.apache.sysds.api.DMLScript.execute(DMLScript.java:400)
2 > org.apache.sysds.api.DMLScript.executeScript(DMLScript.java:269)
2 >
org.apache.sysds.test.AutomatedTestBase.main(AutomatedTestBase.java:1464)
2 >
org.apache.sysds.test.AutomatedTestBase.runTest(AutomatedTestBase.java:1420)
2 >
org.apache.sysds.test.AutomatedTestBase.runTest(AutomatedTestBase.java:1341)
2 >
org.apache.sysds.test.AutomatedTestBase.runTest(AutomatedTestBase.java:1337)
2 >
org.apache.sysds.test.AutomatedTestBase.runTest(AutomatedTestBase.java:1333)
2 >
org.apache.sysds.test.functions.builtin.BuiltinFFNeuralNetworkTest.testClassificationFit(BuiltinFFNeuralNetworkTest.java:91)
2 > sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
2 >
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
2 >
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2 > java.lang.reflect.Method.invoke(Method.java:498)
> ... Stopping Stack Trace at JUnit
````
For now i would suggest not including parameter server functionality in the
builtin, since this provoke the bug and instead focus on the FF, that works.
I have squashed together all your current code in a branch on my fork, if
you want to continue on that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
