[
https://issues.apache.org/jira/browse/SYSTEMML-489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15230867#comment-15230867
]
Deron Eriksson commented on SYSTEMML-489:
-----------------------------------------
Troubleshooting Guide has been checked in with commons-math3 recommendations.
Fixed by [PR104|https://github.com/apache/incubator-systemml/pull/104].
> ClassNotFoundException when training GML on Hadoop 2.0.0-cdh4.2.1
> -----------------------------------------------------------------
>
> Key: SYSTEMML-489
> URL: https://issues.apache.org/jira/browse/SYSTEMML-489
> Project: SystemML
> Issue Type: Bug
> Reporter: Yifan "Ethan" Xu
> Assignee: Deron Eriksson
> Priority: Minor
>
> h4. Summary
> When building a GLM model () on an old hadoop cluster (Hadoop 2.0.0-cdh4.2.1)
> I encountered the following error:
> {code:java}
> java.lang.ClassNotFoundException:org.apache.commons.math3.linear.RealMatrix
> {code}
> The problem was fixed by changing the <scope> of the associate library in the
> pom.xml from "provided" to "compile", and rebuilding the project.
> h4.Details
> *Command and Error Message:*
> {code}
> $hadoop jar ./SystemML.jar -f <parent
> dir>/incubator-systemml/scripts/algorithms/GLM.dml -nvargs X=<location on
> HDFS>/train.data.csv Y=<location on HDFS>/train.labels.csv B=<location on
> HDFS>/logisticResult/B.csv fmt=csv dfam=2 link=2 yneg=2.0 icpt=1 reg=0.0
> tol=0.00000001 moi=100 mii=10
> 16/01/29 10:07:21 INFO api.DMLScript: BEGIN DML run 01/29/2016 10:07:21
> 16/01/29 10:07:21 INFO api.DMLScript: HADOOP_HOME: null
> 16/01/29 10:07:21 WARN conf.DMLConfig: No default SystemML config file
> (./SystemML-config.xml) found
> 16/01/29 10:07:21 WARN conf.DMLConfig: Using default settings in DMLConfig
> 16/01/29 10:07:23 WARN hops.OptimizerUtils: Auto-disable multi-threaded text
> read for 'text' and 'csv' due to thread contention on JRE < 1.8
> (java.version=1.6.0_32).
> 16/01/29 10:07:23 WARN parser.StatementBlock: WARNING:<parent
> dir>/incubator-systemml/scripts/algorithms/GLM.dml -- line 357, column 14 --
> Initialization of w depends on if-else execution
> 16/01/29 10:07:23 WARN parser.StatementBlock: WARNING: <parent
> dir>/incubator-systemml/scripts/algorithms/GLM.dml -- line 357, column 14 --
> Initialization of w depends on while execution
> 16/01/29 10:07:24 WARN parser.StatementBlock: WARNING: <parent
> dir>/incubator-systemml/scripts/algorithms/GLM.dml -- line 161, column 0 --
> Variable termination_code defined with different value type in if and else
> clause.
> BEGIN GLM SCRIPT
> Reading X...
> Reading Y...
> Binomial.logit
> 16/01/29 10:07:25 INFO api.DMLScript: SystemML Statistics:
> Total execution time: 0.111 sec.
> Number of executed MR Jobs: 0.
> 16/01/29 10:07:25 INFO api.DMLScript: END DML run 01/29/2016 10:07:25
> Exception in thread "main" java.lang.NoClassDefFoundError:
> org/apache/commons/math3/linear/RealMatrix
> at
> org.apache.sysml.runtime.instructions.cp.MatrixBuiltinCPInstruction.processInstruction(MatrixBuiltinCPInstruction.java:46)
> at
> org.apache.sysml.runtime.controlprogram.ProgramBlock.executeSingleInstruction(ProgramBlock.java:309)
> at
> org.apache.sysml.runtime.controlprogram.ProgramBlock.executeInstructions(ProgramBlock.java:227)
> at
> org.apache.sysml.runtime.controlprogram.ProgramBlock.execute(ProgramBlock.java:169)
> at
> org.apache.sysml.runtime.controlprogram.IfProgramBlock.execute(IfProgramBlock.java:158)
> at
> org.apache.sysml.runtime.controlprogram.IfProgramBlock.execute(IfProgramBlock.java:175)
> at
> org.apache.sysml.runtime.controlprogram.IfProgramBlock.execute(IfProgramBlock.java:175)
> at
> org.apache.sysml.runtime.controlprogram.IfProgramBlock.execute(IfProgramBlock.java:175)
> at
> org.apache.sysml.runtime.controlprogram.IfProgramBlock.execute(IfProgramBlock.java:158)
> at
> org.apache.sysml.runtime.controlprogram.FunctionProgramBlock.execute(FunctionProgramBlock.java:115)
> at
> org.apache.sysml.runtime.instructions.cp.FunctionCallCPInstruction.processInstruction(FunctionCallCPInstruction.java:177)
> at
> org.apache.sysml.runtime.controlprogram.ProgramBlock.executeSingleInstruction(ProgramBlock.java:309)
> at
> org.apache.sysml.runtime.controlprogram.ProgramBlock.executeInstructions(ProgramBlock.java:227)
> at
> org.apache.sysml.runtime.controlprogram.ProgramBlock.execute(ProgramBlock.java:169)
> at
> org.apache.sysml.runtime.controlprogram.IfProgramBlock.execute(IfProgramBlock.java:158)
> at
> org.apache.sysml.runtime.controlprogram.Program.execute(Program.java:146)
> at org.apache.sysml.api.DMLScript.execute(DMLScript.java:676)
> at org.apache.sysml.api.DMLScript.executeScript(DMLScript.java:338)
> at org.apache.sysml.api.DMLScript.main(DMLScript.java:197)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
> Caused by: java.lang.ClassNotFoundException:
> org.apache.commons.math3.linear.RealMatrix
> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
> ... 24 more
> {code}
> *Hadoop version*
> {code}
> 2.0.0-cdh4.2.1, 144bd548d481c2774fab2bec2ac2645d190f705b
> {code}
> *Changes in {{pom.xml}} that Fixed the Problem*
> Line 773 - 778
> Before:
> {code:xml}
> <dependency>
> <groupId>org.apache.commons</groupId>
> <artifactId>commons-math3</artifactId>
> <version>3.1.1</version>
> <scope>provided</scope>
> </dependency>
> {code}
> After:
> {code:xml}
> <dependency>
> <groupId>org.apache.commons</groupId>
> <artifactId>commons-math3</artifactId>
> <version>3.1.1</version>
> <scope>compile</scope>
> </dependency>
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
