Hi Ethan, The getDouble() method is actually part of org.apache.hadoop.conf.Configuration.java, which is part of hadoop-common but not hadoop-core -- see [1]. Seems like, it used to be part of hadoop-core a long time ago. Also, the pom.xml in SystemML project does specify hadoop-common as the dependency (as provided library). Make sure hadoop-common jar file is in the classpath.
[1] http://grepcode.com/file/repo1.maven.org/maven2/com.ning/metrics.collector/1.3.0/org/apache/hadoop/conf/Configuration.java Shirish On Thu, Feb 4, 2016 at 5:51 PM, Ethan Xu <ethan.yifa...@gmail.com> wrote: > Hello, > > I got an error when running the systemML/scripts/Univar-Stats.dml script on > a hadoop cluster (Cloudera CDH4.2.1) on a 6GB data set. Error message is at > the bottom of the email. The same script ran fine on a smaller sample > (several MB) of the same data set, when MR was not invoked. > > The main error was java.lang.NoSuchMethodError: > org.apache.hadoop.mapred.JobConf.getDouble() > Digging deeper, it looks like the CDH4.2.1 version of MR indeed didn't have > the JobConf.getDouble() method. > > The hadoop-core jar of CDH4.2.1 can be found here: > > https://repository.cloudera.com/artifactory/repo/org/apache/hadoop/hadoop-core/2.0.0-mr1-cdh4.2.1/ > > The calling line of SystemML is line 1194 of > > https://github.com/apache/incubator-systemml/blob/master/src/main/java/org/apache/sysml/runtime/matrix/mapred/MRJobConfiguration.java > > I was wondering, if the finding is accurate, is there a potential fix, or > does this mean the current version of SystemML is not compatible with > CDH4.2.1? > > Thank you, > > Ethan > > > hadoop jar $sysDir/target/SystemML.jar -f > $sysDir/scripts/algorithms/Univar-Stats.dml -nvargs > X=$baseDirHDFS/original-coded.csv > TYPES=$baseDirHDFS/original-coded-type.csv > STATS=$baseDirHDFS/univariate-summary.csv > > 16/02/04 20:35:03 INFO api.DMLScript: BEGIN DML run 02/04/2016 20:35:03 > 16/02/04 20:35:03 INFO api.DMLScript: HADOOP_HOME: null > 16/02/04 20:35:03 WARN conf.DMLConfig: No default SystemML config file > (./SystemML-config.xml) found > 16/02/04 20:35:03 WARN conf.DMLConfig: Using default settings in DMLConfig > 16/02/04 20:35:04 WARN hops.OptimizerUtils: Auto-disable multi-threaded > text read for 'text' and 'csv' due to thread contention on JRE < 1.8 > (java.version=1.7.0_71). > SLF4J: Class path contains multiple SLF4J bindings. > SLF4J: Found binding in > > [jar:file:/usr/lib/zookeeper/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class] > SLF4J: Found binding in > > [jar:file:/usr/local/explorys/datagrid/lib/slf4j-jdk14-1.6.6.jar!/org/slf4j/impl/StaticLoggerBinder.class] > SLF4J: Found binding in > > [jar:file:/usr/local/explorys/datagrid/lib/logback-classic-1.0.6.jar!/org/slf4j/impl/StaticLoggerBinder.class] > SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an > explanation. > 16/02/04 20:35:07 INFO api.DMLScript: SystemML Statistics: > Total execution time: 0.880 sec. > Number of executed MR Jobs: 0. > > 16/02/04 20:35:07 INFO api.DMLScript: END DML run 02/04/2016 20:35:07 > Exception in thread "main" java.lang.NoSuchMethodError: > org.apache.hadoop.mapred.JobConf.getDouble(Ljava/lang/String;D)D > at > > org.apache.sysml.runtime.matrix.mapred.MRJobConfiguration.setUpMultipleInputs(MRJobConfiguration.java:1195) > at > > org.apache.sysml.runtime.matrix.mapred.MRJobConfiguration.setUpMultipleInputs(MRJobConfiguration.java:1129) > at > > org.apache.sysml.runtime.matrix.CSVReblockMR.runAssignRowIDMRJob(CSVReblockMR.java:307) > at > > org.apache.sysml.runtime.matrix.CSVReblockMR.runAssignRowIDMRJob(CSVReblockMR.java:289) > at > org.apache.sysml.runtime.matrix.CSVReblockMR.runJob(CSVReblockMR.java:275) > at > org.apache.sysml.lops.runtime.RunMRJobs.submitJob(RunMRJobs.java:257) > at > > org.apache.sysml.lops.runtime.RunMRJobs.prepareAndSubmitJob(RunMRJobs.java:143) > at > > org.apache.sysml.runtime.instructions.MRJobInstruction.processInstruction(MRJobInstruction.java:1500) > at > > org.apache.sysml.runtime.controlprogram.ProgramBlock.executeSingleInstruction(ProgramBlock.java:309) > at > > org.apache.sysml.runtime.controlprogram.ProgramBlock.executeInstructions(ProgramBlock.java:227) > at > > org.apache.sysml.runtime.controlprogram.ProgramBlock.execute(ProgramBlock.java:169) > at > org.apache.sysml.runtime.controlprogram.Program.execute(Program.java:146) > at org.apache.sysml.api.DMLScript.execute(DMLScript.java:676) > at org.apache.sysml.api.DMLScript.executeScript(DMLScript.java:338) > at org.apache.sysml.api.DMLScript.main(DMLScript.java:197) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.apache.hadoop.util.RunJar.main(RunJar.java:208) >
