Hi Sandeep,
Thank you for contacting the Apache SystemML community. Based on the
following trace fragment:
spark.SparkContext: Running Spark version 2.0.0
it looks like you're running against Spark 2.0 which is not officially
supported with the current master code base. Can you retry with an earlier
Spark version? Let me know if I've misinterpreted your Spark environment.
Cheers,
Glenn
From: "Narayanaswami, Sandeep" <sandeep.narayanasw...@capitalone.com>
To: "dev@systemml.incubator.apache.org"
<dev@systemml.incubator.apache.org>
Date: 10/19/2016 05:00 PM
Subject: Running the bivar-stats example
Hi SystemML community,
I’m trying to run the bivariate stats example script on Spark. I’m only
doing a toy example with very small data, but I have been running into
errors.
My script invocation is as follows:
spark-submit --master local[4] SystemML.jar -f
scripts/algorithms/bivar-stats.dml -exec hybrid_spark -nvargs
X=/data/iqtv.csv index1=/data/index1.csv index2=/data/index2.csv
types1=/data/types1.csv types2=/data/types2.csv OUTDIR=/data/outdata
Am I calling the script incorrectly?
(All the files are on hdfs. Iqtv.csv is a headerless csv with two numeric
fields. The index1 and index2 files are text files containing only the
strings ‘1’ and ‘2’ respectively. The types* files are text files each
containing only the string ‘1’.)
Please excuse the rather long stack trace below, but I’m hoping it will
help diagnose the problem.
Cheers,
Sandeep
================================================================================================================================
16/10/19 16:46:26 INFO api.DMLScript: BEGIN DML run 10/19/2016 16:46:26
ANTLR Tool version 4.3 used for code generation does not match the current
runtime version 4.5.3[…]
16/10/19 16:46:28 INFO spark.SparkContext: Running Spark version 2.0.0
[…]
16/10/19 16:46:29 INFO api.DMLScript: END DML run 10/19/2016 16:46:29
16/10/19 16:46:29 ERROR api.DMLScript: Failed to execute DML script.
org.apache.sysml.runtime.DMLRuntimeException:
org.apache.sysml.runtime.DMLRuntimeException:
org.apache.sysml.runtime.DMLRuntimeException: Unable to recompile program
block hierarchy.
at org.apache.sysml.runtime.controlprogram.Program.execute
(Program.java:152)
at org.apache.sysml.api.DMLScript.execute(DMLScript.java:677)
at org.apache.sysml.api.DMLScript.executeScript(DMLScript.java:337)
at org.apache.sysml.api.DMLScript.main(DMLScript.java:192)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke
(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke
(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy
$SparkSubmit$$runMain(SparkSubmit.scala:729)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1
(SparkSubmit.scala:185)
at org.apache.spark.deploy.SparkSubmit$.submit
(SparkSubmit.scala:210)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:124)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: org.apache.sysml.runtime.DMLRuntimeException:
org.apache.sysml.runtime.DMLRuntimeException: Unable to recompile program
block hierarchy.
at
org.apache.sysml.runtime.controlprogram.parfor.opt.OptimizationWrapper.optimize
(OptimizationWrapper.java:299)
at
org.apache.sysml.runtime.controlprogram.parfor.opt.OptimizationWrapper.optimize
(OptimizationWrapper.java:172)
at
org.apache.sysml.runtime.controlprogram.ParForProgramBlock.execute
(ParForProgramBlock.java:550)
at org.apache.sysml.runtime.controlprogram.Program.execute
(Program.java:145)
... 12 more
Caused by: org.apache.sysml.runtime.DMLRuntimeException: Unable to
recompile program block hierarchy.
at
org.apache.sysml.hops.recompile.Recompiler.recompileProgramBlockHierarchy
(Recompiler.java:541)
at
org.apache.sysml.runtime.controlprogram.parfor.opt.OptimizationWrapper.optimize
(OptimizationWrapper.java:278)
... 15 more
Caused by: org.apache.sysml.hops.HopsException: ERROR: line 342, column 4
-- error constructing Lops for UnaryOp Hop --
at org.apache.sysml.hops.UnaryOp.constructLops(UnaryOp.java:177)
at org.apache.sysml.hops.DataOp.constructLops(DataOp.java:252)
at org.apache.sysml.hops.recompile.Recompiler.recompileHopsDag
(Recompiler.java:209)
at org.apache.sysml.hops.recompile.Recompiler.rRecompileProgramBlock
(Recompiler.java:881)
at org.apache.sysml.hops.recompile.Recompiler.rRecompileProgramBlock
(Recompiler.java:839)
at org.apache.sysml.hops.recompile.Recompiler.rRecompileProgramBlock
(Recompiler.java:839)
at org.apache.sysml.hops.recompile.Recompiler.rRecompileProgramBlock
(Recompiler.java:841)
at org.apache.sysml.hops.recompile.Recompiler.rRecompileProgramBlock
(Recompiler.java:858)
at
org.apache.sysml.hops.recompile.Recompiler.recompileProgramBlockHierarchy
(Recompiler.java:536)
... 16 more
Caused by: org.apache.sysml.hops.HopsException: ERROR: line 0, column 0 --
error constructing Lops for UnaryOp Hop --
at org.apache.sysml.hops.UnaryOp.constructLops(UnaryOp.java:177)
at org.apache.sysml.hops.BinaryOp.constructLopsBinaryDefault
(BinaryOp.java:565)
at org.apache.sysml.hops.BinaryOp.constructLops(BinaryOp.java:166)
at org.apache.sysml.hops.BinaryOp.constructLopsBinaryDefault
(BinaryOp.java:566)
at org.apache.sysml.hops.BinaryOp.constructLops(BinaryOp.java:166)
at org.apache.sysml.hops.UnaryOp.constructLops(UnaryOp.java:142)
... 24 more
Caused by: java.lang.NullPointerException
at
org.apache.sysml.hops.ParameterizedBuiltinOp.constructLopsGroupedAggregate
(ParameterizedBuiltinOp.java:247)
at org.apache.sysml.hops.ParameterizedBuiltinOp.constructLops
(ParameterizedBuiltinOp.java:181)
at org.apache.sysml.hops.BinaryOp.constructLopsBinaryDefault
(BinaryOp.java:590)
at org.apache.sysml.hops.BinaryOp.constructLops(BinaryOp.java:166)
at org.apache.sysml.hops.ReorgOp.constructLops(ReorgOp.java:128)
at org.apache.sysml.hops.AggBinaryOp.constructSparkLopsCPMM
(AggBinaryOp.java:832)
at org.apache.sysml.hops.AggBinaryOp.constructLops
(AggBinaryOp.java:216)
at org.apache.sysml.hops.UnaryOp.constructLops(UnaryOp.java:142)
... 29 more
Exception in thread "main" org.apache.sysml.api.DMLException:
org.apache.sysml.runtime.DMLRuntimeException:
org.apache.sysml.runtime.DMLRuntimeException:
org.apache.sysml.runtime.DMLRuntimeException: Unable to recompile program
block hierarchy.
at org.apache.sysml.api.DMLScript.executeScript(DMLScript.java:351)
at org.apache.sysml.api.DMLScript.main(DMLScript.java:192)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke
(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke
(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy
$SparkSubmit$$runMain(SparkSubmit.scala:729)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1
(SparkSubmit.scala:185)
at org.apache.spark.deploy.SparkSubmit$.submit
(SparkSubmit.scala:210)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:124)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: org.apache.sysml.runtime.DMLRuntimeException:
org.apache.sysml.runtime.DMLRuntimeException:
org.apache.sysml.runtime.DMLRuntimeException: Unable to recompile program
block hierarchy.
at org.apache.sysml.runtime.controlprogram.Program.execute
(Program.java:152)
at org.apache.sysml.api.DMLScript.execute(DMLScript.java:677)
at org.apache.sysml.api.DMLScript.executeScript(DMLScript.java:337)
... 10 more
Caused by: org.apache.sysml.runtime.DMLRuntimeException:
org.apache.sysml.runtime.DMLRuntimeException: Unable to recompile program
block hierarchy.
at
org.apache.sysml.runtime.controlprogram.parfor.opt.OptimizationWrapper.optimize
(OptimizationWrapper.java:299)
at
org.apache.sysml.runtime.controlprogram.parfor.opt.OptimizationWrapper.optimize
(OptimizationWrapper.java:172)
at
org.apache.sysml.runtime.controlprogram.ParForProgramBlock.execute
(ParForProgramBlock.java:550)
at org.apache.sysml.runtime.controlprogram.Program.execute
(Program.java:145)
... 12 more
Caused by: org.apache.sysml.runtime.DMLRuntimeException: Unable to
recompile program block hierarchy.
at
org.apache.sysml.hops.recompile.Recompiler.recompileProgramBlockHierarchy
(Recompiler.java:541)
at
org.apache.sysml.runtime.controlprogram.parfor.opt.OptimizationWrapper.optimize
(OptimizationWrapper.java:278)
... 15 more
Caused by: org.apache.sysml.hops.HopsException: ERROR: line 342, column 4
-- error constructing Lops for UnaryOp Hop --
at org.apache.sysml.hops.UnaryOp.constructLops(UnaryOp.java:177)
at org.apache.sysml.hops.DataOp.constructLops(DataOp.java:252)
at org.apache.sysml.hops.recompile.Recompiler.recompileHopsDag
(Recompiler.java:209)
at org.apache.sysml.hops.recompile.Recompiler.rRecompileProgramBlock
(Recompiler.java:881)
at org.apache.sysml.hops.recompile.Recompiler.rRecompileProgramBlock
(Recompiler.java:839)
at org.apache.sysml.hops.recompile.Recompiler.rRecompileProgramBlock
(Recompiler.java:839)
at org.apache.sysml.hops.recompile.Recompiler.rRecompileProgramBlock
(Recompiler.java:841)
at org.apache.sysml.hops.recompile.Recompiler.rRecompileProgramBlock
(Recompiler.java:858)
at
org.apache.sysml.hops.recompile.Recompiler.recompileProgramBlockHierarchy
(Recompiler.java:536)
... 16 more
Caused by: org.apache.sysml.hops.HopsException: ERROR: line 0, column 0 --
error constructing Lops for UnaryOp Hop --
at org.apache.sysml.hops.UnaryOp.constructLops(UnaryOp.java:177)
at org.apache.sysml.hops.BinaryOp.constructLopsBinaryDefault
(BinaryOp.java:565)
at org.apache.sysml.hops.BinaryOp.constructLops(BinaryOp.java:166)
at org.apache.sysml.hops.BinaryOp.constructLopsBinaryDefault
(BinaryOp.java:566)
at org.apache.sysml.hops.BinaryOp.constructLops(BinaryOp.java:166)
at org.apache.sysml.hops.UnaryOp.constructLops(UnaryOp.java:142)
... 24 more
Caused by: java.lang.NullPointerException
at
org.apache.sysml.hops.ParameterizedBuiltinOp.constructLopsGroupedAggregate
(ParameterizedBuiltinOp.java:247)
at org.apache.sysml.hops.ParameterizedBuiltinOp.constructLops
(ParameterizedBuiltinOp.java:181)
at org.apache.sysml.hops.BinaryOp.constructLopsBinaryDefault
(BinaryOp.java:590)
at org.apache.sysml.hops.BinaryOp.constructLops(BinaryOp.java:166)
at org.apache.sysml.hops.ReorgOp.constructLops(ReorgOp.java:128)
at org.apache.sysml.hops.AggBinaryOp.constructSparkLopsCPMM
(AggBinaryOp.java:832)
at org.apache.sysml.hops.AggBinaryOp.constructLops
(AggBinaryOp.java:216)
at org.apache.sysml.hops.UnaryOp.constructLops(UnaryOp.java:142)
... 29 more
16/10/19 16:46:29 INFO util.ShutdownHookManager: Shutdown hook called
[…]
________________________________________________________
The information contained in this e-mail is confidential and/or proprietary
to Capital One and/or its affiliates and may only be used solely in
performance of work or services for Capital One. The information
transmitted herewith is intended only for use by the individual or entity
to which it is addressed. If the reader of this message is not the intended
recipient, you are hereby notified that any review, retransmission,
dissemination, distribution, copying or other use of, or taking of any
action in reliance upon this information is strictly prohibited. If you
have received this communication in error, please contact the sender and
delete the material from your computer.
