Re: Running the bivar-stats example
That’s correct, I’m using the 0.10.0 release. From: Glenn Weidner Reply-To: "dev@systemml.incubator.apache.org" Date: Thursday, October 27, 2016 at 15:22 To: "dev@systemml.incubator.apache.org" Subject: Re: Running the bivar-stats example Hi Sandeep, No problem and thank you for trying the other scenarios. Just one quick question - are you using the 0.10.0 binary release? Thanks, Glenn [nactive hide details for "Narayanaswami, Sandeep" ---10/27/2016 01:14:26]"Narayanaswami, Sandeep" ---10/27/2016 01:14:26 PM---Hi, sorry I couldn’t get back on this sooner. @Glenn: Thanks for pointing that out. Unfortunately, From: "Narayanaswami, Sandeep" To: "dev@systemml.incubator.apache.org" Date: 10/27/2016 01:14 PM Subject: Re: Running the bivar-stats example Hi, sorry I couldn’t get back on this sooner. @Glenn: Thanks for pointing that out. Unfortunately, switching to Spark 1.6 didn’t make any difference: I got the same error/stacktrace again. @Matthias: I did have the corresponding metadata files. I also tried modifying the script to print the matrix after reading it in, and that works as expected so I don’t think it’s a data read issue. I’d appreciate any other suggestions. Cheers, Sandeep On 10/20/16, 02:30, "Matthias Boehm" wrote: apart from the missing support for Spark 2.x, I would recommend to double check your inputs. Do you have meta data files along with your inputs? If not, then these inputs are assumed to be in default format "text", i.e., matrix market ijv representation (not csv). You can either provide such json meta data files or simply put a 'format="csv"' into the read statements in bivar-stats.dml. Regards, Matthias On 10/20/2016 2:44 AM, Glenn Weidner wrote: > Hi Sandeep, > > Thank you for contacting the Apache SystemML community. Based on the > following trace fragment: > > spark.SparkContext: Running Spark version 2.0.0 > > it looks like you're running against Spark 2.0 which is not officially > supported with the current master code base. Can you retry with an earlier > Spark version? Let me know if I've misinterpreted your Spark environment. > > Cheers, > Glenn > > > > From: "Narayanaswami, Sandeep" > To: "dev@systemml.incubator.apache.org" > > Date: 10/19/2016 05:00 PM > Subject: Running the bivar-stats example > > > > Hi SystemML community, > > I’m trying to run the bivariate stats example script on Spark. I’m only > doing a toy example with very small data, but I have been running into > errors. > My script invocation is as follows: > > spark-submit --master local[4] SystemML.jar -f > scripts/algorithms/bivar-stats.dml -exec hybrid_spark -nvargs > X=/data/iqtv.csv index1=/data/index1.csv index2=/data/index2.csv > types1=/data/types1.csv types2=/data/types2.csv OUTDIR=/data/outdata > > Am I calling the script incorrectly? > (All the files are on hdfs. Iqtv.csv is a headerless csv with two numeric > fields. The index1 and index2 files are text files containing only the > strings ‘1’ and ‘2’ respectively. The types* files are text files each > containing only the string ‘1’.) > > Please excuse the rather long stack trace below, but I’m hoping it will > help diagnose the problem. > > Cheers, > Sandeep > > > > > 16/10/19 16:46:26 INFO api.DMLScript: BEGIN DML run 10/19/2016 16:46:26 > ANTLR Tool version 4.3 used for code generation does not match the current > runtime version 4.5.3[…] > 16/10/19 16:46:28 INFO spark.SparkContext: Running Spark version 2.0.0 > […] > 16/10/19 16:46:29 INFO api.DMLScript: END DML run 10/19/2016 16:46:29 > 16/10/19 16:46:29 ERROR api.DMLScript: Failed to execute DML script. > org.apache.sysml.runtime.DMLRuntimeException: > org.apache.sysml.runtime.DMLRuntimeException: > org.apache.sysml.runtime.DMLRuntimeException: Unable to recompile program > block hierarchy. >at org.apache.sysml.runtime.controlprogram.Program.execute > (Program.java:152) >at org.apache.sysml.api.DMLScript.execute(DMLScript.java:677) >at org.apache.sysml.api.DMLScript.executeScript(DMLScript.java:337) >at org.apache.sysml.api.DMLScript.main(DMLScript.java:192) >at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >at sun.reflect.NativeMethodAccessorImpl.invok
Re: Running the bivar-stats example
Hi Sandeep, No problem and thank you for trying the other scenarios. Just one quick question - are you using the 0.10.0 binary release? Thanks, Glenn From: "Narayanaswami, Sandeep" To: "dev@systemml.incubator.apache.org" Date: 10/27/2016 01:14 PM Subject: Re: Running the bivar-stats example Hi, sorry I couldn’t get back on this sooner. @Glenn: Thanks for pointing that out. Unfortunately, switching to Spark 1.6 didn’t make any difference: I got the same error/stacktrace again. @Matthias: I did have the corresponding metadata files. I also tried modifying the script to print the matrix after reading it in, and that works as expected so I don’t think it’s a data read issue. I’d appreciate any other suggestions. Cheers, Sandeep On 10/20/16, 02:30, "Matthias Boehm" wrote: apart from the missing support for Spark 2.x, I would recommend to double check your inputs. Do you have meta data files along with your inputs? If not, then these inputs are assumed to be in default format "text", i.e., matrix market ijv representation (not csv). You can either provide such json meta data files or simply put a 'format="csv"' into the read statements in bivar-stats.dml. Regards, Matthias On 10/20/2016 2:44 AM, Glenn Weidner wrote: > Hi Sandeep, > > Thank you for contacting the Apache SystemML community. Based on the > following trace fragment: > > spark.SparkContext: Running Spark version 2.0.0 > > it looks like you're running against Spark 2.0 which is not officially > supported with the current master code base. Can you retry with an earlier > Spark version? Let me know if I've misinterpreted your Spark environment. > > Cheers, > Glenn > > > > From: "Narayanaswami, Sandeep" > To:"dev@systemml.incubator.apache.org" > > Date: 10/19/2016 05:00 PM > Subject: Running the bivar-stats example > > > > Hi SystemML community, > > I’m trying to run the bivariate stats example script on Spark. I’m only > doing a toy example with very small data, but I have been running into > errors. > My script invocation is as follows: > > spark-submit --master local[4] SystemML.jar -f > scripts/algorithms/bivar-stats.dml -exec hybrid_spark -nvargs > X=/data/iqtv.csv index1=/data/index1.csv index2=/data/index2.csv > types1=/data/types1.csv types2=/data/types2.csv OUTDIR=/data/outdata > > Am I calling the script incorrectly? > (All the files are on hdfs. Iqtv.csv is a headerless csv with two numeric > fields. The index1 and index2 files are text files containing only the > strings ‘1’ and ‘2’ respectively. The types* files are text files each > containing only the string ‘1’.) > > Please excuse the rather long stack trace below, but I’m hoping it will > help diagnose the problem. > > Cheers, > Sandeep > > > > > 16/10/19 16:46:26 INFO api.DMLScript: BEGIN DML run 10/19/2016 16:46:26 > ANTLR Tool version 4.3 used for code generation does not match the current > runtime version 4.5.3[…] > 16/10/19 16:46:28 INFO spark.SparkContext: Running Spark version 2.0.0 > […] > 16/10/19 16:46:29 INFO api.DMLScript: END DML run 10/19/2016 16:46:29 > 16/10/19 16:46:29 ERROR api.DMLScript: Failed to execute DML script. > org.apache.sysml.runtime.DMLRuntimeException: > org.apache.sysml.runtime.DMLRuntimeException: > org.apache.sysml.runtime.DMLRuntimeException: Unable to recompile program > block hierarchy. >at org.apache.sysml.runtime.controlprogram.Program.execute > (Program.java:152) >at org.apache.sysml.api.DMLScript.execute(DMLScript.java:677) >at org.apache.sysml.api.DMLScript.executeScript (DMLScript.java:337) >at org.apache.sysml.api.DMLScript.main(DMLScript.java:192) >at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >at sun.reflect.NativeMethodAccessorImpl.invoke > (NativeMethodAccessorImpl.java:62) >at sun.reflect.DelegatingMethodAccessorImpl.invoke > (DelegatingMethodAccessorImpl.java:43) >at java.lang.reflect.Method.invoke(Method.java:497) >at org.apache.spark.deploy.SparkSubmit$.org$apache$spark $deploy > $SparkSubmit$$runMain(SparkSubmit.scala:729) >at org.apache.spark.deploy.
Re: Running the bivar-stats example
Hi, sorry I couldn’t get back on this sooner. @Glenn: Thanks for pointing that out. Unfortunately, switching to Spark 1.6 didn’t make any difference: I got the same error/stacktrace again. @Matthias: I did have the corresponding metadata files. I also tried modifying the script to print the matrix after reading it in, and that works as expected so I don’t think it’s a data read issue. I’d appreciate any other suggestions. Cheers, Sandeep On 10/20/16, 02:30, "Matthias Boehm" wrote: apart from the missing support for Spark 2.x, I would recommend to double check your inputs. Do you have meta data files along with your inputs? If not, then these inputs are assumed to be in default format "text", i.e., matrix market ijv representation (not csv). You can either provide such json meta data files or simply put a 'format="csv"' into the read statements in bivar-stats.dml. Regards, Matthias On 10/20/2016 2:44 AM, Glenn Weidner wrote: > Hi Sandeep, > > Thank you for contacting the Apache SystemML community. Based on the > following trace fragment: > > spark.SparkContext: Running Spark version 2.0.0 > > it looks like you're running against Spark 2.0 which is not officially > supported with the current master code base. Can you retry with an earlier > Spark version? Let me know if I've misinterpreted your Spark environment. > > Cheers, > Glenn > > > > From: "Narayanaswami, Sandeep" > To: "dev@systemml.incubator.apache.org" > > Date: 10/19/2016 05:00 PM > Subject: Running the bivar-stats example > > > > Hi SystemML community, > > I’m trying to run the bivariate stats example script on Spark. I’m only > doing a toy example with very small data, but I have been running into > errors. > My script invocation is as follows: > > spark-submit --master local[4] SystemML.jar -f > scripts/algorithms/bivar-stats.dml -exec hybrid_spark -nvargs > X=/data/iqtv.csv index1=/data/index1.csv index2=/data/index2.csv > types1=/data/types1.csv types2=/data/types2.csv OUTDIR=/data/outdata > > Am I calling the script incorrectly? > (All the files are on hdfs. Iqtv.csv is a headerless csv with two numeric > fields. The index1 and index2 files are text files containing only the > strings ‘1’ and ‘2’ respectively. The types* files are text files each > containing only the string ‘1’.) > > Please excuse the rather long stack trace below, but I’m hoping it will > help diagnose the problem. > > Cheers, > Sandeep > > > > > 16/10/19 16:46:26 INFO api.DMLScript: BEGIN DML run 10/19/2016 16:46:26 > ANTLR Tool version 4.3 used for code generation does not match the current > runtime version 4.5.3[…] > 16/10/19 16:46:28 INFO spark.SparkContext: Running Spark version 2.0.0 > […] > 16/10/19 16:46:29 INFO api.DMLScript: END DML run 10/19/2016 16:46:29 > 16/10/19 16:46:29 ERROR api.DMLScript: Failed to execute DML script. > org.apache.sysml.runtime.DMLRuntimeException: > org.apache.sysml.runtime.DMLRuntimeException: > org.apache.sysml.runtime.DMLRuntimeException: Unable to recompile program > block hierarchy. >at org.apache.sysml.runtime.controlprogram.Program.execute > (Program.java:152) >at org.apache.sysml.api.DMLScript.execute(DMLScript.java:677) >at org.apache.sysml.api.DMLScript.executeScript(DMLScript.java:337) >at org.apache.sysml.api.DMLScript.main(DMLScript.java:192) >at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >at sun.reflect.NativeMethodAccessorImpl.invoke > (NativeMethodAccessorImpl.java:62) >at sun.reflect.DelegatingMethodAccessorImpl.invoke > (DelegatingMethodAccessorImpl.java:43) >at java.lang.reflect.Method.invoke(Method.java:497) >at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy > $SparkSubmit$$runMain(SparkSubmit.scala:729) >at org.apache.spark.deploy.SparkSubmit$.doRunMain$1 > (SparkSubmit.scala:185) >at org.apache.spark.deploy.SparkSubmit$.submit > (SparkSubmit.scala:210) >at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:124) >at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) > Caused by: org.apache.sysml.runtime.DMLRuntimeException: > org.apache.sysml.runtime.DMLRuntimeException: Unable to recompile program > block hierarchy. >at > org.apache.sysml.runtime.controlprogram.parfor.opt.OptimizationWrapper.optimize > (OptimizationWrapper.java:299) >at > org.apache.sysml.runtime.control
Re: Running the bivar-stats example
apart from the missing support for Spark 2.x, I would recommend to double check your inputs. Do you have meta data files along with your inputs? If not, then these inputs are assumed to be in default format "text", i.e., matrix market ijv representation (not csv). You can either provide such json meta data files or simply put a 'format="csv"' into the read statements in bivar-stats.dml. Regards, Matthias On 10/20/2016 2:44 AM, Glenn Weidner wrote: Hi Sandeep, Thank you for contacting the Apache SystemML community. Based on the following trace fragment: spark.SparkContext: Running Spark version 2.0.0 it looks like you're running against Spark 2.0 which is not officially supported with the current master code base. Can you retry with an earlier Spark version? Let me know if I've misinterpreted your Spark environment. Cheers, Glenn From: "Narayanaswami, Sandeep" To: "dev@systemml.incubator.apache.org" Date: 10/19/2016 05:00 PM Subject:Running the bivar-stats example Hi SystemML community, I’m trying to run the bivariate stats example script on Spark. I’m only doing a toy example with very small data, but I have been running into errors. My script invocation is as follows: spark-submit --master local[4] SystemML.jar -f scripts/algorithms/bivar-stats.dml -exec hybrid_spark -nvargs X=/data/iqtv.csv index1=/data/index1.csv index2=/data/index2.csv types1=/data/types1.csv types2=/data/types2.csv OUTDIR=/data/outdata Am I calling the script incorrectly? (All the files are on hdfs. Iqtv.csv is a headerless csv with two numeric fields. The index1 and index2 files are text files containing only the strings ‘1’ and ‘2’ respectively. The types* files are text files each containing only the string ‘1’.) Please excuse the rather long stack trace below, but I’m hoping it will help diagnose the problem. Cheers, Sandeep 16/10/19 16:46:26 INFO api.DMLScript: BEGIN DML run 10/19/2016 16:46:26 ANTLR Tool version 4.3 used for code generation does not match the current runtime version 4.5.3[…] 16/10/19 16:46:28 INFO spark.SparkContext: Running Spark version 2.0.0 […] 16/10/19 16:46:29 INFO api.DMLScript: END DML run 10/19/2016 16:46:29 16/10/19 16:46:29 ERROR api.DMLScript: Failed to execute DML script. org.apache.sysml.runtime.DMLRuntimeException: org.apache.sysml.runtime.DMLRuntimeException: org.apache.sysml.runtime.DMLRuntimeException: Unable to recompile program block hierarchy. at org.apache.sysml.runtime.controlprogram.Program.execute (Program.java:152) at org.apache.sysml.api.DMLScript.execute(DMLScript.java:677) at org.apache.sysml.api.DMLScript.executeScript(DMLScript.java:337) at org.apache.sysml.api.DMLScript.main(DMLScript.java:192) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy $SparkSubmit$$runMain(SparkSubmit.scala:729) at org.apache.spark.deploy.SparkSubmit$.doRunMain$1 (SparkSubmit.scala:185) at org.apache.spark.deploy.SparkSubmit$.submit (SparkSubmit.scala:210) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:124) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) Caused by: org.apache.sysml.runtime.DMLRuntimeException: org.apache.sysml.runtime.DMLRuntimeException: Unable to recompile program block hierarchy. at org.apache.sysml.runtime.controlprogram.parfor.opt.OptimizationWrapper.optimize (OptimizationWrapper.java:299) at org.apache.sysml.runtime.controlprogram.parfor.opt.OptimizationWrapper.optimize (OptimizationWrapper.java:172) at org.apache.sysml.runtime.controlprogram.ParForProgramBlock.execute (ParForProgramBlock.java:550) at org.apache.sysml.runtime.controlprogram.Program.execute (Program.java:145) ... 12 more Caused by: org.apache.sysml.runtime.DMLRuntimeException: Unable to recompile program block hierarchy. at org.apache.sysml.hops.recompile.Recompiler.recompileProgramBlockHierarchy (Recompiler.java:541) at org.apache.sysml.runtime.controlprogram.parfor.opt.OptimizationWrapper.optimize (OptimizationWrapper.java:278) ... 15 more Caused by: org.apache.sysml.hops.HopsException: ERROR: line 342, column 4 -- error constructing Lops for UnaryOp Hop -- at org.apache.sysml.hops.UnaryOp.constructLops(UnaryOp.java:177) at org.apache.sysml.hops.DataOp.constructLops(DataOp.java:252) at org.apache.sysml.hops.recompile.Recompiler.recompileHopsDag (Recompiler.java:209) at org.apache.sysml.hops.recompile.Recompiler.rRecompileProgramBl
Re: Running the bivar-stats example
Hi Sandeep, Thank you for contacting the Apache SystemML community. Based on the following trace fragment: spark.SparkContext: Running Spark version 2.0.0 it looks like you're running against Spark 2.0 which is not officially supported with the current master code base. Can you retry with an earlier Spark version? Let me know if I've misinterpreted your Spark environment. Cheers, Glenn From: "Narayanaswami, Sandeep" To: "dev@systemml.incubator.apache.org" Date: 10/19/2016 05:00 PM Subject:Running the bivar-stats example Hi SystemML community, I’m trying to run the bivariate stats example script on Spark. I’m only doing a toy example with very small data, but I have been running into errors. My script invocation is as follows: spark-submit --master local[4] SystemML.jar -f scripts/algorithms/bivar-stats.dml -exec hybrid_spark -nvargs X=/data/iqtv.csv index1=/data/index1.csv index2=/data/index2.csv types1=/data/types1.csv types2=/data/types2.csv OUTDIR=/data/outdata Am I calling the script incorrectly? (All the files are on hdfs. Iqtv.csv is a headerless csv with two numeric fields. The index1 and index2 files are text files containing only the strings ‘1’ and ‘2’ respectively. The types* files are text files each containing only the string ‘1’.) Please excuse the rather long stack trace below, but I’m hoping it will help diagnose the problem. Cheers, Sandeep 16/10/19 16:46:26 INFO api.DMLScript: BEGIN DML run 10/19/2016 16:46:26 ANTLR Tool version 4.3 used for code generation does not match the current runtime version 4.5.3[…] 16/10/19 16:46:28 INFO spark.SparkContext: Running Spark version 2.0.0 […] 16/10/19 16:46:29 INFO api.DMLScript: END DML run 10/19/2016 16:46:29 16/10/19 16:46:29 ERROR api.DMLScript: Failed to execute DML script. org.apache.sysml.runtime.DMLRuntimeException: org.apache.sysml.runtime.DMLRuntimeException: org.apache.sysml.runtime.DMLRuntimeException: Unable to recompile program block hierarchy. at org.apache.sysml.runtime.controlprogram.Program.execute (Program.java:152) at org.apache.sysml.api.DMLScript.execute(DMLScript.java:677) at org.apache.sysml.api.DMLScript.executeScript(DMLScript.java:337) at org.apache.sysml.api.DMLScript.main(DMLScript.java:192) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy $SparkSubmit$$runMain(SparkSubmit.scala:729) at org.apache.spark.deploy.SparkSubmit$.doRunMain$1 (SparkSubmit.scala:185) at org.apache.spark.deploy.SparkSubmit$.submit (SparkSubmit.scala:210) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:124) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) Caused by: org.apache.sysml.runtime.DMLRuntimeException: org.apache.sysml.runtime.DMLRuntimeException: Unable to recompile program block hierarchy. at org.apache.sysml.runtime.controlprogram.parfor.opt.OptimizationWrapper.optimize (OptimizationWrapper.java:299) at org.apache.sysml.runtime.controlprogram.parfor.opt.OptimizationWrapper.optimize (OptimizationWrapper.java:172) at org.apache.sysml.runtime.controlprogram.ParForProgramBlock.execute (ParForProgramBlock.java:550) at org.apache.sysml.runtime.controlprogram.Program.execute (Program.java:145) ... 12 more Caused by: org.apache.sysml.runtime.DMLRuntimeException: Unable to recompile program block hierarchy. at org.apache.sysml.hops.recompile.Recompiler.recompileProgramBlockHierarchy (Recompiler.java:541) at org.apache.sysml.runtime.controlprogram.parfor.opt.OptimizationWrapper.optimize (OptimizationWrapper.java:278) ... 15 more Caused by: org.apache.sysml.hops.HopsException: ERROR: line 342, column 4 -- error constructing Lops for UnaryOp Hop -- at org.apache.sysml.hops.UnaryOp.constructLops(UnaryOp.java:177) at org.apache.sysml.hops.DataOp.constructLops(DataOp.java:252) at org.apache.sysml.hops.recompile.Recompiler.recompileHopsDag (Recompiler.java:209) at org.apache.sysml.hops.recompile.Recompiler.rRecompileProgramBlock (Recompiler.java:881) at org.apache.sysml.hops.recompile.Recompiler.rRecompileProgramBlock (Recompiler.java:839) at org.apache.sysml.hops.recompile.Recompiler.rRecompileProgramBlock (Recompiler.java:839) at org.apache.sysml.hops.recompile.Recompiler.rRecompileProgramBlock (Recompiler.java:841) at org.apache.sysml.hops.recompile.Recompiler.rRecompileProgramBlock (Recompiler.java:858) at org.apache.sysml.hops.recompile.Re