Re: Classpath errors with Breeze
Thanks Xiangrui, that did the trick. Dieterich On Jun 8, 2014, at 10:17 PM, Xiangrui Meng [via Apache Spark User List] ml-node+s1001560n7226...@n3.nabble.com wrote: Hi dlaw, You are using breeze-0.8.1, but the spark assembly jar depends on breeze-0.7. If the spark assembly jar comes the first on the classpath but the method from DenseMatrix is only available in breeze-0.8.1, you get NoSuchMethod. So, a) If you don't need the features in breeze-0.8.1, do not include it as a dependency. or b) Try an experimental features by turning on spark.files.userClassPathFirst in your Spark configuration. Best, Xiangrui On Sun, Jun 8, 2014 at 10:08 PM, dlaw [hidden email] wrote: Thanks for the quick response. No, I actually build my jar via 'sbt package' on EC2 on the master itself. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Classpath-errors-with-Breeze-tp7220p7225.html Sent from the Apache Spark User List mailing list archive at Nabble.com. If you reply to this email, your message will be added to the discussion below: http://apache-spark-user-list.1001560.n3.nabble.com/Classpath-errors-with-Breeze-tp7220p7226.html To unsubscribe from Classpath errors with Breeze, click here. NAML -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Classpath-errors-with-Breeze-tp7220p7232.html Sent from the Apache Spark User List mailing list archive at Nabble.com.
Classpath errors with Breeze
I'm having some trouble getting a basic matrix multiply to work with Breeze. I'm pretty sure it's related to my classpath. My setup is a cluster on AWS with 8 m3.xlarges. To create the cluster I used the provided ec2 scripts and Spark 1.0.0. I've made a gist with the relevant pieces of my app: https://gist.github.com/dieterichlawson/e5e3ab158a09429706e0 The app was created as detailed in the quick start guide. When I run it I get an error that says the method to multiply a dense matrix by a dense matrix does not exist: 14/06/09 04:49:09 WARN scheduler.TaskSetManager: Lost TID 90 (task 0.0:13) 14/06/09 04:49:09 INFO scheduler.TaskSetManager: Loss was due to java.lang.NoSuchMethodError: breeze.linalg.DenseMatrix$.implOpMulMatrix_DMD_DMD_eq_DMD()Lbreeze/linalg/operators/DenseMatrixMultiplyStuff$implOpMulMatrix_DMD_DMD_eq_DMD$; [duplicate 46] I've tried a bunch of different things, including playing with the CLASSPATH and ADD_JARS environment variables, the --jars option on spark-submit, the version of breeze and scala, etc... I've also tried it in the spark-shell. It works there, so I don't really know what's going on. Any thoughts? -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Classpath-errors-with-Breeze-tp7220.html Sent from the Apache Spark User List mailing list archive at Nabble.com.
Re: Classpath errors with Breeze
Hi, I had a similar problem; I was using `sbt assembly` to build a jar containing all my dependencies, but since my file system has a problem with long file names (due to disk encryption), some class files (which correspond to functions in Scala) where not included in the jar I uploaded. Although, thinking about it, that would result in a ClassNotFound exception, not NoSuchMethod. Have you built your code against a different version of the library than the jar you use in EC2? Tobias On Mon, Jun 9, 2014 at 1:52 PM, dlaw dieterich.law...@gmail.com wrote: I'm having some trouble getting a basic matrix multiply to work with Breeze. I'm pretty sure it's related to my classpath. My setup is a cluster on AWS with 8 m3.xlarges. To create the cluster I used the provided ec2 scripts and Spark 1.0.0. I've made a gist with the relevant pieces of my app: https://gist.github.com/dieterichlawson/e5e3ab158a09429706e0 The app was created as detailed in the quick start guide. When I run it I get an error that says the method to multiply a dense matrix by a dense matrix does not exist: 14/06/09 04:49:09 WARN scheduler.TaskSetManager: Lost TID 90 (task 0.0:13) 14/06/09 04:49:09 INFO scheduler.TaskSetManager: Loss was due to java.lang.NoSuchMethodError: breeze.linalg.DenseMatrix$.implOpMulMatrix_DMD_DMD_eq_DMD()Lbreeze/linalg/operators/DenseMatrixMultiplyStuff$implOpMulMatrix_DMD_DMD_eq_DMD$; [duplicate 46] I've tried a bunch of different things, including playing with the CLASSPATH and ADD_JARS environment variables, the --jars option on spark-submit, the version of breeze and scala, etc... I've also tried it in the spark-shell. It works there, so I don't really know what's going on. Any thoughts? -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Classpath-errors-with-Breeze-tp7220.html Sent from the Apache Spark User List mailing list archive at Nabble.com.
Re: Classpath errors with Breeze
Thanks for the quick response. No, I actually build my jar via 'sbt package' on EC2 on the master itself. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Classpath-errors-with-Breeze-tp7220p7225.html Sent from the Apache Spark User List mailing list archive at Nabble.com.
Re: Classpath errors with Breeze
Hi dlaw, You are using breeze-0.8.1, but the spark assembly jar depends on breeze-0.7. If the spark assembly jar comes the first on the classpath but the method from DenseMatrix is only available in breeze-0.8.1, you get NoSuchMethod. So, a) If you don't need the features in breeze-0.8.1, do not include it as a dependency. or b) Try an experimental features by turning on spark.files.userClassPathFirst in your Spark configuration. Best, Xiangrui On Sun, Jun 8, 2014 at 10:08 PM, dlaw dieterich.law...@gmail.com wrote: Thanks for the quick response. No, I actually build my jar via 'sbt package' on EC2 on the master itself. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Classpath-errors-with-Breeze-tp7220p7225.html Sent from the Apache Spark User List mailing list archive at Nabble.com.
Re: Classpath errors with Breeze
Hi Tobias, Which file system and which encryption are you using? Best, Xiangrui On Sun, Jun 8, 2014 at 10:16 PM, Xiangrui Meng men...@gmail.com wrote: Hi dlaw, You are using breeze-0.8.1, but the spark assembly jar depends on breeze-0.7. If the spark assembly jar comes the first on the classpath but the method from DenseMatrix is only available in breeze-0.8.1, you get NoSuchMethod. So, a) If you don't need the features in breeze-0.8.1, do not include it as a dependency. or b) Try an experimental features by turning on spark.files.userClassPathFirst in your Spark configuration. Best, Xiangrui On Sun, Jun 8, 2014 at 10:08 PM, dlaw dieterich.law...@gmail.com wrote: Thanks for the quick response. No, I actually build my jar via 'sbt package' on EC2 on the master itself. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Classpath-errors-with-Breeze-tp7220p7225.html Sent from the Apache Spark User List mailing list archive at Nabble.com.
