Dear Jack,
As is known, Breeze is numerical calculation package wrote by scala , spark
mllib also use it as underlying package for algebra usage.Here I am also
preparing to use Breeze for nonlinear equation optimization, however, it seemed
that I could not find the exact doc or API for Breeze except spark linalg
package...
Could you help some to provide me the official doc or API website for Breeze
?Thank you in advance!
Zhiliang
On Thursday, November 19, 2015 7:32 AM, Jack Yang <j...@uow.edu.au> wrote:
<!--#yiv0801195225 _filtered #yiv0801195225 {font-family:SimSun;panose-1:2 1
6 0 3 1 1 1 1 1;} _filtered #yiv0801195225 {font-family:SimSun;panose-1:2 1 6 0
3 1 1 1 1 1;} _filtered #yiv0801195225 {font-family:Calibri;panose-1:2 15 5 2 2
2 4 3 2 4;} _filtered #yiv0801195225 {font-family:Tahoma;panose-1:2 11 6 4 3 5
4 4 2 4;} _filtered #yiv0801195225 {panose-1:2 1 6 0 3 1 1 1 1
1;}#yiv0801195225 #yiv0801195225 p.yiv0801195225MsoNormal, #yiv0801195225
li.yiv0801195225MsoNormal, #yiv0801195225 div.yiv0801195225MsoNormal
{margin:0cm;margin-bottom:.0001pt;font-size:12.0pt;font-family:"Times New
Roman", "serif";}#yiv0801195225 a:link, #yiv0801195225
span.yiv0801195225MsoHyperlink
{color:blue;text-decoration:underline;}#yiv0801195225 a:visited, #yiv0801195225
span.yiv0801195225MsoHyperlinkFollowed
{color:purple;text-decoration:underline;}#yiv0801195225
p.yiv0801195225MsoAcetate, #yiv0801195225 li.yiv0801195225MsoAcetate,
#yiv0801195225 div.yiv0801195225MsoAcetate
{margin:0cm;margin-bottom:.0001pt;font-size:8.0pt;font-family:"Tahoma",
"sans-serif";}#yiv0801195225 span.yiv0801195225apple-converted-space
{}#yiv0801195225 span.yiv0801195225EmailStyle18 {font-family:"Calibri",
"sans-serif";color:#1F497D;}#yiv0801195225 span.yiv0801195225BalloonTextChar
{font-family:"Tahoma", "sans-serif";}#yiv0801195225 .yiv0801195225MsoChpDefault
{font-size:10.0pt;} _filtered #yiv0801195225 {margin:72.0pt 72.0pt 72.0pt
72.0pt;}#yiv0801195225 div.yiv0801195225WordSection1 {}-->If I tried to change
“provided” to “compile”.. then the error changed to : Exception in thread
"main" java.lang.IncompatibleClassChangeError: Implementing class at
java.lang.ClassLoader.defineClass1(Native Method) at
java.lang.ClassLoader.defineClass(ClassLoader.java:800) at
java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:449) at
java.net.URLClassLoader.access$100(URLClassLoader.java:71) at
java.net.URLClassLoader$1.run(URLClassLoader.java:361) at
java.net.URLClassLoader$1.run(URLClassLoader.java:355) at
java.security.AccessController.doPrivileged(Native Method) at
java.net.URLClassLoader.findClass(URLClassLoader.java:354) at
java.lang.ClassLoader.loadClass(ClassLoader.java:425) at
java.lang.ClassLoader.loadClass(ClassLoader.java:358)
atsmartapp.smart.sparkwithscala.textMingApp.main(textMingApp.scala) at
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606) at
org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:664)
at
org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:169)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:192)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:111) at
org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) 15/11/19 10:28:29
INFO util.Utils: Shutdown hook called Meanwhile, I will prefer to use maven
to compile the jar file rather than sbt, although it is indeed another option.
Best regards, Jack From: Fengdong Yu
[mailto:fengdo...@everstring.com]
Sent: Wednesday, 18 November 2015 7:30 PM
To: Jack Yang
Cc: Ted Yu; user@spark.apache.org
Subject: Re: spark with breeze error of NoClassDefFoundError The simplest
way is remove all “provided” in your pom. then ‘sbt assembly” to build your
final package. then get rid of ‘—jars’ because assembly already includes all
dependencies.
On Nov 18, 2015, at 2:15 PM, Jack Yang <j...@uow.edu.au> wrote: So weird. Is
there anything wrong with the way I made the pom file (I labelled them as
<scope>provided</scope>)? Is there missing jar I forget to add in “--jar”?
See the trace below: Exception in thread "main"
java.lang.NoClassDefFoundError: breeze/storage/DefaultArrayValue at
smartapp.smart.sparkwithscala.textMingApp.main(textMingApp.scala) at
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606) at
org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:664)
at
org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:169)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:192)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:111) at
org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) Caused by:
java.lang.ClassNotFoundException: breeze.storage.DefaultArrayValue at
java.net.URLClassLoader$1.run(URLClassLoader.java:366) at
java.net.URLClassLoader$1.run(URLClassLoader.java:355) at
java.security.AccessController.doPrivileged(Native Method) at
java.net.URLClassLoader.findClass(URLClassLoader.java:354) at
java.lang.ClassLoader.loadClass(ClassLoader.java:425) at
java.lang.ClassLoader.loadClass(ClassLoader.java:358) ... 10 more
15/11/18 17:15:15 INFO util.Utils: Shutdown hook called From: Ted Yu
[mailto:yuzhih...@gmail.com]
Sent: Wednesday, 18 November 2015 4:01 PM
To: Jack Yang
Cc: user@spark.apache.org
Subject: Re: spark with breeze error of NoClassDefFoundError Looking in local
maven repo, breeze_2.10-0.7.jar contains DefaultArrayValue : jar tvf
/Users/tyu/.m2/repository//org/scalanlp/breeze_2.10/0.7/breeze_2.10-0.7.jar |
grep !$ jar tvf
/Users/tyu/.m2/repository//org/scalanlp/breeze_2.10/0.7/breeze_2.10-0.7.jar |
grep DefaultArrayValue 369 Wed Mar 19 11:18:32 PDT 2014
breeze/storage/DefaultArrayValue$mcZ$sp$class.class 309 Wed Mar 19 11:18:32
PDT 2014 breeze/storage/DefaultArrayValue$mcJ$sp.class 2233 Wed Mar 19
11:18:32 PDT 2014
breeze/storage/DefaultArrayValue$DoubleDefaultArrayValue$.class Can you show
the complete stack trace ? FYI On Tue, Nov 17, 2015 at 8:33 PM, Jack Yang
<j...@uow.edu.au> wrote: Hi all, I am using spark 1.4.0, and building my codes
using maven. So in one of my scala, I used: import breeze.linalg._ val v1 =
new breeze.linalg.SparseVector(commonVector.indices, commonVector.values,
commonVector.size) val v2 = new
breeze.linalg.SparseVector(commonVector2.indices, commonVector2.values,
commonVector2.size) println (v1.dot(v2) / (norm(v1) * norm(v2)) ) in my
pom.xml file, I used: <dependency>
<groupId>org.scalanlp</groupId>
<artifactId>breeze-math_2.10</artifactId>
<version>0.4</version>
<scope>provided</scope> </dependency>
<dependency>
<groupId>org.scalanlp</groupId>
<artifactId>breeze_2.10</artifactId>
<version>0.11.2</version>
<scope>provided</scope> </dependency> When
submit, I included breeze jars (breeze_2.10-0.11.2.jar breeze-math_2.10-0.4.jar
breeze-natives_2.10-0.11.2.jar breeze-process_2.10-0.3.jar) using “--jar”
arguments, although I doubt it is necessary to do that. however, the error is
Exception in thread "main" java.lang.NoClassDefFoundError:
breeze/storage/DefaultArrayValue Any thoughts? Best regards, Jack
