Re: spark with breeze error of NoClassDefFoundError
I don't have Spark 1.4 source code on hand. You can use the following command: mvn dependency:tree to find out the answer to your question. Cheers On Wed, Nov 18, 2015 at 10:18 PM, Jack Yang <j...@uow.edu.au> wrote: > Back to my question. If I use “*provided*”, the jar file > will expect some libraries are provided by the system. > > However, the “ *compiled *” is the default setting, which > means the third-party library will be included inside jar file after > compiling. > > So when I use “*provided*”, the error is they cannot find > the Class, but with “compiled” the error is IncompatibleClassChangeError. > > > > Ok, so can someone tell me which version of breeze and breeze-math are > used in spark 1.4? > > > > *From:* Zhiliang Zhu [mailto:zchl.j...@yahoo.com] > *Sent:* Thursday, 19 November 2015 5:10 PM > *To:* Ted Yu > *Cc:* Jack Yang; Fengdong Yu; user@spark.apache.org > > *Subject:* Re: spark with breeze error of NoClassDefFoundError > > > > Dear Ted, > > I just looked at the link you provided, it is great! > > > > For my understanding, I could also directly use other Breeze part (except > spark mllib package linalg ) in spark (scala or java ) program after > importing Breeze package, > > it is right? > > > > Thanks a lot in advance again! > > Zhiliang > > > > > > > > On Thursday, November 19, 2015 1:46 PM, Ted Yu <yuzhih...@gmail.com> > wrote: > > > > Have you looked at > > https://github.com/scalanlp/breeze/wiki > > > > Cheers > > > On Nov 18, 2015, at 9:34 PM, Zhiliang Zhu <zchl.j...@yahoo.com> wrote: > > Dear Jack, > > > > As is known, Breeze is numerical calculation package wrote by scala , > spark mllib also use it as underlying package for algebra usage. > > Here I am also preparing to use Breeze for nonlinear equation > optimization, however, it seemed that I could not find the exact doc or API > for Breeze except spark linalg package... > > > > Could you help some to provide me the official doc or API website for > Breeze ? > > Thank you in advance! > > > > Zhiliang > > > > > > > > On Thursday, November 19, 2015 7:32 AM, Jack Yang <j...@uow.edu.au> wrote: > > > > If I tried to change “provided” to “compile”.. then the error changed to : > > > > Exception in thread "main" java.lang.IncompatibleClassChangeError: > Implementing class > > at java.lang.ClassLoader.defineClass1(Native Method) > > at java.lang.ClassLoader.defineClass(ClassLoader.java:800) > > at > java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) > > at java.net.URLClassLoader.defineClass(URLClassLoader.java:449) > > at java.net.URLClassLoader.access$100(URLClassLoader.java:71) > > at java.net.URLClassLoader$1.run(URLClassLoader.java:361) > > at java.net.URLClassLoader$1.run(URLClassLoader.java:355) > > at java.security.AccessController.doPrivileged(Native Method) > > at java.net.URLClassLoader.findClass(URLClassLoader.java:354) > > at java.lang.ClassLoader.loadClass(ClassLoader.java:425) > > at java.lang.ClassLoader.loadClass(ClassLoader.java:358) > > at > smartapp.smart.sparkwithscala.textMingApp.main(textMingApp.scala) > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > at java.lang.reflect.Method.invoke(Method.java:606) > > at > org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:664) > > at > org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:169) > > at > org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:192) > > at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:111) > > at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) > > 15/11/19 10:28:29 INFO util.Utils: Shutdown hook called > > > > Meanwhile, I will prefer to use maven to compile the jar file rather than > sbt, although it is indeed another option. > > > > Best regards, > > Jack > > > > > > > > *From:* Fengdong Yu [mailto:fengdo...@everstring.com > <fengdo...@everstring.com>] > *Sent:* Wednesday, 18 November 2015 7:30 PM > *To:* Jack Yang > *Cc:* Ted Yu; user@spark.apache.org > *Subject:* Re
Re: spark with breeze error of NoClassDefFoundError
The simplest way is remove all “provided” in your pom. then ‘sbt assembly” to build your final package. then get rid of ‘—jars’ because assembly already includes all dependencies. > On Nov 18, 2015, at 2:15 PM, Jack Yang <j...@uow.edu.au> wrote: > > So weird. Is there anything wrong with the way I made the pom file (I > labelled them as provided)? > > Is there missing jar I forget to add in “--jar”? > > See the trace below: > > > > Exception in thread "main" java.lang.NoClassDefFoundError: > breeze/storage/DefaultArrayValue > at smartapp.smart.sparkwithscala.textMingApp.main(textMingApp.scala) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:664) > at > org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:169) > at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:192) > at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:111) > at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) > Caused by: java.lang.ClassNotFoundException: breeze.storage.DefaultArrayValue > at java.net.URLClassLoader$1.run(URLClassLoader.java:366) > at java.net.URLClassLoader$1.run(URLClassLoader.java:355) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:354) > at java.lang.ClassLoader.loadClass(ClassLoader.java:425) > at java.lang.ClassLoader.loadClass(ClassLoader.java:358) > ... 10 more > 15/11/18 17:15:15 INFO util.Utils: Shutdown hook called > > > From: Ted Yu [mailto:yuzhih...@gmail.com] > Sent: Wednesday, 18 November 2015 4:01 PM > To: Jack Yang > Cc: user@spark.apache.org > Subject: Re: spark with breeze error of NoClassDefFoundError > > Looking in local maven repo, breeze_2.10-0.7.jar contains DefaultArrayValue : > > jar tvf > /Users/tyu/.m2/repository//org/scalanlp/breeze_2.10/0.7/breeze_2.10-0.7.jar | > grep !$ > jar tvf > /Users/tyu/.m2/repository//org/scalanlp/breeze_2.10/0.7/breeze_2.10-0.7.jar | > grep DefaultArrayValue >369 Wed Mar 19 11:18:32 PDT 2014 > breeze/storage/DefaultArrayValue$mcZ$sp$class.class >309 Wed Mar 19 11:18:32 PDT 2014 > breeze/storage/DefaultArrayValue$mcJ$sp.class > 2233 Wed Mar 19 11:18:32 PDT 2014 > breeze/storage/DefaultArrayValue$DoubleDefaultArrayValue$.class > > Can you show the complete stack trace ? > > FYI > > On Tue, Nov 17, 2015 at 8:33 PM, Jack Yang <j...@uow.edu.au > <mailto:j...@uow.edu.au>> wrote: > Hi all, > I am using spark 1.4.0, and building my codes using maven. > So in one of my scala, I used: > > import breeze.linalg._ > val v1 = new breeze.linalg.SparseVector(commonVector.indices, > commonVector.values, commonVector.size) > val v2 = new breeze.linalg.SparseVector(commonVector2.indices, > commonVector2.values, commonVector2.size) > println (v1.dot(v2) / (norm(v1) * norm(v2)) ) > > > > in my pom.xml file, I used: > > org.scalanlp > > breeze-math_2.10 > 0.4 > provided > > > > org.scalanlp > > breeze_2.10 > 0.11.2 > provided > > > > When submit, I included breeze jars (breeze_2.10-0.11.2.jar > breeze-math_2.10-0.4.jar breeze-natives_2.10-0.11.2.jar > breeze-process_2.10-0.3.jar) using “--jar” arguments, although I doubt it is > necessary to do that. > > however, the error is > > Exception in thread "main" java.lang.NoClassDefFoundError: > breeze/storage/DefaultArrayValue > > Any thoughts? > > > > Best regards, > Jack
Re: spark with breeze error of NoClassDefFoundError
Have you tried to change to scope to `compile` ? 2015-11-18 14:15 GMT+08:00 Jack Yang <j...@uow.edu.au>: > So weird. Is there anything wrong with the way I made the pom file (I > labelled them as *provided*)? > > > > Is there missing jar I forget to add in “--jar”? > > > > See the trace below: > > > > > > > > Exception in thread "main" java.lang.NoClassDefFoundError: > breeze/storage/DefaultArrayValue > > at > smartapp.smart.sparkwithscala.textMingApp.main(textMingApp.scala) > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > at java.lang.reflect.Method.invoke(Method.java:606) > > at > org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:664) > > at > org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:169) > > at > org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:192) > > at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:111) > > at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) > > Caused by: java.lang.ClassNotFoundException: > breeze.storage.DefaultArrayValue > > at java.net.URLClassLoader$1.run(URLClassLoader.java:366) > > at java.net.URLClassLoader$1.run(URLClassLoader.java:355) > > at java.security.AccessController.doPrivileged(Native Method) > > at java.net.URLClassLoader.findClass(URLClassLoader.java:354) > > at java.lang.ClassLoader.loadClass(ClassLoader.java:425) > > at java.lang.ClassLoader.loadClass(ClassLoader.java:358) > > ... 10 more > > 15/11/18 17:15:15 INFO util.Utils: Shutdown hook called > > > > > > *From:* Ted Yu [mailto:yuzhih...@gmail.com] > *Sent:* Wednesday, 18 November 2015 4:01 PM > *To:* Jack Yang > *Cc:* user@spark.apache.org > *Subject:* Re: spark with breeze error of NoClassDefFoundError > > > > Looking in local maven repo, breeze_2.10-0.7.jar > contains DefaultArrayValue : > > > > jar tvf > /Users/tyu/.m2/repository//org/scalanlp/breeze_2.10/0.7/breeze_2.10-0.7.jar > | grep !$ > > jar tvf > /Users/tyu/.m2/repository//org/scalanlp/breeze_2.10/0.7/breeze_2.10-0.7.jar > | grep DefaultArrayValue > >369 Wed Mar 19 11:18:32 PDT 2014 > breeze/storage/DefaultArrayValue$mcZ$sp$class.class > >309 Wed Mar 19 11:18:32 PDT 2014 > breeze/storage/DefaultArrayValue$mcJ$sp.class > > 2233 Wed Mar 19 11:18:32 PDT 2014 > breeze/storage/DefaultArrayValue$DoubleDefaultArrayValue$.class > > > > Can you show the complete stack trace ? > > > > FYI > > > > On Tue, Nov 17, 2015 at 8:33 PM, Jack Yang <j...@uow.edu.au> wrote: > > Hi all, > > I am using spark 1.4.0, and building my codes using maven. > > So in one of my scala, I used: > > > > import breeze.linalg._ > > val v1 = new breeze.linalg.SparseVector(commonVector.indices, > commonVector.values, commonVector.size) > > val v2 = new breeze.linalg.SparseVector(commonVector2.indices, > commonVector2.values, commonVector2.size) > > println (v1.dot(v2) / (norm(v1) * norm(v2)) ) > > > > > > > > in my pom.xml file, I used: > > > > > org.scalanlp > > > breeze-math_2.10 > > 0.4 > > *provided* > > > > > > > > > org.scalanlp > > > breeze_2.10 > > 0.11.2 > > *provided* > > > > > > > > When submit, I included breeze jars (breeze_2.10-0.11.2.jar > breeze-math_2.10-0.4.jar breeze-natives_2.10-0.11.2.jar > breeze-process_2.10-0.3.jar) using “--jar” arguments, although I doubt it > is necessary to do that. > > > > however, the error is > > > > Exception in thread "main" java.lang.NoClassDefFoundError: > breeze/storage/DefaultArrayValue > > > > Any thoughts? > > > > > > > > Best regards, > > Jack > > > > >
RE: spark with breeze error of NoClassDefFoundError
If I tried to change “provided” to “compile”.. then the error changed to : Exception in thread "main" java.lang.IncompatibleClassChangeError: Implementing class at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader.java:800) at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) at java.net.URLClassLoader.defineClass(URLClassLoader.java:449) at java.net.URLClassLoader.access$100(URLClassLoader.java:71) at java.net.URLClassLoader$1.run(URLClassLoader.java:361) at java.net.URLClassLoader$1.run(URLClassLoader.java:355) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:354) at java.lang.ClassLoader.loadClass(ClassLoader.java:425) at java.lang.ClassLoader.loadClass(ClassLoader.java:358) at smartapp.smart.sparkwithscala.textMingApp.main(textMingApp.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:664) at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:169) at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:192) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:111) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) 15/11/19 10:28:29 INFO util.Utils: Shutdown hook called Meanwhile, I will prefer to use maven to compile the jar file rather than sbt, although it is indeed another option. Best regards, Jack From: Fengdong Yu [mailto:fengdo...@everstring.com] Sent: Wednesday, 18 November 2015 7:30 PM To: Jack Yang Cc: Ted Yu; user@spark.apache.org Subject: Re: spark with breeze error of NoClassDefFoundError The simplest way is remove all “provided” in your pom. then ‘sbt assembly” to build your final package. then get rid of ‘—jars’ because assembly already includes all dependencies. On Nov 18, 2015, at 2:15 PM, Jack Yang <j...@uow.edu.au<mailto:j...@uow.edu.au>> wrote: So weird. Is there anything wrong with the way I made the pom file (I labelled them as provided)? Is there missing jar I forget to add in “--jar”? See the trace below: Exception in thread "main" java.lang.NoClassDefFoundError: breeze/storage/DefaultArrayValue at smartapp.smart.sparkwithscala.textMingApp.main(textMingApp.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:664) at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:169) at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:192) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:111) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) Caused by: java.lang.ClassNotFoundException: breeze.storage.DefaultArrayValue at java.net.URLClassLoader$1.run(URLClassLoader.java:366) at java.net.URLClassLoader$1.run(URLClassLoader.java:355) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:354) at java.lang.ClassLoader.loadClass(ClassLoader.java:425) at java.lang.ClassLoader.loadClass(ClassLoader.java:358) ... 10 more 15/11/18 17:15:15 INFO util.Utils: Shutdown hook called From: Ted Yu [mailto:yuzhih...@gmail.com] Sent: Wednesday, 18 November 2015 4:01 PM To: Jack Yang Cc: user@spark.apache.org<mailto:user@spark.apache.org> Subject: Re: spark with breeze error of NoClassDefFoundError Looking in local maven repo, breeze_2.10-0.7.jar contains DefaultArrayValue : jar tvf /Users/tyu/.m2/repository//org/scalanlp/breeze_2.10/0.7/breeze_2.10-0.7.jar | grep !$ jar tvf /Users/tyu/.m2/repository//org/scalanlp/breeze_2.10/0.7/breeze_2.10-0.7.jar | grep DefaultArrayValue 369 Wed Mar 19 11:18:32 PDT 2014 breeze/storage/DefaultArrayValue$mcZ$sp$class.class 309 Wed Mar 19 11:18:32 PDT 2014 breeze/storage/DefaultArrayValue$mcJ$sp.class 2233 Wed Mar 19 11:18:32 PDT 2014 breeze/storage/DefaultArrayValue$DoubleDefaultArrayValue$.class Can you show the complete stack trace ? FYI On Tue, Nov 17, 2015 at 8:33 PM, Jack Yang <j...@uow.edu.au<
Re: spark with breeze error of NoClassDefFoundError
Dear Ted,I just looked at the link you provided, it is great! For my understanding, I could also directly use other Breeze part (except spark mllib package linalg ) in spark (scala or java ) program after importing Breeze package,it is right? Thanks a lot in advance again!Zhiliang On Thursday, November 19, 2015 1:46 PM, Ted Yu <yuzhih...@gmail.com> wrote: Have you looked athttps://github.com/scalanlp/breeze/wiki Cheers On Nov 18, 2015, at 9:34 PM, Zhiliang Zhu <zchl.j...@yahoo.com> wrote: Dear Jack, As is known, Breeze is numerical calculation package wrote by scala , spark mllib also use it as underlying package for algebra usage.Here I am also preparing to use Breeze for nonlinear equation optimization, however, it seemed that I could not find the exact doc or API for Breeze except spark linalg package... Could you help some to provide me the official doc or API website for Breeze ?Thank you in advance! Zhiliang On Thursday, November 19, 2015 7:32 AM, Jack Yang <j...@uow.edu.au> wrote: #yiv6155504207 #yiv6155504207 -- filtered {font-family:SimSun;panose-1:2 1 6 0 3 1 1 1 1 1;}#yiv6155504207 filtered {font-family:SimSun;panose-1:2 1 6 0 3 1 1 1 1 1;}#yiv6155504207 filtered {font-family:Calibri;panose-1:2 15 5 2 2 2 4 3 2 4;}#yiv6155504207 filtered {font-family:Tahoma;panose-1:2 11 6 4 3 5 4 4 2 4;}#yiv6155504207 filtered {panose-1:2 1 6 0 3 1 1 1 1 1;}#yiv6155504207 p.yiv6155504207MsoNormal, #yiv6155504207 li.yiv6155504207MsoNormal, #yiv6155504207 div.yiv6155504207MsoNormal {margin:0cm;margin-bottom:.0001pt;font-size:12.0pt;}#yiv6155504207 a:link, #yiv6155504207 span.yiv6155504207MsoHyperlink {color:blue;text-decoration:underline;}#yiv6155504207 a:visited, #yiv6155504207 span.yiv6155504207MsoHyperlinkFollowed {color:purple;text-decoration:underline;}#yiv6155504207 p.yiv6155504207MsoAcetate, #yiv6155504207 li.yiv6155504207MsoAcetate, #yiv6155504207 div.yiv6155504207MsoAcetate {margin:0cm;margin-bottom:.0001pt;font-size:8.0pt;}#yiv6155504207 span.yiv6155504207apple-converted-space {}#yiv6155504207 span.yiv6155504207EmailStyle18 {color:#1F497D;}#yiv6155504207 span.yiv6155504207BalloonTextChar {}#yiv6155504207 .yiv6155504207MsoChpDefault {font-size:10.0pt;}#yiv6155504207 filtered {margin:72.0pt 72.0pt 72.0pt 72.0pt;}#yiv6155504207 div.yiv6155504207WordSection1 {}#yiv6155504207 If I tried to change “provided” to “compile”.. then the error changed to : Exception in thread "main" java.lang.IncompatibleClassChangeError: Implementing class at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader.java:800) at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) at java.net.URLClassLoader.defineClass(URLClassLoader.java:449) at java.net.URLClassLoader.access$100(URLClassLoader.java:71) at java.net.URLClassLoader$1.run(URLClassLoader.java:361) at java.net.URLClassLoader$1.run(URLClassLoader.java:355) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:354) at java.lang.ClassLoader.loadClass(ClassLoader.java:425) at java.lang.ClassLoader.loadClass(ClassLoader.java:358) atsmartapp.smart.sparkwithscala.textMingApp.main(textMingApp.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:664) at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:169) at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:192) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:111) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) 15/11/19 10:28:29 INFO util.Utils: Shutdown hook called Meanwhile, I will prefer to use maven to compile the jar file rather than sbt, although it is indeed another option. Best regards, Jack From: Fengdong Yu [mailto:fengdo...@everstring.com] Sent: Wednesday, 18 November 2015 7:30 PM To: Jack Yang Cc: Ted Yu; user@spark.apache.org Subject: Re: spark with breeze error of NoClassDefFoundError The simplest way is remove all “provided” in your pom. then ‘sbt assembly” to build your final package. then get rid of ‘—jars’ because assembly already includes all dependencies. On Nov 18, 2015, at 2:15 PM, Jack Yang <j...@uow.edu.au> wrote: So weird. Is there anything wrong with the way I made the pom file (I labelled them as provided)? Is there missing jar I forget to add
Re: spark with breeze error of NoClassDefFoundError
Dear Jack, As is known, Breeze is numerical calculation package wrote by scala , spark mllib also use it as underlying package for algebra usage.Here I am also preparing to use Breeze for nonlinear equation optimization, however, it seemed that I could not find the exact doc or API for Breeze except spark linalg package... Could you help some to provide me the official doc or API website for Breeze ?Thank you in advance! Zhiliang On Thursday, November 19, 2015 7:32 AM, Jack Yang <j...@uow.edu.au> wrote: If I tried to change “provided” to “compile”.. then the error changed to : Exception in thread "main" java.lang.IncompatibleClassChangeError: Implementing class at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader.java:800) at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) at java.net.URLClassLoader.defineClass(URLClassLoader.java:449) at java.net.URLClassLoader.access$100(URLClassLoader.java:71) at java.net.URLClassLoader$1.run(URLClassLoader.java:361) at java.net.URLClassLoader$1.run(URLClassLoader.java:355) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:354) at java.lang.ClassLoader.loadClass(ClassLoader.java:425) at java.lang.ClassLoader.loadClass(ClassLoader.java:358) atsmartapp.smart.sparkwithscala.textMingApp.main(textMingApp.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:664) at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:169) at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:192) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:111) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) 15/11/19 10:28:29 INFO util.Utils: Shutdown hook called Meanwhile, I will prefer to use maven to compile the jar file rather than sbt, although it is indeed another option. Best regards, Jack From: Fengdong Yu [mailto:fengdo...@everstring.com] Sent: Wednesday, 18 November 2015 7:30 PM To: Jack Yang Cc: Ted Yu; user@spark.apache.org Subject: Re: spark with breeze error of NoClassDefFoundError The simplest way is remove all “provided” in your pom. then ‘sbt assembly” to build your final package. then get rid of ‘—jars’ because assembly already includes all dependencies. On Nov 18, 2015, at 2:15 PM, Jack Yang <j...@uow.edu.au> wrote: So weird. Is there anything wrong with the way I made the pom file (I labelled them as provided)? Is there missing jar I forget to add in “--jar”? See the trace below: Exception in thread "main" java.lang.NoClassDefFoundError: breeze/storage/DefaultArrayValue at smartapp.smart.sparkwithscala.textMingApp.main(textMingApp.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:664) at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:169) at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:192) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:111) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) Caused by: java.lang.ClassNotFoundException: breeze.storage.DefaultArrayValue at java.net.URLClassLoader$1.run(URLClassLoader.java:366) at java.net.URLClassLoader$1.run(URLClassLoader.java:355) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:354) at java.lang.ClassLoader.loadClass(ClassLoader.java:425) at java.lang.ClassLoader.loadClass(ClassLoader.java:358) ... 10 more 15/11/18 17:15:15 INFO util.Utils: Shutdown hook called From: Ted Yu [mailto:yuzhih...@gmail.com] Sent: Wednesday, 18 November 2015 4:01 PM To: Jack Yang Cc: user@spark.apache.org Subject: Re: spark with breeze error of NoClassDefFoundError Looking in local maven repo, breeze_2.10-0.7.jar contains DefaultArrayValue : jar tvf /Users/tyu/.m2/reposito
Re: spark with breeze error of NoClassDefFoundError
Have you looked at https://github.com/scalanlp/breeze/wiki Cheers > On Nov 18, 2015, at 9:34 PM, Zhiliang Zhu <zchl.j...@yahoo.com> wrote: > > Dear Jack, > > As is known, Breeze is numerical calculation package wrote by scala , spark > mllib also use it as underlying package for algebra usage. > Here I am also preparing to use Breeze for nonlinear equation optimization, > however, it seemed that I could not find the exact doc or API for Breeze > except spark linalg package... > > Could you help some to provide me the official doc or API website for Breeze ? > Thank you in advance! > > Zhiliang > > > > > On Thursday, November 19, 2015 7:32 AM, Jack Yang <j...@uow.edu.au> wrote: > > > If I tried to change “provided” to “compile”.. then the error changed to : > > Exception in thread "main" java.lang.IncompatibleClassChangeError: > Implementing class > at java.lang.ClassLoader.defineClass1(Native Method) > at java.lang.ClassLoader.defineClass(ClassLoader.java:800) > at > java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) > at java.net.URLClassLoader.defineClass(URLClassLoader.java:449) > at java.net.URLClassLoader.access$100(URLClassLoader.java:71) > at java.net.URLClassLoader$1.run(URLClassLoader.java:361) > at java.net.URLClassLoader$1.run(URLClassLoader.java:355) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:354) > at java.lang.ClassLoader.loadClass(ClassLoader.java:425) > at java.lang.ClassLoader.loadClass(ClassLoader.java:358) > at smartapp.smart.sparkwithscala.textMingApp.main(textMingApp.scala) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:664) > at > org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:169) > at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:192) > at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:111) > at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) > 15/11/19 10:28:29 INFO util.Utils: Shutdown hook called > > Meanwhile, I will prefer to use maven to compile the jar file rather than > sbt, although it is indeed another option. > > Best regards, > Jack > > > > From: Fengdong Yu [mailto:fengdo...@everstring.com] > Sent: Wednesday, 18 November 2015 7:30 PM > To: Jack Yang > Cc: Ted Yu; user@spark.apache.org > Subject: Re: spark with breeze error of NoClassDefFoundError > > The simplest way is remove all “provided” in your pom. > > then ‘sbt assembly” to build your final package. then get rid of ‘—jars’ > because assembly already includes all dependencies. > > > > > > > On Nov 18, 2015, at 2:15 PM, Jack Yang <j...@uow.edu.au> wrote: > > So weird. Is there anything wrong with the way I made the pom file (I > labelled them as provided)? > > Is there missing jar I forget to add in “--jar”? > > See the trace below: > > > > Exception in thread "main" java.lang.NoClassDefFoundError: > breeze/storage/DefaultArrayValue > at smartapp.smart.sparkwithscala.textMingApp.main(textMingApp.scala) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:664) > at > org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:169) > at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:192) > at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:111) > at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) > Caused by: java.lang.ClassNotFoundException: breeze.storage.DefaultArrayValue > at java.net.URLClassLoader$1.run(URLClassLoader.java:366) > at java.net.URLClassLoader$1.run(URLClassLoader.java:355) > at java.security.Access
RE: spark with breeze error of NoClassDefFoundError
Back to my question. If I use “provided”, the jar file will expect some libraries are provided by the system. However, the “ compiled ” is the default setting, which means the third-party library will be included inside jar file after compiling. So when I use “provided”, the error is they cannot find the Class, but with “compiled” the error is IncompatibleClassChangeError. Ok, so can someone tell me which version of breeze and breeze-math are used in spark 1.4? From: Zhiliang Zhu [mailto:zchl.j...@yahoo.com] Sent: Thursday, 19 November 2015 5:10 PM To: Ted Yu Cc: Jack Yang; Fengdong Yu; user@spark.apache.org Subject: Re: spark with breeze error of NoClassDefFoundError Dear Ted, I just looked at the link you provided, it is great! For my understanding, I could also directly use other Breeze part (except spark mllib package linalg ) in spark (scala or java ) program after importing Breeze package, it is right? Thanks a lot in advance again! Zhiliang On Thursday, November 19, 2015 1:46 PM, Ted Yu <yuzhih...@gmail.com<mailto:yuzhih...@gmail.com>> wrote: Have you looked at https://github.com/scalanlp/breeze/wiki Cheers On Nov 18, 2015, at 9:34 PM, Zhiliang Zhu <zchl.j...@yahoo.com<mailto:zchl.j...@yahoo.com>> wrote: Dear Jack, As is known, Breeze is numerical calculation package wrote by scala , spark mllib also use it as underlying package for algebra usage. Here I am also preparing to use Breeze for nonlinear equation optimization, however, it seemed that I could not find the exact doc or API for Breeze except spark linalg package... Could you help some to provide me the official doc or API website for Breeze ? Thank you in advance! Zhiliang On Thursday, November 19, 2015 7:32 AM, Jack Yang <j...@uow.edu.au<mailto:j...@uow.edu.au>> wrote: If I tried to change “provided” to “compile”.. then the error changed to : Exception in thread "main" java.lang.IncompatibleClassChangeError: Implementing class at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader.java:800) at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) at java.net.URLClassLoader.defineClass(URLClassLoader.java:449) at java.net.URLClassLoader.access$100(URLClassLoader.java:71) at java.net.URLClassLoader$1.run(URLClassLoader.java:361) at java.net.URLClassLoader$1.run(URLClassLoader.java:355) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:354) at java.lang.ClassLoader.loadClass(ClassLoader.java:425) at java.lang.ClassLoader.loadClass(ClassLoader.java:358) at smartapp.smart.sparkwithscala.textMingApp.main(textMingApp.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:664) at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:169) at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:192) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:111) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) 15/11/19 10:28:29 INFO util.Utils: Shutdown hook called Meanwhile, I will prefer to use maven to compile the jar file rather than sbt, although it is indeed another option. Best regards, Jack From: Fengdong Yu [mailto:fengdo...@everstring.com] Sent: Wednesday, 18 November 2015 7:30 PM To: Jack Yang Cc: Ted Yu; user@spark.apache.org<mailto:user@spark.apache.org> Subject: Re: spark with breeze error of NoClassDefFoundError The simplest way is remove all “provided” in your pom. then ‘sbt assembly” to build your final package. then get rid of ‘—jars’ because assembly already includes all dependencies. On Nov 18, 2015, at 2:15 PM, Jack Yang <j...@uow.edu.au<mailto:j...@uow.edu.au>> wrote: So weird. Is there anything wrong with the way I made the pom file (I labelled them as provided)? Is there missing jar I forget to add in “--jar”? See the trace below: Exception in thread "main" java.lang.NoClassDefFoundError: breeze/storage/DefaultArrayValue at smartapp.smart.sparkwithscala.textMingApp.main(textMingApp.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at
Re: spark with breeze error of NoClassDefFoundError
Looking in local maven repo, breeze_2.10-0.7.jar contains DefaultArrayValue : jar tvf /Users/tyu/.m2/repository//org/scalanlp/breeze_2.10/0.7/breeze_2.10-0.7.jar | grep !$ jar tvf /Users/tyu/.m2/repository//org/scalanlp/breeze_2.10/0.7/breeze_2.10-0.7.jar | grep DefaultArrayValue 369 Wed Mar 19 11:18:32 PDT 2014 breeze/storage/DefaultArrayValue$mcZ$sp$class.class 309 Wed Mar 19 11:18:32 PDT 2014 breeze/storage/DefaultArrayValue$mcJ$sp.class 2233 Wed Mar 19 11:18:32 PDT 2014 breeze/storage/DefaultArrayValue$DoubleDefaultArrayValue$.class Can you show the complete stack trace ? FYI On Tue, Nov 17, 2015 at 8:33 PM, Jack Yangwrote: > Hi all, > > I am using spark 1.4.0, and building my codes using maven. > > So in one of my scala, I used: > > > > import breeze.linalg._ > > val v1 = new breeze.linalg.SparseVector(commonVector.indices, > commonVector.values, commonVector.size) > > val v2 = new breeze.linalg.SparseVector(commonVector2.indices, > commonVector2.values, commonVector2.size) > > println (v1.dot(v2) / (norm(v1) * norm(v2)) ) > > > > > > > > in my pom.xml file, I used: > > > > > org.scalanlp > > > breeze-math_2.10 > > 0.4 > > *provided* > > > > > > > > > org.scalanlp > > > breeze_2.10 > > 0.11.2 > > *provided* > > > > > > > > When submit, I included breeze jars (breeze_2.10-0.11.2.jar > breeze-math_2.10-0.4.jar breeze-natives_2.10-0.11.2.jar > breeze-process_2.10-0.3.jar) using “--jar” arguments, although I doubt it > is necessary to do that. > > > > however, the error is > > > > Exception in thread "main" java.lang.NoClassDefFoundError: > breeze/storage/DefaultArrayValue > > > > Any thoughts? > > > > > > > > Best regards, > > Jack > > >
RE: spark with breeze error of NoClassDefFoundError
So weird. Is there anything wrong with the way I made the pom file (I labelled them as provided)? Is there missing jar I forget to add in “--jar”? See the trace below: Exception in thread "main" java.lang.NoClassDefFoundError: breeze/storage/DefaultArrayValue at smartapp.smart.sparkwithscala.textMingApp.main(textMingApp.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:664) at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:169) at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:192) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:111) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) Caused by: java.lang.ClassNotFoundException: breeze.storage.DefaultArrayValue at java.net.URLClassLoader$1.run(URLClassLoader.java:366) at java.net.URLClassLoader$1.run(URLClassLoader.java:355) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:354) at java.lang.ClassLoader.loadClass(ClassLoader.java:425) at java.lang.ClassLoader.loadClass(ClassLoader.java:358) ... 10 more 15/11/18 17:15:15 INFO util.Utils: Shutdown hook called From: Ted Yu [mailto:yuzhih...@gmail.com] Sent: Wednesday, 18 November 2015 4:01 PM To: Jack Yang Cc: user@spark.apache.org Subject: Re: spark with breeze error of NoClassDefFoundError Looking in local maven repo, breeze_2.10-0.7.jar contains DefaultArrayValue : jar tvf /Users/tyu/.m2/repository//org/scalanlp/breeze_2.10/0.7/breeze_2.10-0.7.jar | grep !$ jar tvf /Users/tyu/.m2/repository//org/scalanlp/breeze_2.10/0.7/breeze_2.10-0.7.jar | grep DefaultArrayValue 369 Wed Mar 19 11:18:32 PDT 2014 breeze/storage/DefaultArrayValue$mcZ$sp$class.class 309 Wed Mar 19 11:18:32 PDT 2014 breeze/storage/DefaultArrayValue$mcJ$sp.class 2233 Wed Mar 19 11:18:32 PDT 2014 breeze/storage/DefaultArrayValue$DoubleDefaultArrayValue$.class Can you show the complete stack trace ? FYI On Tue, Nov 17, 2015 at 8:33 PM, Jack Yang <j...@uow.edu.au<mailto:j...@uow.edu.au>> wrote: Hi all, I am using spark 1.4.0, and building my codes using maven. So in one of my scala, I used: import breeze.linalg._ val v1 = new breeze.linalg.SparseVector(commonVector.indices, commonVector.values, commonVector.size) val v2 = new breeze.linalg.SparseVector(commonVector2.indices, commonVector2.values, commonVector2.size) println (v1.dot(v2) / (norm(v1) * norm(v2)) ) in my pom.xml file, I used: org.scalanlp breeze-math_2.10 0.4 provided org.scalanlp breeze_2.10 0.11.2 provided When submit, I included breeze jars (breeze_2.10-0.11.2.jar breeze-math_2.10-0.4.jar breeze-natives_2.10-0.11.2.jar breeze-process_2.10-0.3.jar) using “--jar” arguments, although I doubt it is necessary to do that. however, the error is Exception in thread "main" java.lang.NoClassDefFoundError: breeze/storage/DefaultArrayValue Any thoughts? Best regards, Jack