It means several jars are missing in the yarn container environment, if you want to submit your application through some other ways besides spark-submit, you have to take care all the environment things yourself. Since we don't know your implementation of java web service, so it is hard to provide useful information. Basically the problem is spark jar is missing in the AM side, so you have to put spark jars into distributed cache ahead of time.
On Mon, Jun 27, 2016 at 11:19 AM, <sychu...@tsmc.com> wrote: > Hi, > > Thanks for reply. It's a java web service resides in a jboss container. > > HY Chung > Best regards, > > S.Y. Chung 鍾學毅 > F14MITD > Taiwan Semiconductor Manufacturing Company, Ltd. > Tel: 06-5056688 Ext: 734-6325 > > > |-------------------------------------> > | Mich Talebzadeh | > | <mich.talebza...@gmail.co| > | m> | > | | > | | > | | > | 2016/06/24 下午 08:05 | > |-------------------------------------> > > >-------------------------------------------------------------------------------------------------------------------------------------------------| > | > | > | > | > | > To| > | sychu...@tsmc.com > | > | > cc| > | "user @spark" <user@spark.apache.org> > | > | > Subject| > | [Spam][SMG] Re: problem running spark with yarn-client not > using spark-submit > | > | > | > | > | > | > | > | > | > | > | > > >-------------------------------------------------------------------------------------------------------------------------------------------------| > > > > > Hi, > Trying to run spark with yarn-client not using spark-submit here > > what are you using to submit the job? spark-shell, spark-sql or anything > else > > > Dr Mich Talebzadeh > > LinkedIn > > https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw > > http://talebzadehmich.wordpress.com > > > > > On 24 June 2016 at 12:01, <sychu...@tsmc.com> wrote: > > Hello guys, > > Trying to run spark with yarn-client not using spark-submit here but the > jobs kept failed while AM launching executor. > The error collected by yarn like below. > Looks like some environment setting is missing? > Could someone help me out with this. > > Thanks in advance! > HY Chung > > Java HotSpot(TM) 64-Bit Server VM warning: ignoring option > MaxPermSize=256m; support was removed in 8.0 > Exception in thread "main" java.lang.NoClassDefFoundError: > org/apache/spark/Logging > at java.lang.ClassLoader.defineClass1(Native Method) > at java.lang.ClassLoader.defineClass > (ClassLoader.java:760) > at java.security.SecureClassLoader.defineClass > (SecureClassLoader.java:142) > at java.net.URLClassLoader.defineClass > (URLClassLoader.java:455) > at java.net.URLClassLoader.access$100 > (URLClassLoader.java:73) > at java.net.URLClassLoader$1.run > (URLClassLoader.java:367) > at java.net.URLClassLoader$1.run > (URLClassLoader.java:361) > at java.security.AccessController.doPrivileged(Native > Method) > at java.net.URLClassLoader.findClass > (URLClassLoader.java:360) > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > at sun.misc.Launcher$AppClassLoader.loadClass > (Launcher.java:308) > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > at org.apache.spark.deploy.yarn.ExecutorLauncher$.main > (ApplicationMaster.scala:674) > at org.apache.spark.deploy.yarn.ExecutorLauncher.main > (ApplicationMaster.scala) > Caused by: java.lang.ClassNotFoundException: org.apache.spark.Logging > at java.net.URLClassLoader$1.run > (URLClassLoader.java:372) > at java.net.URLClassLoader$1.run > (URLClassLoader.java:361) > at java.security.AccessController.doPrivileged(Native > Method) > at java.net.URLClassLoader.findClass > (URLClassLoader.java:360) > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > at sun.misc.Launcher$AppClassLoader.loadClass > (Launcher.java:308) > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > ... 14 more > > > Best regards, > > S.Y. Chung 鍾學毅 > F14MITD > Taiwan Semiconductor Manufacturing Company, Ltd. > Tel: 06-5056688 Ext: 734-6325 > > > --------------------------------------------------------------------------- > > TSMC PROPERTY > This email communication (and any attachments) is proprietary > information > for the sole use of its > intended recipient. Any unauthorized review, use or distribution by > anyone > other than the intended > recipient is strictly prohibited. If you are not the intended > recipient, > please notify the sender by > replying to this email, and then delete this email and any copies of it > immediately. Thank you. > > > --------------------------------------------------------------------------- > > > > > > > --------------------------------------------------------------------------- > TSMC PROPERTY > This email communication (and any attachments) is proprietary information > for the sole use of its > intended recipient. Any unauthorized review, use or distribution by anyone > other than the intended > recipient is strictly prohibited. If you are not the intended recipient, > please notify the sender by > replying to this email, and then delete this email and any copies of it > immediately. Thank you. > > --------------------------------------------------------------------------- > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > >
