Hi, Trying to run spark with yarn-client not using spark-submit here what are you using to submit the job? spark-shell, spark-sql or anything else
Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* http://talebzadehmich.wordpress.com On 24 June 2016 at 12:01, <sychu...@tsmc.com> wrote: > > Hello guys, > > Trying to run spark with yarn-client not using spark-submit here but the > jobs kept failed while AM launching executor. > The error collected by yarn like below. > Looks like some environment setting is missing? > Could someone help me out with this. > > Thanks in advance! > HY Chung > > Java HotSpot(TM) 64-Bit Server VM warning: ignoring option > MaxPermSize=256m; support was removed in 8.0 > Exception in thread "main" java.lang.NoClassDefFoundError: > org/apache/spark/Logging > at java.lang.ClassLoader.defineClass1(Native Method) > at java.lang.ClassLoader.defineClass(ClassLoader.java:760) > at java.security.SecureClassLoader.defineClass > (SecureClassLoader.java:142) > at java.net.URLClassLoader.defineClass > (URLClassLoader.java:455) > at > java.net.URLClassLoader.access$100(URLClassLoader.java:73) > at java.net.URLClassLoader$1.run(URLClassLoader.java:367) > at java.net.URLClassLoader$1.run(URLClassLoader.java:361) > at java.security.AccessController.doPrivileged(Native > Method) > at > java.net.URLClassLoader.findClass(URLClassLoader.java:360) > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > at sun.misc.Launcher$AppClassLoader.loadClass > (Launcher.java:308) > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > at org.apache.spark.deploy.yarn.ExecutorLauncher$.main > (ApplicationMaster.scala:674) > at org.apache.spark.deploy.yarn.ExecutorLauncher.main > (ApplicationMaster.scala) > Caused by: java.lang.ClassNotFoundException: org.apache.spark.Logging > at java.net.URLClassLoader$1.run(URLClassLoader.java:372) > at java.net.URLClassLoader$1.run(URLClassLoader.java:361) > at java.security.AccessController.doPrivileged(Native > Method) > at > java.net.URLClassLoader.findClass(URLClassLoader.java:360) > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > at sun.misc.Launcher$AppClassLoader.loadClass > (Launcher.java:308) > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > ... 14 more > > > Best regards, > > S.Y. Chung 鍾學毅 > F14MITD > Taiwan Semiconductor Manufacturing Company, Ltd. > Tel: 06-5056688 Ext: 734-6325 > > --------------------------------------------------------------------------- > TSMC PROPERTY > This email communication (and any attachments) is proprietary information > for the sole use of its > intended recipient. Any unauthorized review, use or distribution by anyone > other than the intended > recipient is strictly prohibited. If you are not the intended recipient, > please notify the sender by > replying to this email, and then delete this email and any copies of it > immediately. Thank you. > > --------------------------------------------------------------------------- > > >
