Looks like you are bundling hadoop jars in the app package and the version is not same as available in hadoop class path. You should not package hadoop jars in your app package.
Thanks - Gaurav > On Aug 24, 2015, at 2:18 PM, York, Brennon <[email protected]> > wrote: > > Awesome! That¹s what we figured, but we¹re still not understanding why the > containers are failing for YARN. Any thoughts there? > > On 8/24/15, 2:11 PM, "Gaurav Gupta" <[email protected]> wrote: > >> The scope should not be provided. >> >> Thanks >> - Gaurav >> >>> On Aug 24, 2015, at 2:09 PM, York, Brennon >>> <[email protected]> wrote: >>> >>> All, seeing some odd behavior when submitting a job through `dtcli` >>> that, I believe has something to do with Malhar(??). We¹re using both >>> `malhar-contrib` and `malhar-library` in the application and, when both >>> are provided as `scope: provided`, the app will fail with the below >>> (full stack at bottom): >>> >>> java.lang.NoClassDefFoundError: >>> com/datatorrent/lib/io/fs/AbstractFileInputOperator >>> >>> Now, when we package both into the resultant jar, the application then >>> fails at the YARN level with a statement like: >>> >>> Application application_1440084097966_0009 failed 2 times due to AM >>> Container for appattempt_1440084097966_0009_000002 exited with exitCode: >>> 1 >>> >>> From the YARN Resource Manager. Digging deeper into the YARN container >>> logs only provides the following exception: >>> >>> Exception in thread "main" java.lang.AbstractMethodError: >>> org.apache.hadoop.yarn.api.records.ContainerId.setContainerId(J)V >>> at >>> org.apache.hadoop.yarn.api.records.ContainerId.newInstance(ContainerId.ja >>> va:60) >>> at >>> org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(ConverterUtils.j >>> ava:178) >>> at >>> com.datatorrent.stram.StreamingAppMaster.main(StreamingAppMaster.java:87) >>> >>> Any thoughts on this and why, when we jar the malhar packages into our >>> app jar, it fails like this? Could this be a version mismatch? >>> >>> For reference we¹re using the DT Community Edition and building against >>> Malhar v3.0.0 as well as DT v3.0.0. >>> >>> Full stack trace when `scope: provided` for the Malhar packages: >>> java.lang.NoClassDefFoundError: >>> com/datatorrent/lib/io/fs/AbstractFileInputOperator >>> at java.lang.ClassLoader.defineClass1(Native Method) >>> at java.lang.ClassLoader.defineClass(ClassLoader.java:800) >>> at >>> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) >>> at java.net.URLClassLoader.defineClass(URLClassLoader.java:449) >>> at java.net.URLClassLoader.access$100(URLClassLoader.java:71) >>> at java.net.URLClassLoader$1.run(URLClassLoader.java:361) >>> at java.net.URLClassLoader$1.run(URLClassLoader.java:355) >>> at java.security.AccessController.doPrivileged(Native Method) >>> at java.net.URLClassLoader.findClass(URLClassLoader.java:354) >>> at java.lang.ClassLoader.loadClass(ClassLoader.java:425) >>> at java.net.FactoryURLClassLoader.loadClass(URLClassLoader.java:793) >>> at java.lang.ClassLoader.loadClass(ClassLoader.java:358) >>> at >>> com.capitalone.vault8.citadel.H20Application.populateDAG(H20Application.j >>> ava:40) >>> at >>> com.datatorrent.stram.plan.logical.LogicalPlanConfiguration.prepareDAG(Lo >>> gicalPlanConfiguration.java:1171) >>> at >>> com.datatorrent.stram.client.StramAppLauncher$1.createApp(StramAppLaunche >>> r.java:404) >>> at >>> com.datatorrent.stram.client.StramAppLauncher.launchApp(StramAppLauncher. >>> java:479) >>> at >>> com.datatorrent.stram.cli.DTCli$LaunchCommand.execute(DTCli.java:2048) >>> at com.datatorrent.stram.cli.DTCli.launchAppPackage(DTCli.java:3452) >>> at com.datatorrent.stram.cli.DTCli.access$7000(DTCli.java:104) >>> at >>> com.datatorrent.stram.cli.DTCli$LaunchCommand.execute(DTCli.java:1893) >>> at com.datatorrent.stram.cli.DTCli$3.run(DTCli.java:1447) >>> Caused by: java.lang.ClassNotFoundException: >>> com.datatorrent.lib.io.fs.AbstractFileInputOperator >>> at java.net.URLClassLoader$1.run(URLClassLoader.java:366) >>> at java.net.URLClassLoader$1.run(URLClassLoader.java:355) >>> at java.security.AccessController.doPrivileged(Native Method) >>> at java.net.URLClassLoader.findClass(URLClassLoader.java:354) >>> at java.lang.ClassLoader.loadClass(ClassLoader.java:425) >>> at java.net.FactoryURLClassLoader.loadClass(URLClassLoader.java:793) >>> at java.lang.ClassLoader.loadClass(ClassLoader.java:358) >>> ... 21 more >>> ________________________________________________________ >>> >>> The information contained in this e-mail is confidential and/or >>> proprietary to Capital One and/or its affiliates and may only be used >>> solely in performance of work or services for Capital One. The >>> information transmitted herewith is intended only for use by the >>> individual or entity to which it is addressed. If the reader of this >>> message is not the intended recipient, you are hereby notified that any >>> review, retransmission, dissemination, distribution, copying or other >>> use of, or taking of any action in reliance upon this information is >>> strictly prohibited. If you have received this communication in error, >>> please contact the sender and delete the material from your computer. >> > > ________________________________________________________ > > The information contained in this e-mail is confidential and/or proprietary > to Capital One and/or its affiliates and may only be used solely in > performance of work or services for Capital One. The information transmitted > herewith is intended only for use by the individual or entity to which it is > addressed. If the reader of this message is not the intended recipient, you > are hereby notified that any review, retransmission, dissemination, > distribution, copying or other use of, or taking of any action in reliance > upon this information is strictly prohibited. If you have received this > communication in error, please contact the sender and delete the material > from your computer. >
