All, seeing some odd behavior when submitting a job through `dtcli` that, I believe has something to do with Malhar(??). We’re using both `malhar-contrib` and `malhar-library` in the application and, when both are provided as `scope: provided`, the app will fail with the below (full stack at bottom):
java.lang.NoClassDefFoundError: com/datatorrent/lib/io/fs/AbstractFileInputOperator Now, when we package both into the resultant jar, the application then fails at the YARN level with a statement like: Application application_1440084097966_0009 failed 2 times due to AM Container for appattempt_1440084097966_0009_000002 exited with exitCode: 1 >From the YARN Resource Manager. Digging deeper into the YARN container logs >only provides the following exception: Exception in thread "main" java.lang.AbstractMethodError: org.apache.hadoop.yarn.api.records.ContainerId.setContainerId(J)V at org.apache.hadoop.yarn.api.records.ContainerId.newInstance(ContainerId.java:60) at org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(ConverterUtils.java:178) at com.datatorrent.stram.StreamingAppMaster.main(StreamingAppMaster.java:87) Any thoughts on this and why, when we jar the malhar packages into our app jar, it fails like this? Could this be a version mismatch? For reference we’re using the DT Community Edition and building against Malhar v3.0.0 as well as DT v3.0.0. Full stack trace when `scope: provided` for the Malhar packages: java.lang.NoClassDefFoundError: com/datatorrent/lib/io/fs/AbstractFileInputOperator at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader.java:800) at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) at java.net.URLClassLoader.defineClass(URLClassLoader.java:449) at java.net.URLClassLoader.access$100(URLClassLoader.java:71) at java.net.URLClassLoader$1.run(URLClassLoader.java:361) at java.net.URLClassLoader$1.run(URLClassLoader.java:355) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:354) at java.lang.ClassLoader.loadClass(ClassLoader.java:425) at java.net.FactoryURLClassLoader.loadClass(URLClassLoader.java:793) at java.lang.ClassLoader.loadClass(ClassLoader.java:358) at com.capitalone.vault8.citadel.H20Application.populateDAG(H20Application.java:40) at com.datatorrent.stram.plan.logical.LogicalPlanConfiguration.prepareDAG(LogicalPlanConfiguration.java:1171) at com.datatorrent.stram.client.StramAppLauncher$1.createApp(StramAppLauncher.java:404) at com.datatorrent.stram.client.StramAppLauncher.launchApp(StramAppLauncher.java:479) at com.datatorrent.stram.cli.DTCli$LaunchCommand.execute(DTCli.java:2048) at com.datatorrent.stram.cli.DTCli.launchAppPackage(DTCli.java:3452) at com.datatorrent.stram.cli.DTCli.access$7000(DTCli.java:104) at com.datatorrent.stram.cli.DTCli$LaunchCommand.execute(DTCli.java:1893) at com.datatorrent.stram.cli.DTCli$3.run(DTCli.java:1447) Caused by: java.lang.ClassNotFoundException: com.datatorrent.lib.io.fs.AbstractFileInputOperator at java.net.URLClassLoader$1.run(URLClassLoader.java:366) at java.net.URLClassLoader$1.run(URLClassLoader.java:355) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:354) at java.lang.ClassLoader.loadClass(ClassLoader.java:425) at java.net.FactoryURLClassLoader.loadClass(URLClassLoader.java:793) at java.lang.ClassLoader.loadClass(ClassLoader.java:358) ... 21 more ________________________________________________________ The information contained in this e-mail is confidential and/or proprietary to Capital One and/or its affiliates and may only be used solely in performance of work or services for Capital One. The information transmitted herewith is intended only for use by the individual or entity to which it is addressed. If the reader of this message is not the intended recipient, you are hereby notified that any review, retransmission, dissemination, distribution, copying or other use of, or taking of any action in reliance upon this information is strictly prohibited. If you have received this communication in error, please contact the sender and delete the material from your computer.
