strange. I had to do two things to solve this: 1. I added to core-site.xml the property fs.hdfs.impl = org.apache.hadoop.hdfs.DistributedFileSystem I don't really understand why this is needed, but it solved problems with not being able to use hdfs URLs when starting the local Twill Driver and also the on-cluster app.
2. I excluded snappy 1.0.4.1 from twill-core and added a new dependency on 1.0.5, as suggested by a sparse post on StackOverflow. Bernd On Mon, Dec 23, 2013 at 8:19 AM, Bernd Fondermann < [email protected]> wrote: > I've switched YARN log retention on and now see a lot of errors, starting > with java.lang.RuntimeException: java.io.IOException: No FileSystem for > scheme: hdfs > > Do I have a problem with libs or my env vars like HADOOP_CONF? > > Going one step back, how would a Twill job be properly packaged from > Maven? > > Currently, I'm using > maven-assembly-plugin:jar-with-dependencies > which builds one big jar with all libs, which ends up in a libs/ directory > in the twill-bundled jar. > > Bernd > > > > On Sat, Dec 21, 2013 at 11:36 AM, Bernd Fondermann < > [email protected]> wrote: > >> this is the exit log: >> >> 2013-12-17 16:06:48,470 INFO >> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: Application >> application_1387295722390_0002 failed 2 times due to AM Container for >> appattempt_1387295722390_0002_000002 exited with exitCode: 1 due to: >> Exception from container-launch: >> org.apache.hadoop.util.Shell$ExitCodeException: >> at org.apache.hadoop.util.Shell.runCommand(Shell.java:464) >> at org.apache.hadoop.util.Shell.run(Shell.java:379) >> at >> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589) >> at >> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195) >> at >> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:283) >> at >> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:79) >> at >> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) >> at java.util.concurrent.FutureTask.run(FutureTask.java:138) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) >> at java.lang.Thread.run(Thread.java:662) >> >> I'll try do more diagnostics... >> >> Bernd >> >> >> >> On Sat, Dec 21, 2013 at 7:14 AM, Terence Yim <[email protected]> wrote: >> >>> Hi Bernd, >>> >>> I never see this before. Do you know what's the exit code from the >>> YARN log? Currently there is no good way to stage an Twill app besides >>> running a unit test with a mini YARN cluster (see YarnTestSuite in the >>> Twill source). >>> >>> Terence >>> >>> On Wed, Dec 18, 2013 at 6:01 AM, Bernd Fondermann >>> <[email protected]> wrote: >>> > Hi everyone, >>> > >>> > I'm trying to run a Twill app on YARN. Attempts fail immediately with >>> > org.apache.hadoop.util.Shell$ExitCodeException. >>> > >>> > The rm logs report the launch cmd line as: >>> > java -Djava.io.tmpdir=tmp -Dyarn.appId=$YARN_APP_ID_STR >>> > -Dtwill.app=$TWILL_APP_NAME -cp launcher.jar:$HADOOP_CONF_DIR -Xmx362m >>> > org.apache.twill.launcher.TwillLauncher appMaster.jar >>> > org.apache.twill.internal.appmaster.ApplicationMasterMain false >>> > 1><LOG_DIR>/stdout 2><LOG_DIR>/stderr >>> > >>> > Any idea what the problem could be? >>> > Ist there a recommended way to stage a Twill app before running it on a >>> > remote cluster? >>> > >>> > Thanks, >>> > >>> > Bernd >>> >> >> >
