Hey Shekar, You need that samza-yarn file on your RM/NM's classpath, along with scala. We missed this in the docs, and are tracking the issue here:
https://issues.apache.org/jira/browse/SAMZA-456 You'll also need samza-core in the classpath, based on the discussion on SAMZA-456. Sorry about that. If you want to update the tutorial when you get your cluster working, and submit a patch, that'd be great! :) Cheers, Chris On Wed, Mar 11, 2015 at 9:43 PM, Shekar Tippur <ctip...@gmail.com> wrote: > Here is the corresponding log: > > 2015-03-11 20:43:09,665 INFO [AsyncDispatcher event handler] > localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource > http://sprfargas102:8000/hello-samza-0.8.0-dist.tar.gz transitioned from > INIT to DOWNLOADING > > 2015-03-11 20:43:09,665 INFO [AsyncDispatcher event handler] > localizer.ResourceLocalizationService > (ResourceLocalizationService.java:handle(679)) - Created localizer for > container_1426121400423_2587_01_000001 > > 2015-03-11 20:43:09,669 INFO [LocalizerRunner for > container_1426121400423_2587_01_000001] > localizer.ResourceLocalizationService > (ResourceLocalizationService.java:writeCredentials(1107)) - Writing > credentials to the nmPrivate file > > /tmp/hadoop-hadoop/nm-local-dir/nmPrivate/container_1426121400423_2587_01_000001.tokens. > Credentials list: > > 2015-03-11 20:43:09,675 INFO [DeletionService #0] > nodemanager.DefaultContainerExecutor > (DefaultContainerExecutor.java:deleteAsUser(378)) - Deleting path : > /home/hadoop/hadoop-2.5.2/logs/userlogs/application_1426120927668_0010 > > 2015-03-11 20:43:09,676 INFO [LocalizerRunner for > container_1426121400423_2587_01_000001] > nodemanager.DefaultContainerExecutor > (DefaultContainerExecutor.java:createUserCacheDirs(469)) - Initializing > user root > > 2015-03-11 20:43:09,685 INFO [LocalizerRunner for > container_1426121400423_2587_01_000001] > nodemanager.DefaultContainerExecutor > (DefaultContainerExecutor.java:startLocalizer(103)) - Copying from > > /tmp/hadoop-hadoop/nm-local-dir/nmPrivate/container_1426121400423_2587_01_000001.tokens > to > > /tmp/hadoop-hadoop/nm-local-dir/usercache/root/appcache/application_1426121400423_2587/container_1426121400423_2587_01_000001.tokens > > *2015-03-11 20:43:09,685 INFO [LocalizerRunner for > container_1426121400423_2587_01_000001] > nodemanager.DefaultContainerExecutor > (DefaultContainerExecutor.java:startLocalizer(105)) - CWD set to > > /tmp/hadoop-hadoop/nm-local-dir/usercache/root/appcache/application_1426121400423_2587 > = > > file:/tmp/hadoop-hadoop/nm-local-dir/usercache/root/appcache/application_1426121400423_2587* > > *2015-03-11 20:43:09,716 INFO [IPC Server handler 2 on 8040] > localizer.ResourceLocalizationService > (ResourceLocalizationService.java:update(1007)) - DEBUG: FAILED { > http://sprfargas102:8000/hello-samza-0.8.0-dist.tar.gz > <http://sprfargas102:8000/hello-samza-0.8.0-dist.tar.gz>, 0, ARCHIVE, null > }, java.lang.ClassNotFoundException: Class > org.apache.samza.util.hadoop.HttpFileSystem not found* > > *2015-03-11 20:43:09,716 INFO [IPC Server handler 2 on 8040] > localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource > http://sprfargas102:8000/hello-samza-0.8.0-dist.tar.gz(- > <http://sprfargas102:8000/hello-samza-0.8.0-dist.tar.gz(- > >>/tmp/hadoop-hadoop/nm-local-dir/usercache/root/appcache/application_1426121400423_2587/filecache/10/hello-samza-0.8.0-dist.tar.gz) > transitioned from DOWNLOADING to FAILED* > > 2015-03-11 20:43:09,717 INFO [AsyncDispatcher event handler] > container.Container (ContainerImpl.java:handle(918)) - Container > container_1426121400423_2587_01_000001 transitioned from LOCALIZING to > LOCALIZATION_FAILED > > 2015-03-11 20:43:09,717 INFO [AsyncDispatcher event handler] > localizer.LocalResourcesTrackerImpl > (LocalResourcesTrackerImpl.java:handle(151)) - Container > container_1426121400423_2587_01_000001 sent RELEASE event on a resource > request { http://sprfargas102:8000/hello-samza-0.8.0-dist.tar.gz, 0, > ARCHIVE, null } not present in cache. > > 2015-03-11 20:43:09,717 WARN [AsyncDispatcher event handler] > nodemanager.NMAuditLogger (NMAuditLogger.java:logFailure(150)) - > USER=root OPERATION=Container > Finished - Failed TARGET=ContainerImpl RESULT=FAILURE DESCRIPTION=Container > failed with state: LOCALIZATION_FAILED APPID=application_1426121400423_2587 > CONTAINERID=container_1426121400423_2587_01_000001 > > 2015-03-11 20:43:09,717 INFO [AsyncDispatcher event handler] > container.Container (ContainerImpl.java:handle(918)) - Container > container_1426121400423_2587_01_000001 transitioned from > LOCALIZATION_FAILED to DONE > > 2015-03-11 20:43:09,717 INFO [AsyncDispatcher event handler] > application.Application (ApplicationImpl.java:transition(340)) - Removing > container_1426121400423_2587_01_000001 from application > application_1426121400423_2587 > > 2015-03-11 20:43:09,717 INFO [AsyncDispatcher event handler] > containermanager.AuxServices (AuxServices.java:handle(196)) - Got event > CONTAINER_STOP for appId application_1426121400423_2587 > > 2015-03-11 20:43:09,717 INFO [DeletionService #2] > nodemanager.DefaultContainerExecutor > (DefaultContainerExecutor.java:deleteAsUser(369)) - Deleting absolute path > : > > /tmp/hadoop-hadoop/nm-local-dir/usercache/root/appcache/application_1426121400423_2587/container_1426121400423_2587_01_000001 > > 2015-03-11 20:43:09,717 WARN [DeletionService #2] > nodemanager.DefaultContainerExecutor > (DefaultContainerExecutor.java:deleteAsUser(372)) - delete returned false > for path: > > [/tmp/hadoop-hadoop/nm-local-dir/usercache/root/appcache/application_1426121400423_2587/container_1426121400423_2587_01_000001] > > 2015-03-11 20:43:09,718 WARN [LocalizerRunner for > container_1426121400423_2587_01_000001] ipc.Client (Client.java:call(1389)) > - interrupted waiting to send rpc request to server > > java.lang.InterruptedException > > at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:400) > > at java.util.concurrent.FutureTask.get(FutureTask.java:187) > > at org.apache.hadoop.ipc.Client$Connection.sendRpcRequest(Client.java:1030) > > at org.apache.hadoop.ipc.Client.call(Client.java:1384) > > at org.apache.hadoop.ipc.Client.call(Client.java:1364) > > at > > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206) > > at com.sun.proxy.$Proxy29.heartbeat(Unknown Source) > > at > > org.apache.hadoop.yarn.server.nodemanager.api.impl.pb.client.LocalizationProtocolPBClientImpl.heartbeat(LocalizationProtocolPBClientImpl.java:62) > > at > > org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.localizeFiles(ContainerLocalizer.java:255) > > at > > org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.runLocalization(ContainerLocalizer.java:169) > > at > > org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.startLocalizer(DefaultContainerExecutor.java:107) > > at > > org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1073) > > java.io.IOException: java.lang.InterruptedException > > at org.apache.hadoop.ipc.Client.call(Client.java:1390) > > at org.apache.hadoop.ipc.Client.call(Client.java:1364) > > at > > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206) > > at com.sun.proxy.$Proxy29.heartbeat(Unknown Source) > > at > > org.apache.hadoop.yarn.server.nodemanager.api.impl.pb.client.LocalizationProtocolPBClientImpl.heartbeat(LocalizationProtocolPBClientImpl.java:62) > > at > > org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.localizeFiles(ContainerLocalizer.java:255) > > at > > org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.runLocalization(ContainerLocalizer.java:169) > > at > > org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.startLocalizer(DefaultContainerExecutor.java:107) > > at > > org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1073) > > Caused by: java.lang.InterruptedException > > at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:400) > > at java.util.concurrent.FutureTask.get(FutureTask.java:187) > > at org.apache.hadoop.ipc.Client$Connection.sendRpcRequest(Client.java:1030) > > at org.apache.hadoop.ipc.Client.call(Client.java:1384) > > ... 8 more > > On Wed, Mar 11, 2015 at 4:56 PM, Shekar Tippur <ctip...@gmail.com> wrote: > > > Hello, > > > > Sorry to reopen this topic. I had setup yarn couple of months ago and > cant > > seem to replicate this now. > > > > I see that I have done everything listed here > > > http://samza.apache.org/learn/tutorials/0.7.0/run-in-multi-node-yarn.html > > > > I see this error on the application side > > > > Application application_1426115467623_0492 failed 2 times due to AM > > Container for appattempt_1426115467623_0492_000002 exited with exitCode: > > -1000 due to: java.lang.ClassNotFoundException: Class > > org.apache.samza.util.hadoop.HttpFileSystem not found > > .Failing this attempt.. Failing the application. > > > > I see that > > > > /home/hadoop/hadoop-2.5.2/share/hadoop/hdfs/lib/samza-yarn_2.10-0.8.0.jar > > has that particular class > > > > 1739 Tue Nov 25 10:51:40 PST 2014 > > > org/apache/samza/util/hadoop/HttpFileSystem$$anonfun$getFileStatus$1.class > > > > 1570 Tue Nov 25 10:51:40 PST 2014 > > org/apache/samza/util/hadoop/HttpFileSystem$$anonfun$initialize$1.class > > > > 1597 Tue Nov 25 10:51:40 PST 2014 > > org/apache/samza/util/hadoop/HttpFileSystem$$anonfun$open$1.class > > > > 1797 Tue Nov 25 10:51:40 PST 2014 > > org/apache/samza/util/hadoop/HttpFileSystem$$anonfun$open$2.class > > > > 9549 Tue Nov 25 10:51:40 PST 2014 > > org/apache/samza/util/hadoop/HttpFileSystem.class > > > > > > I see that env is set right: > > > > > > HADOOP_YARN_HOME=/home/hadoop/hadoop-2.5.2 > > > > HADOOP_CONF_DIR=/home/hadoop/hadoop-2.5.2/conf > > > > Wondering if I am missing anything... > > - Shekar > > >
