HI Sonali, Try look into the logs folder of YARN (should be something like $YARN_HOME/logs/userlogs/application-xxxxx/container-yyyyyy/ and check the logs in those folder. There are probably some file like samza-container-zzzz.logs. Those are the logs from the actual container and should give me information about the error.
Cheers, Casey On Thu, Mar 13, 2014 at 8:56 PM, <[email protected]> wrote: > Hey Chris, > > I get the same error in the NM logs > 2014-03-13 11:44:14,210 INFO > org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: > Memory usage of ProcessTree 16883 for container-id > container_1391637982288_0039_01_000001: 45.3 MB of 1 GB physical memory > used; 1.3 GB of 2.1 GB virtual memory used > 2014-03-13 11:44:15,144 WARN > org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exit > code from container container_1391637982288_0039_01_000001 is : 1 > 2014-03-13 11:44:15,144 WARN > org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: > Exception from container-launch with container ID: > container_1391637982288_0039_01_000001 and exit code: 1 > org.apache.hadoop.util.Shell$ExitCodeException: > at org.apache.hadoop.util.Shell.runCommand(Shell.java:464) > at org.apache.hadoop.util.Shell.run(Shell.java:379) > at > org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589) > at > org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:283) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:79) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) > 2014-03-13 11:44:15,144 INFO > org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: > 2014-03-13 11:44:15,145 WARN > org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch: > Container exited with a non-zero exit code 1 > 2014-03-13 11:44:15,145 INFO > org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container: > Container container_1391637982288_0039_01_000001 transitioned from RUNNING > to EXITED_WITH_FAILURE > > -----Original Message----- > From: Chris Riccomini [mailto:[email protected]] > Sent: Thursday, March 13, 2014 12:54 PM > To: [email protected] > Subject: Re: Failed to package using mvn > > Hey Sonali, > > Yep, if you applied them both, you should be OK. Though, you still > shouldn't need to add a yarn.rm.hostname to your configuration. The > SamzaAppMaster should automatically pick that up from the yarn-site.xml on > your NM. > > In any case, what does your NM log say? > > Cheers, > Chris > > On 3/13/14 12:51 PM, "[email protected]" > <[email protected]> wrote: > > >Ah I see. I applied them both. That shouldn't matter right? > > > >-----Original Message----- > >From: Chris Riccomini [mailto:[email protected]] > >Sent: Thursday, March 13, 2014 12:38 PM > >To: [email protected] > >Subject: Re: Failed to package using mvn > > > >Hey Sonali, > > > >You should not have to add yarn.rm.hostname to your configuration. You > >have applied the wrong patch. Please apply this patch: > > > > > >https://issues.apache.org/jira/secure/attachment/12634493/SAMZA-182.1.p > >atc > >h > > > > > >Do not apply > >https://issues.apache.org/jira/secure/attachment/12634200/yarn_remote_r > >m.p > >a > >tch, which is the one that adds yarn.rm.hostname. > > > >Regarding the failed containers, can you take a look at the NM's logs, > >and see what it says there? > > > >Cheers, > >Chris > > > >On 3/13/14 12:00 PM, "[email protected]" > ><[email protected]> wrote: > > > >>Hi Chris. > >> > >>I installed the patch and was able to submit the job. (I had to add > >>yarn.rm.hostname in the wikipedia-feed.properties file). The job > >>started running and then I got this error: > >>SamzaAppMasterTaskManager [INFO] Got an exit code of -100. This means > >>that container container_1391637982288_0040_01_000043 was killed by > >>YARN, either due to being released by the application master or being > >>'lost' > >>due to node failures etc. > >>2014-03-13 11:53:03 SamzaAppMasterTaskManager [INFO] Got an exit code > >>of -100. This means that container > >>container_1391637982288_0040_01_000044 > >>was killed by YARN, either due to being released by the application > >>master or being 'lost' due to node failures etc. > >>2014-03-13 11:53:03 SamzaAppMasterTaskManager [INFO] Got an exit code > >>of -100. This means that container > >>container_1391637982288_0040_01_000045 > >>was killed by YARN, either due to being released by the application > >>master or being 'lost' due to node failures etc. > >>2014-03-13 11:53:03 SamzaAppMasterTaskManager [INFO] Got a container > >>from YARN ResourceManager: Container: [ContainerId: > >>container_1391637982288_0040_01_000046, NodeId: svdpdac015:37919, > >>NodeHttpAddress: svdpdac015:8042, Resource: <memory:1024, vCores:1>, > >>Priority: 0, Token: Token { kind: ContainerToken, service: > >>10.1.174.84:37919 }, ] > >>2014-03-13 11:53:03 SamzaAppMasterTaskManager [INFO] Got an extra > >>container from YARN ResourceManager: Container: [ContainerId: > >>container_1391637982288_0040_01_000046, NodeId: svdpdac015:37919, > >>NodeHttpAddress: svdpdac015:8042, Resource: <memory:1024, vCores:1>, > >>Priority: 0, Token: Token { kind: ContainerToken, service: > >>10.1.174.84:37919 }, ] > >>2014-03-13 11:53:04 SamzaAppMasterTaskManager [INFO] Got an exit code > >>of -100. This means that container > >>container_1391637982288_0040_01_000046 > >>was killed by YARN, either due to being released by the application > >>master or being 'lost' due to node failures etc. > >>2014-03-13 11:53:24 SamzaAppMasterTaskManager [INFO] Container > >>container_1391637982288_0040_01_000038 failed with exit code 1 - > >>Exception from container-launch: > >>org.apache.hadoop.util.Shell$ExitCodeException: > >> at org.apache.hadoop.util.Shell.runCommand(Shell.java:464) > >> at org.apache.hadoop.util.Shell.run(Shell.java:379) > >> at > >>org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589) > >> at > >>org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.lau > >>n > >>chC > >>ontainer(DefaultContainerExecutor.java:195) > >> at > >>org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.Co > >>n > >>tai > >>nerLaunch.call(ContainerLaunch.java:283) > >> at > >>org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.Co > >>n > >>tai > >>nerLaunch.call(ContainerLaunch.java:79) > >> at java.util.concurrent.FutureTask.run(FutureTask.java:262) > >> at > >>java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.j > >>ava > >>: > >>1145) > >> at > >>java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor. > >>j > >>ava > >>:615) > >> at java.lang.Thread.run(Thread.java:744) > >> > >> > >>Now, I'm running Hadoop-2.2.0 version of yarn. Do I need to change to > >>YARN 2.3? > >> > >>-----Original Message----- > >>From: Chris Riccomini [mailto:[email protected]] > >>Sent: Thursday, March 13, 2014 11:12 AM > >>To: [email protected] > >>Subject: Re: Failed to package using mvn > >> > >>Hey Sonali, > >> > >>As a follow on, here is EXACTLY what I did: > >> > >># start by downloading YARN and setting it up 1. Download YARN 2.3 > >>from > >>http://mirror.symnds.com/software/Apache/hadoop/common/hadoop-2.3.0/ha > >>d > >>oop > >>- > >>2.3.0.tar.gz to /tmp > >>2. cd /tmp > >>3. tar -xvf hadoop-2.3.0.tar.gz > >>4. cd hadoop-2.3.0 > >>5. export HADOOP_YARN_HOME=$(pwd) > >>6. mkdir conf > >>7. export HADOOP_CONF_DIR=$HADOOP_YARN_HOME/conf > >>8. cp ./etc/hadoop/yarn-site.xml conf > >>9. vi conf/yarn-site.xml > >>10. Add this property to yarn-site.xml: > >> > >> <property> > >> <name>yarn.resourcemanager.hostname</name> > >> <!-- hostname that is accessible from all NMs --> > >> <value>criccomi-mn</value> > >> </property> > >> > >> > >>11. curl > >>http://svn.apache.org/viewvc/hadoop/common/trunk/hadoop-yarn-project/h > >>a > >>doo > >>p > >>-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/resources/c > >>a > >>pac > >>i > >>ty-scheduler.xml?view=co > conf/capacity-scheduler.xml > >> > >># setup http filesystem for YARN (you can skip this and follow > >>SAMZA-181 instead, if you are using HDFS) 12. cd /tmp 13. curl > >>http://www.scala-lang.org/files/archive/scala-2.10.3.tgz > > >>scala-2.10.3.tgz 14. tar -xvf scala-2.10.3.tgz 15. cp > >>/tmp/scala-2.10.3/lib/scala-compiler.jar > >>$HADOOP_YARN_HOME/share/hadoop/hdfs/lib > >>16. cp /tmp/scala-2.10.3/lib/scala-library.jar > >>$HADOOP_YARN_HOME/share/hadoop/hdfs/lib > >>17. curl > >>http://search.maven.org/remotecontent?filepath=org/clapper/grizzled-sl > >>f > >>4j_ > >>2 > >>.10/1.0.1/grizzled-slf4j_2.10-1.0.1.jar > > >>$HADOOP_YARN_HOME/share/hadoop/hdfs/lib/grizzled-slf4j_2.10-1.0.1.jar > >>18. vi $HADOOP_YARN_HOME/conf/core-site.xml > >> > >><?xml-stylesheet type="text/xsl" href="configuration.xsl"?> > >><configuration> <property> <name>fs.http.impl</name> > >><value>org.apache.samza.util.hadoop.HttpFileSystem</value> > >></property> > >></configuration> > >> > >> > >>19. Copy the Hadoop directory to all slave nodes (172.21.100.35, in my > >>case): > >> > >> scp -r . 172.21.100.35:/tmp/hadoop-2.3.0 > >> > >>20. echo 172.21.100.35 > conf/slaves > >>21. sbin/start-yarn.sh > >>22. If you get "172.21.100.35: Error: JAVA_HOME is not set and could > >>not be found.", you'll need to add a conf/hadoop-env.sh file to the > >>machine with the failure (172.21.100.35, in this case), which has > >>"export JAVA_HOME=/export/apps/jdk/JDK-1_6_0_27" (or wherever your > >>JAVA_HOME actually is). > >>23. Validate that your nodes are up by visiting > >>http://criccomi-mn:8088/cluster/nodes > >> > >> > >># now we more or less follow the hello-samza steps. > >>24. cd /tmp > >>25. git clone > >>http://git-wip-us.apache.org/repos/asf/incubator-samza.git > >>26. cd incubator-samza > >>27. curl > >>https://issues.apache.org/jira/secure/attachment/12634493/SAMZA-182.1. > >>p > >>atc > >>h > >> > SAMZA-182.1.patch > >>28. git apply SAMZA-182.1.patch > >>29. ./gradlew clean publishToMavenLocal 30. cd .. > >>31. git clone git://github.com/linkedin/hello-samza.git > >>32. cd hello-samza > >>33. vi samza-job-package/src/main/config/wikipedia-feed.properties > >>34. Change the yarn.package.path property to be: > >> > >> > >>yarn.package.path=http://criccomi-mn:8000/samza-job-package/target/sam > >>z > >>a-j > >>o > >>b-package-0.7.0-dist.tar.gz > >> > >> > >>35. mvn clean package > >>36. mkdir -p deploy/samza > >>37. tar -xvf > >>./samza-job-package/target/samza-job-package-0.7.0-dist.tar.gz -C > >>deploy/samza 38. Open a new terminal, and cd /tmp/hello-samza && > >>python -m SimpleHTTPServer 39. Go back to the original terminal (not > >>the one running the HTTP server) 40. deploy/samza/bin/run-job.sh > >>--config-factory=org.apache.samza.config.factories.PropertiesConfigFac > >>t > >>ory > >>--config-path=file://$PWD/deploy/samza/config/wikipedia-feed.propertie > >>s 41. Go to http://criccomi-mn:8088 and find the wikipedia-feed job. > >>Click on the ApplicationMaster link to see that it's running. > >> > >> > >>I plan to write a tutorial that formalizes this. > >> > >>Cheers, > >>Chris > >> > >>On 3/13/14 11:03 AM, "Chris Riccomini" <[email protected]> wrote: > >> > >>>Hey Sonali, > >>> > >>>Please have a look at: > >>> > >>> https://issues.apache.org/jira/browse/SAMZA-182 > >>> > >>>I have posted a fix there. I have successfully downloaded YARN, setup > >>>a two node grid from scratch, and run hello-samza on it with the > >>>patch on SAMZA-182. Can you give that a shot for me? > >>> > >>>Thanks for your patience! > >>> > >>>Cheers, > >>>Chris > >>> > >>>On 3/13/14 10:58 AM, "[email protected]" > >>><[email protected]> wrote: > >>> > >>>>Hi Chris, > >>>> > >>>>I checked my .bashrc. The variable was set on one of the NMs and not > >>>>on the other. I made the change and restarted the scripts. I still > >>>>get the same error. > >>>> > >>>>Also in my stderr I get: > >>>>Null identity service, trying login service: null Finding identity > >>>>service: null Null identity service, trying login service: null > >>>>Finding identity service: null > >>>> > >>>>-----Original Message----- > >>>>From: Chris Riccomini [mailto:[email protected]] > >>>>Sent: Wednesday, March 12, 2014 7:59 PM > >>>>To: [email protected] > >>>>Subject: Re: Failed to package using mvn > >>>> > >>>>Hey Guys, > >>>> > >>>>I was able to reproduce this problem. > >>>> > >>>>I was also able to fix it (without the patch in SAMZA-182). All I > >>>>needed to do was update ~/.bashrc on my NM's box to have: > >>>> > >>>> export YARN_HOME=/tmp/hadoop-2.3.0 > >>>> > >>>>It appears that the YARN environment variables are somehow getting > >>>>lost or not forwarded from the NM to the AM. Adding this bashrc > >>>>setting makes sure that the NM gets them. > >>>> > >>>> > >>>>I have a feeling upgrading Samza to YARN 2.3.0 will fix this, but I > >>>>haven't validated yet. I will continue to investigate tomorrow. > >>>> > >>>>Cheers, > >>>>Chris > >>>> > >>>>On 3/12/14 6:43 PM, "Yan Fang" <[email protected]> wrote: > >>>> > >>>>>I guess Sonali has the problem is because his NMs do not read the > >>>>>YARN_HOME variable. That may be because the NM machine does not > >>>>>have YARN_HOME set when the NM starts. > >>>>> > >>>>>Check this https://issues.apache.org/jira/browse/SAMZA-182 > >>>>> > >>>>>Thanks, > >>>>> > >>>>>Yan Fang > >>>>> > >>>>>> On Mar 12, 2014, at 6:14 PM, Chris Riccomini > >>>>>><[email protected]> > >>>>>>wrote: > >>>>>> > >>>>>> Hey Sonali, > >>>>>> > >>>>>> I am unfamiliar with the start-yarn.sh. Looking at: > >>>>>> > >>>>>> > >>>>>> > >>>>>>https://svn.apache.org/viewvc/hadoop/common/trunk/hadoop-yarn-proj > >>>>>>e > >>>>>>ct/ > >>>>>>had > >>>>>>oo > >>>>>> p-yarn/bin/stop-yarn.sh?revision=1370666&view=markup > >>>>>> > >>>>>> What version of YARN are you using? > >>>>>> > >>>>>> Cheers, > >>>>>> Chris > >>>>>> > >>>>>> On 3/12/14 5:56 PM, "[email protected]" > >>>>>> <[email protected]> wrote: > >>>>>> > >>>>>>> Hey Chris, > >>>>>>> > >>>>>>> Yes, I have YARN_HOME set in all the NMs pointing to the right > >>>>>>>directories. I also made sure the yarn-site.xml file has the > >>>>>>>hostname set. > >>>>>>> > >>>>>>> I start yarn using start.yarn.sh in the RM and that > >>>>>>>automatically starts the NMs on the slave nodes. Is that the right > way to do it? > >>>>>>> > >>>>>>> -----Original Message----- > >>>>>>> From: Chris Riccomini [mailto:[email protected]] > >>>>>>> Sent: Wednesday, March 12, 2014 5:52 PM > >>>>>>> To: [email protected] > >>>>>>> Subject: Re: Failed to package using mvn > >>>>>>> > >>>>>>> Hey Sonali, > >>>>>>> > >>>>>>> OK, so we've validated that the NMs are able to connect, which > >>>>>>>means they can see the yarn-site.xml. > >>>>>>> > >>>>>>> How are you starting your NMs? Are you running: > >>>>>>> > >>>>>>> export YARN_HOME=/path/to/yarn/home > >>>>>>> > >>>>>>> In the CLI before starting the NM? > >>>>>>> > >>>>>>> For reference, we run: > >>>>>>> > >>>>>>> export YARN_HOME=/path/to/our/yarn-home export > >>>>>>> YARN_CONF_DIR=$YARN_HOME/conf > >>>>>>> > >>>>>>> bin/yarn nodemanager > >>>>>>> > >>>>>>> With YARN_HOME pointing to a directory that has a subdirectory > >>>>>>> called "conf" in it, which has a yarn-site.xml in it: > >>>>>>> > >>>>>>> /path/to/our/yarn-home/conf/yarn-site.xml > >>>>>>> > >>>>>>> This yarn-site.xml has yarn.resourcemanager.hostname set to the > >>>>>>> IP (or > >>>>>>> hostname) of the resource manager: > >>>>>>> > >>>>>>> <property> > >>>>>>> <name>yarn.resourcemanager.hostname</name> > >>>>>>> <value>123.456.789.123</value> > >>>>>>> </property> > >>>>>>> > >>>>>>> > >>>>>>> Cheers, > >>>>>>> Chris > >>>>>>> > >>>>>>> On 3/12/14 5:33 PM, "[email protected]" > >>>>>>> <[email protected]> wrote: > >>>>>>> > >>>>>>>> I see two active nodes (I have 2 NMs running) > >>>>>>>> > >>>>>>>> -----Original Message----- > >>>>>>>> From: Chris Riccomini [mailto:[email protected]] > >>>>>>>> Sent: Wednesday, March 12, 2014 5:24 PM > >>>>>>>> To: [email protected] > >>>>>>>> Subject: Re: Failed to package using mvn > >>>>>>>> > >>>>>>>> Hey Sonali, > >>>>>>>> > >>>>>>>> Can you go to your ResourceManager's UI, and tell me how many > >>>>>>>>active nodes you see? This should be under the "active nodes" > >>>>>>>>heading. > >>>>>>>> > >>>>>>>> It sounds like the SamzaAppMaster is not getting the resource > >>>>>>>> manager host/port from the yarn-site.xml. Usually this is due > >>>>>>>> to not exporting YARN_HOME on the NodeManager before starting it. > >>>>>>>> > >>>>>>>> Cheers, > >>>>>>>> Chris > >>>>>>>> > >>>>>>>> On 3/12/14 5:21 PM, "[email protected]" > >>>>>>>> <[email protected]> wrote: > >>>>>>>> > >>>>>>>>> Okay so I was able to submit the job: > >>>>>>>>> > >>>>>>>>> In the nodemanager I get this error: Specifically it's trying > >>>>>>>>> to connect to 0.0.0.0/8032 instead of the IP I have specified > >>>>>>>>> in the yarn-site.xml file > >>>>>>>>> > >>>>>>>>> 2014-03-12 17:04:47 SamzaAppMaster$ [INFO] got container id: > >>>>>>>>> container_1391637982288_0033_01_000001 > >>>>>>>>> 2014-03-12 17:04:47 SamzaAppMaster$ [INFO] got app attempt id: > >>>>>>>>> appattempt_1391637982288_0033_000001 > >>>>>>>>> 2014-03-12 17:04:47 SamzaAppMaster$ [INFO] got node manager host: > >>>>>>>>> svdpdac001.techlabs.accenture.com > >>>>>>>>> 2014-03-12 17:04:47 SamzaAppMaster$ [INFO] got node manager port: > >>>>>>>>> 38218 > >>>>>>>>> 2014-03-12 17:04:47 SamzaAppMaster$ [INFO] got node manager > >>>>>>>>>http > >>>>>>>>>port: > >>>>>>>>> 8042 > >>>>>>>>> 2014-03-12 17:04:47 SamzaAppMaster$ [INFO] got config: > >>>>>>>>> > >>>>>>>>>{task.inputs=wikipedia.#en.wikipedia,wikipedia.#en.wiktionary,w > >>>>>>>>>i > >>>>>>>>>kip edi a .#e n.wikinews, > >>>>>>>>>systems.wikipedia.host=irc.wikimedia.org, > >>>>>>>>> systems.kafka.producer.batch.num.messages=1, > >>>>>>>>> job.factory.class=org.apache.samza.job.yarn.YarnJobFactory, > >>>>>>>>> systems.wikipedia.port=6667, > >>>>>>>>> systems.kafka.producer.producer.type=sync, > >>>>>>>>> job.name=wikipedia-feed, > >>>>>>>>> > >>>>>>>>>systems.kafka.consumer.zookeeper.connect=svdpdac013.techlabs.ac > >>>>>>>>>c > >>>>>>>>>ent > >>>>>>>>>ure > >>>>>>>>>. > >>>>>>>>> com :2181/, systems.kafka.samza.msg.serde=json, > >>>>>>>>> > >>>>>>>>>serializers.registry.json.class=org.apache.samza.serializers.Js > >>>>>>>>>o > >>>>>>>>>nSe > >>>>>>>>>rde > >>>>>>>>> F > >>>>>>>>> act ory, > >>>>>>>>> > >>>>>>>>>task.class=samza.examples.wikipedia.task.WikipediaFeedStreamTas > >>>>>>>>>k > >>>>>>>>>, > >>>>>>>>> > >>>>>>>>>yarn.package.path=hdfs://10.1.174.85:9000/samza-job-package-0.7. > >>>>>>>>>0-d > >>>>>>>>>ist > >>>>>>>>>. > >>>>>>>>> tar > >>>>>>>>> .gz, > >>>>>>>>> > >>>>>>>>>systems.wikipedia.samza.factory=samza.examples.wikipedia.system. > >>>>>>>>>Wik > >>>>>>>>>ipe > >>>>>>>>> d > >>>>>>>>> iaS > >>>>>>>>> ystemFactory, > >>>>>>>>> > >>>>>>>>>systems.kafka.samza.factory=org.apache.samza.system.kafka.Kafka > >>>>>>>>>S > >>>>>>>>>yst > >>>>>>>>>emF > >>>>>>>>> a > >>>>>>>>> cto > >>>>>>>>> ry, > >>>>>>>>> > >>>>>>>>>systems.kafka.producer.metadata.broker.list=svdpdac001.techlabs. > >>>>>>>>>acc > >>>>>>>>>ent > >>>>>>>>> ure > >>>>>>>>> . > >>>>>>>>> com:6667,svdpdac015.techlabs.accenture.com:6667} > >>>>>>>>> 2014-03-12 17:04:48 ClientHelper [INFO] trying to connect to > >>>>>>>>>RM > >>>>>>>>> 0.0.0.0:8032 > >>>>>>>>> 2014-03-12 17:04:48 NativeCodeLoader [WARN] Unable to load > >>>>>>>>>native-hadoop library for your platform... using builtin-java > >>>>>>>>>classes where applicable > >>>>>>>>> 2014-03-12 17:04:48 RMProxy [INFO] Connecting to > >>>>>>>>>ResourceManager at > >>>>>>>>> /0.0.0.0:8032 > >>>>>>>>> > >>>>>>>>> -----Original Message----- > >>>>>>>>> From: Chris Riccomini [mailto:[email protected]] > >>>>>>>>> Sent: Wednesday, March 12, 2014 4:48 PM > >>>>>>>>> To: [email protected] > >>>>>>>>> Subject: Re: Failed to package using mvn > >>>>>>>>> > >>>>>>>>> Hey Sonali, > >>>>>>>>> > >>>>>>>>> You need to specify a valid HDFS uri. Usually something like: > >>>>>>>>> > >>>>>>>>> hdfs://<hdfs name node ip>:<hdfs name node port>/path/to/tgz > >>>>>>>>> > >>>>>>>>> Right now, Hadoop is trying to use the package name as the > >>>>>>>>>HDFS host. > >>>>>>>>> > >>>>>>>>> Cheers, > >>>>>>>>> Chris > >>>>>>>>> > >>>>>>>>> On 3/12/14 4:45 PM, "[email protected]" > >>>>>>>>> <[email protected]> wrote: > >>>>>>>>> > >>>>>>>>>> I did and I can now see the hadoop-hdfs jar in > >>>>>>>>>> /deploy/samza/lib folder. > >>>>>>>>>> > >>>>>>>>>> I do get a different error now. > >>>>>>>>>> > >>>>>>>>>> I uploaded the samza-job to hdfs and it resides on > >>>>>>>>>> hdfs://samza-job-package-0.7.0-dist.tar.gz > >>>>>>>>>> > >>>>>>>>>> But when I run the job I get this exception: > >>>>>>>>>> > >>>>>>>>>> Exception in thread "main" java.lang.IllegalArgumentException: > >>>>>>>>>> java.net.UnknownHostException: > >>>>>>>>>>samza-job-package-0.7.0-dist.tar.gz > >>>>>>>>>> at > >>>>>>>>>> > >>>>>>>>>>org.apache.hadoop.security.SecurityUtil.buildTokenService(Secu > >>>>>>>>>>r > >>>>>>>>>>ity > >>>>>>>>>>Uti > >>>>>>>>>>l. > >>>>>>>>>> jav > >>>>>>>>>> a:418) > >>>>>>>>>> at > >>>>>>>>>> > >>>>>>>>>>org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(NameNo > >>>>>>>>>>d > >>>>>>>>>>ePr > >>>>>>>>>>oxi > >>>>>>>>>> e > >>>>>>>>>> s > >>>>>>>>>> .ja > >>>>>>>>>> va:231) > >>>>>>>>>> at > >>>>>>>>>> > >>>>>>>>>>org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodePro > >>>>>>>>>>x > >>>>>>>>>>ies > >>>>>>>>>>.ja > >>>>>>>>>> v > >>>>>>>>>> a > >>>>>>>>>> :13 > >>>>>>>>>> 9) > >>>>>>>>>> at > >>>>>>>>>>org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:510) > >>>>>>>>>> at > >>>>>>>>>>org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:453) > >>>>>>>>>> at > >>>>>>>>>> > >>>>>>>>>>org.apache.hadoop.hdfs.DistributedFileSystem.initialize(Distri > >>>>>>>>>>b > >>>>>>>>>>ute > >>>>>>>>>>dFi > >>>>>>>>>> l > >>>>>>>>>> e > >>>>>>>>>> Sys > >>>>>>>>>> tem.java:136) > >>>>>>>>>> at > >>>>>>>>>> > >>>>>>>>>>org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.ja > >>>>>>>>>>v > >>>>>>>>>>a:2 > >>>>>>>>>>433 > >>>>>>>>>>) > >>>>>>>>>> at > >>>>>>>>>> org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:88) > >>>>>>>>>> at > >>>>>>>>>> > >>>>>>>>>>org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.j > >>>>>>>>>>ava > >>>>>>>>>>: > >>>>>>>>>>246 > >>>>>>>>>>7) > >>>>>>>>>> at > >>>>>>>>>> org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2449) > >>>>>>>>>> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:367) > >>>>>>>>>> at org.apache.hadoop.fs.Path.getFileSystem(Path.java:287) > >>>>>>>>>> at > >>>>>>>>>> > >>>>>>>>>>org.apache.samza.job.yarn.ClientHelper.submitApplication(Clien > >>>>>>>>>>t > >>>>>>>>>>Hel > >>>>>>>>>>per > >>>>>>>>>>. > >>>>>>>>>> s > >>>>>>>>>> cal > >>>>>>>>>> a:111) > >>>>>>>>>> at > >>>>>>>>>>org.apache.samza.job.yarn.YarnJob.submit(YarnJob.scala:55) > >>>>>>>>>> at > >>>>>>>>>>org.apache.samza.job.yarn.YarnJob.submit(YarnJob.scala:48) > >>>>>>>>>> at org.apache.samza.job.JobRunner.run(JobRunner.scala:100) > >>>>>>>>>> at org.apache.samza.job.JobRunner$.main(JobRunner.scala:75) > >>>>>>>>>> at org.apache.samza.job.JobRunner.main(JobRunner.scala) > >>>>>>>>>> Caused by: java.net.UnknownHostException: > >>>>>>>>>> samza-job-package-0.7.0-dist.tar.gz > >>>>>>>>>> ... 18 more > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> -----Original Message----- > >>>>>>>>>> From: Yan Fang [mailto:[email protected]] > >>>>>>>>>> Sent: Wednesday, March 12, 2014 4:20 PM > >>>>>>>>>> To: [email protected] > >>>>>>>>>> Subject: Re: Failed to package using mvn > >>>>>>>>>> > >>>>>>>>>> Hi Sonali, > >>>>>>>>>> > >>>>>>>>>> One tip you may miss: > >>>>>>>>>> > >>>>>>>>>> If you had already run > >>>>>>>>>> > >>>>>>>>>> tar -xvf > >>>>>>>>>> ./samza-job-package/target/samza-job-package-0.7.0-dist.tar.g > >>>>>>>>>> z > >>>>>>>>>> -C deploy/samza > >>>>>>>>>> > >>>>>>>>>> before you bundled the jar file to tar.gz. Please also > >>>>>>>>>> remember to put the hdfs jar file to the deploy/samza/lib. > >>>>>>>>>> > >>>>>>>>>> Let me know if you miss this step. > >>>>>>>>>> > >>>>>>>>>> Thanks, > >>>>>>>>>> > >>>>>>>>>> Fang, Yan > >>>>>>>>>> [email protected] > >>>>>>>>>> +1 (206) 849-4108 > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> On Wed, Mar 12, 2014 at 4:10 PM, Chris Riccomini > >>>>>>>>>> <[email protected]>wrote: > >>>>>>>>>> > >>>>>>>>>>> Hey Sonali, > >>>>>>>>>>> > >>>>>>>>>>> Yan has made a step-by-step tutorial for this. Could you > >>>>>>>>>>>confirm that you've followed the instructions, and it's > >>>>>>>>>>>still not working? > >>>>>>>>>>> > >>>>>>>>>>> https://issues.apache.org/jira/browse/SAMZA-181 > >>>>>>>>>>> > >>>>>>>>>>> Cheers, > >>>>>>>>>>> Chris > >>>>>>>>>>> > >>>>>>>>>>> On 3/12/14 3:12 PM, "[email protected]" > >>>>>>>>>>> <[email protected]> wrote: > >>>>>>>>>>> > >>>>>>>>>>>> So sigh! I had some Kafka issues in-between. That's fixed now. > >>>>>>>>>>>> > >>>>>>>>>>>> As suggested, > >>>>>>>>>>>> > >>>>>>>>>>>> 1. I made sure the hadoop-hdfs-2.2.0.jar is bundled with > >>>>>>>>>>>> the samza job tar.gz. > >>>>>>>>>>>> 2. I added the configuration to implement hdfs in the > >>>>>>>>>>>> hdfs-site.xml files both on the NMs and in the /conf > >>>>>>>>>>>> directory for samza > >>>>>>>>>>>> > >>>>>>>>>>>> I still get the No Filesystem for scheme :hdfs error. > >>>>>>>>>>>> > >>>>>>>>>>>> Is there anything else im missing? > >>>>>>>>>>>> Thanks, > >>>>>>>>>>>> Sonali > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> -----Original Message----- > >>>>>>>>>>>> From: Chris Riccomini [mailto:[email protected]] > >>>>>>>>>>>> Sent: Tuesday, March 11, 2014 8:27 PM > >>>>>>>>>>>> To: [email protected] > >>>>>>>>>>>> Subject: Re: Failed to package using mvn > >>>>>>>>>>>> > >>>>>>>>>>>> Hey Yan, > >>>>>>>>>>>> > >>>>>>>>>>>> This looks great! I added a few requests to the JIRA, if > >>>>>>>>>>>> you have > >>>>>>>>>>> time. > >>>>>>>>>>>> > >>>>>>>>>>>> Cheers, > >>>>>>>>>>>> Chris > >>>>>>>>>>>> > >>>>>>>>>>>>> On 3/11/14 7:20 PM, "Yan Fang" <[email protected]> wrote: > >>>>>>>>>>>>> > >>>>>>>>>>>>> Hi Chris, > >>>>>>>>>>>>> > >>>>>>>>>>>>> Has opened an issue > >>>>>>>>>>>>> SAMZA-181<https://issues.apache.org/jira/browse/SAMZA-181> > >>>>>>>>>>>>> a nd also uploaded the patch. Let me know if there is > >>>>>>>>>>>>> something wrong in my tutorial. Thank you! > >>>>>>>>>>>>> > >>>>>>>>>>>>> Cheers, > >>>>>>>>>>>>> > >>>>>>>>>>>>> Fang, Yan > >>>>>>>>>>>>> [email protected] > >>>>>>>>>>>>> +1 (206) 849-4108 > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> On Tue, Mar 11, 2014 at 10:40 AM, > >>>>>>>>>>>>> <[email protected]>wrote: > >>>>>>>>>>>>> > >>>>>>>>>>>>>> Thanks Chris, Yan, > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Let me try that. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> -----Original Message----- > >>>>>>>>>>>>>> From: Chris Riccomini [mailto:[email protected]] > >>>>>>>>>>>>>> Sent: Tuesday, March 11, 2014 10:22 AM > >>>>>>>>>>>>>> To: [email protected] > >>>>>>>>>>>>>> Subject: Re: Failed to package using mvn > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Hey Yan, > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Awesome!The location where you can add your .md is here: > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> docs/learn/tutorials/0.7.0/ > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Here's a link to the code tree: > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> https://git-wip-us.apache.org/repos/asf?p=incubator-samza. > >>>>>>>>>>>>>> git > >>>>>>>>>>>>>> ;a= > >>>>>>>>>>>>>> t > >>>>>>>>>>>>>> r > >>>>>>>>>>>>>> e > >>>>>>>>>>>>>> e;f > >>>>>>>>>>>>>> =do > >>>>>>>>>>>>>> cs > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> /learn/tutorials/0.7.0;h=ef117f4066f14a00f50f0f6fca179031 > >>>>>>>>>>>>>> 3 > >>>>>>>>>>>>>> 044 > >>>>>>>>>>>>>> 831 > >>>>>>>>>>>>>> 2 > >>>>>>>>>>>>>> ; > >>>>>>>>>>>>>> h > >>>>>>>>>>>>>> b=H > >>>>>>>>>>>>>> EAD > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> You can get the code here: > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> git clone > >>>>>>>>>>>>>> http://git-wip-us.apache.org/repos/asf/incubator-samza.gi > >>>>>>>>>>>>>> t > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Once you write the .md, just throw it up on a JIRA, and > >>>>>>>>>>>>>> one of us can merge it in. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Re: hdfs-site.xml, ah ha, that's what I figured. This is > >>>>>>>>>>>>>> good to > >>>>>>>>>>> know. > >>>>>>>>>>>>>> So > >>>>>>>>>>>>>> you just copy your hdfs-site.xml from your NodeManager's > >>>>>>>>>>>>>> conf directory into your local hdfs-site.xml. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Cheers, > >>>>>>>>>>>>>> Chris > >>>>>>>>>>>>>> > >>>>>>>>>>>>>>> On 3/11/14 10:16 AM, "Yan Fang" <[email protected]> > >>>>>>>>>>>>>>>wrote: > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Hi Chris, > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Sure. I just do not know how/where to contribute this > >>>>>>>>>>>>>>> page...*_* > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Oh, I mean the same this as you mentioned in the > >>>>>>>>>>>>>>> *Cluster > >>>>>>>>>>>>>>> Installation*thread: > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> *"2. Get a copy of one of your NM's yarn-site.xml and > >>>>>>>>>>>>>>> put it somewhere > >>>>>>>>>>>>>>> on* > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> *your desktop (I usually use ~/.yarn/conf/yarn-site.xml). > >>>>>>>>>>>>>>> Note that there'sa "conf" directory there. This is > >>>>>>>>>>>>>>> mandatory."* > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> So I just copy the hdfs-site.xml to > >>>>>>>>>>>>>>>~/.yarn/conf/hdfs-site.xml. > >>>>>>>>>>>>>>> Thank > >>>>>>>>>>>>>> you. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Cheers, > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Fang, Yan > >>>>>>>>>>>>>>> [email protected] > >>>>>>>>>>>>>>> +1 (206) 849-4108 > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> On Tue, Mar 11, 2014 at 10:10 AM, Chris Riccomini > >>>>>>>>>>>>>>> <[email protected]>wrote: > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Hey Yan, > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Would you be up for contributing a tutorial page that > >>>>>>>>>>>>>>>> describes > >>>>>>>>>>>>>> this? > >>>>>>>>>>>>>>>> This > >>>>>>>>>>>>>>>> is really useful information. Our docs are just simple > >>>>>>>>>>>>>>>> .md files in the main code base. > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Regarding step (3), is the hdfs-site.xml put into the > >>>>>>>>>>>>>>>> conf folder for the NM boxes, or on the client side > >>>>>>>>>>>>>>>> (where run-job.sh > >>>>>>>>>>> is run)? > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Cheers, > >>>>>>>>>>>>>>>> Chris > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> On 3/11/14 10:07 AM, "Yan Fang" <[email protected]> > >>>>>>>>>>>>>>>>>wrote: > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> Hi Sonali, > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> The way I make Samza run with HDFS is following: > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> 1. include hdfs jar in Samza jar tar.gz. > >>>>>>>>>>>>>>>>> 2. you may also want to make sure the > >>>>>>>>>>>>>>>>>hadoop-common.jar has the same version as your hdfs > >>>>>>>>>>>>>>>>>jar. Otherwise, you may have configuration error popping > out. > >>>>>>>>>>>>>>>>> 3. then put hdfs-site.xml to conf folder, the same > >>>>>>>>>>>>>>>>>folder as the yarn-site.xml 4. all other steps are not > >>>>>>>>>>>>>>>>>changed. > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> Hope this will help. Thank you. > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> Cheers, > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> Fang, Yan > >>>>>>>>>>>>>>>>> [email protected] > >>>>>>>>>>>>>>>>> +1 (206) 849-4108 > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> On Tue, Mar 11, 2014 at 9:25 AM, Chris Riccomini > >>>>>>>>>>>>>>>>> <[email protected]>wrote: > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> Hey Sonali, > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> I believe that you need to make sure that the HDFS > >>>>>>>>>>>>>>>>>> jar is in your .tar.gz file, as you've said. > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> If that doesn't work, you might need to define this > >>>>>>>>>>>>>>>>>> setting in core-site.xml on the machine you're > >>>>>>>>>>>>>>>>>> running > >>>>>>>>>>> run-job.sh on: > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> <property> > >>>>>>>>>>>>>>>>>> <name>fs.hdfs.impl</name> > >>>>>>>>>>> <value>org.apache.hadoop.hdfs.DistributedFileSystem</value> > >>>>>>>>>>>>>>>>>> <description>The FileSystem for hdfs: > >>>>>>>>>>>>>>>>>> uris.</description> </property> > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> You might also need to configure your NodeManagers to > >>>>>>>>>>>>>>>>>> have the HDFS > >>>>>>>>>>>>>>>> file > >>>>>>>>>>>>>>>>>> system impl as well. > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> I've never run Samza with HDFS, so I'm guessing here. > >>>>>>>>>>>>>>>>>> Perhaps someone else on the list has been successful > >>>>>>>>>>>>>>>>>> with > >>>>>>>>>>> this? > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> Cheers, > >>>>>>>>>>>>>>>>>> Chris > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> On 3/10/14 3:59 PM, "[email protected] > " > >>>>>>>>>>>>>>>>>> <[email protected]> wrote: > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> Hello, > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> I fixed this by starting from scratch with gradlew. > >>>>>>>>>>>>>>>>>>> But now when I > >>>>>>>>>>>>>>>> run > >>>>>>>>>>>>>>>>>> my > >>>>>>>>>>>>>>>>>>> job it throws this error: > >>>>>>>>>>>>>>>>>>> Exception in thread "main" java.io.IOException: No > >>>>>>>>>>>>>>>>>>> FileSystem for > >>>>>>>>>>>>>>>>>> scheme: > >>>>>>>>>>>>>>>>>>> hdfs > >>>>>>>>>>>>>>>>>>> at > >>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>org.apache.hadoop.fs.FileSystem.getFileSystemClass(Fi > >>>>>>>>>>>>>>>>>>>l > >>>>>>>>>>>>>>>>>>>eSy > >>>>>>>>>>>>>>>>>>>ste > >>>>>>>>>>>>>>>>>>>m. > >>>>>>>>>>>>>>>>>>> jav > >>>>>>>>>>>>>>>>>>> a: > >>>>>>>>>>>>>>>>>>> 242 > >>>>>>>>>>>>>>>>>>> 1) > >>>>>>>>>>>>>>>>>>> at > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>org.apache.hadoop.fs.FileSystem.createFileSystem(FileSy > >>>>>>>>>>>>>>>>>s > >>>>>>>>>>>>>>>>>tem > >>>>>>>>>>>>>>>>>. > >>>>>>>>>>>>>>>>> j > >>>>>>>>>>>>>>>>> a > >>>>>>>>>>>>>>>>> v > >>>>>>>>>>>>>>>>> a:2 > >>>>>>>>>>>>>>>>> 428 > >>>>>>>>>>>>>>>>> ) > >>>>>>>>>>>>>>>>>>> at > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>org.apache.hadoop.fs.FileSystem.access$200(FileSystem. > >>>>>>>>>>>>>>>>>>j > >>>>>>>>>>>>>>>>>>ava > >>>>>>>>>>>>>>>>>>: > >>>>>>>>>>>>>>>>>> 8 > >>>>>>>>>>>>>>>>>> 8 > >>>>>>>>>>>>>>>>>> ) > >>>>>>>>>>>>>>>>>>> at > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>>org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSyst > >>>>>>>>>>>>>>e > >>>>>>>>>>>>>>m.j > >>>>>>>>>>>>>>ava > >>>>>>>>>>>>>>: > >>>>>>>>>>>>>>>>> 246 > >>>>>>>>>>>>>>>>> 7) > >>>>>>>>>>>>>>>>>>> at > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.j > >>>>>>>>>>>>>>>>>>ava > >>>>>>>>>>>>>>>>>>: > >>>>>>>>>>>>>>>>>> 2 > >>>>>>>>>>>>>>>>>> 4 > >>>>>>>>>>>>>>>>>> 4 > >>>>>>>>>>>>>>>>>> 9) > >>>>>>>>>>>>>>>>>>> at > >>>>>>>>>>>>>> org.apache.hadoop.fs.FileSystem.get(FileSystem.java:367) > >>>>>>>>>>>>>>>>>>> at > >>>>>>>>>>>>>> org.apache.hadoop.fs.Path.getFileSystem(Path.java:287) > >>>>>>>>>>>>>>>>>>> at > >>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> org.apache.samza.job.yarn.ClientHelper.submitApplica > >>>>>>>>>>>>>>>>>>> t > >>>>>>>>>>>>>>>>>>> ion > >>>>>>>>>>>>>>>>>>> (Cl > >>>>>>>>>>>>>>>>>>> i > >>>>>>>>>>>>>>>>>>> e > >>>>>>>>>>>>>>>>>>> n > >>>>>>>>>>>>>>>>>>> tHe > >>>>>>>>>>>>>>>>>>> lpe > >>>>>>>>>>>>>>>>>>> r. > >>>>>>>>>>>>>>>>>>> sc > >>>>>>>>>>>>>>>>>>> al > >>>>>>>>>>>>>>>>>>> a:111) > >>>>>>>>>>>>>>>>>>> at > >>>>>>>>>>>>>>>> org.apache.samza.job.yarn.YarnJob.submit(YarnJob.scala: > >>>>>>>>>>>>>>>> 5 > >>>>>>>>>>>>>>>> 5) > >>>>>>>>>>>>>>>>>>> at > >>>>>>>>>>>>>>>> org.apache.samza.job.yarn.YarnJob.submit(YarnJob.scala: > >>>>>>>>>>>>>>>> 4 > >>>>>>>>>>>>>>>> 8) > >>>>>>>>>>>>>>>>>>> at > >>>>>>>>>>>>>> org.apache.samza.job.JobRunner.run(JobRunner.scala:100) > >>>>>>>>>>>>>>>>>>> at > >>>>>>>>>>>>>> org.apache.samza.job.JobRunner$.main(JobRunner.scala:75) > >>>>>>>>>>>>>>>>>>> at > >>>>>>>>>>>>>>>>>>> org.apache.samza.job.JobRunner.main(JobRunner.scala) > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> I looked at the samza job tar.gz and it doesn't have > >>>>>>>>>>>>>>>>>>> a Hadoop-hdfs > >>>>>>>>>>>>>>>> jar. > >>>>>>>>>>>>>>>>>>> Is that why I get this error? > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> Thanks, > >>>>>>>>>>>>>>>>>>> Sonali > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> From: Parthasarathy, Sonali > >>>>>>>>>>>>>>>>>>> Sent: Monday, March 10, 2014 11:25 AM > >>>>>>>>>>>>>>>>>>> To: [email protected] > >>>>>>>>>>>>>>>>>>> Subject: Failed to package using mvn > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> Hi, > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> When I tried to do a mvn clean package of my > >>>>>>>>>>>>>>>>>>> hello-samza project, I > >>>>>>>>>>>>>>>> get > >>>>>>>>>>>>>>>>>>> the following error. Has anyone seen this before? > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> [ERROR] Failed to execute goal on project > >>>>>>>>>>>>>>>>>>>samza-wikipedia: > >>>>>>>>>>>>>>>>>>> Could not resolve dependencies for project > >>>>>>>>>>>>>> samza:samza-wikipedia:jar:0.7.0: > >>>>>>>>>>>>>>>> Could > >>>>>>>>>>>>>>>>>>> not find artifact > >>>>>>>>>>>>>>>>>>> org.apache.samza:samza-kv_2.10:jar:0.7.0 > >>>>>>>>>>>>>>>>>>> in apache-releases > >>>>>>>>>>>>>>>> (https://repository.apache.org/content/groups/public) > >>>>>>>>>>>>>>>>>> -> > >>>>>>>>>>>>>>>>>>> [Help 1] > >>>>>>>>>>>>>>>>>>> [ERROR] > >>>>>>>>>>>>>>>>>>> [ERROR] To see the full stack trace of the errors, > >>>>>>>>>>>>>>>>>>> re-run Maven with > >>>>>>>>>>>>>>>>>> the > >>>>>>>>>>>>>>>>>>> -e switch. > >>>>>>>>>>>>>>>>>>> [ERROR] Re-run Maven using the -X switch to enable > >>>>>>>>>>>>>>>>>>> full debug > >>>>>>>>>>>>>>>> logging. > >>>>>>>>>>>>>>>>>>> [ERROR] > >>>>>>>>>>>>>>>>>>> [ERROR] For more information about the errors and > >>>>>>>>>>>>>>>>>>> possible > >>>>>>>>>>>>>>>> solutions, > >>>>>>>>>>>>>>>>>>> please read the following articles: > >>>>>>>>>>>>>>>>>>> [ERROR] [Help 1] > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> http://cwiki.apache.org/confluence/display/MAVEN/Depend > >>>>>>>>>>>>>>>> e > >>>>>>>>>>>>>>>> ncy > >>>>>>>>>>>>>>>> Re > >>>>>>>>>>>>>>>> s > >>>>>>>>>>>>>>>> o > >>>>>>>>>>>>>>>> l > >>>>>>>>>>>>>>>> uti > >>>>>>>>>>>>>>>> onE > >>>>>>>>>>>>>>>> xce > >>>>>>>>>>>>>>>>>> p > >>>>>>>>>>>>>>>>>>> tion > >>>>>>>>>>>>>>>>>>> [ERROR] > >>>>>>>>>>>>>>>>>>> [ERROR] After correcting the problems, you can > >>>>>>>>>>>>>>>>>>> resume the build with > >>>>>>>>>>>>>>>>>> the > >>>>>>>>>>>>>>>>>>> command > >>>>>>>>>>>>>>>>>>> [ERROR] mvn <goals> -rf :samza-wikipedia > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> Thanks, > >>>>>>>>>>>>>>>>>>> Sonali > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> Sonali Parthasarathy R&D Developer, Data Insights > >>>>>>>>>>>>>>>>>>> Accenture Technology Labs > >>>>>>>>>>>>>>>>>>> 703-341-7432 > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> ________________________________ > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> This message is for the designated recipient only > >>>>>>>>>>>>>>>>>>> and may contain privileged, proprietary, or > >>>>>>>>>>>>>>>>>>> otherwise confidential > >>>>>>>>>>>>>> information. > >>>>>>>>>>>>>>>>>>> If > >>>>>>>>>>>>>>>> you > >>>>>>>>>>>>>>>>>>> have received it in error, please notify the sender > >>>>>>>>>>>>>>>>>>> immediately and delete the original. Any other use > >>>>>>>>>>>>>>>>>>> of the e-mail by you is > >>>>>>>>>>>>>>>> prohibited. > >>>>>>>>>>>>>>>>>>> Where allowed by local law, electronic > >>>>>>>>>>>>>>>>>>> communications with Accenture > >>>>>>>>>>>>>>>>>> and > >>>>>>>>>>>>>>>>>>> its affiliates, including e-mail and instant > >>>>>>>>>>>>>>>>>>> messaging (including content), may be scanned by our > >>>>>>>>>>>>>>>>>>> systems for the purposes of > >>>>>>>>>>>>>>>> information > >>>>>>>>>>>>>>>>>>> security and assessment of internal compliance with > >>>>>>>>>>>>>>>>>>> Accenture > >>>>>>>>>>>>>>>> policy. > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> ____________________________________________________ > >>>>>>>>>>>>>>>>>>> _ > >>>>>>>>>>>>>>>>>>> ___ > >>>>>>>>>>>>>>>>>>> __ > >>>>>>>>>>>>>>>>>>> _ > >>>>>>>>>>>>>>>>>>> _ > >>>>>>>>>>>>>>>>>>> _ > >>>>>>>>>>>>>>>>>>> ___ > >>>>>>>>>>>>>>>>>>> ___ > >>>>>>>>>>>>>>>>>>> ___ > >>>>>>>>>>>>>>>>>>> __ > >>>>>>>>>>>>>>>>>>> __ > >>>>>>>>>>>>>>>>>>> ____________ > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> www.accenture.com > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> ________________________________ > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> This message is for the designated recipient only and may > >>>>>>>>>>>>>> contain privileged, proprietary, or otherwise > >>>>>>>>>>>>>> confidential information. If you have received it in > >>>>>>>>>>>>>> error, please notify the sender immediately and delete the > original. > >>>>>>>>>>>>>> Any other use of the e-mail by you is prohibited. Where > >>>>>>>>>>>>>> allowed by local law, electronic communications with > >>>>>>>>>>>>>> Accenture and its affiliates, including e-mail and > >>>>>>>>>>>>>> instant messaging (including content), may be scanned by > >>>>>>>>>>>>>> our systems for the purposes of information security and > >>>>>>>>>>>>>> assessment of internal compliance with Accenture > >>>>>>>>>>> policy. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> _________________________________________________________ > >>>>>>>>>>>>>> _ > >>>>>>>>>>>>>> ___ > >>>>>>>>>>>>>> ___ > >>>>>>>>>>>>>> _ > >>>>>>>>>>>>>> _ > >>>>>>>>>>>>>> _ > >>>>>>>>>>>>>> ___ > >>>>>>>>>>>>>> ___ > >>>>>>>>>>>>>> _____________ > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> www.accenture.com > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> ________________________________ > >>>>>>>>>>>> > >>>>>>>>>>>> This message is for the designated recipient only and may > >>>>>>>>>>>> contain privileged, proprietary, or otherwise confidential > >>>>>>>>>>>> information. If you have received it in error, please > >>>>>>>>>>>> notify the sender immediately and delete the original. Any > >>>>>>>>>>>> other use of the e-mail by you is > >>>>>>>>>>> prohibited. > >>>>>>>>>>>> Where allowed by local law, electronic communications with > >>>>>>>>>>>> Accenture and its affiliates, including e-mail and instant > >>>>>>>>>>>> messaging (including content), may be scanned by our > >>>>>>>>>>>> systems for the purposes of information security and > >>>>>>>>>>>> assessment of internal compliance with > >>>>>>>>>>> Accenture policy. > >>>>>>>>>>>> ___________________________________________________________ > >>>>>>>>>>>> _ > >>>>>>>>>>>> ___ > >>>>>>>>>>>> ___ > >>>>>>>>>>>> _ > >>>>>>>>>>>> _ > >>>>>>>>>>>> _ > >>>>>>>>>>>> _____ > >>>>>>>>>>>> ____________ > >>>>>>>>>>>> > >>>>>>>>>>>> www.accenture.com > >>>>>>>>>> > >>>>>>>>>> ________________________________ > >>>>>>>>>> > >>>>>>>>>> This message is for the designated recipient only and may > >>>>>>>>>>contain privileged, proprietary, or otherwise confidential > >>>>>>>>>>information. If you have received it in error, please notify > >>>>>>>>>>the sender immediately and delete the original. Any other use > >>>>>>>>>>of the e-mail by you is prohibited. > >>>>>>>>>> Where allowed by local law, electronic communications with > >>>>>>>>>>Accenture and its affiliates, including e-mail and instant > >>>>>>>>>>messaging (including content), may be scanned by our systems > >>>>>>>>>>for the purposes of information security and assessment of > >>>>>>>>>>internal compliance with Accenture policy. > >>>>>>>>>> > >>>>>>>>>>______________________________________________________________ > >>>>>>>>>>_ > >>>>>>>>>>___ > >>>>>>>>>>___ > >>>>>>>>>> _ > >>>>>>>>>> _ > >>>>>>>>>> ___ > >>>>>>>>>> ____________ > >>>>>>>>>> > >>>>>>>>>> www.accenture.com > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> ________________________________ > >>>>>>>>> > >>>>>>>>> This message is for the designated recipient only and may > >>>>>>>>>contain privileged, proprietary, or otherwise confidential > >>>>>>>>>information. If you have received it in error, please notify > >>>>>>>>>the sender immediately and delete the original. Any other use > >>>>>>>>>of the e-mail by you is prohibited. > >>>>>>>>> Where allowed by local law, electronic communications with > >>>>>>>>>Accenture and its affiliates, including e-mail and instant > >>>>>>>>>messaging (including content), may be scanned by our systems > >>>>>>>>>for the purposes of information security and assessment of > >>>>>>>>>internal compliance with Accenture policy. > >>>>>>>>> > >>>>>>>>>_______________________________________________________________ > >>>>>>>>>_ > >>>>>>>>>___ > >>>>>>>>>___ > >>>>>>>>> _ > >>>>>>>>> ___ > >>>>>>>>> ____________ > >>>>>>>>> > >>>>>>>>> www.accenture.com > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> ________________________________ > >>>>>>>> > >>>>>>>> This message is for the designated recipient only and may > >>>>>>>>contain privileged, proprietary, or otherwise confidential > >>>>>>>>information. If you have received it in error, please notify > >>>>>>>>the sender immediately and delete the original. Any other use > >>>>>>>>of the e-mail by you is prohibited. > >>>>>>>> Where allowed by local law, electronic communications with > >>>>>>>>Accenture and its affiliates, including e-mail and instant > >>>>>>>>messaging (including content), may be scanned by our systems > >>>>>>>>for the purposes of information security and assessment of > >>>>>>>>internal compliance with Accenture policy. > >>>>>>>> > >>>>>>>>________________________________________________________________ > >>>>>>>>_ > >>>>>>>>___ > >>>>>>>>___ > >>>>>>>> ___ > >>>>>>>> ____________ > >>>>>>>> > >>>>>>>> www.accenture.com > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> ________________________________ > >>>>>>> > >>>>>>> This message is for the designated recipient only and may > >>>>>>>contain privileged, proprietary, or otherwise confidential > >>>>>>>information. If you have received it in error, please notify the > >>>>>>>sender immediately and delete the original. Any other use of the > >>>>>>>e-mail by you is prohibited. > >>>>>>> Where allowed by local law, electronic communications with > >>>>>>>Accenture and its affiliates, including e-mail and instant > >>>>>>>messaging (including content), may be scanned by our systems for > >>>>>>>the purposes of information security and assessment of internal > >>>>>>>compliance with Accenture policy. > >>>>>>> > >>>>>>>_________________________________________________________________ > >>>>>>>_ > >>>>>>>___ > >>>>>>>___ > >>>>>>>__ > >>>>>>> ____________ > >>>>>>> > >>>>>>> www.accenture.com > >>>>>> > >>>> > >>>> > >>>> > >>>>________________________________ > >>>> > >>>>This message is for the designated recipient only and may contain > >>>>privileged, proprietary, or otherwise confidential information. If > >>>>you have received it in error, please notify the sender immediately > >>>>and delete the original. Any other use of the e-mail by you is > >>>>prohibited. > >>>>Where allowed by local law, electronic communications with Accenture > >>>>and its affiliates, including e-mail and instant messaging > >>>>(including content), may be scanned by our systems for the purposes > >>>>of information security and assessment of internal compliance with > >>>>Accenture policy. > >>>>____________________________________________________________________ > >>>>_ > >>>>___ > >>>>_ > >>>>_ > >>>>____________ > >>>> > >>>>www.accenture.com > >>>> > >>> > >> > >> > >> > >>________________________________ > >> > >>This message is for the designated recipient only and may contain > >>privileged, proprietary, or otherwise confidential information. If you > >>have received it in error, please notify the sender immediately and > >>delete the original. Any other use of the e-mail by you is prohibited. > >>Where allowed by local law, electronic communications with Accenture > >>and its affiliates, including e-mail and instant messaging (including > >>content), may be scanned by our systems for the purposes of > >>information security and assessment of internal compliance with > Accenture policy. > >>______________________________________________________________________ > >>_ > >>___ > >>____________ > >> > >>www.accenture.com > >> > > > > > > > >________________________________ > > > >This message is for the designated recipient only and may contain > >privileged, proprietary, or otherwise confidential information. If you > >have received it in error, please notify the sender immediately and > >delete the original. Any other use of the e-mail by you is prohibited. > >Where allowed by local law, electronic communications with Accenture > >and its affiliates, including e-mail and instant messaging (including > >content), may be scanned by our systems for the purposes of information > >security and assessment of internal compliance with Accenture policy. > >_______________________________________________________________________ > >___ > >____________ > > > >www.accenture.com > > > > > > ________________________________ > > This message is for the designated recipient only and may contain > privileged, proprietary, or otherwise confidential information. If you have > received it in error, please notify the sender immediately and delete the > original. Any other use of the e-mail by you is prohibited. Where allowed > by local law, electronic communications with Accenture and its affiliates, > including e-mail and instant messaging (including content), may be scanned > by our systems for the purposes of information security and assessment of > internal compliance with Accenture policy. > > ______________________________________________________________________________________ > > www.accenture.com > >
