Hi Anh/Chris I looked in : /hadoop/hadoop-2.2.0/logs/userlogs/application_1391637982288_0040/container_1391637982288_0040_01_000001
I see 1. application-master.log 2. stderr 3. stdout 4. gc.log In the application log is where I see the error. Basically, it claims a container, starts the task ID and then suddenly exists with code -100. It says " This means that container container_1391637982288_0040_01_000004 was killed by YARN, either due to being released by the application master or being 'lost' due to node failures etc." Have you seen this before? -----Original Message----- From: Anh Thu Vu [mailto:[email protected]] Sent: Thursday, March 13, 2014 1:07 PM To: [email protected] Subject: Re: Failed to package using mvn *some typos, most importantly, "more information" not "me information" (shy) On Thu, Mar 13, 2014 at 9:06 PM, Anh Thu Vu <[email protected]> wrote: > HI Sonali, > > Try look into the logs folder of YARN (should be something like > $YARN_HOME/logs/userlogs/application-xxxxx/container-yyyyyy/ and check the > logs in those folder. There are probably some file like > samza-container-zzzz.logs. Those are the logs from the actual container and > should give me information about the error. > > Cheers, > Casey > > > > On Thu, Mar 13, 2014 at 8:56 PM, <[email protected]>wrote: > >> Hey Chris, >> >> I get the same error in the NM logs >> 2014-03-13 11:44:14,210 INFO >> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: >> Memory usage of ProcessTree 16883 for container-id >> container_1391637982288_0039_01_000001: 45.3 MB of 1 GB physical memory >> used; 1.3 GB of 2.1 GB virtual memory used >> 2014-03-13 11:44:15,144 WARN >> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exit >> code from container container_1391637982288_0039_01_000001 is : 1 >> 2014-03-13 11:44:15,144 WARN >> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: >> Exception from container-launch with container ID: >> container_1391637982288_0039_01_000001 and exit code: 1 >> org.apache.hadoop.util.Shell$ExitCodeException: >> at org.apache.hadoop.util.Shell.runCommand(Shell.java:464) >> at org.apache.hadoop.util.Shell.run(Shell.java:379) >> at >> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589) >> at >> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195) >> at >> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:283) >> at >> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:79) >> at java.util.concurrent.FutureTask.run(FutureTask.java:262) >> at >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >> at java.lang.Thread.run(Thread.java:744) >> 2014-03-13 11:44:15,144 INFO >> org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: >> 2014-03-13 11:44:15,145 WARN >> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch: >> Container exited with a non-zero exit code 1 >> 2014-03-13 11:44:15,145 INFO >> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container: >> Container container_1391637982288_0039_01_000001 transitioned from RUNNING >> to EXITED_WITH_FAILURE >> >> -----Original Message----- >> From: Chris Riccomini [mailto:[email protected]] >> Sent: Thursday, March 13, 2014 12:54 PM >> To: [email protected] >> Subject: Re: Failed to package using mvn >> >> Hey Sonali, >> >> Yep, if you applied them both, you should be OK. Though, you still >> shouldn't need to add a yarn.rm.hostname to your configuration. The >> SamzaAppMaster should automatically pick that up from the yarn-site.xml on >> your NM. >> >> In any case, what does your NM log say? >> >> Cheers, >> Chris >> >> On 3/13/14 12:51 PM, "[email protected]" >> <[email protected]> wrote: >> >> >Ah I see. I applied them both. That shouldn't matter right? >> > >> >-----Original Message----- >> >From: Chris Riccomini [mailto:[email protected]] >> >Sent: Thursday, March 13, 2014 12:38 PM >> >To: [email protected] >> >Subject: Re: Failed to package using mvn >> > >> >Hey Sonali, >> > >> >You should not have to add yarn.rm.hostname to your configuration. You >> >have applied the wrong patch. Please apply this patch: >> > >> > >> >https://issues.apache.org/jira/secure/attachment/12634493/SAMZA-182.1.p >> >atc >> >h >> > >> > >> >Do not apply >> >https://issues.apache.org/jira/secure/attachment/12634200/yarn_remote_r >> >m.p >> >a >> >tch, which is the one that adds yarn.rm.hostname. >> > >> >Regarding the failed containers, can you take a look at the NM's logs, >> >and see what it says there? >> > >> >Cheers, >> >Chris >> > >> >On 3/13/14 12:00 PM, "[email protected]" >> ><[email protected]> wrote: >> > >> >>Hi Chris. >> >> >> >>I installed the patch and was able to submit the job. (I had to add >> >>yarn.rm.hostname in the wikipedia-feed.properties file). The job >> >>started running and then I got this error: >> >>SamzaAppMasterTaskManager [INFO] Got an exit code of -100. This means >> >>that container container_1391637982288_0040_01_000043 was killed by >> >>YARN, either due to being released by the application master or being >> >>'lost' >> >>due to node failures etc. >> >>2014-03-13 11:53:03 SamzaAppMasterTaskManager [INFO] Got an exit code >> >>of -100. This means that container >> >>container_1391637982288_0040_01_000044 >> >>was killed by YARN, either due to being released by the application >> >>master or being 'lost' due to node failures etc. >> >>2014-03-13 11:53:03 SamzaAppMasterTaskManager [INFO] Got an exit code >> >>of -100. This means that container >> >>container_1391637982288_0040_01_000045 >> >>was killed by YARN, either due to being released by the application >> >>master or being 'lost' due to node failures etc. >> >>2014-03-13 11:53:03 SamzaAppMasterTaskManager [INFO] Got a container >> >>from YARN ResourceManager: Container: [ContainerId: >> >>container_1391637982288_0040_01_000046, NodeId: svdpdac015:37919, >> >>NodeHttpAddress: svdpdac015:8042, Resource: <memory:1024, vCores:1>, >> >>Priority: 0, Token: Token { kind: ContainerToken, service: >> >>10.1.174.84:37919 }, ] >> >>2014-03-13 11:53:03 SamzaAppMasterTaskManager [INFO] Got an extra >> >>container from YARN ResourceManager: Container: [ContainerId: >> >>container_1391637982288_0040_01_000046, NodeId: svdpdac015:37919, >> >>NodeHttpAddress: svdpdac015:8042, Resource: <memory:1024, vCores:1>, >> >>Priority: 0, Token: Token { kind: ContainerToken, service: >> >>10.1.174.84:37919 }, ] >> >>2014-03-13 11:53:04 SamzaAppMasterTaskManager [INFO] Got an exit code >> >>of -100. This means that container >> >>container_1391637982288_0040_01_000046 >> >>was killed by YARN, either due to being released by the application >> >>master or being 'lost' due to node failures etc. >> >>2014-03-13 11:53:24 SamzaAppMasterTaskManager [INFO] Container >> >>container_1391637982288_0040_01_000038 failed with exit code 1 - >> >>Exception from container-launch: >> >>org.apache.hadoop.util.Shell$ExitCodeException: >> >> at org.apache.hadoop.util.Shell.runCommand(Shell.java:464) >> >> at org.apache.hadoop.util.Shell.run(Shell.java:379) >> >> at >> >> >>org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589) >> >> at >> >>org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.lau >> >>n >> >>chC >> >>ontainer(DefaultContainerExecutor.java:195) >> >> at >> >>org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.Co >> >>n >> >>tai >> >>nerLaunch.call(ContainerLaunch.java:283) >> >> at >> >>org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.Co >> >>n >> >>tai >> >>nerLaunch.call(ContainerLaunch.java:79) >> >> at java.util.concurrent.FutureTask.run(FutureTask.java:262) >> >> at >> >>java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.j >> >>ava >> >>: >> >>1145) >> >> at >> >>java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor. >> >>j >> >>ava >> >>:615) >> >> at java.lang.Thread.run(Thread.java:744) >> >> >> >> >> >>Now, I'm running Hadoop-2.2.0 version of yarn. Do I need to change to >> >>YARN 2.3? >> >> >> >>-----Original Message----- >> >>From: Chris Riccomini [mailto:[email protected]] >> >>Sent: Thursday, March 13, 2014 11:12 AM >> >>To: [email protected] >> >>Subject: Re: Failed to package using mvn >> >> >> >>Hey Sonali, >> >> >> >>As a follow on, here is EXACTLY what I did: >> >> >> >># start by downloading YARN and setting it up 1. Download YARN 2.3 >> >>from >> >>http://mirror.symnds.com/software/Apache/hadoop/common/hadoop-2.3.0/ha >> >>d >> >>oop >> >>- >> >>2.3.0.tar.gz to /tmp >> >>2. cd /tmp >> >>3. tar -xvf hadoop-2.3.0.tar.gz >> >>4. cd hadoop-2.3.0 >> >>5. export HADOOP_YARN_HOME=$(pwd) >> >>6. mkdir conf >> >>7. export HADOOP_CONF_DIR=$HADOOP_YARN_HOME/conf >> >>8. cp ./etc/hadoop/yarn-site.xml conf >> >>9. vi conf/yarn-site.xml >> >>10. Add this property to yarn-site.xml: >> >> >> >> <property> >> >> <name>yarn.resourcemanager.hostname</name> >> >> <!-- hostname that is accessible from all NMs --> >> >> <value>criccomi-mn</value> >> >> </property> >> >> >> >> >> >>11. curl >> >>http://svn.apache.org/viewvc/hadoop/common/trunk/hadoop-yarn-project/h >> >>a >> >>doo >> >>p >> >>-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/resources/c >> >>a >> >>pac >> >>i >> >>ty-scheduler.xml?view=co > conf/capacity-scheduler.xml >> >> >> >># setup http filesystem for YARN (you can skip this and follow >> >>SAMZA-181 instead, if you are using HDFS) 12. cd /tmp 13. curl >> >>http://www.scala-lang.org/files/archive/scala-2.10.3.tgz > >> >>scala-2.10.3.tgz 14. tar -xvf scala-2.10.3.tgz 15. cp >> >>/tmp/scala-2.10.3/lib/scala-compiler.jar >> >>$HADOOP_YARN_HOME/share/hadoop/hdfs/lib >> >>16. cp /tmp/scala-2.10.3/lib/scala-library.jar >> >>$HADOOP_YARN_HOME/share/hadoop/hdfs/lib >> >>17. curl >> >>http://search.maven.org/remotecontent?filepath=org/clapper/grizzled-sl >> >>f >> >>4j_ >> >>2 >> >>.10/1.0.1/grizzled-slf4j_2.10-1.0.1.jar > >> >>$HADOOP_YARN_HOME/share/hadoop/hdfs/lib/grizzled-slf4j_2.10-1.0.1.jar >> >>18. vi $HADOOP_YARN_HOME/conf/core-site.xml >> >> >> >><?xml-stylesheet type="text/xsl" href="configuration.xsl"?> >> >><configuration> <property> <name>fs.http.impl</name> >> >><value>org.apache.samza.util.hadoop.HttpFileSystem</value> >> >></property> >> >></configuration> >> >> >> >> >> >>19. Copy the Hadoop directory to all slave nodes (172.21.100.35, in my >> >>case): >> >> >> >> scp -r . 172.21.100.35:/tmp/hadoop-2.3.0 >> >> >> >>20. echo 172.21.100.35 > conf/slaves >> >>21. sbin/start-yarn.sh >> >>22. If you get "172.21.100.35: Error: JAVA_HOME is not set and could >> >>not be found.", you'll need to add a conf/hadoop-env.sh file to the >> >>machine with the failure (172.21.100.35, in this case), which has >> >>"export JAVA_HOME=/export/apps/jdk/JDK-1_6_0_27" (or wherever your >> >>JAVA_HOME actually is). >> >>23. Validate that your nodes are up by visiting >> >>http://criccomi-mn:8088/cluster/nodes >> >> >> >> >> >># now we more or less follow the hello-samza steps. >> >>24. cd /tmp >> >>25. git clone >> >>http://git-wip-us.apache.org/repos/asf/incubator-samza.git >> >>26. cd incubator-samza >> >>27. curl >> >>https://issues.apache.org/jira/secure/attachment/12634493/SAMZA-182.1. >> >>p >> >>atc >> >>h >> >> > SAMZA-182.1.patch >> >>28. git apply SAMZA-182.1.patch >> >>29. ./gradlew clean publishToMavenLocal 30. cd .. >> >>31. git clone git://github.com/linkedin/hello-samza.git >> >>32. cd hello-samza >> >>33. vi samza-job-package/src/main/config/wikipedia-feed.properties >> >>34. Change the yarn.package.path property to be: >> >> >> >> >> >>yarn.package.path=http://criccomi-mn:8000/samza-job-package/target/sam >> >>z >> >>a-j >> >>o >> >>b-package-0.7.0-dist.tar.gz >> >> >> >> >> >>35. mvn clean package >> >>36. mkdir -p deploy/samza >> >>37. tar -xvf >> >>./samza-job-package/target/samza-job-package-0.7.0-dist.tar.gz -C >> >>deploy/samza 38. Open a new terminal, and cd /tmp/hello-samza && >> >>python -m SimpleHTTPServer 39. Go back to the original terminal (not >> >>the one running the HTTP server) 40. deploy/samza/bin/run-job.sh >> >>--config-factory=org.apache.samza.config.factories.PropertiesConfigFac >> >>t >> >>ory >> >>--config-path=file://$PWD/deploy/samza/config/wikipedia-feed.propertie >> >>s 41. Go to http://criccomi-mn:8088 and find the wikipedia-feed job. >> >>Click on the ApplicationMaster link to see that it's running. >> >> >> >> >> >>I plan to write a tutorial that formalizes this. >> >> >> >>Cheers, >> >>Chris >> >> >> >>On 3/13/14 11:03 AM, "Chris Riccomini" <[email protected]> wrote: >> >> >> >>>Hey Sonali, >> >>> >> >>>Please have a look at: >> >>> >> >>> https://issues.apache.org/jira/browse/SAMZA-182 >> >>> >> >>>I have posted a fix there. I have successfully downloaded YARN, setup >> >>>a two node grid from scratch, and run hello-samza on it with the >> >>>patch on SAMZA-182. Can you give that a shot for me? >> >>> >> >>>Thanks for your patience! >> >>> >> >>>Cheers, >> >>>Chris >> >>> >> >>>On 3/13/14 10:58 AM, "[email protected]" >> >>><[email protected]> wrote: >> >>> >> >>>>Hi Chris, >> >>>> >> >>>>I checked my .bashrc. The variable was set on one of the NMs and not >> >>>>on the other. I made the change and restarted the scripts. I still >> >>>>get the same error. >> >>>> >> >>>>Also in my stderr I get: >> >>>>Null identity service, trying login service: null Finding identity >> >>>>service: null Null identity service, trying login service: null >> >>>>Finding identity service: null >> >>>> >> >>>>-----Original Message----- >> >>>>From: Chris Riccomini [mailto:[email protected]] >> >>>>Sent: Wednesday, March 12, 2014 7:59 PM >> >>>>To: [email protected] >> >>>>Subject: Re: Failed to package using mvn >> >>>> >> >>>>Hey Guys, >> >>>> >> >>>>I was able to reproduce this problem. >> >>>> >> >>>>I was also able to fix it (without the patch in SAMZA-182). All I >> >>>>needed to do was update ~/.bashrc on my NM's box to have: >> >>>> >> >>>> export YARN_HOME=/tmp/hadoop-2.3.0 >> >>>> >> >>>>It appears that the YARN environment variables are somehow getting >> >>>>lost or not forwarded from the NM to the AM. Adding this bashrc >> >>>>setting makes sure that the NM gets them. >> >>>> >> >>>> >> >>>>I have a feeling upgrading Samza to YARN 2.3.0 will fix this, but I >> >>>>haven't validated yet. I will continue to investigate tomorrow. >> >>>> >> >>>>Cheers, >> >>>>Chris >> >>>> >> >>>>On 3/12/14 6:43 PM, "Yan Fang" <[email protected]> wrote: >> >>>> >> >>>>>I guess Sonali has the problem is because his NMs do not read the >> >>>>>YARN_HOME variable. That may be because the NM machine does not >> >>>>>have YARN_HOME set when the NM starts. >> >>>>> >> >>>>>Check this https://issues.apache.org/jira/browse/SAMZA-182 >> >>>>> >> >>>>>Thanks, >> >>>>> >> >>>>>Yan Fang >> >>>>> >> >>>>>> On Mar 12, 2014, at 6:14 PM, Chris Riccomini >> >>>>>><[email protected]> >> >>>>>>wrote: >> >>>>>> >> >>>>>> Hey Sonali, >> >>>>>> >> >>>>>> I am unfamiliar with the start-yarn.sh. Looking at: >> >>>>>> >> >>>>>> >> >>>>>> >> >>>>>>https://svn.apache.org/viewvc/hadoop/common/trunk/hadoop-yarn-proj >> >>>>>>e >> >>>>>>ct/ >> >>>>>>had >> >>>>>>oo >> >>>>>> p-yarn/bin/stop-yarn.sh?revision=1370666&view=markup >> >>>>>> >> >>>>>> What version of YARN are you using? >> >>>>>> >> >>>>>> Cheers, >> >>>>>> Chris >> >>>>>> >> >>>>>> On 3/12/14 5:56 PM, "[email protected]" >> >>>>>> <[email protected]> wrote: >> >>>>>> >> >>>>>>> Hey Chris, >> >>>>>>> >> >>>>>>> Yes, I have YARN_HOME set in all the NMs pointing to the right >> >>>>>>>directories. I also made sure the yarn-site.xml file has the >> >>>>>>>hostname set. >> >>>>>>> >> >>>>>>> I start yarn using start.yarn.sh in the RM and that >> >>>>>>>automatically starts the NMs on the slave nodes. Is that the >> right way to do it? >> >>>>>>> >> >>>>>>> -----Original Message----- >> >>>>>>> From: Chris Riccomini [mailto:[email protected]] >> >>>>>>> Sent: Wednesday, March 12, 2014 5:52 PM >> >>>>>>> To: [email protected] >> >>>>>>> Subject: Re: Failed to package using mvn >> >>>>>>> >> >>>>>>> Hey Sonali, >> >>>>>>> >> >>>>>>> OK, so we've validated that the NMs are able to connect, which >> >>>>>>>means they can see the yarn-site.xml. >> >>>>>>> >> >>>>>>> How are you starting your NMs? Are you running: >> >>>>>>> >> >>>>>>> export YARN_HOME=/path/to/yarn/home >> >>>>>>> >> >>>>>>> In the CLI before starting the NM? >> >>>>>>> >> >>>>>>> For reference, we run: >> >>>>>>> >> >>>>>>> export YARN_HOME=/path/to/our/yarn-home export >> >>>>>>> YARN_CONF_DIR=$YARN_HOME/conf >> >>>>>>> >> >>>>>>> bin/yarn nodemanager >> >>>>>>> >> >>>>>>> With YARN_HOME pointing to a directory that has a subdirectory >> >>>>>>> called "conf" in it, which has a yarn-site.xml in it: >> >>>>>>> >> >>>>>>> /path/to/our/yarn-home/conf/yarn-site.xml >> >>>>>>> >> >>>>>>> This yarn-site.xml has yarn.resourcemanager.hostname set to the >> >>>>>>> IP (or >> >>>>>>> hostname) of the resource manager: >> >>>>>>> >> >>>>>>> <property> >> >>>>>>> <name>yarn.resourcemanager.hostname</name> >> >>>>>>> <value>123.456.789.123</value> >> >>>>>>> </property> >> >>>>>>> >> >>>>>>> >> >>>>>>> Cheers, >> >>>>>>> Chris >> >>>>>>> >> >>>>>>> On 3/12/14 5:33 PM, "[email protected]" >> >>>>>>> <[email protected]> wrote: >> >>>>>>> >> >>>>>>>> I see two active nodes (I have 2 NMs running) >> >>>>>>>> >> >>>>>>>> -----Original Message----- >> >>>>>>>> From: Chris Riccomini [mailto:[email protected]] >> >>>>>>>> Sent: Wednesday, March 12, 2014 5:24 PM >> >>>>>>>> To: [email protected] >> >>>>>>>> Subject: Re: Failed to package using mvn >> >>>>>>>> >> >>>>>>>> Hey Sonali, >> >>>>>>>> >> >>>>>>>> Can you go to your ResourceManager's UI, and tell me how many >> >>>>>>>>active nodes you see? This should be under the "active nodes" >> >>>>>>>>heading. >> >>>>>>>> >> >>>>>>>> It sounds like the SamzaAppMaster is not getting the resource >> >>>>>>>> manager host/port from the yarn-site.xml. Usually this is due >> >>>>>>>> to not exporting YARN_HOME on the NodeManager before starting it. >> >>>>>>>> >> >>>>>>>> Cheers, >> >>>>>>>> Chris >> >>>>>>>> >> >>>>>>>> On 3/12/14 5:21 PM, "[email protected]" >> >>>>>>>> <[email protected]> wrote: >> >>>>>>>> >> >>>>>>>>> Okay so I was able to submit the job: >> >>>>>>>>> >> >>>>>>>>> In the nodemanager I get this error: Specifically it's trying >> >>>>>>>>> to connect to 0.0.0.0/8032 instead of the IP I have specified >> >>>>>>>>> in the yarn-site.xml file >> >>>>>>>>> >> >>>>>>>>> 2014-03-12 17:04:47 SamzaAppMaster$ [INFO] got container id: >> >>>>>>>>> container_1391637982288_0033_01_000001 >> >>>>>>>>> 2014-03-12 17:04:47 SamzaAppMaster$ [INFO] got app attempt id: >> >>>>>>>>> appattempt_1391637982288_0033_000001 >> >>>>>>>>> 2014-03-12 17:04:47 SamzaAppMaster$ [INFO] got node manager >> host: >> >>>>>>>>> svdpdac001.techlabs.accenture.com >> >>>>>>>>> 2014-03-12 17:04:47 SamzaAppMaster$ [INFO] got node manager >> port: >> >>>>>>>>> 38218 >> >>>>>>>>> 2014-03-12 17:04:47 SamzaAppMaster$ [INFO] got node manager >> >>>>>>>>>http >> >>>>>>>>>port: >> >>>>>>>>> 8042 >> >>>>>>>>> 2014-03-12 17:04:47 SamzaAppMaster$ [INFO] got config: >> >>>>>>>>> >> >>>>>>>>>{task.inputs=wikipedia.#en.wikipedia,wikipedia.#en.wiktionary,w >> >>>>>>>>>i >> >>>>>>>>>kip edi a .#e n.wikinews, >> >>>>>>>>>systems.wikipedia.host=irc.wikimedia.org, >> >>>>>>>>> systems.kafka.producer.batch.num.messages=1, >> >>>>>>>>> job.factory.class=org.apache.samza.job.yarn.YarnJobFactory, >> >>>>>>>>> systems.wikipedia.port=6667, >> >>>>>>>>> systems.kafka.producer.producer.type=sync, >> >>>>>>>>> job.name=wikipedia-feed, >> >>>>>>>>> >> >>>>>>>>>systems.kafka.consumer.zookeeper.connect=svdpdac013.techlabs.ac >> >>>>>>>>>c >> >>>>>>>>>ent >> >>>>>>>>>ure >> >>>>>>>>>. >> >>>>>>>>> com :2181/, systems.kafka.samza.msg.serde=json, >> >>>>>>>>> >> >>>>>>>>>serializers.registry.json.class=org.apache.samza.serializers.Js >> >>>>>>>>>o >> >>>>>>>>>nSe >> >>>>>>>>>rde >> >>>>>>>>> F >> >>>>>>>>> act ory, >> >>>>>>>>> >> >>>>>>>>>task.class=samza.examples.wikipedia.task.WikipediaFeedStreamTas >> >>>>>>>>>k >> >>>>>>>>>, >> >>>>>>>>> >> >>>>>>>>>yarn.package.path=hdfs://10.1.174.85:9000/samza-job-package-0.7. >> >>>>>>>>>0-d >> >>>>>>>>>ist >> >>>>>>>>>. >> >>>>>>>>> tar >> >>>>>>>>> .gz, >> >>>>>>>>> >> >>>>>>>>>systems.wikipedia.samza.factory=samza.examples.wikipedia.system. >> >>>>>>>>>Wik >> >>>>>>>>>ipe >> >>>>>>>>> d >> >>>>>>>>> iaS >> >>>>>>>>> ystemFactory, >> >>>>>>>>> >> >>>>>>>>>systems.kafka.samza.factory=org.apache.samza.system.kafka.Kafka >> >>>>>>>>>S >> >>>>>>>>>yst >> >>>>>>>>>emF >> >>>>>>>>> a >> >>>>>>>>> cto >> >>>>>>>>> ry, >> >>>>>>>>> >> >>>>>>>>>systems.kafka.producer.metadata.broker.list=svdpdac001.techlabs. >> >>>>>>>>>acc >> >>>>>>>>>ent >> >>>>>>>>> ure >> >>>>>>>>> . >> >>>>>>>>> com:6667,svdpdac015.techlabs.accenture.com:6667} >> >>>>>>>>> 2014-03-12 17:04:48 ClientHelper [INFO] trying to connect to >> >>>>>>>>>RM >> >>>>>>>>> 0.0.0.0:8032 >> >>>>>>>>> 2014-03-12 17:04:48 NativeCodeLoader [WARN] Unable to load >> >>>>>>>>>native-hadoop library for your platform... using builtin-java >> >>>>>>>>>classes where applicable >> >>>>>>>>> 2014-03-12 17:04:48 RMProxy [INFO] Connecting to >> >>>>>>>>>ResourceManager at >> >>>>>>>>> /0.0.0.0:8032 >> >>>>>>>>> >> >>>>>>>>> -----Original Message----- >> >>>>>>>>> From: Chris Riccomini [mailto:[email protected]] >> >>>>>>>>> Sent: Wednesday, March 12, 2014 4:48 PM >> >>>>>>>>> To: [email protected] >> >>>>>>>>> Subject: Re: Failed to package using mvn >> >>>>>>>>> >> >>>>>>>>> Hey Sonali, >> >>>>>>>>> >> >>>>>>>>> You need to specify a valid HDFS uri. Usually something like: >> >>>>>>>>> >> >>>>>>>>> hdfs://<hdfs name node ip>:<hdfs name node port>/path/to/tgz >> >>>>>>>>> >> >>>>>>>>> Right now, Hadoop is trying to use the package name as the >> >>>>>>>>>HDFS host. >> >>>>>>>>> >> >>>>>>>>> Cheers, >> >>>>>>>>> Chris >> >>>>>>>>> >> >>>>>>>>> On 3/12/14 4:45 PM, "[email protected]" >> >>>>>>>>> <[email protected]> wrote: >> >>>>>>>>> >> >>>>>>>>>> I did and I can now see the hadoop-hdfs jar in >> >>>>>>>>>> /deploy/samza/lib folder. >> >>>>>>>>>> >> >>>>>>>>>> I do get a different error now. >> >>>>>>>>>> >> >>>>>>>>>> I uploaded the samza-job to hdfs and it resides on >> >>>>>>>>>> hdfs://samza-job-package-0.7.0-dist.tar.gz >> >>>>>>>>>> >> >>>>>>>>>> But when I run the job I get this exception: >> >>>>>>>>>> >> >>>>>>>>>> Exception in thread "main" java.lang.IllegalArgumentException: >> >>>>>>>>>> java.net.UnknownHostException: >> >>>>>>>>>>samza-job-package-0.7.0-dist.tar.gz >> >>>>>>>>>> at >> >>>>>>>>>> >> >>>>>>>>>>org.apache.hadoop.security.SecurityUtil.buildTokenService(Secu >> >>>>>>>>>>r >> >>>>>>>>>>ity >> >>>>>>>>>>Uti >> >>>>>>>>>>l. >> >>>>>>>>>> jav >> >>>>>>>>>> a:418) >> >>>>>>>>>> at >> >>>>>>>>>> >> >>>>>>>>>>org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(NameNo >> >>>>>>>>>>d >> >>>>>>>>>>ePr >> >>>>>>>>>>oxi >> >>>>>>>>>> e >> >>>>>>>>>> s >> >>>>>>>>>> .ja >> >>>>>>>>>> va:231) >> >>>>>>>>>> at >> >>>>>>>>>> >> >>>>>>>>>>org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodePro >> >>>>>>>>>>x >> >>>>>>>>>>ies >> >>>>>>>>>>.ja >> >>>>>>>>>> v >> >>>>>>>>>> a >> >>>>>>>>>> :13 >> >>>>>>>>>> 9) >> >>>>>>>>>> at >> >>>>>>>>>>org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:510) >> >>>>>>>>>> at >> >>>>>>>>>>org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:453) >> >>>>>>>>>> at >> >>>>>>>>>> >> >>>>>>>>>>org.apache.hadoop.hdfs.DistributedFileSystem.initialize(Distri >> >>>>>>>>>>b >> >>>>>>>>>>ute >> >>>>>>>>>>dFi >> >>>>>>>>>> l >> >>>>>>>>>> e >> >>>>>>>>>> Sys >> >>>>>>>>>> tem.java:136) >> >>>>>>>>>> at >> >>>>>>>>>> >> >>>>>>>>>>org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.ja >> >>>>>>>>>>v >> >>>>>>>>>>a:2 >> >>>>>>>>>>433 >> >>>>>>>>>>) >> >>>>>>>>>> at >> >>>>>>>>>> org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:88) >> >>>>>>>>>> at >> >>>>>>>>>> >> >>>>>>>>>>org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.j >> >>>>>>>>>>ava >> >>>>>>>>>>: >> >>>>>>>>>>246 >> >>>>>>>>>>7) >> >>>>>>>>>> at >> >>>>>>>>>> org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2449) >> >>>>>>>>>> at >> org.apache.hadoop.fs.FileSystem.get(FileSystem.java:367) >> >>>>>>>>>> at org.apache.hadoop.fs.Path.getFileSystem(Path.java:287) >> >>>>>>>>>> at >> >>>>>>>>>> >> >>>>>>>>>>org.apache.samza.job.yarn.ClientHelper.submitApplication(Clien >> >>>>>>>>>>t >> >>>>>>>>>>Hel >> >>>>>>>>>>per >> >>>>>>>>>>. >> >>>>>>>>>> s >> >>>>>>>>>> cal >> >>>>>>>>>> a:111) >> >>>>>>>>>> at >> >>>>>>>>>>org.apache.samza.job.yarn.YarnJob.submit(YarnJob.scala:55) >> >>>>>>>>>> at >> >>>>>>>>>>org.apache.samza.job.yarn.YarnJob.submit(YarnJob.scala:48) >> >>>>>>>>>> at org.apache.samza.job.JobRunner.run(JobRunner.scala:100) >> >>>>>>>>>> at >> org.apache.samza.job.JobRunner$.main(JobRunner.scala:75) >> >>>>>>>>>> at org.apache.samza.job.JobRunner.main(JobRunner.scala) >> >>>>>>>>>> Caused by: java.net.UnknownHostException: >> >>>>>>>>>> samza-job-package-0.7.0-dist.tar.gz >> >>>>>>>>>> ... 18 more >> >>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>>>> -----Original Message----- >> >>>>>>>>>> From: Yan Fang [mailto:[email protected]] >> >>>>>>>>>> Sent: Wednesday, March 12, 2014 4:20 PM >> >>>>>>>>>> To: [email protected] >> >>>>>>>>>> Subject: Re: Failed to package using mvn >> >>>>>>>>>> >> >>>>>>>>>> Hi Sonali, >> >>>>>>>>>> >> >>>>>>>>>> One tip you may miss: >> >>>>>>>>>> >> >>>>>>>>>> If you had already run >> >>>>>>>>>> >> >>>>>>>>>> tar -xvf >> >>>>>>>>>> ./samza-job-package/target/samza-job-package-0.7.0-dist.tar.g >> >>>>>>>>>> z >> >>>>>>>>>> -C deploy/samza >> >>>>>>>>>> >> >>>>>>>>>> before you bundled the jar file to tar.gz. Please also >> >>>>>>>>>> remember to put the hdfs jar file to the deploy/samza/lib. >> >>>>>>>>>> >> >>>>>>>>>> Let me know if you miss this step. >> >>>>>>>>>> >> >>>>>>>>>> Thanks, >> >>>>>>>>>> >> >>>>>>>>>> Fang, Yan >> >>>>>>>>>> [email protected] >> >>>>>>>>>> +1 (206) 849-4108 >> >>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>>>> On Wed, Mar 12, 2014 at 4:10 PM, Chris Riccomini >> >>>>>>>>>> <[email protected]>wrote: >> >>>>>>>>>> >> >>>>>>>>>>> Hey Sonali, >> >>>>>>>>>>> >> >>>>>>>>>>> Yan has made a step-by-step tutorial for this. Could you >> >>>>>>>>>>>confirm that you've followed the instructions, and it's >> >>>>>>>>>>>still not working? >> >>>>>>>>>>> >> >>>>>>>>>>> https://issues.apache.org/jira/browse/SAMZA-181 >> >>>>>>>>>>> >> >>>>>>>>>>> Cheers, >> >>>>>>>>>>> Chris >> >>>>>>>>>>> >> >>>>>>>>>>> On 3/12/14 3:12 PM, "[email protected]" >> >>>>>>>>>>> <[email protected]> wrote: >> >>>>>>>>>>> >> >>>>>>>>>>>> So sigh! I had some Kafka issues in-between. That's fixed >> now. >> >>>>>>>>>>>> >> >>>>>>>>>>>> As suggested, >> >>>>>>>>>>>> >> >>>>>>>>>>>> 1. I made sure the hadoop-hdfs-2.2.0.jar is bundled with >> >>>>>>>>>>>> the samza job tar.gz. >> >>>>>>>>>>>> 2. I added the configuration to implement hdfs in the >> >>>>>>>>>>>> hdfs-site.xml files both on the NMs and in the /conf >> >>>>>>>>>>>> directory for samza >> >>>>>>>>>>>> >> >>>>>>>>>>>> I still get the No Filesystem for scheme :hdfs error. >> >>>>>>>>>>>> >> >>>>>>>>>>>> Is there anything else im missing? >> >>>>>>>>>>>> Thanks, >> >>>>>>>>>>>> Sonali >> >>>>>>>>>>>> >> >>>>>>>>>>>> >> >>>>>>>>>>>> -----Original Message----- >> >>>>>>>>>>>> From: Chris Riccomini [mailto:[email protected]] >> >>>>>>>>>>>> Sent: Tuesday, March 11, 2014 8:27 PM >> >>>>>>>>>>>> To: [email protected] >> >>>>>>>>>>>> Subject: Re: Failed to package using mvn >> >>>>>>>>>>>> >> >>>>>>>>>>>> Hey Yan, >> >>>>>>>>>>>> >> >>>>>>>>>>>> This looks great! I added a few requests to the JIRA, if >> >>>>>>>>>>>> you have >> >>>>>>>>>>> time. >> >>>>>>>>>>>> >> >>>>>>>>>>>> Cheers, >> >>>>>>>>>>>> Chris >> >>>>>>>>>>>> >> >>>>>>>>>>>>> On 3/11/14 7:20 PM, "Yan Fang" <[email protected]> >> wrote: >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> Hi Chris, >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> Has opened an issue >> >>>>>>>>>>>>> SAMZA-181<https://issues.apache.org/jira/browse/SAMZA-181> >> >>>>>>>>>>>>> a nd also uploaded the patch. Let me know if there is >> >>>>>>>>>>>>> something wrong in my tutorial. Thank you! >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> Cheers, >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> Fang, Yan >> >>>>>>>>>>>>> [email protected] >> >>>>>>>>>>>>> +1 (206) 849-4108 >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> On Tue, Mar 11, 2014 at 10:40 AM, >> >>>>>>>>>>>>> <[email protected]>wrote: >> >>>>>>>>>>>>> >> >>>>>>>>>>>>>> Thanks Chris, Yan, >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> Let me try that. >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> -----Original Message----- >> >>>>>>>>>>>>>> From: Chris Riccomini [mailto:[email protected]] >> >>>>>>>>>>>>>> Sent: Tuesday, March 11, 2014 10:22 AM >> >>>>>>>>>>>>>> To: [email protected] >> >>>>>>>>>>>>>> Subject: Re: Failed to package using mvn >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> Hey Yan, >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> Awesome!The location where you can add your .md is here: >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> docs/learn/tutorials/0.7.0/ >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> Here's a link to the code tree: >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> https://git-wip-us.apache.org/repos/asf?p=incubator-samza. >> >>>>>>>>>>>>>> git >> >>>>>>>>>>>>>> ;a= >> >>>>>>>>>>>>>> t >> >>>>>>>>>>>>>> r >> >>>>>>>>>>>>>> e >> >>>>>>>>>>>>>> e;f >> >>>>>>>>>>>>>> =do >> >>>>>>>>>>>>>> cs >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> /learn/tutorials/0.7.0;h=ef117f4066f14a00f50f0f6fca179031 >> >>>>>>>>>>>>>> 3 >> >>>>>>>>>>>>>> 044 >> >>>>>>>>>>>>>> 831 >> >>>>>>>>>>>>>> 2 >> >>>>>>>>>>>>>> ; >> >>>>>>>>>>>>>> h >> >>>>>>>>>>>>>> b=H >> >>>>>>>>>>>>>> EAD >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> You can get the code here: >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> git clone >> >>>>>>>>>>>>>> http://git-wip-us.apache.org/repos/asf/incubator-samza.gi >> >>>>>>>>>>>>>> t >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> Once you write the .md, just throw it up on a JIRA, and >> >>>>>>>>>>>>>> one of us can merge it in. >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> Re: hdfs-site.xml, ah ha, that's what I figured. This is >> >>>>>>>>>>>>>> good to >> >>>>>>>>>>> know. >> >>>>>>>>>>>>>> So >> >>>>>>>>>>>>>> you just copy your hdfs-site.xml from your NodeManager's >> >>>>>>>>>>>>>> conf directory into your local hdfs-site.xml. >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> Cheers, >> >>>>>>>>>>>>>> Chris >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> On 3/11/14 10:16 AM, "Yan Fang" <[email protected]> >> >>>>>>>>>>>>>>>wrote: >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> Hi Chris, >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> Sure. I just do not know how/where to contribute this >> >>>>>>>>>>>>>>> page...*_* >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> Oh, I mean the same this as you mentioned in the >> >>>>>>>>>>>>>>> *Cluster >> >>>>>>>>>>>>>>> Installation*thread: >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> *"2. Get a copy of one of your NM's yarn-site.xml and >> >>>>>>>>>>>>>>> put it somewhere >> >>>>>>>>>>>>>>> on* >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> *your desktop (I usually use ~/.yarn/conf/yarn-site.xml). >> >>>>>>>>>>>>>>> Note that there'sa "conf" directory there. This is >> >>>>>>>>>>>>>>> mandatory."* >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> So I just copy the hdfs-site.xml to >> >>>>>>>>>>>>>>>~/.yarn/conf/hdfs-site.xml. >> >>>>>>>>>>>>>>> Thank >> >>>>>>>>>>>>>> you. >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> Cheers, >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> Fang, Yan >> >>>>>>>>>>>>>>> [email protected] >> >>>>>>>>>>>>>>> +1 (206) 849-4108 >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> On Tue, Mar 11, 2014 at 10:10 AM, Chris Riccomini >> >>>>>>>>>>>>>>> <[email protected]>wrote: >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>> Hey Yan, >> >>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>> Would you be up for contributing a tutorial page that >> >>>>>>>>>>>>>>>> describes >> >>>>>>>>>>>>>> this? >> >>>>>>>>>>>>>>>> This >> >>>>>>>>>>>>>>>> is really useful information. Our docs are just simple >> >>>>>>>>>>>>>>>> .md files in the main code base. >> >>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>> Regarding step (3), is the hdfs-site.xml put into the >> >>>>>>>>>>>>>>>> conf folder for the NM boxes, or on the client side >> >>>>>>>>>>>>>>>> (where run-job.sh >> >>>>>>>>>>> is run)? >> >>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>> Cheers, >> >>>>>>>>>>>>>>>> Chris >> >>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>> On 3/11/14 10:07 AM, "Yan Fang" <[email protected]> >> >>>>>>>>>>>>>>>>>wrote: >> >>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>> Hi Sonali, >> >>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>> The way I make Samza run with HDFS is following: >> >>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>> 1. include hdfs jar in Samza jar tar.gz. >> >>>>>>>>>>>>>>>>> 2. you may also want to make sure the >> >>>>>>>>>>>>>>>>>hadoop-common.jar has the same version as your hdfs >> >>>>>>>>>>>>>>>>>jar. Otherwise, you may have configuration error >> popping out. >> >>>>>>>>>>>>>>>>> 3. then put hdfs-site.xml to conf folder, the same >> >>>>>>>>>>>>>>>>>folder as the yarn-site.xml 4. all other steps are not >> >>>>>>>>>>>>>>>>>changed. >> >>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>> Hope this will help. Thank you. >> >>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>> Cheers, >> >>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>> Fang, Yan >> >>>>>>>>>>>>>>>>> [email protected] >> >>>>>>>>>>>>>>>>> +1 (206) 849-4108 >> >>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>> On Tue, Mar 11, 2014 at 9:25 AM, Chris Riccomini >> >>>>>>>>>>>>>>>>> <[email protected]>wrote: >> >>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>> Hey Sonali, >> >>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>> I believe that you need to make sure that the HDFS >> >>>>>>>>>>>>>>>>>> jar is in your .tar.gz file, as you've said. >> >>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>> If that doesn't work, you might need to define this >> >>>>>>>>>>>>>>>>>> setting in core-site.xml on the machine you're >> >>>>>>>>>>>>>>>>>> running >> >>>>>>>>>>> run-job.sh on: >> >>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>> <property> >> >>>>>>>>>>>>>>>>>> <name>fs.hdfs.impl</name> >> >>>>>>>>>>> <value>org.apache.hadoop.hdfs.DistributedFileSystem</value> >> >>>>>>>>>>>>>>>>>> <description>The FileSystem for hdfs: >> >>>>>>>>>>>>>>>>>> uris.</description> </property> >> >>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>> You might also need to configure your NodeManagers to >> >>>>>>>>>>>>>>>>>> have the HDFS >> >>>>>>>>>>>>>>>> file >> >>>>>>>>>>>>>>>>>> system impl as well. >> >>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>> I've never run Samza with HDFS, so I'm guessing here. >> >>>>>>>>>>>>>>>>>> Perhaps someone else on the list has been successful >> >>>>>>>>>>>>>>>>>> with >> >>>>>>>>>>> this? >> >>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>> Cheers, >> >>>>>>>>>>>>>>>>>> Chris >> >>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>> On 3/10/14 3:59 PM, " >> [email protected]" >> >>>>>>>>>>>>>>>>>> <[email protected]> wrote: >> >>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>> Hello, >> >>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>> I fixed this by starting from scratch with gradlew. >> >>>>>>>>>>>>>>>>>>> But now when I >> >>>>>>>>>>>>>>>> run >> >>>>>>>>>>>>>>>>>> my >> >>>>>>>>>>>>>>>>>>> job it throws this error: >> >>>>>>>>>>>>>>>>>>> Exception in thread "main" java.io.IOException: No >> >>>>>>>>>>>>>>>>>>> FileSystem for >> >>>>>>>>>>>>>>>>>> scheme: >> >>>>>>>>>>>>>>>>>>> hdfs >> >>>>>>>>>>>>>>>>>>> at >> >>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>org.apache.hadoop.fs.FileSystem.getFileSystemClass(Fi >> >>>>>>>>>>>>>>>>>>>l >> >>>>>>>>>>>>>>>>>>>eSy >> >>>>>>>>>>>>>>>>>>>ste >> >>>>>>>>>>>>>>>>>>>m. >> >>>>>>>>>>>>>>>>>>> jav >> >>>>>>>>>>>>>>>>>>> a: >> >>>>>>>>>>>>>>>>>>> 242 >> >>>>>>>>>>>>>>>>>>> 1) >> >>>>>>>>>>>>>>>>>>> at >> >>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>org.apache.hadoop.fs.FileSystem.createFileSystem(FileSy >> >>>>>>>>>>>>>>>>>s >> >>>>>>>>>>>>>>>>>tem >> >>>>>>>>>>>>>>>>>. >> >>>>>>>>>>>>>>>>> j >> >>>>>>>>>>>>>>>>> a >> >>>>>>>>>>>>>>>>> v >> >>>>>>>>>>>>>>>>> a:2 >> >>>>>>>>>>>>>>>>> 428 >> >>>>>>>>>>>>>>>>> ) >> >>>>>>>>>>>>>>>>>>> at >> >>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>org.apache.hadoop.fs.FileSystem.access$200(FileSystem. >> >>>>>>>>>>>>>>>>>>j >> >>>>>>>>>>>>>>>>>>ava >> >>>>>>>>>>>>>>>>>>: >> >>>>>>>>>>>>>>>>>> 8 >> >>>>>>>>>>>>>>>>>> 8 >> >>>>>>>>>>>>>>>>>> ) >> >>>>>>>>>>>>>>>>>>> at >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>>org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSyst >> >>>>>>>>>>>>>>e >> >>>>>>>>>>>>>>m.j >> >>>>>>>>>>>>>>ava >> >>>>>>>>>>>>>>: >> >>>>>>>>>>>>>>>>> 246 >> >>>>>>>>>>>>>>>>> 7) >> >>>>>>>>>>>>>>>>>>> at >> >>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.j >> >>>>>>>>>>>>>>>>>>ava >> >>>>>>>>>>>>>>>>>>: >> >>>>>>>>>>>>>>>>>> 2 >> >>>>>>>>>>>>>>>>>> 4 >> >>>>>>>>>>>>>>>>>> 4 >> >>>>>>>>>>>>>>>>>> 9) >> >>>>>>>>>>>>>>>>>>> at >> >>>>>>>>>>>>>> org.apache.hadoop.fs.FileSystem.get(FileSystem.java:367) >> >>>>>>>>>>>>>>>>>>> at >> >>>>>>>>>>>>>> org.apache.hadoop.fs.Path.getFileSystem(Path.java:287) >> >>>>>>>>>>>>>>>>>>> at >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>> org.apache.samza.job.yarn.ClientHelper.submitApplica >> >>>>>>>>>>>>>>>>>>> t >> >>>>>>>>>>>>>>>>>>> ion >> >>>>>>>>>>>>>>>>>>> (Cl >> >>>>>>>>>>>>>>>>>>> i >> >>>>>>>>>>>>>>>>>>> e >> >>>>>>>>>>>>>>>>>>> n >> >>>>>>>>>>>>>>>>>>> tHe >> >>>>>>>>>>>>>>>>>>> lpe >> >>>>>>>>>>>>>>>>>>> r. >> >>>>>>>>>>>>>>>>>>> sc >> >>>>>>>>>>>>>>>>>>> al >> >>>>>>>>>>>>>>>>>>> a:111) >> >>>>>>>>>>>>>>>>>>> at >> >>>>>>>>>>>>>>>> org.apache.samza.job.yarn.YarnJob.submit(YarnJob.scala: >> >>>>>>>>>>>>>>>> 5 >> >>>>>>>>>>>>>>>> 5) >> >>>>>>>>>>>>>>>>>>> at >> >>>>>>>>>>>>>>>> org.apache.samza.job.yarn.YarnJob.submit(YarnJob.scala: >> >>>>>>>>>>>>>>>> 4 >> >>>>>>>>>>>>>>>> 8) >> >>>>>>>>>>>>>>>>>>> at >> >>>>>>>>>>>>>> org.apache.samza.job.JobRunner.run(JobRunner.scala:100) >> >>>>>>>>>>>>>>>>>>> at >> >>>>>>>>>>>>>> org.apache.samza.job.JobRunner$.main(JobRunner.scala:75) >> >>>>>>>>>>>>>>>>>>> at >> >>>>>>>>>>>>>>>>>>> org.apache.samza.job.JobRunner.main(JobRunner.scala) >> >>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>> I looked at the samza job tar.gz and it doesn't have >> >>>>>>>>>>>>>>>>>>> a Hadoop-hdfs >> >>>>>>>>>>>>>>>> jar. >> >>>>>>>>>>>>>>>>>>> Is that why I get this error? >> >>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>> Thanks, >> >>>>>>>>>>>>>>>>>>> Sonali >> >>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>> From: Parthasarathy, Sonali >> >>>>>>>>>>>>>>>>>>> Sent: Monday, March 10, 2014 11:25 AM >> >>>>>>>>>>>>>>>>>>> To: [email protected] >> >>>>>>>>>>>>>>>>>>> Subject: Failed to package using mvn >> >>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>> Hi, >> >>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>> When I tried to do a mvn clean package of my >> >>>>>>>>>>>>>>>>>>> hello-samza project, I >> >>>>>>>>>>>>>>>> get >> >>>>>>>>>>>>>>>>>>> the following error. Has anyone seen this before? >> >>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>> [ERROR] Failed to execute goal on project >> >>>>>>>>>>>>>>>>>>>samza-wikipedia: >> >>>>>>>>>>>>>>>>>>> Could not resolve dependencies for project >> >>>>>>>>>>>>>> samza:samza-wikipedia:jar:0.7.0: >> >>>>>>>>>>>>>>>> Could >> >>>>>>>>>>>>>>>>>>> not find artifact >> >>>>>>>>>>>>>>>>>>> org.apache.samza:samza-kv_2.10:jar:0.7.0 >> >>>>>>>>>>>>>>>>>>> in apache-releases >> >>>>>>>>>>>>>>>> (https://repository.apache.org/content/groups/public) >> >>>>>>>>>>>>>>>>>> -> >> >>>>>>>>>>>>>>>>>>> [Help 1] >> >>>>>>>>>>>>>>>>>>> [ERROR] >> >>>>>>>>>>>>>>>>>>> [ERROR] To see the full stack trace of the errors, >> >>>>>>>>>>>>>>>>>>> re-run Maven with >> >>>>>>>>>>>>>>>>>> the >> >>>>>>>>>>>>>>>>>>> -e switch. >> >>>>>>>>>>>>>>>>>>> [ERROR] Re-run Maven using the -X switch to enable >> >>>>>>>>>>>>>>>>>>> full debug >> >>>>>>>>>>>>>>>> logging. >> >>>>>>>>>>>>>>>>>>> [ERROR] >> >>>>>>>>>>>>>>>>>>> [ERROR] For more information about the errors and >> >>>>>>>>>>>>>>>>>>> possible >> >>>>>>>>>>>>>>>> solutions, >> >>>>>>>>>>>>>>>>>>> please read the following articles: >> >>>>>>>>>>>>>>>>>>> [ERROR] [Help 1] >> >>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>> http://cwiki.apache.org/confluence/display/MAVEN/Depend >> >>>>>>>>>>>>>>>> e >> >>>>>>>>>>>>>>>> ncy >> >>>>>>>>>>>>>>>> Re >> >>>>>>>>>>>>>>>> s >> >>>>>>>>>>>>>>>> o >> >>>>>>>>>>>>>>>> l >> >>>>>>>>>>>>>>>> uti >> >>>>>>>>>>>>>>>> onE >> >>>>>>>>>>>>>>>> xce >> >>>>>>>>>>>>>>>>>> p >> >>>>>>>>>>>>>>>>>>> tion >> >>>>>>>>>>>>>>>>>>> [ERROR] >> >>>>>>>>>>>>>>>>>>> [ERROR] After correcting the problems, you can >> >>>>>>>>>>>>>>>>>>> resume the build with >> >>>>>>>>>>>>>>>>>> the >> >>>>>>>>>>>>>>>>>>> command >> >>>>>>>>>>>>>>>>>>> [ERROR] mvn <goals> -rf :samza-wikipedia >> >>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>> Thanks, >> >>>>>>>>>>>>>>>>>>> Sonali >> >>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>> Sonali Parthasarathy R&D Developer, Data Insights >> >>>>>>>>>>>>>>>>>>> Accenture Technology Labs >> >>>>>>>>>>>>>>>>>>> 703-341-7432 >> >>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>> ________________________________ >> >>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>> This message is for the designated recipient only >> >>>>>>>>>>>>>>>>>>> and may contain privileged, proprietary, or >> >>>>>>>>>>>>>>>>>>> otherwise confidential >> >>>>>>>>>>>>>> information. >> >>>>>>>>>>>>>>>>>>> If >> >>>>>>>>>>>>>>>> you >> >>>>>>>>>>>>>>>>>>> have received it in error, please notify the sender >> >>>>>>>>>>>>>>>>>>> immediately and delete the original. Any other use >> >>>>>>>>>>>>>>>>>>> of the e-mail by you is >> >>>>>>>>>>>>>>>> prohibited. >> >>>>>>>>>>>>>>>>>>> Where allowed by local law, electronic >> >>>>>>>>>>>>>>>>>>> communications with Accenture >> >>>>>>>>>>>>>>>>>> and >> >>>>>>>>>>>>>>>>>>> its affiliates, including e-mail and instant >> >>>>>>>>>>>>>>>>>>> messaging (including content), may be scanned by our >> >>>>>>>>>>>>>>>>>>> systems for the purposes of >> >>>>>>>>>>>>>>>> information >> >>>>>>>>>>>>>>>>>>> security and assessment of internal compliance with >> >>>>>>>>>>>>>>>>>>> Accenture >> >>>>>>>>>>>>>>>> policy. >> >>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>> ____________________________________________________ >> >>>>>>>>>>>>>>>>>>> _ >> >>>>>>>>>>>>>>>>>>> ___ >> >>>>>>>>>>>>>>>>>>> __ >> >>>>>>>>>>>>>>>>>>> _ >> >>>>>>>>>>>>>>>>>>> _ >> >>>>>>>>>>>>>>>>>>> _ >> >>>>>>>>>>>>>>>>>>> ___ >> >>>>>>>>>>>>>>>>>>> ___ >> >>>>>>>>>>>>>>>>>>> ___ >> >>>>>>>>>>>>>>>>>>> __ >> >>>>>>>>>>>>>>>>>>> __ >> >>>>>>>>>>>>>>>>>>> ____________ >> >>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>> www.accenture.com >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> ________________________________ >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> This message is for the designated recipient only and may >> >>>>>>>>>>>>>> contain privileged, proprietary, or otherwise >> >>>>>>>>>>>>>> confidential information. If you have received it in >> >>>>>>>>>>>>>> error, please notify the sender immediately and delete the >> original. >> >>>>>>>>>>>>>> Any other use of the e-mail by you is prohibited. Where >> >>>>>>>>>>>>>> allowed by local law, electronic communications with >> >>>>>>>>>>>>>> Accenture and its affiliates, including e-mail and >> >>>>>>>>>>>>>> instant messaging (including content), may be scanned by >> >>>>>>>>>>>>>> our systems for the purposes of information security and >> >>>>>>>>>>>>>> assessment of internal compliance with Accenture >> >>>>>>>>>>> policy. >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> _________________________________________________________ >> >>>>>>>>>>>>>> _ >> >>>>>>>>>>>>>> ___ >> >>>>>>>>>>>>>> ___ >> >>>>>>>>>>>>>> _ >> >>>>>>>>>>>>>> _ >> >>>>>>>>>>>>>> _ >> >>>>>>>>>>>>>> ___ >> >>>>>>>>>>>>>> ___ >> >>>>>>>>>>>>>> _____________ >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> www.accenture.com >> >>>>>>>>>>>> >> >>>>>>>>>>>> >> >>>>>>>>>>>> >> >>>>>>>>>>>> ________________________________ >> >>>>>>>>>>>> >> >>>>>>>>>>>> This message is for the designated recipient only and may >> >>>>>>>>>>>> contain privileged, proprietary, or otherwise confidential >> >>>>>>>>>>>> information. If you have received it in error, please >> >>>>>>>>>>>> notify the sender immediately and delete the original. Any >> >>>>>>>>>>>> other use of the e-mail by you is >> >>>>>>>>>>> prohibited. >> >>>>>>>>>>>> Where allowed by local law, electronic communications with >> >>>>>>>>>>>> Accenture and its affiliates, including e-mail and instant >> >>>>>>>>>>>> messaging (including content), may be scanned by our >> >>>>>>>>>>>> systems for the purposes of information security and >> >>>>>>>>>>>> assessment of internal compliance with >> >>>>>>>>>>> Accenture policy. >> >>>>>>>>>>>> ___________________________________________________________ >> >>>>>>>>>>>> _ >> >>>>>>>>>>>> ___ >> >>>>>>>>>>>> ___ >> >>>>>>>>>>>> _ >> >>>>>>>>>>>> _ >> >>>>>>>>>>>> _ >> >>>>>>>>>>>> _____ >> >>>>>>>>>>>> ____________ >> >>>>>>>>>>>> >> >>>>>>>>>>>> www.accenture.com >> >>>>>>>>>> >> >>>>>>>>>> ________________________________ >> >>>>>>>>>> >> >>>>>>>>>> This message is for the designated recipient only and may >> >>>>>>>>>>contain privileged, proprietary, or otherwise confidential >> >>>>>>>>>>information. If you have received it in error, please notify >> >>>>>>>>>>the sender immediately and delete the original. Any other use >> >>>>>>>>>>of the e-mail by you is prohibited. >> >>>>>>>>>> Where allowed by local law, electronic communications with >> >>>>>>>>>>Accenture and its affiliates, including e-mail and instant >> >>>>>>>>>>messaging (including content), may be scanned by our systems >> >>>>>>>>>>for the purposes of information security and assessment of >> >>>>>>>>>>internal compliance with Accenture policy. >> >>>>>>>>>> >> >>>>>>>>>>______________________________________________________________ >> >>>>>>>>>>_ >> >>>>>>>>>>___ >> >>>>>>>>>>___ >> >>>>>>>>>> _ >> >>>>>>>>>> _ >> >>>>>>>>>> ___ >> >>>>>>>>>> ____________ >> >>>>>>>>>> >> >>>>>>>>>> www.accenture.com >> >>>>>>>>> >> >>>>>>>>> >> >>>>>>>>> >> >>>>>>>>> ________________________________ >> >>>>>>>>> >> >>>>>>>>> This message is for the designated recipient only and may >> >>>>>>>>>contain privileged, proprietary, or otherwise confidential >> >>>>>>>>>information. If you have received it in error, please notify >> >>>>>>>>>the sender immediately and delete the original. Any other use >> >>>>>>>>>of the e-mail by you is prohibited. >> >>>>>>>>> Where allowed by local law, electronic communications with >> >>>>>>>>>Accenture and its affiliates, including e-mail and instant >> >>>>>>>>>messaging (including content), may be scanned by our systems >> >>>>>>>>>for the purposes of information security and assessment of >> >>>>>>>>>internal compliance with Accenture policy. >> >>>>>>>>> >> >>>>>>>>>_______________________________________________________________ >> >>>>>>>>>_ >> >>>>>>>>>___ >> >>>>>>>>>___ >> >>>>>>>>> _ >> >>>>>>>>> ___ >> >>>>>>>>> ____________ >> >>>>>>>>> >> >>>>>>>>> www.accenture.com >> >>>>>>>> >> >>>>>>>> >> >>>>>>>> >> >>>>>>>> ________________________________ >> >>>>>>>> >> >>>>>>>> This message is for the designated recipient only and may >> >>>>>>>>contain privileged, proprietary, or otherwise confidential >> >>>>>>>>information. If you have received it in error, please notify >> >>>>>>>>the sender immediately and delete the original. Any other use >> >>>>>>>>of the e-mail by you is prohibited. >> >>>>>>>> Where allowed by local law, electronic communications with >> >>>>>>>>Accenture and its affiliates, including e-mail and instant >> >>>>>>>>messaging (including content), may be scanned by our systems >> >>>>>>>>for the purposes of information security and assessment of >> >>>>>>>>internal compliance with Accenture policy. >> >>>>>>>> >> >>>>>>>>________________________________________________________________ >> >>>>>>>>_ >> >>>>>>>>___ >> >>>>>>>>___ >> >>>>>>>> ___ >> >>>>>>>> ____________ >> >>>>>>>> >> >>>>>>>> www.accenture.com >> >>>>>>> >> >>>>>>> >> >>>>>>> >> >>>>>>> ________________________________ >> >>>>>>> >> >>>>>>> This message is for the designated recipient only and may >> >>>>>>>contain privileged, proprietary, or otherwise confidential >> >>>>>>>information. If you have received it in error, please notify the >> >>>>>>>sender immediately and delete the original. Any other use of the >> >>>>>>>e-mail by you is prohibited. >> >>>>>>> Where allowed by local law, electronic communications with >> >>>>>>>Accenture and its affiliates, including e-mail and instant >> >>>>>>>messaging (including content), may be scanned by our systems for >> >>>>>>>the purposes of information security and assessment of internal >> >>>>>>>compliance with Accenture policy. >> >>>>>>> >> >>>>>>>_________________________________________________________________ >> >>>>>>>_ >> >>>>>>>___ >> >>>>>>>___ >> >>>>>>>__ >> >>>>>>> ____________ >> >>>>>>> >> >>>>>>> www.accenture.com >> >>>>>> >> >>>> >> >>>> >> >>>> >> >>>>________________________________ >> >>>> >> >>>>This message is for the designated recipient only and may contain >> >>>>privileged, proprietary, or otherwise confidential information. If >> >>>>you have received it in error, please notify the sender immediately >> >>>>and delete the original. Any other use of the e-mail by you is >> >>>>prohibited. >> >>>>Where allowed by local law, electronic communications with Accenture >> >>>>and its affiliates, including e-mail and instant messaging >> >>>>(including content), may be scanned by our systems for the purposes >> >>>>of information security and assessment of internal compliance with >> >>>>Accenture policy. >> >>>>____________________________________________________________________ >> >>>>_ >> >>>>___ >> >>>>_ >> >>>>_ >> >>>>____________ >> >>>> >> >>>>www.accenture.com >> >>>> >> >>> >> >> >> >> >> >> >> >>________________________________ >> >> >> >>This message is for the designated recipient only and may contain >> >>privileged, proprietary, or otherwise confidential information. If you >> >>have received it in error, please notify the sender immediately and >> >>delete the original. Any other use of the e-mail by you is prohibited. >> >>Where allowed by local law, electronic communications with Accenture >> >>and its affiliates, including e-mail and instant messaging (including >> >>content), may be scanned by our systems for the purposes of >> >>information security and assessment of internal compliance with >> Accenture policy. >> >>______________________________________________________________________ >> >>_ >> >>___ >> >>____________ >> >> >> >>www.accenture.com >> >> >> > >> > >> > >> >________________________________ >> > >> >This message is for the designated recipient only and may contain >> >privileged, proprietary, or otherwise confidential information. If you >> >have received it in error, please notify the sender immediately and >> >delete the original. Any other use of the e-mail by you is prohibited. >> >Where allowed by local law, electronic communications with Accenture >> >and its affiliates, including e-mail and instant messaging (including >> >content), may be scanned by our systems for the purposes of information >> >security and assessment of internal compliance with Accenture policy. >> >_______________________________________________________________________ >> >___ >> >____________ >> > >> >www.accenture.com >> > >> >> >> >> ________________________________ >> >> This message is for the designated recipient only and may contain >> privileged, proprietary, or otherwise confidential information. If you have >> received it in error, please notify the sender immediately and delete the >> original. Any other use of the e-mail by you is prohibited. Where allowed >> by local law, electronic communications with Accenture and its affiliates, >> including e-mail and instant messaging (including content), may be scanned >> by our systems for the purposes of information security and assessment of >> internal compliance with Accenture policy. >> >> ______________________________________________________________________________________ >> >> www.accenture.com >> >> > ________________________________ This message is for the designated recipient only and may contain privileged, proprietary, or otherwise confidential information. If you have received it in error, please notify the sender immediately and delete the original. Any other use of the e-mail by you is prohibited. Where allowed by local law, electronic communications with Accenture and its affiliates, including e-mail and instant messaging (including content), may be scanned by our systems for the purposes of information security and assessment of internal compliance with Accenture policy. ______________________________________________________________________________________ www.accenture.com
