Thanks Chris. So I'll have to pull the new code and rebuild the package?
-----Original Message----- From: Chris Riccomini [mailto:[email protected]] Sent: Thursday, March 13, 2014 11:04 AM To: [email protected] Subject: Re: Failed to package using mvn Hey Sonali, Please have a look at: https://issues.apache.org/jira/browse/SAMZA-182 I have posted a fix there. I have successfully downloaded YARN, setup a two node grid from scratch, and run hello-samza on it with the patch on SAMZA-182. Can you give that a shot for me? Thanks for your patience! Cheers, Chris On 3/13/14 10:58 AM, "[email protected]" <[email protected]> wrote: >Hi Chris, > >I checked my .bashrc. The variable was set on one of the NMs and not on >the other. I made the change and restarted the scripts. I still get the >same error. > >Also in my stderr I get: >Null identity service, trying login service: null Finding identity >service: null Null identity service, trying login service: null Finding >identity service: null > >-----Original Message----- >From: Chris Riccomini [mailto:[email protected]] >Sent: Wednesday, March 12, 2014 7:59 PM >To: [email protected] >Subject: Re: Failed to package using mvn > >Hey Guys, > >I was able to reproduce this problem. > >I was also able to fix it (without the patch in SAMZA-182). All I >needed to do was update ~/.bashrc on my NM's box to have: > > export YARN_HOME=/tmp/hadoop-2.3.0 > >It appears that the YARN environment variables are somehow getting lost >or not forwarded from the NM to the AM. Adding this bashrc setting >makes sure that the NM gets them. > > >I have a feeling upgrading Samza to YARN 2.3.0 will fix this, but I >haven't validated yet. I will continue to investigate tomorrow. > >Cheers, >Chris > >On 3/12/14 6:43 PM, "Yan Fang" <[email protected]> wrote: > >>I guess Sonali has the problem is because his NMs do not read the >>YARN_HOME variable. That may be because the NM machine does not have >>YARN_HOME set when the NM starts. >> >>Check this https://issues.apache.org/jira/browse/SAMZA-182 >> >>Thanks, >> >>Yan Fang >> >>> On Mar 12, 2014, at 6:14 PM, Chris Riccomini >>><[email protected]> >>>wrote: >>> >>> Hey Sonali, >>> >>> I am unfamiliar with the start-yarn.sh. Looking at: >>> >>> >>> >>>https://svn.apache.org/viewvc/hadoop/common/trunk/hadoop-yarn-project >>>/ >>>had >>>oo >>> p-yarn/bin/stop-yarn.sh?revision=1370666&view=markup >>> >>> What version of YARN are you using? >>> >>> Cheers, >>> Chris >>> >>> On 3/12/14 5:56 PM, "[email protected]" >>> <[email protected]> wrote: >>> >>>> Hey Chris, >>>> >>>> Yes, I have YARN_HOME set in all the NMs pointing to the right >>>>directories. I also made sure the yarn-site.xml file has the >>>>hostname set. >>>> >>>> I start yarn using start.yarn.sh in the RM and that automatically >>>>starts the NMs on the slave nodes. Is that the right way to do it? >>>> >>>> -----Original Message----- >>>> From: Chris Riccomini [mailto:[email protected]] >>>> Sent: Wednesday, March 12, 2014 5:52 PM >>>> To: [email protected] >>>> Subject: Re: Failed to package using mvn >>>> >>>> Hey Sonali, >>>> >>>> OK, so we've validated that the NMs are able to connect, which >>>>means they can see the yarn-site.xml. >>>> >>>> How are you starting your NMs? Are you running: >>>> >>>> export YARN_HOME=/path/to/yarn/home >>>> >>>> In the CLI before starting the NM? >>>> >>>> For reference, we run: >>>> >>>> export YARN_HOME=/path/to/our/yarn-home export >>>> YARN_CONF_DIR=$YARN_HOME/conf >>>> >>>> bin/yarn nodemanager >>>> >>>> With YARN_HOME pointing to a directory that has a subdirectory >>>> called "conf" in it, which has a yarn-site.xml in it: >>>> >>>> /path/to/our/yarn-home/conf/yarn-site.xml >>>> >>>> This yarn-site.xml has yarn.resourcemanager.hostname set to the IP >>>> (or >>>> hostname) of the resource manager: >>>> >>>> <property> >>>> <name>yarn.resourcemanager.hostname</name> >>>> <value>123.456.789.123</value> >>>> </property> >>>> >>>> >>>> Cheers, >>>> Chris >>>> >>>> On 3/12/14 5:33 PM, "[email protected]" >>>> <[email protected]> wrote: >>>> >>>>> I see two active nodes (I have 2 NMs running) >>>>> >>>>> -----Original Message----- >>>>> From: Chris Riccomini [mailto:[email protected]] >>>>> Sent: Wednesday, March 12, 2014 5:24 PM >>>>> To: [email protected] >>>>> Subject: Re: Failed to package using mvn >>>>> >>>>> Hey Sonali, >>>>> >>>>> Can you go to your ResourceManager's UI, and tell me how many >>>>>active nodes you see? This should be under the "active nodes" >>>>>heading. >>>>> >>>>> It sounds like the SamzaAppMaster is not getting the resource >>>>> manager host/port from the yarn-site.xml. Usually this is due to >>>>> not exporting YARN_HOME on the NodeManager before starting it. >>>>> >>>>> Cheers, >>>>> Chris >>>>> >>>>> On 3/12/14 5:21 PM, "[email protected]" >>>>> <[email protected]> wrote: >>>>> >>>>>> Okay so I was able to submit the job: >>>>>> >>>>>> In the nodemanager I get this error: Specifically it's trying to >>>>>> connect to 0.0.0.0/8032 instead of the IP I have specified in the >>>>>> yarn-site.xml file >>>>>> >>>>>> 2014-03-12 17:04:47 SamzaAppMaster$ [INFO] got container id: >>>>>> container_1391637982288_0033_01_000001 >>>>>> 2014-03-12 17:04:47 SamzaAppMaster$ [INFO] got app attempt id: >>>>>> appattempt_1391637982288_0033_000001 >>>>>> 2014-03-12 17:04:47 SamzaAppMaster$ [INFO] got node manager host: >>>>>> svdpdac001.techlabs.accenture.com >>>>>> 2014-03-12 17:04:47 SamzaAppMaster$ [INFO] got node manager port: >>>>>> 38218 >>>>>> 2014-03-12 17:04:47 SamzaAppMaster$ [INFO] got node manager http >>>>>>port: >>>>>> 8042 >>>>>> 2014-03-12 17:04:47 SamzaAppMaster$ [INFO] got config: >>>>>> >>>>>>{task.inputs=wikipedia.#en.wikipedia,wikipedia.#en.wiktionary,wiki >>>>>>p edi a .#e n.wikinews, systems.wikipedia.host=irc.wikimedia.org, >>>>>> systems.kafka.producer.batch.num.messages=1, >>>>>> job.factory.class=org.apache.samza.job.yarn.YarnJobFactory, >>>>>> systems.wikipedia.port=6667, >>>>>> systems.kafka.producer.producer.type=sync, >>>>>> job.name=wikipedia-feed, >>>>>> >>>>>>systems.kafka.consumer.zookeeper.connect=svdpdac013.techlabs.accen >>>>>>t >>>>>>ure >>>>>>. >>>>>> com :2181/, systems.kafka.samza.msg.serde=json, >>>>>> >>>>>>serializers.registry.json.class=org.apache.samza.serializers.JsonS >>>>>>e >>>>>>rde >>>>>> F >>>>>> act ory, >>>>>> task.class=samza.examples.wikipedia.task.WikipediaFeedStreamTask, >>>>>> >>>>>>yarn.package.path=hdfs://10.1.174.85:9000/samza-job-package-0.7.0- >>>>>>d >>>>>>ist >>>>>>. >>>>>> tar >>>>>> .gz, >>>>>> >>>>>>systems.wikipedia.samza.factory=samza.examples.wikipedia.system.Wi >>>>>>k >>>>>>ipe >>>>>> d >>>>>> iaS >>>>>> ystemFactory, >>>>>> >>>>>>systems.kafka.samza.factory=org.apache.samza.system.kafka.KafkaSys >>>>>>t >>>>>>emF >>>>>> a >>>>>> cto >>>>>> ry, >>>>>> >>>>>>systems.kafka.producer.metadata.broker.list=svdpdac001.techlabs.ac >>>>>>c >>>>>>ent >>>>>> ure >>>>>> . >>>>>> com:6667,svdpdac015.techlabs.accenture.com:6667} >>>>>> 2014-03-12 17:04:48 ClientHelper [INFO] trying to connect to RM >>>>>> 0.0.0.0:8032 >>>>>> 2014-03-12 17:04:48 NativeCodeLoader [WARN] Unable to load >>>>>>native-hadoop library for your platform... using builtin-java >>>>>>classes where applicable >>>>>> 2014-03-12 17:04:48 RMProxy [INFO] Connecting to ResourceManager >>>>>>at >>>>>> /0.0.0.0:8032 >>>>>> >>>>>> -----Original Message----- >>>>>> From: Chris Riccomini [mailto:[email protected]] >>>>>> Sent: Wednesday, March 12, 2014 4:48 PM >>>>>> To: [email protected] >>>>>> Subject: Re: Failed to package using mvn >>>>>> >>>>>> Hey Sonali, >>>>>> >>>>>> You need to specify a valid HDFS uri. Usually something like: >>>>>> >>>>>> hdfs://<hdfs name node ip>:<hdfs name node port>/path/to/tgz >>>>>> >>>>>> Right now, Hadoop is trying to use the package name as the HDFS >>>>>>host. >>>>>> >>>>>> Cheers, >>>>>> Chris >>>>>> >>>>>> On 3/12/14 4:45 PM, "[email protected]" >>>>>> <[email protected]> wrote: >>>>>> >>>>>>> I did and I can now see the hadoop-hdfs jar in /deploy/samza/lib >>>>>>> folder. >>>>>>> >>>>>>> I do get a different error now. >>>>>>> >>>>>>> I uploaded the samza-job to hdfs and it resides on >>>>>>> hdfs://samza-job-package-0.7.0-dist.tar.gz >>>>>>> >>>>>>> But when I run the job I get this exception: >>>>>>> >>>>>>> Exception in thread "main" java.lang.IllegalArgumentException: >>>>>>> java.net.UnknownHostException: samza-job-package-0.7.0-dist.tar.gz >>>>>>> at >>>>>>> >>>>>>>org.apache.hadoop.security.SecurityUtil.buildTokenService(Securit >>>>>>>y >>>>>>>Uti >>>>>>>l. >>>>>>> jav >>>>>>> a:418) >>>>>>> at >>>>>>> >>>>>>>org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(NameNodeP >>>>>>>r >>>>>>>oxi >>>>>>> e >>>>>>> s >>>>>>> .ja >>>>>>> va:231) >>>>>>> at >>>>>>> >>>>>>>org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxie >>>>>>>s >>>>>>>.ja >>>>>>> v >>>>>>> a >>>>>>> :13 >>>>>>> 9) >>>>>>> at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:510) >>>>>>> at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:453) >>>>>>> at >>>>>>> >>>>>>>org.apache.hadoop.hdfs.DistributedFileSystem.initialize(Distribut >>>>>>>e >>>>>>>dFi >>>>>>> l >>>>>>> e >>>>>>> Sys >>>>>>> tem.java:136) >>>>>>> at >>>>>>> >>>>>>>org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java: >>>>>>>2 >>>>>>>433 >>>>>>>) >>>>>>> at >>>>>>> org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:88) >>>>>>> at >>>>>>> >>>>>>>org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java: >>>>>>>246 >>>>>>>7) >>>>>>> at >>>>>>> org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2449) >>>>>>> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:367) >>>>>>> at org.apache.hadoop.fs.Path.getFileSystem(Path.java:287) >>>>>>> at >>>>>>> >>>>>>>org.apache.samza.job.yarn.ClientHelper.submitApplication(ClientHe >>>>>>>l >>>>>>>per >>>>>>>. >>>>>>> s >>>>>>> cal >>>>>>> a:111) >>>>>>> at org.apache.samza.job.yarn.YarnJob.submit(YarnJob.scala:55) >>>>>>> at org.apache.samza.job.yarn.YarnJob.submit(YarnJob.scala:48) >>>>>>> at org.apache.samza.job.JobRunner.run(JobRunner.scala:100) >>>>>>> at org.apache.samza.job.JobRunner$.main(JobRunner.scala:75) >>>>>>> at org.apache.samza.job.JobRunner.main(JobRunner.scala) >>>>>>> Caused by: java.net.UnknownHostException: >>>>>>> samza-job-package-0.7.0-dist.tar.gz >>>>>>> ... 18 more >>>>>>> >>>>>>> >>>>>>> -----Original Message----- >>>>>>> From: Yan Fang [mailto:[email protected]] >>>>>>> Sent: Wednesday, March 12, 2014 4:20 PM >>>>>>> To: [email protected] >>>>>>> Subject: Re: Failed to package using mvn >>>>>>> >>>>>>> Hi Sonali, >>>>>>> >>>>>>> One tip you may miss: >>>>>>> >>>>>>> If you had already run >>>>>>> >>>>>>> tar -xvf >>>>>>> ./samza-job-package/target/samza-job-package-0.7.0-dist.tar.gz >>>>>>> -C deploy/samza >>>>>>> >>>>>>> before you bundled the jar file to tar.gz. Please also remember >>>>>>> to put the hdfs jar file to the deploy/samza/lib. >>>>>>> >>>>>>> Let me know if you miss this step. >>>>>>> >>>>>>> Thanks, >>>>>>> >>>>>>> Fang, Yan >>>>>>> [email protected] >>>>>>> +1 (206) 849-4108 >>>>>>> >>>>>>> >>>>>>> On Wed, Mar 12, 2014 at 4:10 PM, Chris Riccomini >>>>>>> <[email protected]>wrote: >>>>>>> >>>>>>>> Hey Sonali, >>>>>>>> >>>>>>>> Yan has made a step-by-step tutorial for this. Could you >>>>>>>> confirm that you've followed the instructions, and it's still not >>>>>>>> working? >>>>>>>> >>>>>>>> https://issues.apache.org/jira/browse/SAMZA-181 >>>>>>>> >>>>>>>> Cheers, >>>>>>>> Chris >>>>>>>> >>>>>>>> On 3/12/14 3:12 PM, "[email protected]" >>>>>>>> <[email protected]> wrote: >>>>>>>> >>>>>>>>> So sigh! I had some Kafka issues in-between. That's fixed now. >>>>>>>>> >>>>>>>>> As suggested, >>>>>>>>> >>>>>>>>> 1. I made sure the hadoop-hdfs-2.2.0.jar is bundled with the >>>>>>>>> samza job tar.gz. >>>>>>>>> 2. I added the configuration to implement hdfs in the >>>>>>>>> hdfs-site.xml files both on the NMs and in the /conf directory >>>>>>>>> for samza >>>>>>>>> >>>>>>>>> I still get the No Filesystem for scheme :hdfs error. >>>>>>>>> >>>>>>>>> Is there anything else im missing? >>>>>>>>> Thanks, >>>>>>>>> Sonali >>>>>>>>> >>>>>>>>> >>>>>>>>> -----Original Message----- >>>>>>>>> From: Chris Riccomini [mailto:[email protected]] >>>>>>>>> Sent: Tuesday, March 11, 2014 8:27 PM >>>>>>>>> To: [email protected] >>>>>>>>> Subject: Re: Failed to package using mvn >>>>>>>>> >>>>>>>>> Hey Yan, >>>>>>>>> >>>>>>>>> This looks great! I added a few requests to the JIRA, if you >>>>>>>>> have >>>>>>>> time. >>>>>>>>> >>>>>>>>> Cheers, >>>>>>>>> Chris >>>>>>>>> >>>>>>>>>> On 3/11/14 7:20 PM, "Yan Fang" <[email protected]> wrote: >>>>>>>>>> >>>>>>>>>> Hi Chris, >>>>>>>>>> >>>>>>>>>> Has opened an issue >>>>>>>>>> SAMZA-181<https://issues.apache.org/jira/browse/SAMZA-181>and >>>>>>>>>> also uploaded the patch. Let me know if there is something >>>>>>>>>> wrong in my tutorial. Thank you! >>>>>>>>>> >>>>>>>>>> Cheers, >>>>>>>>>> >>>>>>>>>> Fang, Yan >>>>>>>>>> [email protected] >>>>>>>>>> +1 (206) 849-4108 >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Tue, Mar 11, 2014 at 10:40 AM, >>>>>>>>>> <[email protected]>wrote: >>>>>>>>>> >>>>>>>>>>> Thanks Chris, Yan, >>>>>>>>>>> >>>>>>>>>>> Let me try that. >>>>>>>>>>> >>>>>>>>>>> -----Original Message----- >>>>>>>>>>> From: Chris Riccomini [mailto:[email protected]] >>>>>>>>>>> Sent: Tuesday, March 11, 2014 10:22 AM >>>>>>>>>>> To: [email protected] >>>>>>>>>>> Subject: Re: Failed to package using mvn >>>>>>>>>>> >>>>>>>>>>> Hey Yan, >>>>>>>>>>> >>>>>>>>>>> Awesome!The location where you can add your .md is here: >>>>>>>>>>> >>>>>>>>>>> docs/learn/tutorials/0.7.0/ >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Here's a link to the code tree: >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> https://git-wip-us.apache.org/repos/asf?p=incubator-samza.gi >>>>>>>>>>> t >>>>>>>>>>> ;a= >>>>>>>>>>> t >>>>>>>>>>> r >>>>>>>>>>> e >>>>>>>>>>> e;f >>>>>>>>>>> =do >>>>>>>>>>> cs >>>>>>>>>>> >>>>>>>>>>> /learn/tutorials/0.7.0;h=ef117f4066f14a00f50f0f6fca179031304 >>>>>>>>>>> 4 >>>>>>>>>>> 831 >>>>>>>>>>> 2 >>>>>>>>>>> ; >>>>>>>>>>> h >>>>>>>>>>> b=H >>>>>>>>>>> EAD >>>>>>>>>>> >>>>>>>>>>> You can get the code here: >>>>>>>>>>> >>>>>>>>>>> git clone >>>>>>>>>>> http://git-wip-us.apache.org/repos/asf/incubator-samza.git >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Once you write the .md, just throw it up on a JIRA, and one >>>>>>>>>>> of us can merge it in. >>>>>>>>>>> >>>>>>>>>>> Re: hdfs-site.xml, ah ha, that's what I figured. This is >>>>>>>>>>> good to >>>>>>>> know. >>>>>>>>>>> So >>>>>>>>>>> you just copy your hdfs-site.xml from your NodeManager's >>>>>>>>>>> conf directory into your local hdfs-site.xml. >>>>>>>>>>> >>>>>>>>>>> Cheers, >>>>>>>>>>> Chris >>>>>>>>>>> >>>>>>>>>>>> On 3/11/14 10:16 AM, "Yan Fang" <[email protected]> wrote: >>>>>>>>>>>> >>>>>>>>>>>> Hi Chris, >>>>>>>>>>>> >>>>>>>>>>>> Sure. I just do not know how/where to contribute this >>>>>>>>>>>> page...*_* >>>>>>>>>>>> >>>>>>>>>>>> Oh, I mean the same this as you mentioned in the *Cluster >>>>>>>>>>>> Installation*thread: >>>>>>>>>>>> >>>>>>>>>>>> *"2. Get a copy of one of your NM's yarn-site.xml and put >>>>>>>>>>>> it somewhere >>>>>>>>>>>> on* >>>>>>>>>>>> >>>>>>>>>>>> *your desktop (I usually use ~/.yarn/conf/yarn-site.xml). >>>>>>>>>>>> Note that there'sa "conf" directory there. This is >>>>>>>>>>>> mandatory."* >>>>>>>>>>>> >>>>>>>>>>>> So I just copy the hdfs-site.xml to >>>>>>>>>>>>~/.yarn/conf/hdfs-site.xml. >>>>>>>>>>>> Thank >>>>>>>>>>> you. >>>>>>>>>>>> >>>>>>>>>>>> Cheers, >>>>>>>>>>>> >>>>>>>>>>>> Fang, Yan >>>>>>>>>>>> [email protected] >>>>>>>>>>>> +1 (206) 849-4108 >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Tue, Mar 11, 2014 at 10:10 AM, Chris Riccomini >>>>>>>>>>>> <[email protected]>wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Hey Yan, >>>>>>>>>>>>> >>>>>>>>>>>>> Would you be up for contributing a tutorial page that >>>>>>>>>>>>> describes >>>>>>>>>>> this? >>>>>>>>>>>>> This >>>>>>>>>>>>> is really useful information. Our docs are just simple .md >>>>>>>>>>>>> files in the main code base. >>>>>>>>>>>>> >>>>>>>>>>>>> Regarding step (3), is the hdfs-site.xml put into the conf >>>>>>>>>>>>> folder for the NM boxes, or on the client side (where >>>>>>>>>>>>> run-job.sh >>>>>>>> is run)? >>>>>>>>>>>>> >>>>>>>>>>>>> Cheers, >>>>>>>>>>>>> Chris >>>>>>>>>>>>> >>>>>>>>>>>>>> On 3/11/14 10:07 AM, "Yan Fang" <[email protected]> >>>>>>>>>>>>>>wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>> Hi Sonali, >>>>>>>>>>>>>> >>>>>>>>>>>>>> The way I make Samza run with HDFS is following: >>>>>>>>>>>>>> >>>>>>>>>>>>>> 1. include hdfs jar in Samza jar tar.gz. >>>>>>>>>>>>>> 2. you may also want to make sure the hadoop-common.jar >>>>>>>>>>>>>> has the same version as your hdfs jar. Otherwise, you may >>>>>>>>>>>>>> have configuration error popping out. >>>>>>>>>>>>>> 3. then put hdfs-site.xml to conf folder, the same folder >>>>>>>>>>>>>> as the yarn-site.xml 4. all other steps are not changed. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Hope this will help. Thank you. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Cheers, >>>>>>>>>>>>>> >>>>>>>>>>>>>> Fang, Yan >>>>>>>>>>>>>> [email protected] >>>>>>>>>>>>>> +1 (206) 849-4108 >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Tue, Mar 11, 2014 at 9:25 AM, Chris Riccomini >>>>>>>>>>>>>> <[email protected]>wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hey Sonali, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I believe that you need to make sure that the HDFS jar >>>>>>>>>>>>>>> is in your .tar.gz file, as you've said. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> If that doesn't work, you might need to define this >>>>>>>>>>>>>>> setting in core-site.xml on the machine you're running >>>>>>>> run-job.sh on: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> <property> >>>>>>>>>>>>>>> <name>fs.hdfs.impl</name> >>>>>>>> <value>org.apache.hadoop.hdfs.DistributedFileSystem</value> >>>>>>>>>>>>>>> <description>The FileSystem for hdfs: >>>>>>>>>>>>>>> uris.</description> </property> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> You might also need to configure your NodeManagers to >>>>>>>>>>>>>>> have the HDFS >>>>>>>>>>>>> file >>>>>>>>>>>>>>> system impl as well. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I've never run Samza with HDFS, so I'm guessing here. >>>>>>>>>>>>>>> Perhaps someone else on the list has been successful >>>>>>>>>>>>>>> with >>>>>>>> this? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Cheers, >>>>>>>>>>>>>>> Chris >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On 3/10/14 3:59 PM, "[email protected]" >>>>>>>>>>>>>>> <[email protected]> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Hello, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> I fixed this by starting from scratch with gradlew. But >>>>>>>>>>>>>>>> now when I >>>>>>>>>>>>> run >>>>>>>>>>>>>>> my >>>>>>>>>>>>>>>> job it throws this error: >>>>>>>>>>>>>>>> Exception in thread "main" java.io.IOException: No >>>>>>>>>>>>>>>> FileSystem for >>>>>>>>>>>>>>> scheme: >>>>>>>>>>>>>>>> hdfs >>>>>>>>>>>>>>>> at >>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileS >>>>>>>>>>>>>>>>y >>>>>>>>>>>>>>>>ste >>>>>>>>>>>>>>>>m. >>>>>>>>>>>>>>>> jav >>>>>>>>>>>>>>>> a: >>>>>>>>>>>>>>>> 242 >>>>>>>>>>>>>>>> 1) >>>>>>>>>>>>>>>> at >>>>>>>>>>>>>> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem. >>>>>>>>>>>>>> j >>>>>>>>>>>>>> a >>>>>>>>>>>>>> v >>>>>>>>>>>>>> a:2 >>>>>>>>>>>>>> 428 >>>>>>>>>>>>>> ) >>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>> org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java: >>>>>>>>>>>>>>> 8 >>>>>>>>>>>>>>> 8 >>>>>>>>>>>>>>> ) >>>>>>>>>>>>>>>> at >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem. >>>>>>>>>>>j >>>>>>>>>>>ava >>>>>>>>>>>: >>>>>>>>>>>>>> 246 >>>>>>>>>>>>>> 7) >>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>> org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java: >>>>>>>>>>>>>>> 2 >>>>>>>>>>>>>>> 4 >>>>>>>>>>>>>>> 4 >>>>>>>>>>>>>>> 9) >>>>>>>>>>>>>>>> at >>>>>>>>>>> org.apache.hadoop.fs.FileSystem.get(FileSystem.java:367) >>>>>>>>>>>>>>>> at >>>>>>>>>>> org.apache.hadoop.fs.Path.getFileSystem(Path.java:287) >>>>>>>>>>>>>>>> at >>>>>>>>>>> >>>>>>>>>>>>>>>> org.apache.samza.job.yarn.ClientHelper.submitApplicatio >>>>>>>>>>>>>>>> n >>>>>>>>>>>>>>>> (Cl >>>>>>>>>>>>>>>> i >>>>>>>>>>>>>>>> e >>>>>>>>>>>>>>>> n >>>>>>>>>>>>>>>> tHe >>>>>>>>>>>>>>>> lpe >>>>>>>>>>>>>>>> r. >>>>>>>>>>>>>>>> sc >>>>>>>>>>>>>>>> al >>>>>>>>>>>>>>>> a:111) >>>>>>>>>>>>>>>> at >>>>>>>>>>>>> org.apache.samza.job.yarn.YarnJob.submit(YarnJob.scala:55) >>>>>>>>>>>>>>>> at >>>>>>>>>>>>> org.apache.samza.job.yarn.YarnJob.submit(YarnJob.scala:48) >>>>>>>>>>>>>>>> at >>>>>>>>>>> org.apache.samza.job.JobRunner.run(JobRunner.scala:100) >>>>>>>>>>>>>>>> at >>>>>>>>>>> org.apache.samza.job.JobRunner$.main(JobRunner.scala:75) >>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>> org.apache.samza.job.JobRunner.main(JobRunner.scala) >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> I looked at the samza job tar.gz and it doesn't have a >>>>>>>>>>>>>>>> Hadoop-hdfs >>>>>>>>>>>>> jar. >>>>>>>>>>>>>>>> Is that why I get this error? >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>> Sonali >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> From: Parthasarathy, Sonali >>>>>>>>>>>>>>>> Sent: Monday, March 10, 2014 11:25 AM >>>>>>>>>>>>>>>> To: [email protected] >>>>>>>>>>>>>>>> Subject: Failed to package using mvn >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> When I tried to do a mvn clean package of my >>>>>>>>>>>>>>>> hello-samza project, I >>>>>>>>>>>>> get >>>>>>>>>>>>>>>> the following error. Has anyone seen this before? >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> [ERROR] Failed to execute goal on project samza-wikipedia: >>>>>>>>>>>>>>>> Could not resolve dependencies for project >>>>>>>>>>> samza:samza-wikipedia:jar:0.7.0: >>>>>>>>>>>>> Could >>>>>>>>>>>>>>>> not find artifact >>>>>>>>>>>>>>>> org.apache.samza:samza-kv_2.10:jar:0.7.0 >>>>>>>>>>>>>>>> in apache-releases >>>>>>>>>>>>> (https://repository.apache.org/content/groups/public) >>>>>>>>>>>>>>> -> >>>>>>>>>>>>>>>> [Help 1] >>>>>>>>>>>>>>>> [ERROR] >>>>>>>>>>>>>>>> [ERROR] To see the full stack trace of the errors, >>>>>>>>>>>>>>>> re-run Maven with >>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>> -e switch. >>>>>>>>>>>>>>>> [ERROR] Re-run Maven using the -X switch to enable full >>>>>>>>>>>>>>>> debug >>>>>>>>>>>>> logging. >>>>>>>>>>>>>>>> [ERROR] >>>>>>>>>>>>>>>> [ERROR] For more information about the errors and >>>>>>>>>>>>>>>> possible >>>>>>>>>>>>> solutions, >>>>>>>>>>>>>>>> please read the following articles: >>>>>>>>>>>>>>>> [ERROR] [Help 1] >>>>>>>>>>>>> >>>>>>>>>>>>> http://cwiki.apache.org/confluence/display/MAVEN/Dependenc >>>>>>>>>>>>> y >>>>>>>>>>>>> Re >>>>>>>>>>>>> s >>>>>>>>>>>>> o >>>>>>>>>>>>> l >>>>>>>>>>>>> uti >>>>>>>>>>>>> onE >>>>>>>>>>>>> xce >>>>>>>>>>>>>>> p >>>>>>>>>>>>>>>> tion >>>>>>>>>>>>>>>> [ERROR] >>>>>>>>>>>>>>>> [ERROR] After correcting the problems, you can resume >>>>>>>>>>>>>>>> the build with >>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>> command >>>>>>>>>>>>>>>> [ERROR] mvn <goals> -rf :samza-wikipedia >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>> Sonali >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Sonali Parthasarathy >>>>>>>>>>>>>>>> R&D Developer, Data Insights Accenture Technology Labs >>>>>>>>>>>>>>>> 703-341-7432 >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> ________________________________ >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> This message is for the designated recipient only and >>>>>>>>>>>>>>>> may contain privileged, proprietary, or otherwise >>>>>>>>>>>>>>>> confidential >>>>>>>>>>> information. >>>>>>>>>>>>>>>> If >>>>>>>>>>>>> you >>>>>>>>>>>>>>>> have received it in error, please notify the sender >>>>>>>>>>>>>>>> immediately and delete the original. Any other use of >>>>>>>>>>>>>>>> the e-mail by you is >>>>>>>>>>>>> prohibited. >>>>>>>>>>>>>>>> Where allowed by local law, electronic communications >>>>>>>>>>>>>>>> with Accenture >>>>>>>>>>>>>>> and >>>>>>>>>>>>>>>> its affiliates, including e-mail and instant messaging >>>>>>>>>>>>>>>> (including content), may be scanned by our systems for >>>>>>>>>>>>>>>> the purposes of >>>>>>>>>>>>> information >>>>>>>>>>>>>>>> security and assessment of internal compliance with >>>>>>>>>>>>>>>> Accenture >>>>>>>>>>>>> policy. >>>>>>>>>>>>> >>>>>>>>>>>>>>>> _______________________________________________________ >>>>>>>>>>>>>>>> _ >>>>>>>>>>>>>>>> __ >>>>>>>>>>>>>>>> _ >>>>>>>>>>>>>>>> _ >>>>>>>>>>>>>>>> _ >>>>>>>>>>>>>>>> ___ >>>>>>>>>>>>>>>> ___ >>>>>>>>>>>>>>>> ___ >>>>>>>>>>>>>>>> __ >>>>>>>>>>>>>>>> __ >>>>>>>>>>>>>>>> ____________ >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> www.accenture.com >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> ________________________________ >>>>>>>>>>> >>>>>>>>>>> This message is for the designated recipient only and may >>>>>>>>>>> contain privileged, proprietary, or otherwise confidential >>>>>>>>>>> information. If you have received it in error, please >>>>>>>>>>> notify the sender immediately and delete the original. Any >>>>>>>>>>> other use of the e-mail by you is prohibited. Where allowed >>>>>>>>>>> by local law, electronic communications with Accenture and >>>>>>>>>>> its affiliates, including e-mail and instant messaging >>>>>>>>>>> (including content), may be scanned by our systems for the >>>>>>>>>>> purposes of information security and assessment of internal >>>>>>>>>>> compliance with Accenture >>>>>>>> policy. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> ____________________________________________________________ >>>>>>>>>>> _ >>>>>>>>>>> ___ >>>>>>>>>>> _ >>>>>>>>>>> _ >>>>>>>>>>> _ >>>>>>>>>>> ___ >>>>>>>>>>> ___ >>>>>>>>>>> _____________ >>>>>>>>>>> >>>>>>>>>>> www.accenture.com >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> ________________________________ >>>>>>>>> >>>>>>>>> This message is for the designated recipient only and may >>>>>>>>> contain privileged, proprietary, or otherwise confidential >>>>>>>>> information. If you have received it in error, please notify >>>>>>>>> the sender immediately and delete the original. Any other use >>>>>>>>> of the e-mail by you is >>>>>>>> prohibited. >>>>>>>>> Where allowed by local law, electronic communications with >>>>>>>>> Accenture and its affiliates, including e-mail and instant >>>>>>>>> messaging (including content), may be scanned by our systems >>>>>>>>> for the purposes of information security and assessment of >>>>>>>>> internal compliance with >>>>>>>> Accenture policy. >>>>>>>>> ______________________________________________________________ >>>>>>>>> _ >>>>>>>>> ___ >>>>>>>>> _ >>>>>>>>> _ >>>>>>>>> _ >>>>>>>>> _____ >>>>>>>>> ____________ >>>>>>>>> >>>>>>>>> www.accenture.com >>>>>>> >>>>>>> ________________________________ >>>>>>> >>>>>>> This message is for the designated recipient only and may >>>>>>>contain privileged, proprietary, or otherwise confidential >>>>>>>information. If you have received it in error, please notify the >>>>>>>sender immediately and delete the original. Any other use of the >>>>>>>e-mail by you is prohibited. >>>>>>> Where allowed by local law, electronic communications with >>>>>>>Accenture and its affiliates, including e-mail and instant >>>>>>>messaging (including content), may be scanned by our systems for >>>>>>>the purposes of information security and assessment of internal >>>>>>>compliance with Accenture policy. >>>>>>> >>>>>>>_________________________________________________________________ >>>>>>>_ >>>>>>>___ >>>>>>> _ >>>>>>> _ >>>>>>> ___ >>>>>>> ____________ >>>>>>> >>>>>>> www.accenture.com >>>>>> >>>>>> >>>>>> >>>>>> ________________________________ >>>>>> >>>>>> This message is for the designated recipient only and may contain >>>>>>privileged, proprietary, or otherwise confidential information. If >>>>>>you have received it in error, please notify the sender >>>>>>immediately and delete the original. Any other use of the e-mail >>>>>>by you is prohibited. >>>>>> Where allowed by local law, electronic communications with >>>>>>Accenture and its affiliates, including e-mail and instant >>>>>>messaging (including content), may be scanned by our systems for >>>>>>the purposes of information security and assessment of internal >>>>>>compliance with Accenture policy. >>>>>> >>>>>>__________________________________________________________________ >>>>>>_ >>>>>>___ >>>>>> _ >>>>>> ___ >>>>>> ____________ >>>>>> >>>>>> www.accenture.com >>>>> >>>>> >>>>> >>>>> ________________________________ >>>>> >>>>> This message is for the designated recipient only and may contain >>>>>privileged, proprietary, or otherwise confidential information. If >>>>>you have received it in error, please notify the sender >>>>>immediately and delete the original. Any other use of the e-mail >>>>>by you is prohibited. >>>>> Where allowed by local law, electronic communications with >>>>>Accenture and its affiliates, including e-mail and instant >>>>>messaging (including content), may be scanned by our systems for >>>>>the purposes of information security and assessment of internal >>>>>compliance with Accenture policy. >>>>> >>>>>___________________________________________________________________ >>>>>_ >>>>>___ >>>>> ___ >>>>> ____________ >>>>> >>>>> www.accenture.com >>>> >>>> >>>> >>>> ________________________________ >>>> >>>> This message is for the designated recipient only and may contain >>>>privileged, proprietary, or otherwise confidential information. If >>>>you have received it in error, please notify the sender immediately >>>>and delete the original. Any other use of the e-mail by you is >>>>prohibited. >>>> Where allowed by local law, electronic communications with >>>>Accenture and its affiliates, including e-mail and instant >>>>messaging (including content), may be scanned by our systems for >>>>the purposes of information security and assessment of internal >>>>compliance with Accenture policy. >>>> >>>>____________________________________________________________________ >>>>_ >>>>___ >>>>__ >>>> ____________ >>>> >>>> www.accenture.com >>> > > > >________________________________ > >This message is for the designated recipient only and may contain >privileged, proprietary, or otherwise confidential information. If you >have received it in error, please notify the sender immediately and >delete the original. Any other use of the e-mail by you is prohibited. >Where allowed by local law, electronic communications with Accenture >and its affiliates, including e-mail and instant messaging (including >content), may be scanned by our systems for the purposes of information >security and assessment of internal compliance with Accenture policy. >_______________________________________________________________________ >___ >____________ > >www.accenture.com > ________________________________ This message is for the designated recipient only and may contain privileged, proprietary, or otherwise confidential information. If you have received it in error, please notify the sender immediately and delete the original. Any other use of the e-mail by you is prohibited. Where allowed by local law, electronic communications with Accenture and its affiliates, including e-mail and instant messaging (including content), may be scanned by our systems for the purposes of information security and assessment of internal compliance with Accenture policy. ______________________________________________________________________________________ www.accenture.com
