Hey Sonali, Yan has made a step-by-step tutorial for this. Could you confirm that you've followed the instructions, and it's still not working?
https://issues.apache.org/jira/browse/SAMZA-181 Cheers, Chris On 3/12/14 3:12 PM, "[email protected]" <[email protected]> wrote: >So sigh! I had some Kafka issues in-between. That's fixed now. > >As suggested, > >1. I made sure the hadoop-hdfs-2.2.0.jar is bundled with the samza job >tar.gz. >2. I added the configuration to implement hdfs in the hdfs-site.xml files >both on the NMs and in the /conf directory for samza > >I still get the No Filesystem for scheme :hdfs error. > >Is there anything else im missing? >Thanks, >Sonali > > >-----Original Message----- >From: Chris Riccomini [mailto:[email protected]] >Sent: Tuesday, March 11, 2014 8:27 PM >To: [email protected] >Subject: Re: Failed to package using mvn > >Hey Yan, > >This looks great! I added a few requests to the JIRA, if you have time. > >Cheers, >Chris > >On 3/11/14 7:20 PM, "Yan Fang" <[email protected]> wrote: > >>Hi Chris, >> >>Has opened an issue >>SAMZA-181<https://issues.apache.org/jira/browse/SAMZA-181>and also >>uploaded the patch. Let me know if there is something wrong in my >>tutorial. Thank you! >> >>Cheers, >> >>Fang, Yan >>[email protected] >>+1 (206) 849-4108 >> >> >>On Tue, Mar 11, 2014 at 10:40 AM, >><[email protected]>wrote: >> >>> Thanks Chris, Yan, >>> >>> Let me try that. >>> >>> -----Original Message----- >>> From: Chris Riccomini [mailto:[email protected]] >>> Sent: Tuesday, March 11, 2014 10:22 AM >>> To: [email protected] >>> Subject: Re: Failed to package using mvn >>> >>> Hey Yan, >>> >>> Awesome!The location where you can add your .md is here: >>> >>> docs/learn/tutorials/0.7.0/ >>> >>> >>> Here's a link to the code tree: >>> >>> >>> >>>https://git-wip-us.apache.org/repos/asf?p=incubator-samza.git;a=tree;f >>>=do >>>cs >>> >>>/learn/tutorials/0.7.0;h=ef117f4066f14a00f50f0f6fca17903130448312;hb=H >>>EAD >>> >>> You can get the code here: >>> >>> git clone >>> http://git-wip-us.apache.org/repos/asf/incubator-samza.git >>> >>> >>> Once you write the .md, just throw it up on a JIRA, and one of us can >>> merge it in. >>> >>> Re: hdfs-site.xml, ah ha, that's what I figured. This is good to know. >>>So >>> you just copy your hdfs-site.xml from your NodeManager's conf >>>directory into your local hdfs-site.xml. >>> >>> Cheers, >>> Chris >>> >>> On 3/11/14 10:16 AM, "Yan Fang" <[email protected]> wrote: >>> >>> >Hi Chris, >>> > >>> >Sure. I just do not know how/where to contribute this page...*_* >>> > >>> >Oh, I mean the same this as you mentioned in the *Cluster >>> >Installation*thread: >>> > >>> >*"2. Get a copy of one of your NM's yarn-site.xml and put it >>> >somewhere >>> >on* >>> > >>> >*your desktop (I usually use ~/.yarn/conf/yarn-site.xml). Note that >>> >there'sa "conf" directory there. This is mandatory."* >>> > >>> >So I just copy the hdfs-site.xml to ~/.yarn/conf/hdfs-site.xml. >>> >Thank >>>you. >>> > >>> >Cheers, >>> > >>> >Fang, Yan >>> >[email protected] >>> >+1 (206) 849-4108 >>> > >>> > >>> >On Tue, Mar 11, 2014 at 10:10 AM, Chris Riccomini >>> ><[email protected]>wrote: >>> > >>> >> Hey Yan, >>> >> >>> >> Would you be up for contributing a tutorial page that describes >>>this? >>> >>This >>> >> is really useful information. Our docs are just simple .md files >>> >>in the main code base. >>> >> >>> >> Regarding step (3), is the hdfs-site.xml put into the conf folder >>> >>for the NM boxes, or on the client side (where run-job.sh is run)? >>> >> >>> >> Cheers, >>> >> Chris >>> >> >>> >> On 3/11/14 10:07 AM, "Yan Fang" <[email protected]> wrote: >>> >> >>> >> >Hi Sonali, >>> >> > >>> >> >The way I make Samza run with HDFS is following: >>> >> > >>> >> >1. include hdfs jar in Samza jar tar.gz. >>> >> >2. you may also want to make sure the hadoop-common.jar has the >>> >> >same version as your hdfs jar. Otherwise, you may have >>> >> >configuration error popping out. >>> >> >3. then put hdfs-site.xml to conf folder, the same folder as the >>> >> >yarn-site.xml 4. all other steps are not changed. >>> >> > >>> >> >Hope this will help. Thank you. >>> >> > >>> >> >Cheers, >>> >> > >>> >> >Fang, Yan >>> >> >[email protected] >>> >> >+1 (206) 849-4108 >>> >> > >>> >> > >>> >> >On Tue, Mar 11, 2014 at 9:25 AM, Chris Riccomini >>> >> ><[email protected]>wrote: >>> >> > >>> >> >> Hey Sonali, >>> >> >> >>> >> >> I believe that you need to make sure that the HDFS jar is in >>> >> >>your .tar.gz file, as you've said. >>> >> >> >>> >> >> If that doesn't work, you might need to define this setting in >>> >> >> core-site.xml on the machine you're running run-job.sh on: >>> >> >> >>> >> >> <property> >>> >> >> <name>fs.hdfs.impl</name> >>> >> >> <value>org.apache.hadoop.hdfs.DistributedFileSystem</value> >>> >> >> <description>The FileSystem for hdfs: uris.</description> >>> >> >> </property> >>> >> >> >>> >> >> >>> >> >> You might also need to configure your NodeManagers to have the >>> >> >> HDFS >>> >>file >>> >> >> system impl as well. >>> >> >> >>> >> >> I've never run Samza with HDFS, so I'm guessing here. Perhaps >>> >> >>someone else on the list has been successful with this? >>> >> >> >>> >> >> Cheers, >>> >> >> Chris >>> >> >> >>> >> >> On 3/10/14 3:59 PM, "[email protected]" >>> >> >> <[email protected]> wrote: >>> >> >> >>> >> >> >Hello, >>> >> >> > >>> >> >> >I fixed this by starting from scratch with gradlew. But now >>> >> >> >when I >>> >>run >>> >> >>my >>> >> >> >job it throws this error: >>> >> >> >Exception in thread "main" java.io.IOException: No FileSystem >>> >> >> >for >>> >> >>scheme: >>> >> >> >hdfs >>> >> >> > at >>> >> >> >>> >> >>> >>>>>>>>org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java: >>> >>>>>242 >>> >>>>>1) >>> >> >> > at >>> >> >> >>> >>>org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2 >>> >>>428 >>> >>>) >>> >> >> > at >>> >> >>org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:88) >>> >> >> > at >>> >> >> >>> >>>org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java: >>> >>>246 >>> >>>7) >>> >> >> > at >>> >> >>org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2449) >>> >> >> > at >>>org.apache.hadoop.fs.FileSystem.get(FileSystem.java:367) >>> >> >> > at >>>org.apache.hadoop.fs.Path.getFileSystem(Path.java:287) >>> >> >> > at >>> >> >> >>> >> >>> >>>>>>>>org.apache.samza.job.yarn.ClientHelper.submitApplication(ClientHe >>>>>>>>lpe >>>>>>>>r. >>> >>>>>sc >>> >> >>>al >>> >> >> >a:111) >>> >> >> > at >>> >>org.apache.samza.job.yarn.YarnJob.submit(YarnJob.scala:55) >>> >> >> > at >>> >>org.apache.samza.job.yarn.YarnJob.submit(YarnJob.scala:48) >>> >> >> > at >>>org.apache.samza.job.JobRunner.run(JobRunner.scala:100) >>> >> >> > at >>>org.apache.samza.job.JobRunner$.main(JobRunner.scala:75) >>> >> >> > at >>> >> >> > org.apache.samza.job.JobRunner.main(JobRunner.scala) >>> >> >> > >>> >> >> >I looked at the samza job tar.gz and it doesn't have a >>> >> >> >Hadoop-hdfs >>> >>jar. >>> >> >> >Is that why I get this error? >>> >> >> > >>> >> >> >Thanks, >>> >> >> >Sonali >>> >> >> > >>> >> >> >From: Parthasarathy, Sonali >>> >> >> >Sent: Monday, March 10, 2014 11:25 AM >>> >> >> >To: [email protected] >>> >> >> >Subject: Failed to package using mvn >>> >> >> > >>> >> >> >Hi, >>> >> >> > >>> >> >> >When I tried to do a mvn clean package of my hello-samza >>> >> >> >project, I >>> >>get >>> >> >> >the following error. Has anyone seen this before? >>> >> >> > >>> >> >> >[ERROR] Failed to execute goal on project samza-wikipedia: >>> >> >> >Could not resolve dependencies for project >>> samza:samza-wikipedia:jar:0.7.0: >>> >>Could >>> >> >> >not find artifact org.apache.samza:samza-kv_2.10:jar:0.7.0 in >>> >> >> >apache-releases >>> >>(https://repository.apache.org/content/groups/public) >>> >> >>-> >>> >> >> >[Help 1] >>> >> >> >[ERROR] >>> >> >> >[ERROR] To see the full stack trace of the errors, re-run >>> >> >> >Maven with >>> >> >>the >>> >> >> >-e switch. >>> >> >> >[ERROR] Re-run Maven using the -X switch to enable full debug >>> >>logging. >>> >> >> >[ERROR] >>> >> >> >[ERROR] For more information about the errors and possible >>> >>solutions, >>> >> >> >please read the following articles: >>> >> >> >[ERROR] [Help 1] >>> >> >> > >>> >> >> >>> >> >> >>> >> >>> >>http://cwiki.apache.org/confluence/display/MAVEN/DependencyResoluti >>> >>onE >>> >>xce >>> >> >>p >>> >> >> >tion >>> >> >> >[ERROR] >>> >> >> >[ERROR] After correcting the problems, you can resume the >>> >> >> >build with >>> >> >>the >>> >> >> >command >>> >> >> >[ERROR] mvn <goals> -rf :samza-wikipedia >>> >> >> > >>> >> >> >Thanks, >>> >> >> >Sonali >>> >> >> > >>> >> >> >Sonali Parthasarathy >>> >> >> >R&D Developer, Data Insights >>> >> >> >Accenture Technology Labs >>> >> >> >703-341-7432 >>> >> >> > >>> >> >> > >>> >> >> >________________________________ >>> >> >> > >>> >> >> >This message is for the designated recipient only and may >>> >> >> >contain privileged, proprietary, or otherwise confidential >>>information. >>> >> >> >If >>> >>you >>> >> >> >have received it in error, please notify the sender >>> >> >> >immediately and delete the original. Any other use of the >>> >> >> >e-mail by you is >>> >>prohibited. >>> >> >> >Where allowed by local law, electronic communications with >>> >> >> >Accenture >>> >> >>and >>> >> >> >its affiliates, including e-mail and instant messaging >>> >> >> >(including content), may be scanned by our systems for the >>> >> >> >purposes of >>> >>information >>> >> >> >security and assessment of internal compliance with Accenture >>> >>policy. >>> >> >> >>> >> >>> >>>>>________________________________________________________________ >>> >>>>>___ >>> >>>>>___ >>> >>>>>__ >>> >> >>>__ >>> >> >> >____________ >>> >> >> > >>> >> >> >www.accenture.com >>> >> >> >>> >> >> >>> >> >>> >> >>> >>> >>> >>> ________________________________ >>> >>> This message is for the designated recipient only and may contain >>>privileged, proprietary, or otherwise confidential information. If you >>>have received it in error, please notify the sender immediately and >>>delete the original. Any other use of the e-mail by you is >>>prohibited. Where allowed by local law, electronic communications >>>with Accenture and its affiliates, including e-mail and instant >>>messaging (including content), may be scanned by our systems for the >>>purposes of information security and assessment of internal >>>compliance with Accenture policy. >>> >>> >>>______________________________________________________________________ >>>___ >>>_____________ >>> >>> www.accenture.com >>> >>> > > > >________________________________ > >This message is for the designated recipient only and may contain >privileged, proprietary, or otherwise confidential information. If you >have received it in error, please notify the sender immediately and >delete the original. Any other use of the e-mail by you is prohibited. >Where allowed by local law, electronic communications with Accenture and >its affiliates, including e-mail and instant messaging (including >content), may be scanned by our systems for the purposes of information >security and assessment of internal compliance with Accenture policy. >__________________________________________________________________________ >____________ > >www.accenture.com >
