Re: Multiple data centre in Hadoop
Hi Abhishek, 1. Use multiple directories for *dfs.name.dir* *dfs.data.dir* etc * Recommendation: write to *two local directories on different physical volumes*, and to an *NFS-mounted* directory – Data will be preserved even in the event of a total failure of the NameNode machines * Recommendation: *soft-mount the NFS* directory – If the NFS mount goes offline, this will not cause the NameNode to fail 2. *Rack awareness* https://issues.apache.org/jira/secure/attachment/12345251/Rack_aware_HDFS_proposal.pdf On Thu, Apr 12, 2012 at 2:18 AM, Abhishek Pratap Singh manu.i...@gmail.comwrote: Thanks Robert. Is there a best practice or design than can address the High Availability to certain extent? ~Abhishek On Wed, Apr 11, 2012 at 12:32 PM, Robert Evans ev...@yahoo-inc.com wrote: No it does not. Sorry On 4/11/12 1:44 PM, Abhishek Pratap Singh manu.i...@gmail.com wrote: Hi All, Just wanted if hadoop supports more than one data centre. This is basically for DR purposes and High Availability where one centre goes down other can bring up. Regards, Abhishek -- Thanks Regards *Manu S* SI Engineer - OpenSource HPC Wipro Infotech Mob: +91 8861302855Skype: manuspkd www.opensourcetalk.co.in
setting client retry
In the hdfs-site.xml file what argument do I need to set for client retries? Also, what is the default parameter? -- --- Get your facts first, then you can distort them as you please.--
Re: setting client retry
There are several different types of 'client retries'. The following are some that I know of. My guess is that you meant the following one. If so, it is defined in core-site.xml ipc.client.connect.max.retries (default value: 10) - Indicates the number of retries a client will make to establish a server connection. The other type of retries that I can think of on hdfs side: dfs.client.block.write.retries (default value: 3) - As the name suggests, this is the number of times a DFS client retries write to the DataNodes. dfs.client.block.write.locateFollowingBlock.retries (default value: 5) - On certain exceptions, the client might retry when trying to get an additional block from NN and this configuration controls that. There might be more. Feel free to let me know if you meant something else. Regards, John George -Original Message- From: Rita rmorgan...@gmail.com Reply-To: common-user@hadoop.apache.org common-user@hadoop.apache.org Date: Thu, 12 Apr 2012 07:35:43 -0500 To: common-user@hadoop.apache.org common-user@hadoop.apache.org Subject: setting client retry In the hdfs-site.xml file what argument do I need to set for client retries? Also, what is the default parameter? -- --- Get your facts first, then you can distort them as you please.--
Anybody have experience using NetApp Direct Access Storage (DAS) in a Hadoop cluster?
Netapp has some marketing information about their NetApp Open Solution for Hadoop that provides external direct access storage. (See http://www.netapp.com/us/solutions/infrastructure/hadoop.html and the PDF white paper http://www.netapp.com/us/library/supportdocs/ds-3237.html) Anybody have any experience with this type of approach vs. the use of traditional internal storage? Thanks, Kent
UDF compiling
I am trying to compiling a customized WordCount UDF but I get this cannot find symbol error when I compile. And I'm not sure how to resolve this issue. hduser@master:~ javac -classpath /usr/lib/hadoop/hadoop-core-0.20.2-cdh3u3.jar WordCount.java WordCount.java:24: error: cannot find symbol conf.setMapperClass(WordMapper.class); ^ symbol: class WordMapper location: class WordCount WordCount.java:25: error: cannot find symbol conf.setReducerClass(SumReducer.class); ^ symbol: class SumReducer location: class WordCount 2 errors hduser@master:~ ls SumReducer.classWordMapper.class SumReducer.java WordCount.java WordMapper.java hduser@master:~
Re: UDF compiling
The right command for javac usage in your case is: javac -classpath /usr/lib/hadoop/hadoop-core-0.20.2-cdh3u3.jar:. WordCount.java You need . for javac to find the other class files/sources within the same dir. On Fri, Apr 13, 2012 at 12:12 AM, Barry, Sean F sean.f.ba...@intel.com wrote: I am trying to compiling a customized WordCount UDF but I get this cannot find symbol error when I compile. And I'm not sure how to resolve this issue. hduser@master:~ javac -classpath /usr/lib/hadoop/hadoop-core-0.20.2-cdh3u3.jar WordCount.java WordCount.java:24: error: cannot find symbol conf.setMapperClass(WordMapper.class); ^ symbol: class WordMapper location: class WordCount WordCount.java:25: error: cannot find symbol conf.setReducerClass(SumReducer.class); ^ symbol: class SumReducer location: class WordCount 2 errors hduser@master:~ ls SumReducer.class WordMapper.class SumReducer.java WordCount.java WordMapper.java hduser@master:~ -- Harsh J
Re: UDF compiling
HI try adding the directory in which WordCount.class was placed to the -classpath 4/12 Barry, Sean F sean.f.ba...@intel.com I am trying to compiling a customized WordCount UDF but I get this cannot find symbol error when I compile. And I'm not sure how to resolve this issue. hduser@master:~ javac -classpath /usr/lib/hadoop/hadoop-core-0.20.2-cdh3u3.jar WordCount.java WordCount.java:24: error: cannot find symbol conf.setMapperClass(WordMapper.class); ^ symbol: class WordMapper location: class WordCount WordCount.java:25: error: cannot find symbol conf.setReducerClass(SumReducer.class); ^ symbol: class SumReducer location: class WordCount 2 errors hduser@master:~ ls SumReducer.classWordMapper.class SumReducer.java WordCount.java WordMapper.java hduser@master:~
Yuan Jin is out of the office.
I will be out of the office starting 04/13/2012 and will not return until 04/16/2012. I am out of office, and will reply you when I am back. For HAMSTER related things, you can contact Jason(Deng Peng Zhou/China/IBM) or Anthony(Fei Xiong/China/IBM) For CFM related things, you can contact Daniel(Liang SH Su/China/Contr/IBM) For TMB related things, you can contact Flora(Jun Ying Li/China/IBM) For TWB related things, you can contact Kim(Yuan SH Jin/China/IBM) For others, I will reply you when I am back.
Re: Yuan Jin is out of the office.
who cares? On Thu, Apr 12, 2012 at 6:09 PM, Yuan Jin jiny...@cn.ibm.com wrote: I will be out of the office starting 04/13/2012 and will not return until 04/16/2012. I am out of office, and will reply you when I am back. For HAMSTER related things, you can contact Jason(Deng Peng Zhou/China/IBM) or Anthony(Fei Xiong/China/IBM) For CFM related things, you can contact Daniel(Liang SH Su/China/Contr/IBM) For TMB related things, you can contact Flora(Jun Ying Li/China/IBM) For TWB related things, you can contact Kim(Yuan SH Jin/China/IBM) For others, I will reply you when I am back.
Re: Yuan Jin is out of the office.
This is the second time. Pure junk email. Could you avoid sending email to public mail-list, Ms/Mr. Yuan Jin? On Thu, Apr 12, 2012 at 6:22 PM, Chen He airb...@gmail.com wrote: who cares? On Thu, Apr 12, 2012 at 6:09 PM, Yuan Jin jiny...@cn.ibm.com wrote: I will be out of the office starting 04/13/2012 and will not return until 04/16/2012. I am out of office, and will reply you when I am back. For HAMSTER related things, you can contact Jason(Deng Peng Zhou/China/IBM) or Anthony(Fei Xiong/China/IBM) For CFM related things, you can contact Daniel(Liang SH Su/China/Contr/IBM) For TMB related things, you can contact Flora(Jun Ying Li/China/IBM) For TWB related things, you can contact Kim(Yuan SH Jin/China/IBM) For others, I will reply you when I am back.
Re: getting UnknownHostException
Please check contents of /etc/hosts for the hostname and ipaddress mapping. On Thu, Apr 12, 2012 at 11:11 PM, Sujit Dhamale sujitdhamal...@gmail.comwrote: Hi Friends , i am getting UnknownHostException while executing Hadoop Word count program getting below details from job tracker Web page *User:* sujit *Job Name:* word count *Job File:* hdfs://localhost:54310/app/hadoop/tmp/mapred/staging/sujit/.staging/job_201204112234_0002/job.xml http://localhost:50030/jobconf.jsp?jobid=job_201204112234_0002 *Submit Host:* sujit.(null) *Submit Host Address:* 127.0.1.1 *Job-ACLs: All users are allowed* *Job Setup:*None *Status:* Failed *Failure Info:*Job initialization failed: java.net.UnknownHostException: sujit.(null) is not a valid Inet address at org.apache.hadoop.net. NetUtils.verifyHostnames(NetUtils.java:569) at org.apache.hadoop.mapred.JobInProgress.initTasks(JobInProgress.java:711) at org.apache.hadoop.mapred.JobTracker.initJob(JobTracker.java:4207) at org.apache.hadoop.mapred.EagerTaskInitializationListener$InitJob.run(EagerTaskInitializationListener.java:79) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) *Started at:* Wed Apr 11 22:36:46 IST 2012 *Failed at:* Wed Apr 11 22:36:47 IST 2012 *Failed in:* 0sec *Job Cleanup:*None Can some one help me how to resolve this issue . i tried with : http://wiki.apache.org/hadoop/UnknownHost but still not able to resolve issue , please help me out . Hadoop Version: hadoop-1.0.1.tar.gz java version 1.6.0_30 Operating System : Ubuntu 11.10 *Note *: All node were up before starting execution of Program Kind Regards Sujit Dhamale http://wiki.apache.org/hadoop/UnknownHost -- https://github.com/zinnia-phatak-dev/Nectar
Re: hadoop debugging in eclipse
google remote debug hadoop in eclipse“. It is super easy. On Fri, Apr 13, 2012 at 10:42 AM, kewen wang wangkewen...@gmail.com wrote: Hello, Can you give me some advice about how to debug hadoop src codes in eclipse if I want to modify some src codes in hadoop? Thanks. WangKewen -- Best, WU Pengcheng ( Tim )
Anyone successfully run Hadoop in pseudo or cluster model under Cygwin?
If yes. Could u send me an email? Your prompt reply will be appreciated, because I asked two questions in this mailing list in March, but no one reply me. Questions are listed in http://mail-archives.apache.org/mod_mbox/hadoop-common-user/201203.mbox/%3CCA+2n-oGLocSo3YMUiYzhbtOPzO=11g1rl_b45+y-tggyvzk...@mail.gmail.com%3E and http://mail-archives.apache.org/mod_mbox/hadoop-common-user/201203.mbox/%3CCA+2n-oERYM15BKb4dc9KETpjNfucqzd-=coecf2sgclj5u7...@mail.gmail.com%3E -- Best, WU Pengcheng ( Tim )
