Re: Hadoop 2.6.0 - No DataNode to stop

Thanks for your answer. I put the FQDN of the DataNodes in the slaves file on each node (one FQDN per line). Here’s the full DataNode log after the start (the log of the other DataNode is exactly the same): 2015-03-02 00:29:41,841 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:

Re: Permission Denied

David, Thanks for the reply. Taking the questions in the opposite order, my accumulo-site.xml does not have volumes specified. I edited the accumulo-site.xml so it now has property nameinstance.volumes/name valuehdfs://localhost:9000/accumulo/value descriptioncomma separated list

Re: Hadoop 2.6.0 - No DataNode to stop

Hi Did you check your slaves file is correct ? That the datanode process is actually running ? Did you check its log file ? That the datanode is available ? (dfsadmin -report, through the WUI) We need more detail Ulul Le 28/02/2015 22:05, Daniel Klinger a écrit : Thanks but i know how to

Re: cleanup() in hadoop results in aggregation of whole file/not

Edit : instead of buffering in Hash and then emitting at cleanup you can use a combiner. Likely slower but easier to code if speed is not your main concern Le 01/03/2015 13:41, Ulul a écrit : Hi I probably misunderstood your question because my impression is that it's typically a job for a

about the jobid

Hi, all. I run nutch in deploy mode at about 3pm, 02/28/2015, but the jobid is job_201502271057_0251.I found that 201502271057 is the time I start hadoop(by start-all.sh). Is the jobid should be job_201502281500_ ？ what is the problem? system date: [jediael@master history]$ date

Re: cleanup() in hadoop results in aggregation of whole file/not

Hi I probably misunderstood your question because my impression is that it's typically a job for a reducer. Emit local min and max with two keys from each mapper and you will easily get gobal min and max in reducer Ulul Le 28/02/2015 14:10, Shahab Yunus a écrit : As far as I understand

Re: Permission Denied

hadoop fs -mkdir /accumulo hadoop fs -chown accumulo:supergroup /accumulo div Original message /divdivFrom: David Patterson patt...@gmail.com /divdivDate:03/01/2015 7:04 PM (GMT-05:00) /divdivTo: user@hadoop.apache.org /divdivCc: /divdivSubject: Re: Permission Denied

How to troubleshoot failed or stuck jobs

Hi, Wanted to understand, How to troubleshoot failed or stuck jobs ? Thanks Krish

How to find bottlenecks of the cluster ?

Hi, I wanted to understand, how should we find out the bottleneck of the cluster? Thanks Krish

Re: How to troubleshoot failed or stuck jobs

Here are some related discussions and JIRA: http://search-hadoop.com/m/LgpTk2gxrGx http://search-hadoop.com/m/LgpTk2YLArE https://issues.apache.org/jira/browse/MAPREDUCE-6190 Cheers On Sun, Mar 1, 2015 at 8:41 PM, Krish Donald gotomyp...@gmail.com wrote: Hi, Wanted to understand, How to

RE: about the jobid

Hi Yarn application id allocation based on the daemon ResourceManager start time(assuming cluster is MR2 else JobTracker start time). Say if you have 3 job client submitting jobs to Yarn, then application id are application_rm-start-time_0001, application_rm-start-time_0002,

Re: How to troubleshoot failed or stuck jobs

Thanks for Link Ted, However wanted to understand the approach which should be taken when troubleshooting failed or stuck jobs ? On Sun, Mar 1, 2015 at 8:52 PM, Ted Yu yuzhih...@gmail.com wrote: Here are some related discussions and JIRA: http://search-hadoop.com/m/LgpTk2gxrGx

RE: How to troubleshoot failed or stuck jobs

Hi 1. For the Failed jobs, you can directly check the MRAppMaster logs. There you get reason for failed jobs. 2. For the stuck job, you need to do some ground work to identify what is going wrong. It can be either YARN issue or MapReduce issue. 2.1 In a recent time, I have

How to Backup and Restore various components of Hadoop ?

Hi, Wanted to know the backup and recovery strategy for your Hadoop cluster. How do you Backup and Restore various components of Hadoop ? Thanks Krish

Re: How to troubleshoot failed or stuck jobs

Thanks Rohith ... What are the other issue you have seen for failed or stuck jobs? On Sun, Mar 1, 2015 at 10:06 PM, Rohith Sharma K S rohithsharm...@huawei.com wrote: Hi 1. For the Failed jobs, you can directly check the MRAppMaster logs. There you get reason for failed jobs.

how to find corrupt block in java code

hadoop version 2.4.1 I can find corrupt files. $HADOOP_PREFIX/bin/hdfs fsck / -list-corruptfileblocks How to find corrupt block in java code ?

Re: How to find bottlenecks of the cluster ?

This is a vast topic.Can you tell what components are there in your data pipe line and how data flows in to system and the way its processed.There are several inbuilt tests like testDFSIO and terasort that you can run. -Nishan On Sun, Mar 1, 2015 at 9:45 PM, Krish Donald gotomyp...@gmail.com

Re: How to find bottlenecks of the cluster ?

Basically we have 4 points to consider, CPU , Memory, IO and Network So how to see which one is causing the bottleneck ? What parameters we should consider etc ? On Sun, Mar 1, 2015 at 10:57 PM, Nishanth S nishanth.2...@gmail.com wrote: This is a vast topic.Can you tell what components are

Re: how to find corrupt block in java code

Hi, cho I think you may start digging from org.apache.hadoop.hdfs.tools.DFSck.java and org.apache.hadoop.hdfs.server.namenode.FsckServlet.java. Good luck! Drake 민영근 Ph.D kt NexR On Mon, Mar 2, 2015 at 3:22 PM, cho ju il tjst...@kgrid.co.kr wrote: hadoop version 2.4.1 I can find corrupt

Re: Hadoop 2.6.0 - No DataNode to stop

1.Stop the service 2.Change the permissions for log and pid directory once again to hdfs. 3.Start service with hdfs. This will resolve the issue On Sun, Mar 1, 2015 at 6:40 PM, Daniel Klinger d...@web-computing.de wrote: Thanks for your answer. I put the FQDN of the DataNodes in the

答复: how to find corrupt block in java code

$HADOOP_PREFIX/bin/hdfs fsck -list-corruptfileblocks 发件人: Drake민영근 [mailto:drake@nexr.com] 发送时间: 2015年3月2日 15:31 收件人: user; cho ju il 主题: Re: how to find corrupt block in java code Hi, cho I think you may start digging from org.apache.hadoop.hdfs.tools.DFSck.java and

error: [Errno 113] No route to host cloudera

Hi, I tried hard to debug the issue but nothing worked. I am getting error: [Errno 113] No route to host cloudera in cloudera agent log file. Below are some output : [root@snncloudera cloudera-scm-agent]# more /etc/hosts 127.0.0.1 localhost localhost.localdomain localhost4

Re: error: [Errno 113] No route to host cloudera

Stop Iptables services on each datanode. On Sun, Mar 1, 2015 at 12:00 PM, Krish Donald gotomyp...@gmail.com wrote: Hi, I tried hard to debug the issue but nothing worked. I am getting error: [Errno 113] No route to host cloudera in cloudera agent log file. Below are some output :

Permission Denied

I'm trying to create an Accumulo/Hadoop/Zookeeper configuration on a single (Ubuntu) machine, with Hadoop 2.6.0, Zookeeper 3.4.6 and Accumulo 1.6.1. I've got 3 userids for these components that are in the same group and no other users are in that group. I have zookeeper running, and hadoop as

RE: Permission Denied

It looks like / is owned by hadoop.supergroup and the perms are 755. You could precreate /accumulo and chown it appropriately, or set the perms for / to 775. Init is trying to create /accumulo in hdfs as the accumulo user and your perms dont allow it. Do you have instance.volumes set in