Decommissioning a node takes forever

Hi All, I am a novice hadoop user . I tried removing a node from my cluster of 2 nodes by adding the ip in excludes file and running dfsadmin -refreshNodes command . But decommissioning takes a very long time I left it over the weekend and still it was not complete. Your inputs will help -- Wa

build error

Hii all, i am trying to build hadoop using this command "mvn package -Pdist,native -DskipTests=true -Dtar" and error is [INFO] --- maven-antrun-plugin:1.7:run (dist) @ hadoop-hdfs-httpfs --- [INFO] Executing tasks main: [get] Getting: http://archive.apache.org/dist/tomcat/tomcat-6/v6.0.36

Re: Decommissioning a node takes forever

Hi, which version HDFS you used? On Wed, Mar 26, 2014 at 3:17 PM, Bharath Kumar wrote: > > Hi All, > > I am a novice hadoop user . I tried removing a node from my cluster of 2 > nodes by adding the ip in excludes file and running dfsadmin -refreshNodes > command . But decommissioning takes a ve

RE: Decommissioning a node takes forever

can you please elaborate more..? Like how many nodes are there in cluster and what's the replication factor for files..? Normally decommission will be success once all the replica's from the excluded node is replicated another node in cluster(another node should be availble,,).. T

Re: build error

Hi, I think sometimes it will result by your network condition 2014-03-26 15:23 GMT+08:00 Banty Sharma : > Hii all, > > i am trying to build hadoop using this command "mvn package -Pdist,native > -DskipTests=true -Dtar" and error is > > > [INFO] --- maven-antrun-plugin:1.7:run (dist) @ hado

Re: R on hadoop

HI Siddharth, You can try "Big Data Analytics with R and Hadoop " Book, it gives many options and detailed steps to integrate Hadoop and R. If you need this book then mail me to saravanan.nagarajan...@gmail.com. Thanks, Saravanan *linkedin.com/in/saravanan303

Re: R on hadoop

Do you mean (1) running mapreduce jobs from R ? (2) Running R from a mapreduce job ? Without much extra ceremony, for the latter, you could use either MapReduce streaming or pig to call a custom program, as long as R is installed on every node of the cluster itself On Wed, Mar 26, 2014 at 6:39

RE: build error

As Vincent said this is most likely related to your network settings. I would try first a ping [german@gfldesktop tmp]$ ping archive.apache.org …. 64 bytes from eos.apache.org (140.211.11.131): icmp_seq=1 ttl=53 time=69.7 ms If this works, then wget or curl, e.g. [german@gfldeskto

Re: R on hadoop

HI Jay, Below is my understanding of Hadoop+R environment. 1. R contain Many data mining algorithm, to re-use this we have many tools like RHIPE,RHAdoop,etc 2.This tools will convert R algorithm and run in Hadoop map Reduce using RMR,But i am not sure whether it will work for all algorithms in R

re-replication after data node failure

We recently had a node die in our hbase cluster. Afterwards, we saw a huge increase in traffic and I/O as hdfs re-replicated data from the dead node. This negatively affected our application and we are trying to see if there is a way to slow down this process so the app can still run (if a bit slow

Re: re-replication after data node failure

The balancer is not what handles adding extra replicas in the case of a node failure, but it looks like the balancer bandwidth setting is the way to throttle. See: http://mail-archives.apache.org/mod_mbox/hadoop-hdfs-user/201301.mbox/%3c50f870c1.5010...@getjar.com%3E On Wed, Mar 26, 2014 at 10:51

Getting error message from AM container launch

Running a non-MapReduce YARN application, one of the containers launched by the AM is failing with an error message I've never seen. Any ideas? I'm not sure who exactly is running "nice" or why its argument list would be too long. Thanks john Container for appattempt_1395755163053_0030_01

RE: Getting error message from AM container launch

We do have a fairly long container command-line. Not huge, around 200 characters. John From: John Lilley [mailto:john.lil...@redpoint.net] Sent: Wednesday, March 26, 2014 4:38 PM To: user@hadoop.apache.org Subject: Getting error message from AM container launch Running a non-MapReduce YARN appl

RE: Getting error message from AM container launch

On further examination they appear to be 369 characters long. I've read about similar issues showing when the environment exceeds 132KB, but we aren't putting anything significant in the environment. John From: John Lilley [mailto:john.lil...@redpoint.net] Sent: Wednesday, March 26, 2014 4:41

Re: Getting error message from AM container launch

You used 'nice' in your app? Sent from my iPhone5s > On 2014年3月27日, at 6:55, John Lilley wrote: > > On further examination they appear to be 369 characters long. I’ve read > about similar issues showing when the environment exceeds 132KB, but we > aren’t putting anything significant in the

Call for big data proposals

Dear subscribers, My name is Shumin Guo. I am the author of the Hadoop management book "Hadoop Operations and Cluster Management Cookbook

Hadoop 2.2.0 Distributed Cache

Hi Everyone, I'm submitting a MapReduce job using the -files option to copy a text file that contains properties I use for the map and reduce functions. I'm trying to obtain the local cache files in my mapper function using: Path[] paths = context.getLocalCacheFiles(); However, i get an error s

Question on Book

Hello is there a good book on the particulars of learning mapreduce java programming for dummies :O   >

Re: Question on Book

http://hadoopilluminated.com/ On Wed, Mar 26, 2014 at 6:37 PM, Publius wrote: > Hello > > is there a good book on the particulars of learning mapreduce java > programming for dummies :O > > > >

Re: Getting error message from AM container launch

HI John, Typically, this is caused by somewhere in your program set "nice" as AM launching command. You can check the "real" script which YARN used to launch AM. You need set "yarn.nodemanager.delete.debug-delay-sec" in yarn-site.xml on all NMs to a larger value (like 600, 10 min), to make NMs don'

Re: Decommissioning a node takes forever

Hi Brahma, It might be some files have replication factor more than the actual number of datanodes, so namenode will not be able to decommission the machine because it cannot get the replica count settled. Run the following command to check the replication factor of the files on the hdfs, see if a

RE: Getting error message from AM container launch

No, the “nice” is internal to Hadoop, apparently to make sure that the Hadoop infrastructure processes can’t be starved by the applications. Actually we figured out what was going on. There was a very large environment variable being set for the AM, and it exceeded the 132KB or so allowed for

RE: Getting error message from AM container launch

Wangda Tan, Thanks for your reply! We did actually figure out where the problem was coming from, but this is a very helpful technique to know. John From: Wangda Tan [mailto:wheele...@gmail.com] Sent: Wednesday, March 26, 2014 6:35 PM To: user@hadoop.apache.org Subject: Re: Getting error messa

Re: Getting error message from AM container launch

Glad to hear that :) -- Wangda Tan Regards, Wangda Tan On Thu, Mar 27, 2014 at 10:36 AM, John Lilley wrote: > Wangda Tan, > > > > Thanks for your reply! We did actually figure out where the problem was > coming from, but this is a very helpful technique to know. > > > > John > > > > > > *From

yarn application still running but dissapear from UI

Hi Hadoop Users, I'm using hadoop-2.2.0 with YARN. Today I stumble upon a problem with YARN management UI, when I look into cluster/apps, there is one apps running but not showing in the entries. I make sure there is an application running in cluster with command "yarn application -list", could

Re: Hadoop 2.2.0 Distributed Cache

where did you get the error? from the compiler or the runtime? Regards, *Stanley Shi,* On Thu, Mar 27, 2014 at 7:34 AM, Jonathan Poon wrote: > Hi Everyone, > > I'm submitting a MapReduce job using the -files option to copy a text file > that contains properties I use for the map and reduce fu

Re: yarn application still running but dissapear from UI

Sounds like https://issues.apache.org/jira/browse/YARN-1810. +Vinod On Mar 26, 2014, at 7:44 PM, Henry Hung wrote: > Hi Hadoop Users, > > I’m using hadoop-2.2.0 with YARN. > Today I stumble upon a problem with YARN management UI, when I look into > cluster/apps, there is one apps running but

Re: R on hadoop

Try OpenSource h2o.ai - a cran-style package that allows fast & scalable R on Hadoop in-Memory. One can invoke single threaded R from h2o package and the runtime on clusters is Java (not R!) - So you get better memory management. http://docs.0xdata.com/deployment/hadoop.html http://docs.0xdata