Re: Decomssion datanode - no response

I filed this issue at : https://issues.apache.org/jira/browse/HDFS-4959 On Fri, Jul 5, 2013 at 1:06 PM, Azuryy Yu wrote: > Client hasn't any connection problem. > > > On Fri, Jul 5, 2013 at 12:46 PM, Devaraj k wrote: > >> And also could you check whether the client is connecting to NameNode >

Re: Which InputFormat to use?

Using InputFormat under mapreduce package. mapred package is very old package. but generally you can extend from FileInputFormat under o.a.h.mapreduce package. On Fri, Jul 5, 2013 at 1:23 PM, Devaraj k wrote: > Hi Ahmed, > > ** ** > > Hadoop 0.20.0 included the new mapred

RE: Which InputFormat to use?

Hi Ahmed, Hadoop 0.20.0 included the new mapred API, these sometimes refer as context objects. These are designed to make API easier to evolve in future. There are some differences between new & old API's, > The new API's favour abstract classes rather than interfaces, since abs

Re: Decomssion datanode - no response

Client hasn't any connection problem. On Fri, Jul 5, 2013 at 12:46 PM, Devaraj k wrote: > And also could you check whether the client is connecting to NameNode or > any failure in connecting to the NN. > > ** ** > > Thanks > > Devaraj k > > ** ** > > *From:* Azuryy Yu [mailto:azury

RE: Decomssion datanode - no response

And also could you check whether the client is connecting to NameNode or any failure in connecting to the NN. Thanks Devaraj k From: Azuryy Yu [mailto:azury...@gmail.com] Sent: 05 July 2013 09:15 To: user@hadoop.apache.org Subject: Re: Decomssion datanode - no response I added dfs.hosts.exclude

Re: Decomssion datanode - no response

I added dfs.hosts.exclude before NN started. and I updated /usr/local/hadoop/conf/dfs_exclude whith new hosts, but It doesn't decomssion. On Fri, Jul 5, 2013 at 11:39 AM, Devaraj k wrote: > When did you add this configuration in NN conf? > > > dfs.hosts.exclude > /usr/local/ha

RE: Decomssion datanode - no response

When did you add this configuration in NN conf? dfs.hosts.exclude /usr/local/hadoop/conf/dfs_exclude If you have added this configuration after starting NN, it won't take effect and need to restart NN. If you have added this config with the exclude file before NN start, you can up

RE: Which InputFormat to use?

A trainer at Hortonworks told me that org.apache.hadoop.mapred is the old package. So for all intent and purposes use the new one: org.apache.hadoop.mapreduce. Otto out! From: Ahmed Eldawy [mailto:aseld...@gmail.com] Sent: July-04-13 2:30 PM To: user@hadoop.apache.org Subject: Which InputFormat

Re: 答复: Decomssion datanode - no response

It's 20 minutes passed after I ran -refreshNodes, but there is no decomssion nodes showed on the UI. and canno find any hints in the NN and DN logs. On Fri, Jul 5, 2013 at 11:16 AM, Francis.Hu wrote: > I know the default value is 10 minutes and 30 seconds for switching > datanodes from live to

RE: Requesting containers on a specific host

Hi Kishore, hadoop-2.1.0 beta release is in voting process now. You can try out from hadoop-2.1.0 beta RC http://people.apache.org/~acmurthy/hadoop-2.1.0-beta-rc0/ or you could check the same with trunk build. Thanks Devaraj k From: Krishna Kishore Bonagiri [mailto:write2kish...@gmail.com] Se

Re: Decomssion datanode - no response

Thanks Devaraj, There are no any releated logs in the NN log and DN log. On Fri, Jul 5, 2013 at 11:14 AM, Devaraj k wrote: > Do you see any log related to this in Name Node logs when you issue > refreshNodes dfsadmin command? > > ** ** > > Thanks > > Devaraj k > > ** ** > > *From:

答复: Decomssion datanode - no response

I know the default value is 10 minutes and 30 seconds for switching datanodes from live to dead. 发件人: Azuryy Yu [mailto:azury...@gmail.com] 发送时间: Friday, July 05, 2013 10:42 收件人: user@hadoop.apache.org 主题: Decomssion datanode - no response Hi, I am using hadoop-2.0.5-alpha, and I added 5

RE: Decomssion datanode - no response

Do you see any log related to this in Name Node logs when you issue refreshNodes dfsadmin command? Thanks Devaraj k From: Azuryy Yu [mailto:azury...@gmail.com] Sent: 05 July 2013 08:12 To: user@hadoop.apache.org Subject: Decomssion datanode - no response Hi, I am using hadoop-2.0.5-alpha, and I

Decomssion datanode - no response

Hi, I am using hadoop-2.0.5-alpha, and I added 5 datanodes into dfs_exclude, hdfs-site.xml: dfs.hosts.exclude /usr/local/hadoop/conf/dfs_exclude then: hdfs dfsadmin -refreshNodes but there is no decomssion nodes showed on the webUI. and not any releated logs in the datanode log. wh

Re: How to update a file which is in HDFS

The current stable release doesn't support append, not even through the API. If you really want this you have to switch to hadoop 2.x. See this JIRA . Warm Regards, Tariq cloudfront.blogspot.com On Fri, Jul 5, 2013 at 3:05 AM, John Lilley wrote:

Re: Requesting containers on a specific host

John, This feature will available in the upcoming 2.1.0-beta release. The first release candidate (RC) has been cut, but it seems a new RC will be needed. The exact release date is still not known but it should be soon. thanks. On Thu, Jul 4, 2013 at 2:43 PM, John Lilley wrote: > Arun, >

RE: Requesting containers on a specific host

Arun, I'm don't know how to interpret the release schedule from the JIRA. It says that the patch targets 2.1.0 and it is checked into the trunk, does that mean it is likely to be rolled into the first Hadoop 2 GA or will it have to wait for another cycle? Thanks, John From: Arun C Murthy [mail

RE: How to update a file which is in HDFS

Manickam, HDFS supports append; it is the command-line client that does not. You can write a Java application that opens an HDFS-based file for append, and use that instead of the hadoop command line. However, this doesn't completely answer your original question: "How do I move only the delta p

Which InputFormat to use?

Hi I'm developing a new set of InputFormats that are used for a project I'm doing. I found that there are two ways to create a new InputFormat. 1- Extend the abstract class org.apache.hadoop.mapreduce.InputFormat 2- Implement the interface org.apache.hadoop.mapred.InputFormat I don't know why ther

Re: typical JSON data sets

Hi John, I have just started pulling Twitter conversions using Apache Flume. But I have not started processing the pulled data yet. And my answers below: 1) How large is each JSON document? Averages from 100 KB to 2 MB. Flume rolls a new file every 1 minutes (which is configurable). So the

Re: Requesting containers on a specific host

Thanks Arun, it seems to be available with 2.1.0-beta, when will that be released? Or if I want it now, could I get from the trunk? -Kishore On Thu, Jul 4, 2013 at 5:58 PM, Arun C Murthy wrote: > To guarantee nodes on a specific container you need to use the whitelist > feature we added recent

Re: How to update a file which is in HDFS

You can append using WebHDFS.. Following link may help you-- http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/WebHDFS.html#Append_to_a_File On Thu, Jul 4, 2013 at 5:17 PM, Mohammad Tariq wrote: > Hello Manickam, > > Append is currently not possible. > > Warm Regards

DistCp fails on open files

Hi all, I'm trying to copy files from a source HDFS cluster to another. But I have numerous files open for writing, and DistCp fails on thoses ones. I've found a reference of that on jira https://issues.apache.org/jira/browse/MAPREDUCE-2160 Any workaround ? Anyone faced that too ? Thanks

Re: Requesting containers on a specific host

To guarantee nodes on a specific container you need to use the whitelist feature we added recently: https://issues.apache.org/jira/browse/YARN-398 Arun On Jul 4, 2013, at 3:14 AM, Krishna Kishore Bonagiri wrote: > I could get containers on specific nodes using addContainerRequest() on > AMRMC

Re: How to update a file which is in HDFS

Hello Manickam, Append is currently not possible. Warm Regards, Tariq cloudfront.blogspot.com On Thu, Jul 4, 2013 at 4:40 PM, Manickam P wrote: > Hi, > > I have moved my input file into the HDFS location in the cluster setup. > Now i got a new set of file which has some new records al

Dead Node and Under Replication

Hi Guys, How do I tell hadoop to wait longer before fixing Under-Replication when a node dies. Currently I have a node down which crashed, and hadoop detected it down and is busy fixing Under-Replication. The problem is that the extra load required to fix replication is causing our applicat

How to update a file which is in HDFS

Hi, I have moved my input file into the HDFS location in the cluster setup. Now i got a new set of file which has some new records along with the old one. I want to move the delta part alone into HDFS because it will take more time to move the file from my local to HDFS location. Is it possible

Re: Requesting containers on a specific host

I could get containers on specific nodes using addContainerRequest() on AMRMClient. But there are issues with it. I have two nodes, node1 and node2 in my cluster. And, my Application Master is trying to get 3 containers on node1, and 3 containers on node2 in that order. While trying to request on

Re: 答复: interval from Live to Dead about DataNode

Yes. The default is 30s. On Thu, Jul 4, 2013 at 11:10 AM, Francis.Hu wrote: > Nicolas, > > ** ** > > Thanks for your help. had a look at HDFS-3703. > > ** ** > > So I need to turn on dfs.namenode.check.stale.datanode > > and set a shorter time for dfs.namenode.stale.datanode.interv

答复: interval from Live to Dead about DataNode

Nicolas, Thanks for your help. had a look at HDFS-3703. So I need to turn on dfs.namenode.check.stale.datanode and set a shorter time for dfs.namenode.stale.datanode.interval Thanks, Francis.Hu 发件人: Nicolas Liochon [mailto:nkey...@gmail.com] 发送时间: Thursday, July 04, 2013 16:28 收

Re: Could not get additional block while writing hundreds of files

Hye Azuryy, During the import, dfsadmin -report : DFS Used%: 17.72% Moreover, it succeeds from time to time w/ the same data load. It seems that Datanode appears to be down to the Namenode, but why ? On Thu, Jul 4, 2013 at 3:31 AM, Azuryy Yu wrote: > Hi Manuel, > > 2013-07-03 15:03:16,427 W

Re: interval from Live to Dead about DataNode

Hi, It's 10 minutes and 30s. See the stale mode described in HDFS-3703 if you need something shorter. Cheers, Nicolas On Thu, Jul 4, 2013 at 10:05 AM, Francis.Hu wrote: > Hi,All > > ** ** > > Does anyone know when the status of datanode switch from live to dead at > the internal of name

Re: Datanode support different Namespace

It's random. On Jul 4, 2013 3:33 PM, "Bing Jiang" wrote: > If not set cluster id in formatting the Namenode, is there a policy in > hdfs to guarantee the even of distributing DataNodes into different > Namespace, or just randomly? > > > > 2013/7/4 Azuryy Yu > >> Additional, >> >> If these are t

interval from Live to Dead about DataNode

Hi,All Does anyone know when the status of datanode switch from live to dead at the internal of namenode ? the scenario: When i stopped a datanode with command, the status of that datanode in the web UI of namenode displays 'live' and 'In Service' for almost 5 minutes. I know the defaul

Re: Datanode support different Namespace

If not set cluster id in formatting the Namenode, is there a policy in hdfs to guarantee the even of distributing DataNodes into different Namespace, or just randomly? 2013/7/4 Azuryy Yu > Additional, > > If these are two new clusters, then on each namenode, using "hdfs namenode > -format -clu

Re: 【dfs.namenode.shared.edits.dir can support different NameServices】

Thanks for Azuryy's reply. I used configuration: dfs.namenode.shared.edits.dir.hbasecluster.hnn1 qjournal://GS-CIX-SEV0001:8485;GS-CIX-SEV0002:8485;GS-CIX-SEV0003:8485/hbasecluster dfs.namenode.shared.edits.dir.hbasecluster.hnn2 qjournal://GS-CIX-SEV0001:8485;GS-CIX-SEV0002:8485;GS-CIX-S

Re: Datanode support different Namespace

Additional, If these are two new clusters, then on each namenode, using "hdfs namenode -format -clusterID yourID" But if you want to upgrade these two clusters from NonHA to HA, then using "bin/start-dfs.sh -upgrade -clusterID yourID" On Thu, Jul 4, 2013 at 3:14 PM, Azuryy Yu wrote: > > This

Re: Output Directory not getting created

Hi , Please check whether u have permission to the Output directory . Error say that u dont have permission . Regards Syed Thanks and Regards, S SYED ABDUL KATHER On Thu, Jul 4, 2013 at 10:25 AM, devara...@huawei.com [via Lucene] < ml-node+s472066n4075377...@n3.nabble.co

Re: Datanode support different Namespace

This is because you don't use the same clusterID. all data nodes and namenodes should use the same clusterID. On Thu, Jul 4, 2013 at 3:12 PM, Bing Jiang wrote: > Hi, all > > We try to use hadoop-2.0.5-alpha, using two namespaces, one is for hbase > cluster, and the other one is for common use.A

Datanode support different Namespace

Hi, all We try to use hadoop-2.0.5-alpha, using two namespaces, one is for hbase cluster, and the other one is for common use.At the same time, we use Quorum Journal policy as HA. GS-CIX-SEV0001, GS-CIX-SEV0002, namenodes in hbasecluster namespace GS-CIX-SEV0003, GS-CIX-SEV0004, namenodes in c

Re: 【dfs.namenode.shared.edits.dir can support different NameServices】

Hi Bing, HA is not confilct with HDFS federation. for example, you have two name services: cluster1, cluster2, then, dfs.namenode.shared.edits.dir qjournal://n1.com:8485;n2.com:8485/cluster1 dfs.namenode.shared.edits.dir qjournal://n1.com:8485;n2.com:8485/cluster2 On Thu,