Why LineRecordWriter.write(..) is synchronized

Hi, LineRecordWriter.write(..) is synchronized. I did not find any other RecordWriter implementations define the write as synchronized. Any specific reason for this. regards, sathwik

is RM require a lot of memory?

the yarn resource manager require a lot of memory as jobtracker?

is it ok? build hadoop cluster on kvm on product envionment?

hi,all: my company has not much burget for boxes,if i build cluster on kvm ,it will cause a lot of impact on performance??

Re: specify Mapred tasks and slots

Thanks Harsh and all friends response. That's helpful. On Thu, Aug 8, 2013 at 11:55 AM, Harsh J wrote: > What Devaraj said. Except that if you use CapacityScheduler, then you > can bind together memory requests and slot concepts, and be able to have a > task > grab more than one slot for itsel

Re: specify Mapred tasks and slots

What Devaraj said. Except that if you use CapacityScheduler, then you can bind together memory requests and slot concepts, and be able to have a task grab more than one slot for itself when needed. We've discussed this aspect previously at http://search-hadoop.com/m/gnFs91yIg1e On Thu, Aug 8, 2013

RE: specify Mapred tasks and slots

One task can use only one slot, It cannot use more than one slot. If the task is Map task then it will use one map slot and if the task is reduce task the it will use one reduce slot from the configured ones. Thanks Devaraj k From: Azuryy Yu [mailto:azury...@gmail.com] Sent: 08 August 2013 08:2

Re: specify Mapred tasks and slots

My question is can I specify how many slots to be used for each M/R task? On Thu, Aug 8, 2013 at 10:29 AM, Shekhar Sharma wrote: > Slots are decided upon the configuration of machines, RAM etc... > > Regards, > Som Shekhar Sharma > +91-8197243810 > > > On Thu, Aug 8, 2013 at 7:19 AM, Azuryy Yu

Re: specify Mapred tasks and slots

Slots are decided upon the configuration of machines, RAM etc... Regards, Som Shekhar Sharma +91-8197243810 On Thu, Aug 8, 2013 at 7:19 AM, Azuryy Yu wrote: > Hi Dears, > > Can I specify how many slots to use for reduce? > > I know we can specify reduces tasks, but is there one task occupy one

Re: specify Mapred tasks and slots

use mapred.tasktracker.reduce.tasks in mapred-site.xml the default value is 2...Which means that on this task tracker it will not run more than 2 reducer tasks at any given point of time.. Regards, Som Shekhar Sharma +91-8197243810 On Thu, Aug 8, 2013 at 7:19 AM, Azuryy Yu wrote: > Hi Dears,

specify Mapred tasks and slots

Hi Dears, Can I specify how many slots to use for reduce? I know we can specify reduces tasks, but is there one task occupy one slot? it it possible that one tak occupy more than one slot in Hadoop-1.1.2. Thanks.

Re: compatible hadoop version for hbase-0.94.10

Thanks Ted. Regards, OC. On Wed, Aug 7, 2013 at 12:22 PM, Ted Yu wrote: > If you look at pom.xml for 0.94, you should see hadoop-1.1 and hadoop-1.2 > profiles. > > Those hadoop releases (1.1.2 and 1.2.0, respectively) should work. > > > On Wed, Aug 7, 2013 at 12:13 PM, oc tsdb wrote: > >> Hi,

Re: compatible hadoop version for hbase-0.94.10

If you look at pom.xml for 0.94, you should see hadoop-1.1 and hadoop-1.2 profiles. Those hadoop releases (1.1.2 and 1.2.0, respectively) should work. On Wed, Aug 7, 2013 at 12:13 PM, oc tsdb wrote: > Hi, > > I need to create a opentsdb cluster which needs hbase and hadoop. > I picked the lates

compatible hadoop version for hbase-0.94.10

Hi, I need to create a opentsdb cluster which needs hbase and hadoop. I picked the latest hbase supported by opentsdb which is hbase-.0.94.10 Can anybody please suggest which latest version of Hadoop I can use with hbase-0.94.10 Thanks in advance. Regards, VSR.

Re: setLocalResources() on ContainerLaunchContext

Hi Omkar, I will try that. I might have got 2 of '/' wrongly while trying it in different ways to make it work. The file kishore/kk.ksh is accessible to the same user that is running the AM container. And my another questions is to understand what are the exact benefits of using this resource

Re: Datanode doesn't connect to Namenode

Disable the firewall on data node and namenode machines.. Regards, Som Shekhar Sharma +91-8197243810 On Wed, Aug 7, 2013 at 11:33 PM, Jitendra Yadav wrote: > Your hdfs name entry should be same on master and databnodes > > * fs.default.name* > *hdfs://cloud6:54310* > > Thanks > On Wed, Aug 7, 20

Re: setLocalResources() on ContainerLaunchContext

Good that your timestamp worked... Now for hdfs try this hdfs://: now verify that your absolute path is correct. I hope it will work. bin/hadoop fs -ls hdfs://isredeng:8020*//*kishore/kk.ksh... why "//" ?? you have hdfs file at absolute location /kishore/kk.sh? is /kishore and /kishore/kk.sh acc

Re: Datanode doesn't connect to Namenode

Your hdfs name entry should be same on master and databnodes * fs.default.name* *hdfs://cloud6:54310* Thanks On Wed, Aug 7, 2013 at 11:05 PM, Felipe Gutierrez < felipe.o.gutier...@gmail.com> wrote: > on my slave the process is running: > hduser@cloud15:/usr/local/hadoop$ jps > 19025 DataNode > 1

Re: Datanode doesn't connect to Namenode

I'm not able to see tasktraker process on your datanode. On Wed, Aug 7, 2013 at 11:14 PM, Felipe Gutierrez < felipe.o.gutier...@gmail.com> wrote: > yes, in slave I type: > fs.default.name > hdfs://cloud15:54310 > > in master I type: > fs.default.name > hdfs://cloud6:54310 > > If I type cloud6 on

Re: Extra start-up overhead with hadoop-2.1.0-beta

No Ravi, I am not running any MR job. Also, my configuration files are not big. On Wed, Aug 7, 2013 at 11:12 PM, Ravi Prakash wrote: > I believe https://issues.apache.org/jira/browse/MAPREDUCE-5399 causes > performance degradation in cases where there are a lot of reducers. I can > imagine it c

Re: Datanode doesn't connect to Namenode

yes, in slave I type: fs.default.name hdfs://cloud15:54310 in master I type: fs.default.name hdfs://cloud6:54310 If I type cloud6 on both configurations, the slave doesn't start. On Wed, Aug 7, 2013 at 2:40 PM, Sivaram RL wrote: > Hi , > > your configuration of Datanode shows > > fs.defau

Re: Extra start-up overhead with hadoop-2.1.0-beta

I believe https://issues.apache.org/jira/browse/MAPREDUCE-5399 causes performance degradation in cases where there are a lot of reducers. I can imagine it causing degradation if the configuration files are super big / some other weird cases. From: Krishna Ki

Re: Datanode doesn't connect to Namenode

Hi , your configuration of Datanode shows fs.default.name hdfs://cloud15:54310 But you have said Namenode is configured on master (cloud6). Can you check the configuration again ? Regards, Sivaram R L On Wed, Aug 7, 2013 at 10:29 PM, Felipe Gutierrez < felipe.o.gutier...@gmail.com> wrote

Re: Datanode doesn't connect to Namenode

on my slave the process is running: hduser@cloud15:/usr/local/hadoop$ jps 19025 DataNode 19092 Jps On Wed, Aug 7, 2013 at 2:26 PM, Jitendra Yadav wrote: > Hi, > > Your logs showing that the process is creating IPC call not for namenode, > it is hitting datanode itself. > > Check you please check

Re: Datanode doesn't connect to Namenode

Hi, Your logs showing that the process is creating IPC call not for namenode, it is hitting datanode itself. Check you please check you datanode processes status?. Regards Jitendra On Wed, Aug 7, 2013 at 10:29 PM, Felipe Gutierrez < felipe.o.gutier...@gmail.com> wrote: > Hi everyone, > > My sla

Re: Is there any way to use a hdfs file as a Circular buffer?

Hi Shekhar, Thank you for your replies.So far as I know, Storm is a distributed computing framework, but what we need is a storage system, high throughput and concurrency is matters.We have thousands of devices, each device will produce a steady stream of brinary data. The space for every devic

Datanode doesn't connect to Namenode

Hi everyone, My slave machine (cloud15) the datanode shows this log. It doesn't connect to the master (cloud6). 2013-08-07 13:44:03,110 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: cloud15/192.168.188.15:54310. Already tried 9 time(s); retry policy is RetryUpToMaximumCountWithFi

Re: whitelist feature of YARN

Go ahead - it's very close, I'll probably create an rc2 in the next couple of days. On Aug 7, 2013, at 2:06 AM, Krishna Kishore Bonagiri wrote: > Sandy, > Thanks again. I found RC1 for 2.1.0-beta available at > http://people.apache.org/~acmurthy/hadoop-2.1.0-beta-rc1/ >Would this have

Re: Is there any way to use a hdfs file as a Circular buffer?

Use CEP tool like Esper and Storm, you will be able to achieve that ...I can give you more inputs if you can provide me more details of what you are trying to achieve Regards, Som Shekhar Sharma +91-8197243810 On Wed, Aug 7, 2013 at 9:58 PM, Wukang Lin wrote: > Hi Niels and Bertrand, > Than

Re: Is there any way to use a hdfs file as a Circular buffer?

Hi Niels and Bertrand, Thank you for you great advices. In our scenario, we need to store a steady stream of binary data into a circular storage,throughput and concurrency are the most important indicators.The first way seems work, but as hdfs is not friendly for small files, this approche

Re: Extra start-up overhead with hadoop-2.1.0-beta

Hi Omkar, Can you please see if you can answer my question with this info or if you need anything else from me? Also, does resource localization improve or impact any performance? Thanks, Kishore On Thu, Aug 1, 2013 at 11:20 PM, Omkar Joshi wrote: > How are you making these measurements ca

Re: Oozie ssh action error

Hi, I hope below points might help you. *Approach 1#* You need to change the sshd_config file in the remote server (probably in /etc/ssh/sshd_config). Change PasswordAuthentication value. PasswordAuthentication no to PasswordAuthentication yes And then restart the SSHD daemon *Approach 2#* Ch

Oozie ssh action error

Whats the probable cause of the error when the error log of the ssh action reads: Error: permission denied (publickey password) I already have a passphrase-less ssh set. can you guys point me towards the potential reason and solution to the error. Thanks, Kasa.

RE: Compilation problem of Hadoop Projects after Import into Eclipse

Sathwik I experienced something similar a few weeks ago. I reported a JIRA on the documentation of this, please comment there https://issues.apache.org/jira/browse/HADOOP-9771 Regards ./g From: Sathwik B P [mailto:sath...@apache.org] Sent: Tuesday, August 06, 2013 4:46 AM To:

Re: Large-scale collection of logs from multiple Hadoop nodes

Hi, the approach with Flume is the most reliable workflow for, since Flume has a builtin Syslog source as well a loadbalancing channel. On top you can define multiple channels for different sources. Best, Alex sent via my mobile device mapredit.blogspot.com @mapredit > On Aug 7, 2013, at 1

Re: Large-scale collection of logs from multiple Hadoop nodes

We have the same scenario as you described. The following is our solution, just FYI: We installed a local scribe agent on every node of our cluster, and we have several central scribe servers. We extended log4j to support writing logs to the local scribe agent, and the local scribe agents forw

MutableCounterLong metrics display in ganglia

I use hadoop-2.0.5 and config hadoop-metrics2.properties file with below content. *.sink.ganglia.class=org. apache.hadoop.metrics2.sink.ganglia.GangliaSink31 *.sink.ganglia.period=10 *.sink.ganglia.supportsparse=true namenode.sink.ganglia.servers=10.232.98.74:8649 datanode.sink.ganglia.servers=10.2

Re: whitelist feature of YARN

Sandy, Thanks again. I found RC1 for 2.1.0-beta available at http://people.apache.org/~acmurthy/hadoop-2.1.0-beta-rc1/ Would this have the fix for YARN-521? and, can I use that? -Kishore On Wed, Aug 7, 2013 at 12:35 PM, Sandy Ryza wrote: > Responses inline: > > > On Tue, Aug 6, 2013 at 11

Re: Namenode is failing with expception to join

Hi, Did you configured your Name Node to store multiple copies of its metadata?. You can recover your name node in that situation. #hadoop namenode -recover it will ask you whether you want to continue or not, Please follow the instructions. Thanks On Wed, Aug 7, 2013 at 1:44 PM, Manish Bhoge

Re: Namenode is failing with expception to join

I am not using HA here. All I am trying here is to make a 2 node cluster. But before that i wanted to make sure that i am setting up everything right and make the HDFS up on Pseudo distributed mode. However, I am suspecting a mistake in my /etc/hosts file. As, I have rename the local host to myh

Re: Namenode is failing with expception to join

Manish, you stop HDFS then start HDFS on the standby name node right? please looked at https://issues.apache.org/jira/browse/HDFS-5058 there are two solutions: 1) start HDFS on the active name node, nor SBN 2) copy {namenode.name.dir}/* to the SBN I advice #1. On Wed, Aug 7, 2013 at 3:00 PM

Re: whitelist feature of YARN

Responses inline: On Tue, Aug 6, 2013 at 11:55 PM, Krishna Kishore Bonagiri < write2kish...@gmail.com> wrote: > Hi Sandy, > > Thanks for the reply and it is good to know YARN-521 is done! Please > answer my following questions > > 1) when is 2.1.0-beta going to be released? is it soon or do yo

Namenode is failing with expception to join

I have all configuration fine. But whenever i start namenode it fails with a below exception. No clue where to fix this? "2013-08-07 02:56:22,754 FATAL org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join" 2013-08-07 02:56:22,751 INFO org.apache.hadoop.hdfs.server.namenod