Re: HADOOP_SSH_OPTS

On Apr 23, 2010, at 4:01 PM, Hazem Mahmoud wrote: > I have a test setup where (due to the environment I'm testing on) every > system is listening on a different SSH port. From what I can tell, I can use > HADOOP_SSH_OPTS in hadoop-env.sh to specify different SSH options (ie: > specify a differ

HADOOP_SSH_OPTS

I have a test setup where (due to the environment I'm testing on) every system is listening on a different SSH port. From what I can tell, I can use HADOOP_SSH_OPTS in hadoop-env.sh to specify different SSH options (ie: specify a different port to connect to). However, in my case, the grid nodes

Re: Decomishining a node

On Apr 23, 2010, at 2:50 PM, Alex Kozlov wrote: > The best way to resolve an argument is to look at the code: I didn't realize we were having an argument. But I will say this: I've never had a node removed from both dfs.hosts and dfs.hosts.exclude actually disappear from the dead list in the

Re: Decomishining a node

The best way to resolve an argument is to look at the code: */** * Rereads the config to get hosts and exclude list file names. * Rereads the files to update the hosts and exclude lists. It * checks if any of the hosts have changed states: * 1. Added to hosts --> no further work nee

Re: Dynamically determining number of reducers

Thanks! On Fri, Apr 23, 2010 at 1:22 PM, Hong Tang wrote: > JobClient.getClusterStatus().getMaxReduceTasks(). > > > On Apr 23, 2010, at 10:34 AM, Farhan Husain wrote: > > I actually wanted to mean number of tasktrackers. I want to set the number >> of reducers equal to the number of tasktracker

Re: Decomishining a node

On Apr 23, 2010, at 1:56 PM, Alex Kozlov wrote: > I think Raymond says that the machine is already dead... Right. But he wants to re-add it later. So dfs.exclude is still a better way to go. dfs.hosts, iirc, doesn't get re-read so it would require a nn bounce to clear.

Re: Decomishining a node

I think Raymond says that the machine is already dead... At this point, you can just remove it from dfs.hosts list and let HDFS to restore the lost blocks... But before that, if you have the disks intact, you can stop HDFS and manually copy the blocks together with their CRC from the dead machine

Error with distcp: hdfs to S3 bulk transfer

The following error is thrown when distcp ing data from hdfs (hadoop 15.5) to S3 storage. This problem is creeping in after actually applying couple of bug fixes in hadoop 15.5 that were resolved in the later versions. Any thoughts would be greatly helpful. With failures, global counters are in

Re: Decomishining a node

On Apr 23, 2010, at 10:48 AM, Raymond Jennings III wrote: > I've got a dead machine on my cluster. I want to safely update HDFS so that > nothing references this machine then I want to rebuild it and put it back in > service in the cluster. > > Does anyone have any pointers how to do this (th

Re: Dynamically determining number of reducers

JobClient.getClusterStatus().getMaxReduceTasks(). On Apr 23, 2010, at 10:34 AM, Farhan Husain wrote: I actually wanted to mean number of tasktrackers. I want to set the number of reducers equal to the number of tasktrackers present in the cluster and I want to determine the number of tasktr

Decomishining a node

I've got a dead machine on my cluster. I want to safely update HDFS so that nothing references this machine then I want to rebuild it and put it back in service in the cluster. Does anyone have any pointers how to do this (the first part - updating HDFS so that it's no longer referenced.) Tha

Re: Dynamically determining number of reducers

I actually wanted to mean number of tasktrackers. I want to set the number of reducers equal to the number of tasktrackers present in the cluster and I want to determine the number of tasktrackers dynamically. Thanks, Farhan On Fri, Apr 23, 2010 at 12:03 PM, Farhan Husain wrote: > Hello, > > Is

Dynamically determining number of reducers

Hello, Is there any way to determine the number of reducers present in the cluster dynamically? I need to determine it when the job parameters are set up. Thanks, Farhan

Re: Using external library in MapReduce jobs

Hello Mike, I completely agree with you. I think bundling the libraries in the job jar file is the correct way to go. Thanks, Farhan On Thu, Apr 22, 2010 at 9:12 PM, Michael Segel wrote: > > > > > Date: Thu, 22 Apr 2010 17:30:13 -0700 > > Subject: Re: Using external library in MapReduce jobs >

Re: Hadoop performance - xfs and ext4

I've done some research and following mount option sound like optimal , will you interested to give it a try? noatime,data=writeback,barrier=0,nobh On Fri, Apr 23, 2010 at 10:43 PM, Todd Lipcon wrote: > Hi Stephen, > > Can you try mounting ext4 with the nodelalloc option? I've seen the same > im

Re: Try to mount HDFS

Hi Brian, The error is found and the filesystem works now! The error logs of "hadoop fs -put" helped a lot. I tried to copy a small file: # ls -l /tmp/neue_datei.txt -rw-r--r-- 1 root root 5 2010-04-23 14:08 /tmp/neue_datei.txt # hadoop fs -put /tmp/neue_datei.txt /hdfs/ 10/04/23 14:09:02 W

Re: Hadoop performance - xfs and ext4

Hi Stephen, Can you try mounting ext4 with the nodelalloc option? I've seen the same improvement due to delayed allocation butbeen a little nervous about that option (especially in the NN where we currently follow what the kernel people call an antipattern for image rotation). -Todd On Fri, Apr

Host name problem in Hadoop GUI

Having an issue with host names on my new Hadoop cluster. The cluster is currently 1 name node and 2 data nodes, running in a cloud vendor data center. All is well with general operations of the cluster - i.e., name node and data nodes can talk just fine, I can read/write to/from the HDFS, ya

Re: Try to mount HDFS

Hey Christian, Glad to hear things are beginning to click. Can you upload the things you learned into the wiki? In our internal user docs, we have big bold letters saying to watch out for this issue. As far as your writing issues - can you write using "hadoop fs -put"? The nice thing about

Re: Try to mount HDFS

Brian, You got it!!! :-) It works (partly)! i switched to Port 9000. core-site.xml includes now: fs.default.name hdfs://ec2-75-101-210-65.compute-1.amazonaws.com:9000 true $ hadoop fs -ls / Found 1 items drwxr-xr-x - hadoop s

Re: Hadoop performance - xfs and ext4

Steve Loughran wrote: That's really interesting. Do you want to update the bits of the Hadoop wiki that talks about filesystems? I can if people think that would be useful. I'm not sure if my results are neccesarily going to reflect what will happen on other peoples systems and configs though

Re: Hadoop performance - xfs and ext4

Andrew Klochkov wrote: Hi, Just curious - did you try ext3? Can it be faster then ext4? Hadoop wiki suggests ext3 as it's used mostly for hadoop clusters: http://wiki.apache.org/hadoop/DiskSetup For completeness, I rebuilt one more time with ext3 mkfs.ext3 -T largefile4 DEV (mounted with noa

Hadoop Log Collection

Hey everyone, i deal with hadoop since a few weeks to build up a cluster with hdfs. I was looking for several Monitoring tools to observe my cluster and find a good solution with ganglia+nagios. To complete the monitoring part of the cluster, i am looking for an Log collection tool, which store th

data node stops on slave

Hi I am following tutorial running hadoop on ubuntu linux (multinode cluster) * http://www.michael-noll.com/wiki/Running_Hadoop_On_Ubuntu_Linux_(Multi-Node_Cluster) * for configuring 2 node cluster but i am

Re: Try to mount HDFS

Hm, ok, now you have me stumped. One last hunch - can you include the port information, but also switch to port 9000? Additionally, can you do the following: 1) In /var/log/messages and copy out the hdfs/fuse-related messages and post them 2) Using the hadoop clients do, hadoop fs -ls / Brian

failed or killed...

Just wondering about the exact difference between a task declared "failed" and a task declared "killed" Because of a typo, we had a node have WAY more maps it could ever handle., it ran out of memory and got almost totally unresponsive. Some tasks failed, some were killed and I've been wondering ab