Re: Change Map-Red Parameters

Sreekanth Ramakrishnan wrote: Hi Adarsh, I am still not clear about your persistant change requirement. The job client looks at "mapred-site.xml" which is present in classpath and then submits the job. Now if you have a string of jobs which you have to submit: Series-1 of job which requires

Re: decommissioning node woes

Any help? On Wed, Mar 16, 2011 at 9:36 PM, Rita wrote: > Hello, > > I have been struggling with decommissioning data nodes. I have a 50+ data > node cluster (no MR) with each server holding about 2TB of storage. I split > the nodes into 2 racks. > > > I edit the 'exclude' file and then do a -r

Read/Write xml file in Hadoop

Dear all, I am researching on howto read/write an xml file through C++ program in Hadoop Pipes. I need to achieve this as this is the requirement. Please guide if there is a trick to do this. Thanks & best Regards, Adarsh

RE: Could not obtain block

> Can you check the DN logs for "exceeds the limit of concurrent > xcievers"? You may need to bump the dfs.datanode.max.xcievers > parameter in hdfs-site.xml, and also possibly the nfiles ulimit. Thanks Todd, and sorry for the late reply - I missed this message. I didn't see any xciever messages

Re: how to build kmeans

These java files are full of HTML. Are you sure that they are supposed to compile? How did you get these files? On Fri, Mar 18, 2011 at 3:12 AM, MANISH SINGLA wrote: > Hii everyone... > > I am trying to run kmeans on a single node... I have the attached > files with me...I put them in a folder

Re: how to build kmeans

This looks like you took the code from http://code.google.com/p/kmeans-hadoop/ And it looks like you didn't actually download the code, but you cut and pasted the HTML rendition of the code. First, this code is not from a serious project. It is more of a

Re: decommissioning node woes

If nobody else more qualified is willing to jump in, I can at least provide some pointers. What you describe is a bit surprising. I have zero experience with any 0.21 version, but decommissioning was working well in much older versions, so this would be a surprising regression. The observations

Re: decommissioning node woes

Just a note. If you just shut the node off, the blocks will replicate faster. James. On 2011-03-18, at 10:03 AM, Ted Dunning wrote: > If nobody else more qualified is willing to jump in, I can at least provide > some pointers. > > What you describe is a bit surprising. I have zero experience

Re: decommissioning node woes

Unless the last copy is on that node. Decommissioning is the only safe way to shut off 10 nodes at once. Doing them one at a time and waiting for replication to (asymptotically) recover is painful and error prone. On Fri, Mar 18, 2011 at 9:08 AM, James Seigel wrote: > Just a note. If you just

Re: Writable Class with an Array

Maha and I went off-line and resolved this issue, which was just a Java programming error. Cheers, --Matt On Mar 17, 2011, at 4:41 PM, maha wrote: Thanks for your time Matt. Below are my clarifications ... > There are three things missing from your simplified code needed to give a > definiti

RE: decommissioning node woes

Uhmm... If you use the default bandwidth allocation and you have a lot of data on the node you want to decommission you can be waiting for weeks before you can safely take the node out. If you wanted to, you can take the nodes down one by one where you do an fsck in between the removal of node

Kill jobs take awhile to clear off job tracker

Why does it take a long time (fractions of an hour or more sometimes) for kill jobs to disappear from the running jobs list of the job tracker? Clicking on such jobs will often reveal that the tasks themselves are in fact killed, yet the job remains on the front page under running jobs. My con

Re: decommissioning node woes

I like to keep that rather high. If I am decommissioning nodes, I generally want them out of the cluster NOW. That is probably a personality defect on my part. On Fri, Mar 18, 2011 at 9:59 AM, Michael Segel wrote: > Once you see those blocks successfully replicated... you can take down the > ne

Re: decommissioning node woes

I agree. J On 2011-03-18, at 11:34 AM, Ted Dunning wrote: > I like to keep that rather high. If I am decommissioning nodes, I generally > want them out of the cluster NOW. > > That is probably a personality defect on my part. > > On Fri, Mar 18, 2011 at 9:59 AM, Michael Segel > wrote: > >>

Re: decommissioning node woes

On 18/03/11 17:34, Ted Dunning wrote: I like to keep that rather high. If I am decommissioning nodes, I generally want them out of the cluster NOW. Depends on your backbone B/W I guess. And how well the switches really work vs claim to work. One thought here, does the decommissioning give

modification to Hadoop Jobtracker

Hi, For my research project, I need to modify Hadoop JobTracker to collect some statistics of TaskTracker nodes. For example, I would like to piggy-back heartbeat messages sent from TaskTrackers to JobTracker with some extra information related to the resource utilization and other statistics. I am

Public Talks from Yahoo! and LinkedIn in Bristol, England, Friday Mar 25

This isn't relevant for people who don't live in or near South England or Wales, but for those that do, I'm pleased to announce that Owen O'Malley and Sanjay Radia of Yahoo! and Jakob Homan of LinkedIn will all be giving public talks on Hadoop on Friday March 25 at HP Laboratories, in Bristol

Re: modification to Hadoop Jobtracker

Hi, bikash I also worked on this several months ago, If I don't have a wrong memory, the involved files should be: JobTracker.java, TaskTracker.java, the most interesting functions should be JobTracker.heartbeat(), and TaskTracker.transmitHeartbeat(), I think you will also need to add more membe

MapReduce compilation error

Hi, When I am compiling MapReduce source code after checking-in Eclipse, I am getting the following error: The declared package "" does not match the expected package "testjar" ClassWithNoPackage.java Hadoop-MR/src/test/mapred/testjar Any thoughts? Thanks, Bikash

RE: Could not obtain block

My bad! I have done a testrun with Kerberos on the cluster (which worked relatively well...). I was under the impression that just configuring the cluster to NOT use security would revert the process. But it turns out the HDFS deamons were still started using the SecureStarters. This seemed to

running local hadoop job in windows

Hi, guys, I want to give my users a sense of what my hadoop application can do, and I am trying to make it run in Windows, with this command java -jar dist\FreeEed.jar This command runs my hadoop job locally, and it works in Linux. However, in Windows I get the error listed below. Since I am run

Re: running local hadoop job in windows

presumably you ran this under cygwin? 2011/3/18 Mark Kerzner > Hi, guys, > > I want to give my users a sense of what my hadoop application can do, and I > am trying to make it run in Windows, with this command > > java -jar dist\FreeEed.jar > > This command runs my hadoop job locally, and it wor

Re: running local hadoop job in windows

No, I hoped that it is not absolutely necessary for that kind of use. I am not even issuing the "hadoop -jar" command, but it is pure "java -jar". It is true though that my Ubuntu has a Hadoop set up, so maybe it is doing a lot of magic behind my back. I did not want to have my inexperienced Windo

WritableName can't load class ... for custom WritableClasses

Hi, The following was working fine with Hadoop Writables. Now, I'm using my custom Writable class called "TermFreqArrayWritable" to produce a Sequence File with key=LongWritable and value=TermFreqArrayWritable. However, when I try to read the produced Sequence File using its Reader, I get th

RE: decommissioning node woes

Well... When you look at the default value... and compare it to DNs having 7+TB of disk space... The math doesn't look good. If you have 1GBe and a good ToR from Cisco, Blade Networks (now IBM), and a couple of others... they can do it. I had a conversation with a switch provider and he indi