RE: HADOOP-7178 patch is not present in Hadoop-1.0.3

Hi Uma, Yes I checked the api's of Hadoop 0.23.0 which have this patch. But I am not able to find below api in Hadoop-1.0.3 (latest stable) API : public void copyToLocalFile(boolean delSrc, Path src, Path dst, boolean useRawLocalFileSystem) Is this patch not committed in Hadoop-1.0.3 Regards

Re: Integrating hadoop with java UI application deployed on tomcat

Hi, If you are getting the LocalFileSystem, you could try by putting core-site.xml in a directory that's there in the classpath for the Tomcat App (or include such a path in the classpath, if that's possible) Thanks hemanth On Mon, Sep 3, 2012 at 4:01 PM, Visioner Sadak visioner.sa...@gmail.com

custom format

hello user, I am trying to create a map reduce program which will have splits based on a specific length. The content has to be extracted in a way such that the newline(\n) or tab(\t) etc characters will be considered as a byte and not as a mapper instance. is this possible through custom input?

Yarn defaults for local directories

Hi, Is there a reason why Yarn's directory paths are not defaulting to be relative to hadoop.tmp.dir. For e.g. yarn.nodemanager.local-dirs defaults to /tmp/nm-local-dir. Could it be ${hadoop.tmp.dir}/nm-local-dir instead ? Similarly for the log directories, I guess... Thanks hemanth

Re: custom format

Hi Hemanth, Thank you for your valuable reply. Regards, Anit On Mon, Sep 3, 2012 at 4:57 PM, Hemanth Yamijala yhema...@gmail.com wrote: Hi, I found this while trying to see if such a FileFormat or Split already exists:

Re: Integrating hadoop with java UI application deployed on tomcat

Try using hadoopConf.addResource(*new Path(F:/hadoop-0.22.0/conf/core-site.xml)*); instead of hadoopConf.addResource(F:/hadoop-0.22.0/conf/core-site.xml); or you should add your core-site.xml to a location which is in your class path(WEB-INF\classes or WEB-INF\lib in case of a web application)

Re: Integrating hadoop with java UI application deployed on tomcat

Thanks Senthil i tried on trying with new path getting this error do i have to do any ssl setting on tomcat as well * java.io.IOException*: Call to localhost/127.0.0.1:9000 failed on local exception: *java.io.IOException*: An established connection was aborted by the software in your host

knowing the nodes on which reduce tasks will run

Hello, How can one get to know the nodes on which reduce tasks will run? One of my job is running and it's completing all the map tasks. My map tasks write lots of intermediate data. The intermediate directory is getting full on all the nodes. If the reduce task take any node from cluster then

reading a binary file

Hello, I have a binary file of integers and I would like an input format that generates pairs key,value, where value is an integer in the file and key the position of the integer in the file. Which class should I use? (i.e. I'm looking for a kind of TextinputFormat for binary files) Thank you

Re: reading a binary file

Hi Mohammad, SequenceFileInputFormathttp://hadoop.apache.org/mapreduce/docs/r0.21.0/api/org/apache/hadoop/mapreduce/lib/input/SequenceFileInputFormat.html requires the file to be a sequence of key/value stored in binary (i.e., the key is stored in the file). In my case, the key is implicitly

Re: reading a binary file

Hi Francesco TextInputFormat reads line by line based on '\n' by default, there the key values is the position offset and the line contents respectively. But in your case it is just a sequence of integers and also it is Binary. Also you require the offset for each integer value and not offset by

Re: knowing the nodes on which reduce tasks will run

HI Abhay The TaskTrackers on which the reduce tasks are triggered is chosen in random based on the reduce slot availability. So if you don't need the reduce tasks to be scheduled on some particular nodes you need to set 'mapred.tasktracker.reduce.tasks.maximum' on those nodes to 0. The bottleneck

Re: knowing the nodes on which reduce tasks will run

Hi, The reducer is run where there is slot available, the location is not related to where the data is located and it is not possible to choose where the reducer will run (except by tweaking the tasktracker...). Regards Bertrand On Mon, Sep 3, 2012 at 4:19 PM, Abhay Ratnaparkhi

Re: knowing the nodes on which reduce tasks will run

How can I set 'mapred.tasktracker.reduce.tasks.maximum' to 0 in a running tasktracker? Seems that I need to restart the tasktracker and in that case I'll loose the output of map tasks by particular tasktracker. Can I change 'mapred.tasktracker.reduce.tasks.maximum' to 0 without restarting

Re: knowing the nodes on which reduce tasks will run

Hi Abhay You need this value to be changed before you submit your job and restart TT. Modifying this value in mid time won't affect the running jobs. On Mon, Sep 3, 2012 at 9:06 PM, Abhay Ratnaparkhi abhay.ratnapar...@gmail.com wrote: How can I set 'mapred.tasktracker.reduce.tasks.maximum'

Re: knowing the nodes on which reduce tasks will run

Hi, You are right that a change to mapred.tasktracker.reduce.tasks.maximum will require a restart of the tasktrackers. AFAIK, there is no way of modifying this property without restarting. On a different note, could you see if the amount of intermediate data can be reduced using a combiner, or

Error using hadoop in non-distributed mode

Using hadoop with mahout in a local filesystem/non-hdfs config for debugging purposes inside Intellij IDEA. When I run one particular part of the analysis I get the error below. I didn't write the code but we are looking for some hint about what might cause it. This job completes without error

Re: knowing the nodes on which reduce tasks will run

The short answer is no. The longer answer is that you can attempt to force data locality, however even then if an open slot becomes available, its used regardless of what you want to do... On Sep 3, 2012, at 9:19 AM, Abhay Ratnaparkhi abhay.ratnapar...@gmail.com wrote: Hello, How can

Re: knowing the nodes on which reduce tasks will run

All of my map tasks are about to complete and there is not much processing to be done in reducer. The job is running from a week so I don't want the job to fail. Any other suggestion to tackle this is welcome. ~Abhay On Mon, Sep 3, 2012 at 9:26 PM, Hemanth Yamijala

how to execute different tasks on data nodes(simultaneously in hadoop).

genrally in hadoop map function will be exeucted by all the data nodes on the input data set ,against this how can i do the following. i have some filter programs , and what i want to do is each data node(slave) has to execute one filter alogrithm simultaneously, diffent from other data nodes

Re: how to execute different tasks on data nodes(simultaneously in hadoop).

Not sure what you are trying to do... You want to pass through the entire data set on all nodes where each node runs a single filter? You're thinking is orthogonal to how Hadoop works. You would be better off letting each node work on a portion of the data which is local to that node

Re: how to execute different tasks on data nodes(simultaneously in hadoop).

You can check the value of map.input.file in order to apply a different logic for each type of files (in the mapper). More information about your problem/context would help the readers to provide a more extensive reply. Regards Bertrand On Mon, Sep 3, 2012 at 6:25 PM, Michael Segel

Re: Integrating hadoop with java UI application deployed on tomcat

The error says call to 127.0.0.1:9000 fails. It is failing when it tries to contact the namenode (9000 is the default namenode port) configured in core-site.xml. You should also check whether the namenode is configured correctly and also whether the namenode is up. On Mon, Sep 3, 2012 at 7:43

Re: how to execute different tasks on data nodes(simultaneously in hadoop).

Hi, Assuming you have to compute these value for every RGB pixel. Why couldn't you compute all these values at the same time on the same node? Hadoop let you distribute your computation but it doesn't mean each node has to compute only a part of the equations. Each node can compute all equations

Re: Integrating hadoop with java UI application deployed on tomcat

Thanks senthil name node is up and running and in core-site.xml i have configuration property namefs.default.name/name valuehdfs://localhost:9000/value /property /configuration shud i change my ip or any other config?? On Mon, Sep 3, 2012 at 10:11 PM, Senthil Kumar

Re: Hadoop examples don't compile in Eclipse

Hi Mark, Make sure that you enable java 1.6 for your project. More info is available here: http://stackoverflow.com/questions/987973/why-does-eclipse-complain-about-override-on-interface-methods stan On Mon, Sep 3, 2012 at 10:13 PM, Mark Kerzner mark.kerz...@shmsoft.com wrote: Hi, this is

RE: Integrating hadoop with java UI application deployed on tomcat

Hi, Try adding the hadoop/conf directory in the TOMCAT's classpath Ex : CLASSPATH=/usr/local/Apps/hbase-0.90.4/conf:/usr/local/Apps/hadoop-0.20.203.0/conf: Regards, Shobha M From: Visioner Sadak [mailto:visioner.sa...@gmail.com] Sent: 03 September 2012 PM 04:01 To: user@hadoop.apache.org