Re: Hadoop property precedence

What Shumin told is correct,hadoop configurations has been over written through client application. We have faced similar type of issue,Where default replication factor was mentioned 2 in hadoop configuration.But when when ever the client application writes a files,it was having 3 copies in

Re: Hadoop property precedence

Replication, block size, etc. are all per-file and pure client supplied properties. They either take their default from the client config, or directly from an API argument override. On Sun, Jul 14, 2013 at 4:14 PM, varun kumar varun@gmail.com wrote: What Shumin told is correct,hadoop

Re: Hadoop property precedence

Check out how does the writing happens on HDFS... When client issues the command hadoop fs -put local_source hdfs_destination, client contact the namenode that he wants to write and then NameNode creates a blockID and ask three datanodes ( if replication on the client side is set to 3) to host

Map slots and Reduce slots

Does the properties mapred.map.max.tasks=3 and mapred.reduce.max.tasks=4 means that machine has 3 map slots and 4 reduce slots? Or is there any way i can determine the number of map slots and reduce slots that i can allocate for a machine? Let's say on a machine if i have 8GB RAM and dual core

Apache Hadoop 1.1.2 rpm installation issue

Hello Apache Hadoop community, I've just tried to install latest stable Apache Hadoop rpm ( hadoop-1.1.2-1.x86_64.rpm ) on Fedora 19 and it fails with: [root@laptop hadoop]# rpm -i /home/sslavic/Downloads/hadoop-1.1.2-1.x86_64.rpm file /usr/bin from install of hadoop-1.1.2-1.x86_64 conflicts

Running example programs on YARN

Hello, I have a Hadoop 2.0.5 Alpha cluster. In the etc/hadoop/mapred-site.xml, I specify nothing more than that the cluster adopts the YARN architecture so that the cluster is a pure YARN cluster. I also read the source code of hadoop-mapreduce-examples-2.0.5-alpha.jar. They do not have

Re: Running example programs on YARN

Yes, check the code base, there's a thing called MRAppMaster for all MR programs and YARNRunner to enable current JobClient to run on YARN. Jian On Sun, Jul 14, 2013 at 12:55 PM, Chui-Hui Chiu cch...@tigers.lsu.eduwrote: Hello, I have a Hadoop 2.0.5 Alpha cluster. In the

RE: Failed to run wordcount on YARN

Hi Devaraj Thanks a lot for the explanation in detail. Best Regards, Raymond Liu -Original Message- From: Devaraj k [mailto:devara...@huawei.com] Sent: Friday, July 12, 2013 4:24 PM To: user@hadoop.apache.org Subject: RE: Failed to run wordcount on YARN Hi Raymond, In Hadoop

RE: Map slots and Reduce slots

Hi Shekar, I assume you are trying with Hadoop-1. There are no properties with the names 'mapred.map.max.tasks' and 'mapred.reduce.max.tasks'. We have these configuration to control the max no of map/reduce tasks run simultaneously. mapred.tasktracker.map.tasks.maximum - The maximum number

RE: Map slots and Reduce slots

These configurations cannot be changed dynamically. We need to configure these values for Tasks Tracker's before starting and cannot be changed after that. If we want to change these then TT's need to be restarted. You can configure the cluster based on the resources available. You can tune

Re: Running example programs on YARN

The MR (MR2) application would reside in the project/jar called hadoop-mapreduce-client-app-*.jar. In a tarball, this would be at $HADOOP_PREFIX/share/hadoop/mapreduce/ On Mon, Jul 15, 2013 at 1:57 AM, Jian He j...@hortonworks.com wrote: Yes, check the code base, there's a thing called