Re: Dissecting MR output article

[Harsh J]

+1 for Hadoop: The Definitive Guide and other books.

Sidenote: The 3rd Edition of Tom White's Hadoop: The
Definitive Guide does have good details on MRv2 and YARN.

On Sat, Mar 23, 2013 at 11:22 AM, Azuryy Yu  wrote:
> hadoop definition guide.pdf should be helpful. there is a chapter for this.
> but only for MRv1.
>
> On Mar 23, 2013 1:50 PM, "Sai Sai"  wrote:
>>
>>
>> Just wondering if there is any step by step explaination/article of MR
>> output we get when we run a job either in eclipse or ubuntu.
>> Any help is appreciated.
>> Thanks
>> Sai



--
Harsh J

Re: Dissecting MR output article

[Azuryy Yu]

hadoop definition guide.pdf should be helpful. there is a chapter for this.
but only for MRv1.
 On Mar 23, 2013 1:50 PM, "Sai Sai"  wrote:

>
> Just wondering if there is any step by step explaination/article of MR
> output we get when we run a job either in eclipse or ubuntu.
> Any help is appreciated.
> Thanks
> Sai
>

Re: Dissecting MR output article

[Sai Sai]

Just wondering if there is any step by step explaination/article of MR output 
we get when we run a job either in eclipse or ubuntu.Any help is appreciated.
Thanks
Sai

question for commetter

[Azuryy Yu]

is there a way to separate hdfs2 from hadoop2? I want use hdfs2 and
mapreduce1.0.4, exclude yarn. because I need HDFS-HA.

Re: Setup/Cleanup question

[Sai Sai]

Thanks Harsh.
So the setup/cleanup r for the Job and not the Mappers i take it.
Thanks.






 From: Harsh J 
To: "" ; Sai Sai 
 
Sent: Friday, 22 March 2013 10:05 PM
Subject: Re: Setup/Cleanup question
 
Assuming you speak of MRv1 (1.x/0.20.x versions), there is just 1 Job
Setup and 1 Job Cleanup tasks additionally run for each Job.

On Sat, Mar 23, 2013 at 9:10 AM, Sai Sai  wrote:
> When running an MR job/program assuming there r 'n' (=100) Mappers triggered
> then my question is will the setup & cleanup run n number of times which
> means once for each mapper or for all the mappers they will run only once.
> Any help is appreciated.
> Thanks
> Sai



-- 
Harsh J

Re: Cluster lost IP addresses

[Azuryy Yu]

oh,yes,it's not persist, only in memory. so there is no issue.
On Mar 23, 2013 1:13 PM, "Harsh J"  wrote:

> NameNode does not persist block locations; so this is still
> recoverable if the configs are changed to use the new set of hostnames
> to bind to/look up.
>
> On Sat, Mar 23, 2013 at 9:01 AM, Azuryy Yu  wrote:
> > it has issues, namenode save blockid->nodes, using ip addr if your slaves
> > config file using ip addr instead of hostname.
> >
> > On Mar 23, 2013 10:14 AM, "Balaji Narayanan (பாலாஜி நாராயணன்)"
> >  wrote:
> >>
> >> Assuming you are using hostnAmes and not ip address in your config files
> >> What happens when you start the cluster? If you are using IP address in
> your
> >> configs just update them and start. It should work with no issues.
> >>
> >> On Friday, March 22, 2013, John Meza wrote:
> >>>
> >>> I have a 18 node cluster that had to be physically moved.
> >>> Unfortunately all the ip addresses were lost (recreated).
> >>>
> >>> This must have happened to someone before.
> >>> Nothing else on the machines has been changed. Most importantly the
> data
> >>> in HDFS is still sitting there.
> >>>
> >>> Is there a way to recover this cluster to a useable state?
> >>> thanks
> >>> John
> >>
> >>
> >>
> >> --
> >> http://balajin.net/blog
> >> http://flic.kr/balajijegan
>
>
>
> --
> Harsh J
>

Re: Cluster lost IP addresses

[Harsh J]

Hi Chris,

Where exactly are you seeing issues with change of NN/DN IPs? I've
never encountered trouble on IP changes (I keep moving across networks
everyday and the HDFS plus MR I run both stand tall without requiring
a restart). We do not store (generally) nor rely on IP addresses. An
exclusion may apply to files under construction I think, but a
properly shutdown cluster pre-move would not have that and such files
wouldn't matter too much in such a scenario anyway.

Obviously, a hostname change could cause issues.

FWIW, you can easily take any person's fsimage from across the world
and start your NN on top of that and add in new DNs with the block
data under them and setup the HDFS cluster. This is rather painless
and well-built and goes to show that its not really IP dependent in
any way. Please do elaborate.

On Sat, Mar 23, 2013 at 10:52 AM, Chris Embree  wrote:
> Hey John,
>
> Make sure your /etc/hosts ( or DNS) is up to date and  any topology scripts
> are updated.  Unfortunately, NN is pretty dumb about IP's vs. Hostnames.
>
> BTW, NN devs.  Seriously?  You rely on IP addr instead of hostname?  Someone
> should probably be shot or at least be responsible for fixing this
> abomination. Sad that this code was released GA.
>
> Sorry folks.  HDFS/Mapred is really cool tech, I'm just jaded about this
> kind of silliness.
>
> In my Not So Humble Opinion.
> Chris
>
>
> On Sat, Mar 23, 2013 at 1:12 AM, Harsh J  wrote:
>>
>> NameNode does not persist block locations; so this is still
>> recoverable if the configs are changed to use the new set of hostnames
>> to bind to/look up.
>>
>> On Sat, Mar 23, 2013 at 9:01 AM, Azuryy Yu  wrote:
>> > it has issues, namenode save blockid->nodes, using ip addr if your
>> > slaves
>> > config file using ip addr instead of hostname.
>> >
>> > On Mar 23, 2013 10:14 AM, "Balaji Narayanan (பாலாஜி நாராயணன்)"
>> >  wrote:
>> >>
>> >> Assuming you are using hostnAmes and not ip address in your config
>> >> files
>> >> What happens when you start the cluster? If you are using IP address in
>> >> your
>> >> configs just update them and start. It should work with no issues.
>> >>
>> >> On Friday, March 22, 2013, John Meza wrote:
>> >>>
>> >>> I have a 18 node cluster that had to be physically moved.
>> >>> Unfortunately all the ip addresses were lost (recreated).
>> >>>
>> >>> This must have happened to someone before.
>> >>> Nothing else on the machines has been changed. Most importantly the
>> >>> data
>> >>> in HDFS is still sitting there.
>> >>>
>> >>> Is there a way to recover this cluster to a useable state?
>> >>> thanks
>> >>> John
>> >>
>> >>
>> >>
>> >> --
>> >> http://balajin.net/blog
>> >> http://flic.kr/balajijegan
>>
>>
>>
>> --
>> Harsh J
>
>



-- 
Harsh J

Re: Cluster lost IP addresses

[Chris Embree]

Hey John,

Make sure your /etc/hosts ( or DNS) is up to date and  any topology scripts
are updated.  Unfortunately, NN is pretty dumb about IP's vs. Hostnames.

BTW, NN devs.  Seriously?  You rely on IP addr instead of hostname?
 Someone should probably be shot or at least be responsible for fixing
this abomination. Sad that this code was released GA.

Sorry folks.  HDFS/Mapred is really cool tech, I'm just jaded about this
kind of silliness.

In my Not So Humble Opinion.
Chris

On Sat, Mar 23, 2013 at 1:12 AM, Harsh J  wrote:

> NameNode does not persist block locations; so this is still
> recoverable if the configs are changed to use the new set of hostnames
> to bind to/look up.
>
> On Sat, Mar 23, 2013 at 9:01 AM, Azuryy Yu  wrote:
> > it has issues, namenode save blockid->nodes, using ip addr if your slaves
> > config file using ip addr instead of hostname.
> >
> > On Mar 23, 2013 10:14 AM, "Balaji Narayanan (பாலாஜி நாராயணன்)"
> >  wrote:
> >>
> >> Assuming you are using hostnAmes and not ip address in your config files
> >> What happens when you start the cluster? If you are using IP address in
> your
> >> configs just update them and start. It should work with no issues.
> >>
> >> On Friday, March 22, 2013, John Meza wrote:
> >>>
> >>> I have a 18 node cluster that had to be physically moved.
> >>> Unfortunately all the ip addresses were lost (recreated).
> >>>
> >>> This must have happened to someone before.
> >>> Nothing else on the machines has been changed. Most importantly the
> data
> >>> in HDFS is still sitting there.
> >>>
> >>> Is there a way to recover this cluster to a useable state?
> >>> thanks
> >>> John
> >>
> >>
> >>
> >> --
> >> http://balajin.net/blog
> >> http://flic.kr/balajijegan
>
>
>
> --
> Harsh J
>

Re: how to control (or understand) the memory usage in hdfs

[Harsh J]

I run a 128 MB heap size DN for my simple purposes on my Mac and it
runs well for what load I apply on it.

A DN's primary, growing memory consumption comes from the # of blocks
it carries. All of these blocks' file paths are mapped and kept in the
RAM during its lifetime. If your DN has acquired a lot of blocks by
now, like say close to a million or more, then 1 GB may not suffice
anymore to hold them in and you'd need to scale up (add more RAM or
increase heap size if you have more RAM)/scale out (add another node
and run the balancer).

On Sat, Mar 23, 2013 at 10:03 AM, Ted  wrote:
> Hi I'm new to hadoop/hdfs and I'm just running some tests on my local
> machines in a single node setup. I'm encountering out of memory errors
> on the jvm running my data node.
>
> I'm pretty sure I can just increase the heap size to fix the errors,
> but my question is about how memory is actually used.
>
> As an example, with other things like an OS's disk-cache or say
> databases, if you have or let it use as an example 1gb of ram, it will
> "work" with what it has available, if the data is more than 1gb of ram
> it just means it'll swap in and out of memory/disk more often, i.e.
> the cached data is smaller. If you give it 8gb of ram it still
> functions the same, just performance increases.
>
> With my hdfs setup, this does not appear to be true, if I allocate it
> 1gb of heap, it doesn't just perform worst / swap data to disk more.
> It out right fails with out of memory and shuts the data node down.
>
> So my question is... how do I really tune the memory / decide how much
> memory I need to prevent shutdowns? Is 1gb just too small even on a
> single machine test environment with almost no data at all, or is it
> suppose to work like OS-disk caches were it always works but just
> performs better or worst and I just have something configured wrong?.
> Basically my objective isn't performance, it's that the server must
> not shut itself down, it can slow down but not shut off.
>
> --
> Ted.



-- 
Harsh J

Re: Cluster lost IP addresses

[Harsh J]

NameNode does not persist block locations; so this is still
recoverable if the configs are changed to use the new set of hostnames
to bind to/look up.

On Sat, Mar 23, 2013 at 9:01 AM, Azuryy Yu  wrote:
> it has issues, namenode save blockid->nodes, using ip addr if your slaves
> config file using ip addr instead of hostname.
>
> On Mar 23, 2013 10:14 AM, "Balaji Narayanan (பாலாஜி நாராயணன்)"
>  wrote:
>>
>> Assuming you are using hostnAmes and not ip address in your config files
>> What happens when you start the cluster? If you are using IP address in your
>> configs just update them and start. It should work with no issues.
>>
>> On Friday, March 22, 2013, John Meza wrote:
>>>
>>> I have a 18 node cluster that had to be physically moved.
>>> Unfortunately all the ip addresses were lost (recreated).
>>>
>>> This must have happened to someone before.
>>> Nothing else on the machines has been changed. Most importantly the data
>>> in HDFS is still sitting there.
>>>
>>> Is there a way to recover this cluster to a useable state?
>>> thanks
>>> John
>>
>>
>>
>> --
>> http://balajin.net/blog
>> http://flic.kr/balajijegan



-- 
Harsh J

Re: Setup/Cleanup question

[Harsh J]

Assuming you speak of MRv1 (1.x/0.20.x versions), there is just 1 Job
Setup and 1 Job Cleanup tasks additionally run for each Job.

On Sat, Mar 23, 2013 at 9:10 AM, Sai Sai  wrote:
> When running an MR job/program assuming there r 'n' (=100) Mappers triggered
> then my question is will the setup & cleanup run n number of times which
> means once for each mapper or for all the mappers they will run only once.
> Any help is appreciated.
> Thanks
> Sai



-- 
Harsh J

how to control (or understand) the memory usage in hdfs

[Ted]

Hi I'm new to hadoop/hdfs and I'm just running some tests on my local
machines in a single node setup. I'm encountering out of memory errors
on the jvm running my data node.

I'm pretty sure I can just increase the heap size to fix the errors,
but my question is about how memory is actually used.

As an example, with other things like an OS's disk-cache or say
databases, if you have or let it use as an example 1gb of ram, it will
"work" with what it has available, if the data is more than 1gb of ram
it just means it'll swap in and out of memory/disk more often, i.e.
the cached data is smaller. If you give it 8gb of ram it still
functions the same, just performance increases.

With my hdfs setup, this does not appear to be true, if I allocate it
1gb of heap, it doesn't just perform worst / swap data to disk more.
It out right fails with out of memory and shuts the data node down.

So my question is... how do I really tune the memory / decide how much
memory I need to prevent shutdowns? Is 1gb just too small even on a
single machine test environment with almost no data at all, or is it
suppose to work like OS-disk caches were it always works but just
performs better or worst and I just have something configured wrong?.
Basically my objective isn't performance, it's that the server must
not shut itself down, it can slow down but not shut off.

-- 
Ted.

Re: Setup/Cleanup question

[Sai Sai]

When running an MR job/program assuming there r 'n' (=100) Mapperstriggered 
then my question is will the setup & cleanup run n number of times which means 
once for each mapper or for all the mappers they will run only once. 

Any help is appreciated.
Thanks
Sai

Re: Cluster lost IP addresses

[Azuryy Yu]

it has issues, namenode save blockid->nodes, using ip addr if your slaves
config file using ip addr instead of hostname.
On Mar 23, 2013 10:14 AM, "Balaji Narayanan (பாலாஜி நாராயணன்)" <
li...@balajin.net> wrote:

> Assuming you are using hostnAmes and not ip address in your config
> files What happens when you start the cluster? If you are using IP address
> in your configs just update them and start. It should work with no issues.
>
> On Friday, March 22, 2013, John Meza wrote:
>
>> I have a 18 node cluster that had to be physically moved.
>> Unfortunately all the ip addresses were lost (recreated).
>>
>> This must have happened to someone before.
>> Nothing else on the machines has been changed. Most importantly the data
>> in HDFS is still sitting there.
>>
>> Is there a way to recover this cluster to a useable state?
>> thanks
>> John
>>
>
>
> --
> http://balajin.net/blog
> http://flic.kr/balajijegan
>

Re: Cluster lost IP addresses

[பாலாஜி நாராயணன்]

Assuming you are using hostnAmes and not ip address in your config
files What happens when you start the cluster? If you are using IP address
in your configs just update them and start. It should work with no issues.

On Friday, March 22, 2013, John Meza wrote:

> I have a 18 node cluster that had to be physically moved.
> Unfortunately all the ip addresses were lost (recreated).
>
> This must have happened to someone before.
> Nothing else on the machines has been changed. Most importantly the data
> in HDFS is still sitting there.
>
> Is there a way to recover this cluster to a useable state?
> thanks
> John
>


-- 
http://balajin.net/blog
http://flic.kr/balajijegan

Re: Capacity Scheduler question

[Harsh J]

If you're looking to set a fixed # of maps per job and also control
their parallel distributed execution (by numbers), a Scheduler cannot
solve that for you but may assist in the process.

Setting a specific # of maps in a job to match something is certainly
not a Scheduler's work, as it only deals with what task needs to go
where. For you to control your job's # of maps (i.e. input splits),
tweak your Job's InputFormat#getSplits(…). The size of array it
returns dictates the total number of maps your job ends up running.

You are further limited by the fixed task slot behavior in 0.20.x/1.x
releases which use the MR1 framework (i.e. a JobTracker and a
TaskTracker). The property "mapred.tasktracker.map.tasks.maximum"
applies to a TaskTracker and not a per-job one as it name goes, and
isn't what you'd configure to seemingly achieve what you want.

In addition to this, YARN has a slotless NodeManager, wherein you can
ask for a certain amount of resources from your job on a per-task
level and have it granted globally. Meaning, if your NodeManager got
configured to use upto 8 GB, and your job/app requests 8 GB per
task/container, then only 1 such container can at most be run at one
time on any chosen NodeManager that serves 8 GB of memory resources.
Likewise, if your demand becomes 8/18 GB per container/task, then upto
18 containers can run in parallel at most on a given NM.

This is still not rigid though (less than 18 may run at the same time
on an NM as well, depending on the scheduler's distribution of
containers across all nodes), as that isn't MapReduce's goal in the
first place. If you want more rigidity consider writing your own YARN
application that implements such a distribution goal.

On Sat, Mar 23, 2013 at 3:18 AM, jeremy p
 wrote:
> I have two jobs, Job A and Job B.  Job A needs to run with 18 mappers per
> machine, Job B needs to run with 1 mapper per machine.  Hadoop doesn't give
> you a way to specify number of mappers on a per-job basis.
> mapred.tasktracker.map.tasks.maximum and mapred.map.tasks do absolutely
> nothing.  I've been looking into the Capacity Scheduler, but I'm unsure if
> it can help me.  In this documentation, all the settings under "Resource
> Allocation" are cluster-wide.  I need to be able to set the maximum capacity
> on a given machine.  It does look like you have the option to set the
> required amount of memory per slot, but that setting applies to all the
> queues.  If I could set that value on a per-queue basis, that would be
> helpful.
>
> Will the capacity scheduler help me here?  Or am I barking up the wrong
> tree?  If the capacity scheduler won't help me, can you think of anything
> that will?
>
> Thanks!
>
> --Jeremy



-- 
Harsh J

Re: Capacity Scheduler question

[jeremy p]

Thanks for the help.  Sadly, I don't think the Fair Scheduler will help me
here.  It will let you specify the number of
concurrent task slots for a pool, but that applies to the entire cluster.
 For a given pool, I need to set the maximum number of task slots per
machine.

On Fri, Mar 22, 2013 at 3:06 PM, Serge Blazhievsky wrote:

> Take a look at fair scheduler it will do what you ask for
>
> Sent from my iPhone
>
> On Mar 22, 2013, at 2:48 PM, jeremy p 
> wrote:
>
> I have two jobs, Job A and Job B.  Job A needs to run with 18 mappers per
> machine, Job B needs to run with 1 mapper per machine.  Hadoop doesn't give
> you a way to specify number of mappers on a per-job basis.
> mapred.tasktracker.map.tasks.maximum and mapred.map.tasks do absolutely
> nothing.  I've been looking into the Capacity Scheduler, but I'm unsure if
> it can help me.  In 
> thisdocumentation,
>  all the settings under "Resource Allocation" are
> cluster-wide.  I need to be able to set the maximum capacity on a given
> machine.  It does look like you have the option to set the required amount
> of memory per slot, but that setting applies to all the queues.  If I could
> set that value on a per-queue basis, that would be helpful.
>
> Will the capacity scheduler help me here?  Or am I barking up the wrong
> tree?  If the capacity scheduler won't help me, can you think of anything
> that will?
>
> Thanks!
>
> --Jeremy
>
>

Re: Capacity Scheduler question

[Serge Blazhievsky]

Take a look at fair scheduler it will do what you ask for

Sent from my iPhone

On Mar 22, 2013, at 2:48 PM, jeremy p  wrote:

> I have two jobs, Job A and Job B.  Job A needs to run with 18 mappers per 
> machine, Job B needs to run with 1 mapper per machine.  Hadoop doesn't give 
> you a way to specify number of mappers on a per-job basis.  
> mapred.tasktracker.map.tasks.maximum and mapred.map.tasks do absolutely 
> nothing.  I've been looking into the Capacity Scheduler, but I'm unsure if it 
> can help me.  In this documentation, all the settings under "Resource 
> Allocation" are cluster-wide.  I need to be able to set the maximum capacity 
> on a given machine.  It does look like you have the option to set the 
> required amount of memory per slot, but that setting applies to all the 
> queues.  If I could set that value on a per-queue basis, that would be 
> helpful.
> 
> Will the capacity scheduler help me here?  Or am I barking up the wrong tree? 
>  If the capacity scheduler won't help me, can you think of anything that will?
> 
> Thanks!
> 
> --Jeremy

Re: The most newbie question ever

[Keith Thomas]

OK. I have kept battling through, guessing at the gaps in the getting
started page but the final command to run the hadoop-examples.jar has
blocked me. As far as I can tell there is no hadoop-examples.jar file in
the distribution.

At a higher level I must be doing something wrong. The path I've
*followed*makes it seem as if there are effectively zero getting
started docs. This
cannot be the case for such a significant piece of software. Please help
straighten me out.

On Fri, Mar 22, 2013 at 1:30 PM, Keith Thomas wrote:

> Cool. Thanks this should be more straightforward.
>
> The first config file mentioned here,
>
>
> http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/SingleCluster.html
>
> is mapred-site.xml. The instructions say to add some config info to this
> file. Navigating to the recommended directory I don't see this file:
>
> keithomas@ubuntu:~/Development/Software/hadoop-2.0.3-alpha/etc/hadoop$ ls
> -altotal 116
> drwxr-xr-x 2 keithomas keithomas 4096 Mar 21 15:01 .
> drwxr-xr-x 3 keithomas keithomas 4096 Mar 22 12:43 ..
> -rw-r--r-- 1 keithomas keithomas 3578 Feb  6 19:40 capacity-scheduler.xml
> -rw-r--r-- 1 keithomas keithomas 1335 Feb  6 19:40 configuration.xsl
> -rw-r--r-- 1 keithomas keithomas  233 Feb  6 19:40 container-executor.cfg
> -rw-r--r-- 1 keithomas keithomas  774 Feb  6 19:40 core-site.xml
> -rw-r--r-- 1 keithomas keithomas 3397 Feb  6 19:40 hadoop-env.sh
> -rw-r--r-- 1 keithomas keithomas 1774 Feb  6 19:40
> hadoop-metrics2.properties
> -rw-r--r-- 1 keithomas keithomas 2490 Feb  6 19:40
> hadoop-metrics.properties
> -rw-r--r-- 1 keithomas keithomas 9304 Feb  6 19:40 hadoop-policy.xml
> -rw-r--r-- 1 keithomas keithomas  775 Feb  6 19:40 hdfs-site.xml
> -rw-r--r-- 1 keithomas keithomas 1180 Feb  6 19:40 httpfs-env.sh
> -rw-r--r-- 1 keithomas keithomas 1657 Feb  6 19:40 httpfs-log4j.properties
> -rw-r--r-- 1 keithomas keithomas   21 Feb  6 19:40 httpfs-signature.secret
> -rw-r--r-- 1 keithomas keithomas  620 Feb  6 19:40 httpfs-site.xml
> -rw-r--r-- 1 keithomas keithomas 8735 Feb  6 19:40 log4j.properties
> -rw-r--r-- 1 keithomas keithomas 1383 Feb  6 19:40 mapred-env.sh
> -rw-r--r-- 1 keithomas keithomas 4113 Feb  6 19:40
> mapred-queues.xml.template
> -rw-r--r-- 1 keithomas keithomas  758 Feb  6 19:40 mapred-site.xml.template
> -rw-r--r-- 1 keithomas keithomas   10 Feb  6 19:40 slaves
> -rw-r--r-- 1 keithomas keithomas 2316 Feb  6 19:40 ssl-client.xml.example
> -rw-r--r-- 1 keithomas keithomas 2251 Feb  6 19:40 ssl-server.xml.example
> -rw-r--r-- 1 keithomas keithomas 2527 Feb  6 19:40 yarn-env.sh
> -rw-r--r-- 1 keithomas keithomas  690 Feb  6 19:40 yarn-site.xml
>
> Am I supposed to create my own copy from the template file with a similar
> name?
> -Keith
>
>
> On Thu, Mar 21, 2013 at 10:19 PM, Harsh J  wrote:
>
>> You've downloaded a binary inclusive release and can just start using
>> it after configuring it. Mark the extracted path as HADOOP_PREFIX, and
>> look at $HADOOP_PREFIX/etc/hadoop/ for the config files. Run HDFS and
>> YARN via the $HADOOP_PREFIX/sbin/start-{dfs,yarn}.sh scripts, and of
>> course place $HADOOP_PREFIX/bin/ on your PATH.
>>
>> If you instead prefer installing 2.x RPMs or DEBs, head over to the
>> Apache Bigtop project at http://bigtop.apache.org to find the rpm/deb
>> scripts and perhaps also packages from other builders.
>>
>> Out of the box, for 2.0.3, you may also have to configure the
>> CapacityScheduler (since its the default now) to be able to run the
>> YARN services.
>>
>> On Fri, Mar 22, 2013 at 6:14 AM, Keith Thomas 
>> wrote:
>> > I have not checked out the source code. I have downloaded the following
>> file
>> >
>> > hadoop-2.0.3-alpha.tar.gz 07-Feb-2013 03:48   89M
>> >
>> > unzipped it and now I'm trying to find the installation instructions.
>> >
>> > -Keith
>> >
>> >
>> > On Thu, Mar 21, 2013 at 5:41 PM, Hitesh Shah 
>> wrote:
>> >>
>> >> Also, BUILDING.txt can be found at the top level directory of the
>> checked
>> >> out code.
>> >>
>> >> -- Hitesh
>> >>
>> >> On Mar 21, 2013, at 5:39 PM, Hitesh Shah wrote:
>> >>
>> >> > Assuming you have checked out the hadoop source code into
>> >> > /home/keithomas/hadoop-common/ , you need to run the maven command
>> in that
>> >> > directory and not from /home/keithomas/
>> >> >
>> >> > -- Hitesh
>> >> >
>> >> >
>> >> > On Mar 21, 2013, at 5:20 PM, Keith Thomas wrote:
>> >> >
>> >> >> Thank you for your help Mayank. I managed to google for BUILDING.TXT
>> >> >> and discovered I need maven3. I installed maven3.
>> >> >>
>> >> >> When I enter the command on the getting started link you sent me
>> this
>> >> >> happens,
>> >> >>
>> >> >> keithomas@ubuntu:~$ mvn clean install -DskipTests
>> >> >> [INFO] Scanning for projects...
>> >> >> [INFO]
>> >> >>
>> 
>> >> >> [INFO] BUILD FAILURE
>> >> >> [INFO]
>> >> >>
>> 
>> >> >> [I

Cluster lost IP addresses

[John Meza]

I have a 18 node cluster that had to be physically moved.Unfortunately all the 
ip addresses were lost (recreated).
This must have happened to someone before.Nothing else on the machines has been 
changed. Most importantly the data in HDFS is still sitting there.
Is there a way to recover this cluster to a useable state?thanksJohn
  

Capacity Scheduler question

[jeremy p]

I have two jobs, Job A and Job B.  Job A needs to run with 18 mappers per
machine, Job B needs to run with 1 mapper per machine.  Hadoop doesn't give
you a way to specify number of mappers on a per-job basis.
mapred.tasktracker.map.tasks.maximum and mapred.map.tasks do absolutely
nothing.  I've been looking into the Capacity Scheduler, but I'm unsure if
it can help me.  In
thisdocumentation,
all the settings under "Resource Allocation" are
cluster-wide.  I need to be able to set the maximum capacity on a given
machine.  It does look like you have the option to set the required amount
of memory per slot, but that setting applies to all the queues.  If I could
set that value on a per-queue basis, that would be helpful.

Will the capacity scheduler help me here?  Or am I barking up the wrong
tree?  If the capacity scheduler won't help me, can you think of anything
that will?

Thanks!

--Jeremy

Re: What happens when you have fewer input files than mapper slots?

[jeremy p]

Is there a way to force an even spread of data?

On Fri, Mar 22, 2013 at 2:14 PM, jeremy p wrote:

> Apologies -- I don't understand this advice : "If the evenness is the goal
> you can also write your own input format that return empty locations for
> each split and read the small files in map task directly."  How would
> manually reading the files into the map task help me?  Hadoop would still
> spawn multiple mappers per machine, which is what I'm trying to avoid.  I'm
> trying to get one mapper per machine for this job.
>
> --Jeremy
>
>
> On Thu, Mar 21, 2013 at 11:44 AM, Luke Lu  wrote:
>
>>
>> Short version : let's say you have 20 nodes, and each node has 10 mapper
>>> slots.  You start a job with 20 very small input files.  How is the work
>>> distributed to the cluster?  Will it be even, with each node spawning one
>>> mapper task?  Is there any way of predicting or controlling how the work
>>> will be distributed?
>>
>>
>> You're right in expecting that the tasks of the small job will likely be
>> evenly distributed among 20 nodes, if the 20 files are evenly distributed
>> among the nodes and that there are free slots on every node.
>>
>>
>>> Long version : My cluster is currently used for two different jobs.  The
>>> cluster is currently optimized for Job A, so each node has a maximum of 18
>>> mapper slots.  However, I also need to run Job B.  Job B is VERY
>>> cpu-intensive, so we really only want one mapper to run on a node at any
>>> given time.  I've done a bunch of research, and it doesn't seem like Hadoop
>>> gives you any way to set the maximum number of mappers per node on a
>>> per-job basis.  I'm at my wit's end here, and considering some rather
>>> egregious workarounds.  If you can think of anything that can help me, I'd
>>> very much appreciate it.
>>>
>>
>> Are you seeing Job B tasks are not being evenly distributed to each node?
>> You can check the locations of the files by hadoop fsck. If the evenness is
>> the goal you can also write your own input format that return empty
>> locations for each split and read the small files in map task directly. If
>> you're using Hadoop 1.0.x and fair scheduler, you might need to set
>> mapred.fairscheduler.assignmultiple to false in mapred-site.xml (JT restart
>> required) to work around a bug in fairscheduler (MAPREDUCE-2905) that
>> causes the tasks be assigned unevenly. The bug is fixed in Hadoop 1.1+.
>>
>> __Luke
>>
>
>

Re: About running a simple wordcount mapreduce

[Abdelrahman Shettia]

Hi Redwane,

Please run the following command as hdfs user on any datanode. The output
will be something like this. Hope this helps

hadoop dfsadmin -report
Configured Capacity: 81075068925 (75.51 GB)
Present Capacity: 70375292928 (65.54 GB)
DFS Remaining: 69895163904 (65.09 GB)
DFS Used: 480129024 (457.89 MB)
DFS Used%: 0.68%
Under replicated blocks: 0
Blocks with corrupt replicas: 0
Missing blocks: 0

Thanks
-Abdelrahman


On Fri, Mar 22, 2013 at 12:35 PM, reduno1985 wrote:

>
> I have my hosts running on openstack virtual machine instances each
> instance has 10gb hard disc . Is there a way too see how much space is in
> the hdfs without web ui .
>
>
> Sent from Samsung Mobile
>
> Serge Blazhievsky  wrote:
> Check web ui how much space you have on hdfs???
>
> Sent from my iPhone
>
> On Mar 22, 2013, at 11:41 AM, Abdelrahman Shettia <
> ashet...@hortonworks.com> wrote:
>
> Hi Redwane ,
>
> It is possible that the hosts which are running tasks are do not have
> enough space. Those dirs are confiugred in mapred-site.xml
>
>
>
> On Fri, Mar 22, 2013 at 8:42 AM, Redwane belmaati cherkaoui <
> reduno1...@googlemail.com> wrote:
>
>>
>>
>> -- Forwarded message --
>> From: Redwane belmaati cherkaoui 
>> Date: Fri, Mar 22, 2013 at 4:39 PM
>> Subject: About running a simple wordcount mapreduce
>> To: mapreduce-iss...@hadoop.apache.org
>>
>>
>> Hi
>> I am trying to run  a wordcount mapreduce job on several files (<20 mb)
>> using two machines . I get stuck on 0% map 0% reduce.
>> The jobtracker log file shows the following warning:
>>  WARN org.apache.hadoop.mapred.JobInProgress: No room for map task. Node
>> hadoop0.novalocal has 8791384064 bytes free; but we expect map to take
>> 1317624576693539401
>>
>> Please help me ,
>> Best Regards,
>>
>>
>

Re: What happens when you have fewer input files than mapper slots?

[jeremy p]

Apologies -- I don't understand this advice : "If the evenness is the goal
you can also write your own input format that return empty locations for
each split and read the small files in map task directly."  How would
manually reading the files into the map task help me?  Hadoop would still
spawn multiple mappers per machine, which is what I'm trying to avoid.  I'm
trying to get one mapper per machine for this job.

--Jeremy

On Thu, Mar 21, 2013 at 11:44 AM, Luke Lu  wrote:

>
> Short version : let's say you have 20 nodes, and each node has 10 mapper
>> slots.  You start a job with 20 very small input files.  How is the work
>> distributed to the cluster?  Will it be even, with each node spawning one
>> mapper task?  Is there any way of predicting or controlling how the work
>> will be distributed?
>
>
> You're right in expecting that the tasks of the small job will likely be
> evenly distributed among 20 nodes, if the 20 files are evenly distributed
> among the nodes and that there are free slots on every node.
>
>
>> Long version : My cluster is currently used for two different jobs.  The
>> cluster is currently optimized for Job A, so each node has a maximum of 18
>> mapper slots.  However, I also need to run Job B.  Job B is VERY
>> cpu-intensive, so we really only want one mapper to run on a node at any
>> given time.  I've done a bunch of research, and it doesn't seem like Hadoop
>> gives you any way to set the maximum number of mappers per node on a
>> per-job basis.  I'm at my wit's end here, and considering some rather
>> egregious workarounds.  If you can think of anything that can help me, I'd
>> very much appreciate it.
>>
>
> Are you seeing Job B tasks are not being evenly distributed to each node?
> You can check the locations of the files by hadoop fsck. If the evenness is
> the goal you can also write your own input format that return empty
> locations for each split and read the small files in map task directly. If
> you're using Hadoop 1.0.x and fair scheduler, you might need to set
> mapred.fairscheduler.assignmultiple to false in mapred-site.xml (JT restart
> required) to work around a bug in fairscheduler (MAPREDUCE-2905) that
> causes the tasks be assigned unevenly. The bug is fixed in Hadoop 1.1+.
>
> __Luke
>

Re: The most newbie question ever

[Keith Thomas]

Cool. Thanks this should be more straightforward.

The first config file mentioned here,


http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/SingleCluster.html

is mapred-site.xml. The instructions say to add some config info to this
file. Navigating to the recommended directory I don't see this file:

keithomas@ubuntu:~/Development/Software/hadoop-2.0.3-alpha/etc/hadoop$ ls
-altotal 116
drwxr-xr-x 2 keithomas keithomas 4096 Mar 21 15:01 .
drwxr-xr-x 3 keithomas keithomas 4096 Mar 22 12:43 ..
-rw-r--r-- 1 keithomas keithomas 3578 Feb  6 19:40 capacity-scheduler.xml
-rw-r--r-- 1 keithomas keithomas 1335 Feb  6 19:40 configuration.xsl
-rw-r--r-- 1 keithomas keithomas  233 Feb  6 19:40 container-executor.cfg
-rw-r--r-- 1 keithomas keithomas  774 Feb  6 19:40 core-site.xml
-rw-r--r-- 1 keithomas keithomas 3397 Feb  6 19:40 hadoop-env.sh
-rw-r--r-- 1 keithomas keithomas 1774 Feb  6 19:40
hadoop-metrics2.properties
-rw-r--r-- 1 keithomas keithomas 2490 Feb  6 19:40 hadoop-metrics.properties
-rw-r--r-- 1 keithomas keithomas 9304 Feb  6 19:40 hadoop-policy.xml
-rw-r--r-- 1 keithomas keithomas  775 Feb  6 19:40 hdfs-site.xml
-rw-r--r-- 1 keithomas keithomas 1180 Feb  6 19:40 httpfs-env.sh
-rw-r--r-- 1 keithomas keithomas 1657 Feb  6 19:40 httpfs-log4j.properties
-rw-r--r-- 1 keithomas keithomas   21 Feb  6 19:40 httpfs-signature.secret
-rw-r--r-- 1 keithomas keithomas  620 Feb  6 19:40 httpfs-site.xml
-rw-r--r-- 1 keithomas keithomas 8735 Feb  6 19:40 log4j.properties
-rw-r--r-- 1 keithomas keithomas 1383 Feb  6 19:40 mapred-env.sh
-rw-r--r-- 1 keithomas keithomas 4113 Feb  6 19:40
mapred-queues.xml.template
-rw-r--r-- 1 keithomas keithomas  758 Feb  6 19:40 mapred-site.xml.template
-rw-r--r-- 1 keithomas keithomas   10 Feb  6 19:40 slaves
-rw-r--r-- 1 keithomas keithomas 2316 Feb  6 19:40 ssl-client.xml.example
-rw-r--r-- 1 keithomas keithomas 2251 Feb  6 19:40 ssl-server.xml.example
-rw-r--r-- 1 keithomas keithomas 2527 Feb  6 19:40 yarn-env.sh
-rw-r--r-- 1 keithomas keithomas  690 Feb  6 19:40 yarn-site.xml

Am I supposed to create my own copy from the template file with a similar
name?
-Keith

On Thu, Mar 21, 2013 at 10:19 PM, Harsh J  wrote:

> You've downloaded a binary inclusive release and can just start using
> it after configuring it. Mark the extracted path as HADOOP_PREFIX, and
> look at $HADOOP_PREFIX/etc/hadoop/ for the config files. Run HDFS and
> YARN via the $HADOOP_PREFIX/sbin/start-{dfs,yarn}.sh scripts, and of
> course place $HADOOP_PREFIX/bin/ on your PATH.
>
> If you instead prefer installing 2.x RPMs or DEBs, head over to the
> Apache Bigtop project at http://bigtop.apache.org to find the rpm/deb
> scripts and perhaps also packages from other builders.
>
> Out of the box, for 2.0.3, you may also have to configure the
> CapacityScheduler (since its the default now) to be able to run the
> YARN services.
>
> On Fri, Mar 22, 2013 at 6:14 AM, Keith Thomas 
> wrote:
> > I have not checked out the source code. I have downloaded the following
> file
> >
> > hadoop-2.0.3-alpha.tar.gz 07-Feb-2013 03:48   89M
> >
> > unzipped it and now I'm trying to find the installation instructions.
> >
> > -Keith
> >
> >
> > On Thu, Mar 21, 2013 at 5:41 PM, Hitesh Shah 
> wrote:
> >>
> >> Also, BUILDING.txt can be found at the top level directory of the
> checked
> >> out code.
> >>
> >> -- Hitesh
> >>
> >> On Mar 21, 2013, at 5:39 PM, Hitesh Shah wrote:
> >>
> >> > Assuming you have checked out the hadoop source code into
> >> > /home/keithomas/hadoop-common/ , you need to run the maven command in
> that
> >> > directory and not from /home/keithomas/
> >> >
> >> > -- Hitesh
> >> >
> >> >
> >> > On Mar 21, 2013, at 5:20 PM, Keith Thomas wrote:
> >> >
> >> >> Thank you for your help Mayank. I managed to google for BUILDING.TXT
> >> >> and discovered I need maven3. I installed maven3.
> >> >>
> >> >> When I enter the command on the getting started link you sent me this
> >> >> happens,
> >> >>
> >> >> keithomas@ubuntu:~$ mvn clean install -DskipTests
> >> >> [INFO] Scanning for projects...
> >> >> [INFO]
> >> >>
> 
> >> >> [INFO] BUILD FAILURE
> >> >> [INFO]
> >> >>
> 
> >> >> [INFO] Total time: 0.119s
> >> >> [INFO] Finished at: Thu Mar 21 17:14:39 PDT 2013
> >> >> [INFO] Final Memory: 3M/247M
> >> >> [INFO]
> >> >>
> 
> >> >> [ERROR] The goal you specified requires a project to execute but
> there
> >> >> is no POM in this directory (/home/keithomas). Please verify you
> invoked
> >> >> Maven from the correct directory. -> [Help 1]
> >> >>
> >> >> and when I enter the command you kindly sent this happens,
> >> >>
> >> >> keithomas@ubuntu:~$ mvn package -Pdist -DskipTests -Dtar
> >> >> [INFO] Scanning for projects...
> >> >> [INFO]
> >> >>
> -

Re: About running a simple wordcount mapreduce

[reduno1985]

I have my hosts running on openstack virtual machine instances each instance 
has 10gb hard disc . Is there a way too see how much space is in the hdfs 
without web ui .


Sent from Samsung MobileSerge Blazhievsky  wrote:Check web 
ui how much space you have on hdfs???

Sent from my iPhone

On Mar 22, 2013, at 11:41 AM, Abdelrahman Shettia  
wrote:

Hi Redwane , 

It is possible that the hosts which are running tasks are do not have enough 
space. Those dirs are confiugred in mapred-site.xml



On Fri, Mar 22, 2013 at 8:42 AM, Redwane belmaati cherkaoui 
 wrote:


-- Forwarded message --
From: Redwane belmaati cherkaoui 
Date: Fri, Mar 22, 2013 at 4:39 PM
Subject: About running a simple wordcount mapreduce
To: mapreduce-iss...@hadoop.apache.org


Hi 
I am trying to run  a wordcount mapreduce job on several files (<20 mb) using 
two machines . I get stuck on 0% map 0% reduce.
The jobtracker log file shows the following warning:
 WARN org.apache.hadoop.mapred.JobInProgress: No room for map task. Node 
hadoop0.novalocal has 8791384064 bytes free; but we expect map to take 
1317624576693539401

Please help me ,
Best Regards,

Re: About running a simple wordcount mapreduce

[reduno1985]

Thanks . 
Each host has 8gb but hadoop is estimating too much space the number estimated 
is too big for any host in the world ;). My input data are  simple text files 
that do not exceed 20 mb. I do not know why hadooop os estimating that much. 



Sent from Samsung MobileAbdelrahman Shettia  wrote:Hi 
Redwane , 

It is possible that the hosts which are running tasks are do not have enough 
space. Those dirs are confiugred in mapred-site.xml



On Fri, Mar 22, 2013 at 8:42 AM, Redwane belmaati cherkaoui 
 wrote:


-- Forwarded message --
From: Redwane belmaati cherkaoui 
Date: Fri, Mar 22, 2013 at 4:39 PM
Subject: About running a simple wordcount mapreduce
To: mapreduce-iss...@hadoop.apache.org


Hi 
I am trying to run  a wordcount mapreduce job on several files (<20 mb) using 
two machines . I get stuck on 0% map 0% reduce.
The jobtracker log file shows the following warning:
 WARN org.apache.hadoop.mapred.JobInProgress: No room for map task. Node 
hadoop0.novalocal has 8791384064 bytes free; but we expect map to take 
1317624576693539401

Please help me ,
Best Regards,

Re: About running a simple wordcount mapreduce

[Serge Blazhievsky]

Check web ui how much space you have on hdfs???

Sent from my iPhone

On Mar 22, 2013, at 11:41 AM, Abdelrahman Shettia  
wrote:

> Hi Redwane , 
> 
> It is possible that the hosts which are running tasks are do not have enough 
> space. Those dirs are confiugred in mapred-site.xml
> 
> 
> 
> On Fri, Mar 22, 2013 at 8:42 AM, Redwane belmaati cherkaoui 
>  wrote:
>> 
>> 
>> -- Forwarded message --
>> From: Redwane belmaati cherkaoui 
>> Date: Fri, Mar 22, 2013 at 4:39 PM
>> Subject: About running a simple wordcount mapreduce
>> To: mapreduce-iss...@hadoop.apache.org
>> 
>> 
>> Hi 
>> I am trying to run  a wordcount mapreduce job on several files (<20 mb) 
>> using two machines . I get stuck on 0% map 0% reduce.
>> The jobtracker log file shows the following warning:
>>  WARN org.apache.hadoop.mapred.JobInProgress: No room for map task. Node 
>> hadoop0.novalocal has 8791384064 bytes free; but we expect map to take 
>> 1317624576693539401
>> 
>> Please help me ,
>> Best Regards,
> 

Re: About running a simple wordcount mapreduce

[Abdelrahman Shettia]

Hi Redwane ,

It is possible that the hosts which are running tasks are do not have
enough space. Those dirs are confiugred in mapred-site.xml



On Fri, Mar 22, 2013 at 8:42 AM, Redwane belmaati cherkaoui <
reduno1...@googlemail.com> wrote:

>
>
> -- Forwarded message --
> From: Redwane belmaati cherkaoui 
> Date: Fri, Mar 22, 2013 at 4:39 PM
> Subject: About running a simple wordcount mapreduce
> To: mapreduce-iss...@hadoop.apache.org
>
>
> Hi
> I am trying to run  a wordcount mapreduce job on several files (<20 mb)
> using two machines . I get stuck on 0% map 0% reduce.
> The jobtracker log file shows the following warning:
>  WARN org.apache.hadoop.mapred.JobInProgress: No room for map task. Node
> hadoop0.novalocal has 8791384064 bytes free; but we expect map to take
> 1317624576693539401
>
> Please help me ,
> Best Regards,
>
>

Re: Group names for custom Counters

[Michel Segel]

Just a suggestion, look at dynamic counters...

For the group, just create a group name and you are done.

Sent from a remote device. Please excuse any typos...

Mike Segel

On Mar 22, 2013, at 11:17 AM, Tony Burton  wrote:

> Hi list,
> 
> I'm using Hadoop 1.0.3 and creating some custom Counters in my Mapper. I've 
> got an enum that defines the list of counters, and I'm incrementing in the 
> map function using context.getCounter().increment(1).
> 
> I see that there's another implementation of context.getCounter() that takes 
> arguments String groupName and String counterName. I think I'd find it useful 
> to be able to group my Counters - can anyone give me an example of how to 
> collect my Counters in groups?
> 
> Thanks,
> 
> Tony
> 
> 
> 
> *
> P Please consider the environment before printing this email or attachments
> 
> 
> This email and any attachments are confidential, protected by copyright and 
> may be legally privileged. If you are not the intended recipient, then the 
> dissemination or copying of this email is prohibited. If you have received 
> this in error, please notify the sender by replying by email and then delete 
> the email completely from your system. Neither Sporting Index nor the sender 
> accepts responsibility for any virus, or any other defect which might affect 
> any computer or IT system into which the email is received and/or opened. It 
> is the responsibility of the recipient to scan the email and no 
> responsibility is accepted for any loss or damage arising in any way from 
> receipt or use of this email. Sporting Index Ltd is a company registered in 
> England and Wales with company number 2636842, whose registered office is at 
> Gateway House, Milverton Street, London, SE11 4AP. Sporting Index Ltd is 
> authorised and regulated by the UK Financial Services Authority (reg. no. 
> 150404) and Gambling Commission (reg. no. 000-027343-R-308898-001). Any 
> financial promotion contained herein has been issued and approved by Sporting 
> Index Ltd.
> 
> 
> Outbound email has been scanned for viruses and SPAM

Re: disk used percentage is not symmetric on datanodes (balancer)

[Алексей Бабутин]

2013/3/20 Tapas Sarangi 

> Thanks for your reply. Some follow up questions below :
>
> On Mar 20, 2013, at 5:35 AM, Алексей Бабутин 
> wrote:
>
>
>
> dfs.balance.bandwidthPerSec in hdfs-site.xml.I think balancer cant help
> you,because it makes all the nodes equal.They can differ only on balancer
> threshold.Threshold =10 by default.It means,that nodes can differ up to
> 350Tb between each other in 3.5Pb cluster.If Threshold =1 up to 35Tb and so
> on.
>
>
> If we use multiple racks, let's assume we have 10 racks now and they are
> equally divided in size (350 TB each). With a default threshold of 10, any
> two nodes on a given rack will have a maximum difference of 35 TB, is this
> correct ? Also, does this mean the difference between any two racks will
> also go down to 35 TB ?
>

Balancer know about topology,but when calculate balancing it operates only
with nodes not with racks.
You can see how it work in Balancer.java in  BalancerDatanode about string
509.

I was wrong about 350Tb,35Tb it calculates in such way :

For example:
cluster_capacity=3.5Pb
cluster_dfsused=2Pb

avgutil=cluster_dfsused/cluster_capacity*100=57.14% used cluster capacity
Then we know avg node utilization (node_dfsused/node_capacity*100)
.Balancer think that all good if  avgutil
+10>node_utilizazation>=avgutil-10.

Ideal case that all node used avgutl of capacity.but for 12TB node its only
6.5Tb and for 72Tb its about 40Tb.

Balancer cant help you.

Show me http://namenode.rambler.ru:50070/dfsnodelist.jsp?whatNodes=LIVE if
you can.



>
>
> In ideal case with replication factor 2 ,with two nodes 12Tb and 72Tb you
> will be able to have only 12Tb replication data.
>
>
> Yes, this is true for exactly two nodes in the cluster with 12 TB and 72
> TB, but not true for more than two nodes in the cluster.
>
>
> Best way,on my opinion,it is using multiple racks.Nodes in rack must be
> with identical capacity.Racks must be identical capacity.
> For example:
>
> rack1: 1 node with 72Tb
> rack2: 6 nodes with 12Tb
> rack3: 3 nodes with 24Tb
>
> It helps with balancing,because dublicated  block must be another rack.
>
>
> The same question I asked earlier in this message, does multiple racks
> with default threshold for the balancer minimizes the difference between
> racks ?
>
> Why did you select hdfs?May be lustre,cephfs and other is better choise.
>
>
> It wasn't my decision, and I probably can't change it now. I am new to
> this cluster and trying to understand few issues. I will explore other
> options as you mentioned.
>
>
>

Re: Need Help on Hadoop cluster Setup

[Mohammad Tariq]

you are welcome.

Warm Regards,
Tariq
https://mtariq.jux.com/
cloudfront.blogspot.com


On Fri, Mar 22, 2013 at 8:48 PM, MShaik  wrote:

>
> Thank you, Tariq.
> After chang the namesapceID on datanodes, all datanodes are started.
>
>  Thank you once again...!
>
> -Original Message-
> From: Mohammad Tariq 
> To: user 
> Sent: Fri, Mar 22, 2013 8:29 pm
> Subject: Re: Need Help on Hadoop cluster Setup
>
>  sorry for the typo in the second line of the 2nd point. the path will be
> "/dfs.data.dir/current/VERSION".
>
>  Warm Regards,
> Tariq
> https://mtariq.jux.com/
>  cloudfront.blogspot.com
>
>
> On Fri, Mar 22, 2013 at 8:27 PM, Mohammad Tariq wrote:
>
>> have you reformatted the hdfs?if that is the case it was, i think, not
>> proper.
>> were the nodes which you attached serving some other cluster earlier?your
>> logs show that you are facing problems because of mismatch in the IDs of
>> the NN and the IDs which DNs have. to overcome this problem you can
>> follow these steps :
>>
>>  1 - Stop all teh DNs.
>> 2 - Go to the directory which is serving as your dfs.data.dir. Inside
>> this directiry
>> you'll find a subdirectory ". there will be a file named as "VERSION"  in
>> this
>> directory. in this file you can see the namespaceID(probably the second
>> line).
>>  change it to match the namespaceID which is there in
>> "dfs.name.dir/current/VERSION"
>> file.
>> 3 - restart the processes.
>>
>>  HTH
>>
>>
>>  Warm Regards,
>> Tariq
>> https://mtariq.jux.com/
>>  cloudfront.blogspot.com
>>
>>
>>   On Fri, Mar 22, 2013 at 8:04 PM, MShaik  wrote:
>>
>>>  Hi,
>>>
>>>  DataNode is not started on all the nodes, as tasktracker is started on
>>> all the nodes.
>>>
>>>  please find the below datanode log, please let me know the solution.
>>>
>>>  2013-03-22 19:52:27,380 INFO org.apache.hadoop.ipc.RPC: Server at
>>> n1.hc.com/192.168.1.110:54310 not available yet, Z...
>>> 2013-03-22 19:52:29,386 INFO org.apache.hadoop.ipc.Client: Retrying
>>> connect to server: n1.hc.com/192.168.1.110:54310. Already tried 0
>>> time(s).
>>> 2013-03-22 19:52:30,411 INFO org.apache.hadoop.ipc.Client: Retrying
>>> connect to server: n1.hc.com/192.168.1.110:54310. Already tried 1
>>> time(s).
>>> 2013-03-22 19:52:31,416 INFO org.apache.hadoop.ipc.Client: Retrying
>>> connect to server: n1.hc.com/192.168.1.110:54310. Already tried 2
>>> time(s).
>>> 2013-03-22 19:52:32,420 INFO org.apache.hadoop.ipc.Client: Retrying
>>> connect to server: n1.hc.com/192.168.1.110:54310. Already tried 3
>>> time(s).
>>> 2013-03-22 19:52:33,426 INFO org.apache.hadoop.ipc.Client: Retrying
>>> connect to server: n1.hc.com/192.168.1.110:54310. Already tried 4
>>> time(s).
>>> 2013-03-22 19:52:49,162 ERROR
>>> org.apache.hadoop.hdfs.server.datanode.DataNode: java.io.IOException:
>>> Incompatible namespaceIDs in /home/hduser/hadoopdata: namenode namespaceID
>>> = 2050588793; datanode namespaceID = 503772406
>>>  at
>>> org.apache.hadoop.hdfs.server.datanode.DataStorage.doTransition(DataStorage.java:232)
>>>  at
>>> org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:147)
>>>  at
>>> org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:385)
>>>  at
>>> org.apache.hadoop.hdfs.server.datanode.DataNode.(DataNode.java:299)
>>>  at
>>> org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1582)
>>>  at
>>> org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1521)
>>>  at
>>> org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:1539)
>>>  at
>>> org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:1665)
>>>  at
>>> org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:1682)
>>>
>>>  2013-03-22 19:52:49,168 INFO
>>> org.apache.hadoop.hdfs.server.datanode.DataNode: SHUTDOWN_MSG:
>>> /
>>> SHUTDOWN_MSG: Shutting down DataNode at n4.hc.com/192.168.1.113
>>> /
>>>
>>>
>>> Thank's
>>>
>>> -Original Message-
>>> From: Mohammad Tariq 
>>> To: user 
>>> Sent: Fri, Mar 22, 2013 7:07 pm
>>> Subject: Re: Need Help on Hadoop cluster Setup
>>>
>>>  Hello Munavvar,
>>>
>>>It depends on your configuration where your DNs and TTs will
>>> run. If you have configured all your slaves to run both the processes then
>>> they should. If they are not running then there is definitely some problem.
>>> Could you please check your DN logs once and see if you find anything
>>> unusual there. And you have to copy the files across all the machines.
>>>
>>>  You can do one more thing just to cross check. Point your web browser
>>> to the HDFS web UI(master_machine:9000) to do that.
>>>
>>>  Warm Regards,
>>> Tariq
>>> https://mtariq.jux.com/
>>>  cloudfront.blogspot.com
>>>
>>>
>>> On Fri, Mar 22, 2013 at 6:44 PM, Munnavar Sk  wrote:
>>>

 Hi ,

 I am new to Hadoop and I a

RE: problem running multiple native mode map reduce processes concurrently

[Derrick H. Karimi]

Thank you for the response.

Hadoop 0.20.2-cdh3u3

--Derrick H. Karimi
--Software Developer, SEI Innovation Center
--Carnegie Mellon University


-Original Message-
From: Harsh J [mailto:ha...@cloudera.com] 
Sent: Friday, March 22, 2013 1:32 AM
To: 
Subject: Re: problem running multiple native mode map reduce processes 
concurrently

Please post your Hadoop version (command: hadoop version).

On Thu, Mar 21, 2013 at 10:59 PM, Derrick H. Karimi  
wrote:
> Anybody have any ideas?  How can I safely run two native mode rap 
> reduces on one machine at the same time?
>
>
>
> --Derrick H. Karimi
>
> --Software Developer, SEI Innovation Center
>
> --Carnegie Mellon University
>
>
>
> From: Derrick H. Karimi
> Sent: Tuesday, March 19, 2013 10:55 PM
> To: 'user@hadoop.apache.org'
> Subject: problem running multiple native mode map reduce processes 
> concurrently
>
>
>
> Hi,
>
>
>
> I have a MapReduce program I have written and have used it on top of a 
> Hadoop cluster with success.  During development, for quick tests, and 
> when the cluster is not available I run it on machines that have no 
> access to a Hadoop cluster.  I do this with regular command line 
> invocation
>
>
>
> java -cp $MY_HADOOP_JARS:mybuild/app_under_test.jar
>
>
>
> This works fine, until I attempt to run more than one at a time.  When 
> I do launch many at one time I intermittently get failures.  (each 
> invocation is using a separate copy of jars, and has its own working 
> directory and input/output area, they are fully distributable and do not 
> share anything.
> The machines have plenty of disk space too.)  Most commonly I get two 
> exception's in my job's stderr output:
>
>
>
> org.apache.hadoop.util.DiskChecker$DiskErrorException: "Could not find 
> output/file.out in any of the configured local directories"
>
>
>
> when I see this error the job appears to continue on, but in the 
> output I can tell that several of my input files were not processed.  
> I have nothing called "output/file.out" in my job.
>
>
>
> The other error text I do not have handy at the moment, but it appears 
> to be an XML parser error at job startup on some file in the /tmp 
> directory that is not part of any file mentioned in my job.  Here I 
> assume that multiple instances of the native mode implementation of 
> map reduce are trying to write to the same file at startup and it gets 
> corrupted.  In these cases the job fails and I do not get any output.  
> I theorize I can work around this error by sleeping a few seconds between 
> launching my processes.
>
>
>
> I expected to be able to run more than one of these processes at the 
> same time.  It appears I cannot.  Does anyone have any suggestions 
> that would help me do this?
>
>
>
> --Derrick H. Karimi
>
> --Software Developer, SEI Innovation Center
>
> --Carnegie Mellon University
>
>



--
Harsh J

Fwd: About running a simple wordcount mapreduce

[Redwane belmaati cherkaoui]

-- Forwarded message --
From: Redwane belmaati cherkaoui 
Date: Fri, Mar 22, 2013 at 4:39 PM
Subject: About running a simple wordcount mapreduce
To: mapreduce-iss...@hadoop.apache.org


Hi
I am trying to run  a wordcount mapreduce job on several files (<20 mb)
using two machines . I get stuck on 0% map 0% reduce.
The jobtracker log file shows the following warning:
 WARN org.apache.hadoop.mapred.JobInProgress: No room for map task. Node
hadoop0.novalocal has 8791384064 bytes free; but we expect map to take
1317624576693539401

Please help me ,
Best Regards,

Re: Need Help on Hadoop cluster Setup

[MShaik]

Thank you, Tariq.
After chang the namesapceID on datanodes, all datanodes are started.


Thank you once again...!


-Original Message-
From: Mohammad Tariq 
To: user 
Sent: Fri, Mar 22, 2013 8:29 pm
Subject: Re: Need Help on Hadoop cluster Setup


sorry for the typo in the second line of the 2nd point. the path will be 
"/dfs.data.dir/current/VERSION".


Warm Regards,
Tariq
https://mtariq.jux.com/

cloudfront.blogspot.com





On Fri, Mar 22, 2013 at 8:27 PM, Mohammad Tariq  wrote:

have you reformatted the hdfs?if that is the case it was, i think, not proper.
were the nodes which you attached serving some other cluster earlier?your
logs show that you are facing problems because of mismatch in the IDs of
the NN and the IDs which DNs have. to overcome this problem you can 
follow these steps :


1 - Stop all teh DNs.
2 - Go to the directory which is serving as your dfs.data.dir. Inside this 
directiry
you'll find a subdirectory ". there will be a file named as "VERSION"  in this 
directory. in this file you can see the namespaceID(probably the second line).
change it to match the namespaceID which is there in 
"dfs.name.dir/current/VERSION"
file.
3 - restart the processes.


HTH





Warm Regards,
Tariq
https://mtariq.jux.com/

cloudfront.blogspot.com






On Fri, Mar 22, 2013 at 8:04 PM, MShaik  wrote:

Hi,


DataNode is not started on all the nodes, as tasktracker is started on all the 
nodes.


please find the below datanode log, please let me know the solution.


2013-03-22 19:52:27,380 INFO org.apache.hadoop.ipc.RPC: Server at 
n1.hc.com/192.168.1.110:54310 not available yet, Z...
2013-03-22 19:52:29,386 INFO org.apache.hadoop.ipc.Client: Retrying connect to 
server: n1.hc.com/192.168.1.110:54310. Already tried 0 time(s).
2013-03-22 19:52:30,411 INFO org.apache.hadoop.ipc.Client: Retrying connect to 
server: n1.hc.com/192.168.1.110:54310. Already tried 1 time(s).
2013-03-22 19:52:31,416 INFO org.apache.hadoop.ipc.Client: Retrying connect to 
server: n1.hc.com/192.168.1.110:54310. Already tried 2 time(s).
2013-03-22 19:52:32,420 INFO org.apache.hadoop.ipc.Client: Retrying connect to 
server: n1.hc.com/192.168.1.110:54310. Already tried 3 time(s).
2013-03-22 19:52:33,426 INFO org.apache.hadoop.ipc.Client: Retrying connect to 
server: n1.hc.com/192.168.1.110:54310. Already tried 4 time(s).
2013-03-22 19:52:49,162 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: 
java.io.IOException: Incompatible namespaceIDs in /home/hduser/hadoopdata: 
namenode namespaceID = 2050588793; datanode namespaceID = 503772406
at 
org.apache.hadoop.hdfs.server.datanode.DataStorage.doTransition(DataStorage.java:232)
at 
org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:147)
at 
org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:385)
at 
org.apache.hadoop.hdfs.server.datanode.DataNode.(DataNode.java:299)
at 
org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1582)
at 
org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1521)
at 
org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:1539)
at 
org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:1665)
at 
org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:1682)


2013-03-22 19:52:49,168 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: 
SHUTDOWN_MSG: 
/
SHUTDOWN_MSG: Shutting down DataNode at n4.hc.com/192.168.1.113
/


Thank's



-Original Message-
From: Mohammad Tariq 
To: user 
Sent: Fri, Mar 22, 2013 7:07 pm
Subject: Re: Need Help on Hadoop cluster Setup


Hello Munavvar,


  It depends on your configuration where your DNs and TTs will run. If you 
have configured all your slaves to run both the processes then they should. If 
they are not running then there is definitely some problem. Could you please 
check your DN logs once and see if you find anything unusual there. And you 
have to copy the files across all the machines.


You can do one more thing just to cross check. Point your web browser to the 
HDFS web UI(master_machine:9000) to do that.


Warm Regards,
Tariq
https://mtariq.jux.com/

cloudfront.blogspot.com





On Fri, Mar 22, 2013 at 6:44 PM, Munnavar Sk  wrote:


Hi ,



 
I am new to Hadoop and I am fighting with this last 20days, somehowI got very 
good stuff on Hadoop.
 
But, some question are roaming around me...I hope, I can getthe answers from 
your end...!
 
I was setup a cluster in distributed mode with 5 nodes. Ihave configured 
Namenode and DataNodes and all datannodes are able to logingfrom namenode 
without password.
Hadoop and Java installed on same location in all the Nodes.After starting the 
cluster, I was check every node using with "jps"command.
Nam

Group names for custom Counters

[Tony Burton]

Hi list,

I'm using Hadoop 1.0.3 and creating some custom Counters in my Mapper. I've got 
an enum that defines the list of counters, and I'm incrementing in the map 
function using context.getCounter().increment(1).

I see that there's another implementation of context.getCounter() that takes 
arguments String groupName and String counterName. I think I'd find it useful 
to be able to group my Counters - can anyone give me an example of how to 
collect my Counters in groups?

Thanks,

Tony

**
Please consider the environment before printing this email or attachments

This email and any attachments are confidential, protected by copyright and may 
be legally privileged.  If you are not the intended recipient, then the 
dissemination or copying of this email is prohibited. If you have received this 
in error, please notify the sender by replying by email and then delete the 
email completely from your system.  Neither Sporting Index nor the sender 
accepts responsibility for any virus, or any other defect which might affect 
any computer or IT system into which the email is received and/or opened.  It 
is the responsibility of the recipient to scan the email and no responsibility 
is accepted for any loss or damage arising in any way from receipt or use of 
this email.  Sporting Index Ltd is a company registered in England and Wales 
with company number 2636842, whose registered office is at Gateway House, 
Milverton Street, London, SE11 4AP.  Sporting Index Ltd is authorised and 
regulated by the UK Financial Services Authority (reg. no. 150404) and Gambling 
Commission (reg. no. 000-027343-R-308898-001).  Any financial promotion 
contained herein has been issued
and approved by Sporting Index Ltd.

Outbound email has been scanned for viruses and SPAM

Re: Need Help on Hadoop cluster Setup

[Mohammad Tariq]

sorry for the typo in the second line of the 2nd point. the path will be
"/dfs.data.dir/current/VERSION".

Warm Regards,
Tariq
https://mtariq.jux.com/
cloudfront.blogspot.com


On Fri, Mar 22, 2013 at 8:27 PM, Mohammad Tariq  wrote:

> have you reformatted the hdfs?if that is the case it was, i think, not
> proper.
> were the nodes which you attached serving some other cluster earlier?your
> logs show that you are facing problems because of mismatch in the IDs of
> the NN and the IDs which DNs have. to overcome this problem you can
> follow these steps :
>
> 1 - Stop all teh DNs.
> 2 - Go to the directory which is serving as your dfs.data.dir. Inside this
> directiry
> you'll find a subdirectory ". there will be a file named as "VERSION"  in
> this
> directory. in this file you can see the namespaceID(probably the second
> line).
> change it to match the namespaceID which is there in
> "dfs.name.dir/current/VERSION"
> file.
> 3 - restart the processes.
>
> HTH
>
>
> Warm Regards,
> Tariq
> https://mtariq.jux.com/
> cloudfront.blogspot.com
>
>
> On Fri, Mar 22, 2013 at 8:04 PM, MShaik  wrote:
>
>>  Hi,
>>
>>  DataNode is not started on all the nodes, as tasktracker is started on
>> all the nodes.
>>
>>  please find the below datanode log, please let me know the solution.
>>
>>  2013-03-22 19:52:27,380 INFO org.apache.hadoop.ipc.RPC: Server at
>> n1.hc.com/192.168.1.110:54310 not available yet, Z...
>> 2013-03-22 19:52:29,386 INFO org.apache.hadoop.ipc.Client: Retrying
>> connect to server: n1.hc.com/192.168.1.110:54310. Already tried 0
>> time(s).
>> 2013-03-22 19:52:30,411 INFO org.apache.hadoop.ipc.Client: Retrying
>> connect to server: n1.hc.com/192.168.1.110:54310. Already tried 1
>> time(s).
>> 2013-03-22 19:52:31,416 INFO org.apache.hadoop.ipc.Client: Retrying
>> connect to server: n1.hc.com/192.168.1.110:54310. Already tried 2
>> time(s).
>> 2013-03-22 19:52:32,420 INFO org.apache.hadoop.ipc.Client: Retrying
>> connect to server: n1.hc.com/192.168.1.110:54310. Already tried 3
>> time(s).
>> 2013-03-22 19:52:33,426 INFO org.apache.hadoop.ipc.Client: Retrying
>> connect to server: n1.hc.com/192.168.1.110:54310. Already tried 4
>> time(s).
>> 2013-03-22 19:52:49,162 ERROR
>> org.apache.hadoop.hdfs.server.datanode.DataNode: java.io.IOException:
>> Incompatible namespaceIDs in /home/hduser/hadoopdata: namenode namespaceID
>> = 2050588793; datanode namespaceID = 503772406
>>  at
>> org.apache.hadoop.hdfs.server.datanode.DataStorage.doTransition(DataStorage.java:232)
>>  at
>> org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:147)
>>  at
>> org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:385)
>>  at
>> org.apache.hadoop.hdfs.server.datanode.DataNode.(DataNode.java:299)
>>  at
>> org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1582)
>>  at
>> org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1521)
>>  at
>> org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:1539)
>>  at
>> org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:1665)
>>  at
>> org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:1682)
>>
>>  2013-03-22 19:52:49,168 INFO
>> org.apache.hadoop.hdfs.server.datanode.DataNode: SHUTDOWN_MSG:
>> /
>> SHUTDOWN_MSG: Shutting down DataNode at n4.hc.com/192.168.1.113
>> /
>>
>>
>> Thank's
>>
>> -Original Message-
>> From: Mohammad Tariq 
>> To: user 
>> Sent: Fri, Mar 22, 2013 7:07 pm
>> Subject: Re: Need Help on Hadoop cluster Setup
>>
>>  Hello Munavvar,
>>
>>It depends on your configuration where your DNs and TTs will run.
>> If you have configured all your slaves to run both the processes then they
>> should. If they are not running then there is definitely some problem.
>> Could you please check your DN logs once and see if you find anything
>> unusual there. And you have to copy the files across all the machines.
>>
>>  You can do one more thing just to cross check. Point your web browser
>> to the HDFS web UI(master_machine:9000) to do that.
>>
>>  Warm Regards,
>> Tariq
>> https://mtariq.jux.com/
>>  cloudfront.blogspot.com
>>
>>
>> On Fri, Mar 22, 2013 at 6:44 PM, Munnavar Sk  wrote:
>>
>>>
>>> Hi ,
>>>
>>> I am new to Hadoop and I am fighting with this last 20days, somehow I
>>> got very good stuff on Hadoop.
>>>
>>> But, some question are roaming around me...I hope, I can get the answers
>>> from your end...!
>>>
>>> I was setup a cluster in distributed mode with 5 nodes. I have
>>> configured Namenode and DataNodes and all datannodes are able to loging
>>> from namenode without password.
>>> Hadoop and Java installed on same location in all the Nodes. After
>>> starting the cluster, I was check every node using with "jps" command.
>>> NameNode it was shows that all demons
>

Re: Need Help on Hadoop cluster Setup

[Mohammad Tariq]

have you reformatted the hdfs?if that is the case it was, i think, not
proper.
were the nodes which you attached serving some other cluster earlier?your
logs show that you are facing problems because of mismatch in the IDs of
the NN and the IDs which DNs have. to overcome this problem you can
follow these steps :

1 - Stop all teh DNs.
2 - Go to the directory which is serving as your dfs.data.dir. Inside this
directiry
you'll find a subdirectory ". there will be a file named as "VERSION"  in
this
directory. in this file you can see the namespaceID(probably the second
line).
change it to match the namespaceID which is there in
"dfs.name.dir/current/VERSION"
file.
3 - restart the processes.

HTH


Warm Regards,
Tariq
https://mtariq.jux.com/
cloudfront.blogspot.com


On Fri, Mar 22, 2013 at 8:04 PM, MShaik  wrote:

>  Hi,
>
>  DataNode is not started on all the nodes, as tasktracker is started on
> all the nodes.
>
>  please find the below datanode log, please let me know the solution.
>
>  2013-03-22 19:52:27,380 INFO org.apache.hadoop.ipc.RPC: Server at
> n1.hc.com/192.168.1.110:54310 not available yet, Z...
> 2013-03-22 19:52:29,386 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: n1.hc.com/192.168.1.110:54310. Already tried 0 time(s).
> 2013-03-22 19:52:30,411 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: n1.hc.com/192.168.1.110:54310. Already tried 1 time(s).
> 2013-03-22 19:52:31,416 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: n1.hc.com/192.168.1.110:54310. Already tried 2 time(s).
> 2013-03-22 19:52:32,420 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: n1.hc.com/192.168.1.110:54310. Already tried 3 time(s).
> 2013-03-22 19:52:33,426 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: n1.hc.com/192.168.1.110:54310. Already tried 4 time(s).
> 2013-03-22 19:52:49,162 ERROR
> org.apache.hadoop.hdfs.server.datanode.DataNode: java.io.IOException:
> Incompatible namespaceIDs in /home/hduser/hadoopdata: namenode namespaceID
> = 2050588793; datanode namespaceID = 503772406
>  at
> org.apache.hadoop.hdfs.server.datanode.DataStorage.doTransition(DataStorage.java:232)
>  at
> org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:147)
>  at
> org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:385)
>  at
> org.apache.hadoop.hdfs.server.datanode.DataNode.(DataNode.java:299)
>  at
> org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1582)
>  at
> org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1521)
>  at
> org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:1539)
>  at
> org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:1665)
>  at
> org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:1682)
>
>  2013-03-22 19:52:49,168 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: SHUTDOWN_MSG:
> /
> SHUTDOWN_MSG: Shutting down DataNode at n4.hc.com/192.168.1.113
> /
>
>
> Thank's
>
> -Original Message-
> From: Mohammad Tariq 
> To: user 
> Sent: Fri, Mar 22, 2013 7:07 pm
> Subject: Re: Need Help on Hadoop cluster Setup
>
>  Hello Munavvar,
>
>It depends on your configuration where your DNs and TTs will run.
> If you have configured all your slaves to run both the processes then they
> should. If they are not running then there is definitely some problem.
> Could you please check your DN logs once and see if you find anything
> unusual there. And you have to copy the files across all the machines.
>
>  You can do one more thing just to cross check. Point your web browser to
> the HDFS web UI(master_machine:9000) to do that.
>
>  Warm Regards,
> Tariq
> https://mtariq.jux.com/
>  cloudfront.blogspot.com
>
>
> On Fri, Mar 22, 2013 at 6:44 PM, Munnavar Sk  wrote:
>
>>
>> Hi ,
>>
>> I am new to Hadoop and I am fighting with this last 20days, somehow I got
>> very good stuff on Hadoop.
>>
>> But, some question are roaming around me...I hope, I can get the answers
>> from your end...!
>>
>> I was setup a cluster in distributed mode with 5 nodes. I have configured
>> Namenode and DataNodes and all datannodes are able to loging from namenode
>> without password.
>> Hadoop and Java installed on same location in all the Nodes. After
>> starting the cluster, I was check every node using with "jps" command.
>> NameNode it was shows that all demons
>> running(NameNode,JobTracker,SecondryNameNode).
>> Same process is I applied for Datanodes. But, Some nodes only showing
>> that TaskTracer running, only one node shows that DataNode and TaskTracker
>> runs perfectly.
>> My Question is that the configuration files are required to copy all the
>> nodes which is located in $HADOOP_HOME/conf directory?
>> And why that DataNode is not running 

Re: Need Help on Hadoop cluster Setup

[MShaik]

Hi,


DataNode is not started on all the nodes, as tasktracker is started on all the 
nodes.


please find the below datanode log, please let me know the solution.


2013-03-22 19:52:27,380 INFO org.apache.hadoop.ipc.RPC: Server at 
n1.hc.com/192.168.1.110:54310 not available yet, Z...
2013-03-22 19:52:29,386 INFO org.apache.hadoop.ipc.Client: Retrying connect to 
server: n1.hc.com/192.168.1.110:54310. Already tried 0 time(s).
2013-03-22 19:52:30,411 INFO org.apache.hadoop.ipc.Client: Retrying connect to 
server: n1.hc.com/192.168.1.110:54310. Already tried 1 time(s).
2013-03-22 19:52:31,416 INFO org.apache.hadoop.ipc.Client: Retrying connect to 
server: n1.hc.com/192.168.1.110:54310. Already tried 2 time(s).
2013-03-22 19:52:32,420 INFO org.apache.hadoop.ipc.Client: Retrying connect to 
server: n1.hc.com/192.168.1.110:54310. Already tried 3 time(s).
2013-03-22 19:52:33,426 INFO org.apache.hadoop.ipc.Client: Retrying connect to 
server: n1.hc.com/192.168.1.110:54310. Already tried 4 time(s).
2013-03-22 19:52:49,162 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: 
java.io.IOException: Incompatible namespaceIDs in /home/hduser/hadoopdata: 
namenode namespaceID = 2050588793; datanode namespaceID = 503772406
at 
org.apache.hadoop.hdfs.server.datanode.DataStorage.doTransition(DataStorage.java:232)
at 
org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:147)
at 
org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:385)
at 
org.apache.hadoop.hdfs.server.datanode.DataNode.(DataNode.java:299)
at 
org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1582)
at 
org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1521)
at 
org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:1539)
at 
org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:1665)
at 
org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:1682)


2013-03-22 19:52:49,168 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: 
SHUTDOWN_MSG: 
/
SHUTDOWN_MSG: Shutting down DataNode at n4.hc.com/192.168.1.113
/


Thank's


-Original Message-
From: Mohammad Tariq 
To: user 
Sent: Fri, Mar 22, 2013 7:07 pm
Subject: Re: Need Help on Hadoop cluster Setup


Hello Munavvar,


  It depends on your configuration where your DNs and TTs will run. If you 
have configured all your slaves to run both the processes then they should. If 
they are not running then there is definitely some problem. Could you please 
check your DN logs once and see if you find anything unusual there. And you 
have to copy the files across all the machines.


You can do one more thing just to cross check. Point your web browser to the 
HDFS web UI(master_machine:9000) to do that.


Warm Regards,
Tariq
https://mtariq.jux.com/

cloudfront.blogspot.com





On Fri, Mar 22, 2013 at 6:44 PM, Munnavar Sk  wrote:


Hi ,



 
I am new to Hadoop and I am fighting with this last 20days, somehowI got very 
good stuff on Hadoop.
 
But, some question are roaming around me...I hope, I can getthe answers from 
your end...!
 
I was setup a cluster in distributed mode with 5 nodes. Ihave configured 
Namenode and DataNodes and all datannodes are able to logingfrom namenode 
without password.
Hadoop and Java installed on same location in all the Nodes.After starting the 
cluster, I was check every node using with "jps"command.
NameNode it was shows that all 
demonsrunning(NameNode,JobTracker,SecondryNameNode). 
Same process is I applied for Datanodes. But, Some nodesonly showing that 
TaskTracer running, only one node shows that DataNode andTaskTracker runs 
perfectly.
My Question is that the configuration files are required tocopy all the nodes 
which is located in $HADOOP_HOME/conf directory?
And why that DataNode is not running on remaining nodes?
 
Please clarify this doubts, so that I can able to moveahead... :)
 
Thank you,
M Shaik
--
 
 




 

Re: Need Help on Hadoop cluster Setup

[Mohammad Tariq]

Hello Munavvar,

  It depends on your configuration where your DNs and TTs will run. If
you have configured all your slaves to run both the processes then they
should. If they are not running then there is definitely some problem.
Could you please check your DN logs once and see if you find anything
unusual there. And you have to copy the files across all the machines.

You can do one more thing just to cross check. Point your web browser to
the HDFS web UI(master_machine:9000) to do that.

Warm Regards,
Tariq
https://mtariq.jux.com/
cloudfront.blogspot.com


On Fri, Mar 22, 2013 at 6:44 PM, Munnavar Sk  wrote:

>
> Hi ,
>
> I am new to Hadoop and I am fighting with this last 20days, somehow I got
> very good stuff on Hadoop.
>
> But, some question are roaming around me...I hope, I can get the answers
> from your end...!
>
> I was setup a cluster in distributed mode with 5 nodes. I have configured
> Namenode and DataNodes and all datannodes are able to loging from namenode
> without password.
> Hadoop and Java installed on same location in all the Nodes. After
> starting the cluster, I was check every node using with "jps" command.
> NameNode it was shows that all demons
> running(NameNode,JobTracker,SecondryNameNode).
> Same process is I applied for Datanodes. But, Some nodes only showing that
> TaskTracer running, only one node shows that DataNode and TaskTracker runs
> perfectly.
> My Question is that the configuration files are required to copy all the
> nodes which is located in $HADOOP_HOME/conf directory?
> And why that DataNode is not running on remaining nodes?
>
> Please clarify this doubts, so that I can able to move ahead... :)
>
> Thank you,
> M Shaik
> --
>

Need Help on Hadoop cluster Setup

[Munnavar Sk]

Hi Techies,
 
I am new to Hadoop and I am fighting with this last 20days, somehowI got very 
good stuff on Hadoop.
 
But, some question are roaming around me...I hope, I can getthe answers from 
your end...!
 
I was setup a cluster in distributed mode with 5 nodes. Ihave configured 
Namenode and DataNodes and all datannodes are able to logingfrom namenode 
without password.
Hadoop and Java installed on same location in all the Nodes.After starting the 
cluster, I was check every node using with "jps"command.
NameNode it was shows that all 
demonsrunning(NameNode,JobTracker,SecondryNameNode). 
Same process is I applied for Datanodes. But, Some nodesonly showing that 
TaskTracer running, only one node shows that DataNode andTaskTracker runs 
perfectly.
My Question is that the configuration files are required tocopy all the nodes 
which is located in $HADOOP_HOME/conf directory?
And why that DataNode is not running on remaining nodes?
 
Please clarify this doubts, so that I can able to moveahead... :)
 
Thank you,
M Shaik
--
 

Fwd: Need Help on Hadoop cluster Setup

[Munnavar Sk]

Hi ,



 
I am new to Hadoop and I am fighting with this last 20days, somehowI got very 
good stuff on Hadoop.
 
But, some question are roaming around me...I hope, I can getthe answers from 
your end...!
 
I was setup a cluster in distributed mode with 5 nodes. Ihave configured 
Namenode and DataNodes and all datannodes are able to logingfrom namenode 
without password.
Hadoop and Java installed on same location in all the Nodes.After starting the 
cluster, I was check every node using with "jps"command.
NameNode it was shows that all 
demonsrunning(NameNode,JobTracker,SecondryNameNode). 
Same process is I applied for Datanodes. But, Some nodesonly showing that 
TaskTracer running, only one node shows that DataNode andTaskTracker runs 
perfectly.
My Question is that the configuration files are required tocopy all the nodes 
which is located in $HADOOP_HOME/conf directory?
And why that DataNode is not running on remaining nodes?
 
Please clarify this doubts, so that I can able to moveahead... :)
 
Thank you,
M Shaik
--
 
 

MapReduce Failed and Killed

[Jinchun Kim]

Hi, All.

I'm trying to create category-based splits of Wikipedia dataset(41GB) and
the training data set(5GB) using Mahout.
I'm using following command.

$MAHOUT_HOME/bin/mahout wikipediaDataSetCreator -i wikipedia/chunks -o
wikipediainput -c $MAHOUT_HOME/examples/temp/categories.txt

I had no problem with the training data set, but Hadoop showed following
messages
when I tried to do a same job with Wikipedia dataset,

.
13/03/21 22:31:00 INFO mapred.JobClient:  map 27% reduce 1%
13/03/21 22:40:31 INFO mapred.JobClient:  map 27% reduce 2%
13/03/21 22:58:49 INFO mapred.JobClient:  map 27% reduce 3%
13/03/21 23:22:57 INFO mapred.JobClient:  map 27% reduce 4%
13/03/21 23:46:32 INFO mapred.JobClient:  map 27% reduce 5%
13/03/22 00:27:14 INFO mapred.JobClient:  map 27% reduce 6%
13/03/22 01:06:55 INFO mapred.JobClient:  map 27% reduce 7%
13/03/22 01:14:06 INFO mapred.JobClient:  map 27% reduce 3%
13/03/22 01:15:35 INFO mapred.JobClient: Task Id :
attempt_201303211339_0002_r_00_1, Status : FAILED
Task attempt_201303211339_0002_r_00_1 failed to report status for 1200
seconds. Killing!
13/03/22 01:20:09 INFO mapred.JobClient:  map 27% reduce 4%
13/03/22 01:33:35 INFO mapred.JobClient: Task Id :
attempt_201303211339_0002_m_37_1, Status : FAILED
Task attempt_201303211339_0002_m_37_1 failed to report status for 1228
seconds. Killing!
13/03/22 01:35:12 INFO mapred.JobClient:  map 27% reduce 5%
13/03/22 01:40:38 INFO mapred.JobClient:  map 27% reduce 6%
13/03/22 01:52:28 INFO mapred.JobClient:  map 27% reduce 7%
13/03/22 02:16:27 INFO mapred.JobClient:  map 27% reduce 8%
13/03/22 02:19:02 INFO mapred.JobClient: Task Id :
attempt_201303211339_0002_m_18_1, Status : FAILED
Task attempt_201303211339_0002_m_18_1 failed to report status for 1204
seconds. Killing!
13/03/22 02:49:03 INFO mapred.JobClient:  map 27% reduce 9%
13/03/22 02:52:04 INFO mapred.JobClient:  map 28% reduce 9%


Because I just started to learn how to run Hadoop, I have no idea how to
solve
this problem...
Does anyone have an idea how to handle this weird thing?

-- 
*Jinchun Kim*

Application Master getting killed randomly reporting excess usage of memory

[Krishna Kishore Bonagiri]

Hi,

  I am running a date command using the Distributed Shell example in a loop
of 500 times. It ran successfully all the times except one time where it
gave the following error.

2013-03-22 04:33:25,280 INFO  [main] distributedshell.Client
(Client.java:monitorApplication(605)) - Got application report from ASM
for, appId=222, clientToken=null, appDiagnostics=Application
application_1363938200742_0222 failed 1 times due to AM Container for
appattempt_1363938200742_0222_01 exited with  exitCode: 143 due to:
Container [pid=21141,containerID=container_1363938200742_0222_01_01] is
running beyond virtual memory limits. Current usage: 47.3 Mb of 128 Mb
physical memory used; 611.6 Mb of 268.8 Mb virtual memory used. Killing
container.
Dump of the process-tree for container_1363938200742_0222_01_01 :
|- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS)
SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
|- 21147 21141 21141 21141 (java) 244 12 532643840 11802
/home_/dsadm/yarn/jdk//bin/java -Xmx128m
org.apache.hadoop.yarn.applications.distributedshell.ApplicationMaster
--container_memory 10 --num_containers 2 --priority 0 --shell_command date
|- 21141 8433 21141 21141 (bash) 0 0 108642304 298 /bin/bash -c
/home_/dsadm/yarn/jdk//bin/java -Xmx128m
org.apache.hadoop.yarn.applications.distributedshell.ApplicationMaster
--container_memory 10 --num_containers 2 --priority 0 --shell_command date
1>/tmp/logs/application_1363938200742_0222/container_1363938200742_0222_01_01/AppMaster.stdout
2>/tmp/logs/application_1363938200742_0222/container_1363938200742_0222_01_01/AppMaster.stderr


  Any ideas if it is a known issue? I am using the latest version of
hadoop, i.e. hadoop-2.0.3-alpha.

Thanks,
Kishore

Re: Error starting ResourceManager with hadoop-2.0.3-alpha

[Krishna Kishore Bonagiri]

Thanks Hitesh, it worked. I just copied the capacity-scheduler.xml from
that link and added a property yarn.scheduler.capacity.child.queues similar
to yarn.scheduler.capacity.root.queues which is already there.

Thanks again,
Kishore


On Thu, Mar 21, 2013 at 10:55 PM, Hitesh Shah wrote:

> You are missing the capacity-scheduler.xml in your config directory or in
> your classpath. You can pull a default example copy of it from here:
>
> http://svn.apache.org/viewvc/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/resources/capacity-scheduler.xml
>
> -- Hitesh
>
> On Mar 21, 2013, at 7:57 AM, Krishna Kishore Bonagiri wrote:
>
> > Hi,
> >
> >   I had been using hadoop-2.0.0-alpha since quite sometime which I think
> I downloaded around June/July last year. Now I am trying to shift to the
> latest version hadoop-2.0.3-alpha and while starting it up in the similar
> way as the old one, I am getting the following error in Resource Manger's
> log file and it is not starting up.
> >
> >
> > 2013-03-21 10:25:18,811 FATAL [main] resourcemanager.ResourceManager
> (ResourceManager.java:main(713)) - Error starting ResourceManager
> > java.lang.IllegalStateException: Queue configuration missing child queue
> names for root
> > at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.parseQueue(CapacityScheduler.java:328)
> > at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initializeQueues(CapacityScheduler.java:255)
> > at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.reinitialize(CapacityScheduler.java:220)
> > at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.init(ResourceManager.java:226)
> > at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:710)
> > 2013-03-21 10:25:18,817 INFO  [Thread-5] service.CompositeService
> (CompositeService.java:run(124)) - Error stopping ResourceManager
> > java.lang.NullPointerException
> > at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.stop(ResourceManager.java:590)
> > at
> org.apache.hadoop.yarn.service.CompositeService$CompositeServiceShutdownHook.run(CompositeService.java:122)
> > at
> org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:54)
> > 2013-03-21 10:25:18,817 INFO  [Thread-5] resourcemanager.ResourceManager
> (StringUtils.java:run(595)) - SHUTDOWN_MSG:
> > /
> >
> >
> >
> > I don' t know what is chiled queue names and where to set that
> information. Please help me, what am I missing?
> >
> > Thanks,
> > Kishore
>
>