date:20090821

Re: MR job scheduler

2009-08-21 Thread Harish Mallipeddi

On Fri, Aug 21, 2009 at 12:11 PM, bharath vissapragada <
bharathvissapragada1...@gmail.com> wrote:

> Yes , My doubt is that how is the location of the reducer selected . Is it
> selected arbitrarily or is selected on a particular machine which has
> already the more values (corresponding to the key of that reducer) which
> reduces the cost of transferring data across the network(because already
> many values to that key are on that machine where the map phase
> completed)..
>

I think what you're asking for is whether a ReduceTask is scheduled on a
node which has the largest partition among all the mapoutput partitions
(p1-pN) that the ReduceTask has to fetch in order to do its job. The answer
is "no" - the ReduceTasks are assigned arbitrarily (no such optimization is
done and I think this can really be an optimization only if 1 of your
partitions is heavily skewed for some reason). Also as Amogh pointed out,
the ReduceTasks start fetching their mapoutput-partitions (shuffle phase) as
and when they hear about completed ones. So it would not be possible to
schedule ReduceTasks only on nodes with the largest partitions.

-- 
Harish Mallipeddi
http://blog.poundbang.in

RE: MR job scheduler

2009-08-21 Thread Amogh Vasekar

Let me rephrase,

1. Copy phase starts after reducer initialization, which happens before all 
maps have completed.
2. Which mapper has maximum values for a particular key wont be known until all 
mappers have completed ( to be more precise, until a particular percentage of 
running mappers is completed as we have the "current" maximum value mapper).
Also, there is no rule which says one record can go to only one reducer.

Thanks,
Amogh

-Original Message-
From: bharath vissapragada [mailto:bharathvissapragada1...@gmail.com] 
Sent: Friday, August 21, 2009 12:12 PM
To: common-user@hadoop.apache.org
Subject: Re: MR job scheduler

Yes , My doubt is that how is the location of the reducer selected . Is it
selected arbitrarily or is selected on a particular machine which has
already the more values (corresponding to the key of that reducer) which
reduces the cost of transferring data across the network(because already
many values to that key are on that machine where the map phase completed)..

2009/8/21 Amogh Vasekar 

> Yes, but the copy phase starts with the initialization for a reducer, after
> which it would keep polling for completed map tasks to fetch the respective
> outputs.
>
> -Original Message-
> From: bharath vissapragada [mailto:bharathvissapragada1...@gmail.com]
> Sent: Friday, August 21, 2009 12:00 PM
> To: common-user@hadoop.apache.org
> Subject: Re: MR job scheduler
>
> Amogh
>
> i think Reduce phase starts only when all the map phases are completed .
> Because it needs all the values corresponding to a particular key!
>
> 2009/8/21 Amogh Vasekar 
>
> > I'm not sure that is the case with Hadoop. I think its assigning reduce
> > task to an available tasktracker at any instant; Since a reducer polls JT
> > for completed maps. And if it were the case as you said, a reducer wont
> be
> > initialized until all maps have completed , after which copy phase would
> > start.
> >
> > Thanks,
> > Amogh
> >
> > -Original Message-
> > From: bharath vissapragada [mailto:bharathvissapragada1...@gmail.com]
> > Sent: Friday, August 21, 2009 9:50 AM
> > To: common-user@hadoop.apache.org
> > Subject: Re: MR job scheduler
> >
> > OK i'll be a bit more specific ,
> >
> > Suppose map outputs 100 different keys .
> >
> > Consider a key "K" whose correspoding values may be on N diff datanodes.
> > Consider a datanode "D" which have maximum number of values . So instead
> of
> > moving the values on "D"
> > to other systems , it is useful to bring in the values from other
> datanodes
> > to "D" to minimize the data movement and
> > also the delay. Similar is the case with All the other keys . How does
> the
> > scheduler take care of this ?
> > 2009/8/21 zjffdu 
> >
> > > Add some detials:
> > >
> > > 1. #map is determined by the block size and InputFormat (whether you
> can
> > > want to split or not split)
> > >
> > > 2. The default scheduler for Hadoop is FIFO, and the Fair Scheduler and
> > > Capacity Scheduler are other two options as I know.  JobTracker has the
> > > scheduler.
> > >
> > > 3. Once the map task is done, it will tell its own tasktracker, and the
> > > tasktracker will tell jobtracker, so jobtracker manage all the tasks,
> and
> > > it
> > > will decide how to and when to start the reduce task
> > >
> > >
> > >
> > > -Original Message-
> > > From: Arun C Murthy [mailto:a...@yahoo-inc.com]
> > > Sent: 2009年8月20日 11:41
> > > To: common-user@hadoop.apache.org
> > > Subject: Re: MR job scheduler
> > >
> > >
> > > On Aug 20, 2009, at 9:00 AM, bharath vissapragada wrote:
> > >
> > > > Hi all,
> > > >
> > > > Can anyone tell me how the MR scheduler schedule the MR jobs?
> > > > How does it decide where t create MAP tasks and how many to create.
> > > > Once the MAP tasks are over how does it decide to move the keys to
> the
> > > > reducer efficiently(minimizing the data movement across the network).
> > > > Is there any doc available which describes this scheduling process
> > > > quite
> > > > efficiently
> > > >
> > >
> > > The #maps is decided by the application. The scheduler decides where
> > > to execute them.
> > >
> > > Once the map is done, the reduce tasks connect to the tasktracker (on
> > > the node where the map-task executed) and copies the entire output
> > > over http.
> > >
> > > Arun
> > >
> > >
> >
>

Job design question

2009-08-21 Thread udiw


Hi all,
I'm trying to design an MR job for processing "walks-on-graph" data from
database. The idea is that I have a list of random walks on a graph (which
is unknown).

I have two tables ("walk ids" and "hops"): 
- the first holds the list of random-walk ids, one row per walk, each is
unique id (increasing).
- the second holds, for each walk (identified by the uid) the list of hops
(vertices) traversed in the walk (one hop per row) 
-- these two tables are in a "one-to-many" structure, with the walk uid used
as a foreign key in the hops table.

Meaning, walks should be split between nodes but hops per walk must not.
How would you suggest handling this structure? is it even possible with
DBInputFormat?

Second, assuming it is possible to have this split in an MR job, I would
like to have different reducers that operate on the data during reading (I
want to avoid multiple reading since it can take a long time).
For example, one Reducer should create the actual graph: (Source Node,Dest
Node)-->(num_walks).
Another one should create a length analysis: (Origin Node, Final
Node)-->distance
etc.

Any comments and thoughts will help!
Thanks.
-- 
View this message in context: 
http://www.nabble.com/Job-design-question-tp25076132p25076132.html
Sent from the Hadoop core-user mailing list archive at Nabble.com.

Re: Running hadoop jobs from a client and tuning (was Re: How does hadoop deal with hadoop-site.xml?)

2009-08-21 Thread stephen mulcahy

Hi Amogh,

Thanks for your reply. Some comments below.

Amogh Vasekar wrote:

AFAIK,
hadoop.tmp.dir : Used by NN and DN for directory listings and metadata ( don't
have much info on this )

I've been running some test jobs against a local hadoop cluster from
eclipse using the eclipse plugin. The eclipse plugin manages the client
equivalent of hadoop-site.xml in it's settings. One of the settings
there is hadoop.tmp.dir. When running a hadoop job through eclipse,
there are sometimes items created on the client machine in the
designated hadoop.tmp.dir so there is a client notion of a
hadoop.tmp.dir aswell.

Some of my confusion is arising from trying to get client jobs working
on our cluster while running as someone other than the superuser - what
I thought were permissions errors may have been caused by blindly
copying the hadoop.tmp.dir setting from a cluster node for client use -
instead of setting this to something client specific like
/tmp/hadoop-${user}

java.opts & ulimit : ulimit defines the maximum limit of virtual mem for task launched. java.opts is the amount of memory reserved for a task.
When setting you need to account for memory set aside for hadoop daemons like tasktracker etc.

Right. This is the one tunable I mostly understand :)

mapred.map.tasks and mapred.reduce.tasks : these are job wide configurations
and not per-task configurations for a node. Acts as a hint to the hadoop
framework and explicitly setting them might not be always recommended, unless
you want to run a no-reducer job.

I notice in the Hadoop samples like WordCount, the mappers and reducers
are being explicitly set in the code. Is there any standard approach to
this? Is it better to set this in the client's hadoop-site.xml after
understanding the capacity of the cluster or do developers normally make
their own call on this? As a hadoop cluster admin, should I let
developers worry about this themselves and concentrate on the
per-machine limits below?

mapred.tasktracker.(map | reduce )tasks.maximum : Limit on concurrent tasks running
on a machine, typically set according to cores & memory each map/reduce task
will be using.

Right, so as an admin, these are probably the more interesting ones to
worry.

Also, typically client and datanodes will be the same.

Given my comments above, is this correct?

Thanks,

-stephen

--
Stephen Mulcahy, DI2, Digital Enterprise Research Institute,
NUI Galway, IDA Business Park, Lower Dangan, Galway, Ireland
http://di2.deri.iehttp://webstar.deri.iehttp://sindice.com

How can I copy files from S3 to my local hadoop cluster

2009-08-21 Thread zhang jianfeng

Hi all,


I found hadoop has a filesystem implementation for S3, So how can I copy
files from S3 to my local hadoop cluster ?
Is there any Java API examples?


Thank you.

Jeff Zhang

Help.

2009-08-21 Thread qiu tian

Hi everyone.
I installed hadoop among three pcs. When I ran the command 'start-all.sh', I 
only could start the jobtracker and tasktrackers. I use 192.*.*.x as master and 
use 192.*.*.y and 192.*.*.z as slaves.

The namenode log from the master 192.*.*.x is following like this:

2009-08-18 10:48:44,543 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* 
NameSystem.registerDatanode: node 192.*.*.y:50010 is replaced by 
192.*.*.x:50010 with the same storageID 
DS-1120429845-127.0.0.1-50010-1246697164684
2009-08-18 10:48:44,543 INFO org.apache.hadoop.net.NetworkTopology: Removing a 
node: /default-rack/192.*.*.y:50010
2009-08-18 10:48:44,543 INFO org.apache.hadoop.net.NetworkTopology: Adding a 
new node: /default-rack/192.*.*.x:50010
2009-08-18 10:48:45,932 FATAL org.apache.hadoop.hdfs.StateChange: BLOCK* 
NameSystem.getDatanode: Data node 192.*.*.z:50010 is attempting to report 
storage ID DS-1120429845-127.0.0.1-50010-1246697164684. Node 192.*.*.x:50010 is 
expected to serve this storage.
2009-08-18 10:48:45,932 INFO org.apache.hadoop.ipc.Server: IPC Server handler 8 
on 9000, call blockReport(DatanodeRegistration(192.*.*.z:50010, 
storageID=DS-1120429845-127.0.0.1-50010-1246697164684, infoPort=50075, 
ipcPort=50020), [...@1b8ebe3) from 192.*.*.z:33177: error: 
org.apache.hadoop.hdfs.protocol.UnregisteredDatanodeException: Data node 
192.*.*.z:50010 is attempting to report storage ID 
DS-1120429845-127.0.0.1-50010-1246697164684. Node 192.*.*.x:50010 is expected 
to serve this storage.
org.apache.hadoop.hdfs.protocol.UnregisteredDatanodeException: Data node 
192.*.*.z:50010 is attempting to report storage ID 
DS-1120429845-127.0.0.1-50010-1246697164684. Node 192.*.*.x:50010 is expected 
to serve this storage.
    at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getDatanode(FSNamesystem.java:3800)
    at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.processReport(FSNamesystem.java:2771)
    at 
org.apache.hadoop.hdfs.server.namenode.NameNode.blockReport(NameNode.java:636)
    at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source)
    at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:452)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:892)
2009-08-18 10:48:46,398 FATAL org.apache.hadoop.hdfs.StateChange: BLOCK* 
NameSystem.getDatanode: Data node 192.*.*.y:50010 is attempting to report 
storage ID DS-1120429845-127.0.0.1-50010-1246697164684. Node 192.*.*.x:50010 is 
expected to serve this storage.
2009-08-18 10:48:46,398 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 
on 9000, call blockReport(DatanodeRegistration(192.9.200.y:50010, 
storageID=DS-1120429845-127.0.0.1-50010-1246697164684, infoPort=50075, 
ipcPort=50020), [...@186b634) from 192.*.*.y:47367: error: 
org.apache.hadoop.hdfs.protocol.UnregisteredDatanodeException: Data node 
192.*.*.y:50010 is attempting to report storage ID 
DS-1120429845-127.0.0.1-50010-1246697164684. Node 192.*.*.x:50010 is expected 
to serve this storage.
org.apache.hadoop.hdfs.protocol.UnregisteredDatanodeException: Data node 
192.*.*.y:50010 is attempting to report storage ID 
DS-1120429845-127.0.0.1-50010-1246697164684. Node 192.*.*.x:50010 is expected 
to serve this storage.
    at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getDatanode(FSNamesystem.java:3800)
    at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.processReport(FSNamesystem.java:2771)
    at 
org.apache.hadoop.hdfs.server.namenode.NameNode.blockReport(NameNode.java:636)
    at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source)
    at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:452)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:892)
2009-08-18 10:48:47,000 INFO 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from 
192.*.*.x
~    

The message on the shell looks like this:
192.*.*.x: Exception in thread "main" java.io.IOException: Cannot lock storage 
/home/gaojun/HadoopInstall/tmp/dfs/namesecondary. The directory is already 
locked.
192.*.*.x:     at 
org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.lock(Storage.java:510)
192.*.*.x:     at 
org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.analyzeStorage(Storage.java:363)
192.*.*.x:     at 
org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$CheckpointStorage.recoverCreate(SecondaryNameNode.java:517)
192.*.*.x:     at 
org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.initialize(SecondaryNameNode.java:145)
192.*.*.x:     at 
org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.(SecondaryNameNode.java:115)
192.*.*.x:     at 
org.apache.hadoop.hdfs.server.name

Re: Faster alternative to FSDataInputStream

2009-08-21 Thread Jason Venner

It may be some kind of hostname name or reverse lookup delay, either on the
origination or destination side.

On Thu, Aug 20, 2009 at 10:43 AM, Raghu Angadi wrote:

> Ananth T. Sarathy wrote:
>
>> it's on s3. and it always happens.
>>
>
> I have no experience with S3. You might want to check out S3 forums. It
> can't be normal for S3 either.. there must be something missing
> (configuration, ACLs... ).
>
> Raghu.
>
>
>  Ananth T Sarathy
>>
>>
>> On Wed, Aug 19, 2009 at 4:35 PM, Raghu Angadi 
>> wrote:
>>
>>  Ananth T. Sarathy wrote:
>>>
>>>  Also, I just want to clear... the delay seems to at the intial

 (read = in.read(buf))

  It the file on HDFS (over S3) or S3?
>>>
>>> Does it always happen?
>>>
>>> Raghu.
>>>
>>>
>>>  after the first time into the loop it flies...
>>>
 Ananth T Sarathy


 On Wed, Aug 19, 2009 at 1:58 PM, Raghu Angadi 
 wrote:

  Edward Capriolo wrote:

>  On Wed, Aug 19, 2009 at 11:11 AM, Edward Capriolo <
>
>> edlinuxg...@gmail.com
>>
>>  wrote:
>>>  It would be as fast as underlying filesystem goes.
>>>
>>>  I would not agree with that statement. There is overhead.
 You might be misinterpreting my comment. There is of course some
 over

>>> head
> (at the least the procedure calls).. depending on you underlying
> filesystem,
> there could be extra buffer copies and CRC overhead. But none of that
> explains transfer as slow as 1 MBps (if my interpretation of of results
> is
> correct).
>
> Raghu.
>
>
>
>
>>
>


-- 
Pro Hadoop, a book to guide you from beginner to hadoop mastery,
http://www.amazon.com/dp/1430219424?tag=jewlerymall
www.prohadoopbook.com a community for Hadoop Professionals

Re: Help.

2009-08-21 Thread Jason Venner

It may be that the individual datanodes get different names for their ip
addresses than the namenode does.
It may also be that some subset of your namenode/datanodes do not have write
access to the hdfs storage directories.


On Mon, Aug 17, 2009 at 10:05 PM, qiu tian  wrote:

> Hi everyone.
> I installed hadoop among three pcs. When I ran the command 'start-all.sh',
> I only could start the jobtracker and tasktrackers. I use 192.*.*.x as
> master and use 192.*.*.y and 192.*.*.z as slaves.
>
> The namenode log from the master 192.*.*.x is following like this:
>
> 2009-08-18 10:48:44,543 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
> NameSystem.registerDatanode: node 192.*.*.y:50010 is replaced by
> 192.*.*.x:50010 with the same storageID
> DS-1120429845-127.0.0.1-50010-1246697164684
> 2009-08-18 10:48:44,543 INFO org.apache.hadoop.net.NetworkTopology:
> Removing a node: /default-rack/192.*.*.y:50010
> 2009-08-18 10:48:44,543 INFO org.apache.hadoop.net.NetworkTopology: Adding
> a new node: /default-rack/192.*.*.x:50010
> 2009-08-18 10:48:45,932 FATAL org.apache.hadoop.hdfs.StateChange: BLOCK*
> NameSystem.getDatanode: Data node 192.*.*.z:50010 is attempting to report
> storage ID DS-1120429845-127.0.0.1-50010-1246697164684. Node 192.*.*.x:50010
> is expected to serve this storage.
> 2009-08-18 10:48:45,932 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 8 on 9000, call blockReport(DatanodeRegistration(192.*.*.z:50010,
> storageID=DS-1120429845-127.0.0.1-50010-1246697164684, infoPort=50075,
> ipcPort=50020), [...@1b8ebe3) from 192.*.*.z:33177: error:
> org.apache.hadoop.hdfs.protocol.UnregisteredDatanodeException: Data node
> 192.*.*.z:50010 is attempting to report storage ID
> DS-1120429845-127.0.0.1-50010-1246697164684. Node 192.*.*.x:50010 is
> expected to serve this storage.
> org.apache.hadoop.hdfs.protocol.UnregisteredDatanodeException: Data node
> 192.*.*.z:50010 is attempting to report storage ID
> DS-1120429845-127.0.0.1-50010-1246697164684. Node 192.*.*.x:50010 is
> expected to serve this storage.
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getDatanode(FSNamesystem.java:3800)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.processReport(FSNamesystem.java:2771)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.blockReport(NameNode.java:636)
> at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:452)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:892)
> 2009-08-18 10:48:46,398 FATAL org.apache.hadoop.hdfs.StateChange: BLOCK*
> NameSystem.getDatanode: Data node 192.*.*.y:50010 is attempting to report
> storage ID DS-1120429845-127.0.0.1-50010-1246697164684. Node 192.*.*.x:50010
> is expected to serve this storage.
> 2009-08-18 10:48:46,398 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 0 on 9000, call blockReport(DatanodeRegistration(192.9.200.y:50010,
> storageID=DS-1120429845-127.0.0.1-50010-1246697164684, infoPort=50075,
> ipcPort=50020), [...@186b634) from 192.*.*.y:47367: error:
> org.apache.hadoop.hdfs.protocol.UnregisteredDatanodeException: Data node
> 192.*.*.y:50010 is attempting to report storage ID
> DS-1120429845-127.0.0.1-50010-1246697164684. Node 192.*.*.x:50010 is
> expected to serve this storage.
> org.apache.hadoop.hdfs.protocol.UnregisteredDatanodeException: Data node
> 192.*.*.y:50010 is attempting to report storage ID
> DS-1120429845-127.0.0.1-50010-1246697164684. Node 192.*.*.x:50010 is
> expected to serve this storage.
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getDatanode(FSNamesystem.java:3800)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.processReport(FSNamesystem.java:2771)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.blockReport(NameNode.java:636)
> at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:452)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:892)
> 2009-08-18 10:48:47,000 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
> 192.*.*.x
> ~
>
> The message on the shell looks like this:
> 192.*.*.x: Exception in thread "main" java.io.IOException: Cannot lock
> storage /home/gaojun/HadoopInstall/tmp/dfs/namesecondary. The directory is
> already locked.
> 192.*.*.x: at
> org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.lock(Storage.java:510)
> 192.*.*.x: at
> org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.analyzeStorage(Storage.java:363)
> 192.*.*.x: at
>

Re: Help.

2009-08-21 Thread Sujith Vellat




Sent from my iPhone

On Aug 21, 2009, at 9:25 AM, Jason Venner   
wrote:


It may be that the individual datanodes get different names for  
their ip

addresses than the namenode does.
It may also be that some subset of your namenode/datanodes do not  
have write

access to the hdfs storage directories.


On Mon, Aug 17, 2009 at 10:05 PM, qiu tian  
 wrote:



Hi everyone.
I installed hadoop among three pcs. When I ran the command 'start- 
all.sh',
I only could start the jobtracker and tasktrackers. I use 192.*.*.x  
as

master and use 192.*.*.y and 192.*.*.z as slaves.

The namenode log from the master 192.*.*.x is following like this:

2009-08-18 10:48:44,543 INFO org.apache.hadoop.hdfs.StateChange:  
BLOCK*

NameSystem.registerDatanode: node 192.*.*.y:50010 is replaced by
192.*.*.x:50010 with the same storageID
DS-1120429845-127.0.0.1-50010-1246697164684
2009-08-18 10:48:44,543 INFO org.apache.hadoop.net.NetworkTopology:
Removing a node: /default-rack/192.*.*.y:50010
2009-08-18 10:48:44,543 INFO org.apache.hadoop.net.NetworkTopology:  
Adding

a new node: /default-rack/192.*.*.x:50010
2009-08-18 10:48:45,932 FATAL org.apache.hadoop.hdfs.StateChange:  
BLOCK*
NameSystem.getDatanode: Data node 192.*.*.z:50010 is attempting to  
report
storage ID DS-1120429845-127.0.0.1-50010-1246697164684. Node  
192.*.*.x:50010

is expected to serve this storage.
2009-08-18 10:48:45,932 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 8 on 9000, call blockReport(DatanodeRegistration(192.*.*.z: 
50010,
storageID=DS-1120429845-127.0.0.1-50010-1246697164684,  
infoPort=50075,

ipcPort=50020), [...@1b8ebe3) from 192.*.*.z:33177: error:
org.apache.hadoop.hdfs.protocol.UnregisteredDatanodeException: Data  
node

192.*.*.z:50010 is attempting to report storage ID
DS-1120429845-127.0.0.1-50010-1246697164684. Node 192.*.*.x:50010 is
expected to serve this storage.
org.apache.hadoop.hdfs.protocol.UnregisteredDatanodeException: Data  
node

192.*.*.z:50010 is attempting to report storage ID
DS-1120429845-127.0.0.1-50010-1246697164684. Node 192.*.*.x:50010 is
expected to serve this storage.
   at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getDatanode 
(FSNamesystem.java:3800)

   at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.processReport 
(FSNamesystem.java:2771)

   at
org.apache.hadoop.hdfs.server.namenode.NameNode.blockReport 
(NameNode.java:636)
   at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown  
Source)

   at
sun.reflect.DelegatingMethodAccessorImpl.invoke 
(DelegatingMethodAccessorImpl.java:25)

   at java.lang.reflect.Method.invoke(Method.java:597)
   at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:452)
   at org.apache.hadoop.ipc.Server$Handler.run(Server.java:892)
2009-08-18 10:48:46,398 FATAL org.apache.hadoop.hdfs.StateChange:  
BLOCK*
NameSystem.getDatanode: Data node 192.*.*.y:50010 is attempting to  
report
storage ID DS-1120429845-127.0.0.1-50010-1246697164684. Node  
192.*.*.x:50010

is expected to serve this storage.
2009-08-18 10:48:46,398 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 0 on 9000, call blockReport(DatanodeRegistration 
(192.9.200.y:50010,
storageID=DS-1120429845-127.0.0.1-50010-1246697164684,  
infoPort=50075,

ipcPort=50020), [...@186b634) from 192.*.*.y:47367: error:
org.apache.hadoop.hdfs.protocol.UnregisteredDatanodeException: Data  
node

192.*.*.y:50010 is attempting to report storage ID
DS-1120429845-127.0.0.1-50010-1246697164684. Node 192.*.*.x:50010 is
expected to serve this storage.
org.apache.hadoop.hdfs.protocol.UnregisteredDatanodeException: Data  
node

192.*.*.y:50010 is attempting to report storage ID
DS-1120429845-127.0.0.1-50010-1246697164684. Node 192.*.*.x:50010 is
expected to serve this storage.
   at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getDatanode 
(FSNamesystem.java:3800)

   at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.processReport 
(FSNamesystem.java:2771)

   at
org.apache.hadoop.hdfs.server.namenode.NameNode.blockReport 
(NameNode.java:636)
   at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown  
Source)

   at
sun.reflect.DelegatingMethodAccessorImpl.invoke 
(DelegatingMethodAccessorImpl.java:25)

   at java.lang.reflect.Method.invoke(Method.java:597)
   at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:452)
   at org.apache.hadoop.ipc.Server$Handler.run(Server.java:892)
2009-08-18 10:48:47,000 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log  
from

192.*.*.x
~

The message on the shell looks like this:
192.*.*.x: Exception in thread "main" java.io.IOException: Cannot  
lock
storage /home/gaojun/HadoopInstall/tmp/dfs/namesecondary. The  
directory is

already locked.
192.*.*.x: at
org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.lock 
(Storage.java:510)

192.*.*.x: at
org.apache.hadoop.hdfs.server.common.Storage 
$StorageDirectory.analyzeStorage(Storage.java:363)

192.*.*.x: at
org.apa

NSF/Google/IBM CLuE PI Meeting: October 5, 2009 in Mountain View, California

2009-08-21 Thread Jimmy Lin


==CLuE PI Meeting 2009==

Monday, October 5, 2009
Mountain View, California (Exact Location TBA)

Sponsored by the National Science Foundation, Google, IBM
Organized by the University of Maryland Cloud Computing Center

Website: https://wiki.umiacs.umd.edu/ccc/index.php/CLuE_PI_Meeting_2009
Registration: http://clue2009.eventbrite.com/
  (Early-bird registration ends 8/31)

= What's this event about?

In October 2007, Google and IBM announced the first pilot phase of the 
Academic Cloud Computing Initiative (ACCI), which granted several 
prominent U.S. universities access to a large computer cluster running 
Hadoop, an open source distributed computing platform inspired by 
Google’s file system and MapReduce programming model. In February 2008, 
the ACCI partnered with the National Science Foundation to provide grant 
funding to academic researchers interested in exploring large-data 
applications that could take advantage of this infrastructure. This 
resulted in the creation of the Cluster Exploratory (CLuE) program led 
by Dr. Jim French, which currently funds 14 projects from 17 universities.


Nearing the two year anniversary of this collaboration, the National 
Science Foundation, Google, and IBM will be jointly sponsoring a meeting 
for the CLuE project principal investigators (PIs). This will event will 
be open to the public—in fact, the explicit goal of this event is to 
showcase the exciting research currently underway in academia and 
promote closer ties with the broader "cloud computing" community in the 
bay area.  Register now!


= Who's speaking?

Most of the meeting will consist of plenary talks by the following people:

* Daniel Abadi (Yale University): "HadoopDB An Architectural Hybrid of 
MapReduce and DBMS Technologies for Analytical Workloads"


* Jamie Callan (Carnegie Mellon University): "Topic-Partitioned Search 
Engine Indexes"


* Andrew Connolly (University of Washington): "Scaling the Universe 
through MapReduce"


* Bill Howe (University of Washington) and Claudio Silva (University of 
Utah)


* Chen Li (University of California, Irvine): "Large-Scale Data Cleaning 
Using Hadoop"


* Jimmy Lin (University of Maryland): "Data-Intensive Text Processing 
with MapReduce"


* Sam Madden (MIT): "A Performance and Usability Comparison of Hadoop 
and Relational Database Systems"


* Mihai Pop (University of Maryland): "Commodity Computing in Genomics 
Research"


* Naphtali Rishe (Florida International University): "Experience with 
Geospatial Data in MapReduce"


* Suresh Jagannathan and Ananth Grama (Purdue University): "Relaxed 
Synchronization and Eager Scheduling in MapReduce"


* Stephan Vogel (Carnegie Mellon University)

* Ben Zhao and Xifeng Yan (University of California, Santa Barbara): 
"Scalable Graph Processing in Data Center Environments"


We are also anticipating keynotes from both Google and IBM.

The meeting will be capped off with a poster reception in the early 
evening, where representatives of all CLuE projects will present their 
work in a more informal setting. The speakers above will be joined by 
the follow presenters in the poster session:


* James Allan (University of Massachusetts, Amherst)

* Jason Lawrence (University of Virginia)

* Chaitanya Baru and Sriram Krishnan (San Diego Supercomputer 
Center/University of California, San Diego)

CFP of 3rd Hadoop in China event (Hadoop World:Beijing)

2009-08-21 Thread He Yongqiang


http://www.hadooper.cn/hadoop/cgi-bin/moin.cgi/thirdcfp

Time : Sunday, November 15, 2009
City: Beijing, China

Sponsored by Yahoo!, Cloudera
Organized by hadooper.cn

Website: http://www.hadooper.cn/hadoop/cgi-bin/moin.cgi/thirdcfp

  Sorry for the cross posting. Have a good day!  =

Thanks,
Yongqiang

Re: CFP of 3rd Hadoop in China event (Hadoop World:Beijing)

2009-08-21 Thread He Yongqiang

Hi all,
   
Please do not directly reply this announce email. Please send all your
messages to the secretary email in the CFP.

  Sorry for the cross posting. Have a good day!  =

Thanks,
Yongqiang

On 09-8-22 上午12:21, "He Yongqiang"  wrote:

> 
> http://www.hadooper.cn/hadoop/cgi-bin/moin.cgi/thirdcfp
> 
> Time : Sunday, November 15, 2009
> City: Beijing, China
> 
> Sponsored by Yahoo!, Cloudera
> Organized by hadooper.cn
> 
> Website: http://www.hadooper.cn/hadoop/cgi-bin/moin.cgi/thirdcfp
> 
>   Sorry for the cross posting. Have a good day!  =
> 
> Thanks,
> Yongqiang

Re: MR job scheduler

2009-08-21 Thread bharath vissapragada

I discussed the same doubt in Hbase forums .. Iam pasting the reply i got
(for those who aren't subscribed to that list)

Regarding optimizing the reduce phase(similar to what harish was pointing
out)

I got the following reply .. frm Ryan

"I think people are confused about how optimal map reduces have to be.
Keeping all the data super-local on each machine is not always helping
you, since you have to read via a socket anyways. Going remote doesn't
actually make things that much slower, since on a modern lan ping
times are < 0.1ms.  If your entire cluster is hanging off a single
switch, there is nearly unlimited bandwidth between all nodes
(certainly much higher than any single system could push).  Only once
you go multi-switch then switch-locality (aka rack locality) becomes
important.

Remember, hadoop isn't about the instantaneous speed of any job, but
about running jobs in a highly scalable manner that works on tens or
tens of thousands of nodes. You end up blocking on single machine
limits anyways, and the r=3 of HDFS helps you transcend a single
machine read speed for large files. Keeping the data transfer local in
this case results in lower performance."

Just FYI!
Thanks

On Fri, Aug 21, 2009 at 1:43 PM, Harish Mallipeddi <
harish.mallipe...@gmail.com> wrote:

> On Fri, Aug 21, 2009 at 12:11 PM, bharath vissapragada <
> bharathvissapragada1...@gmail.com> wrote:
>
> > Yes , My doubt is that how is the location of the reducer selected . Is
> it
> > selected arbitrarily or is selected on a particular machine which has
> > already the more values (corresponding to the key of that reducer) which
> > reduces the cost of transferring data across the network(because already
> > many values to that key are on that machine where the map phase
> > completed)..
> >
>

I discussed the same issue on hbase forums and one of its developers
answered my questi

>
> I think what you're asking for is whether a ReduceTask is scheduled on a
> node which has the largest partition among all the mapoutput partitions
> (p1-pN) that the ReduceTask has to fetch in order to do its job. The answer
> is "no" - the ReduceTasks are assigned arbitrarily (no such optimization is
> done and I think this can really be an optimization only if 1 of your
> partitions is heavily skewed for some reason). Also as Amogh pointed out,
> the ReduceTasks start fetching their mapoutput-partitions (shuffle phase)
> as
> and when they hear about completed ones. So it would not be possible to
> schedule ReduceTasks only on nodes with the largest partitions.
>
> --
> Harish Mallipeddi
> http://blog.poundbang.in
>

Re: Exception when starting namenode

2009-08-21 Thread Alex Loddengaard

Have you tampered with anything in dfs.name.dir?  This exception occurs when
your image files in dfs.name.dir are corrupt.  What have you set
dfs.name.dir to?  If it's set to /tmp, then I imagine tmpwatch might have
deleted your HDFS metadata.

Hope this helps.

Alex

On Thu, Aug 20, 2009 at 10:08 PM, Zheng Lv wrote:

> Hello,
>
>I got these exceptions when I started the cluster, any suggestions?
>I used hadoop 0.15.2.
>2009-08-21 12:12:53,463 ERROR org.apache.hadoop.dfs.NameNode:
> java.io.EOFException
>at java.io.DataInputStream.readInt(DataInputStream.java:375)
>at org.apache.hadoop.dfs.FSImage.loadFSImage(FSImage.java:650)
>at org.apache.hadoop.dfs.FSImage.loadFSImage(FSImage.java:614)
>at
> org.apache.hadoop.dfs.FSImage.recoverTransitionRead(FSImage.java:222)
>at
> org.apache.hadoop.dfs.FSDirectory.loadFSImage(FSDirectory.java:76)
>at org.apache.hadoop.dfs.FSNamesystem.(FSNamesystem.java:221)
>at org.apache.hadoop.dfs.NameNode.init(NameNode.java:130)
>at org.apache.hadoop.dfs.NameNode.(NameNode.java:168)
>at org.apache.hadoop.dfs.NameNode.createNameNode(NameNode.java:804)
>at org.apache.hadoop.dfs.NameNode.main(NameNode.java:813)
>Thank you,
>LvZheng
>

Re: How can I copy files from S3 to my local hadoop cluster

2009-08-21 Thread Alex Loddengaard

Hi Jeff,

You can use distcp.  Something like "hadoop distcp s3n//bucket/object
foo/bar".  Read more here: 

Alex

On Fri, Aug 21, 2009 at 3:19 AM, zhang jianfeng  wrote:

> Hi all,
>
>
> I found hadoop has a filesystem implementation for S3, So how can I copy
> files from S3 to my local hadoop cluster ?
> Is there any Java API examples?
>
>
> Thank you.
>
> Jeff Zhang
>

Logging when running locally

2009-08-21 Thread Mark Kerzner

Hi,
when I run either on EC2 or inside the NetBeans IDE, I get a lot of logs.
But when I execute the same job from the command line on my machine, the
_logs in HDFS is empty. Do I need to set some switch?

Thank you,
Mark

Re: Writing to a db with DBOutputFormat spits out IOException Error

2009-08-21 Thread ishwar ramani

For future reference.

This is a class not found exception for the mysql driver.  The
DBOuputFormat converts
it into an IO exception gr.

I had the mysql-connector in both $HADOOP/lib and $HADOOP_CLASSPATH.
That did not help.

I had to pkg the mysql jar into my map reduce jar to fix this problem.

Hope that saves a day for some one!

On Thu, Aug 20, 2009 at 4:52 PM, ishwar ramani wrote:
> Hi,
>
> I am trying to run a simple map reduce that writes the result from the
> reducer to a mysql db.
>
> I Keep getting
>
> 09/08/20 15:44:59 INFO mapred.JobClient: Task Id :
> attempt_200908201210_0013_r_00_0, Status : FAILED
> java.io.IOException: com.mysql.jdbc.Driver
>        at 
> org.apache.hadoop.mapred.lib.db.DBOutputFormat.getRecordWriter(DBOutputFormat.java:162)
>        at 
> org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:435)
>        at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:413)
>        at org.apache.hadoop.mapred.Child.main(Child.java:170)
>
> when the reducer is run.
>
> Here is my code. The user name and password are valid and works fine.
> Is there any way get more info on this exception?
>
>
>
> static class MyWritable implements Writable, DBWritable {
>  long id;
>  String description;
>
>  MyWritable(long mid, String mdescription) {
>    id = mid;
>    description = mdescription;
>  }
>
>  public void readFields(DataInput in) throws IOException {
>    this.id = in.readLong();
>    this.description = Text.readString(in);
>  }
>
>  public void readFields(ResultSet resultSet)
>      throws SQLException {
>    this.id = resultSet.getLong(1);
>    this.description = resultSet.getString(2);
>  }
>
>  public void write(DataOutput out) throws IOException {
>    out.writeLong(this.id);
>    Text.writeString(out, this.description);
>  }
>
>  public void write(PreparedStatement stmt) throws SQLException {
>    stmt.setLong(1, this.id);
>    stmt.setString(2, this.description);
>  }
> }
>
>
>
>
>
>
> public static class Reduce extends MapReduceBase implements
> Reducer {
>  public void reduce(Text key, Iterator values,
> OutputCollector output, Reporter reporter)
> throws IOException {
>    int sum = 0;
>    while (values.hasNext()) {
>      sum += values.next().get();
>    }
>
>    output.collect(new MyWritable(sum, key.toString()), new IntWritable(sum));
>  }
> }
>
>
>
>
>
> public static void main(String[] args) throws Exception {
>  JobConf conf = new JobConf(WordCount.class);
>  conf.setJobName("wordcount");
>
>  conf.setMapperClass(Map.class);
>
>  conf.setReducerClass(Reduce.class);
>
>  DBConfiguration.configureDB(conf, "com.mysql.jdbc.Driver",
> "jdbc:mysql://localhost:8100/testvmysqlsb", "dummy", "pass");
>
>
>  String fields[] = {"id", "description"};
>  DBOutputFormat.setOutput(conf, "funtable", fields);
>
>
>
>  conf.setNumMapTasks(1);
>  conf.setNumReduceTasks(1);
>
>  conf.setMapOutputKeyClass(Text.class);
>  conf.setMapOutputValueClass(IntWritable.class);
>
>
>  conf.setOutputKeyClass(MyWritable.class);
>  conf.setOutputValueClass(IntWritable.class);
>
>  conf.setInputFormat(TextInputFormat.class);
>
>
>
>
>  FileInputFormat.setInputPaths(conf, new Path(args[0]));
>
>
>  JobClient.runJob(conf);
> }
>

Hadoop User Group (Bay Area) - Sep 23rd at Yahoo!

2009-08-21 Thread Dekel Tankel


Hi all,

Thank you for those who attended the meeting this week, I hope you found it 
interesting and helpful.

RSVP is now open for the next monthly Bay Area Hadoop user group at the Yahoo! 
Sunnyvale Campus, planed for Sep 23rd.

Registration is available here
http://www.meetup.com/hadoop/calendar/11166700/?from=list&offset=0

We would like to include a session describing an application using Hadoop 
and/or related technologies. If you are interested to present please send me an 
email with a short description.

Looking forward to seeing you at Sep 23rd



Dekel

Re: Cluster Disk Usage

2009-08-21 Thread Todd Lipcon

Hi Arvind,
Check the source code in DFSAdmin which handles dfsadmin -report. It uses
the same API that the namenode web UI does - I think it's called
getClusterStatus or something if my memory serves me correctly. Here's
example output on my pseudodistributed cluster:

Datanodes available: 1 (1 total, 0 dead)

Name: 127.0.0.1:50010
Decommission Status : Normal
Configured Capacity: 37265149952 (34.71 GB)
DFS Used: 1556480 (1.48 MB)
Non DFS Used: 29779308544 (27.73 GB)
DFS Remaining: 7484284928(6.97 GB)
DFS Used%: 0%
DFS Remaining%: 20.08%
Last contact: Fri Aug 21 13:33:53 PDT 2009

-Todd

2009/8/21 zjffdu 

> Arvind,
>
> You can use this API to get the size of file system used
>
> FileSystem.getUsed();
>
>
> But, I do not find the API for calculate the remaining space. You can write
> some code to create a API,
>
> The remaining disk space = Total of disk space - operate system space -
> FileSystem.getUsed()
>
>
>
> -Original Message-
> From: Arvind Sharma [mailto:arvind...@yahoo.com]
> Sent: 2009年8月20日 16:45
> To: common-user@hadoop.apache.org
> Subject: Re: Cluster Disk Usage
>
> Sorry, I also sent a direct e-mail to one response
>
> there I asked one question - what is the cost of these APIs ???  Are they
> too expensive calls ?  Is the API only going to the NN which stores this
> data ?
>
> Thanks!
> Arvind
>
>
>
>
> 
> From: Arvind Sharma 
> To: common-user@hadoop.apache.org
> Sent: Thursday, August 20, 2009 4:01:02 PM
> Subject: Re: Cluster Disk Usage
>
> Using hadoop-0.19.2
>
>
>
>
> 
> From: Arvind Sharma 
> To: common-user@hadoop.apache.org
> Sent: Thursday, August 20, 2009 3:56:53 PM
> Subject: Cluster Disk Usage
>
> Is there a way to find out how much disk space - overall or per Datanode
> basis - is available before creating a file ?
>
> I am trying to address an issue where the disk got full (config error) and
> the client was not able to create a file on the HDFS.
>
> I want to be able to check if  there space left on the grid before trying
> to
> create the file.
>
> Arvind
>
>
>
>
>

JVM reuse

2009-08-21 Thread roman kolcun

Hello everyone,
I would like to ask whether there is any particular reason why JVM is not
reused by default? According to some of my benchmarks it can speed up
execution significantly. Is there any reason why it is not a good idea to
reuse it?

Roman Kolcun

Re: JVM reuse

2009-08-21 Thread Jason Venner

I think simply because it was a new feature, and it really only helps for
jobs where there are a large number of tasks compared to the available task
slots, coupled with the concern that the subsequent tasks run in the jvm may
not run identically to running in a fresh jvm.

On Fri, Aug 21, 2009 at 6:22 PM, roman kolcun  wrote:

> Hello everyone,
> I would like to ask whether there is any particular reason why JVM is not
> reused by default? According to some of my benchmarks it can speed up
> execution significantly. Is there any reason why it is not a good idea to
> reuse it?
>
> Roman Kolcun
>

-- 
Pro Hadoop, a book to guide you from beginner to hadoop mastery,
http://www.amazon.com/dp/1430219424?tag=jewlerymall
www.prohadoopbook.com a community for Hadoop Professionals

Re: JVM reuse

2009-08-21 Thread roman kolcun

Thanks for the reply. It definitely makes sense to reuse the JVM only if
there are much more tasks than available tasks slots. But isn't it the
common scenario? I thought that MapReduce is generally used to process large
amount of data (=> large amount of tasks) on a cluster with several (tens)
of nodes. So there always will be much more tasks than task slots.

Roman Kolcun

On Sat, Aug 22, 2009 at 2:30 AM, Jason Venner wrote:

> I think simply because it was a new feature, and it really only helps for
> jobs where there are a large number of tasks compared to the available task
> slots, coupled with the concern that the subsequent tasks run in the jvm
> may
> not run identically to running in a fresh jvm.
>
> On Fri, Aug 21, 2009 at 6:22 PM, roman kolcun 
> wrote:
>
> > Hello everyone,
> > I would like to ask whether there is any particular reason why JVM is not
> > reused by default? According to some of my benchmarks it can speed up
> > execution significantly. Is there any reason why it is not a good idea to
> > reuse it?
> >
> > Roman Kolcun
> >
>
>
>
> --
> Pro Hadoop, a book to guide you from beginner to hadoop mastery,
> http://www.amazon.com/dp/1430219424?tag=jewlerymall
> www.prohadoopbook.com a community for Hadoop Professionals
>

Re: MR job scheduler

RE: MR job scheduler

Job design question

Re: Running hadoop jobs from a client and tuning (was Re: How does hadoop deal with hadoop-site.xml?)

How can I copy files from S3 to my local hadoop cluster

Help.

Re: Faster alternative to FSDataInputStream

Re: Help.

Re: Help.

NSF/Google/IBM CLuE PI Meeting: October 5, 2009 in Mountain View, California

CFP of 3rd Hadoop in China event (Hadoop World:Beijing)

Re: CFP of 3rd Hadoop in China event (Hadoop World:Beijing)

Re: MR job scheduler

Re: Exception when starting namenode

Re: How can I copy files from S3 to my local hadoop cluster

Logging when running locally

Re: Writing to a db with DBOutputFormat spits out IOException Error

Hadoop User Group (Bay Area) - Sep 23rd at Yahoo!

Re: Cluster Disk Usage

JVM reuse

Re: JVM reuse

Re: JVM reuse

22 matches

Site Navigation

Mail list logo

Footer information