date:20091013

fuse:The ratio of reported blocks 0.0000 has not reached the threshold 0.9990

2009-10-13 Thread yibo820217


hi,recently,i got some problem.
at first,I start the hadoop
#bin/start-all.sh
then,I mount hdfs to local
#fuse_dfs_wrapper.sh dfs://cent52ip32:9000/ /dfs-test/
In /dfs-test,I do some work,like touch,mkdir,rm
then I look the logs
#cat hadoop-root-namenode-cent52ip32.log 

2009-10-13 22:12:29,733 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit: 

ugi=root,root,bin,daemon,sys,adm,disk,wheelip=/127.0.0.1  
cmd=listStatus  src=/tmp/hadoop-root/mapred/system

  dst=null  perm=null
2009-10-13 22:12:29,736 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 5 on 9000, call delete(/tmp/hadoop-

root/mapred/system, true) from 127.0.0.1:37371: error:
org.apache.hadoop.hdfs.server.namenode.SafeModeException: 

Cannot delete /tmp/hadoop-root/mapred/system. Name node is in safe mode.
The ratio of reported blocks 0. has not reached the threshold 0.9990.
Safe mode will be turned off 
automatically.
org.apache.hadoop.hdfs.server.namenode.SafeModeException: Cannot delete
/tmp/hadoop-root/mapred/system. Name node is 

in safe mode.
The ratio of reported blocks 0. has not reached the threshold 0.9990.
Safe mode will be turned off 

automatically.
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.deleteInternal(FSNamesystem.java:1696)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.delete(FSNamesystem.java:1676)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.delete(NameNode.java:517)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)


I don't know the reason,please help me.
thanks very much.
-- 
View this message in context: 
http://www.nabble.com/fuse%3AThe-ratio-of-reported-blocks-0.-has-not-reached-the-threshold-0.9990-tp25868687p25868687.html
Sent from the Hadoop core-user mailing list archive at Nabble.com.

Re: Announce: Karmasphere Studio for Hadoop

2009-10-13 Thread Bogdan M. Maryniuk

On Mon, Oct 12, 2009 at 5:56 AM, Martin Hall  wrote:
> The license on the beta product is open-ended and we're committed to a free
> version of the product in final release with at least as much functionality
> as you see in the product today.
>
> We're a business and have to find a way to make money. Like other companies
> in this space, we're working on figuring out what those paths look like but
> want to balance commercial offerings with other contributions that anyone
> can pick up and use.

Martin,
thanks to the more light here. So will be there free version as well
all the time? (yes/no: _)

:)

-- 
Kind regards, BM

Things, that are stupid at the beginning, rarely ends up wisely.

Re: fuse:The ratio of reported blocks 0.0000 has not reached the threshold 0.9990

2009-10-13 Thread Eason.Lee

seams no datanode is coming~~no block is reported, so hdfs is in safemode
check if datanode is up, and network between namenode and datanode is fine~~

2009/10/13 yibo820217 

>
> hi,recently,i got some problem.
> at first,I start the hadoop
> #bin/start-all.sh
> then,I mount hdfs to local
> #fuse_dfs_wrapper.sh dfs://cent52ip32:9000/ /dfs-test/
> In /dfs-test,I do some work,like touch,mkdir,rm
> then I look the logs
> #cat hadoop-root-namenode-cent52ip32.log
>
> 2009-10-13 22:12:29,733 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit:
>
> ugi=root,root,bin,daemon,sys,adm,disk,wheelip=/127.0.0.1
> cmd=listStatus  src=/tmp/hadoop-root/mapred/system
>
>  dst=null  perm=null
> 2009-10-13 22:12:29,736 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 5 on 9000, call delete(/tmp/hadoop-
>
> root/mapred/system, true) from 127.0.0.1:37371: error:
> org.apache.hadoop.hdfs.server.namenode.SafeModeException:
>
> Cannot delete /tmp/hadoop-root/mapred/system. Name node is in safe mode.
> The ratio of reported blocks 0. has not reached the threshold 0.9990.
> Safe mode will be turned off
> automatically.
> org.apache.hadoop.hdfs.server.namenode.SafeModeException: Cannot delete
> /tmp/hadoop-root/mapred/system. Name node is
>
> in safe mode.
> The ratio of reported blocks 0. has not reached the threshold 0.9990.
> Safe mode will be turned off
>
> automatically.
>at
>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.deleteInternal(FSNamesystem.java:1696)
>at
>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.delete(FSNamesystem.java:1676)
>at
> org.apache.hadoop.hdfs.server.namenode.NameNode.delete(NameNode.java:517)
>at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>at
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>at java.lang.reflect.Method.invoke(Method.java:597)
>at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
>at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
>at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
>at java.security.AccessController.doPrivileged(Native Method)
>at javax.security.auth.Subject.doAs(Subject.java:396)
>at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
>
>
> I don't know the reason,please help me.
> thanks very much.
> --
> View this message in context:
> http://www.nabble.com/fuse%3AThe-ratio-of-reported-blocks-0.-has-not-reached-the-threshold-0.9990-tp25868687p25868687.html
> Sent from the Hadoop core-user mailing list archive at Nabble.com.
>
>

Re: Delete replicated blocks?

2009-10-13 Thread Stas Oskin

Hi.

Any idea about having replication value at 2?

Was this fixed in the patches for 0.18.3, and if yes, which patch is this?

Thanks.

On Thu, Aug 27, 2009 at 8:18 PM, Stas Oskin  wrote:

> Hi.
>
> Following on this issue, any idea if all the bugs were worked out in 0.20,
> with replication value of 2?
>
> I remember 0.18.3 had some issues with this, and actually caused a lost of
> data to some uni.
>
> Regards.
>
> 2009/8/27 Alex Loddengaard 
>
> I don't know for sure, but running the rebalancer might do this for you.
>>
>> <
>>
>> http://hadoop.apache.org/common/docs/r0.20.0/hdfs_user_guide.html#Rebalancer
>> >
>>
>> Alex
>>
>> On Thu, Aug 27, 2009 at 9:18 AM, Michael Thomas > >wrote:
>>
>> > dfs.replication is only used by the client at the time the files are
>> > written.  Changing this setting will not automatically change the
>> > replication level on existing files.  To do that, you need to use the
>> > hadoop cli:
>> >
>> > hadoop fs -setrep -R 1 /
>> >
>> > --Mike
>> >
>> >
>> > Vladimir Klimontovich wrote:
>> > > This will happen automatically.
>> > > On Aug 27, 2009, at 6:04 PM, Andy Liu wrote:
>> > >
>> > >> I'm running a test Hadoop cluster, which had a dfs.replication value
>> > >> of 3.
>> > >> I'm now running out of disk space, so I've reduced dfs.replication to
>> > >> 1 and
>> > >> restarted my datanodes.  Is there a way to free up the
>> over-replicated
>> > >> blocks, or does this happen automatically at some point?
>> > >>
>> > >> Thanks,
>> > >> Andy
>> > >
>> > > ---
>> > > Vladimir Klimontovich,
>> > > skype: klimontovich
>> > > GoogleTalk/Jabber: klimontov...@gmail.com
>> > > Cell phone: +7926 890 2349
>> > >
>> >
>> >
>>
>

could only be replicated to 0 nodes, instead of 1 (java.io.EOFException)

2009-10-13 Thread tim robertson

Hi all,

I have just done a fresh install of hadoop-0.20.1 on a small cluster
and can't get it to start up.

Could someone please help me diagnose where I might be going wrong?
Below are the snippets of logs from the namenode, a datanode and a
tasktrasker.

I have successfully formated the namenode:
09/10/13 15:18:51 INFO common.Storage: Storage directory /hadoop/name
has been successfully formatted.

Any advice is greatly appreciated and please let me know if there is
more info I can to provide.

Thanks
Tim



The namenode is reporting:
---
2009-10-13 15:00:24,758 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 3 on 8020, call
addBlock(/hadoop/mapred/system/jobtracker.info, DFSClient_-262825200)
from 192.38.28.30:49642: error: java.io.IOException: File
/hadoop/mapred/system/jobtracker.info could only be replicated
to 0 nodes, instead of 1
java.io.IOException: File /hadoop/mapred/system/jobtracker.info could
only be replicated to 0 nodes, instead of 1
at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1267)
at 
org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:422)
---


And the datanodes are reporting repeatedly:
---
2009-10-13 15:20:40,773 INFO org.apache.hadoop.ipc.RPC: Server at
hdfs-master.local/169.254.97.194:8020 not available yet, Z...
2009-10-13 15:20:42,774 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: hdfs-master.local/169.254.97.194:8020. Already
tried 0 time(s).
---


The task trackers are reporting:
---
2009-10-13 15:06:27,034 ERROR org.apache.hadoop.mapred.TaskTracker:
Can not start task tracker because java.io.IOException: Call to
hdfs-master.local/169.254.97.194:50070 failed on local exception:
java.io.EOFException
at org.apache.hadoop.ipc.Client.wrapException(Client.java:774)
at org.apache.hadoop.ipc.Client.call(Client.java:742)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
at org.apache.hadoop.mapred.$Proxy4.getProtocolVersion(Unknown Source)
at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:359)
at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:346)
at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:383)
at org.apache.hadoop.ipc.RPC.waitForProxy(RPC.java:314)
at org.apache.hadoop.ipc.RPC.waitForProxy(RPC.java:291)
at org.apache.hadoop.mapred.TaskTracker.initialize(TaskTracker.java:514)
at org.apache.hadoop.mapred.TaskTracker.(TaskTracker.java:934)
at org.apache.hadoop.mapred.TaskTracker.main(TaskTracker.java:2833)
Caused by: java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:375)
at 
org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:508)
at org.apache.hadoop.ipc.Client$Connection.run(Client.java:446)
---

Re: Could only be replicated to 0 nodes, instead of 1

2009-10-13 Thread Stas Oskin

Hi.

I wonder if there was any progress with this issue?

Regards.

On Thu, May 21, 2009 at 9:01 PM, Raghu Angadi  wrote:

>
> I think you should file a jira on this. Most likely this is what is
> happening :
>
>  * two out of 3 dns can not take anymore blocks.
>  * While picking nodes for a new block, NN mostly skips the third dn as
> well since '# active writes' on it is larger than '2 * avg'.
>  * Even if there is one other block is being written on the 3rd, it is
> still greater than (2 * 1/3).
>
> To test this, if you write just one block to an idle cluster it should
> succeed.
>
> Writing from the client on the 3rd dn succeeds since local node is always
> favored.
>
> This particular problem is not that severe on a large cluster but HDFS
> should do the sensible thing.
>
> Raghu.
>
>
> Stas Oskin wrote:
>
>> Hi.
>>
>> I'm testing Hadoop in our lab, and started getting the following message
>> when trying to copy a file:
>> Could only be replicated to 0 nodes, instead of 1
>>
>> I have the following setup:
>>
>> * 3 machines, 2 of them with only 80GB of space, and 1 with 1.5GB
>> * Two clients are copying files all the time (one of them is the 1.5GB
>> machine)
>> * The replication is set on 2
>> * I let the space on 2 smaller machines to end, to test the behavior
>>
>> Now, one of the clients (the one located on 1.5GB) works fine, and the
>> other
>> one - the external, unable to copy and displays the error + the exception
>> below
>>
>> Any idea if this expected on my scenario? Or how it can be solved?
>>
>> Thanks in advance.
>>
>>
>>
>> 09/05/21 10:51:03 WARN dfs.DFSClient: NotReplicatedYetException sleeping
>> /test/test.bin retries left 1
>>
>> 09/05/21 10:51:06 WARN dfs.DFSClient: DataStreamer Exception:
>> org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>> /test/test.bin could only be replicated to 0 nodes, instead of 1
>>
>>at
>>
>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1123
>> )
>>
>>at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:330)
>>
>>at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
>>
>>at
>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25
>> )
>>
>>at java.lang.reflect.Method.invoke(Method.java:597)
>>
>>at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>
>>at org.apache.hadoop.ipc.Server$Handler.run(Server.java:890)
>>
>>
>>
>>at org.apache.hadoop.ipc.Client.call(Client.java:716)
>>
>>at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:216)
>>
>>at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>>
>>at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>
>>at
>>
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39
>> )
>>
>>at
>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25
>> )
>>
>>at java.lang.reflect.Method.invoke(Method.java:597)
>>
>>at
>>
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82
>> )
>>
>>at
>>
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59
>> )
>>
>>at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>>
>>at
>>
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2450
>> )
>>
>>at
>>
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2333
>> )
>>
>>at
>>
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1800(DFSClient.java:1745
>> )
>>
>>at
>>
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1922
>> )
>>
>>
>>
>> 09/05/21 10:51:06 WARN dfs.DFSClient: Error Recovery for block null bad
>> datanode[0]
>>
>> java.io.IOException: Could not get block locations. Aborting...
>>
>>at
>>
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2153
>> )
>>
>>at
>>
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1400(DFSClient.java:1745
>> )
>>
>>at
>>
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1899
>> )
>>
>>
>

0.20.1 Cluster Setup Problem

2009-10-13 Thread Tejas Lagvankar


Hi,


We are trying to set up a cluster (starting with 2 machines) using the  
new 0.20.1 version.


On the master machine, just after the server starts, the name node  
dies off with the following exception:


2009-10-13 01:22:24,740 ERROR  
org.apache.hadoop.hdfs.server.namenode.NameNode: java.io.IOException:  
Incomplete HDFS URI, no host: hdfs://master_hadoop
at org.apache.hadoop.hdfs.DistributedFileSystem.initialize 
(DistributedFileSystem.java:78)
at org.apache.hadoop.fs.FileSystem.createFileSystem 
(FileSystem.java:1373)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java: 
66)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java: 
1385)

at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:191)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:95)
at org.apache.hadoop.fs.Trash.(Trash.java:62)
at  
org.apache.hadoop.hdfs.server.namenode.NameNode.startTrashEmptier 
(NameNode.java:208)
at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize 
(NameNode.java:204)
at org.apache.hadoop.hdfs.server.namenode.NameNode. 
(NameNode.java:279)
at  
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode 
(NameNode.java:956)
at org.apache.hadoop.hdfs.server.namenode.NameNode.main 
(NameNode.java:965)


Can anyone help ?  Also can anyone send across example configuration  
files for 0.20.1 if they are different than we are using ?


The detail log file is attached along with.




The configuration files are as follows:

MASTER CONFIG
-- conf/masters ---
master_hadoop

-- conf/slaves ---
master_hadoop
slave_hadoop

-- core-site.xml ---



  fs.default.name
  hdfs://master_hadoop



  hadoop.tmp.dir
  /opt/hadoop-0.20.1/tmp


-- hdfs-site.xml ---

  dfs.replication
  2



-- mapred-site.xml ---

  mapred.job.tracker
  tejas_hadoop:9001






SLAVE CONFIG
-- core-site.xml ---

  hadoop.tmp.dir
  /opt/hadoop-0.20.1/tmp/




  fs.default.name
  hdfs://master_hadoop



-- hdfs-site.xml ---

  dfs.replication
  2


-- mapred-site.xml ---

  mapred.job.tracker
  tejas_hadoop:9001




Regards,

Tejas Lagvankar
meette...@umbc.edu
www.umbc.edu/~tej2

Trends in JClouds

2009-10-13 Thread Mikio Uzawa

Hi all,

I posted below three topics:

NTT focuses on the social infrastructure with clouds
A major common paper ASAHI talked the about cloud
NetWorld will dive into the cloud market with Bplats

http://jclouds.wordpress.com/

Thanks,

/mikio uzawa

Re: 0.20.1 Cluster Setup Problem

2009-10-13 Thread Kevin Sweeney

Hi Tejas,
I just upgraded to 20.1 as well and you config all looks the same as mine
except in the core-site.xml I have:


  
fs.default.name
hdfs://localhost:9000
  


Maybe you need to add the port on yours. I haven't seen that error before,
but it seems to be suggesting it can't resolve the host. I'd say
double-check your names and that they resolve.

Hope that helps,
Kevin

On Tue, Oct 13, 2009 at 2:17 PM, Tejas Lagvankar  wrote:

> Hi,
>
>
> We are trying to set up a cluster (starting with 2 machines) using the new
> 0.20.1 version.
>
> On the master machine, just after the server starts, the name node dies off
> with the following exception:
>
> 2009-10-13 01:22:24,740 ERROR
> org.apache.hadoop.hdfs.server.namenode.NameNode: java.io.IOException:
> Incomplete HDFS URI, no host: hdfs://master_hadoop
>at
> org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:78)
>at
> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1373)
>at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66)
>at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1385)
>at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:191)
>at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:95)
>at org.apache.hadoop.fs.Trash.(Trash.java:62)
>at
> org.apache.hadoop.hdfs.server.namenode.NameNode.startTrashEmptier(NameNode.java:208)
>at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:204)
>at
> org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:279)
>at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:956)
>at
> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:965)
>
> Can anyone help ?  Also can anyone send across example configuration files
> for 0.20.1 if they are different than we are using ?
>
> The detail log file is attached along with.
>
>
>
>
> The configuration files are as follows:
>
> MASTER CONFIG
> -- conf/masters ---
> master_hadoop
>
> -- conf/slaves ---
> master_hadoop
> slave_hadoop
>
> -- core-site.xml ---
> 
>
> 
>  fs.default.name
>  hdfs://master_hadoop
> 
>
> 
>  hadoop.tmp.dir
>  /opt/hadoop-0.20.1/tmp
> 
>
> -- hdfs-site.xml ---
> 
>  dfs.replication
>  2
> 
>
>
> -- mapred-site.xml ---
> 
>  mapred.job.tracker
>  tejas_hadoop:9001
> 
>
>
>
>
>
> SLAVE CONFIG
> -- core-site.xml ---
> 
>  hadoop.tmp.dir
>  /opt/hadoop-0.20.1/tmp/
> 
>
>
> 
>  fs.default.name
>  hdfs://master_hadoop
> 
>
>
> -- hdfs-site.xml ---
> 
>  dfs.replication
>  2
> 
>
> -- mapred-site.xml ---
> 
>  mapred.job.tracker
>  tejas_hadoop:9001
> 
>
>
>
> Regards,
>
> Tejas Lagvankar
> meette...@umbc.edu
> www.umbc.edu/~tej2
>
>
>
>
>

Re: 0.20.1 Cluster Setup Problem

2009-10-13 Thread jun hu

I think you should edit the core-site.xml .

(master and slave machine)

> -- core-site.xml ---
> 
>
> 
>  fs.default.name
>  hdfs://master_hadoop: 54310

  

> 
>
> 
>  hadoop.tmp.dir
>  /opt/hadoop-0.20.1/tmp
> 



On Tue, Oct 13, 2009 at 10:17 PM, Tejas Lagvankar  wrote:

> Hi,


>
> We are trying to set up a cluster (starting with 2 machines) using the new
> 0.20.1 version.
>
> On the master machine, just after the server starts, the name node dies off
> with the following exception:
>
> 2009-10-13 01:22:24,740 ERROR
> org.apache.hadoop.hdfs.server.namenode.NameNode: java.io.IOException:
> Incomplete HDFS URI, no host: hdfs://master_hadoop
>at
> org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:78)
>at
> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1373)
>at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66)
>at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1385)
>at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:191)
>at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:95)
>at org.apache.hadoop.fs.Trash.(Trash.java:62)
>at
> org.apache.hadoop.hdfs.server.namenode.NameNode.startTrashEmptier(NameNode.java:208)
>at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:204)
>at
> org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:279)
>at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:956)
>at
> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:965)
>
> Can anyone help ?  Also can anyone send across example configuration files
> for 0.20.1 if they are different than we are using ?
>
> The detail log file is attached along with.
>
>
>
>
> The configuration files are as follows:
>
> MASTER CONFIG
> -- conf/masters ---
> master_hadoop
>
> -- conf/slaves ---
> master_hadoop
> slave_hadoop
>
> -- core-site.xml ---
> 
>
> 
>  fs.default.name
>  hdfs://master_hadoop
> 
>
> 
>  hadoop.tmp.dir
>  /opt/hadoop-0.20.1/tmp
> 
>
> -- hdfs-site.xml ---
> 
>  dfs.replication
>  2
> 
>
>
> -- mapred-site.xml ---
> 
>  mapred.job.tracker
>  tejas_hadoop:9001
> 
>
>
>
>
>
> SLAVE CONFIG
> -- core-site.xml ---
> 
>  hadoop.tmp.dir
>  /opt/hadoop-0.20.1/tmp/
> 
>
>
> 
>  fs.default.name
>  hdfs://master_hadoop
> 
>
>
> -- hdfs-site.xml ---
> 
>  dfs.replication
>  2
> 
>
> -- mapred-site.xml ---
> 
>  mapred.job.tracker
>  tejas_hadoop:9001
> 
>
>
>
> Regards,
>
> Tejas Lagvankar
> meette...@umbc.edu
> www.umbc.edu/~tej2 
>
>
>
>
>


-- 
Best Regards!
胡俊

Optimization of cpu and i/o usage / other bottlenecks?

2009-10-13 Thread Chris Seline

I am using the 0.3 Cloudera scripts to start a Hadoop cluster on EC2 of 
11 c1.xlarge instances (1 master, 10 slaves), that is the biggest 
instance available with 20 compute units and 4x 400gb disks.


I wrote some scripts to test many (100's) of configurations running a 
particular Hive query to try to make it as fast as possible, but no 
matter what I don't seem to be able to get above roughly 45% cpu 
utilization on the slaves, and not more than about 1.5% wait state. I 
have also measured network traffic and there don't seem to be 
bottlenecks there at all.


Here are some typical CPU utilization lines from top on a slave when 
running a query:
Cpu(s): 33.9%us,  7.4%sy,  0.0%ni, 56.8%id,  0.6%wa,  0.0%hi,  0.5%si,  
0.7%st
Cpu(s): 33.6%us,  5.9%sy,  0.0%ni, 58.7%id,  0.9%wa,  0.0%hi,  0.4%si,  
0.5%st
Cpu(s): 33.9%us,  7.2%sy,  0.0%ni, 56.8%id,  0.5%wa,  0.0%hi,  0.6%si,  
1.0%st
Cpu(s): 38.6%us,  8.7%sy,  0.0%ni, 50.8%id,  0.5%wa,  0.0%hi,  0.7%si,  
0.7%st
Cpu(s): 36.8%us,  7.4%sy,  0.0%ni, 53.6%id,  0.4%wa,  0.0%hi,  0.5%si,  
1.3%st


It seems like if tuned properly, I should be able to max out my cpu (or 
my disk) and get roughly twice the performance I am seeing now. None of 
the parameters I am tuning seem to be able to achieve this. Adjusting 
mapred.map.tasks and mapred.reduce.tasks does help somewhat, and setting 
the io.file.buffer.size to 4096 does better than the default, but the 
rest of the values I am testing seem to have little positive  effect.


These are the parameters I am testing, and the values tried:

io.sort.factor=2,3,4,5,10,15,20,25,30,50,100
mapred.job.shuffle.merge.percent=0.10,0.20,0.30,0.40,0.50,0.60,0.70,0.80,0.90,0.93,0.95,0.97,0.98,0.99
io.bytes.per.checksum=256,512,1024,2048,4192
mapred.output.compress=true,false
hive.exec.compress.intermediate=true,false
hive.map.aggr.hash.min.reduction=0.10,0.20,0.30,0.40,0.50,0.60,0.70,0.80,0.90,0.93,0.95,0.97,0.98,0.99
mapred.map.tasks=1,2,3,4,5,6,8,10,12,15,20,25,30,40,50,60,75,100,150,200
mapred.child.java.opts=-Xmx400m,-Xmx500m,-Xmx600m,-Xmx700m,-Xmx800m,-Xmx900m,-Xmx1000m,-Xmx1200m,-Xmx1400m,-Xmx1600m,-Xmx2000m
mapred.reduce.tasks=5,10,15,20,25,30,35,40,50,60,70,80,100,125,150,200
mapred.merge.recordsBeforeProgress=5000,1,2,3
mapred.job.shuffle.input.buffer.percent=0.10,0.20,0.30,0.40,0.50,0.60,0.70,0.80,0.90,0.93,0.95,0.99
io.sort.spill.percent=0.10,0.20,0.30,0.40,0.50,0.60,0.70,0.80,0.90,0.93,0.95,0.99
mapred.job.tracker.handler.count=3,4,5,7,10,15,25
hive.merge.size.per.task=6400,12800,16800,25600,3,4
hive.optimize.ppd=true,false
hive.merge.mapredfiles=false,true
io.sort.record.percent=0.10,0.20,0.30,0.40,0.50,0.60,0.70,0.80,0.90,0.93,0.95,0.97,0.98,0.99
hive.map.aggr.hash.percentmemory=0.10,0.20,0.30,0.40,0.50,0.60,0.70,0.80,0.90,0.93,0.95,0.97,0.98,0.99
mapred.tasktracker.reduce.tasks.maximum=1,2,3,4,5,6,8,10,12,15,20,30
mapred.reduce.parallel.copies=1,2,4,6,8,10,13,16,20,25,30,50
io.seqfile.lazydecompress=true,false
io.sort.mb=20,50,75,100,150,200,250,350,500
mapred.compress.map.output=true,false
io.file.buffer.size=1024,2048,4096,8192,16384,32768,65536,131072,262144
hive.exec.reducers.bytes.per.reducer=10
dfs.datanode.handler.count=1,2,3,4,5,6,8,10,15
mapred.tasktracker.map.tasks.maximum=5,8,12,20

Anyone have any thoughts for other parameters I might try? Am I going 
about this the wrong way? Am I missing some other bottleneck?


thanks

Chris Seline

Re: 0.20.1 Cluster Setup Problem

2009-10-13 Thread Chandan Tamrakar

I think you need to specify the port as well for following port


 fs.default.name
 hdfs://master_hadoop



On Tue, Oct 13, 2009 at 7:17 AM, Tejas Lagvankar  wrote:

> Hi,
>
>
> We are trying to set up a cluster (starting with 2 machines) using the new
> 0.20.1 version.
>
> On the master machine, just after the server starts, the name node dies off
> with the following exception:
>
> 2009-10-13 01:22:24,740 ERROR
> org.apache.hadoop.hdfs.server.namenode.NameNode: java.io.IOException:
> Incomplete HDFS URI, no host: hdfs://master_hadoop
>at
> org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:78)
>at
> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1373)
>at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66)
>at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1385)
>at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:191)
>at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:95)
>at org.apache.hadoop.fs.Trash.(Trash.java:62)
>at
> org.apache.hadoop.hdfs.server.namenode.NameNode.startTrashEmptier(NameNode.java:208)
>at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:204)
>at
> org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:279)
>at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:956)
>at
> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:965)
>
> Can anyone help ?  Also can anyone send across example configuration files
> for 0.20.1 if they are different than we are using ?
>
> The detail log file is attached along with.
>
>
>
>
> The configuration files are as follows:
>
> MASTER CONFIG
> -- conf/masters ---
> master_hadoop
>
> -- conf/slaves ---
> master_hadoop
> slave_hadoop
>
> -- core-site.xml ---
> 
>
> 
>  fs.default.name
>  hdfs://master_hadoop
> 
>
> 
>  hadoop.tmp.dir
>  /opt/hadoop-0.20.1/tmp
> 
>
> -- hdfs-site.xml ---
> 
>  dfs.replication
>  2
> 
>
>
> -- mapred-site.xml ---
> 
>  mapred.job.tracker
>  tejas_hadoop:9001
> 
>
>
>
>
>
> SLAVE CONFIG
> -- core-site.xml ---
> 
>  hadoop.tmp.dir
>  /opt/hadoop-0.20.1/tmp/
> 
>
>
> 
>  fs.default.name
>  hdfs://master_hadoop
> 
>
>
> -- hdfs-site.xml ---
> 
>  dfs.replication
>  2
> 
>
> -- mapred-site.xml ---
> 
>  mapred.job.tracker
>  tejas_hadoop:9001
> 
>
>
>
> Regards,
>
> Tejas Lagvankar
> meette...@umbc.edu
> www.umbc.edu/~tej2 
>
>
>
>
>


-- 
Chandan Tamrakar

Re: 0.20.1 Cluster Setup Problem

2009-10-13 Thread Tejas Lagvankar



I get the same error even if i specify the port number. I have tried  
with port numbers 54310 as well as 9000.



Regards,
Tejas

On Oct 13, 2009, at 12:12 PM, Chandan Tamrakar wrote:


I think you need to specify the port as well for following port


fs.default.name
hdfs://master_hadoop



On Tue, Oct 13, 2009 at 7:17 AM, Tejas Lagvankar   
wrote:



Hi,


We are trying to set up a cluster (starting with 2 machines) using  
the new

0.20.1 version.

On the master machine, just after the server starts, the name node  
dies off

with the following exception:

2009-10-13 01:22:24,740 ERROR
org.apache.hadoop.hdfs.server.namenode.NameNode: java.io.IOException:
Incomplete HDFS URI, no host: hdfs://master_hadoop
  at
org.apache.hadoop.hdfs.DistributedFileSystem.initialize 
(DistributedFileSystem.java:78)

  at
org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java: 
1373)
  at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java: 
66)
  at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java: 
1385)

  at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:191)
  at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:95)
  at org.apache.hadoop.fs.Trash.(Trash.java:62)
  at
org.apache.hadoop.hdfs.server.namenode.NameNode.startTrashEmptier 
(NameNode.java:208)

  at
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize 
(NameNode.java:204)

  at
org.apache.hadoop.hdfs.server.namenode.NameNode. 
(NameNode.java:279)

  at
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode 
(NameNode.java:956)

  at
org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java: 
965)


Can anyone help ?  Also can anyone send across example  
configuration files

for 0.20.1 if they are different than we are using ?

The detail log file is attached along with.




The configuration files are as follows:

MASTER CONFIG
-- conf/masters ---
master_hadoop

-- conf/slaves ---
master_hadoop
slave_hadoop

-- core-site.xml ---



fs.default.name
hdfs://master_hadoop



hadoop.tmp.dir
/opt/hadoop-0.20.1/tmp


-- hdfs-site.xml ---

dfs.replication
2



-- mapred-site.xml ---

mapred.job.tracker
tejas_hadoop:9001






SLAVE CONFIG
-- core-site.xml ---

hadoop.tmp.dir
/opt/hadoop-0.20.1/tmp/




fs.default.name
hdfs://master_hadoop



-- hdfs-site.xml ---

dfs.replication
2


-- mapred-site.xml ---

mapred.job.tracker
tejas_hadoop:9001




Regards,

Tejas Lagvankar
meette...@umbc.edu
www.umbc.edu/~tej2 








--
Chandan Tamrakar


Tejas Lagvankar
meette...@umbc.edu
www.umbc.edu/~tej2

Re: 0.20.1 Cluster Setup Problem

2009-10-13 Thread Kevin Sweeney

did you verify the name resolution?

On Tue, Oct 13, 2009 at 4:34 PM, Tejas Lagvankar  wrote:

>
> I get the same error even if i specify the port number. I have tried with
> port numbers 54310 as well as 9000.
>
>
> Regards,
> Tejas
>
>
> On Oct 13, 2009, at 12:12 PM, Chandan Tamrakar wrote:
>
>  I think you need to specify the port as well for following port
>>
>> 
>> fs.default.name
>> hdfs://master_hadoop
>> 
>>
>>
>> On Tue, Oct 13, 2009 at 7:17 AM, Tejas Lagvankar  wrote:
>>
>>  Hi,
>>>
>>>
>>> We are trying to set up a cluster (starting with 2 machines) using the
>>> new
>>> 0.20.1 version.
>>>
>>> On the master machine, just after the server starts, the name node dies
>>> off
>>> with the following exception:
>>>
>>> 2009-10-13 01:22:24,740 ERROR
>>> org.apache.hadoop.hdfs.server.namenode.NameNode: java.io.IOException:
>>> Incomplete HDFS URI, no host: hdfs://master_hadoop
>>>  at
>>>
>>> org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:78)
>>>  at
>>> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1373)
>>>  at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66)
>>>  at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1385)
>>>  at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:191)
>>>  at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:95)
>>>  at org.apache.hadoop.fs.Trash.(Trash.java:62)
>>>  at
>>>
>>> org.apache.hadoop.hdfs.server.namenode.NameNode.startTrashEmptier(NameNode.java:208)
>>>  at
>>>
>>> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:204)
>>>  at
>>> org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:279)
>>>  at
>>>
>>> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:956)
>>>  at
>>> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:965)
>>>
>>> Can anyone help ?  Also can anyone send across example configuration
>>> files
>>> for 0.20.1 if they are different than we are using ?
>>>
>>> The detail log file is attached along with.
>>>
>>>
>>>
>>>
>>> The configuration files are as follows:
>>>
>>> MASTER CONFIG
>>> -- conf/masters ---
>>> master_hadoop
>>>
>>> -- conf/slaves ---
>>> master_hadoop
>>> slave_hadoop
>>>
>>> -- core-site.xml ---
>>> 
>>>
>>> 
>>> fs.default.name
>>> hdfs://master_hadoop
>>> 
>>>
>>> 
>>> hadoop.tmp.dir
>>> /opt/hadoop-0.20.1/tmp
>>> 
>>>
>>> -- hdfs-site.xml ---
>>> 
>>> dfs.replication
>>> 2
>>> 
>>>
>>>
>>> -- mapred-site.xml ---
>>> 
>>> mapred.job.tracker
>>> tejas_hadoop:9001
>>> 
>>>
>>>
>>>
>>>
>>>
>>> SLAVE CONFIG
>>> -- core-site.xml ---
>>> 
>>> hadoop.tmp.dir
>>> /opt/hadoop-0.20.1/tmp/
>>> 
>>>
>>>
>>> 
>>> fs.default.name
>>> hdfs://master_hadoop
>>> 
>>>
>>>
>>> -- hdfs-site.xml ---
>>> 
>>> dfs.replication
>>> 2
>>> 
>>>
>>> -- mapred-site.xml ---
>>> 
>>> mapred.job.tracker
>>> tejas_hadoop:9001
>>> 
>>>
>>>
>>>
>>> Regards,
>>>
>>> Tejas Lagvankar
>>> meette...@umbc.edu
>>> www.umbc.edu/~tej2 
>>>
>>>
>>>
>>>
>>>
>>>
>>
>> --
>> Chandan Tamrakar
>>
>
> Tejas Lagvankar
> meette...@umbc.edu
> www.umbc.edu/~tej2
>
>
>
>

Re: 0.20.1 Cluster Setup Problem

2009-10-13 Thread Tejas Lagvankar

By name resolution, I assume that you mean the name mentioned in /etc/ 
hosts.  Yes, in the logs, the IP address appears in the beginning.

Correct me if I'm wrong
I will also try with using just the IP's instead of the aliases.

On Oct 13, 2009, at 12:37 PM, Kevin Sweeney wrote:


did you verify the name resolution?

On Tue, Oct 13, 2009 at 4:34 PM, Tejas Lagvankar   
wrote:


I get the same error even if i specify the port number. I have tried  
with port numbers 54310 as well as 9000.



Regards,
Tejas


On Oct 13, 2009, at 12:12 PM, Chandan Tamrakar wrote:

I think you need to specify the port as well for following port


fs.default.name
hdfs://master_hadoop



On Tue, Oct 13, 2009 at 7:17 AM, Tejas Lagvankar   
wrote:


Hi,


We are trying to set up a cluster (starting with 2 machines) using  
the new

0.20.1 version.

On the master machine, just after the server starts, the name node  
dies off

with the following exception:

2009-10-13 01:22:24,740 ERROR
org.apache.hadoop.hdfs.server.namenode.NameNode: java.io.IOException:
Incomplete HDFS URI, no host: hdfs://master_hadoop
 at
org.apache.hadoop.hdfs.DistributedFileSystem.initialize 
(DistributedFileSystem.java:78)

 at
org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1373)
 at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66)
 at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java: 
1385)

 at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:191)
 at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:95)
 at org.apache.hadoop.fs.Trash.(Trash.java:62)
 at
org.apache.hadoop.hdfs.server.namenode.NameNode.startTrashEmptier 
(NameNode.java:208)

 at
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize 
(NameNode.java:204)

 at
org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java: 
279)

 at
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode 
(NameNode.java:956)

 at
org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java: 
965)


Can anyone help ?  Also can anyone send across example configuration  
files

for 0.20.1 if they are different than we are using ?

The detail log file is attached along with.




The configuration files are as follows:

MASTER CONFIG
-- conf/masters ---
master_hadoop

-- conf/slaves ---
master_hadoop
slave_hadoop

-- core-site.xml ---



fs.default.name
hdfs://master_hadoop



hadoop.tmp.dir
/opt/hadoop-0.20.1/tmp


-- hdfs-site.xml ---

dfs.replication
2



-- mapred-site.xml ---

mapred.job.tracker
tejas_hadoop:9001






SLAVE CONFIG
-- core-site.xml ---

hadoop.tmp.dir
/opt/hadoop-0.20.1/tmp/




fs.default.name
hdfs://master_hadoop



-- hdfs-site.xml ---

dfs.replication
2


-- mapred-site.xml ---

mapred.job.tracker
tejas_hadoop:9001




Regards,

Tejas Lagvankar
meette...@umbc.edu
www.umbc.edu/~tej2 







--
Chandan Tamrakar

Tejas Lagvankar
meette...@umbc.edu
www.umbc.edu/~tej2









Tejas Lagvankar
meette...@umbc.edu
www.umbc.edu/~tej2

Re: 0.20.1 Cluster Setup Problem

2009-10-13 Thread Tejas Lagvankar


Hey Kevin,

You were right...
I changed all my aliases to IP addresses. It worked !

Thank you all again :)

Regards,
Tejas

On Oct 13, 2009, at 12:41 PM, Tejas Lagvankar wrote:

By name resolution, I assume that you mean the name mentioned in / 
etc/hosts.  Yes, in the logs, the IP address appears in the beginning.

Correct me if I'm wrong
I will also try with using just the IP's instead of the aliases.

On Oct 13, 2009, at 12:37 PM, Kevin Sweeney wrote:


did you verify the name resolution?

On Tue, Oct 13, 2009 at 4:34 PM, Tejas Lagvankar   
wrote:


I get the same error even if i specify the port number. I have  
tried with port numbers 54310 as well as 9000.



Regards,
Tejas


On Oct 13, 2009, at 12:12 PM, Chandan Tamrakar wrote:

I think you need to specify the port as well for following port


fs.default.name
hdfs://master_hadoop



On Tue, Oct 13, 2009 at 7:17 AM, Tejas Lagvankar   
wrote:


Hi,


We are trying to set up a cluster (starting with 2 machines) using  
the new

0.20.1 version.

On the master machine, just after the server starts, the name node  
dies off

with the following exception:

2009-10-13 01:22:24,740 ERROR
org.apache.hadoop.hdfs.server.namenode.NameNode: java.io.IOException:
Incomplete HDFS URI, no host: hdfs://master_hadoop
at
org.apache.hadoop.hdfs.DistributedFileSystem.initialize 
(DistributedFileSystem.java:78)

at
org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java: 
1373)

at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java: 
1385)

at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:191)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:95)
at org.apache.hadoop.fs.Trash.(Trash.java:62)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.startTrashEmptier 
(NameNode.java:208)

at
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize 
(NameNode.java:204)

at
org.apache.hadoop.hdfs.server.namenode.NameNode. 
(NameNode.java:279)

at
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode 
(NameNode.java:956)

at
org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java: 
965)


Can anyone help ?  Also can anyone send across example  
configuration files

for 0.20.1 if they are different than we are using ?

The detail log file is attached along with.




The configuration files are as follows:

MASTER CONFIG
-- conf/masters ---
master_hadoop

-- conf/slaves ---
master_hadoop
slave_hadoop

-- core-site.xml ---



fs.default.name
hdfs://master_hadoop



hadoop.tmp.dir
/opt/hadoop-0.20.1/tmp


-- hdfs-site.xml ---

dfs.replication
2



-- mapred-site.xml ---

mapred.job.tracker
tejas_hadoop:9001






SLAVE CONFIG
-- core-site.xml ---

hadoop.tmp.dir
/opt/hadoop-0.20.1/tmp/




fs.default.name
hdfs://master_hadoop



-- hdfs-site.xml ---

dfs.replication
2


-- mapred-site.xml ---

mapred.job.tracker
tejas_hadoop:9001




Regards,

Tejas Lagvankar
meette...@umbc.edu
www.umbc.edu/~tej2 







--
Chandan Tamrakar

Tejas Lagvankar
meette...@umbc.edu
www.umbc.edu/~tej2









Tejas Lagvankar
meette...@umbc.edu
www.umbc.edu/~tej2





Tejas Lagvankar
meette...@umbc.edu
www.umbc.edu/~tej2

Re: 0.20.1 Cluster Setup Problem

2009-10-13 Thread Todd Lipcon

Your issue was probably that slave_hadoop and master_hadoop are not valid
host names:

RFCs  mandate that a
hostname's labels may contain only the
ASCIIletters 'a' through 'z'
(case-insensitive), the digits '0' through '9', and
the hyphen. Hostname labels cannot begin or end with a hyphen. No other
symbols, punctuation characters, or blank spaces are permitted.

from http://en.wikipedia.org/wiki/Hostname

-Todd

On Tue, Oct 13, 2009 at 10:01 AM, Tejas Lagvankar  wrote:

> Hey Kevin,
>
> You were right...
> I changed all my aliases to IP addresses. It worked !
>
> Thank you all again :)
>
> Regards,
> Tejas
>
>
> On Oct 13, 2009, at 12:41 PM, Tejas Lagvankar wrote:
>
>  By name resolution, I assume that you mean the name mentioned in
>> /etc/hosts.  Yes, in the logs, the IP address appears in the beginning.
>> Correct me if I'm wrong
>> I will also try with using just the IP's instead of the aliases.
>>
>> On Oct 13, 2009, at 12:37 PM, Kevin Sweeney wrote:
>>
>>  did you verify the name resolution?
>>>
>>> On Tue, Oct 13, 2009 at 4:34 PM, Tejas Lagvankar  wrote:
>>>
>>> I get the same error even if i specify the port number. I have tried with
>>> port numbers 54310 as well as 9000.
>>>
>>>
>>> Regards,
>>> Tejas
>>>
>>>
>>> On Oct 13, 2009, at 12:12 PM, Chandan Tamrakar wrote:
>>>
>>> I think you need to specify the port as well for following port
>>>
>>> 
>>> fs.default.name
>>> hdfs://master_hadoop
>>> 
>>>
>>>
>>> On Tue, Oct 13, 2009 at 7:17 AM, Tejas Lagvankar  wrote:
>>>
>>> Hi,
>>>
>>>
>>> We are trying to set up a cluster (starting with 2 machines) using the
>>> new
>>> 0.20.1 version.
>>>
>>> On the master machine, just after the server starts, the name node dies
>>> off
>>> with the following exception:
>>>
>>> 2009-10-13 01:22:24,740 ERROR
>>> org.apache.hadoop.hdfs.server.namenode.NameNode: java.io.IOException:
>>> Incomplete HDFS URI, no host: hdfs://master_hadoop
>>>at
>>>
>>> org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:78)
>>>at
>>> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1373)
>>>at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66)
>>>at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1385)
>>>at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:191)
>>>at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:95)
>>>at org.apache.hadoop.fs.Trash.(Trash.java:62)
>>>at
>>>
>>> org.apache.hadoop.hdfs.server.namenode.NameNode.startTrashEmptier(NameNode.java:208)
>>>at
>>>
>>> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:204)
>>>at
>>> org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:279)
>>>at
>>>
>>> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:956)
>>>at
>>> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:965)
>>>
>>> Can anyone help ?  Also can anyone send across example configuration
>>> files
>>> for 0.20.1 if they are different than we are using ?
>>>
>>> The detail log file is attached along with.
>>>
>>>
>>>
>>>
>>> The configuration files are as follows:
>>>
>>> MASTER CONFIG
>>> -- conf/masters ---
>>> master_hadoop
>>>
>>> -- conf/slaves ---
>>> master_hadoop
>>> slave_hadoop
>>>
>>> -- core-site.xml ---
>>> 
>>>
>>> 
>>> fs.default.name
>>> hdfs://master_hadoop
>>> 
>>>
>>> 
>>> hadoop.tmp.dir
>>> /opt/hadoop-0.20.1/tmp
>>> 
>>>
>>> -- hdfs-site.xml ---
>>> 
>>> dfs.replication
>>> 2
>>> 
>>>
>>>
>>> -- mapred-site.xml ---
>>> 
>>> mapred.job.tracker
>>> tejas_hadoop:9001
>>> 
>>>
>>>
>>>
>>>
>>>
>>> SLAVE CONFIG
>>> -- core-site.xml ---
>>> 
>>> hadoop.tmp.dir
>>> /opt/hadoop-0.20.1/tmp/
>>> 
>>>
>>>
>>> 
>>> fs.default.name
>>> hdfs://master_hadoop
>>> 
>>>
>>>
>>> -- hdfs-site.xml ---
>>> 
>>> dfs.replication
>>> 2
>>> 
>>>
>>> -- mapred-site.xml ---
>>> 
>>> mapred.job.tracker
>>> tejas_hadoop:9001
>>> 
>>>
>>>
>>>
>>> Regards,
>>>
>>> Tejas Lagvankar
>>> meette...@umbc.edu
>>> www.umbc.edu/~tej2  <
>>> http://www.umbc.edu/%7Etej2>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> --
>>> Chandan Tamrakar
>>>
>>> Tejas Lagvankar
>>> meette...@umbc.edu
>>> www.umbc.edu/~tej2 
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>> Tejas Lagvankar
>> meette...@umbc.edu
>> www.umbc.edu/~tej2 
>>
>>
>>
>>
> Tejas Lagvankar
> meette...@umbc.edu
> www.umbc.edu/~tej2 
>
>
>
>

Re: 0.20.1 Cluster Setup Problem

2009-10-13 Thread Tejas Lagvankar


Thanks Todd,

I never thought of that !!

Regards,
Tejas
On Oct 13, 2009, at 1:50 PM, Todd Lipcon wrote:

Your issue was probably that slave_hadoop and master_hadoop are not  
valid

host names:

RFCs  mandate  
that a

hostname's labels may contain only the
ASCIIletters 'a' through 'z'
(case-insensitive), the digits '0' through '9', and
the hyphen. Hostname labels cannot begin or end with a hyphen. No  
other

symbols, punctuation characters, or blank spaces are permitted.

from http://en.wikipedia.org/wiki/Hostname

-Todd

On Tue, Oct 13, 2009 at 10:01 AM, Tejas Lagvankar   
wrote:



Hey Kevin,

You were right...
I changed all my aliases to IP addresses. It worked !

Thank you all again :)

Regards,
Tejas


On Oct 13, 2009, at 12:41 PM, Tejas Lagvankar wrote:

By name resolution, I assume that you mean the name mentioned in
/etc/hosts.  Yes, in the logs, the IP address appears in the  
beginning.

Correct me if I'm wrong
I will also try with using just the IP's instead of the aliases.

On Oct 13, 2009, at 12:37 PM, Kevin Sweeney wrote:

did you verify the name resolution?


On Tue, Oct 13, 2009 at 4:34 PM, Tejas Lagvankar   
wrote:


I get the same error even if i specify the port number. I have  
tried with

port numbers 54310 as well as 9000.


Regards,
Tejas


On Oct 13, 2009, at 12:12 PM, Chandan Tamrakar wrote:

I think you need to specify the port as well for following port


fs.default.name
hdfs://master_hadoop



On Tue, Oct 13, 2009 at 7:17 AM, Tejas Lagvankar   
wrote:


Hi,


We are trying to set up a cluster (starting with 2 machines)  
using the

new
0.20.1 version.

On the master machine, just after the server starts, the name  
node dies

off
with the following exception:

2009-10-13 01:22:24,740 ERROR
org.apache.hadoop.hdfs.server.namenode.NameNode:  
java.io.IOException:

Incomplete HDFS URI, no host: hdfs://master_hadoop
  at

org.apache.hadoop.hdfs.DistributedFileSystem.initialize 
(DistributedFileSystem.java:78)

  at
org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java: 
1373)

  at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66)
  at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java: 
1385)

  at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:191)
  at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:95)
  at org.apache.hadoop.fs.Trash.(Trash.java:62)
  at

org.apache.hadoop.hdfs.server.namenode.NameNode.startTrashEmptier 
(NameNode.java:208)

  at

org.apache.hadoop.hdfs.server.namenode.NameNode.initialize 
(NameNode.java:204)

  at
org.apache.hadoop.hdfs.server.namenode.NameNode. 
(NameNode.java:279)

  at

org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode 
(NameNode.java:956)

  at
org.apache.hadoop.hdfs.server.namenode.NameNode.main 
(NameNode.java:965)


Can anyone help ?  Also can anyone send across example  
configuration

files
for 0.20.1 if they are different than we are using ?

The detail log file is attached along with.




The configuration files are as follows:

MASTER CONFIG
-- conf/masters ---
master_hadoop

-- conf/slaves ---
master_hadoop
slave_hadoop

-- core-site.xml ---



fs.default.name
hdfs://master_hadoop



hadoop.tmp.dir
/opt/hadoop-0.20.1/tmp


-- hdfs-site.xml ---

dfs.replication
2



-- mapred-site.xml ---

mapred.job.tracker
tejas_hadoop:9001






SLAVE CONFIG
-- core-site.xml ---

hadoop.tmp.dir
/opt/hadoop-0.20.1/tmp/




fs.default.name
hdfs://master_hadoop



-- hdfs-site.xml ---

dfs.replication
2


-- mapred-site.xml ---

mapred.job.tracker
tejas_hadoop:9001




Regards,

Tejas Lagvankar
meette...@umbc.edu
www.umbc.edu/~tej2  <
http://www.umbc.edu/%7Etej2>







--
Chandan Tamrakar

Tejas Lagvankar
meette...@umbc.edu
www.umbc.edu/~tej2 









Tejas Lagvankar
meette...@umbc.edu
www.umbc.edu/~tej2 





Tejas Lagvankar
meette...@umbc.edu
www.umbc.edu/~tej2 






Tejas Lagvankar
meette...@umbc.edu
www.umbc.edu/~tej2

New to hadoop

2009-10-13 Thread shwitzu


Hello,

I am new to hadoop. I have 4 databases with different files in each one of
them. Say, images in one, video, documents etc. My task is to develop a web
service which accepts the keyword from the client and it should come to HDFS
and the output/ the file has to go back to the client.

I have the following questions:

1) How should I start with the design?
2)  Should I upload all the files and create Map, Reduce and Driver code and
once I run my application will it automatically go the file system and get
back the results to me?

Please let me know how I should proceed.

Thanks in anticipation. 
-- 
View this message in context: 
http://www.nabble.com/New-to-hadoop-tp25878445p25878445.html
Sent from the Hadoop core-user mailing list archive at Nabble.com.

Re: DC Hadoop Users Group Meetup - October 16th, 2009 6:30 PM

2009-10-13 Thread Rebecca Owen

Is there a Bay Area (Silicon Valley) user group that gets together?

Thanks!

--- On Thu, 10/8/09, Lalit Kapoor  wrote:

From: Lalit Kapoor 
Subject: DC Hadoop Users Group Meetup - October 16th, 2009 6:30 PM
To: common-user@hadoop.apache.org
Date: Thursday, October 8, 2009, 12:16 PM

Greetings,
    I would like to let everyone know that we will be having our 4th Hadoop
Users Group DC Meetup Friday, October 16, 2009 at 6:30 PM.

    This meetup will have a couple talks about some new non-relational
(#nosql, #altsql, #lesql) databases that are upcoming in the community as
well as a recap of events at Hadoop World in NYC (several Hadoop-DC members
attended and spoke there). Food and refreshments will be provided. There
will be giveaways & prizes.

    As always, attendees of all expertise levels are welcome to talk and ask
questions.

    Please confirm your attendance here:
http://www.meetup.com/Hadoop-DC/calendar/11303458/

    More about our meetups: http://www.meetup.com/Hadoop-DC/

*Location*
Sheraton 
Reston
11810 Sunrise Valley Drive
Reston, VA 20191
(703) 620-9000

*Agenda*

   - Mike Dirolf (http://twitter.com/mdirolf) , from MongoDB, is confirmed
   and will discuss Hadoop and MongoDB
   - Aaron Cordova (http://twitter.com/aaroncordova) will be talking about
   designing scalable column-oriented database
   - Joe Alex (http://twitter.com/joealex) will talk about Cassandra
   - Sharing what happened at the Hadoop World NYC

Sincerely,
     Lalit Kapoor (http://twitter.com/idefine)

Re: DC Hadoop Users Group Meetup - October 16th, 2009 6:30 PM

2009-10-13 Thread Jeff Hammerbacher

Hey Rebecca,

Yes, see http://www.meetup.com/hadoop/.

Regards,
Jeff

On Tue, Oct 13, 2009 at 3:09 PM, Rebecca Owen wrote:

> Is there a Bay Area (Silicon Valley) user group that gets together?
>
> Thanks!
>
> --- On Thu, 10/8/09, Lalit Kapoor  wrote:
>
>
> From: Lalit Kapoor 
> Subject: DC Hadoop Users Group Meetup - October 16th, 2009 6:30 PM
> To: common-user@hadoop.apache.org
> Date: Thursday, October 8, 2009, 12:16 PM
>
>
> Greetings,
> I would like to let everyone know that we will be having our 4th Hadoop
> Users Group DC Meetup Friday, October 16, 2009 at 6:30 PM.
>
> This meetup will have a couple talks about some new non-relational
> (#nosql, #altsql, #lesql) databases that are upcoming in the community as
> well as a recap of events at Hadoop World in NYC (several Hadoop-DC members
> attended and spoke there). Food and refreshments will be provided. There
> will be giveaways & prizes.
>
> As always, attendees of all expertise levels are welcome to talk and
> ask
> questions.
>
> Please confirm your attendance here:
> http://www.meetup.com/Hadoop-DC/calendar/11303458/
>
> More about our meetups: http://www.meetup.com/Hadoop-DC/
>
> *Location*
> Sheraton Reston<
> http://www.meetup.com/Hadoop-DC/venue/1069086/?eventId=11303458&popup=true
> >
> 11810 Sunrise Valley Drive
> Reston, VA 20191
> (703) 620-9000
>
> *Agenda*
>
>- Mike Dirolf (http://twitter.com/mdirolf) , from MongoDB, is confirmed
>and will discuss Hadoop and MongoDB
>- Aaron Cordova (http://twitter.com/aaroncordova) will be talking about
>designing scalable column-oriented database
>- Joe Alex (http://twitter.com/joealex) will talk about Cassandra
>- Sharing what happened at the Hadoop World NYC
>
>
> Sincerely,
>  Lalit Kapoor (http://twitter.com/idefine)
>
>
>
>
>

Database to use with Hadoop

2009-10-13 Thread Mark Kerzner

Hi,
I run Elastic MapReduce. The output of my application is a text file, where
each line is essentially a set of fields. It will fit very nicely into a
simple database, but which database

   1. Is persistent after cluster shutdown;
   2. Can be written to by many reducers?

Amazon SimpleDB could do - but does it work with Hadoop?

Thank you,
Mark

Re: Database to use with Hadoop

2009-10-13 Thread Amandeep Khurana

You can put into Hbase. Or you can use the DBOutputFormat and interface with
an RDBMS.


Amandeep Khurana
Computer Science Graduate Student
University of California, Santa Cruz


On Tue, Oct 13, 2009 at 3:12 PM, Mark Kerzner  wrote:

> Hi,
> I run Elastic MapReduce. The output of my application is a text file, where
> each line is essentially a set of fields. It will fit very nicely into a
> simple database, but which database
>
>   1. Is persistent after cluster shutdown;
>   2. Can be written to by many reducers?
>
> Amazon SimpleDB could do - but does it work with Hadoop?
>
> Thank you,
> Mark
>

Re: Database to use with Hadoop

2009-10-13 Thread Jeff Hammerbacher

Hey Mark,

You will probably get some mileage from
http://developer.amazonwebservices.com/connect/entry.jspa?externalID=2571.

Regards,
Jeff

On Tue, Oct 13, 2009 at 3:19 PM, Amandeep Khurana  wrote:

> You can put into Hbase. Or you can use the DBOutputFormat and interface
> with
> an RDBMS.
>
>
> Amandeep Khurana
> Computer Science Graduate Student
> University of California, Santa Cruz
>
>
> On Tue, Oct 13, 2009 at 3:12 PM, Mark Kerzner 
> wrote:
>
> > Hi,
> > I run Elastic MapReduce. The output of my application is a text file,
> where
> > each line is essentially a set of fields. It will fit very nicely into a
> > simple database, but which database
> >
> >   1. Is persistent after cluster shutdown;
> >   2. Can be written to by many reducers?
> >
> > Amazon SimpleDB could do - but does it work with Hadoop?
> >
> > Thank you,
> > Mark
> >
>

Re: Database to use with Hadoop

2009-10-13 Thread Mark Kerzner

Thank you, all. It looks like SimpleDB may be good enough for my needs. The
forums claim that you can write to it from all reducers at once, being that
it is highly optimized for concurrent access.

On Tue, Oct 13, 2009 at 5:30 PM, Jeff Hammerbacher wrote:

> Hey Mark,
>
> You will probably get some mileage from
> http://developer.amazonwebservices.com/connect/entry.jspa?externalID=2571.
>
> Regards,
> Jeff
>
> On Tue, Oct 13, 2009 at 3:19 PM, Amandeep Khurana 
> wrote:
>
> > You can put into Hbase. Or you can use the DBOutputFormat and interface
> > with
> > an RDBMS.
> >
> >
> > Amandeep Khurana
> > Computer Science Graduate Student
> > University of California, Santa Cruz
> >
> >
> > On Tue, Oct 13, 2009 at 3:12 PM, Mark Kerzner 
> > wrote:
> >
> > > Hi,
> > > I run Elastic MapReduce. The output of my application is a text file,
> > where
> > > each line is essentially a set of fields. It will fit very nicely into
> a
> > > simple database, but which database
> > >
> > >   1. Is persistent after cluster shutdown;
> > >   2. Can be written to by many reducers?
> > >
> > > Amazon SimpleDB could do - but does it work with Hadoop?
> > >
> > > Thank you,
> > > Mark
> > >
> >
>

How to get IP address of the machine where map task runs

2009-10-13 Thread Long Van Nguyen Dinh

Hi all,

Given a map task, I need to know the IP address of the machine where
that task is running. Is there any existing method to get that
information?

Thank you,
Van

Re: Optimization of cpu and i/o usage / other bottlenecks?

2009-10-13 Thread Jason Venner

are your network interface or the namenode/jobtracker/datanodes saturated


On Tue, Oct 13, 2009 at 9:05 AM, Chris Seline  wrote:

> I am using the 0.3 Cloudera scripts to start a Hadoop cluster on EC2 of 11
> c1.xlarge instances (1 master, 10 slaves), that is the biggest instance
> available with 20 compute units and 4x 400gb disks.
>
> I wrote some scripts to test many (100's) of configurations running a
> particular Hive query to try to make it as fast as possible, but no matter
> what I don't seem to be able to get above roughly 45% cpu utilization on the
> slaves, and not more than about 1.5% wait state. I have also measured
> network traffic and there don't seem to be bottlenecks there at all.
>
> Here are some typical CPU utilization lines from top on a slave when
> running a query:
> Cpu(s): 33.9%us,  7.4%sy,  0.0%ni, 56.8%id,  0.6%wa,  0.0%hi,  0.5%si,
>  0.7%st
> Cpu(s): 33.6%us,  5.9%sy,  0.0%ni, 58.7%id,  0.9%wa,  0.0%hi,  0.4%si,
>  0.5%st
> Cpu(s): 33.9%us,  7.2%sy,  0.0%ni, 56.8%id,  0.5%wa,  0.0%hi,  0.6%si,
>  1.0%st
> Cpu(s): 38.6%us,  8.7%sy,  0.0%ni, 50.8%id,  0.5%wa,  0.0%hi,  0.7%si,
>  0.7%st
> Cpu(s): 36.8%us,  7.4%sy,  0.0%ni, 53.6%id,  0.4%wa,  0.0%hi,  0.5%si,
>  1.3%st
>
> It seems like if tuned properly, I should be able to max out my cpu (or my
> disk) and get roughly twice the performance I am seeing now. None of the
> parameters I am tuning seem to be able to achieve this. Adjusting
> mapred.map.tasks and mapred.reduce.tasks does help somewhat, and setting the
> io.file.buffer.size to 4096 does better than the default, but the rest of
> the values I am testing seem to have little positive  effect.
>
> These are the parameters I am testing, and the values tried:
>
> io.sort.factor=2,3,4,5,10,15,20,25,30,50,100
>
> mapred.job.shuffle.merge.percent=0.10,0.20,0.30,0.40,0.50,0.60,0.70,0.80,0.90,0.93,0.95,0.97,0.98,0.99
> io.bytes.per.checksum=256,512,1024,2048,4192
> mapred.output.compress=true,false
> hive.exec.compress.intermediate=true,false
>
> hive.map.aggr.hash.min.reduction=0.10,0.20,0.30,0.40,0.50,0.60,0.70,0.80,0.90,0.93,0.95,0.97,0.98,0.99
> mapred.map.tasks=1,2,3,4,5,6,8,10,12,15,20,25,30,40,50,60,75,100,150,200
>
> mapred.child.java.opts=-Xmx400m,-Xmx500m,-Xmx600m,-Xmx700m,-Xmx800m,-Xmx900m,-Xmx1000m,-Xmx1200m,-Xmx1400m,-Xmx1600m,-Xmx2000m
> mapred.reduce.tasks=5,10,15,20,25,30,35,40,50,60,70,80,100,125,150,200
> mapred.merge.recordsBeforeProgress=5000,1,2,3
>
> mapred.job.shuffle.input.buffer.percent=0.10,0.20,0.30,0.40,0.50,0.60,0.70,0.80,0.90,0.93,0.95,0.99
>
> io.sort.spill.percent=0.10,0.20,0.30,0.40,0.50,0.60,0.70,0.80,0.90,0.93,0.95,0.99
> mapred.job.tracker.handler.count=3,4,5,7,10,15,25
>
> hive.merge.size.per.task=6400,12800,16800,25600,3,4
> hive.optimize.ppd=true,false
> hive.merge.mapredfiles=false,true
>
> io.sort.record.percent=0.10,0.20,0.30,0.40,0.50,0.60,0.70,0.80,0.90,0.93,0.95,0.97,0.98,0.99
>
> hive.map.aggr.hash.percentmemory=0.10,0.20,0.30,0.40,0.50,0.60,0.70,0.80,0.90,0.93,0.95,0.97,0.98,0.99
> mapred.tasktracker.reduce.tasks.maximum=1,2,3,4,5,6,8,10,12,15,20,30
> mapred.reduce.parallel.copies=1,2,4,6,8,10,13,16,20,25,30,50
> io.seqfile.lazydecompress=true,false
> io.sort.mb=20,50,75,100,150,200,250,350,500
> mapred.compress.map.output=true,false
> io.file.buffer.size=1024,2048,4096,8192,16384,32768,65536,131072,262144
> hive.exec.reducers.bytes.per.reducer=10
> dfs.datanode.handler.count=1,2,3,4,5,6,8,10,15
> mapred.tasktracker.map.tasks.maximum=5,8,12,20
>
> Anyone have any thoughts for other parameters I might try? Am I going about
> this the wrong way? Am I missing some other bottleneck?
>
> thanks
>
> Chris Seline
>



-- 
Pro Hadoop, a book to guide you from beginner to hadoop mastery,
http://www.amazon.com/dp/1430219424?tag=jewlerymall
www.prohadoopbook.com a community for Hadoop Professionals

DataNode is shutting down

2009-10-13 Thread yibo820217


hi all,there is my problem.
when add a datanode to hadoop,the way is;
1.in namenode add the new datanode to conf/slave

2.in new datanode cd $HADOOP_HOME then
$ bin/hadoop-daemon.sh start datanode 
$ bin/hadoop-daemon.sh start tasktracker

3.in namenode,
$bin/hadoop  balancer

and the new datanode is added to hadoop,but a old datanode is shutdown
and here is the logs in datanode which is shutdown

2009-10-14 13:16:30,604 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: BlockReport of 3 blocks got
processed in 5 msecs
2009-10-14 13:48:44,395 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeCommand action:
DNA_REGISTER
2009-10-14 13:48:47,402 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeCommand action:
DNA_REGISTER
2009-10-14 13:48:50,403 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeCommand action:
DNA_REGISTER
2009-10-14 13:48:53,407 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeCommand action:
DNA_REGISTER
2009-10-14 13:48:56,418 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeCommand action:
DNA_REGISTER
2009-10-14 13:48:59,415 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeCommand action:
DNA_REGISTER
2009-10-14 13:49:02,420 WARN
org.apache.hadoop.hdfs.server.datanode.DataNode: DataNode is shutting down:
org.apache.hadoop.ipc.RemoteException:
org.apache.hadoop.hdfs.protocol.UnregisteredDatanodeException: Data node
100.207.100.33:50010 is attempting to report storage ID
DS-1277539940-100.207.100.33-50010-1255486116525. Node 100.207.100.25:50010
is expected to serve this storage.
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getDatanode(FSNamesystem.java:3914)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.processReport(FSNamesystem.java:2885)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.blockReport(NameNode.java:715)
at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)

at org.apache.hadoop.ipc.Client.call(Client.java:739)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
at $Proxy4.blockReport(Unknown Source)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:756)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1186)
at java.lang.Thread.run(Thread.java:619)

2009-10-14 13:49:02,527 INFO org.apache.hadoop.ipc.Server: Stopping server
on 50020
2009-10-14 13:49:02,528 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 1 on 50020: exiting
2009-10-14 13:49:02,529 INFO org.apache.hadoop.ipc.Server: Stopping IPC
Server Responder
2009-10-14 13:49:02,529 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: Waiting for threadgroup to
exit, active threads is 1
2009-10-14 13:49:02,528 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 0 on 50020: exiting
2009-10-14 13:49:02,528 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 2 on 50020: exiting
2009-10-14 13:49:02,529 WARN
org.apache.hadoop.hdfs.server.datanode.DataNode:
DatanodeRegistration(10.207.0.33:50010,
storageID=DS-1277539940-100.207.100.33-50010-1255486116525, infoPort=50075,
ipcPort=50020):DataXceiveServer:
java.nio.channels.AsynchronousCloseException
at
java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:185)
at
sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:152)
at
sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:84)
at
org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:130)
at java.lang.Thread.run(Thread.java:619)

2009-10-14 13:49:02,530 INFO org.apache.hadoop.ipc.Server: Stopping IPC
Server listener on 50020
2009-10-14 13:49:03,267 INFO
org.apache.hadoop.hdfs.server.datanode.DataBlockScanner: Exiting
DataBlockScanner thread.
2009-10-14 13:49:03,530 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: Waiting for threadgroup to
exit, active threads is 0
2009-10-14 13:49:03,635 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode:
DatanodeRegistration(10.207.0.33:50010,
storageID=DS-1277539940-100.207.100.33-50010-1255486116525, infoPort=50075,
ipcPort=50020):Finishing DataNode in:
FSDataset{dirpath='/data0/hadoop/hadoopfs/data/current'}
2009-10-14 13:49:03,635 INFO org.apache.hadoop.ipc.Server: Stopping server
on 50020
2009-10-14 13:49:03,635 INFO
org.apache.hadoop.hdfs.server.da

fuse:The ratio of reported blocks 0.0000 has not reached the threshold 0.9990

Re: Announce: Karmasphere Studio for Hadoop

Re: fuse:The ratio of reported blocks 0.0000 has not reached the threshold 0.9990

Re: Delete replicated blocks?

could only be replicated to 0 nodes, instead of 1 (java.io.EOFException)

Re: Could only be replicated to 0 nodes, instead of 1

0.20.1 Cluster Setup Problem

Trends in JClouds

Re: 0.20.1 Cluster Setup Problem

Re: 0.20.1 Cluster Setup Problem

Optimization of cpu and i/o usage / other bottlenecks?

Re: 0.20.1 Cluster Setup Problem

Re: 0.20.1 Cluster Setup Problem

Re: 0.20.1 Cluster Setup Problem

Re: 0.20.1 Cluster Setup Problem

Re: 0.20.1 Cluster Setup Problem

Re: 0.20.1 Cluster Setup Problem

Re: 0.20.1 Cluster Setup Problem

New to hadoop

Re: DC Hadoop Users Group Meetup - October 16th, 2009 6:30 PM

Re: DC Hadoop Users Group Meetup - October 16th, 2009 6:30 PM

Database to use with Hadoop

Re: Database to use with Hadoop

Re: Database to use with Hadoop

Re: Database to use with Hadoop

How to get IP address of the machine where map task runs

Re: Optimization of cpu and i/o usage / other bottlenecks?

DataNode is shutting down

28 matches

Site Navigation

Mail list logo

Footer information