Re: Is there any smart ways to give arguments to mappers reducers from a main job?
correct.
Just like this:
Configuration conf = new Configuration();
conf.setStrings(test, test);
Job job = new Job(conf, job name);
On Thu, Feb 10, 2011 at 6:42 PM, Harsh J qwertyman...@gmail.com wrote:
Your 'Job' must reference this Configuration object for it to take
those values. If it does not know about it, it would not work,
logically :-)
For example, create your Configuration and set things into it, and
only then do new Job(ConfigurationObj) to make it use your configured
object for this job.
On Thu, Feb 10, 2011 at 3:19 PM, Jun Young Kim juneng...@gmail.com
wrote:
Hi, all
in my job, I wanna pass some arguments to mappers and reducers from a
main
job.
I googled some references to do that by using Configuration.
but, it's not working.
code)
job)
Configuration conf = new Configuration();
conf.set(test, value);
mapper)
doMap() extends Mapper... {
System.out.println(context.getConfiguration.get(test));
/// -- this printed out null
}
How could I do that to make it working?--
Junyoung Kim (juneng...@gmail.com)
--
Harsh J
www.harshj.com
--
-李平
Re: How do I insert a new node while running a MapReduce hadoop?
of course you can. What is the node type, datanode?job tracker?task tracker? Let's say you are trying to add a datanode. You can modify the xml file let the datanode pointed to the NameNode, JobTracker, TaskTracker. property namefs.default.name/name valuehdfs://:9000//value /property property namemapred.job.tracker/name valueip:port/value descriptionThe host and port that the MapReduce job tracker runs at. If local, then jobs are run in-process as a single map and reduce task. /description /property In most cases, the tasktracker and datanode are running on the same machine (to get the best performance). After doing this, you can start the hdfs by command start-dfs.sh On Fri, Feb 11, 2011 at 11:13 AM, Sandro Simas sandro.csi...@gmail.comwrote: Hi, i started using hadoop now and I'm doing some tests on a cluster of three machines. I wanted to insert a new node after the MapReduce started, is this possible? How do I? -- -李平
Re: Hadoop Multi user - Cluster Setup
If can check this property in hdfs-site.xml
property
namedfs.permissions/name
valuetrue/value
description
If true, enable permission checking in HDFS.
If false, permission checking is turned off,
but all other behavior is unchanged.
Switching from one parameter value to the other does not change the
mode,
owner or group of files or directories.
/description
/property
You can disable this option.
the second way is:
running the command in hadoop. hadoop fs -chmod o+w /
It has the same effect with first one
On Thu, Feb 10, 2011 at 3:12 AM, Kumar, Amit H. ahku...@odu.edu wrote:
Dear All,
I am trying to setup Hadoop for multiple users in a class, on our cluster.
For some reason I don't seem to get it right. If only one user is running it
works great.
I would want to have all of the users submit a Hadoop job to the existing
DataNode and on the cluster, not sure if this is right.
Do I need to start a DataNode for every user, if so I was not able to do
because I ran into issues of port already being used.
Please advise. Below are few of the config files.
Also I have tired searching for other documents, that tell us to create a
user Hadoop and a group Hadoop and then start the daemons as Hadoop
user. This didn't work for me as well. I am sure I am doing something
wrong. Could anyone please thrown in some more ideas.
=List of env changed in Hadoop-env.sh:
export HADOOP_LOG_DIR=/scratch/$USER/hadoop-logs
export HADOOP_PID_DIR=/scratch/$USER/.var/hadoop/pids
#cat core-site.xml
configuration
property
namefs.default.name/name
valuehdfs://frontend:9000/value
/property
property
namehadoop.tmp.dir/name
value/scratch/${user.name}/hadoop-FS/value
descriptionA base for other temporary directories./description
/property
/configuration
# cat hdfs-site.xml
configuration
property
namedfs.replication/name
value1/value
/property
property
namedfs.name.dir/name
value/scratch/${user.name}/.hadoop/.transaction/.edits/value
/property
/configuration
# cat mapred-site.xml
configuration
property
namemapred.job.tracker/name
valuefrontend:9001/value
/property
property
namemapreduce.tasktracker.map.tasks.maximum/name
value2/value
/property
property
namemapreduce.tasktracker.reduce.tasks.maximum/name
value2/value
/property
/configuration
Thank you,
Amit
--
-李平
Re: Multiuser access to hdfs instance
I don't think reformat is needed. On Wed, Feb 9, 2011 at 9:10 PM, som shekha kolluru.sha...@wipro.com wrote: Hi, i have two node hadoop cluster up and running, but now i am trying to see how the user authentication is weak, that means i want to check whether other user can delete the files on the HDFS created by another user. I have the user hud which currently runs the hadoop. i have created another user fraud and put this user int the hud group. useradd fraud usermod -G hud fraud After this i change the permission of hud directory (previoulsy 700) to 770, so that other users belonging to user groups can see... I searched blogs and literature but then i was unable to give the user fraud access to the HDFS... in the blogs they mentioned to change the fs.group.supergroup and fs.permission parameter and then format the namenode, but i have already put lot of data on the HDFS so if i cant format it .. Any suggestions how should i go ahead. Thanks for the inputs. Regards, Shekhar -- View this message in context: http://hadoop-common.472056.n3.nabble.com/Multiuser-access-to-hdfs-instance-tp2458319p2458319.html Sent from the Users mailing list archive at Nabble.com. -- -李平
Re: Why do I get SocketTimeoutException?
It could be the exception caused by connection between NN and DN On Sat, Jan 29, 2011 at 8:11 AM, hadoop user iphul...@gmail.com wrote: What are possible causes due to which I might get SocketTimeoutException ? 11/01/28 19:01:36 INFO hdfs.DFSClient: Exception in createBlockOutputStream java.net.SocketTimeoutException: 69000 millis timeout while waiting for channel to be ready for connect. ch : java.nio.channels.SocketChannel[connection-pending remote=/XX.XXX.XX.X:50010] 11/01/28 19:01:36 INFO hdfs.DFSClient: Abandoning block blk_987175206123664825_1215418 Thanks, Ravi -- -李平
Re: Cannot copy files to HDFS
Please double check the node is alive. and you have the permission to connect to. On Wed, Jan 26, 2011 at 4:58 PM, Karthik Kumar karthik84ku...@gmail.comwrote: Hi, I am new to Hadoop. I am using Hadoop 0.20.2 version. I tried to copy a file of size 300 MB from local to HDFS. It showed the error as below. Please help me in solving this issue. 11/01/26 13:01:52 WARN hdfs.DFSClient: DataStreamer Exception: java.io.IOException: An existing connection was forcibly closed by the remote host at sun.nio.ch.SocketDispatcher.write0(Native Method) at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:33) at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:104) at sun.nio.ch.IOUtil.write(IOUtil.java:75) at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334) at org.apache.hadoop.net.SocketOutputStream$Writer.performIO(SocketOutputStream.java:55) at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142) at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:146) at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:107) at java.io.BufferedOutputStream.write(BufferedOutputStream.java:105) at java.io.DataOutputStream.write(DataOutputStream.java:90) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2314) 11/01/26 13:01:52 WARN hdfs.DFSClient: Error Recovery for block blk_4184614741505116937_1012 bad datanode[0] 160.110.184.114:50010 11/01/26 13:01:52 WARN hdfs.DFSClient: Error Recovery for block blk_4184614741505116937_1012 in pipeline 160.110.184.114:50010, 160.110.184.111:50010: bad datanode 160.110.184.114:50010 11/01/26 13:01:55 WARN hdfs.DFSClient: Error Recovery for block blk_4184614741505116937_1012 failed because recovery from primary datanode 160.110.184.111:50010 failed 1 times. Pipeline was 160.110.184.114:50010, 160.110.184.111:50010. Will retry... 11/01/26 13:01:55 WARN hdfs.DFSClient: Error Recovery for block blk_4184614741505116937_1012 bad datanode[0] 160.110.184.114:50010 11/01/26 13:01:55 WARN hdfs.DFSClient: Error Recovery for block blk_4184614741505116937_1012 in pipeline 160.110.184.114:50010, 160.110.184.111:50010: bad datanode 160.110.184.114:50010 11/01/26 13:02:28 WARN hdfs.DFSClient: DataStreamer Exception: java.io.IOException: An existing connection was forcibly closed by the remote host at sun.nio.ch.SocketDispatcher.write0(Native Method) at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:33) at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:104) at sun.nio.ch.IOUtil.write(IOUtil.java:75) at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334) at org.apache.hadoop.net.SocketOutputStream$Writer.performIO(SocketOutputStream.java:55) at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142) at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:146) at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:107) at java.io.BufferedOutputStream.write(BufferedOutputStream.java:105) at java.io.DataOutputStream.write(DataOutputStream.java:90) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2314) 11/01/26 13:02:28 WARN hdfs.DFSClient: Error Recovery for block blk_4184614741505116937_1013 bad datanode[0] 160.110.184.111:50010 copyFromLocal: All datanodes 160.110.184.111:50010 are bad. Aborting... 11/01/26 13:02:28 ERROR hdfs.DFSClient: Exception closing file /hdfs/data/input/cdr10M.csv : java.io.IOException: All datanodes 160.110.184.111:50010 are bad. Aborting... java.io.IOException: All datanodes 160.110.184.111:50010 are bad. Aborting... at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2556) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:2102) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2265) -- With Regards, Karthik -- -李平
Re: have a problem to run a hadoop with a jar.
It is a NoSuchMethodError error. Perhaps, the jar that you are using does not contain the method. Please double check it. On Tue, Jan 25, 2011 at 10:44 AM, Jun Young Kim juneng...@gmail.com wrote: Hi, I got this error when I executed a hadoop with a my jar application. $ hadoop jar test-hdeploy.jar Test Exception in thread main java.lang.NoSuchMethodError: org.slf4j.spi.LocationAwareLogger.log(Lorg/slf4j/Marker;Ljava/lang/String;ILjava/lang/String;[Ljava/lang/Object;Ljava/lang/Throwable;)V at org.apache.commons.logging.impl.SLF4JLocationAwareLog.debug(SLF4JLocationAwareLog.java:133) at org.apache.hadoop.security.UnixUserGroupInformation.login(UnixUserGroupInformation.java:301) at org.apache.hadoop.mapred.JobClient.getUGI(JobClient.java:679) at org.apache.hadoop.mapred.JobClient.createRPCProxy(JobClient.java:429) at org.apache.hadoop.mapred.JobClient.init(JobClient.java:423) at org.apache.hadoop.mapred.JobClient.init(JobClient.java:410) at org.apache.hadoop.mapreduce.Job.init(Job.java:50) at org.apache.hadoop.mapreduce.Job.init(Job.java:54) at com.naver.shopping.feeder.hadoop.EPComparatorJob.run(EPComparatorJob.java:78) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79) at com.naver.shopping.feeder.hadoop.EPComparatorJob.main(EPComparatorJob.java:54) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:156) a hadoop already has dependecies with slf libraries. (slf4j-log4j12-1.4.3.jar, slf4j-api-1.4.3.jar) so my jar file doesn't need to include it. do you know how I can fix it? -- Junyoung Kim (juneng...@gmail.com) -- -李平
Re: installation of Hadoop 0.21
The exception java.io.IOException: NameNode is not formatted. indicated you should format the NameNode first. hadoop -fs namenode -format On Tue, Jan 25, 2011 at 12:47 PM, Jim X jim.p...@gmail.com wrote: I am trying to install Hadoop by following the instruction from http://alans.se/blog/2010/hadoop-hbase-cygwin-windows-7-x64/. 1. I can not open http://localhost:9100 or http://localhost:9101 after I run bin/start-dfs.sh and bin/start-mapred.sh without any error message being printed. 2. I shutdown cygwin shell. 3. I start another cygwin shell, run bin/start-dfs.sh and get the following message from the shell. $ bin/start-dfs.sh starting namenode, logging to C:\cygwin\hadoop\0.21.0\logs/hadoop-Jim-namenode-Jim-PC.out localhost: datanode running as process 6908. Stop it first. localhost: secondarynamenode running as process 6156. Stop it first. Log message in C:\cygwin\hadoop\0.21.0\logs/hadoop-Jim-namenode-Jim-PC.out are listed as below: 2011-01-24 23:10:11,202 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: NameNode up at: 127.0.0.1/127.0.0.1:9100 2011-01-24 23:10:36,187 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 9100: readAndProcess threw exception java.io.IOException: Unable to read authentication method. Count of bytes read: 0 java.io.IOException: Unable to read authentication method at org.apache.hadoop.ipc.Server$Connection.readAndProcess(Server.java:1079) at org.apache.hadoop.ipc.Server$Listener.doRead(Server.java:525) at org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:332) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:619) 2011-01-24 23:41:47,815 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Registered FSNamesystemStatusMBean 2011-01-24 23:41:47,915 ERROR org.apache.hadoop.hdfs.server.namenode.FSNamesystem: FSNamesystem initialization failed. java.io.IOException: NameNode is not formatted. at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:434) at org.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSDirectory.java:110) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesystem.java:291) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.init(FSNamesystem.java:270) at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:271) at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:303) at org.apache.hadoop.hdfs.server.namenode.NameNode.init(NameNode.java:433) at org.apache.hadoop.hdfs.server.namenode.NameNode.init(NameNode.java:421) at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1359) at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1368) I am running Window Vista, JDK 1.6. I appreciate your help. Jim -- -李平
Re: Why Hadoop uses HTTP for file transmission between Map and Reduce?
That is also my concerns. Is it efficient for data transmission. On Thu, Jan 13, 2011 at 4:27 PM, Nan Zhu zhunans...@gmail.com wrote: Hi, all I have a question about the file transmission between Map and Reduce stage, in current implementation, the Reducers get the results generated by Mappers through HTTP Get, I don't understand why HTTP is selected, why not FTP, or a self-developed protocal? Just for HTTP's simple? thanks Nan -- -李平
Re: ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: java.io.IOException: Datanode state: LV = -19 CTime = 1294051643891 is newer than the namespace state: LV = -19 CTime = 0
Are you sure you have formated the NN successfully. If you really do not need the data, you can remove the directory, and then reformat it. On Sun, Jan 9, 2011 at 5:09 AM, Shuja Rehman shujamug...@gmail.com wrote: hi i have format the name node and now when i restart the cluster, i am getting the strange error. kindly let me know how to fix it. thnx / STARTUP_MSG: Starting DataNode STARTUP_MSG: host = hadoop.zoniversal.com/10.0.3.85 STARTUP_MSG: args = [] STARTUP_MSG: version = 0.20.2+737 STARTUP_MSG: build = -r 98c55c28258aa6f42250569bd7fa431ac657bdbd; compiled by 'root' on Mon Oct 11 13:14:05 EDT 2010 / 2011-01-08 12:55:58,586 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: /10.0.3.85:8020. Already tried 0 time(s). 2011-01-08 12:55:59,598 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: /10.0.3.85:8020. Already tried 1 time(s). 2011-01-08 12:56:00,608 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: /10.0.3.85:8020. Already tried 2 time(s). 2011-01-08 12:56:01,618 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: /10.0.3.85:8020. Already tried 3 time(s). 2011-01-08 12:56:03,540 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: java.io.IOException: Datanode state: LV = -19 CTime = 1294051643891 is newer than the namespace state: LV = -19 CTime = 0 at org.apache.hadoop.hdfs.server.datanode.DataStorage.doTransition(DataStorage.java:249) at org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:148) at org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:356) at org.apache.hadoop.hdfs.server.datanode.DataNode.init(DataNode.java:272) at org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1492) at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1432) at org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:1450) at org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:1575) at org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:1585) 2011-01-08 12:56:03,541 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: SHUTDOWN_MSG: / SHUTDOWN_MSG: Shutting down DataNode at hadoop.zoniversal.com/10.0.3.85 / 2011-01-08 13:04:17,579 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: STARTUP_MSG: / STARTUP_MSG: Starting DataNode STARTUP_MSG: host = hadoop.zoniversal.com/10.0.3.85 STARTUP_MSG: args = [] STARTUP_MSG: version = 0.20.2+737 STARTUP_MSG: build = -r 98c55c28258aa6f42250569bd7fa431ac657bdbd; compiled by 'root' on Mon Oct 11 13:14:05 EDT 2010 / 2011-01-08 13:04:19,028 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: /10.0.3.85:8020. Already tried 0 time(s). 2011-01-08 13:04:20,038 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: /10.0.3.85:8020. Already tried 1 time(s). 2011-01-08 13:04:21,049 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: /10.0.3.85:8020. Already tried 2 time(s). 2011-01-08 13:04:22,060 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: /10.0.3.85:8020. Already tried 3 time(s). 2011-01-08 13:04:24,601 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: java.io.IOException: Incompatible namespaceIDs in /var/lib/hadoop-0.20/cache/hdfs/dfs/data: namenode namespaceID = 125812142; datanode namespaceID = 1083940884 at org.apache.hadoop.hdfs.server.datanode.DataStorage.doTransition(DataStorage.java:233) at org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:148) at org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:356) at org.apache.hadoop.hdfs.server.datanode.DataNode.init(DataNode.java:272) at org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1492) at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1432) at org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:1450) at org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:1575) at org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:1585) 2011-01-08 13:04:24,602 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: SHUTDOWN_MSG: / SHUTDOWN_MSG: Shutting down DataNode at hadoop.zoniversal.com/10.0.3.85 -- Regards
Re: HDFS FS Commands Hanging System
I suggest you should look through the logs to see if there is any error. And the second point that I need to point out is which node you run the command hadoop fs -ls . If you run the command on Node A, the configuration item fs.default.name should point to the HDFS. On Sat, Jan 1, 2011 at 3:20 AM, Jon Lederman jon2...@gmail.com wrote: Hi Michael, Thanks for your response. It doesn't seem to be an issue with safemode. Even when I try the command dfsadmin -safemode get, the system hangs. I am unable to execute any FS shell commands other than hadoop fs -help. I am wondering whether this an issue with communication between the daemons? What should I be looking at there? Or could it be something else? When I do jps, I do see all the daemons listed. Any other thoughts. Thanks again and happy new year. -Jon On Dec 31, 2010, at 9:09 AM, Black, Michael (IS) wrote: Try checking your dfs status hadoop dfsadmin -safemode get Probably says ON hadoop dfsadmin -safemode leave Somebody else can probably say how to make this happen every reboot Michael D. Black Senior Scientist Advanced Analytics Directorate Northrop Grumman Information Systems From: Jon Lederman [mailto:jon2...@gmail.com] Sent: Fri 12/31/2010 11:00 AM To: common-user@hadoop.apache.org Subject: EXTERNAL:HDFS FS Commands Hanging System Hi All, I have been working on running Hadoop on a new microprocessor architecture in pseudo-distributed mode. I have been successful in getting SSH configured. I am also able to start a namenode, secondary namenode, tasktracker, jobtracker and datanode as evidenced by the response I get from jps. However, when I attempt to interact with the file system in any way such as the simple command hadoop fs -ls, the system hangs. So it appears to me that some communication is not occurring properly. Does anyone have any suggestions what I look into in order to fix this problem? Thanks in advance. -Jon -- -李平
Re: Retrying connect to server
make sure your /etc/hosts file contains the correct ip/hostname pair. This is very important 2010/12/30 Cavus,M.,Fa. Post Direkt m.ca...@postdirekt.de I process this ./hadoop jar ../../hadoopjar/hd.jar org.postdirekt.hadoop.WordCount gutenberg gutenberg-output I get this Dıd anyone know why I get this Error? 10/12/30 16:48:59 INFO security.Groups: Group mapping impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping; cacheTimeout=30 10/12/30 16:49:01 INFO ipc.Client: Retrying connect to server: localhost/ 127.0.0.1:9001. Already tried 0 time(s). 10/12/30 16:49:02 INFO ipc.Client: Retrying connect to server: localhost/ 127.0.0.1:9001. Already tried 1 time(s). 10/12/30 16:49:03 INFO ipc.Client: Retrying connect to server: localhost/ 127.0.0.1:9001. Already tried 2 time(s). 10/12/30 16:49:04 INFO ipc.Client: Retrying connect to server: localhost/ 127.0.0.1:9001. Already tried 3 time(s). 10/12/30 16:49:05 INFO ipc.Client: Retrying connect to server: localhost/ 127.0.0.1:9001. Already tried 4 time(s). 10/12/30 16:49:06 INFO ipc.Client: Retrying connect to server: localhost/ 127.0.0.1:9001. Already tried 5 time(s). 10/12/30 16:49:07 INFO ipc.Client: Retrying connect to server: localhost/ 127.0.0.1:9001. Already tried 6 time(s). 10/12/30 16:49:08 INFO ipc.Client: Retrying connect to server: localhost/ 127.0.0.1:9001. Already tried 7 time(s). 10/12/30 16:49:09 INFO ipc.Client: Retrying connect to server: localhost/ 127.0.0.1:9001. Already tried 8 time(s). 10/12/30 16:49:10 INFO ipc.Client: Retrying connect to server: localhost/ 127.0.0.1:9001. Already tried 9 time(s). Exception in thread main java.net.ConnectException: Call to localhost/ 127.0.0.1:9001 failed on connection exception: java.net.ConnectException: Connection refused at org.apache.hadoop.ipc.Client.wrapException(Client.java:932) at org.apache.hadoop.ipc.Client.call(Client.java:908) at org.apache.hadoop.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:198) at $Proxy0.getProtocolVersion(Unknown Source) at org.apache.hadoop.ipc.WritableRpcEngine.getProxy(WritableRpcEngine.java:228) at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:224) at org.apache.hadoop.mapreduce.Cluster.createRPCProxy(Cluster.java:82) at org.apache.hadoop.mapreduce.Cluster.createClient(Cluster.java:94) at org.apache.hadoop.mapreduce.Cluster.init(Cluster.java:70) at org.apache.hadoop.mapreduce.Job.init(Job.java:129) at org.apache.hadoop.mapreduce.Job.init(Job.java:134) at org.postdirekt.hadoop.WordCount.main(WordCount.java:19) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:192) Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:373) at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:417) at org.apache.hadoop.ipc.Client$Connection.access$1900(Client.java:207) at org.apache.hadoop.ipc.Client.getConnection(Client.java:1025) at org.apache.hadoop.ipc.Client.call(Client.java:885) ... 15 more -- -李平
Re: Flow of control
On Fri, Dec 31, 2010 at 9:28 AM, maha m...@umail.ucsb.edu wrote: Hi, (1) I declared a global variable in my hadoop mainClass which gets initialized in the 'run' function of this mainClass. When I try to access this global static variable from the MapperClass, it appears to be uninitialized. Why is that? Is it because of the parallel execution of Hadoop functions ? but , isn't the 'run' function supposed to be the one to run first and prepare all the job configurations before the Maps even start? The Mapper will run on a remote machine, in other JVM, so the variable you set in Main class can not be shared with other VM. (2) Fig 4.5 in http://developer.yahoo.com/hadoop/tutorial/module4.html shows the inputFormat to be the one running before the maps. My question is in which node? The JobTracker node? I think it should run on JobTracker, The inputFormat will split the file and the map function will read the every splited file. Thank you, Maha -- -李平
Re: HDFS disk consumption.
If you remove the file from the command line. The file will be moved to trash folder(if you enable the feature).otherwise the file will be moved immediately. On Thu, Dec 30, 2010 at 1:09 AM, Jane Chen jxchen_us_1...@yahoo.com wrote: You are right. There's only one replica. When does the deleted file space get reclaimed? --- On Tue, 12/28/10, Hemanth Yamijala yhema...@gmail.com wrote: From: Hemanth Yamijala yhema...@gmail.com Subject: Re: HDFS disk consumption. To: common-user@hadoop.apache.org Date: Tuesday, December 28, 2010, 8:43 PM Hi, On Wed, Dec 29, 2010 at 5:51 AM, Jane Chen jxchen_us_1...@yahoo.com wrote: Is setting dfs.replication to 1 sufficient to stop replication? How do I verify that? I have a pseudo cluster running 0.21.0. It seems that the hdfs disk consumption triples the amount of data stored. Setting to 1 is sufficient to stop replication. Can you check if the web UI for NameNode has a way to show the replicas of blocks for a file ? Thanks, Jane -- -李平
Re: Task fails: starts over with first input key?
I think the *org.apache.hadoop.mapred.SkipBadRecords* is you are looking for. On Tue, Dec 14, 2010 at 8:51 AM, Keith Wiley kwi...@keithwiley.com wrote: I think I am seeing a behavior in which if a mapper task fails (crashes) on one input key/value, the entire task is rescheduled and rerun, starting over again from the first input key/value even if all of the inputs preceding the troublesome input were processed successfully. Am I correct about this or am I seeing something that isn't there? If I am correct, what happens to the outputs of the successful duplicate map() calls? Which output key/value is the one that is sent to shuffle (and a reducer): Is it the result of the first attempt on the input in question or the result of the last attempt? Is there any way to prevent it from recalculating those duplicate inputs other than something manual on the side like keeping a job-log of the map attempts and scanning the log at the beginning of each map() call? Thanks. Keith Wiley kwi...@keithwiley.com www.keithwiley.com I used to be with it, but then they changed what it was. Now, what I'm with isn't it, and what's it seems weird and scary to me. -- Abe (Grandpa) Simpson -- -李平
Re: exceptions copying files into HDFS
That's right. You have to make sure the datanode is running. If you are using the virtual machine, like Virtual-box, sometime, you should wait for a moment until the datanode is active. seems like the performance issue, the datanode in vm will be active after several mins. On Sun, Dec 12, 2010 at 2:48 PM, Varadharajan Mukundan srinath...@gmail.com wrote: Hi, org.apache.hadoop.ipc.RemoteException: java.io.IOException: File /user/rock/input/fair-scheduler.xml could only be replicated to 0 nodes, instead of 1 I think none of your datanodes are actually running. why not use jps and make sure whether they are running. Also check the datanode logs in the nodes. -- Thanks, M. Varadharajan Experience is what you get when you didn't get what you wanted -By Prof. Randy Pausch in The Last Lecture My Journal :- www.thinkasgeek.wordpress.com -- -李平
Re: how to run jobs every 30 minutes?
submit the job every 30 mins. On Tue, Dec 7, 2010 at 4:55 PM, edward choi mp2...@gmail.com wrote: Hi, I'm planning to crawl a certain web site every 30 minutes. How would I get it done in Hadoop? In pure Java, I used Thread.sleep() method, but I guess this won't work in Hadoop. Or if it could work, could anyone show me an example? Ed. -- -李平
Re: Help: 1) Hadoop processes still are running after we stopped hadoop.2) How to exclude a dead node?
I am not sure I have fully understand your post. You mean the conf/slaves only be used for stop/start script to start or stop the datanode/tasktracker? And the conf/master only contains the information about the secondary namenode? Thanks On Wed, Dec 8, 2010 at 1:44 PM, Sudhir Vallamkondu sudhir.vallamko...@icrossing.com wrote: There is a proper decommissioning process to remove dead nodes. See the FAQ link here: http://wiki.apache.org/hadoop/FAQ#I_want_to_make_a_large_cluster_smaller_by_ taking_out_a_bunch_of_nodes_simultaneously._How_can_this_be_done.3F For a fact $HADOOP_HOME/conf/slaves is not used by the name node to keep track of datanodes/tasktracker. It is merely used by the stop/start hadoop scripts to know which nodes to start datanode / tasktracker services. Similarly there is confusion regarding understanding the $HADOOP_HOME/conf/master file. That file contains the details of the machine where secondary name node is running, not the name node/job tracker. With regards to not all java/hadoop processes getting killed, this may be happening due to hadoop loosing track of pid files. By default the pid files are configured to be created in the /tmp directory. If these pid files get deleted then stop/start scripts cannot detect running hadoop processes. I suggest changing location of pid files to a persistent location like /var/hadoop/. The $HADOOP_HOME/conf/hadoop-env.sh file has details on configuring the PID location - Sudhir On 12/7/10 5:07 PM, common-user-digest-h...@hadoop.apache.org common-user-digest-h...@hadoop.apache.org wrote: From: Tali K ncherr...@hotmail.com Date: Tue, 7 Dec 2010 10:40:16 -0800 To: core-u...@hadoop.apache.org Subject: Help: 1) Hadoop processes still are running after we stopped hadoop.2) How to exclude a dead node? 1)When I stopped hadoop, we checked all the nodes and found that 2 or 3 java/hadoop processes were still running on each node. So we went to each node and did a 'killall java' - in some cases I had to do 'killall -9 java'. My question : why is is this happening and what would be recommendations , how to make sure that there is no hadoop processes running after I stopped hadoop with stop-all.sh? 2) Also we have a dead node. We removed this node from $HADOOP_HOME/conf/slaves. This file is supposed to tell the namenode which machines are supposed to be datanodes/tasktrackers. We started hadoop again, and were surprised to see a dead node in hadoop 'report' ($HADOOP_HOME/bin/hadoop dfsadmin -report|less) It is only after blocking a deadnode and restarting hadoop, deadnode no longer showed up in hreport. Any recommendations, how to deal with dead nodes? iCrossing Privileged and Confidential Information This email message is for the sole use of the intended recipient(s) and may contain confidential and privileged information of iCrossing. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message. -- -李平
Re: Abandoning Block
Make sure the VMs can reach each other (e.g,IPtables). And the DNS/ip is correct. On Mon, Dec 6, 2010 at 7:05 PM, Adarsh Sharma adarsh.sha...@orkash.comwrote: Dear all, I am facing below problem while running Hadoop on VM's. I am using hadoop0-.20.2 with JDK6 My jobtracker log says that :-2010-12-06 15:16:06,618 INFO org.apache.hadoop.mapred.JobTracker: JobTracker up at: 54311 2010-12-06 15:16:06,618 INFO org.apache.hadoop.mapred.JobTracker: JobTracker webserver: 50030 2010-12-06 15:16:06,738 INFO org.apache.hadoop.mapred.JobTracker: Cleaning up the system directory 2010-12-06 15:16:06,801 INFO org.apache.hadoop.mapred.CompletedJobStatusStore: Completed job store is inactive 2010-12-06 15:17:15,830 INFO org.apache.hadoop.hdfs.DFSClient: Exception in createBlockOutputStream java.net.SocketTimeoutException: 69000 millis timeout while waiting for channel to be ready for connect. ch : java.nio.channels.SocketChannel[connection-pending remote=/ 192.168.0.56:50010] 2010-12-06 15:17:15,830 INFO org.apache.hadoop.hdfs.DFSClient: Abandoning block blk_377241628391316172_1001 2010-12-06 15:17:15,832 INFO org.apache.hadoop.hdfs.DFSClient: Waiting to find target node: 192.168.0.56:50010 2010-12-06 15:18:30,836 INFO org.apache.hadoop.hdfs.DFSClient: Exception in createBlockOutputStream java.net.SocketTimeoutException: 69000 millis timeout while waiting for channel to be ready for connect. ch : java.nio.channels.SocketChannel[connection-pending remote=/ 192.168.0.56:50010] 2010-12-06 15:18:30,836 INFO org.apache.hadoop.hdfs.DFSClient: Abandoning block blk_2025622418653738085_1001 2010-12-06 15:18:30,838 INFO org.apache.hadoop.hdfs.DFSClient: Waiting to find target node: 192.168.0.56:50010 2010-12-06 15:19:45,842 INFO org.apache.hadoop.hdfs.DFSClient: Exception in createBlockOutputStream java.net.SocketTimeoutException: 69000 millis timeout while waiting for channel to be ready for connect. ch : java.nio.channels.SocketChannel[connection-pending remote=/ 192.168.0.61:50010] 2010-12-06 15:19:45,843 INFO org.apache.hadoop.hdfs.DFSClient: Abandoning block blk_696328516245550547_1001 2010-12-06 15:19:45,845 INFO org.apache.hadoop.hdfs.DFSClient: Waiting to find target node: 192.168.0.61:50010 2010-12-06 15:21:00,849 INFO org.apache.hadoop.hdfs.DFSClient: Exception in createBlockOutputStream java.net.SocketTimeoutException: 69000 millis timeout while waiting for channel to be ready for connect. ch : java.nio.channels.SocketChannel[connection-pending remote=/ 192.168.0.55:50010] 2010-12-06 15:21:00,849 INFO org.apache.hadoop.hdfs.DFSClient: Abandoning block blk_6110605884701761678_1001 2010-12-06 15:21:00,853 INFO org.apache.hadoop.hdfs.DFSClient: Waiting to find target node: 192.168.0.55:50010 2010-12-06 15:21:06,854 WARN org.apache.hadoop.hdfs.DFSClient: DataStreamer Exception: java.io.IOException: Unable to create new block. at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2845) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2102) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2288) 2010-12-06 15:21:06,855 WARN org.apache.hadoop.hdfs.DFSClient: Error Recovery for block blk_6110605884701761678_1001 bad datanode[0] nodes == null 2010-12-06 15:21:06,855 WARN org.apache.hadoop.hdfs.DFSClient: Could not get block locations. Source file /home/hadoop/mapred/system/ jobtracker.info - Aborting... 2010-12-06 15:21:06,855 WARN org.apache.hadoop.mapred.JobTracker: Writing to file hdfs://ws-test:54310/home/hadoop/mapred/system/jobtracker.infofailed! 41,1 5% tem/jobtracker.info failed! 2010-12-06 15:21:06,855 WARN org.apache.hadoop.mapred.JobTracker: FileSystem is not ready yet! 2010-12-06 15:21:06,862 WARN org.apache.hadoop.mapred.JobTracker: Failed to initialize recovery manager. java.net.SocketTimeoutException: 69000 millis timeout while waiting for channel to be ready for connect. ch : java.nio.channels.SocketChannel[connection-pending remote=/ 192.168.0.55:50010] at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:213) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:404) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.createBlockOutputStream(DFSClient.java:2870) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2826) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2102) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2288) 2010-12-06 15:21:16,864 WARN org.apache.hadoop.mapred.JobTracker: Retrying... 2010-12-06 15:22:25,879 INFO org.apache.hadoop.hdfs.DFSClient: Exception in createBlockOutputStream java.net.SocketTimeoutException: 69000 millis timeout while waiting for channel to be ready for
Re: delay the execution of reducers
org.apache.hadoop.mapred.JobInProgress Maybe you find this class. On Mon, Nov 29, 2010 at 4:36 AM, Da Zheng zhengda1...@gmail.com wrote: I have a problem with subscribing mapreduce mailing list. I use hadoop-0.20.2. I have added this parameter to mapred-site.xml. Is there any way for me to check whether the parameter has been read and activated? BTW, what do you mean by opening a jira? Thanks, Da On 11/28/2010 05:03 AM, Arun C Murthy wrote: Moving to mapreduce-user@, bcc common-u...@. Please use project specific lists. mapreduce.reduce.slowstart.completed.maps is the right knob. Which version of hadoop are you running? If it isn't working, please open a jira. Thanks. Arun On Nov 27, 2010, at 11:40 PM, Da Zheng wrote: Hello, I found in Hadoop that reducers starts when a fraction of the number of mappers is complete. However, in my case, I hope reducers to start only when all mappers are complete. I searched for Hadoop configuration parameters, and found mapred.reduce.slowstart.completed.maps, which seems to do what I want. But no matter what value (0.99, 1.00, etc) I set to mapred.reduce.slowstart.completed.maps, reducers always start to execute when about 10% of mappers are complete. Do I set the right parameter? Is there any other parameter I can use for this purpose? Thanks, Da -- -李平
Re: InputSplit is confusing me .. Any clarifications ??
That depends on which InputFormat class you use.
For example, The *org.apache.hadoop.mapreduce.lib.input.TextInputFormat. *This
TextInputFormat class will break the file into lines, and the key is the
position of the line. the value is the text of the line.
On Sat, Nov 27, 2010 at 6:08 AM, maha m...@umail.ucsb.edu wrote:
Hi Everyone,
What I really wish for Thanksgiving is some one giving me clarification
of how the inputSplit is working.
Eg.
public void map(LongWritable key, Text value, OutputCollectorText, Text
output, Reporter reporter) throws IOException {
String line = value.toString();
How did we know that map in this case is taking a line and not the whole
input document ?
Happy Thanksgiving everyone,
Maha
--
-李平
