Hadoop-Fuse issue
Dear All, I have installed Hadoop-fuse to mount the HDFS filesystem locally for more flexibility. I could mount the HDFS without any issues.But I am not able to do any file operations like *delete, copy, move* orations directly. The directory ownership automatically changed to nobody:nobody while mounting. [root@namenode ~]# ls -ld /hdfs_mount/ drwxr-xr-x 2 *root root* 4096 Apr 3 16:22 /hdfs_mount/ [root@namenode ~]# *hadoop-fuse-dfs dfs://namenode:8020 /hdfs_mount/* INFO fuse_options.c:162 Adding FUSE arg /hdfs_mount/ [root@namenode ~]# ls -ld /hdfs_mount/ drwxrwxr-x 10* nobody nobody* 4096 Apr 5 13:22 /hdfs_mount/ I tried the same with pseudo-distributed node, its working fine. I can do any normal file operations after mounting the HDFS. Appreciate your help on the same. -- Thanks Regards *Manu S* SI Engineer - OpenSource HPC Wipro Infotech Mob: +91 8861302855Skype: manuspkd www.opensourcetalk.co.in
Re: map task execution time
On 04/04/2012 11:20 PM, bikash sharma wrote: Hi, Is it possible to get the execution time of the constituent map/reduce tasks of a MapReduce job (say sort) at the end of a job run? Preferably, can we obtain this programatically? Thanks, Bikash You could parse the logs inside the job's output directory (output dir/_logs) You may also find this tool to be interesting: http://www.cs.duke.edu/starfish/ It can extract all that task info for you, after doing a profiling run. -- Luca Pireddu CRS4 - Distributed Computing Group Loc. Pixina Manna Edificio 1 09010 Pula (CA), Italy Tel: +39 0709250452
Data Node is not Started
Hi all, my DataNode is not started . even after deleting hadoop*.pid file from /tmp , But still Data node is not started , Hadoop Version: hadoop-1.0.1.tar.gz Java version : java version 1.6.0_26 Operating System : Ubuntu 11.10 i did below procedure *hduser@sujit:~/Desktop/hadoop/bin$ jps* 11455 Jps *hduser@sujit:~/Desktop/hadoop/bin$ start-all.sh* Warning: $HADOOP_HOME is deprecated. starting namenode, logging to /home/hduser/Desktop/hadoop/libexec/../logs/hadoop-hduser-namenode-sujit.out localhost: starting datanode, logging to /home/hduser/Desktop/hadoop/libexec/../logs/hadoop-hduser-datanode-sujit.out localhost: starting secondarynamenode, logging to /home/hduser/Desktop/hadoop/libexec/../logs/hadoop-hduser-secondarynamenode-sujit.out starting jobtracker, logging to /home/hduser/Desktop/hadoop/libexec/../logs/hadoop-hduser-jobtracker-sujit.out localhost: starting tasktracker, logging to /home/hduser/Desktop/hadoop/libexec/../logs/hadoop-hduser-tasktracker-sujit.out *hduser@sujit:~/Desktop/hadoop/bin$ jps* 11528 NameNode 12019 SecondaryNameNode 12355 TaskTracker 12115 JobTracker 12437 Jps *hduser@sujit:~/Desktop/hadoop/bin$ stop-all.sh* Warning: $HADOOP_HOME is deprecated. stopping jobtracker localhost: stopping tasktracker stopping namenode localhost: no datanode to stop localhost: stopping secondarynamenode *hduser@sujit:~/Desktop/hadoop/bin$ jps* 13127 Jps *hduser@sujit:~/Desktop/hadoop/bin$ ls /tmp* hadoop-hduser-datanode.pid hsperfdata_hduserkeyring-meecr7 ssh-JXYCAJsX1324 hadoop-hduser-jobtracker.pid hsperfdata_sujit plugtmp unity_support_test.0 hadoop-hduser-namenode.pid Jetty_0_0_0_0_50030_jobyn7qmkpulse-2L9K88eMlGn7 virtual-hduser.Q8j5nJ hadoop-hduser-secondarynamenode.pid Jetty_0_0_0_0_50070_hdfsw2cu08 pulse-Ob9vyJcXyHZz hadoop-hduser-tasktracker.pid Jetty_0_0_0_0_50090_secondaryy6aanv pulse-PKdhtXMmr18n *Deleted *.pid file :) hduser@sujit:~$ ls /tmp* hsperfdata_hduserpulse-2L9K88eMlGn7 hsperfdata_sujit pulse-Ob9vyJcXyHZz Jetty_0_0_0_0_50030_jobyn7qmkpulse-PKdhtXMmr18n Jetty_0_0_0_0_50070_hdfsw2cu08 ssh-JXYCAJsX1324 Jetty_0_0_0_0_50090_secondaryy6aanv unity_support_test.0 keyring-meecr7 virtual-hduser.Q8j5nJ plugtmp *hduser@sujit:~/Desktop/hadoop$ bin/hadoop namenode -format* Warning: $HADOOP_HOME is deprecated. 12/04/06 23:23:22 INFO namenode.NameNode: STARTUP_MSG: / STARTUP_MSG: Starting NameNode STARTUP_MSG: host = sujit.(null)/127.0.1.1 STARTUP_MSG: args = [-format] STARTUP_MSG: version = 1.0.1 STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0 -r 1243785; compiled by 'hortonfo' on Tue Feb 14 08:15:38 UTC 2012 / Re-format filesystem in /app/hadoop/tmp/dfs/name ? (Y or N) Y 12/04/06 23:23:25 INFO util.GSet: VM type = 32-bit 12/04/06 23:23:25 INFO util.GSet: 2% max memory = 17.77875 MB 12/04/06 23:23:25 INFO util.GSet: capacity = 2^22 = 4194304 entries 12/04/06 23:23:25 INFO util.GSet: recommended=4194304, actual=4194304 12/04/06 23:23:25 INFO namenode.FSNamesystem: fsOwner=hduser 12/04/06 23:23:25 INFO namenode.FSNamesystem: supergroup=supergroup 12/04/06 23:23:25 INFO namenode.FSNamesystem: isPermissionEnabled=true 12/04/06 23:23:25 INFO namenode.FSNamesystem: dfs.block.invalidate.limit=100 12/04/06 23:23:25 INFO namenode.FSNamesystem: isAccessTokenEnabled=false accessKeyUpdateInterval=0 min(s), accessTokenLifetime=0 min(s) 12/04/06 23:23:25 INFO namenode.NameNode: Caching file names occuring more than 10 times 12/04/06 23:23:26 INFO common.Storage: Image file of size 112 saved in 0 seconds. 12/04/06 23:23:26 INFO common.Storage: Storage directory /app/hadoop/tmp/dfs/name has been successfully formatted. 12/04/06 23:23:26 INFO namenode.NameNode: SHUTDOWN_MSG: / SHUTDOWN_MSG: Shutting down NameNode at sujit.(null)/127.0.1.1 / hduser@sujit:~/Desktop/hadoop$ bin/start-all.sh Warning: $HADOOP_HOME is deprecated. starting namenode, logging to /home/hduser/Desktop/hadoop/libexec/../logs/hadoop-hduser-namenode-sujit.out localhost: starting datanode, logging to /home/hduser/Desktop/hadoop/libexec/../logs/hadoop-hduser-datanode-sujit.out localhost: starting secondarynamenode, logging to /home/hduser/Desktop/hadoop/libexec/../logs/hadoop-hduser-secondarynamenode-sujit.out starting jobtracker, logging to /home/hduser/Desktop/hadoop/libexec/../logs/hadoop-hduser-jobtracker-sujit.out localhost: starting tasktracker, logging to /home/hduser/Desktop/hadoop/libexec/../logs/hadoop-hduser-tasktracker-sujit.out *hduser@sujit:~/Desktop/hadoop$ jps* 14157 JobTracker 14492 Jps 14397 TaskTracker 14063 SecondaryNameNode
Re: Data Node is not Started
Can you check the datanode logs? May its an incompatible namespace issue. On Apr 6, 2012, at 11:13 AM, Sujit Dhamale sujitdhamal...@gmail.com wrote: Hi all, my DataNode is not started . even after deleting hadoop*.pid file from /tmp , But still Data node is not started , Hadoop Version: hadoop-1.0.1.tar.gz Java version : java version 1.6.0_26 Operating System : Ubuntu 11.10 i did below procedure *hduser@sujit:~/Desktop/hadoop/bin$ jps* 11455 Jps *hduser@sujit:~/Desktop/hadoop/bin$ start-all.sh* Warning: $HADOOP_HOME is deprecated. starting namenode, logging to /home/hduser/Desktop/hadoop/libexec/../logs/hadoop-hduser-namenode-sujit.out localhost: starting datanode, logging to /home/hduser/Desktop/hadoop/libexec/../logs/hadoop-hduser-datanode-sujit.out localhost: starting secondarynamenode, logging to /home/hduser/Desktop/hadoop/libexec/../logs/hadoop-hduser-secondarynamenode-sujit.out starting jobtracker, logging to /home/hduser/Desktop/hadoop/libexec/../logs/hadoop-hduser-jobtracker-sujit.out localhost: starting tasktracker, logging to /home/hduser/Desktop/hadoop/libexec/../logs/hadoop-hduser-tasktracker-sujit.out *hduser@sujit:~/Desktop/hadoop/bin$ jps* 11528 NameNode 12019 SecondaryNameNode 12355 TaskTracker 12115 JobTracker 12437 Jps *hduser@sujit:~/Desktop/hadoop/bin$ stop-all.sh* Warning: $HADOOP_HOME is deprecated. stopping jobtracker localhost: stopping tasktracker stopping namenode localhost: no datanode to stop localhost: stopping secondarynamenode *hduser@sujit:~/Desktop/hadoop/bin$ jps* 13127 Jps *hduser@sujit:~/Desktop/hadoop/bin$ ls /tmp* hadoop-hduser-datanode.pid hsperfdata_hduserkeyring-meecr7 ssh-JXYCAJsX1324 hadoop-hduser-jobtracker.pid hsperfdata_sujit plugtmp unity_support_test.0 hadoop-hduser-namenode.pid Jetty_0_0_0_0_50030_jobyn7qmkpulse-2L9K88eMlGn7 virtual-hduser.Q8j5nJ hadoop-hduser-secondarynamenode.pid Jetty_0_0_0_0_50070_hdfsw2cu08 pulse-Ob9vyJcXyHZz hadoop-hduser-tasktracker.pid Jetty_0_0_0_0_50090_secondaryy6aanv pulse-PKdhtXMmr18n *Deleted *.pid file :) hduser@sujit:~$ ls /tmp* hsperfdata_hduserpulse-2L9K88eMlGn7 hsperfdata_sujit pulse-Ob9vyJcXyHZz Jetty_0_0_0_0_50030_jobyn7qmkpulse-PKdhtXMmr18n Jetty_0_0_0_0_50070_hdfsw2cu08 ssh-JXYCAJsX1324 Jetty_0_0_0_0_50090_secondaryy6aanv unity_support_test.0 keyring-meecr7 virtual-hduser.Q8j5nJ plugtmp *hduser@sujit:~/Desktop/hadoop$ bin/hadoop namenode -format* Warning: $HADOOP_HOME is deprecated. 12/04/06 23:23:22 INFO namenode.NameNode: STARTUP_MSG: / STARTUP_MSG: Starting NameNode STARTUP_MSG: host = sujit.(null)/127.0.1.1 STARTUP_MSG: args = [-format] STARTUP_MSG: version = 1.0.1 STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0 -r 1243785; compiled by 'hortonfo' on Tue Feb 14 08:15:38 UTC 2012 / Re-format filesystem in /app/hadoop/tmp/dfs/name ? (Y or N) Y 12/04/06 23:23:25 INFO util.GSet: VM type = 32-bit 12/04/06 23:23:25 INFO util.GSet: 2% max memory = 17.77875 MB 12/04/06 23:23:25 INFO util.GSet: capacity = 2^22 = 4194304 entries 12/04/06 23:23:25 INFO util.GSet: recommended=4194304, actual=4194304 12/04/06 23:23:25 INFO namenode.FSNamesystem: fsOwner=hduser 12/04/06 23:23:25 INFO namenode.FSNamesystem: supergroup=supergroup 12/04/06 23:23:25 INFO namenode.FSNamesystem: isPermissionEnabled=true 12/04/06 23:23:25 INFO namenode.FSNamesystem: dfs.block.invalidate.limit=100 12/04/06 23:23:25 INFO namenode.FSNamesystem: isAccessTokenEnabled=false accessKeyUpdateInterval=0 min(s), accessTokenLifetime=0 min(s) 12/04/06 23:23:25 INFO namenode.NameNode: Caching file names occuring more than 10 times 12/04/06 23:23:26 INFO common.Storage: Image file of size 112 saved in 0 seconds. 12/04/06 23:23:26 INFO common.Storage: Storage directory /app/hadoop/tmp/dfs/name has been successfully formatted. 12/04/06 23:23:26 INFO namenode.NameNode: SHUTDOWN_MSG: / SHUTDOWN_MSG: Shutting down NameNode at sujit.(null)/127.0.1.1 / hduser@sujit:~/Desktop/hadoop$ bin/start-all.sh Warning: $HADOOP_HOME is deprecated. starting namenode, logging to /home/hduser/Desktop/hadoop/libexec/../logs/hadoop-hduser-namenode-sujit.out localhost: starting datanode, logging to /home/hduser/Desktop/hadoop/libexec/../logs/hadoop-hduser-datanode-sujit.out localhost: starting secondarynamenode, logging to /home/hduser/Desktop/hadoop/libexec/../logs/hadoop-hduser-secondarynamenode-sujit.out starting jobtracker, logging to
Re: Data Node is not Started
Below are DataNode logs hduser@sujit:~/Desktop/hadoop/logs$ cat hadoop-hduser-datanode-sujit.log 2012-04-06 22:11:34,566 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: STARTUP_MSG: / STARTUP_MSG: Starting DataNode STARTUP_MSG: host = sujit.(null)/127.0.1.1 STARTUP_MSG: args = [] STARTUP_MSG: version = 1.0.1 STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0 -r 1243785; compiled by 'hortonfo' on Tue Feb 14 08:15:38 UTC 2012 / 2012-04-06 22:11:34,749 INFO org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties 2012-04-06 22:11:34,768 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source MetricsSystem,sub=Stats registered. 2012-04-06 22:11:34,769 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s). 2012-04-06 22:11:34,769 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: DataNode metrics system started 2012-04-06 22:11:34,950 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source ugi registered. 2012-04-06 22:11:34,956 WARN org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name ugi already exists! 2012-04-06 22:11:36,149 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: localhost/127.0.0.1:54310. Already tried 0 time(s). 2012-04-06 22:11:41,923 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: java.io.IOException: Incompatible namespaceIDs in /app/hadoop/tmp/dfs/data: namenode namespaceID = 387652554; datanode namespaceID = 1269725409 at org.apache.hadoop.hdfs.server.datanode.DataStorage.doTransition(DataStorage.java:232) at org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:147) at org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:385) at org.apache.hadoop.hdfs.server.datanode.DataNode.init(DataNode.java:299) at org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1582) at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1521) at org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:1539) at org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:1665) at org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:1682) 2012-04-06 22:11:41,924 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: SHUTDOWN_MSG: / SHUTDOWN_MSG: Shutting down DataNode at sujit.(null)/127.0.1.1 / 2012-04-06 22:30:04,713 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: STARTUP_MSG: / STARTUP_MSG: Starting DataNode STARTUP_MSG: host = sujit.(null)/127.0.1.1 STARTUP_MSG: args = [] STARTUP_MSG: version = 1.0.1 STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0 -r 1243785; compiled by 'hortonfo' on Tue Feb 14 08:15:38 UTC 2012 / 2012-04-06 22:30:04,870 INFO org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties 2012-04-06 22:30:04,883 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source MetricsSystem,sub=Stats registered. 2012-04-06 22:30:04,884 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s). 2012-04-06 22:30:04,884 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: DataNode metrics system started 2012-04-06 22:30:05,046 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source ugi registered. 2012-04-06 22:30:05,051 WARN org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name ugi already exists! 2012-04-06 22:30:06,273 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: localhost/127.0.0.1:54310. Already tried 0 time(s). 2012-04-06 22:30:11,073 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: java.io.IOException: Incompatible namespaceIDs in /app/hadoop/tmp/dfs/data: namenode namespaceID = 361514980; datanode namespaceID = 1269725409 at org.apache.hadoop.hdfs.server.datanode.DataStorage.doTransition(DataStorage.java:232) at org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:147) at org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:385) at org.apache.hadoop.hdfs.server.datanode.DataNode.init(DataNode.java:299) at org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1582) at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1521) at
Re: Data Node is not Started
according to the logs the namespace id in the datanode data directories is incompatible. Since you formatted the namenode these id's do not match. Clean up the contents of the data dir (/app/hadoop/tmp/dfs/data) and then start the datanode. -- Arpit Gupta Hortonworks Inc. http://hortonworks.com/ On Apr 6, 2012, at 11:27 AM, Sujit Dhamale wrote: Below are DataNode logs hduser@sujit:~/Desktop/hadoop/logs$ cat hadoop-hduser-datanode-sujit.log 2012-04-06 22:11:34,566 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: STARTUP_MSG: / STARTUP_MSG: Starting DataNode STARTUP_MSG: host = sujit.(null)/127.0.1.1 STARTUP_MSG: args = [] STARTUP_MSG: version = 1.0.1 STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0 -r 1243785; compiled by 'hortonfo' on Tue Feb 14 08:15:38 UTC 2012 / 2012-04-06 22:11:34,749 INFO org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties 2012-04-06 22:11:34,768 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source MetricsSystem,sub=Stats registered. 2012-04-06 22:11:34,769 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s). 2012-04-06 22:11:34,769 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: DataNode metrics system started 2012-04-06 22:11:34,950 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source ugi registered. 2012-04-06 22:11:34,956 WARN org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name ugi already exists! 2012-04-06 22:11:36,149 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: localhost/127.0.0.1:54310. Already tried 0 time(s). 2012-04-06 22:11:41,923 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: java.io.IOException: Incompatible namespaceIDs in /app/hadoop/tmp/dfs/data: namenode namespaceID = 387652554; datanode namespaceID = 1269725409 at org.apache.hadoop.hdfs.server.datanode.DataStorage.doTransition(DataStorage.java:232) at org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:147) at org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:385) at org.apache.hadoop.hdfs.server.datanode.DataNode.init(DataNode.java:299) at org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1582) at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1521) at org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:1539) at org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:1665) at org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:1682) 2012-04-06 22:11:41,924 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: SHUTDOWN_MSG: / SHUTDOWN_MSG: Shutting down DataNode at sujit.(null)/127.0.1.1 / 2012-04-06 22:30:04,713 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: STARTUP_MSG: / STARTUP_MSG: Starting DataNode STARTUP_MSG: host = sujit.(null)/127.0.1.1 STARTUP_MSG: args = [] STARTUP_MSG: version = 1.0.1 STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0 -r 1243785; compiled by 'hortonfo' on Tue Feb 14 08:15:38 UTC 2012 / 2012-04-06 22:30:04,870 INFO org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties 2012-04-06 22:30:04,883 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source MetricsSystem,sub=Stats registered. 2012-04-06 22:30:04,884 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s). 2012-04-06 22:30:04,884 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: DataNode metrics system started 2012-04-06 22:30:05,046 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source ugi registered. 2012-04-06 22:30:05,051 WARN org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name ugi already exists! 2012-04-06 22:30:06,273 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: localhost/127.0.0.1:54310. Already tried 0 time(s). 2012-04-06 22:30:11,073 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: java.io.IOException: Incompatible namespaceIDs in /app/hadoop/tmp/dfs/data: namenode namespaceID = 361514980; datanode namespaceID = 1269725409 at org.apache.hadoop.hdfs.server.datanode.DataStorage.doTransition(DataStorage.java:232) at
Namespace logs : a common issue?
Hi guys : I'm noticing that namespace conflicts or differences are a common theme in hadoop both in my experience and now on this list Serv. Does anyone have any thoughts on why this is such a common issue and how it will be dealt with in new releases? Jay Vyas MMSB UCHC
Re: Namespace logs : a common issue?
If you are talking about the Incompatible namespace ID issues, then look at https://issues.apache.org/jira/browse/HDFS-107 for a discussion on the topic. It is a common issue with newcomers as they end up formatting at the first sight of trouble, or skip reading most of the tutorial that otherwise helps get configurations right. On Sat, Apr 7, 2012 at 12:31 AM, JAX jayunit...@gmail.com wrote: Hi guys : I'm noticing that namespace conflicts or differences are a common theme in hadoop both in my experience and now on this list Serv. Does anyone have any thoughts on why this is such a common issue and how it will be dealt with in new releases? Jay Vyas MMSB UCHC -- Harsh J
Reading properties file from command line + passing parameters from main to mapper/reducer?
Dear friends I am new to hadoop on aws. I am using AWS Elastic mapreduce. I am trying to convert an old java program to elastic mapreduce. I would appreciate your help. My question is about: 1. How do I pass a Properties file which i used to pass via command line paramters - to elastic mapreduce? 2. how do I pass parameters from this properties file from main() to the Mapper and Reducer? Details: 1. properties file: My program used to read a properties file as follows: org.apache.commons.configuration.Configuration config = new PropertiesConfiguration(my_app.properties); How do I read this in Amazon EMR? I tried launching the program with : jar file: s3n://my.bucket.name/myjar-job.jar args: s3n://my.bucket.name/my_app.properties I get an stderr of: org.apache.commons.configuration.ConfigurationException: Cannot locate configuration source s3n://my.bucket.name/my_app.properties In order to debug, I tried to do the following: String c = FileUtils.readFileToString(new File(remainingArgs[0])); I got an exception: Exception in thread main java.io.FileNotFoundException: File 's3n:/my.bucket.name/my_app.propertiess' does not exist Notice that the file name - even though I entered s3n:// is shown as s3n:/ - could the // be getting dropped? 2. Passing the properties file from my main() to Mapper and Reducer. How do I do that? Or pass other generic parameters? -Sky
Reading properties file from command line + passing parameters from main to mapper/reducer?
Dear friends I am new to hadoop on aws. I am using AWS Elastic mapreduce. I am trying to convert an old java program to elastic mapreduce. I would appreciate your help. My question is about: 1. How do I pass a Properties file which i used to pass via command line paramters - to elastic mapreduce? 2. how do I pass parameters from this properties file from main() to the Mapper and Reducer? Details: 1. properties file: My program used to read a properties file as follows: org.apache.commons.configuration.Configuration config = new PropertiesConfiguration(my_app.properties); How do I read this in Amazon EMR? I tried launching the program with : jar file: s3n://my.bucket.name/myjar-job.jar args: s3n://my.bucket.name/my_app.properties I get an stderr of: org.apache.commons.configuration.ConfigurationException: Cannot locate configuration source s3n://my.bucket.name/my_app.properties In order to debug, I tried to do the following: String c = FileUtils.readFileToString(new File(remainingArgs[0])); I got an exception: Exception in thread main java.io.FileNotFoundException: File 's3n:/my.bucket.name/my_app.propertiess' does not exist Notice that the file name - even though I entered s3n:// is shown as s3n:/ - could the // be getting dropped? 2. Passing the properties file from my main() to Mapper and Reducer. How do I do that? Or pass other generic parameters? -Sky
Reading properties file from command line + passing parameters from main to mapper/reducer?
Dear friends I am new to hadoop on aws. I am using AWS Elastic mapreduce. I am trying to convert an old java program to elastic mapreduce. I would appreciate your help. My question is about: 1. How do I pass a Properties file which i used to pass via command line paramters - to elastic mapreduce? 2. how do I pass parameters from this properties file from main() to the Mapper and Reducer? Details: 1. properties file: My program used to read a properties file as follows: org.apache.commons.configuration.Configuration config = new PropertiesConfiguration(my_app.properties); How do I read this in Amazon EMR? I tried launching the program with : jar file: s3n://my.bucket.name/myjar-job.jar args: s3n://my.bucket.name/my_app.properties I get an stderr of: org.apache.commons.configuration.ConfigurationException: Cannot locate configuration source s3n://my.bucket.name/my_app.properties In order to debug, I tried to do the following: String c = FileUtils.readFileToString(new File(remainingArgs[0])); I got an exception: Exception in thread main java.io.FileNotFoundException: File 's3n:/my.bucket.name/my_app.propertiess' does not exist Notice that the file name - even though I entered s3n:// is shown as s3n:/ - could the // be getting dropped? 2. Passing the properties file from my main() to Mapper and Reducer. How do I do that? Or pass other generic parameters? -Sky
