Re: Unexpected termination of a job
Have you tried after increasing HEAP memory to your process ? Arvind From: Rakhi Khatwani rkhatw...@gmail.com To: common-user@hadoop.apache.org Sent: Wed, March 3, 2010 10:38:43 PM Subject: Re: Unexpected termination of a job Hi, I tried running it on eclipse, the job starts... but somehow it terminates throwing an exception, Job Failed. thats why i wanted to run on jobtracker to check the logs but the execution terminates even before the job starts(during the preprocessing). How do i ensure that the job runs in jobtracker mode? Regards Raakhi On Thu, Mar 4, 2010 at 2:25 AM, Aaron Kimball aa...@cloudera.com wrote: If it's terminating before you even run a job, then you're in luck -- it's all still running on the local machine. Try running it in Eclipse and use the debugger to trace its execution. - Aaron On Wed, Mar 3, 2010 at 4:13 AM, Rakhi Khatwani rkhatw...@gmail.com wrote: Hi, I am running a job which has lotta preprocessing involved. so whn i run my class from a jarfile, somehow it terminates after sometime without giving any exception, i have tried running the same program several times, and everytime it terminates at different locations in the code(during the preprocessing... haven't configured a job as yet). Probably it terminaits after a fixed interval). No idea why this is happeneing, Any Pointers?? Regards, Raakhi Khatwani
should data be evenly distributed to each (physical) node
I am building a small two node cluster following http://www.michael-noll.com/wiki/Running_Hadoop_On_Ubuntu_Linux_(Multi-Node_Cluster) Every thing seems to be working, except I notice the data are NOT evenly distributed to each physical box. e.g., when I hadoop dfs -put 6G data. I am expecting ~3G on each node (take turns every ~64MB), however, I checked dfshealth.jsp and du -k on local box, and found the uploaded data are ONLY residing on the physical box where I start dfs -put. That defeats the whole (data locality) purpose of hadoop?! Please help. Thanks -- View this message in context: http://old.nabble.com/should-data-be-evenly-distributed-to-each-%28physical%29-node-tp27782215p27782215.html Sent from the Hadoop core-user mailing list archive at Nabble.com.
Re: should data be evenly distributed to each (physical) node
On Thu, Mar 4, 2010 at 10:25 AM, openresearch qiming...@openresearchinc.com wrote: I am building a small two node cluster following http://www.michael-noll.com/wiki/Running_Hadoop_On_Ubuntu_Linux_(Multi-Node_Cluster) Every thing seems to be working, except I notice the data are NOT evenly distributed to each physical box. e.g., when I hadoop dfs -put 6G data. I am expecting ~3G on each node (take turns every ~64MB), however, I checked dfshealth.jsp and du -k on local box, and found the uploaded data are ONLY residing on the physical box where I start dfs -put. That defeats the whole (data locality) purpose of hadoop?! Please help. Thanks -- View this message in context: http://old.nabble.com/should-data-be-evenly-distributed-to-each-%28physical%29-node-tp27782215p27782215.html Sent from the Hadoop core-user mailing list archive at Nabble.com. The distribution of data on each datanode will not be exactly even, however blocks should be located on different boxes. Check the namenode webinterface http://nn-ip:50070 make sure all the datanodes are listed. Make sure the java DataNode process is running on all the nodes it should be. Also check the logs of the datanode on the servers with no blocks. You probably have a misconfiguration. Shameless plug..sorry.. Take a look at http://www.jointhegrid.com/acod/index.jsp I made it to generate and push out hadoop configurations. One of the target audiences was first time multi-node setups. If you get a chance give it a try and let me know if it helps or makes things worse.
Re: should data be evenly distributed to each (physical) node
There's nothing like reading the manual: http://hadoop.apache.org/common/docs/r0.20.0/hdfs_design.html#Replica+Placement%3A+The+First+Baby+Steps Quote: For the common case, when the replication factor is three, HDFS’s placement policy is to put one replica on one node in the local rack, another on a different node in the local rack, and the last on a different node in a different rack. So if you write the data from only 1 machine, every block will have 1 replica on that machine (although you can run the balancer afterwards). J-D On Thu, Mar 4, 2010 at 7:25 AM, openresearch qiming...@openresearchinc.com wrote: I am building a small two node cluster following http://www.michael-noll.com/wiki/Running_Hadoop_On_Ubuntu_Linux_(Multi-Node_Cluster) Every thing seems to be working, except I notice the data are NOT evenly distributed to each physical box. e.g., when I hadoop dfs -put 6G data. I am expecting ~3G on each node (take turns every ~64MB), however, I checked dfshealth.jsp and du -k on local box, and found the uploaded data are ONLY residing on the physical box where I start dfs -put. That defeats the whole (data locality) purpose of hadoop?! Please help. Thanks -- View this message in context: http://old.nabble.com/should-data-be-evenly-distributed-to-each-%28physical%29-node-tp27782215p27782215.html Sent from the Hadoop core-user mailing list archive at Nabble.com.
Re: can't start namenode
We have a single dfs.name.dir directory, in case it's useful the contents are: [m...@carr name]$ ls -l total 8 drwxrwxr-x 2 mike mike 4096 Mar 4 11:18 current drwxrwxr-x 2 mike mike 4096 Oct 8 16:38 image On Thu, Mar 4, 2010 at 12:00 PM, Todd Lipcon t...@cloudera.com wrote: Hi Mike, Was your namenode configured with multiple dfs.name.dir settings? If so, can you please reply with ls -l from each dfs.name.dir? Thanks -Todd On Thu, Mar 4, 2010 at 8:57 AM, mike anderson saidthero...@gmail.com wrote: Our hadoop cluster went down last night when the namenode ran out of hard drive space. Trying to restart fails with this exception (see below). Since I don't really care that much about losing a days worth of data or so I'm fine with blowing away the edits file if that's what it takes (we don't have a secondary namenode to restore from). I tried removing the edits file from the namenode directory, but then it complained about not finding an edits file. I touched a blank edits file and I got the exact same exception. Any thoughts? I googled around a bit, but to no avail. -mike 2010-03-04 10:50:44,768 INFO org.apache.hadoop.ipc.metrics.RpcMetrics: Initializing RPC Metrics with hostName=NameNode, port=54310 2010-03-04 10:50:44,772 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: Namenode up at: carr.projectlounge.com/10.0.16.91:54310 2010-03-04 http://carr.projectlounge.com/10.0.16.91:54310%0A2010-03-0410:50:44,773 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing JVM Metrics with processName=NameNode, sessionId=null 2010-03-04 10:50:44,774 INFO org.apache.hadoop.hdfs.server.namenode.metrics.NameNodeMetrics: Initializing NameNodeMeterics using context object:org.apache.hadoop.metrics.spi.NullContext 2010-03-04 10:50:44,816 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner=pubget,pubget 2010-03-04 10:50:44,817 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup=supergroup 2010-03-04 10:50:44,817 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled=true 2010-03-04 10:50:44,823 INFO org.apache.hadoop.hdfs.server.namenode.metrics.FSNamesystemMetrics: Initializing FSNamesystemMetrics using context object:org.apache.hadoop.metrics.spi.NullContext 2010-03-04 10:50:44,825 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Registered FSNamesystemStatusMBean 2010-03-04 10:50:44,849 INFO org.apache.hadoop.hdfs.server.common.Storage: Number of files = 2687 2010-03-04 10:50:45,092 INFO org.apache.hadoop.hdfs.server.common.Storage: Number of files under construction = 7 2010-03-04 10:50:45,095 INFO org.apache.hadoop.hdfs.server.common.Storage: Image file of size 347821 loaded in 0 seconds. 2010-03-04 10:50:45,104 INFO org.apache.hadoop.hdfs.server.common.Storage: Edits file /mnt/hadoop/name/current/edits of size 4653 edits # 39 loaded in 0 seconds. 2010-03-04 10:50:45,114 ERROR org.apache.hadoop.hdfs.server.namenode.NameNode: java.lang.NumberFormatException: For input string: at java.lang.NumberFormatException.forInputString(NumberFormatException.java:48) at java.lang.Long.parseLong(Long.java:424) at java.lang.Long.parseLong(Long.java:461) at org.apache.hadoop.hdfs.server.namenode.FSEditLog.readLong(FSEditLog.java:1273) at org.apache.hadoop.hdfs.server.namenode.FSEditLog.loadFSEdits(FSEditLog.java:670) at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSEdits(FSImage.java:997) at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:812) at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:364) at org.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSDirectory.java:87) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesystem.java:311) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.init(FSNamesystem.java:292) at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:201) at org.apache.hadoop.hdfs.server.namenode.NameNode.init(NameNode.java:279) at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:956) at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:965) 2010-03-04 10:50:45,115 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG: / SHUTDOWN_MSG: Shutting down NameNode at carr.projectlounge.com/10.0.16.91 /
Re: Hadoop as master's thesis
On Mon, Mar 1, 2010 at 3:01 PM, Tonci Buljan tonci.bul...@gmail.com wrote: Hello everyone, I'm thinking of using Hadoop as a subject in my master's thesis in Computer Science. I'm supposed to solve some kind of a problem with Hadoop, but can't think of any :)). Here is an overview of hadoop/mapreduce algorithms that might be of inspiration when finding a problem to solve: http://atbrox.com/2010/02/12/mapreduce-hadoop-algorithms-in-academic-papers-updated/ A new dataset related to machine learning: http://learningtorankchallenge.yahoo.com/ Best regards, Amund We have a lab with 10-15 computers and I tough of installing Hadoop on those computers, and now I should write some kind of a program to run on my cluster. I really hope you understood my problem :). I really need any kind of suggestion. P.S. Sorry for my bad English, I'm from Croatia. -- http://atbrox.com - +47 416 26 572
Re: Hbase VS Hive
On 03/04/2010 01:19 AM, Michael Segel wrote: Date: Thu, 4 Mar 2010 00:42:11 +0700 From: fitrah.fird...@gmail.com To: common-user@hadoop.apache.org Subject: Hbase VS Hive Hello Everyone I want to ask about Hbase and Hive. What is the different between Hbase and Hive? and then what is the consideration for choose between Hbase or Hive? Hi, HBase is a column oriented database that sits on top of HDFS. You really want to use it if you're thinking about 'transactional' or random access of data within the data set. Hive sits on HDFS and generates Hadoop jobs to process data. I believe that Hive supports a SQL query type language (or am I confusing it with Pig) so you tend to write a query to walk through your data sets and perform a map reduce. HBase, you want to pull subsets or even individual rows of data from a very large data set. It can be used as part of Hadoop jobs or as a separate application. If you're asking if you want to choose one or the other, I'd say think about knowing both. Also how do you want to persist the data? In HDFS as flat files, or within a column oriented database. HTH -Mike Thanks for Your Reply, Based on your explain,if I want to build Data mining project for Decision Support System,I should choose hive,is it correct? Kind Regards Firdaus
Re: Will interactive password authentication fail talk between namenode-datanode/jobtracker-tasktracker?
On 3/3/10 3:38 PM, jiang licht licht_ji...@yahoo.com wrote: Here's my question, I have to type my password (not PASSPHRASE for key) due to some reverse name resolution problem when I do either SSH MASTER from SLAVE or SSH SLAVE from MASTER. Since my system admin told me all ports are open between them, I am wondering will this interactive authentication prevents datanode/tasktracker from talking to namenode/jobtracker? If reverse name resolution isn't working, then you are going to have all sorts of problems. DNS needs to be properly configured. [I wish I knew where the whole you don't need reverses configured for dns thing came from amongst admins.]
Re: Hbase VS Hive
On Thu, Mar 4, 2010 at 12:13 PM, Fitrah Elly Firdaus fitrah.fird...@gmail.com wrote: On 03/04/2010 01:19 AM, Michael Segel wrote: Date: Thu, 4 Mar 2010 00:42:11 +0700 From: fitrah.fird...@gmail.com To: common-user@hadoop.apache.org Subject: Hbase VS Hive Hello Everyone I want to ask about Hbase and Hive. What is the different between Hbase and Hive? and then what is the consideration for choose between Hbase or Hive? Hi, HBase is a column oriented database that sits on top of HDFS. You really want to use it if you're thinking about 'transactional' or random access of data within the data set. Hive sits on HDFS and generates Hadoop jobs to process data. I believe that Hive supports a SQL query type language (or am I confusing it with Pig) so you tend to write a query to walk through your data sets and perform a map reduce. HBase, you want to pull subsets or even individual rows of data from a very large data set. It can be used as part of Hadoop jobs or as a separate application. If you're asking if you want to choose one or the other, I'd say think about knowing both. Also how do you want to persist the data? In HDFS as flat files, or within a column oriented database. HTH -Mike Thanks for Your Reply, Based on your explain,if I want to build Data mining project for Decision Support System,I should choose hive,is it correct? Kind Regards Firdaus Hive and Hbase are very different. I am basically doing a rehash of what some people have said above with my own thoughts. It is hard to sum up complex projects in few words. To some people, Hbase looks like a distributed, persistent memcached with hadoop as a storage backend. It is designed for fast put, fast scans, fast gets and auto-sharding for linear scalability and performance. It works in the real time. If you have a data store that you need to scale past a couple of nodes, but need real-time put/get/scan hbase might be for you. To some people, Hive looks like a distributed, RDBMS, with hadoop as a storage backend. Hive has a query language that looks like SQL. HQL is missing some things that relation databases do, but adds some things that relation database can not easily do. http://wiki.apache.org/hadoop/Hive/LanguageManual Hive is not a real time system. It goals are not to support Real time, select , insertions, no updates. Hive allows you to wrap a schema around a file in HDFS and treat is as a table and then query it. These files can be (and usually are) really really really large. Hive can effectively parallelize queries using map/reduce, something that traditional single node database can not normally do. Because hive can work over files in HDFS it can deal with input in many formats. Below is a new feature being added that will allow hive to do queries over HBase data (Hbase input format) https://issues.apache.org/jira/browse/HIVE-705 This is something the Hive community is very excited about. It opens up many doors, for both hbase and Hive.
some doubts
I am using ubuntu Linux. I was able to get the standalone hadoop cluster running and run the wordcount example. before i start writing hadoop programs i wanted to compile the wordcount example on my own. So this is what i did to make the jar file on my own. javac -classpath /home/varun/hadoop/hadoop-0.20.1/hadoop-0.20.1-core.jar WordCount.java jar -cvf wordcount.jar -C /media/d/iproggys/java/Hadoop/src/wordcount/ . Is this the correct way to do it? I had one more doubt while running the example.This is what i do to run the mapreduce job. bin/hadoop jar hadoop-0.20.1-examples.jar wordcount gutenberg gutenberg-output what is wordcount? gutenberg being the input dir. gutenberg-output being the out dir. -- Regards, Varun Thacker http://varunthacker.wordpress.com
Re: Pipelining data from map to reduce
Bharath, This idea is kicking around in academia.. not made into apache yet.. https://issues.apache.org/jira/browse/MAPREDUCE-1211 You can get a working prototype from: http://code.google.com/p/hop/ Ashutosh On Thu, Mar 4, 2010 at 09:06, E. Sammer e...@lifeless.net wrote: On 3/4/10 12:00 PM, bharath v wrote: Hi , Can we pipeline the map output directly into reduce phase without storing it in the local filesystem (avoiding disk IOs). If yes , how to do that ? Bharath: No, there's no way to avoid going to disk after the mappers. -- Eric Sammer e...@lifeless.net http://esammer.blogspot.com
Re: can't start namenode
Hi Mike, Since you removed the edits, you restored to an earlier version of the namesystem. Thus, any files that were deleted since the last checkpoint will have come back. But, the blocks will have been removed from the datanodes. So, the NN is complaining since there are some files that have missing blocks. That is to say, some of your files are corrupt (ie unreadable because the data is gone but the metadata is still there) In order to force it out of safemode, you can run hadoop dfsadmin -safemode leave You should also run hadoop fsck in order to determine which files are broken, and then probably use the -delete option to remove their metadata. Thanks -Todd On Thu, Mar 4, 2010 at 11:37 AM, mike anderson saidthero...@gmail.comwrote: Removing edits.new and starting worked, though it didn't seem that happy about it. It started up nonetheless, in safe mode. Saying that The ratio of reported blocks 0.9948 has not reached the threshold 0.9990. Safe mode will be turned off automatically. Unfortunately this is holding up the restart of hbase. About how long does it take to exit safe mode? is there anything I can do to expedite the process? On Thu, Mar 4, 2010 at 1:54 PM, Todd Lipcon t...@cloudera.com wrote: Sorry, I actually meant ls -l from name.dir/current/ Having only one dfs.name.dir isn't recommended - after you get your system back up and running I would strongly suggest running with at least two, preferably with one on a separate server via NFS. Thanks -Todd On Thu, Mar 4, 2010 at 9:05 AM, mike anderson saidthero...@gmail.com wrote: We have a single dfs.name.dir directory, in case it's useful the contents are: [m...@carr name]$ ls -l total 8 drwxrwxr-x 2 mike mike 4096 Mar 4 11:18 current drwxrwxr-x 2 mike mike 4096 Oct 8 16:38 image On Thu, Mar 4, 2010 at 12:00 PM, Todd Lipcon t...@cloudera.com wrote: Hi Mike, Was your namenode configured with multiple dfs.name.dir settings? If so, can you please reply with ls -l from each dfs.name.dir? Thanks -Todd On Thu, Mar 4, 2010 at 8:57 AM, mike anderson saidthero...@gmail.com wrote: Our hadoop cluster went down last night when the namenode ran out of hard drive space. Trying to restart fails with this exception (see below). Since I don't really care that much about losing a days worth of data or so I'm fine with blowing away the edits file if that's what it takes (we don't have a secondary namenode to restore from). I tried removing the edits file from the namenode directory, but then it complained about not finding an edits file. I touched a blank edits file and I got the exact same exception. Any thoughts? I googled around a bit, but to no avail. -mike 2010-03-04 10:50:44,768 INFO org.apache.hadoop.ipc.metrics.RpcMetrics: Initializing RPC Metrics with hostName=NameNode, port=54310 2010-03-04 10:50:44,772 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: Namenode up at: carr.projectlounge.com/10.0.16.91:54310 2010-03-04 http://carr.projectlounge.com/10.0.16.91:54310%0A2010-03-04 10:50:44,773 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing JVM Metrics with processName=NameNode, sessionId=null 2010-03-04 10:50:44,774 INFO org.apache.hadoop.hdfs.server.namenode.metrics.NameNodeMetrics: Initializing NameNodeMeterics using context object:org.apache.hadoop.metrics.spi.NullContext 2010-03-04 10:50:44,816 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner=pubget,pubget 2010-03-04 10:50:44,817 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup=supergroup 2010-03-04 10:50:44,817 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled=true 2010-03-04 10:50:44,823 INFO org.apache.hadoop.hdfs.server.namenode.metrics.FSNamesystemMetrics: Initializing FSNamesystemMetrics using context object:org.apache.hadoop.metrics.spi.NullContext 2010-03-04 10:50:44,825 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Registered FSNamesystemStatusMBean 2010-03-04 10:50:44,849 INFO org.apache.hadoop.hdfs.server.common.Storage: Number of files = 2687 2010-03-04 10:50:45,092 INFO org.apache.hadoop.hdfs.server.common.Storage: Number of files under construction = 7 2010-03-04 10:50:45,095 INFO org.apache.hadoop.hdfs.server.common.Storage: Image file of size 347821 loaded in 0 seconds. 2010-03-04 10:50:45,104 INFO org.apache.hadoop.hdfs.server.common.Storage: Edits file /mnt/hadoop/name/current/edits of size 4653 edits # 39 loaded in 0 seconds. 2010-03-04 10:50:45,114 ERROR org.apache.hadoop.hdfs.server.namenode.NameNode: java.lang.NumberFormatException:
Re: can't start namenode
Todd, That did the trick. Thanks to everyone for the quick responses and effective suggestions. -Mike On Thu, Mar 4, 2010 at 2:50 PM, Todd Lipcon t...@cloudera.com wrote: Hi Mike, Since you removed the edits, you restored to an earlier version of the namesystem. Thus, any files that were deleted since the last checkpoint will have come back. But, the blocks will have been removed from the datanodes. So, the NN is complaining since there are some files that have missing blocks. That is to say, some of your files are corrupt (ie unreadable because the data is gone but the metadata is still there) In order to force it out of safemode, you can run hadoop dfsadmin -safemode leave You should also run hadoop fsck in order to determine which files are broken, and then probably use the -delete option to remove their metadata. Thanks -Todd On Thu, Mar 4, 2010 at 11:37 AM, mike anderson saidthero...@gmail.comwrote: Removing edits.new and starting worked, though it didn't seem that happy about it. It started up nonetheless, in safe mode. Saying that The ratio of reported blocks 0.9948 has not reached the threshold 0.9990. Safe mode will be turned off automatically. Unfortunately this is holding up the restart of hbase. About how long does it take to exit safe mode? is there anything I can do to expedite the process? On Thu, Mar 4, 2010 at 1:54 PM, Todd Lipcon t...@cloudera.com wrote: Sorry, I actually meant ls -l from name.dir/current/ Having only one dfs.name.dir isn't recommended - after you get your system back up and running I would strongly suggest running with at least two, preferably with one on a separate server via NFS. Thanks -Todd On Thu, Mar 4, 2010 at 9:05 AM, mike anderson saidthero...@gmail.com wrote: We have a single dfs.name.dir directory, in case it's useful the contents are: [m...@carr name]$ ls -l total 8 drwxrwxr-x 2 mike mike 4096 Mar 4 11:18 current drwxrwxr-x 2 mike mike 4096 Oct 8 16:38 image On Thu, Mar 4, 2010 at 12:00 PM, Todd Lipcon t...@cloudera.com wrote: Hi Mike, Was your namenode configured with multiple dfs.name.dir settings? If so, can you please reply with ls -l from each dfs.name.dir? Thanks -Todd On Thu, Mar 4, 2010 at 8:57 AM, mike anderson saidthero...@gmail.com wrote: Our hadoop cluster went down last night when the namenode ran out of hard drive space. Trying to restart fails with this exception (see below). Since I don't really care that much about losing a days worth of data or so I'm fine with blowing away the edits file if that's what it takes (we don't have a secondary namenode to restore from). I tried removing the edits file from the namenode directory, but then it complained about not finding an edits file. I touched a blank edits file and I got the exact same exception. Any thoughts? I googled around a bit, but to no avail. -mike 2010-03-04 10:50:44,768 INFO org.apache.hadoop.ipc.metrics.RpcMetrics: Initializing RPC Metrics with hostName=NameNode, port=54310 2010-03-04 10:50:44,772 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: Namenode up at: carr.projectlounge.com/10.0.16.91:54310 2010-03-04 http://carr.projectlounge.com/10.0.16.91:54310%0A2010-03-04 10:50:44,773 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing JVM Metrics with processName=NameNode, sessionId=null 2010-03-04 10:50:44,774 INFO org.apache.hadoop.hdfs.server.namenode.metrics.NameNodeMetrics: Initializing NameNodeMeterics using context object:org.apache.hadoop.metrics.spi.NullContext 2010-03-04 10:50:44,816 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner=pubget,pubget 2010-03-04 10:50:44,817 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup=supergroup 2010-03-04 10:50:44,817 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled=true 2010-03-04 10:50:44,823 INFO org.apache.hadoop.hdfs.server.namenode.metrics.FSNamesystemMetrics: Initializing FSNamesystemMetrics using context object:org.apache.hadoop.metrics.spi.NullContext 2010-03-04 10:50:44,825 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Registered FSNamesystemStatusMBean 2010-03-04 10:50:44,849 INFO org.apache.hadoop.hdfs.server.common.Storage: Number of files = 2687 2010-03-04 10:50:45,092 INFO org.apache.hadoop.hdfs.server.common.Storage: Number of files under construction = 7 2010-03-04 10:50:45,095 INFO org.apache.hadoop.hdfs.server.common.Storage: Image file of size 347821 loaded in 0 seconds. 2010-03-04 10:50:45,104 INFO org.apache.hadoop.hdfs.server.common.Storage: Edits file
Re: Hadoop on Azure
You can take a look at www.zidata.com which provides a full windows experience on top of Hadoop. On Thu, Mar 4, 2010 at 12:41 PM, jawaid ekram jek...@hotmail.com wrote: Is there a Hadoop impletmentation on Azure cloud?
Re: Sorting
Sample your input data and use the sample to drive your partitioner. Please take a look at TeraSort example in org.apache.hadoop.examples.terasort. Arun On Mar 3, 2010, at 9:21 AM, Aayush Garg wrote: Hi, Suppose I do need to sort a big file(in GB). How would I accomplish this task using hadoop. My main problem is how to merge the output of individual reduce phases? thanks
Re: Pipelining data from map to reduce
Also see Breaking the MapReduce Stage Barrier from UIUC: http://www.ideals.illinois.edu/bitstream/handle/2142/14819/breaking.pdf On Thu, Mar 4, 2010 at 11:41 AM, Ashutosh Chauhan ashutosh.chau...@gmail.com wrote: Bharath, This idea is kicking around in academia.. not made into apache yet.. https://issues.apache.org/jira/browse/MAPREDUCE-1211 You can get a working prototype from: http://code.google.com/p/hop/ Ashutosh On Thu, Mar 4, 2010 at 09:06, E. Sammer e...@lifeless.net wrote: On 3/4/10 12:00 PM, bharath v wrote: Hi , Can we pipeline the map output directly into reduce phase without storing it in the local filesystem (avoiding disk IOs). If yes , how to do that ? Bharath: No, there's no way to avoid going to disk after the mappers. -- Eric Sammer e...@lifeless.net http://esammer.blogspot.com
Re: Pipelining data from map to reduce
Interesting article. It claims to have the same fault tolerance but I don't see any explanation of how that can be. If a single mapper fails part-way through a task when it has transmitted partial results to a reducer, the whole job is corrupted. With the current barrier between map and reduce, a job can recover from partially completed tasks and speculatively execute. I would imagine that small low latency tasks can benefit greatly from such an approach, but larger tasks need the barrier or will not be very fault tolerant. However, there is still a lot of optimizations to dot in Hadoop for low latency tasks while maintaining the barrier. On Mar 4, 2010, at 2:18 PM, Jeff Hammerbacher wrote: Also see Breaking the MapReduce Stage Barrier from UIUC: http://www.ideals.illinois.edu/bitstream/handle/2142/14819/breaking.pdf On Thu, Mar 4, 2010 at 11:41 AM, Ashutosh Chauhan ashutosh.chau...@gmail.com wrote: Bharath, This idea is kicking around in academia.. not made into apache yet.. https://issues.apache.org/jira/browse/MAPREDUCE-1211 You can get a working prototype from: http://code.google.com/p/hop/ Ashutosh On Thu, Mar 4, 2010 at 09:06, E. Sammer e...@lifeless.net wrote: On 3/4/10 12:00 PM, bharath v wrote: Hi , Can we pipeline the map output directly into reduce phase without storing it in the local filesystem (avoiding disk IOs). If yes , how to do that ? Bharath: No, there's no way to avoid going to disk after the mappers. -- Eric Sammer e...@lifeless.net http://esammer.blogspot.com
Re: Pipelining data from map to reduce
I guess if you emmitted the key as task-id+ key you would have more overhead but if the data replayed the reducer could detect dups. Ed On 3/4/10, Scott Carey sc...@richrelevance.com wrote: Interesting article. It claims to have the same fault tolerance but I don't see any explanation of how that can be. If a single mapper fails part-way through a task when it has transmitted partial results to a reducer, the whole job is corrupted. With the current barrier between map and reduce, a job can recover from partially completed tasks and speculatively execute. I would imagine that small low latency tasks can benefit greatly from such an approach, but larger tasks need the barrier or will not be very fault tolerant. However, there is still a lot of optimizations to dot in Hadoop for low latency tasks while maintaining the barrier. On Mar 4, 2010, at 2:18 PM, Jeff Hammerbacher wrote: Also see Breaking the MapReduce Stage Barrier from UIUC: http://www.ideals.illinois.edu/bitstream/handle/2142/14819/breaking.pdf On Thu, Mar 4, 2010 at 11:41 AM, Ashutosh Chauhan ashutosh.chau...@gmail.com wrote: Bharath, This idea is kicking around in academia.. not made into apache yet.. https://issues.apache.org/jira/browse/MAPREDUCE-1211 You can get a working prototype from: http://code.google.com/p/hop/ Ashutosh On Thu, Mar 4, 2010 at 09:06, E. Sammer e...@lifeless.net wrote: On 3/4/10 12:00 PM, bharath v wrote: Hi , Can we pipeline the map output directly into reduce phase without storing it in the local filesystem (avoiding disk IOs). If yes , how to do that ? Bharath: No, there's no way to avoid going to disk after the mappers. -- Eric Sammer e...@lifeless.net http://esammer.blogspot.com
Data node cannot talk to name node Re: Will interactive password authentication fail talk between namenode-datanode/jobtracker-tasktracker?
Thanks Edward. Since the string of reverse mapping ... is just a warning, I guess it won't be a issue. Now, the namenode A is listening on port a. No data node sitting on a different box can talk to a...@a to join the cluster. But assign A also as a datanode is ok and this datanode can join the cluster since it sits on the same box as namenode. Telnet from other machines to a...@a fails. To verify that the slave box is allowed, I also added both IP and name (according to boolean inHostsList(DatanodeID node, String ipAddr) in FSNamesystem.java which does access control) to slaves and the file specified for dfs.hosts but I didn't use a exclude list. Namenode box can ssh to itself and to slaves. Also I verified the IP address of slave is the ip address listed in slaves and the file specified for dfs.hosts. And the ports are open for access from slaves on the namenode machine. For example, from slaves, telnet to 50070/50...@namenode is allowed and by firing a GET / returns correct html pages. So, ssh is ok and access control is ok. Really confusing. What may cause this problem which prevents data nodes from talking to namenode? Any thoughts? Thanks, -- Michael --- On Thu, 3/4/10, Edward Capriolo edlinuxg...@gmail.com wrote: From: Edward Capriolo edlinuxg...@gmail.com Subject: Re: Will interactive password authentication fail talk between namenode-datanode/jobtracker-tasktracker? To: common-user@hadoop.apache.org Date: Thursday, March 4, 2010, 3:23 PM On Thu, Mar 4, 2010 at 2:01 PM, Edson Ramiro erlfi...@gmail.com wrote: You don't need DNS. You can use the /etc/hosts. I'm using it here and it's working well. Edson Ramiro On 4 March 2010 14:39, Allen Wittenauer awittena...@linkedin.com wrote: On 3/3/10 3:38 PM, jiang licht licht_ji...@yahoo.com wrote: Here's my question, I have to type my password (not PASSPHRASE for key) due to some reverse name resolution problem when I do either SSH MASTER from SLAVE or SSH SLAVE from MASTER. Since my system admin told me all ports are open between them, I am wondering will this interactive authentication prevents datanode/tasktracker from talking to namenode/jobtracker? If reverse name resolution isn't working, then you are going to have all sorts of problems. DNS needs to be properly configured. [I wish I knew where the whole you don't need reverses configured for dns thing came from amongst admins.] Adding an entry to DNS adds the entry to your forward and reverse lookup table, thus fixing/hiding the problem. Host file is not a winning strategy long term for obvious reasons :)
