Re: dfs.name.dir capacity for namenode backup?
On Mon, May 17, 2010 at 5:10 PM, jiang licht wrote: > I am considering to use a machine to save a > redundant copy of HDFS metadata through setting dfs.name.dir in > hdfs-site.xml like this (as in YDN): > > >dfs.name.dir >/home/hadoop/dfs/name,/mnt/namenode-backup >true > > > where the two folders are on different machines so that > /mnt/namenode-backup keeps a copy of hdfs file system information and its > machine can be used to replace the first machine that fails as namenode. > > So, my question is how big this hdfs metatdata will consume? I guess it is > proportional to the hdfs capacity. What ratio is that or what size will be > for 150TB hdfs? > On the order of a few GB, max (you really need double the size of your image, so it has tmp space when downloading a checkpoint or performing an upgrade). But on any disk you can buy these days you'll have plenty of space. -Todd -- Todd Lipcon Software Engineer, Cloudera
Data node decommission doesn't seem to be working correctly
I followed the steps mentioned here: http://developer.yahoo.com/hadoop/tutorial/module2.html#decommission to decommission a data node. What I see from the namenode is the hostname of the machine that I decommissioned shows up in both the list of dead nodes but also live nodes where its admin status is marked as 'In Service'. It's been twelve hours and there is no sign in the namenode logs that the node has been decommissioned. Any suggestions of what might be the problem and what to try to ensure that this node gets safely taken down? thanks in advance, Scott
Re: distcp of small number of really large files
There is a new feature called concat(), which concatenates files consisting of full blocks. So the ideas is to copy individual blocks in parallel, then concatenate them once they are copied back into original files. You will have to write some code to do this or modify distcp. This is in 0.22/21, but not in 0.20. --Konstantin On 5/17/2010 5:10 PM, Mridul Muralidharan wrote: Hi, Is there a way to parallelize copy of really large files ? From my understanding, currently a each map in distcp copies one file. So for really large files, this would be pretty slow if number of files is really large. Thanks, Mridul
distcp of small number of really large files
Hi, Is there a way to parallelize copy of really large files ? From my understanding, currently a each map in distcp copies one file. So for really large files, this would be pretty slow if number of files is really large. Thanks, Mridul
dfs.name.dir capacity for namenode backup?
I am considering to use a machine to save a redundant copy of HDFS metadata through setting dfs.name.dir in hdfs-site.xml like this (as in YDN): dfs.name.dir /home/hadoop/dfs/name,/mnt/namenode-backup true where the two folders are on different machines so that /mnt/namenode-backup keeps a copy of hdfs file system information and its machine can be used to replace the first machine that fails as namenode. So, my question is how big this hdfs metatdata will consume? I guess it is proportional to the hdfs capacity. What ratio is that or what size will be for 150TB hdfs? Thanks, Michael
Re: Setting up a second cluster and getting a weird issue
Sorry for bothering everyone, I accidentally configured my dfs.data.dir and mapred.local.dir to the same directory... Bad copy/paste job. Thanks for everyone's help!
Re: Setting up a second cluster and getting a weird issue
So I pulled everything of NFS and I'm still getting the original error with a FileNotFoundException for current/VERSION. I only have 4 slaves and scp'ed the Hadoop directory to all 4 slaves. Any other ideas? On May 14, 2010, at 7:41 PM, Hemanth Yamijala wrote: > Andrew, > >> Just to be clear, I'm only sharing the Hadoop binaries and config files via >> NFS. I don't see how this would cause a conflict - do you have any >> additional information? > > FWIW, we had an experience where we were storing config files on NFS > on a large cluster. Randomly, (and we guess due to NFS problems), > Hadoop would fail picking up the config files on NFS and instead use > its defaults. The config values for some directory paths defined in > default being different from the actual config values was resulting in > very odd errors. We were able to eventually solve the problem by > moving the config files off NFS. Of course, the size of the cluster > (several hundreds of slaves) was probably a reason. But nevertheless, > you may want to try pulling everything off NFS. > > Thanks > Hemanth > >> >> The referenced path in the error below (/srv/hadoop/dfs/1) is not being >> shared via NFS... >> >> Thanks, >> Andrew >> >> On May 13, 2010, at 6:51 PM, Jeff Zhang wrote: >> >>> It is not suggested to deploy hadoop on NFS, there will be conflict >>> between data nodes, because NFS share the same namespace of file >>> system. >>> >>> >>> >>> On Thu, May 13, 2010 at 9:52 PM, Andrew Nguyen wrote: Yes, in this deployment, I'm attempting to share the hadoop files via NFS. The log and pid directories are local. Thanks! --Andrew On May 12, 2010, at 7:40 PM, Jeff Zhang wrote: > These 4 nodes share NFS ？ > > > On Thu, May 13, 2010 at 8:19 AM, Andrew Nguyen > wrote: >> I'm working on bringing up a second test cluster and am getting these >> intermittent errors on the DataNodes: >> >> 2010-05-12 17:17:15,094 ERROR >> org.apache.hadoop.hdfs.server.datanode.DataNode: >> java.io.FileNotFoundException: /srv/hadoop/dfs/1/current/VERSION (No >> such file or directory) >>at java.io.RandomAccessFile.open(Native Method) >>at java.io.RandomAccessFile.(RandomAccessFile.java:212) >>at >> org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.write(Storage.java:249) >>at >> org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.write(Storage.java:243) >>at >> org.apache.hadoop.hdfs.server.common.Storage.writeAll(Storage.java:689) >>at >> org.apache.hadoop.hdfs.server.datanode.DataNode.register(DataNode.java:560) >>at >> org.apache.hadoop.hdfs.server.datanode.DataNode.runDatanodeDaemon(DataNode.java:1230) >>at >> org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:1273) >>at >> org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:1394) >> >> >> There are 4 slaves and sometimes 1 or 2 have the error but the specific >> nodes change. Sometimes it's slave1, sometimes it's slave4, etc. >> >> Any thoughts? >> >> Thanks! >> >> --Andrew > > > > -- > Best Regards > > Jeff Zhang >>> >>> >>> >>> -- >>> Best Regards >>> >>> Jeff Zhang >> >>
Re: Build a indexing and search service with Hadoop
http://lucene.apache.org/java/3_0_1/api/all/org/apache/lucene/search/ParallelMultiSearcher.html Ian Aécio writes: > Thanks for the replies. > > I'm already investigating how katta works and how I can extend it. > What do you mean by distributed search capability? Lucene provives any way > to "merge" hits from diferent indexes? > > 2010/5/14 Ian Soboroff > >> Aécio writes: >> >> > 2. Search >> > - The query received is used as input of the map function. This function >> > would search the document on the local shard using our custom library and >> > emit the hits. The reduce function would group the hits from all shards. >> >> There is no way you can do interactive searches via MapReduce in Hadoop, >> because the JVM start time will kill you. If your shard backend is >> Lucene, just use the distributed search capability already there, or >> look at Katta. >> >> Ian >>
Re: NameNode deadlocked (help?)
Brian Bockelman wrote: On May 17, 2010, at 5:25 AM, Steve Loughran wrote: Brian Bockelman wrote: On May 14, 2010, at 8:27 PM, Todd Lipcon wrote: Hey Brian, Yep, excessive GC definitely sounds like a likely culprit. I'm surprised you didn't see OOMEs in the log, though. We didn't until the third restart today. I have no clue why we haven't seen this in the past 9 months of this cluster though... Anyhow, it looks like this might have done the trick... the sysadmin is heading over to kick over a few errant datanodes, and we should be able to get out of safemode soon. Luckily, it's a 4-day weekend in Europe and otherwise a Friday evening in the US, so there's only a few folks using it. good thing we europeans have long weekends. :) Indeed If you want to monitor GC, I'd recommend adding -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps to your java options - occasionally useful for times like this. What are your current GC options? Played with compressed object pointers yet? I've been eyeballing them, but haven't had any chance yet. We probably won't mess with them until we start to run out of RAM on the machine themselves. This particular instance was a simple oversight - there was no need to try and fit the NN into a 1GB heap on a dedicated machine. I tell folks 1GB RAM per 1M objects. It's almost always an over-estimate but, better safe than deadlocked on a Friday evening... I've been using compressed pointers on JRockit for a long time, a very nice JVM that doesn't ever seem to run out of stack when you accidentally tail recurse without end. The Sun JVM pointers are newer, not had any problems with that part of the JVM, and the benefits in both memory consumption and possibly in cache hits make it very appealing.
Re: NameNode deadlocked (help?)
On May 17, 2010, at 5:25 AM, Steve Loughran wrote: > Brian Bockelman wrote: >> On May 14, 2010, at 8:27 PM, Todd Lipcon wrote: >>> Hey Brian, >>> >>> Yep, excessive GC definitely sounds like a likely culprit. I'm surprised you >>> didn't see OOMEs in the log, though. >>> >> We didn't until the third restart today. I have no clue why we haven't seen >> this in the past 9 months of this cluster though... >> Anyhow, it looks like this might have done the trick... the sysadmin is >> heading over to kick over a few errant datanodes, and we should be able to >> get out of safemode soon. Luckily, it's a 4-day weekend in Europe and >> otherwise a Friday evening in the US, so there's only a few folks using it. > > good thing we europeans have long weekends. > :) Indeed >>> If you want to monitor GC, I'd recommend adding -verbose:gc >>> -XX:+PrintGCDetails -XX:+PrintGCDateStamps to your java options - >>> occasionally useful for times like this. >>> > > What are your current GC options? Played with compressed object pointers yet? I've been eyeballing them, but haven't had any chance yet. We probably won't mess with them until we start to run out of RAM on the machine themselves. This particular instance was a simple oversight - there was no need to try and fit the NN into a 1GB heap on a dedicated machine. I tell folks 1GB RAM per 1M objects. It's almost always an over-estimate but, better safe than deadlocked on a Friday evening... Brian smime.p7s Description: S/MIME cryptographic signature
Re: NameNode deadlocked (help?)
Brian Bockelman wrote: On May 14, 2010, at 8:27 PM, Todd Lipcon wrote: Hey Brian, Yep, excessive GC definitely sounds like a likely culprit. I'm surprised you didn't see OOMEs in the log, though. We didn't until the third restart today. I have no clue why we haven't seen this in the past 9 months of this cluster though... Anyhow, it looks like this might have done the trick... the sysadmin is heading over to kick over a few errant datanodes, and we should be able to get out of safemode soon. Luckily, it's a 4-day weekend in Europe and otherwise a Friday evening in the US, so there's only a few folks using it. good thing we europeans have long weekends. If you want to monitor GC, I'd recommend adding -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps to your java options - occasionally useful for times like this. What are your current GC options? Played with compressed object pointers yet?
