dfs.block.size vs avg block size
Hello, I checked the ML archives and the Wiki, as well as the HDFS user guide, but could not find information about how to change block size of an existing HDFS. After running fsck I can see that my avg. block size is 12706144 B (cca 12MB), and that's a lot smaller than what I have configured: dfs.block.size=67108864 B Is the difference between the configured block size and actual (avg) block size results effectively wasted space? If so, is there a way to change the DFS block size and have Hadoop shrink all the existing blocks? I am OK with not running any jobs on the cluster for a day or two if I can do something to free up the wasted disk space. Thanks, Otis -- Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
Re: java.io.IOException: Could not obtain block / java.io.IOException: Could not get block locations
Hi dhruba, we are running the latest Sun Java 6u10-beta, and the namenode runs with 25 threads on a quad core machine. Cu on the 'net, Bye - bye, < André èrbnA > Dhruba Borthakur wrote: What version of java are you using? How may threads are you running on the namenode? How many cores does your machines have? thanks, dhruba On Fri, May 16, 2008 at 6:02 AM, André Martin <[EMAIL PROTECTED]> wrote: Hi Hadoopers, we are experiencing a lot of "Could not obtain block / Could not get block locations IOExceptions" when processing a 400 GB large Map/Red job using our 6 nodes DFS & MapRed (v. 0.16.4) cluster. Each node is equipped with a 400GB Sata HDD and running Suse Linux Enterprise Edition. While processing this "huge" MapRed job, the name node doesn't seem to receive heartbeats from datanodes for up to a couple of minutes and thus marks those nodes as dead even they are still alive and serving blocks according to their logs. We first suspected network congestion and measured the inter-node bandwidth using scp - receiving throughputs of 30MB/s. CPU utilization is about 100% when processing the job, however, the tasktracker instances shouldn't cause such datanode drop outs? In the datanode logs, we see a lot of java.io.IOException: Block blk_-7943096461180653598 is valid, and cannot be written to. errors... Any ideas? Thanks in advance. Cu on the 'net, Bye - bye, < André èrbnA >
Re: Making the case for Hadoop
thanks but I read the list before I posted. I was hoping for examples a bit closer to what we're planning to use it for, i.e. as the storage for media assets. Most people seem to use it rather for large amounts of not-so-critical or processing of temporary data. lohit wrote: You could also find some info about companies/projects using Hadoop at PoweredBy page http://wiki.apache.org/hadoop/PoweredBy Thanks, Lohit - Original Message From: Ted Dunning <[EMAIL PROTECTED]> To: core-user@hadoop.apache.org Sent: Friday, May 16, 2008 10:02:25 AM Subject: Re: Making the case for Hadoop Nothing is the best case! On 5/16/08 7:00 AM, "Edward Capriolo" <[EMAIL PROTECTED]> wrote: So hadoop is a fact. My advice for convincing IT executives. Ask them to present their alternative. (usually its nothing)
Re: Mirroring data to a non-Hadoop FS
The reasoning was that in the event of system-inherent failures (i.e. bugs in HDFS which corrupt the files) a system set up with a completely different technology would protect from that type of failure would prevent it from becoming catastrophic. Sounds (and probably in our case is) a bit paranoid but is common practice e.g. in the aerospace industry for really critical systems. Ted Dunning wrote: Why not go to the next step and use a second cluster as the backup? On 5/16/08 6:33 AM, "Robert Krüger" <[EMAIL PROTECTED]> wrote: Hi, what are the options to keep a copy of data from an HDFS instance in sync with a backup file system which is not HDFS? Are there Rsync-like tools that allow only to transfer deltas or would one have to implement that oneself (e.g. by writing a java program that accesses both filesystems)? Thanks in advance, Robert P.S.: Why would one want that? E.g. to have a completely redundant copy which in case of systematic failure (e.g. data corruption due to a bug) offers a backup not affected by that problem.
Re: About HDFS`s Certification and Authorization?
Release 0.15 does not have any permission/security control. Release 0.16 supports permission control. An initial design of user authentication is coming soon. A jira issue regarding this will open in the next couple of weeks. Please contribute if you have any ideas. Hairong On 5/16/08 1:32 AM, "wangxiaowei" <[EMAIL PROTECTED]> wrote: > hi,all > I now use hadoop-0.15.3.Does it`s HDFS have the functionality of > certification and authorization? So that one user can just access one part of > HDFS,and cann`t access other parts without permitting?If it does ,how can I > implement it? > Thanks a lot.
Re: Mirroring data to a non-Hadoop FS
There was some chatter on the Hbase list about a dual hdfs/s3 driver class which would write to both but only read from hdfs. Of course, having this functionality at the hadoop level would be better than in a subsidiary project. Maybe the ability to specify a secondary filesystem in the hadoop-site.xml? Candidates might include S3, NFS, or of course, another HDFS in a geographically isolated location. -- Jim R. Wilson (jimbojw) On Fri, May 16, 2008 at 12:06 PM, Ted Dunning <[EMAIL PROTECTED]> wrote: > > Why not go to the next step and use a second cluster as the backup? > > > On 5/16/08 6:33 AM, "Robert Krüger" <[EMAIL PROTECTED]> wrote: > >> >> Hi, >> >> what are the options to keep a copy of data from an HDFS instance in >> sync with a backup file system which is not HDFS? Are there Rsync-like >> tools that allow only to transfer deltas or would one have to implement >> that oneself (e.g. by writing a java program that accesses both >> filesystems)? >> >> Thanks in advance, >> >> Robert >> >> P.S.: Why would one want that? E.g. to have a completely redundant copy >> which in case of systematic failure (e.g. data corruption due to a bug) >> offers a backup not affected by that problem. > >
Re: Making the case for Hadoop
You could also find some info about companies/projects using Hadoop at PoweredBy page http://wiki.apache.org/hadoop/PoweredBy Thanks, Lohit - Original Message From: Ted Dunning <[EMAIL PROTECTED]> To: core-user@hadoop.apache.org Sent: Friday, May 16, 2008 10:02:25 AM Subject: Re: Making the case for Hadoop Nothing is the best case! On 5/16/08 7:00 AM, "Edward Capriolo" <[EMAIL PROTECTED]> wrote: > So hadoop is a fact. My advice for convincing IT executives. Ask them > to present their alternative. (usually its nothing)
Re: java.io.IOException: Could not obtain block / java.io.IOException: Could not get block locations
What version of java are you using? How may threads are you running on the namenode? How many cores does your machines have? thanks, dhruba On Fri, May 16, 2008 at 6:02 AM, André Martin <[EMAIL PROTECTED]> wrote: > Hi Hadoopers, > we are experiencing a lot of "Could not obtain block / Could not get block > locations IOExceptions" when processing a 400 GB large Map/Red job using our > 6 nodes DFS & MapRed (v. 0.16.4) cluster. Each node is equipped with a 400GB > Sata HDD and running Suse Linux Enterprise Edition. While processing this > "huge" MapRed job, the name node doesn't seem to receive heartbeats from > datanodes for up to a couple of minutes and thus marks those nodes as dead > even they are still alive and serving blocks according to their logs. We > first suspected network congestion and measured the inter-node bandwidth > using scp - receiving throughputs of 30MB/s. CPU utilization is about 100% > when processing the job, however, the tasktracker instances shouldn't cause > such datanode drop outs? > In the datanode logs, we see a lot of java.io.IOException: Block > blk_-7943096461180653598 is valid, and cannot be written to. errors... > > Any ideas? Thanks in advance. > > Cu on the 'net, > Bye - bye, > > < André èrbnA > > > >
Re: Mirroring data to a non-Hadoop FS
Why not go to the next step and use a second cluster as the backup? On 5/16/08 6:33 AM, "Robert Krüger" <[EMAIL PROTECTED]> wrote: > > Hi, > > what are the options to keep a copy of data from an HDFS instance in > sync with a backup file system which is not HDFS? Are there Rsync-like > tools that allow only to transfer deltas or would one have to implement > that oneself (e.g. by writing a java program that accesses both > filesystems)? > > Thanks in advance, > > Robert > > P.S.: Why would one want that? E.g. to have a completely redundant copy > which in case of systematic failure (e.g. data corruption due to a bug) > offers a backup not affected by that problem.
Re: Making the case for Hadoop
Nothing is the best case! On 5/16/08 7:00 AM, "Edward Capriolo" <[EMAIL PROTECTED]> wrote: > So hadoop is a fact. My advice for convincing IT executives. Ask them > to present their alternative. (usually its nothing)
Re: How do people keep their client configurations in sync with the remote cluster(s)
That is all that almost all of my arms-length clients need. With 18, all clients should be able to ask for the default configuration if they have a root URL which will make the amount of information needed for any and all clients very small. On 5/16/08 2:03 AM, "Steve Loughran" <[EMAIL PROTECTED]> wrote: > I agree. I think right now clients need a bit too much info about the > name node; its URL should be all they need, and presumably who to log in > as.
Re: How do people keep their client configurations in sync with the remote cluster(s)
I would think that this would cover about 80% of the newbie problem reports. It would be especially good if it included ssh'ed commands run on the slaves that look back at the namenode and job tracker. Forward and reverse name lookups are also important. This is worth a Jira. On 5/16/08 2:03 AM, "Steve Loughran" <[EMAIL PROTECTED]> wrote: > For Hadoop, something that prints out the config and looks for good/bad > problems, maybe even does nslookup() on the hosts, checks the ports are > open, etc, would be nice, especially if it provides hints when things > are screwed up.
Re: Making the case for Hadoop
Here at Veoh, we have committed to this style of file system in a very big way. We currently have around a billion files that we manage using replicated file storage. We didn't go with HDFS for this, but the reasons probably do not apply in your case. In our case, we have lots (as in LOTS) of files, many of which are relatively small. The sheer number of files made it better for us to go with an alternative (MogileFS, heavily patched). That choice had issues as well since Mogile was not very well engineered for scaling to true web scale. We felt then, and I think that this is still true, that putting the effort into Mogile to stabilize it was a bit easier than putting the effort into Hadoop to scale it. We also run Hadoop for log processing and are very happy. Our experience with the replicated file store lifestyle has been excellent. Both Mogile and HDFS have been really excellent in terms of reliable file storage and nearly 100% uptime. For your application, I would be pretty confident that HDFS would be a very effective solution and would probably be much better than Mogile. The advantage over Mogile would largely be due to the fact that HDFS breaks files across servers so the quantum of file management would be much smaller for your application. Hiring Hadoop experienced engineers is still difficult, but we have found that it takes very little time for engineers to come up to speed on using Hadoop and HDFS concepts take much less time. With recent versions have file system level interfaces integrated, it should be even easier to manage these systems. We are beginning to see Hadoop experience on resumes, but I would guess that there will be a lag in Europe before you begin to see that. There will also be some bias in our favor because we have a relatively high profile as "cool" place to work. On 5/16/08 1:22 AM, "Robert Krüger" <[EMAIL PROTECTED]> wrote: > > Hi, > > I'm currently trying to make the case for using Hadoop (or more > precisely HDFS) as part of a storage architecture for a large media > asset repository. HDFS will be used for storing up to total of 1 PB of > high-resolution video (average file size will be > 1GB). We think HDFS > is a perfect match for these requirements but have to convince a rather > conservative IT executive who is worried about > > - Hadoop not being mature enough (reference customers or case studies > for similiar projects, e.g. TV stations running their video archives > using HDFS would probably help) > - Hadoop knowledge being not widely available should our customer choose > to change their hosting partner or systems integrator (a list of > consulting/hosting firms having Hadoop expertise would help) > > Thanks in advance for any pointers which help me make the case. > > Robert > > > >
Re: HDFS corrupt...how to proceed?
I second that request... I use DRDB for another project where I work and definitely see it's benefits, but I haven't tried it with hadoop yet. Thanks On Tue, May 13, 2008 at 11:17 AM, Otis Gospodnetic < [EMAIL PROTECTED]> wrote: > Hi, > > I'd love to see the DRBD+Hadoop write up! Not only would this be useful > for Hadoop, I can see this being useful for Solr (master replication). > > > Thanks, > Otis > -- > Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch > > > - Original Message > > From: C G <[EMAIL PROTECTED]> > > To: core-user@hadoop.apache.org > > Sent: Monday, May 12, 2008 2:40:57 PM > > Subject: Re: HDFS corrupt...how to proceed? > > > > Thanks to everyone who responded. Things are back on the air now - all > the > > replication issues seem to have gone away. I am wading through a > detailed fsck > > output now looking for specific problems on a file-by-file basis. > > > > Just in case anybody is interested, we mirror our master nodes using > DRBD. It > > performed very well in this first "real world" test. If there is > interest I can > > write up how we protect our master nodes in more detail and share w/the > > community. > > > > Thanks, > > C G > > > > Ted Dunning wrote: > > > > > > You don't need to correct over-replicated files. > > > > The under-replicated files should cure themselves, but there is a problem > on > > old versions where that doesn't happen quite right. > > > > You can use hadoop fsck / to get a list of the files that are broken and > > there are options to copy what remains of them to lost+found or to delete > > them. > > > > Other than that, things should correct themselves fairly quickly. > > > > > > On 5/11/08 8:23 PM, "C G" > > wrote: > > > > > Hi All: > > > > > > We had a primary node failure over the weekend. When we brought the > node > > > back up and I ran Hadoop fsck, I see the file system is corrupt. I'm > unsure > > > how best to proceed. Any advice is greatly appreciated. If I've missed > a > > > Wiki page or documentation somewhere please feel free to tell me to > RTFM and > > > let me know where to look. > > > > > > Specific question: how to clear under and over replicated files? Is the > > > correct procedure to copy the file locally, delete from HDFS, and then > copy > > > back to HDFS? > > > > > > The fsck output is long, but the final summary is: > > > > > > Total size: 4899680097382 B > > > Total blocks: 994252 (avg. block size 4928006 B) > > > Total dirs: 47404 > > > Total files: 952070 > > > > > > CORRUPT FILES: 2 > > > MISSING BLOCKS: 24 > > > MISSING SIZE: 1501009630 B > > > > > > Over-replicated blocks: 1 (1.0057812E-4 %) > > > Under-replicated blocks: 14958 (1.5044476 %) > > > Target replication factor: 3 > > > Real replication factor: 2.9849212 > > > > > > The filesystem under path '/' is CORRUPT > > > > > > > > > - > > > Be a better friend, newshound, and know-it-all with Yahoo! Mobile. Try > it > > > now. > > > > > > > > > > - > > Be a better friend, newshound, and know-it-all with Yahoo! Mobile. Try > it now. > >
Re: Making the case for Hadoop
Conservative IT executiveSounds like your working at my last job. :) Yahoo uses hadoop. For a very large cluster. http://developer.yahoo.com/blogs/hadoop/ And afterall hadoop is a work alike of the Google File System, google uses that for all types of satelite data, The new york times is using hadoop http://open.blogs.nytimes.com/tag/hadoop/ Raskpsace a big hosting provider users hadoop too http://highscalability.com/how-rackspace-now-uses-mapreduce-and-hadoop-query-terabytes-data So hadoop is a fact. My advice for convincing IT executives. Ask them to present their alternative. (usually its nothing) On Fri, May 16, 2008 at 4:22 AM, Robert Krüger <[EMAIL PROTECTED]> wrote: > > Hi, > > I'm currently trying to make the case for using Hadoop (or more precisely > HDFS) as part of a storage architecture for a large media asset repository. > HDFS will be used for storing up to total of 1 PB of high-resolution video > (average file size will be > 1GB). We think HDFS is a perfect match for > these requirements but have to convince a rather conservative IT executive > who is worried about > > - Hadoop not being mature enough (reference customers or case studies for > similiar projects, e.g. TV stations running their video archives using HDFS > would probably help) > - Hadoop knowledge being not widely available should our customer choose to > change their hosting partner or systems integrator (a list of > consulting/hosting firms having Hadoop expertise would help) > > Thanks in advance for any pointers which help me make the case. > > Robert > > > > >
Mirroring data to a non-Hadoop FS
Hi, what are the options to keep a copy of data from an HDFS instance in sync with a backup file system which is not HDFS? Are there Rsync-like tools that allow only to transfer deltas or would one have to implement that oneself (e.g. by writing a java program that accesses both filesystems)? Thanks in advance, Robert P.S.: Why would one want that? E.g. to have a completely redundant copy which in case of systematic failure (e.g. data corruption due to a bug) offers a backup not affected by that problem.
java.io.IOException: Could not obtain block / java.io.IOException: Could not get block locations
Hi Hadoopers, we are experiencing a lot of "Could not obtain block / Could not get block locations IOExceptions" when processing a 400 GB large Map/Red job using our 6 nodes DFS & MapRed (v. 0.16.4) cluster. Each node is equipped with a 400GB Sata HDD and running Suse Linux Enterprise Edition. While processing this "huge" MapRed job, the name node doesn't seem to receive heartbeats from datanodes for up to a couple of minutes and thus marks those nodes as dead even they are still alive and serving blocks according to their logs. We first suspected network congestion and measured the inter-node bandwidth using scp - receiving throughputs of 30MB/s. CPU utilization is about 100% when processing the job, however, the tasktracker instances shouldn't cause such datanode drop outs? In the datanode logs, we see a lot of java.io.IOException: Block blk_-7943096461180653598 is valid, and cannot be written to. errors... Any ideas? Thanks in advance. Cu on the 'net, Bye - bye, < André èrbnA >
Re: How do people keep their client configurations in sync with the remote cluster(s)
Ted Dunning wrote: I use several strategies: A) avoid dependency on hadoop's configuration by using http access to files. I use this, for example, where we have a PHP or grails or oracle app that needs to read a data file or three from HDFS. B) rsync early and often and lock down the config directory. C) get a really good sysop who does (b) and shoots people who mess up D) (we don't do this yet) establish a configuration repository using zookeeper or a webdav or a (horrors) NFS file system. At the very least, I would like to be able to get namenode address and port. I think in a single organisation, you can get away with SVN management of conf files -if you build everything in. With an HTTP-svn bridge you could aways pull in the file over http during startup. Mostly, our apps are in the cluster and covered by b+c or very out of the cluster and covered by a. Many of our apps are pure import or pure export. The import side really only needs to know where the namenode is and the pure export only really needs the http access. That makes the configuration management task vastly easier. Another serious (as in SERIOUS) problem is how you keep data-processing elements from a QA or staging data chain from inserting bogus data into the production data chain, but still have them work in production with minimal reconfiguration on final deploy. That's a problem on all projects. One team I was on once had a classic disaster where a test cluster bonded to the production MSSQL database (Remember, windows likes flat naming \\database-3 style names) and caused plenty of damage. Still, its good to test your backup strategy works -at least in hindsight. It never seems so good at the time, though. We don't have a particularly good solution for that yet, but are planning on using zookeeper host based permissions to good effect there. That should let us have data mirrors that shadow the production data feed system so that staged systems can process live data, but be unable to insert it back into the production setting. The mirror will have read-only access to the feed meta-data and the staging machines will have no access to the production feed meta-data and these limitations will be imposed by a single configuration on the zookeeper rather than on each machine. This should allow us to keep it cleaner than these things normally wind up. But the short answer is that this is a hard problem to get really, really right. I agree. I think right now clients need a bit too much info about the name node; its URL should be all they need, and presumably who to log in as. In a local cluster you can use discovery services to get a list of machines offering the service too, though its that kind of automatic binding that leads to the erased database problem I've hit before. One thing that might be useful over time would be more client-side diagnostics. if you type ant -diagnostics (or run ant runs through a set of health checks that have caused problems in the past -mixed up JAR versions -tmp dir unwriteable -tmp dir on a filesystem with a different clock/TZ from the local machines -bad proxy settings -wrong XML parser Some you can test, some it just prints. What it prints out is enough for a support email/bugrep. For Hadoop, something that prints out the config and looks for good/bad problems, maybe even does nslookup() on the hosts, checks the ports are open, etc, would be nice, especially if it provides hints when things are screwed up. -Steve -- Steve Loughran http://www.1060.org/blogxter/publish/5 Author: Ant in Action http://antbook.org/
About HDFS`s Certification and Authorization?
hi,all I now use hadoop-0.15.3.Does it`s HDFS have the functionality of certification and authorization? So that one user can just access one part of HDFS,and cann`t access other parts without permitting?If it does ,how can I implement it? Thanks a lot.
Making the case for Hadoop
Hi, I'm currently trying to make the case for using Hadoop (or more precisely HDFS) as part of a storage architecture for a large media asset repository. HDFS will be used for storing up to total of 1 PB of high-resolution video (average file size will be > 1GB). We think HDFS is a perfect match for these requirements but have to convince a rather conservative IT executive who is worried about - Hadoop not being mature enough (reference customers or case studies for similiar projects, e.g. TV stations running their video archives using HDFS would probably help) - Hadoop knowledge being not widely available should our customer choose to change their hosting partner or systems integrator (a list of consulting/hosting firms having Hadoop expertise would help) Thanks in advance for any pointers which help me make the case. Robert
