On Tue, Dec 13, 2011 at 10:42 PM, M. C. Srivas <mcsri...@gmail.com> wrote: > Any simple file meta-data test will cause the NN to spiral to death with > infinite GC. For example, try create many many files. Or even simple > "stat" a bunch of file continuously.
Sure. If I run "dd if=/dev/zero of=foo" my laptop will "spiral to death" also. I think this is what you're referring to -- continuously write files until it is out of RAM. This is a well understood design choice of HDFS. It is not designed as general purpose storage for small files, and if you run tests against it assuming it is, you'll get bad results. I agree there. > > The real FUD going on is refusing to acknowledge that there is indeed a > real problem. Yes, if you use HDFS for workloads for which it was never designed, you'll have a problem. If you stick to commonly accepted best practices I think you'll find the same thing that hundreds of other companies have found: HDFS is stable and reliable and has no such "GC of death" problems when used as intended. -Todd -- Todd Lipcon Software Engineer, Cloudera
