Re: NameNode per-block memory usage?
how much blocks a file has depends on your data status,please consider it according to your data status 2012/1/18 prasenjit mukherjee > Does it mean that on an average 1 file has only 2 blocks ( with > replication=1 ) ? > > > On 1/18/12, M. C. Srivas wrote: > > Konstantin's paper > > http://www.usenix.org/publications/login/2010-04/openpdfs/shvachko.pdf > > > > mentions that on average a file consumes about 600 bytes of memory in the > > name-node (1 file object + 2 block objects). > > > > To quote from his paper (see page 9) > > > > ".. in order to store 100 million files (referencing 200 million blocks) > a > > name-node should have at least 60GB of RAM. This matches observations on > > deployed clusters". > > > > > > > > On Tue, Jan 17, 2012 at 7:08 AM, Otis Gospodnetic < > > otis_gospodne...@yahoo.com> wrote: > > > >> Hello, > >> > >> How much memory/JVM heap does NameNode use for each block? > >> > >> I've tried locating this in the FAQ and on search-hadoop.com, but > >> couldn't find a ton of concrete numbers, just these two: > >> > >> http://search-hadoop.com/m/RmxWMVyVvK1 - 150 bytes/block? > >> http://search-hadoop.com/m/O886P1VyVvK1 - 1 GB heap for every object? > >> > >> Thanks, > >> Otis > >> > > > > -- > Sent from my mobile device >
Re: NameNode per-block memory usage?
Does it mean that on an average 1 file has only 2 blocks ( with replication=1 ) ? On 1/18/12, M. C. Srivas wrote: > Konstantin's paper > http://www.usenix.org/publications/login/2010-04/openpdfs/shvachko.pdf > > mentions that on average a file consumes about 600 bytes of memory in the > name-node (1 file object + 2 block objects). > > To quote from his paper (see page 9) > > ".. in order to store 100 million files (referencing 200 million blocks) a > name-node should have at least 60GB of RAM. This matches observations on > deployed clusters". > > > > On Tue, Jan 17, 2012 at 7:08 AM, Otis Gospodnetic < > otis_gospodne...@yahoo.com> wrote: > >> Hello, >> >> How much memory/JVM heap does NameNode use for each block? >> >> I've tried locating this in the FAQ and on search-hadoop.com, but >> couldn't find a ton of concrete numbers, just these two: >> >> http://search-hadoop.com/m/RmxWMVyVvK1 - 150 bytes/block? >> http://search-hadoop.com/m/O886P1VyVvK1 - 1 GB heap for every object? >> >> Thanks, >> Otis >> > -- Sent from my mobile device
Re: NameNode per-block memory usage?
Does it mean that on an average 1 file has only 2 blocks ( with replication=1 ) ? On 1/18/12, M. C. Srivas wrote: > Konstantin's paper > http://www.usenix.org/publications/login/2010-04/openpdfs/shvachko.pdf > > mentions that on average a file consumes about 600 bytes of memory in the > name-node (1 file object + 2 block objects). > > To quote from his paper (see page 9) > > ".. in order to store 100 million files (referencing 200 million blocks) a > name-node should have at least 60GB of RAM. This matches observations on > deployed clusters". > > > > On Tue, Jan 17, 2012 at 7:08 AM, Otis Gospodnetic < > otis_gospodne...@yahoo.com> wrote: > >> Hello, >> >> How much memory/JVM heap does NameNode use for each block? >> >> I've tried locating this in the FAQ and on search-hadoop.com, but >> couldn't find a ton of concrete numbers, just these two: >> >> http://search-hadoop.com/m/RmxWMVyVvK1 - 150 bytes/block? >> http://search-hadoop.com/m/O886P1VyVvK1 - 1 GB heap for every object? >> >> Thanks, >> Otis >> > -- Sent from my mobile device
Re: NameNode per-block memory usage?
Konstantin's paper http://www.usenix.org/publications/login/2010-04/openpdfs/shvachko.pdf mentions that on average a file consumes about 600 bytes of memory in the name-node (1 file object + 2 block objects). To quote from his paper (see page 9) ".. in order to store 100 million files (referencing 200 million blocks) a name-node should have at least 60GB of RAM. This matches observations on deployed clusters". On Tue, Jan 17, 2012 at 7:08 AM, Otis Gospodnetic < otis_gospodne...@yahoo.com> wrote: > Hello, > > How much memory/JVM heap does NameNode use for each block? > > I've tried locating this in the FAQ and on search-hadoop.com, but > couldn't find a ton of concrete numbers, just these two: > > http://search-hadoop.com/m/RmxWMVyVvK1 - 150 bytes/block? > http://search-hadoop.com/m/O886P1VyVvK1 - 1 GB heap for every object? > > Thanks, > Otis >
Re: NameNode per-block memory usage?
On Tue, Jan 17, 2012 at 10:08 AM, Otis Gospodnetic < otis_gospodne...@yahoo.com> wrote: > Hello, > > How much memory/JVM heap does NameNode use for each block? > > I've tried locating this in the FAQ and on search-hadoop.com, but > couldn't find a ton of concrete numbers, just these two: > > http://search-hadoop.com/m/RmxWMVyVvK1 - 150 bytes/block? > http://search-hadoop.com/m/O886P1VyVvK1 - 1 GB heap for every object? > > Thanks, > Otis > Some real world statistics. From NN web Interface. replication factor=2 Cluster Summary 22,061,605 files and directories, 22,151,870 blocks = 44,213,475 total. Heap Size is 10.85 GB / 16.58 GB (65%) compressedOOps is enabled.
Re: NameNode per-block memory usage?
> How much memory/JVM heap does NameNode use for each block? I don't remember the exact number, it also depends on which version of Hadoop you're using > http://search-hadoop.com/m/O886P1VyVvK1 - 1 GB heap for every object? It's 1 GB for every *million* objects (files, blocks, etc.). This is a good rule of thumb, at least for the 0.20.x/1.0.0 series. Is there a reason you need a more exact estimate? -Joey -- Joseph Echeverria Cloudera, Inc. 443.305.9434
NameNode per-block memory usage?
Hello, How much memory/JVM heap does NameNode use for each block? I've tried locating this in the FAQ and on search-hadoop.com, but couldn't find a ton of concrete numbers, just these two: http://search-hadoop.com/m/RmxWMVyVvK1 - 150 bytes/block? http://search-hadoop.com/m/O886P1VyVvK1 - 1 GB heap for every object? Thanks, Otis
