subject:"Block Size"

Got it. Thankyou Harsh. BTW
It is `hadoop dfs -Ddfs.blocksize=size -put file file`. No dot between
"block" and "size"

On Wed, May 4, 2011 at 3:18 PM, He Chen  wrote:

> Tried second solution. Does not work, still 2 64M blocks. h
>
>
> On Wed, May 4, 2011 at 3:16 PM, He Chen  wrote:
>
>> Hi Harsh
>>
>> Thank you for the reply.
>>
>> Actually, the hadoop directory is on my NFS server, every node reads the
>> same file from NFS server. I think this is not a problem.
>>
>> I like your second solution. But I am not sure, whether the namenode
>> will divide those 128MB
>>
>>  blocks to smaller ones in future or not.
>>
>> Chen
>>
>> On Wed, May 4, 2011 at 3:00 PM, Harsh J  wrote:
>>
>>> Your client (put) machine must have the same block size configuration
>>> during upload as well.
>>>
>>> Alternatively, you may do something explicit like `hadoop dfs
>>> -Ddfs.block.size=size -put file file`
>>>
>>> On Thu, May 5, 2011 at 12:59 AM, He Chen  wrote:
>>> > Hi all
>>> >
>>> > I met a problem about changing block size from 64M to 128M. I am sure I
>>> > modified the correct configuration file hdfs-site.xml. Because I can
>>> change
>>> > the replication number correctly. However, it does not work on block
>>> size
>>> > changing.
>>> >
>>> > For example:
>>> >
>>> > I change the dfs.block.size to 134217728 bytes.
>>> >
>>> > I upload a file which is 128M and use "fsck" to find how many blocks
>>> this
>>> > file has. It shows:
>>> > /user/file1/file 134217726 bytes, 2 blocks(s): OK
>>> > 0. blk_xx len=67108864 repl=2 [192.168.0.3:50010,
>>> 192.168.0.32:50010
>>> > ]
>>> > 1. blk_xx len=67108862 repl=2 [192.168.0.9:50010,
>>> 192.168.0.8:50010]
>>> >
>>> > The hadoop version is 0.21. Any suggestion will be appreciated!
>>> >
>>> > thanks
>>> >
>>> > Chen
>>> >
>>>
>>>
>>>
>>> --
>>> Harsh J
>>>
>>
>>
>

Re: Change block size from 64M to 128M does not work on Hadoop-0.21

Tried second solution. Does not work, still 2 64M blocks. h

On Wed, May 4, 2011 at 3:16 PM, He Chen  wrote:

> Hi Harsh
>
> Thank you for the reply.
>
> Actually, the hadoop directory is on my NFS server, every node reads the
> same file from NFS server. I think this is not a problem.
>
> I like your second solution. But I am not sure, whether the namenode
> will divide those 128MB
>
>  blocks to smaller ones in future or not.
>
> Chen
>
> On Wed, May 4, 2011 at 3:00 PM, Harsh J  wrote:
>
>> Your client (put) machine must have the same block size configuration
>> during upload as well.
>>
>> Alternatively, you may do something explicit like `hadoop dfs
>> -Ddfs.block.size=size -put file file`
>>
>> On Thu, May 5, 2011 at 12:59 AM, He Chen  wrote:
>> > Hi all
>> >
>> > I met a problem about changing block size from 64M to 128M. I am sure I
>> > modified the correct configuration file hdfs-site.xml. Because I can
>> change
>> > the replication number correctly. However, it does not work on block
>> size
>> > changing.
>> >
>> > For example:
>> >
>> > I change the dfs.block.size to 134217728 bytes.
>> >
>> > I upload a file which is 128M and use "fsck" to find how many blocks
>> this
>> > file has. It shows:
>> > /user/file1/file 134217726 bytes, 2 blocks(s): OK
>> > 0. blk_xx len=67108864 repl=2 [192.168.0.3:50010,
>> 192.168.0.32:50010
>> > ]
>> > 1. blk_xx len=67108862 repl=2 [192.168.0.9:50010,
>> 192.168.0.8:50010]
>> >
>> > The hadoop version is 0.21. Any suggestion will be appreciated!
>> >
>> > thanks
>> >
>> > Chen
>> >
>>
>>
>>
>> --
>> Harsh J
>>
>
>

Re: Change block size from 64M to 128M does not work on Hadoop-0.21

Hi Harsh

Thank you for the reply.

Actually, the hadoop directory is on my NFS server, every node reads the
same file from NFS server. I think this is not a problem.

I like your second solution. But I am not sure, whether the namenode
will divide those 128MB

 blocks to smaller ones in future or not.

Chen

On Wed, May 4, 2011 at 3:00 PM, Harsh J  wrote:

> Your client (put) machine must have the same block size configuration
> during upload as well.
>
> Alternatively, you may do something explicit like `hadoop dfs
> -Ddfs.block.size=size -put file file`
>
> On Thu, May 5, 2011 at 12:59 AM, He Chen  wrote:
> > Hi all
> >
> > I met a problem about changing block size from 64M to 128M. I am sure I
> > modified the correct configuration file hdfs-site.xml. Because I can
> change
> > the replication number correctly. However, it does not work on block size
> > changing.
> >
> > For example:
> >
> > I change the dfs.block.size to 134217728 bytes.
> >
> > I upload a file which is 128M and use "fsck" to find how many blocks this
> > file has. It shows:
> > /user/file1/file 134217726 bytes, 2 blocks(s): OK
> > 0. blk_xx len=67108864 repl=2 [192.168.0.3:50010,
> 192.168.0.32:50010
> > ]
> > 1. blk_xx len=67108862 repl=2 [192.168.0.9:50010,
> 192.168.0.8:50010]
> >
> > The hadoop version is 0.21. Any suggestion will be appreciated!
> >
> > thanks
> >
> > Chen
> >
>
>
>
> --
> Harsh J
>

Re: Change block size from 64M to 128M does not work on Hadoop-0.21

2011-05-04 Thread Harsh J

Your client (put) machine must have the same block size configuration
during upload as well.

Alternatively, you may do something explicit like `hadoop dfs
-Ddfs.block.size=size -put file file`

On Thu, May 5, 2011 at 12:59 AM, He Chen  wrote:
> Hi all
>
> I met a problem about changing block size from 64M to 128M. I am sure I
> modified the correct configuration file hdfs-site.xml. Because I can change
> the replication number correctly. However, it does not work on block size
> changing.
>
> For example:
>
> I change the dfs.block.size to 134217728 bytes.
>
> I upload a file which is 128M and use "fsck" to find how many blocks this
> file has. It shows:
> /user/file1/file 134217726 bytes, 2 blocks(s): OK
> 0. blk_xx len=67108864 repl=2 [192.168.0.3:50010, 192.168.0.32:50010
> ]
> 1. blk_xx len=67108862 repl=2 [192.168.0.9:50010, 192.168.0.8:50010]
>
> The hadoop version is 0.21. Any suggestion will be appreciated!
>
> thanks
>
> Chen
>



-- 
Harsh J

Change block size from 64M to 128M does not work on Hadoop-0.21

Hi all

I met a problem about changing block size from 64M to 128M. I am sure I
modified the correct configuration file hdfs-site.xml. Because I can change
the replication number correctly. However, it does not work on block size
changing.

For example:

I change the dfs.block.size to 134217728 bytes.

I upload a file which is 128M and use "fsck" to find how many blocks this
file has. It shows:
/user/file1/file 134217726 bytes, 2 blocks(s): OK
0. blk_xx len=67108864 repl=2 [192.168.0.3:50010, 192.168.0.32:50010
]
1. blk_xx len=67108862 repl=2 [192.168.0.9:50010, 192.168.0.8:50010]

The hadoop version is 0.21. Any suggestion will be appreciated!

thanks

Chen

Re: io.sort.mb based on HDFS block size

2011-04-16 Thread 顾荣

Hi Shrinivas,
 sry for this late reply.

yeah,I can understand what you mean.I also don't  mean the io.sort.mb is
equal to the block size.The point is that the data in buffer are spilled to
HDFS by several times,and each time just spill a little.Before writing to
HDFS ,the spilled data will be combined,if there is a combine
function.So,the data size is really uncertain during the process .From
HDFS's pespective,it can just feel that the data come group by group ,no
idea about the io.sort.mb which is the buffer's total size.

that's why I think setting HDFS block size to config the io.sort.mb is  kind
of meaningless.However this is a very interesting idea.

Regards
Walker Gu

2011/4/15 Shrinivas Joshi 

> Hi Walker,
>
> Thanks for your feedback. I was actually thinking that io.sort.mb could be
> some factor of block size and not equal to block size. This will avoid
> re-tuning of sort buffer sizes and spill threshold values for different
> HDFS
> block sizes. Am I missing something?
>
> Thanks,
> -Shrinivas
>
> On Thu, Apr 14, 2011 at 10:35 AM, 顾荣  wrote:
>
> > Hi Shrinivas,
> >  I also used to think about this.However,the data in buffer are spilled
> > into
> > HDFS when the can reach the threshold,not copy the entire data int to
> > HDFS.And also the data in HDFS may not has the same size as they are in
> > buffer,because
> > if there is a combiner that works they can be shrinked to some degree
> which
> > we are not sure.
> > In one word,the data's finally size are uncertain.so,the this fact to
> > config
> > HDFS block size kind of meaningless.
> >
> > Good Luck
> > Walker Gu.
> >
> >
> > 2011/4/13 Shrinivas Joshi 
> >
> > > Looking at workloads like TeraSort where intermediate map output is
> > > proportional to HDFS block size, I was wondering whether it would be
> > > beneficial to have a mechanism for setting buffer spaces like
> io.sort.mb
> > to
> > > be a certain factor of HDFS block size? I am sure there are other
> config
> > > parameters that could benefit from such expression type values.
> > >
> > > Please let me know your thoughts on this.
> > >
> > > Thanks,
> > > -Shrinivas
> > >
> >
>

Re: io.sort.mb based on HDFS block size

2011-04-14 Thread Shrinivas Joshi

Hi Walker,

Thanks for your feedback. I was actually thinking that io.sort.mb could be
some factor of block size and not equal to block size. This will avoid
re-tuning of sort buffer sizes and spill threshold values for different HDFS
block sizes. Am I missing something?

Thanks,
-Shrinivas

On Thu, Apr 14, 2011 at 10:35 AM, 顾荣  wrote:

> Hi Shrinivas,
>  I also used to think about this.However,the data in buffer are spilled
> into
> HDFS when the can reach the threshold,not copy the entire data int to
> HDFS.And also the data in HDFS may not has the same size as they are in
> buffer,because
> if there is a combiner that works they can be shrinked to some degree which
> we are not sure.
> In one word,the data's finally size are uncertain.so,the this fact to
> config
> HDFS block size kind of meaningless.
>
> Good Luck
> Walker Gu.
>
>
> 2011/4/13 Shrinivas Joshi 
>
> > Looking at workloads like TeraSort where intermediate map output is
> > proportional to HDFS block size, I was wondering whether it would be
> > beneficial to have a mechanism for setting buffer spaces like io.sort.mb
> to
> > be a certain factor of HDFS block size? I am sure there are other config
> > parameters that could benefit from such expression type values.
> >
> > Please let me know your thoughts on this.
> >
> > Thanks,
> > -Shrinivas
> >
>

Re: io.sort.mb based on HDFS block size

2011-04-14 Thread 顾荣

Hi Shrinivas,
 I also used to think about this.However,the data in buffer are spilled into
HDFS when the can reach the threshold,not copy the entire data int to
HDFS.And also the data in HDFS may not has the same size as they are in
buffer,because
if there is a combiner that works they can be shrinked to some degree which
we are not sure.
In one word,the data's finally size are uncertain.so,the this fact to config
HDFS block size kind of meaningless.

Good Luck
Walker Gu.


2011/4/13 Shrinivas Joshi 

> Looking at workloads like TeraSort where intermediate map output is
> proportional to HDFS block size, I was wondering whether it would be
> beneficial to have a mechanism for setting buffer spaces like io.sort.mb to
> be a certain factor of HDFS block size? I am sure there are other config
> parameters that could benefit from such expression type values.
>
> Please let me know your thoughts on this.
>
> Thanks,
> -Shrinivas
>

Does changing the block size of MiniDFSCluster work?

2011-04-12 Thread Jason Rutherglen

I'm using the append 0.20.3 branch and am wondering why the following
fails, where setting the block size either in the Configuration or the
DFSClient.create method causes a failure later on when writing a file
out.

Configuration conf = new Configuration();
long blockSize = (long)32 * 1024 * 1024 * 1024;
conf.setLong("dfs.block.size", blockSize); // doesn't work
MiniDFSCluster cluster = new MiniDFSCluster(conf, 1, true, null);
FileSystem fileSys = cluster.getFileSystem();
fileSys.create(path, false, bufferSize, replication, blockSize); // doesn't work
fileSys.create(path); //works

Output: http://pastebin.com/MrQJcbJr

io.sort.mb based on HDFS block size

2011-04-12 Thread Shrinivas Joshi

Looking at workloads like TeraSort where intermediate map output is
proportional to HDFS block size, I was wondering whether it would be
beneficial to have a mechanism for setting buffer spaces like io.sort.mb to
be a certain factor of HDFS block size? I am sure there are other config
parameters that could benefit from such expression type values.

Please let me know your thoughts on this.

Thanks,
-Shrinivas

Re: Setting a larger block size at runtime in the DFSClient

2011-04-12 Thread Jason Rutherglen

Harsh, thanks, and sounds good!

On Tue, Apr 12, 2011 at 7:08 AM, Harsh J  wrote:
> Hey Jason,
>
> On Tue, Apr 12, 2011 at 7:06 PM, Jason Rutherglen
>  wrote:
>> Are there performance implications to setting the block size to 1 GB
>> or higher (via the DFSClient.create method)?
>
> You'll be streaming 1 complete GB per block to a DN with that value
> (before the next block gets scheduled on another), shouldn't be any
> differences beyond that.
>
> --
> Harsh J
>

Re: Setting a larger block size at runtime in the DFSClient

2011-04-12 Thread Harsh J

Hey Jason,

On Tue, Apr 12, 2011 at 7:06 PM, Jason Rutherglen
 wrote:
> Are there performance implications to setting the block size to 1 GB
> or higher (via the DFSClient.create method)?

You'll be streaming 1 complete GB per block to a DN with that value
(before the next block gets scheduled on another), shouldn't be any
differences beyond that.

-- 
Harsh J

Setting a larger block size at runtime in the DFSClient

2011-04-12 Thread Jason Rutherglen

Are there performance implications to setting the block size to 1 GB
or higher (via the DFSClient.create method)?

Re: Question regardin the block size and the way that a block is used in Hadoop

2011-03-12 Thread James Seigel

Yes.  Just your "FAT" is consumed.

Sent from my mobile. Please excuse the typos.

On 2011-03-12, at 11:04 AM, Florin Picioroaga  wrote:

> Hello!
>  I've been reading in the "Hadoop Definitive guide" by Tom White
> about the block emptiness when a file is not large enough to occupy the full 
> size of the block.
>
> From the statement (cite from the book)
> "Unlike a filesystem for a single disk, a file in HDFS that is smaller than a 
> single block does not occupy a full block’s worth of underlying storage." I 
> can understand the physical space left from the initial block size will be 
> free. My question is can the underlying operating reuse/write this remained 
> free space?
> I'll look forward for your answers.
> Thank you,
>  Florin
>
>
>
>

Question regardin the block size and the way that a block is used in Hadoop

2011-03-12 Thread Florin Picioroaga

Hello!
  I've been reading in the "Hadoop Definitive guide" by Tom White 
about the block emptiness when a file is not large enough to occupy the full 
size of the block.

From the statement (cite from the book)
"Unlike a filesystem for a single disk, a file in HDFS that is smaller than a 
single block does not occupy a full block’s worth of underlying storage." I can 
understand the physical space left from the initial block size will be free. My 
question is can the underlying operating reuse/write this remained free space?
 I'll look forward for your answers.
 Thank you,
  Florin

Re: How I can assume the proper a block size if the input file size is dynamic?

2011-02-22 Thread Jun Young Kim


currenly, I got a problem to reduce the output of mappers.

11/02/23 09:57:45 INFO input.FileInputFormat: Total input paths to 
process : 4157
11/02/23 09:57:47 WARN conf.Configuration: mapred.map.tasks is 
deprecated. Instead, use mapreduce.job.maps

11/02/23 09:57:47 INFO mapreduce.JobSubmitter: number of splits:4309

input file sizes are so dynamic now.
based on this files, a hadoop creates so many splits to map them.

here is my result of M/R.

Kind 	Total Tasks(successful+failed+killed) 	Successful tasks 	Failed 
tasks 	Killed tasks 	Start Time 	Finish Time
Setup 	1 
 
	1 
 
	0 
 
	0 
 
	22-2-2011 22:10:07 	22-2-2011 22:10:08 (1sec)
Map 	4309 
 
	4309 
 
	0 
 
	0 
 
	22-2-2011 22:10:11 	22-2-2011 22:18:51 (8mins, 40sec)
Reduce 	5 
 
	0 
 
	4 
 
	1 
 
	22-2-2011 22:11:00 	22-2-2011 22:36:51 (25mins, 50sec)
Cleanup 	1 
 
	1 
 
	0 
 
	0 
 
	22-2-2011 22:36:47 	22-2-2011 22:37:51 (1mins, 4sec)




in the step of Reduce. there are failed/killed tasks.
the reason of them are this.

org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in 
shuffle in fetcher#3 at 
org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:124) at 
org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:362) at 
org.apache.hadoop.mapred.Child$4.run(Child.java:217) at 
java.security.AccessController.doPrivileged(Native Method) at 
javax.security.auth.Subject.doAs(Subject.java:396) at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:742) 
at org.apache.hadoop.mapred.Child.main(Child.java:211) Caused by: 
java.lang.OutOfMemoryError: Java heap space at 
org.apache.hadoop.io.BoundedByteArrayOutputStream.(BoundedByteArrayOutputStream.java:58) 
at 
org.apache.hadoop.io.BoundedByteArrayOutputStream.(BoundedByteArrayOutputStream.java:45) 
at 
org.apache.hadoop.mapreduce.task.reduce.MapOutput.(MapOutput.java:104) 
at 
org.apache.hadoop.mapreduce.task.reduce.MergeManager.unconditionalReserve(MergeManager.java:267) 
at org.apache.hadoop.mapreduce.task.re


yes.

Re: How I can assume the proper a block size if the input file size is dynamic?

2011-02-22 Thread Tish Heyssel

Yeah,

That's not gonna work.  You need to pre-process your input files to
concatenate them into larger files and then set your dfs.blocksize
accordingly.  Otherwise your jobs will be slow, slow slow.

tish

On Tue, Feb 22, 2011 at 3:57 AM, Jun Young Kim  wrote:

> hi, all.
>
> I know dfs.blocksize key can affect the performance of a hadoop.
>
> in my case, I have thousands of directories which are including so many
> different sized input files.
> (file sizes are from 10K to 1G).
>
> in this case, How I can assume the dfs.blocksize to get a best performance?
>
> 11/02/22 17:45:49 INFO input.FileInputFormat: Total input paths to process
> : *15407*
> 11/02/22 17:45:54 WARN conf.Configuration: mapred.map.tasks is deprecated.
> Instead, use mapreduce.job.maps
> 11/02/22 17:45:54 INFO mapreduce.JobSubmitter: number of splits:*15411*
> 11/02/22 17:45:54 INFO mapreduce.JobSubmitter: adding the following
> namenodes' delegation tokens:null
> 11/02/22 17:45:54 INFO mapreduce.Job: Running job: job_201102221737_0002
> 11/02/22 17:45:55 INFO mapreduce.Job:  map 0% reduce 0%
>
> thanks.
>
> --
> Junyoung Kim (juneng...@gmail.com)
>
>


-- 
Tish Heyssel
Peterson Burnett Technologies
Please use: tish...@gmail.com
Alternate email: t...@speakeasy.net
p...@pbtechnologies.com

How I can assume the proper a block size if the input file size is dynamic?

2011-02-22 Thread Jun Young Kim


hi, all.

I know dfs.blocksize key can affect the performance of a hadoop.

in my case, I have thousands of directories which are including so many 
different sized input files.

(file sizes are from 10K to 1G).

in this case, How I can assume the dfs.blocksize to get a best performance?

11/02/22 17:45:49 INFO input.FileInputFormat: Total input paths to 
process : *15407*
11/02/22 17:45:54 WARN conf.Configuration: mapred.map.tasks is 
deprecated. Instead, use mapreduce.job.maps

11/02/22 17:45:54 INFO mapreduce.JobSubmitter: number of splits:*15411*
11/02/22 17:45:54 INFO mapreduce.JobSubmitter: adding the following 
namenodes' delegation tokens:null

11/02/22 17:45:54 INFO mapreduce.Job: Running job: job_201102221737_0002
11/02/22 17:45:55 INFO mapreduce.Job:  map 0% reduce 0%

thanks.

--
Junyoung Kim (juneng...@gmail.com)

Re: HDFS block size v.s. mapred.min.split.size

2011-02-17 Thread Koji Noguchi

> (mapred.min.split.size can be only set to larger than HDFS block size)
>
I haven't tried this on a new mapreduce API, but

-Dmapred.min.split.size= -Dmapred.map.tasks=1

I think this would let you set a split size smaller than the hdfs block size :)

Koji

On 2/17/11 2:32 PM, "Jim Falgout"  wrote:

Generally, if you have large files, setting the block size to 128M or larger is 
helpful. You can do that on a per file basis or set the block size for the 
whole filesystem. The larger block size cuts down on the number of map tasks 
required to handle the overall data size. I've experimented with 
mapred.min.split.size also and have usually found that the larger the split 
size, the better the overall run time. Of course there is a cut off point, 
especially with a very large cluster where larger split sizes will hurt overall 
scalability.

On tests I've run on a 10 and 20 node cluster though, setting the split size as 
high as 1GB has allows the overall Hadoop jobs to run faster, sometimes quite a 
bit faster. You will lose some locality, but it seems a trade off with the 
number of files that have to be shuffled for the reduction step.

-Original Message-
From: Boduo Li [mailto:birdeey...@gmail.com]
Sent: Thursday, February 17, 2011 12:01 PM
To: common-user@hadoop.apache.org
Subject: HDFS block size v.s. mapred.min.split.size

Hi,

I'm recently benchmarking Hadoop. I know two ways to control the input data 
size for each map task(): by changing the HDFS block size (have to reload data 
into HDFS in this case), or by setting mapred.min.split.size.

For my benchmarking task, I need to change the input size for a map task 
frequently. Changing HDFS block size and reloading data is really painful.
But using mapred.min.split.size seems to be problematic. I did some simple test 
to verify if Hadoop has similar performance in the following cases:

(1) HDFS block size = 32MB, mapred.min.split.size=64MB (mapred.min.split.size 
can be only set to larger than HDFS block size)

(2) HDFS block size = 64MB, mapred.min.split.size is not set

I ran the same job under these settings. Setting (1) takes 1374s to finish.
Setting (2) takes 1412s to finish.

I do understand that, given smaller HDFS block size, the I/O is more random.
But the 50-second difference seems too much for random I/O of input data.
Does anyone have any insight of it? Or does anyone know a better way to control 
the input size of each map task?

Thanks.

RE: HDFS block size v.s. mapred.min.split.size

2011-02-17 Thread Jim Falgout

Generally, if you have large files, setting the block size to 128M or larger is 
helpful. You can do that on a per file basis or set the block size for the 
whole filesystem. The larger block size cuts down on the number of map tasks 
required to handle the overall data size. I've experimented with 
mapred.min.split.size also and have usually found that the larger the split 
size, the better the overall run time. Of course there is a cut off point, 
especially with a very large cluster where larger split sizes will hurt overall 
scalability.

On tests I've run on a 10 and 20 node cluster though, setting the split size as 
high as 1GB has allows the overall Hadoop jobs to run faster, sometimes quite a 
bit faster. You will lose some locality, but it seems a trade off with the 
number of files that have to be shuffled for the reduction step.

-Original Message-
From: Boduo Li [mailto:birdeey...@gmail.com] 
Sent: Thursday, February 17, 2011 12:01 PM
To: common-user@hadoop.apache.org
Subject: HDFS block size v.s. mapred.min.split.size

Hi,

I'm recently benchmarking Hadoop. I know two ways to control the input data 
size for each map task(): by changing the HDFS block size (have to reload data 
into HDFS in this case), or by setting mapred.min.split.size.

For my benchmarking task, I need to change the input size for a map task 
frequently. Changing HDFS block size and reloading data is really painful.
But using mapred.min.split.size seems to be problematic. I did some simple test 
to verify if Hadoop has similar performance in the following cases:

(1) HDFS block size = 32MB, mapred.min.split.size=64MB (mapred.min.split.size 
can be only set to larger than HDFS block size)

(2) HDFS block size = 64MB, mapred.min.split.size is not set

I ran the same job under these settings. Setting (1) takes 1374s to finish.
Setting (2) takes 1412s to finish.

I do understand that, given smaller HDFS block size, the I/O is more random.
But the 50-second difference seems too much for random I/O of input data.
Does anyone have any insight of it? Or does anyone know a better way to control 
the input size of each map task?

Thanks.

HDFS block size v.s. mapred.min.split.size

2011-02-17 Thread Boduo Li

Hi,

I'm recently benchmarking Hadoop. I know two ways to control the input data
size for each map task(): by changing the HDFS block size (have to reload
data into HDFS in this case), or by setting mapred.min.split.size.

For my benchmarking task, I need to change the input size for a map task
frequently. Changing HDFS block size and reloading data is really painful.
But using mapred.min.split.size seems to be problematic. I did some simple
test to verify if Hadoop has similar performance in the following cases:

(1) HDFS block size = 32MB, mapred.min.split.size=64MB
(mapred.min.split.size can be only set to larger than HDFS block size)

(2) HDFS block size = 64MB, mapred.min.split.size is not set

I ran the same job under these settings. Setting (1) takes 1374s to finish.
Setting (2) takes 1412s to finish.

I do understand that, given smaller HDFS block size, the I/O is more random.
But the 50-second difference seems too much for random I/O of input data.
Does anyone have any insight of it? Or does anyone know a better way to
control the input size of each map task?

Thanks.

Re: BUG: Anyone use block size more than 2GB before?

2010-10-21 Thread Milind A Bhandarkar

That's correct. That is why teragen, the program that generates data to be 
sorted in terasort is a MR program :-)

- Milind

On Oct 21, 2010, at 9:47 PM, elton sky wrote:

> Milind,
> 
> You are right. But that only happens when your client is one of the data
> nodes in HDFS. otherwise a random node will be picked up for the first
> replica.
> 
> On Fri, Oct 22, 2010 at 3:37 PM, Milind A Bhandarkar
> wrote:
> 
>> If a file of say, 12.5 GB were produced by a single task with replication
>> 3, the default replication policy will ensure that the first replica of each
>> block will be created on local datanode. So, there will be one datanode in
>> the cluster that contains one replica of all blocks of that file. Map
>> placement hint specifies that node.
>> 
>> It's evil, I know :-)
>> 
>> - Milind
>> 
>> On Oct 21, 2010, at 1:30 PM, Alex Kozlov wrote:
>> 
>>> Hmm, this is interesting: how did it manage to keep the blocks local?
>> Why
>>> performance was better?
>>> 
>>> On Thu, Oct 21, 2010 at 11:43 AM, Owen O'Malley 
>> wrote:
>>> 
 The block sizes were 2G. The input format made splits that were more
>> than a
 block because that led to better performance.
 
 -- Owen
 
>> 
>> --
>> Milind Bhandarkar
>> (mailto:mili...@yahoo-inc.com)
>> (phone: 408-203-5213 W)
>> 
>> 
>> 

--
Milind Bhandarkar
(mailto:mili...@yahoo-inc.com)
(phone: 408-203-5213 W)

Re: BUG: Anyone use block size more than 2GB before?

2010-10-21 Thread elton sky

Milind,

You are right. But that only happens when your client is one of the data
nodes in HDFS. otherwise a random node will be picked up for the first
replica.

On Fri, Oct 22, 2010 at 3:37 PM, Milind A Bhandarkar
wrote:

> If a file of say, 12.5 GB were produced by a single task with replication
> 3, the default replication policy will ensure that the first replica of each
> block will be created on local datanode. So, there will be one datanode in
> the cluster that contains one replica of all blocks of that file. Map
> placement hint specifies that node.
>
> It's evil, I know :-)
>
> - Milind
>
> On Oct 21, 2010, at 1:30 PM, Alex Kozlov wrote:
>
> > Hmm, this is interesting: how did it manage to keep the blocks local?
>  Why
> > performance was better?
> >
> > On Thu, Oct 21, 2010 at 11:43 AM, Owen O'Malley 
> wrote:
> >
> >> The block sizes were 2G. The input format made splits that were more
> than a
> >> block because that led to better performance.
> >>
> >> -- Owen
> >>
>
> --
> Milind Bhandarkar
> (mailto:mili...@yahoo-inc.com)
> (phone: 408-203-5213 W)
>
>
>

Re: BUG: Anyone use block size more than 2GB before?

2010-10-21 Thread Milind A Bhandarkar

If a file of say, 12.5 GB were produced by a single task with replication 3, 
the default replication policy will ensure that the first replica of each block 
will be created on local datanode. So, there will be one datanode in the 
cluster that contains one replica of all blocks of that file. Map placement 
hint specifies that node.

It's evil, I know :-)

- Milind

On Oct 21, 2010, at 1:30 PM, Alex Kozlov wrote:

> Hmm, this is interesting: how did it manage to keep the blocks local?  Why
> performance was better?
> 
> On Thu, Oct 21, 2010 at 11:43 AM, Owen O'Malley  wrote:
> 
>> The block sizes were 2G. The input format made splits that were more than a
>> block because that led to better performance.
>> 
>> -- Owen
>> 

--
Milind Bhandarkar
(mailto:mili...@yahoo-inc.com)
(phone: 408-203-5213 W)

Re: BUG: Anyone use block size more than 2GB before?

2010-10-21 Thread Alex Kozlov

Hmm, this is interesting: how did it manage to keep the blocks local?  Why
performance was better?

On Thu, Oct 21, 2010 at 11:43 AM, Owen O'Malley  wrote:

> The block sizes were 2G. The input format made splits that were more than a
> block because that led to better performance.
>
> -- Owen
>

Re: BUG: Anyone use block size more than 2GB before?

2010-10-21 Thread Owen O'Malley

The block sizes were 2G. The input format made splits that were more than a
block because that led to better performance.

-- Owen

Re: BUG: Anyone use block size more than 2GB before?

2010-10-21 Thread M. C. Srivas

I thought the petasort benchmark you published used 12.5G block sizes. How
did you make that work?

On Mon, Oct 18, 2010 at 4:27 PM, Owen O'Malley  wrote:

> Block sizes larger than 2**31 are known to not work. I haven't ever tracked
> down the problem, just set my block size to be smaller than that.
>
> -- Owen
>

Re: BUG: Anyone use block size more than 2GB before?

2010-10-19 Thread Steve Loughran


On 18/10/10 23:07, Michael Segel wrote:


Ok, I'll bite.
Why would you want to use a block size of>  2GB?



1. Some of the events coming off large physics devices are single 
self-contained files of 3+ GB size; having a block size which has an 
event in a single block guarantees locality for the entirety of the 
operation.


2. Some very fast clusters are finding problems with smaller block sizes 
(256MB) causing TT's to finish too fast and so overload the JT. It's a 
combination of TT-JVM-Reuse, cluster size and improved disk and CPU 
performance.

Re: BUG: Anyone use block size more than 2GB before?

2010-10-18 Thread elton sky

I am curious, any specific reason to make it smaller than 2**31?

On Tue, Oct 19, 2010 at 10:27 AM, Owen O'Malley  wrote:

> Block sizes larger than 2**31 are known to not work. I haven't ever tracked
> down the problem, just set my block size to be smaller than that.
>
> -- Owen
>

Re: BUG: Anyone use block size more than 2GB before?

2010-10-18 Thread Allen Wittenauer


On Oct 18, 2010, at 4:08 PM, elton sky wrote:

>> Why would you want to use a block size of > 2GB?
> For keeping a maps input split in a single block~

Just use mapred.min.split.size + multifileinputformat.

Re: BUG: Anyone use block size more than 2GB before?

2010-10-18 Thread James Seigel

If there is a hard requirement for input split being one block you could just 
make your input split fit a smaller block size. 

Just saying, in case you can't overcome the 2G ceiling

J

Sent from my mobile. Please excuse the typos.

On 2010-10-18, at 5:08 PM, "elton sky"  wrote:

>> Why would you want to use a block size of > 2GB?
> For keeping a maps input split in a single block~
> 
> On Tue, Oct 19, 2010 at 9:07 AM, Michael Segel 
> wrote:
> 
>> 
>> Ok, I'll bite.
>> Why would you want to use a block size of > 2GB?
>> 
>> 
>> 
>>> Date: Mon, 18 Oct 2010 21:33:34 +1100
>>> Subject: BUG: Anyone use block size more than 2GB before?
>>> From: eltonsky9...@gmail.com
>>> To: common-user@hadoop.apache.org
>>> 
>>> Hello,
>>> 
>>> In
>>> 
>> hdfs.org.apache.hadoop.hdfs.DFSClient
>>> 
>> .DFSOutputStream.writeChunk(byte[]
>>> b, int offset, int len, byte[] checksum)
>>> The second last line:
>>> 
>>> int psize = Math.min((int)(blockSize-bytesCurBlock), writePacketSize);
>>> 
>>> When I use blockSize  bigger than 2GB, which is out of the boundary of
>>> integer something weird would happen. For example, for a 3GB block it
>> will
>>> create more than 2Million packets.
>>> 
>>> Anyone noticed this before?
>>> 
>>> Elton
>> 
>>

Re: BUG: Anyone use block size more than 2GB before?

2010-10-18 Thread Owen O'Malley

Block sizes larger than 2**31 are known to not work. I haven't ever  
tracked down the problem, just set my block size to be smaller than  
that.


-- Owen

Re: BUG: Anyone use block size more than 2GB before?

2010-10-18 Thread elton sky

>Why would you want to use a block size of > 2GB?
For keeping a maps input split in a single block~

On Tue, Oct 19, 2010 at 9:07 AM, Michael Segel wrote:

>
> Ok, I'll bite.
> Why would you want to use a block size of > 2GB?
>
>
>
> > Date: Mon, 18 Oct 2010 21:33:34 +1100
> > Subject: BUG: Anyone use block size more than 2GB before?
> > From: eltonsky9...@gmail.com
> > To: common-user@hadoop.apache.org
> >
> > Hello,
> >
> > In
> >
> hdfs.org.apache.hadoop.hdfs.DFSClient
> >
> .DFSOutputStream.writeChunk(byte[]
> > b, int offset, int len, byte[] checksum)
> > The second last line:
> >
> > int psize = Math.min((int)(blockSize-bytesCurBlock), writePacketSize);
> >
> > When I use blockSize  bigger than 2GB, which is out of the boundary of
> > integer something weird would happen. For example, for a 3GB block it
> will
> > create more than 2Million packets.
> >
> > Anyone noticed this before?
> >
> > Elton
>
>

RE: BUG: Anyone use block size more than 2GB before?

2010-10-18 Thread Michael Segel


Ok, I'll bite.
Why would you want to use a block size of > 2GB?



> Date: Mon, 18 Oct 2010 21:33:34 +1100
> Subject: BUG: Anyone use block size more than 2GB before?
> From: eltonsky9...@gmail.com
> To: common-user@hadoop.apache.org
> 
> Hello,
> 
> In
> hdfs.org.apache.hadoop.hdfs.DFSClient
> .DFSOutputStream.writeChunk(byte[]
> b, int offset, int len, byte[] checksum)
> The second last line:
> 
> int psize = Math.min((int)(blockSize-bytesCurBlock), writePacketSize);
> 
> When I use blockSize  bigger than 2GB, which is out of the boundary of
> integer something weird would happen. For example, for a 3GB block it will
> create more than 2Million packets.
> 
> Anyone noticed this before?
> 
> Elton

Re: BUG: Anyone use block size more than 2GB before?

2010-10-18 Thread Allen Wittenauer


On Oct 18, 2010, at 3:33 AM, elton sky wrote:
> 
> 
> When I use blockSize  bigger than 2GB, which is out of the boundary of
> integer something weird would happen. For example, for a 3GB block it will
> create more than 2Million packets.
> 
> Anyone noticed this before?



https://issues.apache.org/jira/browse/HDFS-96

BUG: Anyone use block size more than 2GB before?

2010-10-18 Thread elton sky

Hello,

In
hdfs.org.apache.hadoop.hdfs.DFSClient
.DFSOutputStream.writeChunk(byte[]
b, int offset, int len, byte[] checksum)
The second last line:

int psize = Math.min((int)(blockSize-bytesCurBlock), writePacketSize);

When I use blockSize  bigger than 2GB, which is out of the boundary of
integer something weird would happen. For example, for a 3GB block it will
create more than 2Million packets.

Anyone noticed this before?

Elton

Re: change HDFS block size

2010-09-08 Thread Gang Luo

That makes sense. Thanks Alex and Jeff.

-Gang




- 原始邮件 
发件人： Alex Kozlov 
收件人： common-user@hadoop.apache.org
发送日期： 2010/9/8 (周三) 1:31:14 下午
主   题： Re: change HDFS block size

The block size is a per-file property, so it will change only for the newly
created files.  If you want to change the block size for the 'legacy' files,
you'll need to recreate them, for example with the distcp command (for the
new block size 512M):
*
hadoop distcp -D dfs.block.size=536870912 
*

and then rm the old file.

-- 
Alex Kozlov
Solutions Architect
Cloudera, Inc
twitter: alexvk2009

Hadoop World 2010, October 12, New York City - Register now:
http://www.cloudera.com/company/press-center/hadoop-world-nyc/

On Tue, Sep 7, 2010 at 8:03 PM, Jeff Zhang  wrote:

> Those lagacy files won't change block size (NameNode have the mapping
> between block and file)
> only the new added files will have the block size of 128m
>
>
> On Tue, Sep 7, 2010 at 7:27 PM, Gang Luo  wrote:
> > Hi all,
> > I need to change the block size (from 128m to 64m) and have to shut down
> the
> > cluster first. I was wondering what will happen to the current files on
> HDFS
> > (with 128M block size). Are they still there and usable? If so, what is
> the
> > block size of those lagacy files?
> >
> > Thanks,
> > -Gang
> >
> >
> >
> >
> >
>
>
>
> --
> Best Regards
>
> Jeff Zhang
>

Re: change HDFS block size

2010-09-08 Thread Alex Kozlov

The block size is a per-file property, so it will change only for the newly
created files.  If you want to change the block size for the 'legacy' files,
you'll need to recreate them, for example with the distcp command (for the
new block size 512M):
*
hadoop distcp -D dfs.block.size=536870912 
*

and then rm the old file.

-- 
Alex Kozlov
Solutions Architect
Cloudera, Inc
twitter: alexvk2009

Hadoop World 2010, October 12, New York City - Register now:
http://www.cloudera.com/company/press-center/hadoop-world-nyc/

On Tue, Sep 7, 2010 at 8:03 PM, Jeff Zhang  wrote:

> Those lagacy files won't change block size (NameNode have the mapping
> between block and file)
> only the new added files will have the block size of 128m
>
>
> On Tue, Sep 7, 2010 at 7:27 PM, Gang Luo  wrote:
> > Hi all,
> > I need to change the block size (from 128m to 64m) and have to shut down
> the
> > cluster first. I was wondering what will happen to the current files on
> HDFS
> > (with 128M block size). Are they still there and usable? If so, what is
> the
> > block size of those lagacy files?
> >
> > Thanks,
> > -Gang
> >
> >
> >
> >
> >
>
>
>
> --
> Best Regards
>
> Jeff Zhang
>

Re: change HDFS block size

2010-09-07 Thread Jeff Zhang

Those lagacy files won't change block size (NameNode have the mapping
between block and file)
only the new added files will have the block size of 128m



On Tue, Sep 7, 2010 at 7:27 PM, Gang Luo  wrote:
> Hi all,
> I need to change the block size (from 128m to 64m) and have to shut down the
> cluster first. I was wondering what will happen to the current files on HDFS
> (with 128M block size). Are they still there and usable? If so, what is the
> block size of those lagacy files?
>
> Thanks,
> -Gang
>
>
>
>
>



-- 
Best Regards

Jeff Zhang

change HDFS block size

2010-09-07 Thread Gang Luo

Hi all,
I need to change the block size (from 128m to 64m) and have to shut down the 
cluster first. I was wondering what will happen to the current files on HDFS 
(with 128M block size). Are they still there and usable? If so, what is the 
block size of those lagacy files?

Thanks,
-Gang

Re: Any possible to set hdfs block size to a value smaller than 64MB?

2010-05-20 Thread Nyamul Hassan

Thank you Brian and Konstantin for the two very interesting reads.  I had to
read some lines quite a few times to get the proper understanding of what
they tried to explain.  That interview was a class of its own.

It does seem that Google has heavily used BigTable to overcome some of the
shortcomings of the GFS.  Is that the same as what Konstantin is referring
to HBase over HDFS?

Regards
HASSAN



On Thu, May 20, 2010 at 04:11, Konstantin Shvachko wrote:

> Hi Brian,
>
> Interesting observations.
> This is probably in line with the "client side mount table" approach, [soon
> to be] proposed in HDFS-1053.
> Another way to provide personalized view of the file system would be to use
> symbolic links, which is
> available now in 0.21/22.
> For 4K files I would probably use h-archives, especially if the data, as
> you describe it, is not changing.
> But high-energy physicist should be considering using HBase over HDFS.
>
> --konst
>
>
>
> On 5/18/2010 11:57 AM, Brian Bockelman wrote:
>
>>
>> Hey Konstantin,
>>
>> Interesting paper :)
>>
>> One thing which I've been kicking around lately is "at what scale does the
>> file/directory paradigm break down?"
>>
>> At some point, I think the human mind can no longer comprehend so many
>> files (certainly, I can barely organize the few thousand files on my
>> laptop).  Thus, file growth comes from (a) having lots of humans use a
>> single file system or (b) automated programs generate the files.  For (a),
>> you don't need a central global namespace, you just need the ability to have
>> a "local" namespace per-person that can be shared among friends.  For (b), a
>> program isn't going to be upset if you replace a file system with a database
>> / dataset object / bucket.
>>
>> Two examples:
>> - structural biology: I've seen a lot of different analysis workflows
>> (such as autodock) that compares a protein against a "database" of ligands,
>> where the database is 80,000 O(4KB) files.  Each file represents a known
>> ligand that the biologist might come back and examine if it is relevant to
>> the study of their protein.
>> - high-energy physics: Each detector can produce millions of a events a
>> night, and experiments will produce many billions of events.  These are
>> saved into files (each file containing hundreds or thousands of events);
>> these files are kept in collections (called datasets, data blocks, lumi
>> sections, or runs, depending on what you're doing).  Tasks are run against
>> datasets; they will output smaller datasets which the physicists will
>> iterate upon until they get some dataset which fits onto their laptop.
>> Here's my Claim: The biologists have a small enough number of objects to
>> manage each one as a separate file; they do this because it's easier for
>> humans navigating around in a terminal.  The physicists have such a huge
>> number of objects that there's no way to manage them using one file per
>> object, so they utilize files only as a mechanism to serialize bytes of data
>> and have higher-order data structures for management
>> Here's my Question: at what point do you move from the biologist's model
>> (named objects, managed independently, single files) to the physicist's
>> model (anonymous objects, managed in large groups, files are only used
>> because we save data on file systems)?
>>
>> Another way to look at this is to consider DNS.  DNS maintains the
>> namespace of the globe, but appears to do this just fine without a single
>> central catalog.  If you start with a POSIX filesystem namespace (and the
>> guarantees it implies), what rules must you relax in order to arrive at DNS?
>>  On the scale of managing million (billion? ten billion? trillion?) files,
>> are any of the assumptions relevant?
>>
>> I don't know the answers to these questions, but I suspect they become
>> important over the next 10 years.
>>
>> Brian
>>
>> PS - I starting thinking along these lines during MSST when the LLNL guy
>> was speculating about what it meant to "fsck" a file system with 1 trillion
>> files.
>>
>> On May 18, 2010, at 12:56 PM, Konstantin Shvachko wrote:
>>
>>  You can also get some performance numbers and answers to the block size
>>> dilemma problem here:
>>>
>>>
>>> http://developer.yahoo.net/blogs/hadoop/2010/05/scalability_of_the_hadoop_dist.html
>>>
>>> I remember some people were using Hadoop for storing or streaming videos.
>>> Don&

Re: Any possible to set hdfs block size to a value smaller than 64MB?

2010-05-19 Thread Konstantin Shvachko

Hi Brian,

Interesting observations.
This is probably in line with the "client side mount table" approach, [soon to
be] proposed in HDFS-1053.
Another way to provide personalized view of the file system would be to use
symbolic links, which is
available now in 0.21/22.
For 4K files I would probably use h-archives, especially if the data, as you
describe it, is not changing.
But high-energy physicist should be considering using HBase over HDFS.

--konst

On 5/18/2010 11:57 AM, Brian Bockelman wrote:

Hey Konstantin,

Interesting paper :)

One thing which I've been kicking around lately is "at what scale does the
file/directory paradigm break down?"

At some point, I think the human mind can no longer comprehend so many files (certainly,
I can barely organize the few thousand files on my laptop). Thus, file growth comes from
(a) having lots of humans use a single file system or (b) automated programs generate the
files. For (a), you don't need a central global namespace, you just need the ability to
have a "local" namespace per-person that can be shared among friends. For (b),
a program isn't going to be upset if you replace a file system with a database / dataset
object / bucket.

Two examples:
- structural biology: I've seen a lot of different analysis workflows (such as autodock)
that compares a protein against a "database" of ligands, where the database is
80,000 O(4KB) files. Each file represents a known ligand that the biologist might come
back and examine if it is relevant to the study of their protein.
- high-energy physics: Each detector can produce millions of a events a night,
and experiments will produce many billions of events. These are saved into
files (each file containing hundreds or thousands of events); these files are
kept in collections (called datasets, data blocks, lumi sections, or runs,
depending on what you're doing). Tasks are run against datasets; they will
output smaller datasets which the physicists will iterate upon until they get
some dataset which fits onto their laptop.
Here's my Claim: The biologists have a small enough number of objects to manage
each one as a separate file; they do this because it's easier for humans
navigating around in a terminal. The physicists have such a huge number of
objects that there's no way to manage them using one file per object, so they
utilize files only as a mechanism to serialize bytes of data and have
higher-order data structures for management
Here's my Question: at what point do you move from the biologist's model (named
objects, managed independently, single files) to the physicist's model
(anonymous objects, managed in large groups, files are only used because we
save data on file systems)?

Another way to look at this is to consider DNS. DNS maintains the namespace of
the globe, but appears to do this just fine without a single central catalog.
If you start with a POSIX filesystem namespace (and the guarantees it implies),
what rules must you relax in order to arrive at DNS? On the scale of managing
million (billion? ten billion? trillion?) files, are any of the assumptions
relevant?

I don't know the answers to these questions, but I suspect they become
important over the next 10 years.

Brian

PS - I starting thinking along these lines during MSST when the LLNL guy was speculating
about what it meant to "fsck" a file system with 1 trillion files.

On May 18, 2010, at 12:56 PM, Konstantin Shvachko wrote:

You can also get some performance numbers and answers to the block size dilemma
problem here:

http://developer.yahoo.net/blogs/hadoop/2010/05/scalability_of_the_hadoop_dist.html

I remember some people were using Hadoop for storing or streaming videos.
Don't know how well that worked.
It would be interesting to learn about your experience.

Thanks,
--Konstantin

On 5/18/2010 8:41 AM, Brian Bockelman wrote:

Hey Hassan,

1) The overhead is pretty small, measured in a small number of milliseconds on
average
2) HDFS is not designed for "online latency". Even though the average is small, if
something "bad happens", your clients might experience a lot of delays while going
through the retry stack. The initial design was for batch processing, and latency-sensitive
applications came later.

Additionally since the NN is a SPOF, you might want to consider your uptime
requirements. Each organization will have to balance these risks with the
advantages (such as much cheaper hardware).

There's a nice interview with the GFS authors here where they touch upon the
latency issues:

http://queue.acm.org/detail.cfm?id=1594206

As GFS and HDFS share many design features, the theoretical parts of their
discussion might be useful for you.

As far as overall throughput of the system goes, it depends heavily upon your
implementation and hardware. Our

Re: Any possible to set hdfs block size to a value smaller than 64MB?

2010-05-19 Thread Pierre ANCELOT

Okay, sorry then, I misunderstood.
I think I could aswell run it on empty files, I would only get task startup
overhead.
Thank you.

On Tue, May 18, 2010 at 11:36 PM, Patrick Angeles wrote:

> That wasn't sarcasm. This is what you do:
>
> - Run your mapreduce job on 30k small files.
> - Consolidate your 30k small files into larger files.
> - Run mapreduce ok the larger files.
> - Compare the running time
>
> The difference in runtime is made up by your task startup and seek
> overhead.
>
> If you want to get the 'average' overhead per task, divide the total times
> for each job by the number of map tasks. This won't be a true average
> because with larger chunks of data, you will have longer running map tasks
> that will hold up the shuffle phase. But the average doesn't really matter
> here because you always have that trade-off going from small to large
> chunks
> of data.
>
>
> On Tue, May 18, 2010 at 7:31 PM, Pierre ANCELOT 
> wrote:
>
> > Thanks for the sarcasm but with 3 small files and so, 3 Mapper
> > instatiations, even though it's not (and never did I say it was) he only
> > metric that matters, it seem to me lie something very interresting to
> check
> > out...
> > I have hierarchy over me and they will be happy to understand my choices
> > with real numbers to base their understanding on.
> > Thanks.
> >
> >
> > On Tue, May 18, 2010 at 5:00 PM, Patrick Angeles  > >wrote:
> >
> > > Should be evident in the total job running time... that's the only
> metric
> > > that really matters :)
> > >
> > > On Tue, May 18, 2010 at 10:39 AM, Pierre ANCELOT  > > >wrote:
> > >
> > > > Thank you,
> > > > Any way I can measure the startup overhead in terms of time?
> > > >
> > > >
> > > > On Tue, May 18, 2010 at 4:27 PM, Patrick Angeles <
> patr...@cloudera.com
> > > > >wrote:
> > > >
> > > > > Pierre,
> > > > >
> > > > > Adding to what Brian has said (some things are not explicitly
> > mentioned
> > > > in
> > > > > the HDFS design doc)...
> > > > >
> > > > > - If you have small files that take up < 64MB you do not actually
> use
> > > the
> > > > > entire 64MB block on disk.
> > > > > - You *do* use up RAM on the NameNode, as each block represents
> > > meta-data
> > > > > that needs to be maintained in-memory in the NameNode.
> > > > > - Hadoop won't perform optimally with very small block sizes.
> Hadoop
> > > I/O
> > > > is
> > > > > optimized for high sustained throughput per single file/block.
> There
> > is
> > > a
> > > > > penalty for doing too many seeks to get to the beginning of each
> > block.
> > > > > Additionally, you will have a MapReduce task per small file. Each
> > > > MapReduce
> > > > > task has a non-trivial startup overhead.
> > > > > - The recommendation is to consolidate your small files into large
> > > files.
> > > > > One way to do this is via SequenceFiles... put the filename in the
> > > > > SequenceFile key field, and the file's bytes in the SequenceFile
> > value
> > > > > field.
> > > > >
> > > > > In addition to the HDFS design docs, I recommend reading this blog
> > > post:
> > > > > http://www.cloudera.com/blog/2009/02/the-small-files-problem/
> > > > >
> > > > > Happy Hadooping,
> > > > >
> > > > > - Patrick
> > > > >
> > > > > On Tue, May 18, 2010 at 9:11 AM, Pierre ANCELOT <
> pierre...@gmail.com
> > >
> > > > > wrote:
> > > > >
> > > > > > Okay, thank you :)
> > > > > >
> > > > > >
> > > > > > On Tue, May 18, 2010 at 2:48 PM, Brian Bockelman <
> > > bbock...@cse.unl.edu
> > > > > > >wrote:
> > > > > >
> > > > > > >
> > > > > > > On May 18, 2010, at 7:38 AM, Pierre ANCELOT wrote:
> > > > > > >
> > > > > > > > Hi, thanks for this fast answer :)
> > > > > > > > If so, what do you mean by blocks? If a file has to be
> > splitted,
> > > it
> > > > > > will
> > > > > > > be
> > > > > > > > splitted when larger than 64MB?
> > > > > > > >
> > > > > > >
> > > > > > > For every 64MB of the file, Hadoop will create a separate
> block.
> > >  So,
> > > > > if
> > > > > > > you have a 32KB file, there will be one block of 32KB.  If the
> > file
> > > > is
> > > > > > 65MB,
> > > > > > > then it will have one block of 64MB and another block of 1MB.
> > > > > > >
> > > > > > > Splitting files is very useful for load-balancing and
> > distributing
> > > > I/O
> > > > > > > across multiple nodes.  At 32KB / file, you don't really need
> to
> > > > split
> > > > > > the
> > > > > > > files at all.
> > > > > > >
> > > > > > > I recommend reading the HDFS design document for background
> > issues
> > > > like
> > > > > > > this:
> > > > > > >
> > > > > > > http://hadoop.apache.org/common/docs/r0.20.0/hdfs_design.html
> > > > > > >
> > > > > > > Brian
> > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Tue, May 18, 2010 at 2:34 PM, Brian Bockelman <
> > > > > bbock...@cse.unl.edu
> > > > > > > >wrote:
> > > > > > > >
> > > > > > > >> Hey Pierre,
> > > > > > > >>
> > > > > > > >> These are not traditional filesystem blocks - if you save a
> > file
> > > >

Re: Any possible to set hdfs block size to a value smaller than 64MB?

2010-05-18 Thread Todd Lipcon

On Tue, May 18, 2010 at 2:50 PM, Jones, Nick  wrote:

> I'm not familiar with how to use/create them, but shouldn't a HAR (Hadoop
> Archive) work well in this situation?  I thought it was designed to collect
> several small files together through another level indirection to avoid the
> NN load and decreasing the HDFS block size.
>
>
Yes, or CombineFileInputFormat. JVM reuse also helps somewhat, so long as
you're not talking about hundreds of thousands of files (in which case it
starts to hurt JT load with that many tasks in jobs)

There are a number of ways to combat the issue, but rule of thumb is that
you shouldn't try to use HDFS to store tons of small files :)

-Todd

-Original Message-
> From: patrickange...@gmail.com [mailto:patrickange...@gmail.com] On Behalf
> Of Patrick Angeles
> Sent: Tuesday, May 18, 2010 4:36 PM
> To: common-user@hadoop.apache.org
> Subject: Re: Any possible to set hdfs block size to a value smaller than
> 64MB?
>
> That wasn't sarcasm. This is what you do:
>
> - Run your mapreduce job on 30k small files.
> - Consolidate your 30k small files into larger files.
> - Run mapreduce ok the larger files.
> - Compare the running time
>
> The difference in runtime is made up by your task startup and seek
> overhead.
>
> If you want to get the 'average' overhead per task, divide the total times
> for each job by the number of map tasks. This won't be a true average
> because with larger chunks of data, you will have longer running map tasks
> that will hold up the shuffle phase. But the average doesn't really matter
> here because you always have that trade-off going from small to large
> chunks
> of data.
>
>
> On Tue, May 18, 2010 at 7:31 PM, Pierre ANCELOT 
> wrote:
>
> > Thanks for the sarcasm but with 3 small files and so, 3 Mapper
> > instatiations, even though it's not (and never did I say it was) he only
> > metric that matters, it seem to me lie something very interresting to
> check
> > out...
> > I have hierarchy over me and they will be happy to understand my choices
> > with real numbers to base their understanding on.
> > Thanks.
> >
> >
> > On Tue, May 18, 2010 at 5:00 PM, Patrick Angeles  > >wrote:
> >
> > > Should be evident in the total job running time... that's the only
> metric
> > > that really matters :)
> > >
> > > On Tue, May 18, 2010 at 10:39 AM, Pierre ANCELOT  > > >wrote:
> > >
> > > > Thank you,
> > > > Any way I can measure the startup overhead in terms of time?
> > > >
> > > >
> > > > On Tue, May 18, 2010 at 4:27 PM, Patrick Angeles <
> patr...@cloudera.com
> > > > >wrote:
> > > >
> > > > > Pierre,
> > > > >
> > > > > Adding to what Brian has said (some things are not explicitly
> > mentioned
> > > > in
> > > > > the HDFS design doc)...
> > > > >
> > > > > - If you have small files that take up < 64MB you do not actually
> use
> > > the
> > > > > entire 64MB block on disk.
> > > > > - You *do* use up RAM on the NameNode, as each block represents
> > > meta-data
> > > > > that needs to be maintained in-memory in the NameNode.
> > > > > - Hadoop won't perform optimally with very small block sizes.
> Hadoop
> > > I/O
> > > > is
> > > > > optimized for high sustained throughput per single file/block.
> There
> > is
> > > a
> > > > > penalty for doing too many seeks to get to the beginning of each
> > block.
> > > > > Additionally, you will have a MapReduce task per small file. Each
> > > > MapReduce
> > > > > task has a non-trivial startup overhead.
> > > > > - The recommendation is to consolidate your small files into large
> > > files.
> > > > > One way to do this is via SequenceFiles... put the filename in the
> > > > > SequenceFile key field, and the file's bytes in the SequenceFile
> > value
> > > > > field.
> > > > >
> > > > > In addition to the HDFS design docs, I recommend reading this blog
> > > post:
> > > > > http://www.cloudera.com/blog/2009/02/the-small-files-problem/
> > > > >
> > > > > Happy Hadooping,
> > > > >
> > > > > - Patrick
> > > > >
> > > > > On Tue, May 18, 2010 at 9:11 AM, Pierre ANCELOT <
> pierre...@g

RE: Any possible to set hdfs block size to a value smaller than 64MB?

2010-05-18 Thread Jones, Nick

I'm not familiar with how to use/create them, but shouldn't a HAR (Hadoop 
Archive) work well in this situation?  I thought it was designed to collect 
several small files together through another level indirection to avoid the NN 
load and decreasing the HDFS block size.

Nick Jones
-Original Message-
From: patrickange...@gmail.com [mailto:patrickange...@gmail.com] On Behalf Of 
Patrick Angeles
Sent: Tuesday, May 18, 2010 4:36 PM
To: common-user@hadoop.apache.org
Subject: Re: Any possible to set hdfs block size to a value smaller than 64MB?

That wasn't sarcasm. This is what you do:

- Run your mapreduce job on 30k small files.
- Consolidate your 30k small files into larger files.
- Run mapreduce ok the larger files.
- Compare the running time

The difference in runtime is made up by your task startup and seek overhead.

If you want to get the 'average' overhead per task, divide the total times
for each job by the number of map tasks. This won't be a true average
because with larger chunks of data, you will have longer running map tasks
that will hold up the shuffle phase. But the average doesn't really matter
here because you always have that trade-off going from small to large chunks
of data.


On Tue, May 18, 2010 at 7:31 PM, Pierre ANCELOT  wrote:

> Thanks for the sarcasm but with 3 small files and so, 3 Mapper
> instatiations, even though it's not (and never did I say it was) he only
> metric that matters, it seem to me lie something very interresting to check
> out...
> I have hierarchy over me and they will be happy to understand my choices
> with real numbers to base their understanding on.
> Thanks.
>
>
> On Tue, May 18, 2010 at 5:00 PM, Patrick Angeles  >wrote:
>
> > Should be evident in the total job running time... that's the only metric
> > that really matters :)
> >
> > On Tue, May 18, 2010 at 10:39 AM, Pierre ANCELOT  > >wrote:
> >
> > > Thank you,
> > > Any way I can measure the startup overhead in terms of time?
> > >
> > >
> > > On Tue, May 18, 2010 at 4:27 PM, Patrick Angeles  > > >wrote:
> > >
> > > > Pierre,
> > > >
> > > > Adding to what Brian has said (some things are not explicitly
> mentioned
> > > in
> > > > the HDFS design doc)...
> > > >
> > > > - If you have small files that take up < 64MB you do not actually use
> > the
> > > > entire 64MB block on disk.
> > > > - You *do* use up RAM on the NameNode, as each block represents
> > meta-data
> > > > that needs to be maintained in-memory in the NameNode.
> > > > - Hadoop won't perform optimally with very small block sizes. Hadoop
> > I/O
> > > is
> > > > optimized for high sustained throughput per single file/block. There
> is
> > a
> > > > penalty for doing too many seeks to get to the beginning of each
> block.
> > > > Additionally, you will have a MapReduce task per small file. Each
> > > MapReduce
> > > > task has a non-trivial startup overhead.
> > > > - The recommendation is to consolidate your small files into large
> > files.
> > > > One way to do this is via SequenceFiles... put the filename in the
> > > > SequenceFile key field, and the file's bytes in the SequenceFile
> value
> > > > field.
> > > >
> > > > In addition to the HDFS design docs, I recommend reading this blog
> > post:
> > > > http://www.cloudera.com/blog/2009/02/the-small-files-problem/
> > > >
> > > > Happy Hadooping,
> > > >
> > > > - Patrick
> > > >
> > > > On Tue, May 18, 2010 at 9:11 AM, Pierre ANCELOT  >
> > > > wrote:
> > > >
> > > > > Okay, thank you :)
> > > > >
> > > > >
> > > > > On Tue, May 18, 2010 at 2:48 PM, Brian Bockelman <
> > bbock...@cse.unl.edu
> > > > > >wrote:
> > > > >
> > > > > >
> > > > > > On May 18, 2010, at 7:38 AM, Pierre ANCELOT wrote:
> > > > > >
> > > > > > > Hi, thanks for this fast answer :)
> > > > > > > If so, what do you mean by blocks? If a file has to be
> splitted,
> > it
> > > > > will
> > > > > > be
> > > > > > > splitted when larger than 64MB?
> > > > > > >
> > > > > >
> > > > > > For every 64MB of the file, Hadoop will create a separate block.
> >  So,
> > >

Re: Any possible to set hdfs block size to a value smaller than 64MB?

2010-05-18 Thread Patrick Angeles

That wasn't sarcasm. This is what you do:

- Run your mapreduce job on 30k small files.
- Consolidate your 30k small files into larger files.
- Run mapreduce ok the larger files.
- Compare the running time

The difference in runtime is made up by your task startup and seek overhead.

If you want to get the 'average' overhead per task, divide the total times
for each job by the number of map tasks. This won't be a true average
because with larger chunks of data, you will have longer running map tasks
that will hold up the shuffle phase. But the average doesn't really matter
here because you always have that trade-off going from small to large chunks
of data.


On Tue, May 18, 2010 at 7:31 PM, Pierre ANCELOT  wrote:

> Thanks for the sarcasm but with 3 small files and so, 3 Mapper
> instatiations, even though it's not (and never did I say it was) he only
> metric that matters, it seem to me lie something very interresting to check
> out...
> I have hierarchy over me and they will be happy to understand my choices
> with real numbers to base their understanding on.
> Thanks.
>
>
> On Tue, May 18, 2010 at 5:00 PM, Patrick Angeles  >wrote:
>
> > Should be evident in the total job running time... that's the only metric
> > that really matters :)
> >
> > On Tue, May 18, 2010 at 10:39 AM, Pierre ANCELOT  > >wrote:
> >
> > > Thank you,
> > > Any way I can measure the startup overhead in terms of time?
> > >
> > >
> > > On Tue, May 18, 2010 at 4:27 PM, Patrick Angeles  > > >wrote:
> > >
> > > > Pierre,
> > > >
> > > > Adding to what Brian has said (some things are not explicitly
> mentioned
> > > in
> > > > the HDFS design doc)...
> > > >
> > > > - If you have small files that take up < 64MB you do not actually use
> > the
> > > > entire 64MB block on disk.
> > > > - You *do* use up RAM on the NameNode, as each block represents
> > meta-data
> > > > that needs to be maintained in-memory in the NameNode.
> > > > - Hadoop won't perform optimally with very small block sizes. Hadoop
> > I/O
> > > is
> > > > optimized for high sustained throughput per single file/block. There
> is
> > a
> > > > penalty for doing too many seeks to get to the beginning of each
> block.
> > > > Additionally, you will have a MapReduce task per small file. Each
> > > MapReduce
> > > > task has a non-trivial startup overhead.
> > > > - The recommendation is to consolidate your small files into large
> > files.
> > > > One way to do this is via SequenceFiles... put the filename in the
> > > > SequenceFile key field, and the file's bytes in the SequenceFile
> value
> > > > field.
> > > >
> > > > In addition to the HDFS design docs, I recommend reading this blog
> > post:
> > > > http://www.cloudera.com/blog/2009/02/the-small-files-problem/
> > > >
> > > > Happy Hadooping,
> > > >
> > > > - Patrick
> > > >
> > > > On Tue, May 18, 2010 at 9:11 AM, Pierre ANCELOT  >
> > > > wrote:
> > > >
> > > > > Okay, thank you :)
> > > > >
> > > > >
> > > > > On Tue, May 18, 2010 at 2:48 PM, Brian Bockelman <
> > bbock...@cse.unl.edu
> > > > > >wrote:
> > > > >
> > > > > >
> > > > > > On May 18, 2010, at 7:38 AM, Pierre ANCELOT wrote:
> > > > > >
> > > > > > > Hi, thanks for this fast answer :)
> > > > > > > If so, what do you mean by blocks? If a file has to be
> splitted,
> > it
> > > > > will
> > > > > > be
> > > > > > > splitted when larger than 64MB?
> > > > > > >
> > > > > >
> > > > > > For every 64MB of the file, Hadoop will create a separate block.
> >  So,
> > > > if
> > > > > > you have a 32KB file, there will be one block of 32KB.  If the
> file
> > > is
> > > > > 65MB,
> > > > > > then it will have one block of 64MB and another block of 1MB.
> > > > > >
> > > > > > Splitting files is very useful for load-balancing and
> distributing
> > > I/O
> > > > > > across multiple nodes.  At 32KB / file, you don't really need to
> > > split
> > > > > the
> > > > > > files at all.
> > > > > >
> > > > > > I recommend reading the HDFS design document for background
> issues
> > > like
> > > > > > this:
> > > > > >
> > > > > > http://hadoop.apache.org/common/docs/r0.20.0/hdfs_design.html
> > > > > >
> > > > > > Brian
> > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Tue, May 18, 2010 at 2:34 PM, Brian Bockelman <
> > > > bbock...@cse.unl.edu
> > > > > > >wrote:
> > > > > > >
> > > > > > >> Hey Pierre,
> > > > > > >>
> > > > > > >> These are not traditional filesystem blocks - if you save a
> file
> > > > > smaller
> > > > > > >> than 64MB, you don't lose 64MB of file space..
> > > > > > >>
> > > > > > >> Hadoop will use 32KB to store a 32KB file (ok, plus a KB of
> > > metadata
> > > > > or
> > > > > > >> so), not 64MB.
> > > > > > >>
> > > > > > >> Brian
> > > > > > >>
> > > > > > >> On May 18, 2010, at 7:06 AM, Pierre ANCELOT wrote:
> > > > > > >>
> > > > > > >>> Hi,
> > > > > > >>> I'm porting a legacy application to hadoop and it uses a
> bunch
> > of
> > > > > small
> > > > > > >>> files.
> > > > > > >>> I'm aware that having such

Re: Any possible to set hdfs block size to a value smaller than 64MB?

Thanks for the sarcasm but with 3 small files and so, 3 Mapper
instatiations, even though it's not (and never did I say it was) he only
metric that matters, it seem to me lie something very interresting to check
out...
I have hierarchy over me and they will be happy to understand my choices
with real numbers to base their understanding on.
Thanks.


On Tue, May 18, 2010 at 5:00 PM, Patrick Angeles wrote:

> Should be evident in the total job running time... that's the only metric
> that really matters :)
>
> On Tue, May 18, 2010 at 10:39 AM, Pierre ANCELOT  >wrote:
>
> > Thank you,
> > Any way I can measure the startup overhead in terms of time?
> >
> >
> > On Tue, May 18, 2010 at 4:27 PM, Patrick Angeles  > >wrote:
> >
> > > Pierre,
> > >
> > > Adding to what Brian has said (some things are not explicitly mentioned
> > in
> > > the HDFS design doc)...
> > >
> > > - If you have small files that take up < 64MB you do not actually use
> the
> > > entire 64MB block on disk.
> > > - You *do* use up RAM on the NameNode, as each block represents
> meta-data
> > > that needs to be maintained in-memory in the NameNode.
> > > - Hadoop won't perform optimally with very small block sizes. Hadoop
> I/O
> > is
> > > optimized for high sustained throughput per single file/block. There is
> a
> > > penalty for doing too many seeks to get to the beginning of each block.
> > > Additionally, you will have a MapReduce task per small file. Each
> > MapReduce
> > > task has a non-trivial startup overhead.
> > > - The recommendation is to consolidate your small files into large
> files.
> > > One way to do this is via SequenceFiles... put the filename in the
> > > SequenceFile key field, and the file's bytes in the SequenceFile value
> > > field.
> > >
> > > In addition to the HDFS design docs, I recommend reading this blog
> post:
> > > http://www.cloudera.com/blog/2009/02/the-small-files-problem/
> > >
> > > Happy Hadooping,
> > >
> > > - Patrick
> > >
> > > On Tue, May 18, 2010 at 9:11 AM, Pierre ANCELOT 
> > > wrote:
> > >
> > > > Okay, thank you :)
> > > >
> > > >
> > > > On Tue, May 18, 2010 at 2:48 PM, Brian Bockelman <
> bbock...@cse.unl.edu
> > > > >wrote:
> > > >
> > > > >
> > > > > On May 18, 2010, at 7:38 AM, Pierre ANCELOT wrote:
> > > > >
> > > > > > Hi, thanks for this fast answer :)
> > > > > > If so, what do you mean by blocks? If a file has to be splitted,
> it
> > > > will
> > > > > be
> > > > > > splitted when larger than 64MB?
> > > > > >
> > > > >
> > > > > For every 64MB of the file, Hadoop will create a separate block.
>  So,
> > > if
> > > > > you have a 32KB file, there will be one block of 32KB.  If the file
> > is
> > > > 65MB,
> > > > > then it will have one block of 64MB and another block of 1MB.
> > > > >
> > > > > Splitting files is very useful for load-balancing and distributing
> > I/O
> > > > > across multiple nodes.  At 32KB / file, you don't really need to
> > split
> > > > the
> > > > > files at all.
> > > > >
> > > > > I recommend reading the HDFS design document for background issues
> > like
> > > > > this:
> > > > >
> > > > > http://hadoop.apache.org/common/docs/r0.20.0/hdfs_design.html
> > > > >
> > > > > Brian
> > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Tue, May 18, 2010 at 2:34 PM, Brian Bockelman <
> > > bbock...@cse.unl.edu
> > > > > >wrote:
> > > > > >
> > > > > >> Hey Pierre,
> > > > > >>
> > > > > >> These are not traditional filesystem blocks - if you save a file
> > > > smaller
> > > > > >> than 64MB, you don't lose 64MB of file space..
> > > > > >>
> > > > > >> Hadoop will use 32KB to store a 32KB file (ok, plus a KB of
> > metadata
> > > > or
> > > > > >> so), not 64MB.
> > > > > >>
> > > > > >> Brian
> > > > > >>
> > > > > >> On May 18, 2010, at 7:06 AM, Pierre ANCELOT wrote:
> > > > > >>
> > > > > >>> Hi,
> > > > > >>> I'm porting a legacy application to hadoop and it uses a bunch
> of
> > > > small
> > > > > >>> files.
> > > > > >>> I'm aware that having such small files ain't a good idea but
> I'm
> > > not
> > > > > >> doing
> > > > > >>> the technical decisions and the port has to be done for
> > > yesterday...
> > > > > >>> Of course such small files are a problem, loading 64MB blocks
> for
> > a
> > > > few
> > > > > >>> lines of text is an evident loss.
> > > > > >>> What will happen if I set a smaller, or even way smaller (32kB)
> > > > blocks?
> > > > > >>>
> > > > > >>> Thank you.
> > > > > >>>
> > > > > >>> Pierre ANCELOT.
> > > > > >>
> > > > > >>
> > > > > >
> > > > > >
> > > > > > --
> > > > > > http://www.neko-consulting.com
> > > > > > Ego sum quis ego servo
> > > > > > "Je suis ce que je protège"
> > > > > > "I am what I protect"
> > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > > http://www.neko-consulting.com
> > > > Ego sum quis ego servo
> > > > "Je suis ce que je protège"
> > > > "I am what I protect"
> > > >
> > >
> >
> >
> >
> > --
> > http://www.neko-consulting.com
> > Ego sum quis ego servo
> > "Je suis ce que je protèg

Re: Any possible to set hdfs block size to a value smaller than 64MB?


Hey Konstantin,

Interesting paper :)

One thing which I've been kicking around lately is "at what scale does the 
file/directory paradigm break down?"

At some point, I think the human mind can no longer comprehend so many files 
(certainly, I can barely organize the few thousand files on my laptop).  Thus, 
file growth comes from (a) having lots of humans use a single file system or 
(b) automated programs generate the files.  For (a), you don't need a central 
global namespace, you just need the ability to have a "local" namespace 
per-person that can be shared among friends.  For (b), a program isn't going to 
be upset if you replace a file system with a database / dataset object / bucket.

Two examples:
- structural biology: I've seen a lot of different analysis workflows (such as 
autodock) that compares a protein against a "database" of ligands, where the 
database is 80,000 O(4KB) files.  Each file represents a known ligand that the 
biologist might come back and examine if it is relevant to the study of their 
protein.
- high-energy physics: Each detector can produce millions of a events a night, 
and experiments will produce many billions of events.  These are saved into 
files (each file containing hundreds or thousands of events); these files are 
kept in collections (called datasets, data blocks, lumi sections, or runs, 
depending on what you're doing).  Tasks are run against datasets; they will 
output smaller datasets which the physicists will iterate upon until they get 
some dataset which fits onto their laptop.
Here's my Claim: The biologists have a small enough number of objects to manage 
each one as a separate file; they do this because it's easier for humans 
navigating around in a terminal.  The physicists have such a huge number of 
objects that there's no way to manage them using one file per object, so they 
utilize files only as a mechanism to serialize bytes of data and have 
higher-order data structures for management
Here's my Question: at what point do you move from the biologist's model (named 
objects, managed independently, single files) to the physicist's model 
(anonymous objects, managed in large groups, files are only used because we 
save data on file systems)?

Another way to look at this is to consider DNS.  DNS maintains the namespace of 
the globe, but appears to do this just fine without a single central catalog.  
If you start with a POSIX filesystem namespace (and the guarantees it implies), 
what rules must you relax in order to arrive at DNS?  On the scale of managing 
million (billion? ten billion? trillion?) files, are any of the assumptions 
relevant?

I don't know the answers to these questions, but I suspect they become 
important over the next 10 years.

Brian

PS - I starting thinking along these lines during MSST when the LLNL guy was 
speculating about what it meant to "fsck" a file system with 1 trillion files.

On May 18, 2010, at 12:56 PM, Konstantin Shvachko wrote:

> You can also get some performance numbers and answers to the block size 
> dilemma problem here:
> 
> http://developer.yahoo.net/blogs/hadoop/2010/05/scalability_of_the_hadoop_dist.html
> 
> I remember some people were using Hadoop for storing or streaming videos.
> Don't know how well that worked.
> It would be interesting to learn about your experience.
> 
> Thanks,
> --Konstantin
> 
> 
> On 5/18/2010 8:41 AM, Brian Bockelman wrote:
>> Hey Hassan,
>> 
>> 1) The overhead is pretty small, measured in a small number of milliseconds 
>> on average
>> 2) HDFS is not designed for "online latency".  Even though the average is 
>> small, if something "bad happens", your clients might experience a lot of 
>> delays while going through the retry stack.  The initial design was for 
>> batch processing, and latency-sensitive applications came later.
>> 
>> Additionally since the NN is a SPOF, you might want to consider your uptime 
>> requirements.  Each organization will have to balance these risks with the 
>> advantages (such as much cheaper hardware).
>> 
>> There's a nice interview with the GFS authors here where they touch upon the 
>> latency issues:
>> 
>> http://queue.acm.org/detail.cfm?id=1594206
>> 
>> As GFS and HDFS share many design features, the theoretical parts of their 
>> discussion might be useful for you.
>> 
>> As far as overall throughput of the system goes, it depends heavily upon 
>> your implementation and hardware.  Our HDFS routinely serves 5-10 Gbps.
>> 
>> Brian
>> 
>> On May 18, 2010, at 10:29 AM, Nyamul Hassan wrote:
>> 
>>> This is a very interesting thread to us, as we are thinking about deploying
>>> H

Re: Any possible to set hdfs block size to a value smaller than 64MB?

2010-05-18 Thread Konstantin Boudnik

I had an experiment with block size of 10 bytes (sic!). This was _very_ slow
on NN side. Like writing 5 Mb was happening for 25 minutes or so :( No fun to
say the least...

On Tue, May 18, 2010 at 10:56AM, Konstantin Shvachko wrote:
> You can also get some performance numbers and answers to the block size 
> dilemma problem here:
> 
> http://developer.yahoo.net/blogs/hadoop/2010/05/scalability_of_the_hadoop_dist.html
> 
> I remember some people were using Hadoop for storing or streaming videos.
> Don't know how well that worked.
> It would be interesting to learn about your experience.
> 
> Thanks,
> --Konstantin
> 
> 
> On 5/18/2010 8:41 AM, Brian Bockelman wrote:
> > Hey Hassan,
> >
> > 1) The overhead is pretty small, measured in a small number of milliseconds 
> > on average
> > 2) HDFS is not designed for "online latency".  Even though the average is 
> > small, if something "bad happens", your clients might experience a lot of 
> > delays while going through the retry stack.  The initial design was for 
> > batch processing, and latency-sensitive applications came later.
> >
> > Additionally since the NN is a SPOF, you might want to consider your uptime 
> > requirements.  Each organization will have to balance these risks with the 
> > advantages (such as much cheaper hardware).
> >
> > There's a nice interview with the GFS authors here where they touch upon 
> > the latency issues:
> >
> > http://queue.acm.org/detail.cfm?id=1594206
> >
> > As GFS and HDFS share many design features, the theoretical parts of their 
> > discussion might be useful for you.
> >
> > As far as overall throughput of the system goes, it depends heavily upon 
> > your implementation and hardware.  Our HDFS routinely serves 5-10 Gbps.
> >
> > Brian
> >
> > On May 18, 2010, at 10:29 AM, Nyamul Hassan wrote:
> >
> >> This is a very interesting thread to us, as we are thinking about deploying
> >> HDFS as a massive online storage for a on online university, and then
> >> serving the video files to students who want to view them.
> >>
> >> We cannot control the size of the videos (and some class work files), as
> >> they will mostly be uploaded by the teachers providing the classes.
> >>
> >> How would the overall through put of HDFS be affected in such a solution?
> >> Would HDFS be feasible at all for such a setup?
> >>
> >> Regards
> >> HASSAN
> >>
> >>
> >>
> >> On Tue, May 18, 2010 at 21:11, He Chen  wrote:
> >>
> >>> If you know how to use AspectJ to do aspect oriented programming. You can
> >>> write a aspect class. Let it just monitors the whole process of MapReduce
> >>>
> >>> On Tue, May 18, 2010 at 10:00 AM, Patrick Angeles >>>> wrote:
> >>>
> >>>> Should be evident in the total job running time... that's the only metric
> >>>> that really matters :)
> >>>>
> >>>> On Tue, May 18, 2010 at 10:39 AM, Pierre ANCELOT >>>>> wrote:
> >>>>
> >>>>> Thank you,
> >>>>> Any way I can measure the startup overhead in terms of time?
> >>>>>
> >>>>>
> >>>>> On Tue, May 18, 2010 at 4:27 PM, Patrick Angeles >>>>>> wrote:
> >>>>>
> >>>>>> Pierre,
> >>>>>>
> >>>>>> Adding to what Brian has said (some things are not explicitly
> >>> mentioned
> >>>>> in
> >>>>>> the HDFS design doc)...
> >>>>>>
> >>>>>> - If you have small files that take up<  64MB you do not actually use
> >>>> the
> >>>>>> entire 64MB block on disk.
> >>>>>> - You *do* use up RAM on the NameNode, as each block represents
> >>>> meta-data
> >>>>>> that needs to be maintained in-memory in the NameNode.
> >>>>>> - Hadoop won't perform optimally with very small block sizes. Hadoop
> >>>> I/O
> >>>>> is
> >>>>>> optimized for high sustained throughput per single file/block. There
> >>> is
> >>>> a
> >>>>>> penalty for doing too many seeks to get to the beginning of each
> >>> block.
> >>>>>> Additionally, you will have a MapReduce task per small file. Each
> >>>>>

Re: Any possible to set hdfs block size to a value smaller than 64MB?

2010-05-18 Thread Konstantin Shvachko

You can also get some performance numbers and answers to the block size dilemma
problem here:

http://developer.yahoo.net/blogs/hadoop/2010/05/scalability_of_the_hadoop_dist.html

I remember some people were using Hadoop for storing or streaming videos.
Don't know how well that worked.
It would be interesting to learn about your experience.

Thanks,
--Konstantin

On 5/18/2010 8:41 AM, Brian Bockelman wrote:

Hey Hassan,

Additionally since the NN is a SPOF, you might want to consider your uptime
requirements. Each organization will have to balance these risks with the
advantages (such as much cheaper hardware).

There's a nice interview with the GFS authors here where they touch upon the
latency issues:

http://queue.acm.org/detail.cfm?id=1594206

As GFS and HDFS share many design features, the theoretical parts of their
discussion might be useful for you.

As far as overall throughput of the system goes, it depends heavily upon your
implementation and hardware. Our HDFS routinely serves 5-10 Gbps.

Brian

On May 18, 2010, at 10:29 AM, Nyamul Hassan wrote:

This is a very interesting thread to us, as we are thinking about deploying
HDFS as a massive online storage for a on online university, and then
serving the video files to students who want to view them.

We cannot control the size of the videos (and some class work files), as
they will mostly be uploaded by the teachers providing the classes.

How would the overall through put of HDFS be affected in such a solution?
Would HDFS be feasible at all for such a setup?

Regards
HASSAN

On Tue, May 18, 2010 at 21:11, He Chen wrote:

If you know how to use AspectJ to do aspect oriented programming. You can
write a aspect class. Let it just monitors the whole process of MapReduce

On Tue, May 18, 2010 at 10:00 AM, Patrick Angeles
wrote:

Should be evident in the total job running time... that's the only metric
that really matters :)

On Tue, May 18, 2010 at 10:39 AM, Pierre ANCELOT
wrote:

Thank you,
Any way I can measure the startup overhead in terms of time?

On Tue, May 18, 2010 at 4:27 PM, Patrick Angeles
wrote:

Pierre,

Adding to what Brian has said (some things are not explicitly

mentioned

the HDFS design doc)...

- If you have small files that take up< 64MB you do not actually use

the

entire 64MB block on disk.
- You *do* use up RAM on the NameNode, as each block represents

meta-data

that needs to be maintained in-memory in the NameNode.
- Hadoop won't perform optimally with very small block sizes. Hadoop

I/O

optimized for high sustained throughput per single file/block. There

penalty for doing too many seeks to get to the beginning of each

block.

Additionally, you will have a MapReduce task per small file. Each

MapReduce

task has a non-trivial startup overhead.
- The recommendation is to consolidate your small files into large

files.

One way to do this is via SequenceFiles... put the filename in the
SequenceFile key field, and the file's bytes in the SequenceFile

value

field.

In addition to the HDFS design docs, I recommend reading this blog

post:

http://www.cloudera.com/blog/2009/02/the-small-files-problem/

Happy Hadooping,

- Patrick

On Tue, May 18, 2010 at 9:11 AM, Pierre ANCELOT

wrote:

Okay, thank you :)

On Tue, May 18, 2010 at 2:48 PM, Brian Bockelman<

bbock...@cse.unl.edu

wrote:

On May 18, 2010, at 7:38 AM, Pierre ANCELOT wrote:

Hi, thanks for this fast answer :)
If so, what do you mean by blocks? If a file has to be

splitted,

will

splitted when larger than 64MB?

For every 64MB of the file, Hadoop will create a separate block.

So,

you have a 32KB file, there will be one block of 32KB. If the

file

65MB,

then it will have one block of 64MB and another block of 1MB.

Splitting files is very useful for load-balancing and

distributing

I/O

across multiple nodes. At 32KB / file, you don't really need to

split

the

files at all.

I recommend reading the HDFS design document for background

issues

this:

http://hadoop.apache.org/common/docs/r0.20.0/hdfs_design.html

Brian

On Tue, May 18, 2010 at 2:34 PM, Brian Bockelman<

bbock...@cse.unl.edu

wrote:

Hey Pierre,

These are not traditional filesystem blocks - if you save a

file

smaller

than 64MB, you don't lose 64MB of file space..

Hadoop will use 32KB to store a 32KB file (ok, plus a KB of

metadata

so), not 64MB.

Brian

On May 18, 2010, at 7:06 AM, Pierre ANCELOT wrote:

Hi,
I'm porting a legacy application to hadoop and it uses a

Re: Any possible to set hdfs block size to a value smaller than 64MB?

Hey Hassan,

1) The overhead is pretty small, measured in a small number of milliseconds on 
average
2) HDFS is not designed for "online latency".  Even though the average is 
small, if something "bad happens", your clients might experience a lot of 
delays while going through the retry stack.  The initial design was for batch 
processing, and latency-sensitive applications came later.

Additionally since the NN is a SPOF, you might want to consider your uptime 
requirements.  Each organization will have to balance these risks with the 
advantages (such as much cheaper hardware).

There's a nice interview with the GFS authors here where they touch upon the 
latency issues:

http://queue.acm.org/detail.cfm?id=1594206

As GFS and HDFS share many design features, the theoretical parts of their 
discussion might be useful for you.

As far as overall throughput of the system goes, it depends heavily upon your 
implementation and hardware.  Our HDFS routinely serves 5-10 Gbps.

Brian

On May 18, 2010, at 10:29 AM, Nyamul Hassan wrote:

> This is a very interesting thread to us, as we are thinking about deploying
> HDFS as a massive online storage for a on online university, and then
> serving the video files to students who want to view them.
> 
> We cannot control the size of the videos (and some class work files), as
> they will mostly be uploaded by the teachers providing the classes.
> 
> How would the overall through put of HDFS be affected in such a solution?
> Would HDFS be feasible at all for such a setup?
> 
> Regards
> HASSAN
> 
> 
> 
> On Tue, May 18, 2010 at 21:11, He Chen  wrote:
> 
>> If you know how to use AspectJ to do aspect oriented programming. You can
>> write a aspect class. Let it just monitors the whole process of MapReduce
>> 
>> On Tue, May 18, 2010 at 10:00 AM, Patrick Angeles >> wrote:
>> 
>>> Should be evident in the total job running time... that's the only metric
>>> that really matters :)
>>> 
>>> On Tue, May 18, 2010 at 10:39 AM, Pierre ANCELOT >>> wrote:
>>> 
 Thank you,
 Any way I can measure the startup overhead in terms of time?
 
 
 On Tue, May 18, 2010 at 4:27 PM, Patrick Angeles  wrote:
 
> Pierre,
> 
> Adding to what Brian has said (some things are not explicitly
>> mentioned
 in
> the HDFS design doc)...
> 
> - If you have small files that take up < 64MB you do not actually use
>>> the
> entire 64MB block on disk.
> - You *do* use up RAM on the NameNode, as each block represents
>>> meta-data
> that needs to be maintained in-memory in the NameNode.
> - Hadoop won't perform optimally with very small block sizes. Hadoop
>>> I/O
 is
> optimized for high sustained throughput per single file/block. There
>> is
>>> a
> penalty for doing too many seeks to get to the beginning of each
>> block.
> Additionally, you will have a MapReduce task per small file. Each
 MapReduce
> task has a non-trivial startup overhead.
> - The recommendation is to consolidate your small files into large
>>> files.
> One way to do this is via SequenceFiles... put the filename in the
> SequenceFile key field, and the file's bytes in the SequenceFile
>> value
> field.
> 
> In addition to the HDFS design docs, I recommend reading this blog
>>> post:
> http://www.cloudera.com/blog/2009/02/the-small-files-problem/
> 
> Happy Hadooping,
> 
> - Patrick
> 
> On Tue, May 18, 2010 at 9:11 AM, Pierre ANCELOT >> 
> wrote:
> 
>> Okay, thank you :)
>> 
>> 
>> On Tue, May 18, 2010 at 2:48 PM, Brian Bockelman <
>>> bbock...@cse.unl.edu
>>> wrote:
>> 
>>> 
>>> On May 18, 2010, at 7:38 AM, Pierre ANCELOT wrote:
>>> 
 Hi, thanks for this fast answer :)
 If so, what do you mean by blocks? If a file has to be
>> splitted,
>>> it
>> will
>>> be
 splitted when larger than 64MB?
 
>>> 
>>> For every 64MB of the file, Hadoop will create a separate block.
>>> So,
> if
>>> you have a 32KB file, there will be one block of 32KB.  If the
>> file
 is
>> 65MB,
>>> then it will have one block of 64MB and another block of 1MB.
>>> 
>>> Splitting files is very useful for load-balancing and
>> distributing
 I/O
>>> across multiple nodes.  At 32KB / file, you don't really need to
 split
>> the
>>> files at all.
>>> 
>>> I recommend reading the HDFS design document for background
>> issues
 like
>>> this:
>>> 
>>> http://hadoop.apache.org/common/docs/r0.20.0/hdfs_design.html
>>> 
>>> Brian
>>> 
 
 
 
 On Tue, May 18, 2010 at 2:34 PM, Brian Bockelman <
> bbock...@cse.unl.edu
 wrote:
 
> Hey Pierre,
> 
> These are not traditional filesystem blocks - if you save a
>> file
>> smaller
> than 64MB, you don't lose 64MB of file space..
>>>

Re: Any possible to set hdfs block size to a value smaller than 64MB?

2010-05-18 Thread Nyamul Hassan

This is a very interesting thread to us, as we are thinking about deploying
HDFS as a massive online storage for a on online university, and then
serving the video files to students who want to view them.

We cannot control the size of the videos (and some class work files), as
they will mostly be uploaded by the teachers providing the classes.

How would the overall through put of HDFS be affected in such a solution?
 Would HDFS be feasible at all for such a setup?

Regards
HASSAN



On Tue, May 18, 2010 at 21:11, He Chen  wrote:

> If you know how to use AspectJ to do aspect oriented programming. You can
> write a aspect class. Let it just monitors the whole process of MapReduce
>
> On Tue, May 18, 2010 at 10:00 AM, Patrick Angeles  >wrote:
>
> > Should be evident in the total job running time... that's the only metric
> > that really matters :)
> >
> > On Tue, May 18, 2010 at 10:39 AM, Pierre ANCELOT  > >wrote:
> >
> > > Thank you,
> > > Any way I can measure the startup overhead in terms of time?
> > >
> > >
> > > On Tue, May 18, 2010 at 4:27 PM, Patrick Angeles  > > >wrote:
> > >
> > > > Pierre,
> > > >
> > > > Adding to what Brian has said (some things are not explicitly
> mentioned
> > > in
> > > > the HDFS design doc)...
> > > >
> > > > - If you have small files that take up < 64MB you do not actually use
> > the
> > > > entire 64MB block on disk.
> > > > - You *do* use up RAM on the NameNode, as each block represents
> > meta-data
> > > > that needs to be maintained in-memory in the NameNode.
> > > > - Hadoop won't perform optimally with very small block sizes. Hadoop
> > I/O
> > > is
> > > > optimized for high sustained throughput per single file/block. There
> is
> > a
> > > > penalty for doing too many seeks to get to the beginning of each
> block.
> > > > Additionally, you will have a MapReduce task per small file. Each
> > > MapReduce
> > > > task has a non-trivial startup overhead.
> > > > - The recommendation is to consolidate your small files into large
> > files.
> > > > One way to do this is via SequenceFiles... put the filename in the
> > > > SequenceFile key field, and the file's bytes in the SequenceFile
> value
> > > > field.
> > > >
> > > > In addition to the HDFS design docs, I recommend reading this blog
> > post:
> > > > http://www.cloudera.com/blog/2009/02/the-small-files-problem/
> > > >
> > > > Happy Hadooping,
> > > >
> > > > - Patrick
> > > >
> > > > On Tue, May 18, 2010 at 9:11 AM, Pierre ANCELOT  >
> > > > wrote:
> > > >
> > > > > Okay, thank you :)
> > > > >
> > > > >
> > > > > On Tue, May 18, 2010 at 2:48 PM, Brian Bockelman <
> > bbock...@cse.unl.edu
> > > > > >wrote:
> > > > >
> > > > > >
> > > > > > On May 18, 2010, at 7:38 AM, Pierre ANCELOT wrote:
> > > > > >
> > > > > > > Hi, thanks for this fast answer :)
> > > > > > > If so, what do you mean by blocks? If a file has to be
> splitted,
> > it
> > > > > will
> > > > > > be
> > > > > > > splitted when larger than 64MB?
> > > > > > >
> > > > > >
> > > > > > For every 64MB of the file, Hadoop will create a separate block.
> >  So,
> > > > if
> > > > > > you have a 32KB file, there will be one block of 32KB.  If the
> file
> > > is
> > > > > 65MB,
> > > > > > then it will have one block of 64MB and another block of 1MB.
> > > > > >
> > > > > > Splitting files is very useful for load-balancing and
> distributing
> > > I/O
> > > > > > across multiple nodes.  At 32KB / file, you don't really need to
> > > split
> > > > > the
> > > > > > files at all.
> > > > > >
> > > > > > I recommend reading the HDFS design document for background
> issues
> > > like
> > > > > > this:
> > > > > >
> > > > > > http://hadoop.apache.org/common/docs/r0.20.0/hdfs_design.html
> > > > > >
> > > > > > Brian
> > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Tue, May 18, 2010 at 2:34 PM, Brian Bockelman <
> > > > bbock...@cse.unl.edu
> > > > > > >wrote:
> > > > > > >
> > > > > > >> Hey Pierre,
> > > > > > >>
> > > > > > >> These are not traditional filesystem blocks - if you save a
> file
> > > > > smaller
> > > > > > >> than 64MB, you don't lose 64MB of file space..
> > > > > > >>
> > > > > > >> Hadoop will use 32KB to store a 32KB file (ok, plus a KB of
> > > metadata
> > > > > or
> > > > > > >> so), not 64MB.
> > > > > > >>
> > > > > > >> Brian
> > > > > > >>
> > > > > > >> On May 18, 2010, at 7:06 AM, Pierre ANCELOT wrote:
> > > > > > >>
> > > > > > >>> Hi,
> > > > > > >>> I'm porting a legacy application to hadoop and it uses a
> bunch
> > of
> > > > > small
> > > > > > >>> files.
> > > > > > >>> I'm aware that having such small files ain't a good idea but
> > I'm
> > > > not
> > > > > > >> doing
> > > > > > >>> the technical decisions and the port has to be done for
> > > > yesterday...
> > > > > > >>> Of course such small files are a problem, loading 64MB blocks
> > for
> > > a
> > > > > few
> > > > > > >>> lines of text is an evident loss.
> > > > > > >>> What will happen if I set a smaller, or even way smaller
> (32kB)

Re: Any possible to set hdfs block size to a value smaller than 64MB?

2010-05-18 Thread He Chen

If you know how to use AspectJ to do aspect oriented programming. You can
write a aspect class. Let it just monitors the whole process of MapReduce

On Tue, May 18, 2010 at 10:00 AM, Patrick Angeles wrote:

> Should be evident in the total job running time... that's the only metric
> that really matters :)
>
> On Tue, May 18, 2010 at 10:39 AM, Pierre ANCELOT  >wrote:
>
> > Thank you,
> > Any way I can measure the startup overhead in terms of time?
> >
> >
> > On Tue, May 18, 2010 at 4:27 PM, Patrick Angeles  > >wrote:
> >
> > > Pierre,
> > >
> > > Adding to what Brian has said (some things are not explicitly mentioned
> > in
> > > the HDFS design doc)...
> > >
> > > - If you have small files that take up < 64MB you do not actually use
> the
> > > entire 64MB block on disk.
> > > - You *do* use up RAM on the NameNode, as each block represents
> meta-data
> > > that needs to be maintained in-memory in the NameNode.
> > > - Hadoop won't perform optimally with very small block sizes. Hadoop
> I/O
> > is
> > > optimized for high sustained throughput per single file/block. There is
> a
> > > penalty for doing too many seeks to get to the beginning of each block.
> > > Additionally, you will have a MapReduce task per small file. Each
> > MapReduce
> > > task has a non-trivial startup overhead.
> > > - The recommendation is to consolidate your small files into large
> files.
> > > One way to do this is via SequenceFiles... put the filename in the
> > > SequenceFile key field, and the file's bytes in the SequenceFile value
> > > field.
> > >
> > > In addition to the HDFS design docs, I recommend reading this blog
> post:
> > > http://www.cloudera.com/blog/2009/02/the-small-files-problem/
> > >
> > > Happy Hadooping,
> > >
> > > - Patrick
> > >
> > > On Tue, May 18, 2010 at 9:11 AM, Pierre ANCELOT 
> > > wrote:
> > >
> > > > Okay, thank you :)
> > > >
> > > >
> > > > On Tue, May 18, 2010 at 2:48 PM, Brian Bockelman <
> bbock...@cse.unl.edu
> > > > >wrote:
> > > >
> > > > >
> > > > > On May 18, 2010, at 7:38 AM, Pierre ANCELOT wrote:
> > > > >
> > > > > > Hi, thanks for this fast answer :)
> > > > > > If so, what do you mean by blocks? If a file has to be splitted,
> it
> > > > will
> > > > > be
> > > > > > splitted when larger than 64MB?
> > > > > >
> > > > >
> > > > > For every 64MB of the file, Hadoop will create a separate block.
>  So,
> > > if
> > > > > you have a 32KB file, there will be one block of 32KB.  If the file
> > is
> > > > 65MB,
> > > > > then it will have one block of 64MB and another block of 1MB.
> > > > >
> > > > > Splitting files is very useful for load-balancing and distributing
> > I/O
> > > > > across multiple nodes.  At 32KB / file, you don't really need to
> > split
> > > > the
> > > > > files at all.
> > > > >
> > > > > I recommend reading the HDFS design document for background issues
> > like
> > > > > this:
> > > > >
> > > > > http://hadoop.apache.org/common/docs/r0.20.0/hdfs_design.html
> > > > >
> > > > > Brian
> > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Tue, May 18, 2010 at 2:34 PM, Brian Bockelman <
> > > bbock...@cse.unl.edu
> > > > > >wrote:
> > > > > >
> > > > > >> Hey Pierre,
> > > > > >>
> > > > > >> These are not traditional filesystem blocks - if you save a file
> > > > smaller
> > > > > >> than 64MB, you don't lose 64MB of file space..
> > > > > >>
> > > > > >> Hadoop will use 32KB to store a 32KB file (ok, plus a KB of
> > metadata
> > > > or
> > > > > >> so), not 64MB.
> > > > > >>
> > > > > >> Brian
> > > > > >>
> > > > > >> On May 18, 2010, at 7:06 AM, Pierre ANCELOT wrote:
> > > > > >>
> > > > > >>> Hi,
> > > > > >>> I'm porting a legacy application to hadoop and it uses a bunch
> of
> > > > small
> > > > > >>> files.
> > > > > >>> I'm aware that having such small files ain't a good idea but
> I'm
> > > not
> > > > > >> doing
> > > > > >>> the technical decisions and the port has to be done for
> > > yesterday...
> > > > > >>> Of course such small files are a problem, loading 64MB blocks
> for
> > a
> > > > few
> > > > > >>> lines of text is an evident loss.
> > > > > >>> What will happen if I set a smaller, or even way smaller (32kB)
> > > > blocks?
> > > > > >>>
> > > > > >>> Thank you.
> > > > > >>>
> > > > > >>> Pierre ANCELOT.
> > > > > >>
> > > > > >>
> > > > > >
> > > > > >
> > > > > > --
> > > > > > http://www.neko-consulting.com
> > > > > > Ego sum quis ego servo
> > > > > > "Je suis ce que je protège"
> > > > > > "I am what I protect"
> > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > > http://www.neko-consulting.com
> > > > Ego sum quis ego servo
> > > > "Je suis ce que je protège"
> > > > "I am what I protect"
> > > >
> > >
> >
> >
> >
> > --
> > http://www.neko-consulting.com
> > Ego sum quis ego servo
> > "Je suis ce que je protège"
> > "I am what I protect"
> >
>



-- 
Best Wishes！
顺送商祺！

－－
Chen He
(402)613-9298
PhD. student of CSE Dept.
Holland Computing Center
University of Nebraska-Lincoln
Lincoln NE 68588

Re: Any possible to set hdfs block size to a value smaller than 64MB?

2010-05-18 Thread Patrick Angeles

Should be evident in the total job running time... that's the only metric
that really matters :)

On Tue, May 18, 2010 at 10:39 AM, Pierre ANCELOT wrote:

> Thank you,
> Any way I can measure the startup overhead in terms of time?
>
>
> On Tue, May 18, 2010 at 4:27 PM, Patrick Angeles  >wrote:
>
> > Pierre,
> >
> > Adding to what Brian has said (some things are not explicitly mentioned
> in
> > the HDFS design doc)...
> >
> > - If you have small files that take up < 64MB you do not actually use the
> > entire 64MB block on disk.
> > - You *do* use up RAM on the NameNode, as each block represents meta-data
> > that needs to be maintained in-memory in the NameNode.
> > - Hadoop won't perform optimally with very small block sizes. Hadoop I/O
> is
> > optimized for high sustained throughput per single file/block. There is a
> > penalty for doing too many seeks to get to the beginning of each block.
> > Additionally, you will have a MapReduce task per small file. Each
> MapReduce
> > task has a non-trivial startup overhead.
> > - The recommendation is to consolidate your small files into large files.
> > One way to do this is via SequenceFiles... put the filename in the
> > SequenceFile key field, and the file's bytes in the SequenceFile value
> > field.
> >
> > In addition to the HDFS design docs, I recommend reading this blog post:
> > http://www.cloudera.com/blog/2009/02/the-small-files-problem/
> >
> > Happy Hadooping,
> >
> > - Patrick
> >
> > On Tue, May 18, 2010 at 9:11 AM, Pierre ANCELOT 
> > wrote:
> >
> > > Okay, thank you :)
> > >
> > >
> > > On Tue, May 18, 2010 at 2:48 PM, Brian Bockelman  > > >wrote:
> > >
> > > >
> > > > On May 18, 2010, at 7:38 AM, Pierre ANCELOT wrote:
> > > >
> > > > > Hi, thanks for this fast answer :)
> > > > > If so, what do you mean by blocks? If a file has to be splitted, it
> > > will
> > > > be
> > > > > splitted when larger than 64MB?
> > > > >
> > > >
> > > > For every 64MB of the file, Hadoop will create a separate block.  So,
> > if
> > > > you have a 32KB file, there will be one block of 32KB.  If the file
> is
> > > 65MB,
> > > > then it will have one block of 64MB and another block of 1MB.
> > > >
> > > > Splitting files is very useful for load-balancing and distributing
> I/O
> > > > across multiple nodes.  At 32KB / file, you don't really need to
> split
> > > the
> > > > files at all.
> > > >
> > > > I recommend reading the HDFS design document for background issues
> like
> > > > this:
> > > >
> > > > http://hadoop.apache.org/common/docs/r0.20.0/hdfs_design.html
> > > >
> > > > Brian
> > > >
> > > > >
> > > > >
> > > > >
> > > > > On Tue, May 18, 2010 at 2:34 PM, Brian Bockelman <
> > bbock...@cse.unl.edu
> > > > >wrote:
> > > > >
> > > > >> Hey Pierre,
> > > > >>
> > > > >> These are not traditional filesystem blocks - if you save a file
> > > smaller
> > > > >> than 64MB, you don't lose 64MB of file space..
> > > > >>
> > > > >> Hadoop will use 32KB to store a 32KB file (ok, plus a KB of
> metadata
> > > or
> > > > >> so), not 64MB.
> > > > >>
> > > > >> Brian
> > > > >>
> > > > >> On May 18, 2010, at 7:06 AM, Pierre ANCELOT wrote:
> > > > >>
> > > > >>> Hi,
> > > > >>> I'm porting a legacy application to hadoop and it uses a bunch of
> > > small
> > > > >>> files.
> > > > >>> I'm aware that having such small files ain't a good idea but I'm
> > not
> > > > >> doing
> > > > >>> the technical decisions and the port has to be done for
> > yesterday...
> > > > >>> Of course such small files are a problem, loading 64MB blocks for
> a
> > > few
> > > > >>> lines of text is an evident loss.
> > > > >>> What will happen if I set a smaller, or even way smaller (32kB)
> > > blocks?
> > > > >>>
> > > > >>> Thank you.
> > > > >>>
> > > > >>> Pierre ANCELOT.
> > > > >>
> > > > >>
> > > > >
> > > > >
> > > > > --
> > > > > http://www.neko-consulting.com
> > > > > Ego sum quis ego servo
> > > > > "Je suis ce que je protège"
> > > > > "I am what I protect"
> > > >
> > > >
> > >
> > >
> > > --
> > > http://www.neko-consulting.com
> > > Ego sum quis ego servo
> > > "Je suis ce que je protège"
> > > "I am what I protect"
> > >
> >
>
>
>
> --
> http://www.neko-consulting.com
> Ego sum quis ego servo
> "Je suis ce que je protège"
> "I am what I protect"
>

Re: Any possible to set hdfs block size to a value smaller than 64MB?

Thank you,
Any way I can measure the startup overhead in terms of time?


On Tue, May 18, 2010 at 4:27 PM, Patrick Angeles wrote:

> Pierre,
>
> Adding to what Brian has said (some things are not explicitly mentioned in
> the HDFS design doc)...
>
> - If you have small files that take up < 64MB you do not actually use the
> entire 64MB block on disk.
> - You *do* use up RAM on the NameNode, as each block represents meta-data
> that needs to be maintained in-memory in the NameNode.
> - Hadoop won't perform optimally with very small block sizes. Hadoop I/O is
> optimized for high sustained throughput per single file/block. There is a
> penalty for doing too many seeks to get to the beginning of each block.
> Additionally, you will have a MapReduce task per small file. Each MapReduce
> task has a non-trivial startup overhead.
> - The recommendation is to consolidate your small files into large files.
> One way to do this is via SequenceFiles... put the filename in the
> SequenceFile key field, and the file's bytes in the SequenceFile value
> field.
>
> In addition to the HDFS design docs, I recommend reading this blog post:
> http://www.cloudera.com/blog/2009/02/the-small-files-problem/
>
> Happy Hadooping,
>
> - Patrick
>
> On Tue, May 18, 2010 at 9:11 AM, Pierre ANCELOT 
> wrote:
>
> > Okay, thank you :)
> >
> >
> > On Tue, May 18, 2010 at 2:48 PM, Brian Bockelman  > >wrote:
> >
> > >
> > > On May 18, 2010, at 7:38 AM, Pierre ANCELOT wrote:
> > >
> > > > Hi, thanks for this fast answer :)
> > > > If so, what do you mean by blocks? If a file has to be splitted, it
> > will
> > > be
> > > > splitted when larger than 64MB?
> > > >
> > >
> > > For every 64MB of the file, Hadoop will create a separate block.  So,
> if
> > > you have a 32KB file, there will be one block of 32KB.  If the file is
> > 65MB,
> > > then it will have one block of 64MB and another block of 1MB.
> > >
> > > Splitting files is very useful for load-balancing and distributing I/O
> > > across multiple nodes.  At 32KB / file, you don't really need to split
> > the
> > > files at all.
> > >
> > > I recommend reading the HDFS design document for background issues like
> > > this:
> > >
> > > http://hadoop.apache.org/common/docs/r0.20.0/hdfs_design.html
> > >
> > > Brian
> > >
> > > >
> > > >
> > > >
> > > > On Tue, May 18, 2010 at 2:34 PM, Brian Bockelman <
> bbock...@cse.unl.edu
> > > >wrote:
> > > >
> > > >> Hey Pierre,
> > > >>
> > > >> These are not traditional filesystem blocks - if you save a file
> > smaller
> > > >> than 64MB, you don't lose 64MB of file space..
> > > >>
> > > >> Hadoop will use 32KB to store a 32KB file (ok, plus a KB of metadata
> > or
> > > >> so), not 64MB.
> > > >>
> > > >> Brian
> > > >>
> > > >> On May 18, 2010, at 7:06 AM, Pierre ANCELOT wrote:
> > > >>
> > > >>> Hi,
> > > >>> I'm porting a legacy application to hadoop and it uses a bunch of
> > small
> > > >>> files.
> > > >>> I'm aware that having such small files ain't a good idea but I'm
> not
> > > >> doing
> > > >>> the technical decisions and the port has to be done for
> yesterday...
> > > >>> Of course such small files are a problem, loading 64MB blocks for a
> > few
> > > >>> lines of text is an evident loss.
> > > >>> What will happen if I set a smaller, or even way smaller (32kB)
> > blocks?
> > > >>>
> > > >>> Thank you.
> > > >>>
> > > >>> Pierre ANCELOT.
> > > >>
> > > >>
> > > >
> > > >
> > > > --
> > > > http://www.neko-consulting.com
> > > > Ego sum quis ego servo
> > > > "Je suis ce que je protège"
> > > > "I am what I protect"
> > >
> > >
> >
> >
> > --
> > http://www.neko-consulting.com
> > Ego sum quis ego servo
> > "Je suis ce que je protège"
> > "I am what I protect"
> >
>



-- 
http://www.neko-consulting.com
Ego sum quis ego servo
"Je suis ce que je protège"
"I am what I protect"

Re: Any possible to set hdfs block size to a value smaller than 64MB?

2010-05-18 Thread Patrick Angeles

Pierre,

Adding to what Brian has said (some things are not explicitly mentioned in
the HDFS design doc)...

- If you have small files that take up < 64MB you do not actually use the
entire 64MB block on disk.
- You *do* use up RAM on the NameNode, as each block represents meta-data
that needs to be maintained in-memory in the NameNode.
- Hadoop won't perform optimally with very small block sizes. Hadoop I/O is
optimized for high sustained throughput per single file/block. There is a
penalty for doing too many seeks to get to the beginning of each block.
Additionally, you will have a MapReduce task per small file. Each MapReduce
task has a non-trivial startup overhead.
- The recommendation is to consolidate your small files into large files.
One way to do this is via SequenceFiles... put the filename in the
SequenceFile key field, and the file's bytes in the SequenceFile value
field.

In addition to the HDFS design docs, I recommend reading this blog post:
http://www.cloudera.com/blog/2009/02/the-small-files-problem/

Happy Hadooping,

- Patrick

On Tue, May 18, 2010 at 9:11 AM, Pierre ANCELOT  wrote:

> Okay, thank you :)
>
>
> On Tue, May 18, 2010 at 2:48 PM, Brian Bockelman  >wrote:
>
> >
> > On May 18, 2010, at 7:38 AM, Pierre ANCELOT wrote:
> >
> > > Hi, thanks for this fast answer :)
> > > If so, what do you mean by blocks? If a file has to be splitted, it
> will
> > be
> > > splitted when larger than 64MB?
> > >
> >
> > For every 64MB of the file, Hadoop will create a separate block.  So, if
> > you have a 32KB file, there will be one block of 32KB.  If the file is
> 65MB,
> > then it will have one block of 64MB and another block of 1MB.
> >
> > Splitting files is very useful for load-balancing and distributing I/O
> > across multiple nodes.  At 32KB / file, you don't really need to split
> the
> > files at all.
> >
> > I recommend reading the HDFS design document for background issues like
> > this:
> >
> > http://hadoop.apache.org/common/docs/r0.20.0/hdfs_design.html
> >
> > Brian
> >
> > >
> > >
> > >
> > > On Tue, May 18, 2010 at 2:34 PM, Brian Bockelman  > >wrote:
> > >
> > >> Hey Pierre,
> > >>
> > >> These are not traditional filesystem blocks - if you save a file
> smaller
> > >> than 64MB, you don't lose 64MB of file space..
> > >>
> > >> Hadoop will use 32KB to store a 32KB file (ok, plus a KB of metadata
> or
> > >> so), not 64MB.
> > >>
> > >> Brian
> > >>
> > >> On May 18, 2010, at 7:06 AM, Pierre ANCELOT wrote:
> > >>
> > >>> Hi,
> > >>> I'm porting a legacy application to hadoop and it uses a bunch of
> small
> > >>> files.
> > >>> I'm aware that having such small files ain't a good idea but I'm not
> > >> doing
> > >>> the technical decisions and the port has to be done for yesterday...
> > >>> Of course such small files are a problem, loading 64MB blocks for a
> few
> > >>> lines of text is an evident loss.
> > >>> What will happen if I set a smaller, or even way smaller (32kB)
> blocks?
> > >>>
> > >>> Thank you.
> > >>>
> > >>> Pierre ANCELOT.
> > >>
> > >>
> > >
> > >
> > > --
> > > http://www.neko-consulting.com
> > > Ego sum quis ego servo
> > > "Je suis ce que je protège"
> > > "I am what I protect"
> >
> >
>
>
> --
> http://www.neko-consulting.com
> Ego sum quis ego servo
> "Je suis ce que je protège"
> "I am what I protect"
>

Re: Any possible to set hdfs block size to a value smaller than 64MB?

Okay, thank you :)


On Tue, May 18, 2010 at 2:48 PM, Brian Bockelman wrote:

>
> On May 18, 2010, at 7:38 AM, Pierre ANCELOT wrote:
>
> > Hi, thanks for this fast answer :)
> > If so, what do you mean by blocks? If a file has to be splitted, it will
> be
> > splitted when larger than 64MB?
> >
>
> For every 64MB of the file, Hadoop will create a separate block.  So, if
> you have a 32KB file, there will be one block of 32KB.  If the file is 65MB,
> then it will have one block of 64MB and another block of 1MB.
>
> Splitting files is very useful for load-balancing and distributing I/O
> across multiple nodes.  At 32KB / file, you don't really need to split the
> files at all.
>
> I recommend reading the HDFS design document for background issues like
> this:
>
> http://hadoop.apache.org/common/docs/r0.20.0/hdfs_design.html
>
> Brian
>
> >
> >
> >
> > On Tue, May 18, 2010 at 2:34 PM, Brian Bockelman  >wrote:
> >
> >> Hey Pierre,
> >>
> >> These are not traditional filesystem blocks - if you save a file smaller
> >> than 64MB, you don't lose 64MB of file space..
> >>
> >> Hadoop will use 32KB to store a 32KB file (ok, plus a KB of metadata or
> >> so), not 64MB.
> >>
> >> Brian
> >>
> >> On May 18, 2010, at 7:06 AM, Pierre ANCELOT wrote:
> >>
> >>> Hi,
> >>> I'm porting a legacy application to hadoop and it uses a bunch of small
> >>> files.
> >>> I'm aware that having such small files ain't a good idea but I'm not
> >> doing
> >>> the technical decisions and the port has to be done for yesterday...
> >>> Of course such small files are a problem, loading 64MB blocks for a few
> >>> lines of text is an evident loss.
> >>> What will happen if I set a smaller, or even way smaller (32kB) blocks?
> >>>
> >>> Thank you.
> >>>
> >>> Pierre ANCELOT.
> >>
> >>
> >
> >
> > --
> > http://www.neko-consulting.com
> > Ego sum quis ego servo
> > "Je suis ce que je protège"
> > "I am what I protect"
>
>


-- 
http://www.neko-consulting.com
Ego sum quis ego servo
"Je suis ce que je protège"
"I am what I protect"

Re: Any possible to set hdfs block size to a value smaller than 64MB?


On May 18, 2010, at 7:38 AM, Pierre ANCELOT wrote:

> Hi, thanks for this fast answer :)
> If so, what do you mean by blocks? If a file has to be splitted, it will be
> splitted when larger than 64MB?
> 

For every 64MB of the file, Hadoop will create a separate block.  So, if you 
have a 32KB file, there will be one block of 32KB.  If the file is 65MB, then 
it will have one block of 64MB and another block of 1MB.

Splitting files is very useful for load-balancing and distributing I/O across 
multiple nodes.  At 32KB / file, you don't really need to split the files at 
all.

I recommend reading the HDFS design document for background issues like this:

http://hadoop.apache.org/common/docs/r0.20.0/hdfs_design.html

Brian

> 
> 
> 
> On Tue, May 18, 2010 at 2:34 PM, Brian Bockelman wrote:
> 
>> Hey Pierre,
>> 
>> These are not traditional filesystem blocks - if you save a file smaller
>> than 64MB, you don't lose 64MB of file space..
>> 
>> Hadoop will use 32KB to store a 32KB file (ok, plus a KB of metadata or
>> so), not 64MB.
>> 
>> Brian
>> 
>> On May 18, 2010, at 7:06 AM, Pierre ANCELOT wrote:
>> 
>>> Hi,
>>> I'm porting a legacy application to hadoop and it uses a bunch of small
>>> files.
>>> I'm aware that having such small files ain't a good idea but I'm not
>> doing
>>> the technical decisions and the port has to be done for yesterday...
>>> Of course such small files are a problem, loading 64MB blocks for a few
>>> lines of text is an evident loss.
>>> What will happen if I set a smaller, or even way smaller (32kB) blocks?
>>> 
>>> Thank you.
>>> 
>>> Pierre ANCELOT.
>> 
>> 
> 
> 
> -- 
> http://www.neko-consulting.com
> Ego sum quis ego servo
> "Je suis ce que je protège"
> "I am what I protect"



smime.p7s
Description: S/MIME cryptographic signature

Re: Any possible to set hdfs block size to a value smaller than 64MB?

... and by slices of 64MB then I mean...
?

On Tue, May 18, 2010 at 2:38 PM, Pierre ANCELOT  wrote:

> Hi, thanks for this fast answer :)
> If so, what do you mean by blocks? If a file has to be splitted, it will be
> splitted when larger than 64MB?
>
>
>
>
>
> On Tue, May 18, 2010 at 2:34 PM, Brian Bockelman wrote:
>
>> Hey Pierre,
>>
>> These are not traditional filesystem blocks - if you save a file smaller
>> than 64MB, you don't lose 64MB of file space..
>>
>> Hadoop will use 32KB to store a 32KB file (ok, plus a KB of metadata or
>> so), not 64MB.
>>
>> Brian
>>
>> On May 18, 2010, at 7:06 AM, Pierre ANCELOT wrote:
>>
>> > Hi,
>> > I'm porting a legacy application to hadoop and it uses a bunch of small
>> > files.
>> > I'm aware that having such small files ain't a good idea but I'm not
>> doing
>> > the technical decisions and the port has to be done for yesterday...
>> > Of course such small files are a problem, loading 64MB blocks for a few
>> > lines of text is an evident loss.
>> > What will happen if I set a smaller, or even way smaller (32kB) blocks?
>> >
>> > Thank you.
>> >
>> > Pierre ANCELOT.
>>
>>
>
>
> --
> http://www.neko-consulting.com
> Ego sum quis ego servo
> "Je suis ce que je protège"
> "I am what I protect"
>
>


-- 
http://www.neko-consulting.com
Ego sum quis ego servo
"Je suis ce que je protège"
"I am what I protect"

Re: Any possible to set hdfs block size to a value smaller than 64MB?

Hi, thanks for this fast answer :)
If so, what do you mean by blocks? If a file has to be splitted, it will be
splitted when larger than 64MB?




On Tue, May 18, 2010 at 2:34 PM, Brian Bockelman wrote:

> Hey Pierre,
>
> These are not traditional filesystem blocks - if you save a file smaller
> than 64MB, you don't lose 64MB of file space..
>
> Hadoop will use 32KB to store a 32KB file (ok, plus a KB of metadata or
> so), not 64MB.
>
> Brian
>
> On May 18, 2010, at 7:06 AM, Pierre ANCELOT wrote:
>
> > Hi,
> > I'm porting a legacy application to hadoop and it uses a bunch of small
> > files.
> > I'm aware that having such small files ain't a good idea but I'm not
> doing
> > the technical decisions and the port has to be done for yesterday...
> > Of course such small files are a problem, loading 64MB blocks for a few
> > lines of text is an evident loss.
> > What will happen if I set a smaller, or even way smaller (32kB) blocks?
> >
> > Thank you.
> >
> > Pierre ANCELOT.
>
>


-- 
http://www.neko-consulting.com
Ego sum quis ego servo
"Je suis ce que je protège"
"I am what I protect"

Re: Any possible to set hdfs block size to a value smaller than 64MB?

Hey Pierre,

These are not traditional filesystem blocks - if you save a file smaller than 
64MB, you don't lose 64MB of file space..

Hadoop will use 32KB to store a 32KB file (ok, plus a KB of metadata or so), 
not 64MB.

Brian

On May 18, 2010, at 7:06 AM, Pierre ANCELOT wrote:

> Hi,
> I'm porting a legacy application to hadoop and it uses a bunch of small
> files.
> I'm aware that having such small files ain't a good idea but I'm not doing
> the technical decisions and the port has to be done for yesterday...
> Of course such small files are a problem, loading 64MB blocks for a few
> lines of text is an evident loss.
> What will happen if I set a smaller, or even way smaller (32kB) blocks?
> 
> Thank you.
> 
> Pierre ANCELOT.

smime.p7s
Description: S/MIME cryptographic signature

Any possible to set hdfs block size to a value smaller than 64MB?