Expanding on Alan’s post:
Files are intended to span many blocks and a single file may be read by many
mappers. In order for a file to be read by many mappers, it goes through a
process called input splits which splits the input around hdfs block boundaries.
If a unit of data within a file crosses a hdfs block, a portion of that unit of
data must be sent from the node which contains block/mapper of one portion to
the node that contains the block/mapper of the other portion. Take a csv file
for example, in this case a unit of data is a line, and transferring a portion
of a line between boxes is no big deal.
This changes a bit for orc files as the unit of data is a stripe. An orc stripe
is typically a few hundred MB. Without some additional logic, a substantial
part of data locality would be lost; however, orc has such additional logic.
The stripe size of the orc file should be set a few MB below the hdfs block
size and padding enable to produce a 1:1 relationship between an orc stripe and
an hdfs block. How many stripes or blocks that are “in" a single file is of no
consequence so long as this 1:1 relationship is maintained.
Below is an example config for 128mb hdfs blocks.
Configuration writerConf = new Configuration();
// other config
OrcFile.WriterOptions writerOptions =
OrcFile.writerOptions(writerConf);
writerOptions.blockPadding(true);
writerOptions.stripeSize(122 * 1024 * 1024);
// other options
Writer writer = OrcFile.createWriter(path, writerOptions);
[http://www.cisco.com/web/europe/images/email/signature/est2014/logo_06.png?ct=1398192119726]
Grant Overby
Software Engineer
Cisco.com<http://www.cisco.com/>
[email protected]<mailto:[email protected]>
Mobile: 865 724 4910
[http://www.cisco.com/assets/swa/img/thinkbeforeyouprint.gif] Think before you
print.
This email may contain confidential and privileged material for the sole use of
the intended recipient. Any review, use, distribution or disclosure by others
is strictly prohibited. If you are not the intended recipient (or authorized to
receive for the recipient), please contact the sender by reply email and delete
all copies of this message.
Please click
here<http://www.cisco.com/web/about/doing_business/legal/cri/index.html> for
Company Registration Information.
From: Alan Gates <[email protected]<mailto:[email protected]>>
Reply-To: "[email protected]<mailto:[email protected]>"
<[email protected]<mailto:[email protected]>>
Date: Monday, April 27, 2015 at 2:05 PM
To: "[email protected]<mailto:[email protected]>"
<[email protected]<mailto:[email protected]>>
Subject: Re: ORC file across multiple HDFS blocks
to cross blocks and hence n
