[
https://issues.apache.org/jira/browse/PIG-480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ying He updated PIG-480:
------------------------
Attachment: PIG_480.patch
patch to use identity map.
An IdentityMapOptimizer is applied when a MR plan contains at least 2 MRs. It
evaluates each MR job, if its reducer uses POStore to dump a tmp file, and the
mapper of next MR only contains a POLocalRearrange and a POLoad to load the tmp
file, then the POLocalRearrange of next mapper is moved up to the reducer of
this MR, and the mapper of next MR job is changed to use identity map.
In this case, the reducer of the MR job output (key, tuple) pairs to the tmp
file by using a different OutputFormat, PigBinaryValueOutputFormat. It uses a
different record writer to dump data, the format is
delimiter (3 bytes,, 0x01, 0x02, 0x03)
key
length of byte[] for tuple
byte[] for tuple
the next MR job that uses identity map uses a different InputFormat,
PigBinaryValueInputFormat, which returns a different RecordReader, to read in
data as (key, tuple) pairs. But the tuple is kept in byte[] form. The identity
map does nothing except passing the (key, tuple) through and writing them to
disk. When reducer picks them up, the tuple is de-serialized for processing.
The reason of doing this is performance. Because the tuple reading in and
writing out of identity map are in byte[] form, we saved a de-serialization and
serialization of tuples in mapper.
A use case is following:
a = load 'f' as (id, v);
b = load 's' as (id, v);
c = join a by id, b by id;
d = group c by a::id;
dump d;
this example contains 2 MR jobs. After optimization, the first job output
(key, tuple) pairs, and second job uses identity map.
> PERFORMANCE: Use identity mapper in a chain of M-R jobs
> -------------------------------------------------------
>
> Key: PIG-480
> URL: https://issues.apache.org/jira/browse/PIG-480
> Project: Pig
> Issue Type: Improvement
> Affects Versions: 0.2.0
> Reporter: Olga Natkovich
> Attachments: PIG_480.patch
>
>
> For jobs with two or more MR jobs, use identity mapper wherever possible in
> second and subsequent MR jobs. Identity mapper is about 50% than pig empty
> map job because it doesn't parse the data.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
