[
https://issues.apache.org/jira/browse/PIG-1472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12885835#action_12885835
]
Thejas M Nair commented on PIG-1472:
------------------------------------
I ran the pigmix1 queries with changes to specify types for columns in load
function, with this patch, and I see significant performance improvements for
following queries -
The time is in secs, and the time in table is the best out of 3 runs. The
queries that show significant improvement (> 10%) are ones where bag/map
columns are written across Map/Reduce boundaries. (In pigmix v1 L2, L3 queries
don't prune the bag and map columns before group-by or join.)
||pigmix1 query || before patch || after patch|| % diff ||
|L1 | 211 | 198 | 6%|
| L2 | 514|424 | 17% |
| L3| 670|541 | 19%|
| L4 | 133|123 | 7.5%|
| L5| 118|113 | |
| L6 | 139|134 | |
| L7 | 114|114 | |
| L8 | 68|69 | |
| L9| 1113|957 | 14%|
| L10 | 1153| 998| 13.4%|
| L11 | 317| 317| |
| L12 | 124| 123| |
> Optimize serialization/deserialization between Map and Reduce and between MR
> jobs
> ---------------------------------------------------------------------------------
>
> Key: PIG-1472
> URL: https://issues.apache.org/jira/browse/PIG-1472
> Project: Pig
> Issue Type: Improvement
> Affects Versions: 0.8.0
> Reporter: Thejas M Nair
> Assignee: Thejas M Nair
> Fix For: 0.8.0
>
> Attachments: PIG-1472.patch
>
>
> In certain types of pig queries most of the execution time is spent in
> serializing/deserializing (sedes) records between Map and Reduce and between
> MR jobs.
> For example, if PigMix queries are modified to specify types for all the
> fields in the load statement schema, some of the queries (L2,L3,L9, L10 in
> pigmix v1) that have records with bags and maps being transmitted across map
> or reduce boundaries run a lot longer (runtime increase of few times has been
> seen.
> There are a few optimizations that have shown to improve the performance of
> sedes in my tests -
> 1. Use smaller number of bytes to store length of the column . For example if
> a bytearray is smaller than 255 bytes , a byte can be used to store the
> length instead of the integer that is currently used.
> 2. Instead of custom code to do sedes on Strings, use DataOutput.writeUTF and
> DataInput.readUTF. This reduces the cost of serialization by more than 1/2.
> Zebra and BinStorage are known to use DefaultTuple sedes functionality. The
> serialization format that these loaders use cannot change, so after the
> optimization their format is going to be different from the format used
> between M/R boundaries.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
