[
https://issues.apache.org/jira/browse/PIG-2293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13120577#comment-13120577
]
Aaron Klish commented on PIG-2293:
----------------------------------
Here is some performance data:
merge-sparse (Using IndexedStorage)
------------
SLOTS_MILLIS_MAPS 1,711,722
HDFS_BYTES_READ 13,305,558,069
HDFS_BYTES_WRITTEN 485,765,545
Execution Time: (2mins, 46sec)
merge (Using standard PigStorage)
-----
SLOTS_MILLIS_MAPS 3,633,465
HDFS_BYTES_READ 91,896,826,321
HDFS_BYTES_WRITTEN 485,765,545
Execution Time: (6mins, 53sec) This does not include the index phase.
Left Table
131,540 Records
Right Table
28,476,640 Records
90,549,549,781 bytes
Join
96,174 Records
The join key was 37 bytes.
Sparseness by records: 0.34%
Sparseness by bytes: 0.0039%
Other runs have similar data.
> Pig should support a more efficient merge join against data sources that
> natively support point lookups or where the join is against large, sparse
> tables.
> ----------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: PIG-2293
> URL: https://issues.apache.org/jira/browse/PIG-2293
> Project: Pig
> Issue Type: New Feature
> Components: impl
> Affects Versions: 0.9.0
> Reporter: Aaron Klish
> Assignee: Aaron Klish
> Fix For: 0.10
>
> Attachments: PIG-2293-1.patch, PIG-2293-2.patch, PIG-2293-3.patch,
> PIG-2293-4.patch, PIG-2293-5.patch, e2e_test.txt, patch.txt, patch.txt
>
> Original Estimate: 336h
> Remaining Estimate: 336h
>
> The existing PIG merge join has the following limitations:
> 1. It assumes the right side of the table must be accessed sequentially -
> record by record.
> 2. It does not perform well against large, sparse tables.
> The current implementation of the merge join introduced the interface
> IndexableLoadFunc. This 'LoadFunc'
> supports the ability to 'seekNear' a given key (before reading the next
> record).
> The merge join physical operator only calls 'seekNear' for the first key in
> each split (effectively eliminating splits
> where the first and subsequent keys will not be found). Subsequent joins are
> found by reading sequentially through
> the records on the right table looking for matches from the left table.
> While this method works well for dense join tables - it performs poorly
> against large sparse tables or data sources that support
> point lookups natively (HBase for example).
> The proposed enhancement is to add a new join type - 'merge-sparse' to PIG
> latin. When specified in the PIG script, this join type
> will cause the merge join operator to call seekNear on each and every key
> (rather than just the first in each split).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
