[ https://issues.apache.org/jira/browse/PIG-2293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13120577#comment-13120577 ]
Aaron Klish commented on PIG-2293: ---------------------------------- Here is some performance data: merge-sparse (Using IndexedStorage) ------------ SLOTS_MILLIS_MAPS 1,711,722 HDFS_BYTES_READ 13,305,558,069 HDFS_BYTES_WRITTEN 485,765,545 Execution Time: (2mins, 46sec) merge (Using standard PigStorage) ----- SLOTS_MILLIS_MAPS 3,633,465 HDFS_BYTES_READ 91,896,826,321 HDFS_BYTES_WRITTEN 485,765,545 Execution Time: (6mins, 53sec) This does not include the index phase. Left Table 131,540 Records Right Table 28,476,640 Records 90,549,549,781 bytes Join 96,174 Records The join key was 37 bytes. Sparseness by records: 0.34% Sparseness by bytes: 0.0039% Other runs have similar data. > Pig should support a more efficient merge join against data sources that > natively support point lookups or where the join is against large, sparse > tables. > ---------------------------------------------------------------------------------------------------------------------------------------------------------- > > Key: PIG-2293 > URL: https://issues.apache.org/jira/browse/PIG-2293 > Project: Pig > Issue Type: New Feature > Components: impl > Affects Versions: 0.9.0 > Reporter: Aaron Klish > Assignee: Aaron Klish > Fix For: 0.10 > > Attachments: PIG-2293-1.patch, PIG-2293-2.patch, PIG-2293-3.patch, > PIG-2293-4.patch, PIG-2293-5.patch, e2e_test.txt, patch.txt, patch.txt > > Original Estimate: 336h > Remaining Estimate: 336h > > The existing PIG merge join has the following limitations: > 1. It assumes the right side of the table must be accessed sequentially - > record by record. > 2. It does not perform well against large, sparse tables. > The current implementation of the merge join introduced the interface > IndexableLoadFunc. This 'LoadFunc' > supports the ability to 'seekNear' a given key (before reading the next > record). > The merge join physical operator only calls 'seekNear' for the first key in > each split (effectively eliminating splits > where the first and subsequent keys will not be found). Subsequent joins are > found by reading sequentially through > the records on the right table looking for matches from the left table. > While this method works well for dense join tables - it performs poorly > against large sparse tables or data sources that support > point lookups natively (HBase for example). > The proposed enhancement is to add a new join type - 'merge-sparse' to PIG > latin. When specified in the PIG script, this join type > will cause the merge join operator to call seekNear on each and every key > (rather than just the first in each split). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira