[ https://issues.apache.org/jira/browse/PIG-1458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Richard Ding updated PIG-1458: ------------------------------ Attachment: PIG-1458.patch This patch uses the new multi-file-combiner (PIG-1518) to concatenate many small files for replicated join. This is based on the assumption that the total size of the replicated files should be small enough to fit into main memory. > aggregate files for replicated join > ----------------------------------- > > Key: PIG-1458 > URL: https://issues.apache.org/jira/browse/PIG-1458 > Project: Pig > Issue Type: Improvement > Reporter: Olga Natkovich > Assignee: Richard Ding > Fix For: 0.8.0 > > Attachments: PIG-1458.patch > > > We have noticed that if the smaller data in replicated join has many files, > this puts unneeded burden on the name node. pre-aggregating the files can > improve the situation -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.