[ https://issues.apache.org/jira/browse/DATAFU-116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15187523#comment-15187523 ]
Matthew Hayes commented on DATAFU-116: -------------------------------------- bq. but the worst case is all of them, which is no worse than the current implementation. I think it would be worse than the current implementation actually. Pig does not keep the entire input bags in memory. I'm not an expert on Pig internals, but I believe as you iterate through the members of a DataBag it loads the data in chunks from disk. Without doing this it wouldn't be possible to operate on bags larger than what can fit in memory. > Make SetIntersect and SetDifference implement Accumulator > --------------------------------------------------------- > > Key: DATAFU-116 > URL: https://issues.apache.org/jira/browse/DATAFU-116 > Project: DataFu > Issue Type: Improvement > Affects Versions: 1.3.0 > Reporter: Eyal Allweil > > SetIntersect and SetDifference accept only sorted bags, and the output is > always smaller than the inputs. Therefore an accumulator implementation > should be possible and it will improve memory usage (somewhat) and allow Pig > to optimize loops with these operations better. -- This message was sent by Atlassian JIRA (v6.3.4#6332)