[
https://issues.apache.org/jira/browse/DATAFU-116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15187523#comment-15187523
]
Matthew Hayes commented on DATAFU-116:
--------------------------------------
bq. but the worst case is all of them, which is no worse than the current
implementation.
I think it would be worse than the current implementation actually. Pig does
not keep the entire input bags in memory. I'm not an expert on Pig internals,
but I believe as you iterate through the members of a DataBag it loads the data
in chunks from disk. Without doing this it wouldn't be possible to operate on
bags larger than what can fit in memory.
> Make SetIntersect and SetDifference implement Accumulator
> ---------------------------------------------------------
>
> Key: DATAFU-116
> URL: https://issues.apache.org/jira/browse/DATAFU-116
> Project: DataFu
> Issue Type: Improvement
> Affects Versions: 1.3.0
> Reporter: Eyal Allweil
>
> SetIntersect and SetDifference accept only sorted bags, and the output is
> always smaller than the inputs. Therefore an accumulator implementation
> should be possible and it will improve memory usage (somewhat) and allow Pig
> to optimize loops with these operations better.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
