[
https://issues.apache.org/jira/browse/PIG-2918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dmitriy V. Ryaboy updated PIG-2918:
-----------------------------------
Resolution: Fixed
Fix Version/s: 0.11
Status: Resolved (was: Patch Available)
Committed to 0.11 (trunk). Thanks for the review Alan.
Btw, I profiled this change -- prior to this patch, calling TOBAG() on about
60,000 tuples resulted in 530Kb worth of DefaultDataBags, and 2Mb of
WeakReferences (!). After the patch, practically no WeakReferences, and only
175K NonSpillableDataBags. Turns out this sort of thing adds up fast.
> Avoid Spillable bag overhead where possible
> -------------------------------------------
>
> Key: PIG-2918
> URL: https://issues.apache.org/jira/browse/PIG-2918
> Project: Pig
> Issue Type: Bug
> Reporter: Dmitriy V. Ryaboy
> Assignee: Dmitriy V. Ryaboy
> Fix For: 0.11
>
> Attachments: PIG-2918.patch
>
>
> We use BagFactory.newDefaultBag() liberally, and pay a price -- each such bag
> registers with the spillable memory manager, and if we allocate a lot of tiny
> bags, we wind up paying for maintaining and cleaning up the internal linked
> list of weak references.
> In many cases, we know a-priori that the bags are smal, and should probably
> be creating non-spillable bags for those cases.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
