[ https://issues.apache.org/jira/browse/PIG-1037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12772772#action_12772772 ]
Alan Gates commented on PIG-1037: --------------------------------- The difference is much more than switching from dumping one tuple at a time to multiple tuples. It is about how spilling is activated. In the past, spilling was passive; it was done when the JVM informed us that memory was getting low. This did not work well as the JVM only checks memory usage when it garbage collects. So by the time pig was notified of a low memory condition it was often too late. We often ran out of memory while trying to spill. Now instead, spilling is active. Pig sets aside a buffer for a bag to put its tuples in. For default bags, once this buffer is full any additional tuples are written to disk. For sorted or distinct bags, once the buffer is full it is sorted and dumped to disk, and new records go into the buffer. This particular patch only adds the change for sorted and distinct bags. PIG-975 contains the original patch for default bags. > better memory layout and spill for sorted and distinct bags > ----------------------------------------------------------- > > Key: PIG-1037 > URL: https://issues.apache.org/jira/browse/PIG-1037 > Project: Pig > Issue Type: Improvement > Reporter: Olga Natkovich > Assignee: Ying He > Fix For: 0.6.0 > > Attachments: PIG-1037.patch, PIG-1037.patch2, PIG-1037.patch3 > > -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.