[
https://issues.apache.org/jira/browse/PIG-1313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12848442#action_12848442
]
Bill Graham commented on PIG-1313:
----------------------------------
Thanks for the pointer to PIG-240. The patch included make it look like that
bug is focusing specifically on the LogicalPlan refactor.
In my case I do need a multi-threaded PigServer instance. My VM is running a
scheduler which wakes up and invokes scripts periodically and it is possible
that one script will be kicked off while another is already running. As a
result, calling FileLocalizer.deleteTempFiles() can cause problem. This is
because the toDelete Stack and others are static.
One possible fix would be to bind a toDelete Stack to each thread local
instance, or to make FileLocalizer non-static. The later seems cleaner but
would require modifying a lot more code.
> PigServer leaks memory over time
> --------------------------------
>
> Key: PIG-1313
> URL: https://issues.apache.org/jira/browse/PIG-1313
> Project: Pig
> Issue Type: Bug
> Reporter: Bill Graham
>
> When {{PigServer}} runs it creates temporary files using the
> {{FileLocalizer.getTemporaryPath(..)}}. This static method creates and
> returns a handle to a temporary file (as an instance of
> {{ElementDescriptor}}). The {{ElementDescriptors}} returned by this method
> are kept on a static {{Stack}} named {{toDelete}}. The items on {{toDelete}}
> get removed by the {{FileLocalizer.deleteTempFile()}} method.
> The only place in the code where I see {{FileLocalizer.deleteTempFile()}}
> called is in the Main class. {{PigServer}} does not call that method though,
> so a long-running VM that repeatedly uses instances of {{PigServer}} to run
> jobs will leak memory via {{toDelete}}.
> One suggested fix is to have {{PigServer.shutdown()}} call
> {{FileLocalizer.deleteTempFile()}}, but this would cause problems in a
> multi-threaded environment, since it seems {{ElementDescriptors}} are pushed
> onto the {{toDelete}} stack before they're used, not once they're done with.
> With this approach, running multiple instances of {{PigServer}} in separate
> threads could cause one completed job to clobber the other's still-in-use
> temp files.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
