[
https://issues.apache.org/jira/browse/PIG-1313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12848391#action_12848391
]
Daniel Dai commented on PIG-1313:
---------------------------------
Yes, your observation is valid. When we design PigServer, we do not put long
running and multi-thread in mind. We already have
[PIG-240|https://issues.apache.org/jira/browse/PIG-240] in an attempt to make
PigServer multi-thread. In your case, it seems all you need is to
intermittently instantiate PigServer and run script within one java VM. So we
need to ensure when the first PigServer is done, we reclaim all the resources
local to that particular PigServer instance. And PigServer.shutdown sounds like
a reasonable place to do that.
FileLocalizer.toDelete and FileLocalizer.deleteOnFail are such resources local
to PigServer instance. Seems that we should move them into PigServer so that we
can clear them safely during shutdown.
> PigServer leaks memory over time
> --------------------------------
>
> Key: PIG-1313
> URL: https://issues.apache.org/jira/browse/PIG-1313
> Project: Pig
> Issue Type: Bug
> Reporter: Bill Graham
>
> When {{PigServer}} runs it creates temporary files using the
> {{FileLocalizer.getTemporaryPath(..)}}. This static method creates and
> returns a handle to a temporary file (as an instance of
> {{ElementDescriptor}}). The {{ElementDescriptors}} returned by this method
> are kept on a static {{Stack}} named {{toDelete}}. The items on {{toDelete}}
> get removed by the {{FileLocalizer.deleteTempFile()}} method.
> The only place in the code where I see {{FileLocalizer.deleteTempFile()}}
> called is in the Main class. {{PigServer}} does not call that method though,
> so a long-running VM that repeatedly uses instances of {{PigServer}} to run
> jobs will leak memory via {{toDelete}}.
> One suggested fix is to have {{PigServer.shutdown()}} call
> {{FileLocalizer.deleteTempFile()}}, but this would cause problems in a
> multi-threaded environment, since it seems {{ElementDescriptors}} are pushed
> onto the {{toDelete}} stack before they're used, not once they're done with.
> With this approach, running multiple instances of {{PigServer}} in separate
> threads could cause one completed job to clobber the other's still-in-use
> temp files.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
