[
https://issues.apache.org/jira/browse/OOZIE-3396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16706999#comment-16706999
]
Andras Piros commented on OOZIE-3396:
-------------------------------------
Thanks for the ideas and the accurate root cause analysis [~zuston]! Now I see
the use case behind OOZIE-3379: you're using {{AuthOozieClient}} from inside a
server JVM, where it's practically never stopped / restarted. For that cases,
usage of {{File#deleteOnExit()}} is not recommended.
I suggest following points while delivering the needed functionality while
keeping commits / Jiras atomic:
# finish patch for OOZIE-3379 while using
{{currentToken.toString().equals(readToken.toString())}} instead of
{{currentToken.equals(readToken)}}. Create also unit tests that cover the
functionality change: when authentication URL is the same, auth token file
shouldn't be recreated or rewritten within its TTL
# in this Jira OOZIE-3396 create a mechanism that periodically purges auth
token cache files from file system and from the heap. Usage of Guava's
{{LoadingCache#removalListener()}} is [much
recommended|https://github.com/google/guava/wiki/CachesExplained]. I'd also
create unit / integration tests there, e.g. checking different combinations of
short- and long-running {{AuthOozieClient}} instances opening all variations of
same / different authentication URLs
> AuthOozieClient leak memeory
> ----------------------------
>
> Key: OOZIE-3396
> URL: https://issues.apache.org/jira/browse/OOZIE-3396
> Project: Oozie
> Issue Type: Bug
> Affects Versions: 5.0.0
> Reporter: Junfan Zhang
> Assignee: Junfan Zhang
> Priority: Major
>
> Phenomenon:
> Recently we upgraded the AuthOozieClient and added the OOZIE-2447 (Illegal
> character 0x0 oozie client) patch. When we were running online, the server
> memory usage continued to increase. Through the jmap and jhat command
> analysis, it was found that the deleteOnExist method in the writeAuthToken
> method under the AuthOozieClient class occupies a large amount of memory,
> which is caused by
> oozie-2447(https://issues.apache.org/jira/browse/OOZIE-2447).
> Also, because I have applied the oozie-3379 patch in the production
> environment. I still find that the .oozie-auth-token-xxxx file update time
> has changed, but the file content has not changed.
> After reviewing the code, I found that currentToken.equals(readToken) in the
> AuthOozieClient class is always false because the AuthenticatedURL.token does
> not have an equals method.
> The reason is because currentToken.equals(readToken) is always false, causing
> the tmp file to be created continuously, but there is a memory leak due to
> the deleteOnExist method
> (https://stackoverflow.com/questions/40119188/memory-leak-on-deleteonexithook)
>
> Solution:
> 1. In the AuthOozieClient, change to
> currentToken.toString().equals(readToken.toString())
> 2. In addition, the deleteOnExist method should be changed to the delete() in
> AuthOozieClient class.
> Doubt:
> Why delete the equals method in the AuthenticatedURL.Token class in Hadoop?
> See: https://issues.apache.org/jira/browse/HADOOP-10771
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
