[
https://issues.apache.org/jira/browse/OOZIE-3396?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Junfan Zhang updated OOZIE-3396:
--------------------------------
Description:
Phenomenon:
Recently we upgraded the {{AuthOozieClient}} and added the OOZIE-2447 (Illegal
character 0x0 oozie client) patch. When we were running online, the server
memory usage continued to increase. Through the jmap and jhat command analysis,
it was found that the {{deleteOnExist}} method in the writeAuthToken method
under the {{AuthOozieClient}} class occupies a large amount of memory, which is
caused by OOZIE-2447. Also, because I have applied the OOZIE-3379 in the
production environment. I still find that the .oozie-auth-token-xxxx file
update time has changed, but the file content has not changed.
After reviewing the code, I found that {{currentToken.equals(readToken)}} in
the {{AuthOozieClient}} class is always false because the
{{AuthenticatedURL.token}} does not have an equals method. The reason is
because {{currentToken.equals(readToken)}} is always false, causing the tmp
file to be created continuously, but there is a memory leak due to the
{{deleteOnExist}} method
([https://stackoverflow.com/questions/40119188/memory-leak-on-deleteonexithook])
Solution:
1. In {{AuthOozieClient}}, change to
{{currentToken.toString().equals(readToken.toString())}}
2. In addition, the deleteOnExist method should be changed to the delete() in
AuthOozieClient class.
Doubt:
Why delete the equals method in the {{AuthenticatedURL.Token}} class in Hadoop?
See: https://issues.apache.org/jira/browse/HADOOP-10771
was:
Phenomenon:
Recently we upgraded the AuthOozieClient and added the OOZIE-2447 (Illegal
character 0x0 oozie client) patch. When we were running online, the server
memory usage continued to increase. Through the jmap and jhat command analysis,
it was found that the deleteOnExist method in the writeAuthToken method under
the AuthOozieClient class occupies a large amount of memory, which is caused by
oozie-2447(https://issues.apache.org/jira/browse/OOZIE-2447).
Also, because I have applied the oozie-3379 patch in the production
environment. I still find that the .oozie-auth-token-xxxx file update time has
changed, but the file content has not changed.
After reviewing the code, I found that currentToken.equals(readToken) in the
AuthOozieClient class is always false because the AuthenticatedURL.token does
not have an equals method.
The reason is because currentToken.equals(readToken) is always false, causing
the tmp file to be created continuously, but there is a memory leak due to the
deleteOnExist method
(https://stackoverflow.com/questions/40119188/memory-leak-on-deleteonexithook)
Solution:
1. In the AuthOozieClient, change to
currentToken.toString().equals(readToken.toString())
2. In addition, the deleteOnExist method should be changed to the delete() in
AuthOozieClient class.
Doubt:
Why delete the equals method in the AuthenticatedURL.Token class in Hadoop?
See: https://issues.apache.org/jira/browse/HADOOP-10771
> AuthOozieClient leaks memory
> ----------------------------
>
> Key: OOZIE-3396
> URL: https://issues.apache.org/jira/browse/OOZIE-3396
> Project: Oozie
> Issue Type: Bug
> Affects Versions: 5.0.0
> Reporter: Junfan Zhang
> Assignee: Junfan Zhang
> Priority: Major
>
> Phenomenon:
> Recently we upgraded the {{AuthOozieClient}} and added the OOZIE-2447
> (Illegal character 0x0 oozie client) patch. When we were running online, the
> server memory usage continued to increase. Through the jmap and jhat command
> analysis, it was found that the {{deleteOnExist}} method in the
> writeAuthToken method under the {{AuthOozieClient}} class occupies a large
> amount of memory, which is caused by OOZIE-2447. Also, because I have applied
> the OOZIE-3379 in the production environment. I still find that the
> .oozie-auth-token-xxxx file update time has changed, but the file content has
> not changed.
> After reviewing the code, I found that {{currentToken.equals(readToken)}} in
> the {{AuthOozieClient}} class is always false because the
> {{AuthenticatedURL.token}} does not have an equals method. The reason is
> because {{currentToken.equals(readToken)}} is always false, causing the tmp
> file to be created continuously, but there is a memory leak due to the
> {{deleteOnExist}} method
> ([https://stackoverflow.com/questions/40119188/memory-leak-on-deleteonexithook])
> Solution:
> 1. In {{AuthOozieClient}}, change to
> {{currentToken.toString().equals(readToken.toString())}}
> 2. In addition, the deleteOnExist method should be changed to the delete() in
> AuthOozieClient class.
> Doubt:
> Why delete the equals method in the {{AuthenticatedURL.Token}} class in
> Hadoop? See: https://issues.apache.org/jira/browse/HADOOP-10771
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
