[
https://issues.apache.org/jira/browse/RANGER-3386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
caozhiqiang updated RANGER-3386:
--------------------------------
Description:
org.apache.commons.collections.CollectionUtils' performance is too poor and
reduce the performance of ranger plugins, particularly with hdfs. There are too
much places used CollectionUtils.isNotEmpty and CollectionUtils.isEmpty, and we
should replace them.
We can see many CollectionUtils call is take too much time in namenode
benchmark's result.
In this patch, I replace almost CollectionUtils functions in agents-common.
After add this patch, in create file scenario in hdfs benchmark, the
performance can improve from creating 7000 files to 7600 files per second.
By the way, I write a simple test code below. collection.isEmpty is almost take
0 milliseconds, but CollectionUtils.isNotEmpty take 5 milliseconds.
{code:java}
List<Integer> list = new ArrayList<Integer>();
for(int i = 0; i<1000; i++)
list.add(i);
long startTime = System.currentTimeMillis();
if(list != null && !list.isEmpty()){
}
long endTime = System.currentTimeMillis();
long totalTime = endTime - startTime;
System.out.println(totalTime);
long startTime2 = System.currentTimeMillis();
CollectionUtils.isNotEmpty(list);
long endTime2 = System.currentTimeMillis();
long totalTime2 = endTime2 - startTime2;
System.out.println(totalTime2);
{code}
!CollectionUtils.png|width=680,height=324!
was:
org.apache.commons.collections.CollectionUtils' performance is too poor and
reduce the performance of ranger plugins, particularly with hdfs. There are too
much places used CollectionUtils.isNotEmpty and CollectionUtils.isEmpty, and we
should replace them.
We can see many CollectionUtils call is take too much time in namenode
benchmark's result.
By the way, I write a simple test code below. collection.isEmpty is almost take
0 milliseconds, but CollectionUtils.isNotEmpty take 5 milliseconds.
{code:java}
List<Integer> list = new ArrayList<Integer>();
for(int i = 0; i<1000; i++)
list.add(i);
long startTime = System.currentTimeMillis();
if(list != null && !list.isEmpty()){
}
long endTime = System.currentTimeMillis();
long totalTime = endTime - startTime;
System.out.println(totalTime);
long startTime2 = System.currentTimeMillis();
CollectionUtils.isNotEmpty(list);
long endTime2 = System.currentTimeMillis();
long totalTime2 = endTime2 - startTime2;
System.out.println(totalTime2);
{code}
In this patch, I replace almost CollectionUtils functions in agents-common.
After add this patch, in create file scenario in hdfs benchmark, the
performance can improve from creating 7000 files to 7600 files per second.
!CollectionUtils.png|width=680,height=324!
> apache CollectionUtils package reduce ranger agent performance and should be
> replaced
> -------------------------------------------------------------------------------------
>
> Key: RANGER-3386
> URL: https://issues.apache.org/jira/browse/RANGER-3386
> Project: Ranger
> Issue Type: Improvement
> Components: plugins
> Affects Versions: 2.0.1
> Reporter: caozhiqiang
> Assignee: caozhiqiang
> Priority: Major
> Attachments: CollectionUtils.png, RANGER-3386-branch-2.0.1.001.patch
>
>
> org.apache.commons.collections.CollectionUtils' performance is too poor and
> reduce the performance of ranger plugins, particularly with hdfs. There are
> too much places used CollectionUtils.isNotEmpty and CollectionUtils.isEmpty,
> and we should replace them.
> We can see many CollectionUtils call is take too much time in namenode
> benchmark's result.
> In this patch, I replace almost CollectionUtils functions in agents-common.
> After add this patch, in create file scenario in hdfs benchmark, the
> performance can improve from creating 7000 files to 7600 files per second.
> By the way, I write a simple test code below. collection.isEmpty is almost
> take 0 milliseconds, but CollectionUtils.isNotEmpty take 5 milliseconds.
>
> {code:java}
> List<Integer> list = new ArrayList<Integer>();
> for(int i = 0; i<1000; i++)
> list.add(i);
> long startTime = System.currentTimeMillis();
> if(list != null && !list.isEmpty()){
> }
> long endTime = System.currentTimeMillis();
> long totalTime = endTime - startTime;
> System.out.println(totalTime);
> long startTime2 = System.currentTimeMillis();
> CollectionUtils.isNotEmpty(list);
> long endTime2 = System.currentTimeMillis();
> long totalTime2 = endTime2 - startTime2;
> System.out.println(totalTime2);
> {code}
>
> !CollectionUtils.png|width=680,height=324!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
