[
https://issues.apache.org/jira/browse/HDDS-5384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei-Chiu Chuang updated HDDS-5384:
----------------------------------
Attachment: om_liststatus_alloc_before.svg
om_liststatus_cpu_before.svg
om_listatus_alloc_after.svg
om_liststatus_cpu_after.svg
> OM refreshPipeline should not invoke the expensive
> OmKeyLocationInfoGroup.getLocationList()
> -------------------------------------------------------------------------------------------
>
> Key: HDDS-5384
> URL: https://issues.apache.org/jira/browse/HDDS-5384
> Project: Apache Ozone
> Issue Type: Improvement
> Components: OM
> Reporter: Wei-Chiu Chuang
> Assignee: Wei-Chiu Chuang
> Priority: Major
> Attachments: om_listatus_alloc_after.svg,
> om_liststatus_alloc_before.svg, om_liststatus_cpu_after.svg,
> om_liststatus_cpu_before.svg
>
>
> The OM's refreshPipeline (used by liststatus) implementation iterates over
> OmKeyLocationInfoGroup.getLocationList(), which is a very expensive call. It
> iterate over a collection of list of objects, allocates a new list, perform
> operations on each of them. In short, it's an O(n) method in terms of space
> and time complexity.
> There are many places in the Ozone code that uses this method. Most usages
> iterates over the generated list, without modifying the list. We should
> instead return the collection of lists, which is O(1).
> I have a client that issues many listStatus calls to examine the effect.
> Before the change, refreshPipeline costs 8.65% of heap usage. After: 1.95%.
> CPU cost: before: 8.18% after: 4.22%
> We should refrain from invoking getLocationList() as much as possible. But
> given the wide usage in the code, I elect not to remove the usage completely
> to avoid destabilizing it. Instead, I changed the usage in refreshPipeline to
> demonstrate its impact.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@ozone.apache.org
For additional commands, e-mail: issues-h...@ozone.apache.org
