[
https://issues.apache.org/jira/browse/HDDS-4403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aravindan Vijayan updated HDDS-4403:
------------------------------------
Description:
Recon tracks the history for every container replica on the Ozone cluster in
its SQL DB (By default, this is Derby). To track this, it keeps track of the
last timestamp of a replica on a DN through reports. This becomes a SQL DB scan
+ write operation for every container report received. Even though there is
async hand off from the report to EventQueue, the event queue handler itself by
default uses 1 thread per event type (report type). Hence, there is implicit
blocking behavior here which is pushed down to DNs.
This has to be changed into a lazy update of DB to support better scalability.
Details on how to achieve this will be added to the JIRA later.
was:
Recon tracks the history for every container replica on the Ozone cluster. To
track this, it keeps track of the last timestamp of a replica on a DN through
reports. This becomes a SQL DB scan + write operation for every container
report received. On a cluster with million containers, this amounts to 1
million per scan + write operations per minute.
This has to be changed into a lazy update of DB to support better scalability.
Details on how to achieve this will be added to the JIRA later.
> Update the container replica history to the Recon DB lazily instead of for
> every report.
> ----------------------------------------------------------------------------------------
>
> Key: HDDS-4403
> URL: https://issues.apache.org/jira/browse/HDDS-4403
> Project: Hadoop Distributed Data Store
> Issue Type: Task
> Reporter: Aravindan Vijayan
> Priority: Major
> Fix For: 1.1.0
>
>
> Recon tracks the history for every container replica on the Ozone cluster in
> its SQL DB (By default, this is Derby). To track this, it keeps track of the
> last timestamp of a replica on a DN through reports. This becomes a SQL DB
> scan + write operation for every container report received. Even though
> there is async hand off from the report to EventQueue, the event queue
> handler itself by default uses 1 thread per event type (report type). Hence,
> there is implicit blocking behavior here which is pushed down to DNs.
> This has to be changed into a lazy update of DB to support better
> scalability. Details on how to achieve this will be added to the JIRA later.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
