[
https://issues.apache.org/jira/browse/HDDS-17655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-17655:
------------------------------
Description:
Snapshot DBs and OM DB checkpoints are opened through
{{OmMetadataManagerImpl#initializeOmTables}} with {{CacheType.PARTIAL_CACHE}}.
If a table ignores that argument and uses {{FULL_CACHE}}, every open loads the
whole table and starts a {{FullTableCache}} cleanup thread that is never shut
down (HDDS-17460), so the OM leaks one thread per snapshot DB open.
HDDS-14091 did exactly this for {{s3RevokedStsTokenTable}}. Master is not
affected, because the STS branch merge f7c62ae2329 changed that line to
{{cacheType}}, but nothing prevents it from happening again. Branches that
picked up HDDS-14091 without that merge leaked about 170K threads per OM in a
snapshot scale test.
Proposal:
* Replace {{OmMetadataManagerImpl.TableInitializer#get(definition, cacheType)}}
with {{getWithDbCache(definition)}}, which always uses the cache type the DB
was opened with, so a table cannot pick {{FULL_CACHE}} in a snapshot or
checkpoint DB through {{TableInitializer}}.
* Add a unit test that repeatedly opens a snapshot DB and a checkpoint DB,
checks that no {{FullTableCache}} cleanup thread starts while they are open,
and checks that no thread is left behind after they are closed.
was:
Snapshot DBs and OM DB checkpoints are opened through
{{OmMetadataManagerImpl#initializeOmTables}} with {{CacheType.PARTIAL_CACHE}}.
If a table ignores that argument and uses {{FULL_CACHE}}, every open loads the
whole table and starts a {{FullTableCache}} cleanup thread that is never shut
down (HDDS-17460), so the OM leaks one thread per snapshot DB open.
HDDS-14091 did exactly this for {{s3RevokedStsTokenTable}}. Master is not
affected, because the STS branch merge f7c62ae2329 changed that line to
{{cacheType}}, but nothing prevents it from happening again. Branches that
picked up HDDS-14091 without that merge leaked about 170K threads per OM in a
snapshot scale test.
Proposal:
* Make {{OmMetadataManagerImpl.TableInitializer}} reject a {{FULL_CACHE}} table
in a DB opened with {{PARTIAL_CACHE}}, so the mistake fails on the first
snapshot DB open in any test.
* Add a unit test that repeatedly opens and closes a snapshot DB and a
checkpoint DB and checks that no thread is left behind.
> Prevent FULL_CACHE tables in snapshot and checkpoint DBs
> --------------------------------------------------------
>
> Key: HDDS-17655
> URL: https://issues.apache.org/jira/browse/HDDS-17655
> Project: Apache Ozone
> Issue Type: Improvement
> Components: OM, Snapshot
> Reporter: Siyao Meng
> Assignee: Siyao Meng
> Priority: Major
> Labels: pull-request-available
>
> Snapshot DBs and OM DB checkpoints are opened through
> {{OmMetadataManagerImpl#initializeOmTables}} with
> {{CacheType.PARTIAL_CACHE}}. If a table ignores that argument and uses
> {{FULL_CACHE}}, every open loads the whole table and starts a
> {{FullTableCache}} cleanup thread that is never shut down (HDDS-17460), so
> the OM leaks one thread per snapshot DB open.
> HDDS-14091 did exactly this for {{s3RevokedStsTokenTable}}. Master is not
> affected, because the STS branch merge f7c62ae2329 changed that line to
> {{cacheType}}, but nothing prevents it from happening again. Branches that
> picked up HDDS-14091 without that merge leaked about 170K threads per OM in a
> snapshot scale test.
> Proposal:
> * Replace {{OmMetadataManagerImpl.TableInitializer#get(definition,
> cacheType)}} with {{getWithDbCache(definition)}}, which always uses the cache
> type the DB was opened with, so a table cannot pick {{FULL_CACHE}} in a
> snapshot or checkpoint DB through {{TableInitializer}}.
> * Add a unit test that repeatedly opens a snapshot DB and a checkpoint DB,
> checks that no {{FullTableCache}} cleanup thread starts while they are open,
> and checks that no thread is left behind after they are closed.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
