[ https://issues.apache.org/jira/browse/HDDS-1232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16787555#comment-16787555 ]
Aravindan Vijayan commented on HDDS-1232: ----------------------------------------- *A summary of what the patch contains* A basic requirement from Recon is that it should be able to tell which keys are part of a container. The Recon Container DB is designed to hold this information which is basically a reverse map of the OM DB information. Instead of storing the entire Container -> Key mapping, it stores the map of (Container, Key Prefix) -> Count of keys that match that prefix in the container. For example, if a container 'C1' has the following keys * Vol1/Bucket1/Prefix1ABC * Vol1/Bucket1/Prefix1EFG * Vol1/Bucket1/Prefix2DEF * Vol1/Bucket2/Prefix1HIG This information is stored as follows in the Recon container DB. C1-Vol1/Bucket1/Prefix1 -> 2 C1-Vol1/Bucket1/Prefix2 -> 1 C1-Vol1/Bucket2/Prefix1 -> 1 Using the above mapping and the OM metadata DB snapshot (HDDS-1233), we can get the mapping of Container -> Key and Container -> BlockId. The interface ContainerDBServiceProvider provides APIs to store / get this information. The ContainerDBServiceProviderImpl is the implementation of this interface and is instantiated through injection. It holds an instance of RocksDB which it uses to store the above information. This patch also contains other related changes like * wiring up hadoop-ozone-recon to the other modules in hadoop-ozone distribution. * Adding testing of Recon config keys to the TestConfigurationFields utility. * Refactor a utility method to be re-usable in org.apache.hadoop.hdds.server.ServerUtils which attempts to get the DB dir for a given component from config, and falls back to ozone.metadata.dirs if the config is not present. * Add a prefix seek iterator in the org.apache.hadoop.utils.MetaStoreIterator wrapper around the underlying RocksDB/LevelDB iterator + corresponding unit test. > Recon Container DB service definition > ------------------------------------- > > Key: HDDS-1232 > URL: https://issues.apache.org/jira/browse/HDDS-1232 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Recon > Reporter: Aravindan Vijayan > Assignee: Aravindan Vijayan > Priority: Major > Labels: pull-request-available > Fix For: 0.5.0 > > Attachments: HDDS-1232-000.patch, HDDS-1232-001.patch, > HDDS-1232-002.patch, HDDS-1232-003.patch, HDDS-1232-004.patch > > Time Spent: 1h > Remaining Estimate: 0h > > * Define the Ozone Recon container DB service definition. -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org