[ https://issues.apache.org/jira/browse/HDDS-330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Anu Engineer updated HDDS-330: ------------------------------ Labels: alpha2 (was: ) > Ozone: number of keys/values/buckets to KSMMetrics > -------------------------------------------------- > > Key: HDDS-330 > URL: https://issues.apache.org/jira/browse/HDDS-330 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: Ozone Manager > Reporter: Elek, Marton > Priority: Major > Labels: alpha2 > > During my last ozone test with 100 node ozone cluster I see a problem to > track how many keys/volumes/buckets do I have. > I opened this jira to start a discussion about extending KSM metrics (but let > me know if this is already planned somewhere else) and add number of > keys/volumes/buckets to the metrics interface. > These counters could be added to anywhere else (for example as a client call) > but I think it is an important number and would be worth to monitor it. > I see multiple ways to achieve it: > 1. Extend the `org.apache.hadoop.utils.MetadaStore` class with an additional > count() method. As I know there is no easy way to implement it with leveldb > but with rocksdb there is a posibility to get the _estimated_ number of keys. > On the other hand KSM stores volumes/buckets/keys in the same db, so we can't > use it without splitting the ksm.db to separated dbs. > 2. Create a background task to iterate over all the keys and count ozone > key/volume/bucket numbers: > pro: it would be independent from the existing program flow > con: doesn't provided up-to-date information. > con: it uses more resources to scan the whole db frequently > 3. During the startup we can iterate over the whole ksm.db and count the > current metrics, and later we can update the numbers in case of new > create/delete calls. It uses additional resources during the startup (should > be checked how much time is to parse a db with millions of keys) but after > that it would be fast. Also we can introduce new confguration variables to > skip the initial scan. In that case the numbers will be valid only from the > last restart but the startup would be fast. > I suggest to use the 3rd approach, could you please comment about your > opinion? -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org