errose28 opened a new pull request #1298: URL: https://github.com/apache/hadoop-ozone/pull/1298
## What changes were proposed in this pull request? This pull request adds column family support for Datanode RocksDB instances. Originally, datanodes placed all of their data under the default column family in RocksDB. This differs from OM and SCM which organize their data into different column families based on its type. This feature will move the datanode code off of the database utilities in the hadoop.hdds.utils package (which has no column family support), and move them to the newer utilities used by OM and SCM in the hadoop.hdds.utils.db package (which has column family support). It then divides data for new datanode containers into three column families. This is implemented in a backwards compatible way, so that containers written in the old format can still be used. Since this is a rather large pull request, a breakdown of the different components is provided below. ### Migration From Old to New Database Interfaces In order to divide container data into different column families, datanode code had to first be moved off of the old database utilities in the hadoop.hdds.utils package (which has no column family support), and moved to the newer utilities used by OM and SCM in the hadoop.hdds.utils.db package (which has column family support). In addition to the ability to use column families, this also provides a strong typing layer to the tables. The majority of file changes are just swapping calls made using the old interface to their equivalent versions in the new interface. This allowed removing the byte stream conversions that previously had to be done when interacting with the database, since the new interface provides codecs to handle this. The code using the new interface should exhibit identical behavior to that using the old interface. In cases where the new interface lacked functionality present in the old interface, that functionality was added to the new interface. Apart from these minor extensions, this pull request does not modify the new interface or existing implementations, but adds new implementations to the interface needed by the datanode. ### New Container Layout Any new containers created with this code will divide data in to three column families: 1. **block_data** - Key type: `String` - Block ID with optional prefix. - Value type: `BlockData` 2. **metadata** - Key type: `String` - Name of metadata field. - Value type: `Long` - The value of the field. 3. **deleted_blocks** - Key type: `String` - Block ID with optional prefix. - Value type: `ChunkInfoList` - A new type created that allows encoding/decoding and saving the chunk information associated with blocks that have been deleted. - This value is not currently used by the code (except in tests), but is included for potential future use. - The underlying chunks are still deleted, only the information about them is retained. ### Backwards Compatibility To distinguish between containers created in the original layout with only a default column family, and containers created in the new layout with three column families, a new field was added to the .container files, called *schemaVersion*. *schemaVersion* 1, or a missing *schemaVersion* value, indicates that the container was created in the old layout. *schemaVersion* 2 indicates that the container was created in the new layout. The code will use the proper `DatanodeStore` implementation for the corresponding schema version, and callers do not need to change their operations based on the schema version. Unfortunately, the existing design of the interface requires callers to specify which column families (called `Table`s by the interface) they interact with, which makes it difficult to abstract out this piece of the implementation. In order to allow the same set of calls to work with both schema versions, callers always specify tables as if they are using schema version 2. If they are in fact using schema version 2, database interactions proceed as expected. If they are using schema version 1, however, calls into the *block_data*, *metadata*, or *deleted_blocks* tables will be redirected to the one *default* table. This comes with a few issues that had to be resolved: 1. Since deleted blocks are in their own table in schema version 2, they do not need a prefix to separate them from regular blocks. - In schema version 1, however, the prefix is still necessary, otherwise block ID keys for deleted and regular blocks would have identical format and indistinguishable at the default table level. - This is solved with the `SchemaOneDeletedBlocksTable` class, which is returned as the `Table` implementation when callers ask for the deleted blocks table but are using schema version 1. - This class automatically adds the `#deleted#` prefix to caller data, so callers can read and write as if their data is actually in a separate table. - This class will not preprocess and modify iterator results, however, so scanning the deleted blocks table will return the keys with the `#deleted#` prefix. - This is documented above the appropriate methods, and the current code never reads iterator values, only their size. - The definition of the `#deleted#` prefix was moved out of the `OzoneConsts` and `MetadataKeyFilters` classes and into `SchemaOneDeletedBlocksTable`, since this field is now specific only to this case. 2. Databases that had blocks deleted from them before this pull request will have the deleted block ID key mapped to the block ID value, instead of having the chunk information for that block saved as the value. - This is solved by the `SchemaOneChunkInfoListCodec` class, which will chain the default exception thrown by a trying to read a block ID as a `ChunkInfoList` to another `IOException` with a more detailed error message explaining why the chunk information may be invalid. - Callers were already forced to handle checked IOExceptions from this method according to the `Codec` interface, so there should be no surprise behavior if this occurs. 3. Calls to `Table#iterator` will return an iterator over the whole underlying table structure. - On schema version 1, this means calls like `blockDataTable.iterator()` will iterate all data in the default table, not just that pertaining to block information. - Without an explicit list of key formats defined for each table (which is fragile if prefixes or metadata are added/changed), this method cannot function properly. - Since this method is not used by datanode code, it was left unaltered. - An alternate design choice would be to always return a table implementation that throws `UnsupportedOperationException` for this call. ### Guide to File Changes - The following *new* files were created to supply the datanode with necessary functionality: - For defining the database layout: - AbstractDatanodeDBDefinition.java - DatanodeSchemaOneDBDefinition.java - DatanodeSchemaTwoDBDefinition.java - For interacting the with the database: - DatanodeStore.java - AbstractDatanodeStore.java - DatanodeStoreSchemaOneImpl.java - DatanodeStoreSchemaTwoImpl.java - For encoding/decoding byte streams to/from the database: - ChunkInfoList.java - ChunkInfoListCodec.java - BlockDataCodec.java - Wrappers for backwards compatibility: - SchemaOneDeletedBlocksTable.java - SchemaOneChunkInfoListCodec.java - Unit testing: - TestSchemaOneBackwardsCompatibility.java - Additionally, a RocksDB database and associated .container file were created using the original code to test backwards compatibility. - These files are internal to RocksDB and can be ignored, even though they show up in the diff. - Their contents are documented in `TestSchemaOneBackwardsCompatibility#TestDB`. - Most other changes are switching existing code over to the new interface, or adding equivalent functionality that was present in the old interface to the new interface. ## What is the link to the Apache JIRA HDDS-3869 ## How was this patch tested? - To ensure equivalent functionality between the old and new interfaces and their implementations, the code was run through the existing unit test framework. - Since this creates all containers from scratch, these tests were all run on using databases in schema version 2. - To ensure backwards compatibility, a new set of tests was created in `TestSchemaOneBackwardsCompatibility.java.` - These tests use a RocksDB database and container file that were generated using the original code before this pull request. - This tests reading all data in the database, and writing everything except new block data. - Since all containers will be closed before upgrade, no new blocks will be added to containers encountered with schema version 1. - In `TestBlockDeletingService#testDeletedChunkInfo` was added to verify that when using a database in schema version 2, chunk information from deleted blocks is saved correctly. - In `TestKeyValueBlockIterator`, use of the `#deleted#` blocks prefix was replaced with the `#BCSID#` prefix, since deleted blocks were moved to their own column family outside of the block data table. - The specific prefixes used in these tests is arbitrary, as long as they are two different prefixes present in the block data table. - The new unit test `testKeyValueBlockIteratorWithAdvancedFilter` was also added to this class. - This makes sure that the iterator still functions properly when keys matching a given filter may occur in the middle of the key set. - This condition was not guaranteed to occur in any of the existing unit tests, but it could happen when using the schema version 2 layout. ## Notes Leaving as draft until HDDS-4061 is merged, since this pull request incorporates changes from that issue. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org