minxhe opened a new pull request, #29572: URL: https://github.com/apache/flink/pull/29572
## What is the purpose of the change When `RocksDBKeyedStateBackend` restores a checkpoint produced by the asynchronous ForSt state backend (`ForStKeyedStateBackend`, which snapshots state metadata with `BackendStateType.KEY_VALUE_V2`), `RocksDBHandle`, `RestoredDBInstance`, and `DistributeStateHandlerHelper` instantiate `org.apache.flink.runtime.state.v2.RegisteredKeyValueStateBackendMetaInfo` (or `RegisteredKeyAndUserKeyValueStateBackendMetaInfo`) via `RegisteredStateMetaInfoBase.fromMetaInfoSnapshot`. Subsequent state registration in `RocksDBKeyedStateBackend#tryRegisterKvStateInformation` casts `oldStateInfo.metaInfo` to V1 `org.apache.flink.runtime.state.RegisteredKeyValueStateBackendMetaInfo` and fails with a `ClassCastException`, while `getKeys` and `getKeysAndNamespaces` skip unaccessed `KEY_VALUE_V2` states. `ForStIncrementalRestoreOperation` already calls `RegisteredKeyValueStateBackendMetaInfo.fromMetaInfoSnapshot`, which converts both `KEY_VALUE` and `KEY_VALUE_V2` snapshots into V1 `RegisteredKeyValueStateBackendMetaInfo` instances (converting V2 `MAP` state's `UserKeySerializer` and value `StateSerializer` into a V1 `MapSerializer`). This pull request aligns RocksDB's three restore sites with `ForStIncrementalRestoreOperation`. Jira: https://issues.apache.org/jira/browse/FLINK-41092 ## Brief change log - Use `RegisteredKeyValueStateBackendMetaInfo.fromMetaInfoSnapshot` instead of `RegisteredStateMetaInfoBase.fromMetaInfoSnapshot` in `RocksDBHandle#getOrRegisterStateColumnFamilyHandle`, `RestoredDBInstance#createColumnFamilyDescriptors`, and `DistributeStateHandlerHelper`. - Add `RocksDBRestoreKeyValueV2MetadataTest` covering `KEY_VALUE_V2` metadata for `ValueState`, `ListState`, `MapState`, and unaccessed `getKeys` across single-handle (`RocksDBHandle`), temporary-DB rescale (`RestoredDBInstance`), and ingest-DB rescale (`DistributeStateHandlerHelper`) restore modes. ## Verifying this change This change added tests and can be verified as follows: - On unmodified `master` (`8d01380ff5bf9892462ef9808b1b56fd9723f54d`), all 3 parameterized restore modes in `RocksDBRestoreKeyValueV2MetadataTest` fail with `ClassCastException: class org.apache.flink.runtime.state.v2.RegisteredKeyAndUserKeyValueStateBackendMetaInfo cannot be cast to class org.apache.flink.runtime.state.RegisteredKeyValueStateBackendMetaInfo` (and `getKeys` returns empty on unaccessed `KEY_VALUE_V2` state). - With the fix, all 8 tests across `RocksDBRestoreKeyValueV2MetadataTest`, `RocksIncrementalCheckpointRescalingTest`, and `DistributeStateHandlerHelperTest` pass. ```sh ./mvnw -B -ntp -pl flink-state-backends/flink-statebackend-rocksdb -Dflink.forkCountUnitTest=1 \ -Dtest=RocksDBRestoreKeyValueV2MetadataTest,RocksIncrementalCheckpointRescalingTest,DistributeStateHandlerHelperTest test # 8 run, 0 failures ./mvnw -B -ntp -pl flink-state-backends/flink-statebackend-rocksdb \ spotless:check checkstyle:check # 0 violations ``` ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): no - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no - The serializers: no - The runtime per-record code paths (performance sensitive): no - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: yes (RocksDB restore of `KEY_VALUE_V2` state metadata) - The S3 file system connector: no ## Documentation - Does this pull request introduce a new feature? no - If yes, how is the feature documented? not applicable --- ##### Was generative AI tooling used to co-author this PR? - [X] Yes (please specify the tool below) Generated-by: GitHub Copilot CLI 1.0.99-2 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
