amaliujia commented on code in PR #10503: URL: https://github.com/apache/ozone/pull/10503#discussion_r3450098619
########## hadoop-hdds/docs/content/design/leader-execution/components.md: ########## @@ -0,0 +1,257 @@ +--- +title: Leader-Planned Execution — Components +summary: ManagedIndex, LeaderPlanner, ChangeRecorder, ApplyEngine, and State Machine integration +date: 2026-06-12 +jira: HDDS-11898 +status: proposed +author: Abhishek Pal +--- +<!-- + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. See accompanying LICENSE file. +--> + +# Components + +This document describes the core components that implement leader-planned execution. For the high-level architecture and motivation, see the [overview](./overview.md). + +## Table of Contents + +1. [Managed Index Service](#1-managed-index-service) +2. [Leader Planning Framework](#2-leader-planning-framework) +3. [Apply Engine](#3-apply-engine) +4. [State Machine Integration](#4-state-machine-integration) + +--- + +## 1. Managed Index Service + +**Problem:** With the current architecture, `objectID` for new keys/volumes comes from the Ratis `transactionLogIndex`. In the new model, planning happens in `startTransaction()` (before the log entry is written), so the Ratis index is not yet available. + +**Why must objectId be decided during planning?** + +The `objectID` is part of the value that gets serialized into the DB delta. For example, when we create a new key: +1. We build an `OmKeyInfo` object — this object contains the `objectID` field +2. We serialize this `OmKeyInfo` to raw bytes via the codec +3. These raw bytes go into an `Operation` message in the `Batch` proto + +The raw bytes are frozen at planning time. They get written as-is to RocksDB during apply. We cannot "fill in" the objectId later during apply because the bytes are already serialized and embedded in the Ratis log entry. + +In the current architecture, this is not a problem: `validateAndUpdateCache` runs during `applyTransaction` where the Ratis log index is already known. So it uses `transactionLogIndex` directly as the objectId. + +In the new architecture, planning runs in `startTransaction` which happens **before** Ratis assigns a log index. So we need our own counter. + +**Solution:** An OM-managed `AtomicLong` counter (`ManagedIndexService`) that: +- Returns a unique, always-increasing value via `getAndIncrement()`. +- Is saved together with each DB batch (stored in `TransactionInfoTable` under key `#MANAGED_INDEX`). +- Survives leader switchover: `onBecomeLeader(lastCommitted)` calls `max(currentIndex, lastCommittedIndex)`. Review Comment: Following up our conversion: I am seeing a potential issue of monotonic increasing OM Object ID allocation here. Because we have decoupled the OM Object ID allocation (in StartTransaction) and OM Object ID persistence, there is a gap in between where an ID is allocated, not yet reach RockDB, and the OM leader crashes. On concrete example I am thinking is: 1. Old OM leader allocates 1, 2, 3, 4, 5. 2. The DB deltas with ID 1, 2, 3, 4, 5 have been appended to Ratis logs. 3. Only DB deltas with ID 1,2 are applied to DB. Then assuming old OM leader crashes and new OM leader wins, and assuming last committed Index is 4, then new OM leader does a max(currentIndex, lastCommittedIndex) which is max(2, 4) which becomes 4. So OM becomes the next Object ID is 5, however 5 is already been used and in the Ratis log, thus cause a conflict. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
