[
https://issues.apache.org/jira/browse/HDDS-17059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Devesh Kumar Singh updated HDDS-17059:
--------------------------------------
Description:
Introduce a Jobworker service that allows OM to dispatch background commands to
registered workers and receive execution results. This provides reusable
infrastructure for operations such as key migration without requiring OM to
perform the underlying data processing.
The framework should be independent of storage-policy definitions, lifecycle
rules, and policy-aware key rewriting, allowing it to be developed against
master.
*Scope:*
- Add Jobworker service startup, shutdown, configuration, packaging, and OM
connectivity.
- Introduce the shared registration, heartbeat, command, and result-reporting
protocols.
- Track registered workers, their health, and their availability.
- Deliver queued commands through heartbeat responses.
- Provide extensible command-handler registration and asynchronous execution.
- Track command status, execution results, and reporting timeouts.
- Integrate with OM leadership changes and worker re-registration.
- Provide metrics and diagnostics for worker health and command processing.
Migration-specific commands, task tables, scheduling, and storage-policy
behavior belong to the dependent JIRA.
*Acceptance criteria:*
- A worker can register with OM and maintain its health through heartbeats.
- OM can dispatch a test command, and the worker can execute it and report its
result.
- Worker restart, heartbeat loss, execution failure, duplicate reports, and
command timeouts have defined and tested behavior.
- Leadership changes prevent stale dispatch and allow workers to reconnect to
the active OM.
- Shutdown releases connections, threads, and other resources.
- Tests exercise the complete registration → dispatch → execution → result flow
without storage-policy dependencies.
was:
Updating a bucket’s storage policy changes the desired placement for applicable
writes, but existing keys remain on their current storage volumes. For example,
changing a bucket from HOT to COLD does not move existing RATIS THREE keys from
SSD to ARCHIVE.
Introduce asynchronous key migration using Jobworker to rewrite existing keys
onto the target storage tier while preserving their replication configuration.
Bucket-policy updates through CLI or native APIs and migration submission are
separate operations; updating the policy alone must not be represented as
completed data migration.
*Acceptance criteria:*
- A RATIS THREE key can migrate from SSD to ARCHIVE while retaining its name,
content, replication configuration, and required attributes.
- Placement respects the target policy and configured fallback rules.
- Failed or incomplete copies cannot replace the committed source key.
- Concurrent overwrites or deletions cannot be overwritten by stale migration
work.
- Tasks recover across OM failover and Jobworker restart without losing work or
double-counting results.
- Progress, failures, retries, and cancellation outcomes are observable.
- Integration tests verify actual DN placement, data correctness, failure
recovery, and unchanged replication.
The migration flow should:
1. Persist migration tasks and batches of key names, observed generations, and
target storage policies in OM.
2. Allow lifecycle processing to create migration work.
3. Use StoragePolicySatisfierService to schedule tasks, select healthy
Jobworkers, and dispatch commands through heartbeat responses.
4. Have Jobworker read each source key and invoke policy-aware rewriteKey,
retaining the source replication configuration.
5. Reuse the OM → SCM → datanode write path to allocate and write target-tier
blocks.
6. Validate the source generation before committing the replacement, protecting
concurrent user updates.
7. Report per-key outcomes, retry eligible failures, and persist task progress
through OM Ratis.
> Introduce a generic Jobworker framework for OM-managed background jobs.
> -----------------------------------------------------------------------
>
> Key: HDDS-17059
> URL: https://issues.apache.org/jira/browse/HDDS-17059
> Project: Apache Ozone
> Issue Type: Sub-task
> Components: Ozone Datanode, Ozone Filesystem, Ozone Manager
> Reporter: Devesh Kumar Singh
> Assignee: Devesh Kumar Singh
> Priority: Major
>
> Introduce a Jobworker service that allows OM to dispatch background commands
> to registered workers and receive execution results. This provides reusable
> infrastructure for operations such as key migration without requiring OM to
> perform the underlying data processing.
> The framework should be independent of storage-policy definitions, lifecycle
> rules, and policy-aware key rewriting, allowing it to be developed against
> master.
> *Scope:*
> - Add Jobworker service startup, shutdown, configuration, packaging, and OM
> connectivity.
> - Introduce the shared registration, heartbeat, command, and result-reporting
> protocols.
> - Track registered workers, their health, and their availability.
> - Deliver queued commands through heartbeat responses.
> - Provide extensible command-handler registration and asynchronous execution.
> - Track command status, execution results, and reporting timeouts.
> - Integrate with OM leadership changes and worker re-registration.
> - Provide metrics and diagnostics for worker health and command processing.
> Migration-specific commands, task tables, scheduling, and storage-policy
> behavior belong to the dependent JIRA.
> *Acceptance criteria:*
> - A worker can register with OM and maintain its health through heartbeats.
> - OM can dispatch a test command, and the worker can execute it and report
> its result.
> - Worker restart, heartbeat loss, execution failure, duplicate reports, and
> command timeouts have defined and tested behavior.
> - Leadership changes prevent stale dispatch and allow workers to reconnect to
> the active OM.
> - Shutdown releases connections, threads, and other resources.
> - Tests exercise the complete registration → dispatch → execution → result
> flow without storage-policy dependencies.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
