smengcl commented on code in PR #593: URL: https://github.com/apache/ozone-site/pull/593#discussion_r4430560635
########## src/pages/release-notes/2.2.2.md: ########## @@ -8,6 +8,95 @@ Apache Ozone 2.2.2 is a maintenance release on top of Ozone 2.2.1 with bug fixes and dependency updates. +## Upgrade from 2.2.1 + +Ozone 2.2.2 introduces no new OM or HDDS layout features and does not change +the layout version relative to 2.2.1. No layout finalization is required solely +for this patch upgrade. If your cluster already has +an unfinished upgrade from an earlier release, assess that upgrade separately. +Follow the non-rolling procedure in the +[upgrade and downgrade guide](https://ozone.apache.org/docs/administrator-guide/operations/upgrade-and-downgrade/), +with the release-specific checks below. + +### Before stopping the cluster + +1. Verify cluster health. Back up the current configuration, service keytab + files, certificates, and metadata. Preserve the existing + OM, SCM, Datanode, and Recon storage directories during the upgrade. Retain + the 2.2.1 binaries for recovery. +2. On secure clusters, review the following principal properties in the + configuration distributed to **every SCM**, including SCM instances that + issue or renew certificates. Values must match the service principals used + by the corresponding daemons. Copying these principal settings does not + require copying the other daemons' private keys or keytab files to SCM. + + | Property on SCM | Used for | + | --- | --- | + | `ozone.om.kerberos.principal` | OM certificate requests and block allocation/deletion | + | `hdds.datanode.kerberos.principal` | Datanode certificate requests | + | `ozone.recon.kerberos.principal` | Recon certificate requests | + | `hdds.scm.kerberos.principal` | SCM certificate requests | + + SCM now checks the requesting service identity. Matching considers the + service name, whether a host component is present, and the realm when one + is configured; individual hostnames are not compared. Proxy identities + do not qualify as service identities. A mismatch can prevent certificate + issuance or renewal. Incorrect OM principal configuration can also prevent + new block allocation or deletion when admin authorization is enabled. + Correct the principal configuration rather than granting service accounts + broad administrator access as a workaround. +3. If security and gRPC TLS are enabled, review the Recon known issue and + conditional workaround below **before restarting with 2.2.2**. Preserve a + usable Recon database. If you require a full SCM checkpoint download, defer + the upgrade until a release or vendor-supported build fixes this issue. + +### Known issue: Recon SCM checkpoint download with gRPC TLS + +[HDDS-17235](https://issues.apache.org/jira/browse/HDDS-17235) requires SCM peer +certificates on SCM Ratis and inter-SCM gRPC connections. Recon downloads a +full SCM database checkpoint through the same inter-SCM gRPC service, but +presents a Recon leaf certificate. With both `ozone.security.enabled=true` +and `hdds.grpc.tls.enabled=true`, SCM 2.2.2 rejects that certificate. + +This affects full SCM checkpoint downloads from both Recon 2.2.1 and Recon +2.2.2 to SCM 2.2.2. Logs may contain `Not an SCM certificate` or +`Rocks DB checkpoint downloading failed`; Recon may report +`GET_SCM_DB_SNAPSHOT_FAILED`. Review Comment: nit: `Not an SCM certificate` is logged by SCM (TLS handshake), while the other two come from Recon. Saying which log to look in might save some searching. ```suggestion 2.2.2 to SCM 2.2.2. The SCM log may contain `Not an SCM certificate`, and the Recon log `Rocks DB checkpoint downloading failed`; Recon may report `GET_SCM_DB_SNAPSHOT_FAILED`. ``` ########## src/pages/release-notes/2.2.2.md: ########## @@ -8,6 +8,95 @@ Apache Ozone 2.2.2 is a maintenance release on top of Ozone 2.2.1 with bug fixes and dependency updates. +## Upgrade from 2.2.1 + +Ozone 2.2.2 introduces no new OM or HDDS layout features and does not change +the layout version relative to 2.2.1. No layout finalization is required solely +for this patch upgrade. If your cluster already has +an unfinished upgrade from an earlier release, assess that upgrade separately. +Follow the non-rolling procedure in the +[upgrade and downgrade guide](https://ozone.apache.org/docs/administrator-guide/operations/upgrade-and-downgrade/), +with the release-specific checks below. + +### Before stopping the cluster + +1. Verify cluster health. Back up the current configuration, service keytab + files, certificates, and metadata. Preserve the existing + OM, SCM, Datanode, and Recon storage directories during the upgrade. Retain + the 2.2.1 binaries for recovery. +2. On secure clusters, review the following principal properties in the + configuration distributed to **every SCM**, including SCM instances that + issue or renew certificates. Values must match the service principals used + by the corresponding daemons. Copying these principal settings does not + require copying the other daemons' private keys or keytab files to SCM. + + | Property on SCM | Used for | + | --- | --- | + | `ozone.om.kerberos.principal` | OM certificate requests and block allocation/deletion | + | `hdds.datanode.kerberos.principal` | Datanode certificate requests | + | `ozone.recon.kerberos.principal` | Recon certificate requests | + | `hdds.scm.kerberos.principal` | SCM certificate requests | + + SCM now checks the requesting service identity. Matching considers the + service name, whether a host component is present, and the realm when one + is configured; individual hostnames are not compared. Proxy identities + do not qualify as service identities. A mismatch can prevent certificate + issuance or renewal. Incorrect OM principal configuration can also prevent + new block allocation or deletion when admin authorization is enabled. + Correct the principal configuration rather than granting service accounts + broad administrator access as a workaround. +3. If security and gRPC TLS are enabled, review the Recon known issue and + conditional workaround below **before restarting with 2.2.2**. Preserve a + usable Recon database. If you require a full SCM checkpoint download, defer + the upgrade until a release or vendor-supported build fixes this issue. + +### Known issue: Recon SCM checkpoint download with gRPC TLS + +[HDDS-17235](https://issues.apache.org/jira/browse/HDDS-17235) requires SCM peer +certificates on SCM Ratis and inter-SCM gRPC connections. Recon downloads a +full SCM database checkpoint through the same inter-SCM gRPC service, but +presents a Recon leaf certificate. With both `ozone.security.enabled=true` +and `hdds.grpc.tls.enabled=true`, SCM 2.2.2 rejects that certificate. + +This affects full SCM checkpoint downloads from both Recon 2.2.1 and Recon +2.2.2 to SCM 2.2.2. Logs may contain `Not an SCM certificate` or +`Rocks DB checkpoint downloading failed`; Recon may report +`GET_SCM_DB_SNAPSHOT_FAILED`. + +Disabling `ozone.recon.scm.snapshot.enabled` is **not required for every +2.2.2 deployment**. This issue applies only when security and gRPC TLS are +both enabled. With this setting enabled, Recon attempts a full checkpoint +download at startup only when the difference between its container count and +SCM's exceeds the configured threshold. An affected download fails; a restart +that does not need a full checkpoint does not encounter this failure. Review Comment: One detail changes the picture quite a bit: `ozone.recon.scm.container.threshold` defaults to `1000000` (`ozone-default.xml`). With defaults, Recon only attempts the startup download when its container count is more than a million off from SCM's. The only other trigger is an explicit `POST /api/v1/triggerdbsync/scm/snapshot`; the Recon UI only calls `/api/v1/triggerdbsync/om`. Could we state the default and narrow "affected" accordingly? ```suggestion Disabling `ozone.recon.scm.snapshot.enabled` is **not required for most 2.2.2 deployments**. This issue applies only when security and gRPC TLS are both enabled, and only to a full checkpoint download. Recon attempts one only: - at startup, when `ozone.recon.scm.snapshot.enabled` is true (the default) and its container count differs from SCM's by more than `ozone.recon.scm.container.threshold` (default `1000000`), or - when someone calls `POST /api/v1/triggerdbsync/scm/snapshot`. With the default threshold, most clusters never attempt this download. A restart that does not need a full checkpoint does not encounter this failure. ``` ########## src/pages/release-notes/2.2.2.md: ########## @@ -8,6 +8,95 @@ Apache Ozone 2.2.2 is a maintenance release on top of Ozone 2.2.1 with bug fixes and dependency updates. +## Upgrade from 2.2.1 + +Ozone 2.2.2 introduces no new OM or HDDS layout features and does not change +the layout version relative to 2.2.1. No layout finalization is required solely +for this patch upgrade. If your cluster already has +an unfinished upgrade from an earlier release, assess that upgrade separately. +Follow the non-rolling procedure in the +[upgrade and downgrade guide](https://ozone.apache.org/docs/administrator-guide/operations/upgrade-and-downgrade/), +with the release-specific checks below. + +### Before stopping the cluster + +1. Verify cluster health. Back up the current configuration, service keytab + files, certificates, and metadata. Preserve the existing + OM, SCM, Datanode, and Recon storage directories during the upgrade. Retain + the 2.2.1 binaries for recovery. +2. On secure clusters, review the following principal properties in the + configuration distributed to **every SCM**, including SCM instances that + issue or renew certificates. Values must match the service principals used + by the corresponding daemons. Copying these principal settings does not + require copying the other daemons' private keys or keytab files to SCM. + + | Property on SCM | Used for | + | --- | --- | + | `ozone.om.kerberos.principal` | OM certificate requests and block allocation/deletion | + | `hdds.datanode.kerberos.principal` | Datanode certificate requests | + | `ozone.recon.kerberos.principal` | Recon certificate requests | + | `hdds.scm.kerberos.principal` | SCM certificate requests | + + SCM now checks the requesting service identity. Matching considers the + service name, whether a host component is present, and the realm when one + is configured; individual hostnames are not compared. Proxy identities + do not qualify as service identities. A mismatch can prevent certificate + issuance or renewal. Incorrect OM principal configuration can also prevent + new block allocation or deletion when admin authorization is enabled. Review Comment: I think this one deserves stronger wording. If the OM principal doesn't match on SCM and the OM user isn't an SCM admin, `SCMBlockProtocolServer` rejects every `allocateBlock` and `deleteKeyBlocks` from OM while `ozone.authorization.enabled` is true, which is the default. In practice all key writes fail and deleted keys stop freeing space right after the upgrade. The certificate side is the opposite: existing certificates keep working, so a mismatch may only show up at renewal or when a node is added. A short post upgrade check could catch both early. ```suggestion do not qualify as service identities. A mismatch prevents certificate issuance or renewal for that role. Existing certificates keep working, so this may only appear at renewal or when adding a node. If the OM principal does not match and the OM user is not an SCM administrator, SCM rejects all block allocation and deletion from OM while `ozone.authorization.enabled` is true (the default): key writes fail and deleted keys stop freeing space. After the upgrade, write and delete a test key, and check the SCM logs for `can allocate or delete blocks` or `is not allowed to request a`. ``` ########## src/pages/release-notes/2.2.2.md: ########## @@ -8,6 +8,95 @@ Apache Ozone 2.2.2 is a maintenance release on top of Ozone 2.2.1 with bug fixes and dependency updates. +## Upgrade from 2.2.1 + +Ozone 2.2.2 introduces no new OM or HDDS layout features and does not change +the layout version relative to 2.2.1. No layout finalization is required solely +for this patch upgrade. If your cluster already has +an unfinished upgrade from an earlier release, assess that upgrade separately. +Follow the non-rolling procedure in the +[upgrade and downgrade guide](https://ozone.apache.org/docs/administrator-guide/operations/upgrade-and-downgrade/), +with the release-specific checks below. + +### Before stopping the cluster + +1. Verify cluster health. Back up the current configuration, service keytab + files, certificates, and metadata. Preserve the existing + OM, SCM, Datanode, and Recon storage directories during the upgrade. Retain + the 2.2.1 binaries for recovery. +2. On secure clusters, review the following principal properties in the + configuration distributed to **every SCM**, including SCM instances that + issue or renew certificates. Values must match the service principals used + by the corresponding daemons. Copying these principal settings does not + require copying the other daemons' private keys or keytab files to SCM. + + | Property on SCM | Used for | + | --- | --- | + | `ozone.om.kerberos.principal` | OM certificate requests and block allocation/deletion | + | `hdds.datanode.kerberos.principal` | Datanode certificate requests | + | `ozone.recon.kerberos.principal` | Recon certificate requests | + | `hdds.scm.kerberos.principal` | SCM certificate requests | + + SCM now checks the requesting service identity. Matching considers the + service name, whether a host component is present, and the realm when one + is configured; individual hostnames are not compared. Proxy identities + do not qualify as service identities. A mismatch can prevent certificate + issuance or renewal. Incorrect OM principal configuration can also prevent + new block allocation or deletion when admin authorization is enabled. + Correct the principal configuration rather than granting service accounts + broad administrator access as a workaround. +3. If security and gRPC TLS are enabled, review the Recon known issue and + conditional workaround below **before restarting with 2.2.2**. Preserve a + usable Recon database. If you require a full SCM checkpoint download, defer + the upgrade until a release or vendor-supported build fixes this issue. Review Comment: Two other 2.2.2 changes that operators may run into right after this upgrade, and that the changelog doesn't mention: S3 Gateway now rejects SigV4 requests signed with an `Authorization` header whose `X-Amz-Date` is more than 15 minutes off its clock (2.2.1 accepted a credential date within one day), and with security and gRPC TLS enabled, OM now fails to start if it can't set up TLS for the S3 gRPC endpoint (2.2.1 served it without TLS). A few operations also need an admin now. The suggestion below adds a short "Behavior changes" list for these, and narrows item 3 to the deployments the Recon issue can actually reach (see the threshold comment further down). ```suggestion 3. If security and gRPC TLS are enabled, and you lowered `ozone.recon.scm.container.threshold` or trigger Recon's SCM DB sync through its API, review the Recon known issue and conditional workaround below **before restarting with 2.2.2**. 4. Keep the clocks of S3 clients and S3 Gateway hosts synchronized. S3 Gateway 2.2.2 rejects requests signed with an `Authorization` header whose `X-Amz-Date` is more than 15 minutes off its clock (`RequestTimeTooSkewed`, HTTP 403). 2.2.1 accepted a credential date within one day. ### Behavior changes - With security and gRPC TLS enabled, OM fails to start if it cannot set up TLS for the S3 gRPC endpoint, instead of serving the endpoint without TLS. - On secure clusters, OM rejects S3 Gateway gRPC requests that carry no S3 authentication. `ozone.om.s3.grpc.auth.required` (default `true`) controls this. - Forcing SCM out of safe mode through the Ozone file system `SafeMode` API and the OM bootstrap request require an OM administrator. When authorization is enabled, downloading the SCM DB checkpoint over HTTP (`/dbCheckpoint`) requires an SCM administrator. ``` ########## src/pages/release-notes/2.2.2.md: ########## @@ -8,6 +8,95 @@ Apache Ozone 2.2.2 is a maintenance release on top of Ozone 2.2.1 with bug fixes and dependency updates. +## Upgrade from 2.2.1 + +Ozone 2.2.2 introduces no new OM or HDDS layout features and does not change +the layout version relative to 2.2.1. No layout finalization is required solely +for this patch upgrade. If your cluster already has +an unfinished upgrade from an earlier release, assess that upgrade separately. +Follow the non-rolling procedure in the +[upgrade and downgrade guide](https://ozone.apache.org/docs/administrator-guide/operations/upgrade-and-downgrade/), +with the release-specific checks below. + +### Before stopping the cluster + +1. Verify cluster health. Back up the current configuration, service keytab + files, certificates, and metadata. Preserve the existing + OM, SCM, Datanode, and Recon storage directories during the upgrade. Retain + the 2.2.1 binaries for recovery. +2. On secure clusters, review the following principal properties in the + configuration distributed to **every SCM**, including SCM instances that + issue or renew certificates. Values must match the service principals used + by the corresponding daemons. Copying these principal settings does not + require copying the other daemons' private keys or keytab files to SCM. + + | Property on SCM | Used for | + | --- | --- | + | `ozone.om.kerberos.principal` | OM certificate requests and block allocation/deletion | + | `hdds.datanode.kerberos.principal` | Datanode certificate requests | + | `ozone.recon.kerberos.principal` | Recon certificate requests | + | `hdds.scm.kerberos.principal` | SCM certificate requests | + + SCM now checks the requesting service identity. Matching considers the + service name, whether a host component is present, and the realm when one + is configured; individual hostnames are not compared. Proxy identities + do not qualify as service identities. A mismatch can prevent certificate + issuance or renewal. Incorrect OM principal configuration can also prevent + new block allocation or deletion when admin authorization is enabled. + Correct the principal configuration rather than granting service accounts + broad administrator access as a workaround. +3. If security and gRPC TLS are enabled, review the Recon known issue and + conditional workaround below **before restarting with 2.2.2**. Preserve a + usable Recon database. If you require a full SCM checkpoint download, defer + the upgrade until a release or vendor-supported build fixes this issue. + +### Known issue: Recon SCM checkpoint download with gRPC TLS + +[HDDS-17235](https://issues.apache.org/jira/browse/HDDS-17235) requires SCM peer +certificates on SCM Ratis and inter-SCM gRPC connections. Recon downloads a +full SCM database checkpoint through the same inter-SCM gRPC service, but +presents a Recon leaf certificate. With both `ozone.security.enabled=true` +and `hdds.grpc.tls.enabled=true`, SCM 2.2.2 rejects that certificate. + +This affects full SCM checkpoint downloads from both Recon 2.2.1 and Recon +2.2.2 to SCM 2.2.2. Logs may contain `Not an SCM certificate` or +`Rocks DB checkpoint downloading failed`; Recon may report +`GET_SCM_DB_SNAPSHOT_FAILED`. + +Disabling `ozone.recon.scm.snapshot.enabled` is **not required for every +2.2.2 deployment**. This issue applies only when security and gRPC TLS are +both enabled. With this setting enabled, Recon attempts a full checkpoint +download at startup only when the difference between its container count and +SCM's exceeds the configured threshold. An affected download fails; a restart +that does not need a full checkpoint does not encounter this failure. + +For an existing Recon instance with a usable database, you can avoid that +startup download by setting the following in **Recon's** `ozone-site.xml` +before restarting Recon: + +```xml +<property> + <name>ozone.recon.scm.snapshot.enabled</name> + <value>false</value> +</property> +``` + +This is a limited workaround: it skips startup SCM snapshot initialization, +while periodic container reconciliation continues to run. It does not repair +full checkpoint downloads or prevent manual full snapshot requests from being +attempted. Preserve Recon's database and avoid requesting a full SCM snapshot +refresh. The workaround does not provide a way to initialize a new Recon +instance or rebuild a missing or unusable database through a full checkpoint. Review Comment: Given the default threshold, a new Recon on a cluster with fewer than a million containers doesn't attempt a full checkpoint at all: `initializeSCMDB()` falls through to `initializePipelinesFromScm()`, and the periodic container sync fills in from SCM. So this sentence reads more alarming than it needs to for most clusters. I followed the code path but haven't run it end to end, so it may be worth a quick check before taking the second sentence. ```suggestion refresh. The workaround does not provide a way to rebuild Recon's database through a full checkpoint. With the default threshold, a new Recon on a cluster with fewer than 1,000,000 containers does not attempt one; it relies on periodic container reconciliation with SCM instead. ``` ########## src/pages/release-notes/2.2.2.md: ########## @@ -8,6 +8,95 @@ Apache Ozone 2.2.2 is a maintenance release on top of Ozone 2.2.1 with bug fixes and dependency updates. +## Upgrade from 2.2.1 + +Ozone 2.2.2 introduces no new OM or HDDS layout features and does not change +the layout version relative to 2.2.1. No layout finalization is required solely +for this patch upgrade. If your cluster already has +an unfinished upgrade from an earlier release, assess that upgrade separately. +Follow the non-rolling procedure in the +[upgrade and downgrade guide](https://ozone.apache.org/docs/administrator-guide/operations/upgrade-and-downgrade/), +with the release-specific checks below. + +### Before stopping the cluster + +1. Verify cluster health. Back up the current configuration, service keytab + files, certificates, and metadata. Preserve the existing + OM, SCM, Datanode, and Recon storage directories during the upgrade. Retain + the 2.2.1 binaries for recovery. +2. On secure clusters, review the following principal properties in the + configuration distributed to **every SCM**, including SCM instances that + issue or renew certificates. Values must match the service principals used + by the corresponding daemons. Copying these principal settings does not + require copying the other daemons' private keys or keytab files to SCM. + + | Property on SCM | Used for | + | --- | --- | + | `ozone.om.kerberos.principal` | OM certificate requests and block allocation/deletion | + | `hdds.datanode.kerberos.principal` | Datanode certificate requests | + | `ozone.recon.kerberos.principal` | Recon certificate requests | + | `hdds.scm.kerberos.principal` | SCM certificate requests | Review Comment: Would "set explicitly" fit better than "review" here? When one of these keys is missing from SCM's configuration, SCM falls back to the `ozone-default.xml` placeholders, and none of them can match a real principal. A "Default if unset" column might make that obvious at a glance. ```suggestion 2. On secure clusters, set the following principal properties explicitly in the configuration distributed to **every SCM**, including SCM instances that issue or renew certificates. Values must match the service principals used by the corresponding daemons. If a property is unset, SCM uses its default, a placeholder that does not match any real principal. Copying these principal settings does not require copying the other daemons' private keys or keytab files to SCM. | Property on SCM | Default if unset | Used for | | --- | --- | --- | | `ozone.om.kerberos.principal` | `OM/_HOST@REALM` | OM certificate requests and block allocation/deletion | | `hdds.datanode.kerberos.principal` | `dn/_HOST@REALM` | Datanode certificate requests | | `ozone.recon.kerberos.principal` | (empty) | Recon certificate requests | | `hdds.scm.kerberos.principal` | `SCM/_HOST@REALM` | SCM certificate requests | ``` ########## src/pages/release-notes/2.2.2.md: ########## @@ -8,6 +8,95 @@ Apache Ozone 2.2.2 is a maintenance release on top of Ozone 2.2.1 with bug fixes and dependency updates. +## Upgrade from 2.2.1 + +Ozone 2.2.2 introduces no new OM or HDDS layout features and does not change +the layout version relative to 2.2.1. No layout finalization is required solely +for this patch upgrade. If your cluster already has +an unfinished upgrade from an earlier release, assess that upgrade separately. +Follow the non-rolling procedure in the +[upgrade and downgrade guide](https://ozone.apache.org/docs/administrator-guide/operations/upgrade-and-downgrade/), +with the release-specific checks below. + +### Before stopping the cluster + +1. Verify cluster health. Back up the current configuration, service keytab + files, certificates, and metadata. Preserve the existing + OM, SCM, Datanode, and Recon storage directories during the upgrade. Retain + the 2.2.1 binaries for recovery. +2. On secure clusters, review the following principal properties in the + configuration distributed to **every SCM**, including SCM instances that + issue or renew certificates. Values must match the service principals used + by the corresponding daemons. Copying these principal settings does not + require copying the other daemons' private keys or keytab files to SCM. + + | Property on SCM | Used for | + | --- | --- | + | `ozone.om.kerberos.principal` | OM certificate requests and block allocation/deletion | + | `hdds.datanode.kerberos.principal` | Datanode certificate requests | + | `ozone.recon.kerberos.principal` | Recon certificate requests | + | `hdds.scm.kerberos.principal` | SCM certificate requests | + + SCM now checks the requesting service identity. Matching considers the + service name, whether a host component is present, and the realm when one + is configured; individual hostnames are not compared. Proxy identities + do not qualify as service identities. A mismatch can prevent certificate + issuance or renewal. Incorrect OM principal configuration can also prevent + new block allocation or deletion when admin authorization is enabled. + Correct the principal configuration rather than granting service accounts + broad administrator access as a workaround. +3. If security and gRPC TLS are enabled, review the Recon known issue and + conditional workaround below **before restarting with 2.2.2**. Preserve a + usable Recon database. If you require a full SCM checkpoint download, defer + the upgrade until a release or vendor-supported build fixes this issue. + +### Known issue: Recon SCM checkpoint download with gRPC TLS + +[HDDS-17235](https://issues.apache.org/jira/browse/HDDS-17235) requires SCM peer +certificates on SCM Ratis and inter-SCM gRPC connections. Recon downloads a +full SCM database checkpoint through the same inter-SCM gRPC service, but +presents a Recon leaf certificate. With both `ozone.security.enabled=true` +and `hdds.grpc.tls.enabled=true`, SCM 2.2.2 rejects that certificate. + +This affects full SCM checkpoint downloads from both Recon 2.2.1 and Recon +2.2.2 to SCM 2.2.2. Logs may contain `Not an SCM certificate` or +`Rocks DB checkpoint downloading failed`; Recon may report +`GET_SCM_DB_SNAPSHOT_FAILED`. + +Disabling `ozone.recon.scm.snapshot.enabled` is **not required for every +2.2.2 deployment**. This issue applies only when security and gRPC TLS are +both enabled. With this setting enabled, Recon attempts a full checkpoint +download at startup only when the difference between its container count and +SCM's exceeds the configured threshold. An affected download fails; a restart +that does not need a full checkpoint does not encounter this failure. + +For an existing Recon instance with a usable database, you can avoid that +startup download by setting the following in **Recon's** `ozone-site.xml` +before restarting Recon: + +```xml +<property> + <name>ozone.recon.scm.snapshot.enabled</name> + <value>false</value> +</property> +``` + +This is a limited workaround: it skips startup SCM snapshot initialization, +while periodic container reconciliation continues to run. It does not repair +full checkpoint downloads or prevent manual full snapshot requests from being +attempted. Preserve Recon's database and avoid requesting a full SCM snapshot +refresh. The workaround does not provide a way to initialize a new Recon +instance or rebuild a missing or unusable database through a full checkpoint. + +Ozone 2.2.2 has no configuration workaround that restores full SCM checkpoint +downloads while keeping gRPC TLS enabled. If you require this capability, +defer the upgrade until a release or vendor-supported build fixes the issue. Review Comment: Following from the threshold point, deferring the upgrade seems needed only for deployments that lowered `ozone.recon.scm.container.threshold` or rely on the SCM snapshot endpoint. nit: "vendor-supported build" is a bit unusual in ASF release notes; "a later release" might be enough. ```suggestion Ozone 2.2.2 has no configuration workaround that restores full SCM checkpoint downloads while keeping gRPC TLS enabled. If you lowered the threshold or rely on the SCM snapshot endpoint, defer the upgrade until a later release fixes the issue. ``` ########## src/pages/release-notes/2.2.2.md: ########## @@ -8,6 +8,95 @@ Apache Ozone 2.2.2 is a maintenance release on top of Ozone 2.2.1 with bug fixes and dependency updates. +## Upgrade from 2.2.1 + +Ozone 2.2.2 introduces no new OM or HDDS layout features and does not change +the layout version relative to 2.2.1. No layout finalization is required solely +for this patch upgrade. If your cluster already has +an unfinished upgrade from an earlier release, assess that upgrade separately. +Follow the non-rolling procedure in the +[upgrade and downgrade guide](https://ozone.apache.org/docs/administrator-guide/operations/upgrade-and-downgrade/), +with the release-specific checks below. + +### Before stopping the cluster + +1. Verify cluster health. Back up the current configuration, service keytab + files, certificates, and metadata. Preserve the existing + OM, SCM, Datanode, and Recon storage directories during the upgrade. Retain + the 2.2.1 binaries for recovery. +2. On secure clusters, review the following principal properties in the + configuration distributed to **every SCM**, including SCM instances that + issue or renew certificates. Values must match the service principals used + by the corresponding daemons. Copying these principal settings does not + require copying the other daemons' private keys or keytab files to SCM. + + | Property on SCM | Used for | + | --- | --- | + | `ozone.om.kerberos.principal` | OM certificate requests and block allocation/deletion | + | `hdds.datanode.kerberos.principal` | Datanode certificate requests | + | `ozone.recon.kerberos.principal` | Recon certificate requests | + | `hdds.scm.kerberos.principal` | SCM certificate requests | + + SCM now checks the requesting service identity. Matching considers the + service name, whether a host component is present, and the realm when one + is configured; individual hostnames are not compared. Proxy identities + do not qualify as service identities. A mismatch can prevent certificate + issuance or renewal. Incorrect OM principal configuration can also prevent + new block allocation or deletion when admin authorization is enabled. + Correct the principal configuration rather than granting service accounts + broad administrator access as a workaround. +3. If security and gRPC TLS are enabled, review the Recon known issue and + conditional workaround below **before restarting with 2.2.2**. Preserve a + usable Recon database. If you require a full SCM checkpoint download, defer + the upgrade until a release or vendor-supported build fixes this issue. + +### Known issue: Recon SCM checkpoint download with gRPC TLS + +[HDDS-17235](https://issues.apache.org/jira/browse/HDDS-17235) requires SCM peer +certificates on SCM Ratis and inter-SCM gRPC connections. Recon downloads a +full SCM database checkpoint through the same inter-SCM gRPC service, but +presents a Recon leaf certificate. With both `ozone.security.enabled=true` +and `hdds.grpc.tls.enabled=true`, SCM 2.2.2 rejects that certificate. + +This affects full SCM checkpoint downloads from both Recon 2.2.1 and Recon +2.2.2 to SCM 2.2.2. Logs may contain `Not an SCM certificate` or +`Rocks DB checkpoint downloading failed`; Recon may report +`GET_SCM_DB_SNAPSHOT_FAILED`. + +Disabling `ozone.recon.scm.snapshot.enabled` is **not required for every +2.2.2 deployment**. This issue applies only when security and gRPC TLS are +both enabled. With this setting enabled, Recon attempts a full checkpoint +download at startup only when the difference between its container count and +SCM's exceeds the configured threshold. An affected download fails; a restart +that does not need a full checkpoint does not encounter this failure. + +For an existing Recon instance with a usable database, you can avoid that +startup download by setting the following in **Recon's** `ozone-site.xml` +before restarting Recon: + +```xml +<property> + <name>ozone.recon.scm.snapshot.enabled</name> + <value>false</value> +</property> +``` + +This is a limited workaround: it skips startup SCM snapshot initialization, +while periodic container reconciliation continues to run. It does not repair +full checkpoint downloads or prevent manual full snapshot requests from being +attempted. Preserve Recon's database and avoid requesting a full SCM snapshot +refresh. The workaround does not provide a way to initialize a new Recon +instance or rebuild a missing or unusable database through a full checkpoint. + +Ozone 2.2.2 has no configuration workaround that restores full SCM checkpoint +downloads while keeping gRPC TLS enabled. If you require this capability, +defer the upgrade until a release or vendor-supported build fixes the issue. +Once a fix is available, follow its upgrade instructions before restoring +`ozone.recon.scm.snapshot.enabled=true` and verifying a full checkpoint download. + +This issue and the workaround are based on a review of the released source; +they have not been validated in a live secure-cluster upgrade test. Review Comment: nit: for published release notes, this disclaimer may undercut the guidance above. Maybe keep it in the PR description instead, or validate on a secure cluster with gRPC TLS before merging. ```suggestion ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
