rajeshkumarchandolu opened a new pull request, #11994: URL: https://github.com/apache/ozone/pull/11994
## What changes are proposed in this pull request? Ozone Manager spans today stop at the RPC server span, so a slow or failed request cannot be attributed to a stage (validation, lock wait, ACL check, key lookup, double buffer commit). This PR adds OpenTelemetry coverage for OM on top of the tracing helpers merged in HDDS-17581 (#11852). It is part of the tracing epic HDDS-13679. The SCM, datanode, client and Freon changes are in separate PRs. **This branch includes the OM prerequisite commits**, which go away from the diff once they are merged: #11849 (HDDS-17595, `KeyManager` interface instead of `KeyManagerImpl` casts), #11850 (HDDS-17596, `TestOMSortDatanodes` through `KeyManager#listStatus`), #11851 (HDDS-17597, request handler fault injection as a test decorator), and `IOMClientRequest` for callers of `OzoneManagerRatisUtils#createClientRequest` (HDDS-17598). The OM tracing change itself is the last 9 commits. **Spans added (service `OzoneManager`)** | Area | Span | How | |---|---|---| | Request handler | `OzoneManagerRequestHandler.handleReadRequest` / `handleWriteRequest` / `validateRequest` | `createProxy` | | Client requests | `<Request>.preExecute` / `validateAndUpdateCache` / `handleRequestFailure`, for example `OMKeyCommitRequestWithFSO.validateAndUpdateCache` | `createProxy` through `IOMClientRequest` | | Managers | `KeyManagerImpl.getKeyInfo` / `getFileStatus` / `listStatus`, `BucketManagerImpl.getBucketInfo`, `VolumeManagerImpl.getVolumeInfo` / `checkAccess` | `createProxy` | | Authorizer | `OzoneNativeAuthorizer.checkAccess` (only when ACLs are enabled) | `createProxy` | | Key read steps | `OM.readKeyInfo`, `OM.refreshPipelineFromCache`, EDEK, bucket checks, snapshot checkpoint | local spans | | Double buffer | `DB-commitWriteBatch`, `DB-addWriteBatch-transactioninfo` (fixed names, the batch size is an attribute) | renamed spans | **Attributes added** | Where | Attributes | |---|---| | OM server span of each RPC | `om.cmd_type`, `om.client_id`, `om.call_id`, `ozone.status` | | Request and handler spans | `volume`, `bucket`, `key`, `dataSize`, `replicationType`, `replicationFactor`, `bucketLayout`, `quotaInBytes`, `quotaInNamespace`, `snapshotName`, `srcKey`, `dstKey`, `isVersionEnabled`, and the other values already put in the audit map | | Write requests | `om.lock.wait_ns`, `om.lock.read_ns`, `om.lock.write_ns`, `om.bucket_layout`, `om.role` | | ACL checks | `om.acl.granted` | | Key requests | `om.key.requested_blocks`, `om.key.committed_blocks`, `om.key.hsync`, `om.key.recovery`, `om.key.container_count`, `om.key.force_refresh` | | Double buffer | `om.batch_size` | | Failures | exception recorded as a span event, span status ERROR, `ozone.status` holds the `OMException` result code | Audit identity fields such as owner and admin are not added as span attributes. ## What is the link to the Apache JIRA https://issues.apache.org/jira/browse/HDDS-17586 ## How was this patch tested? * `mvn clean install -DskipTests -DskipShade -DskipRecon -DskipDocs`: BUILD SUCCESS * Unit tests with `TestSpanCollector` assertions, 211 tests in 12 classes pass: `TestOMMetadataReader`, `TestOzoneManagerDoubleBuffer`, `TestOzoneManagerRatisRequest`, `TestOMResponse`, `TestOMBucketCreateRequest`, `TestOMBucketDeleteRequest`, `TestOMKeyCommitRequest(WithFSO)`, `TestOMKeyRenameRequest(WithFSO)`, `TestOMSnapshotCreateResponse`, `TestKeyManagerUnit` * `checkstyle.sh`: clean. `rat.sh`: no findings in the changed files * Build in the fork: [`build-branch` workflow run on `tracing-om-spans`](https://github.com/rajeshkumarchandolu/ozone/actions/runs/42639967763) (in progress) * Manual check in Jaeger, master versus this branch: The Jaeger runs used the docker compose cluster (`ozone` with `monitoring.yaml`, 3 datanodes) and the same commands on `origin/master` and on this branch. Screenshots are on the [tracing-proofs](https://github.com/rajeshkumarchandolu/ozone/tree/tracing-proofs) branch. Click an image to open it at full size. | Command | Spans on master | Spans with this PR | Jaeger screenshot | |---|---|---|---| | `ozone sh volume create` | 9 | 16 | [before](https://raw.githubusercontent.com/rajeshkumarchandolu/ozone/tracing-proofs/om/sh-volume-create-before.png) / [after](https://raw.githubusercontent.com/rajeshkumarchandolu/ozone/tracing-proofs/om/sh-volume-create-after.png) / [side by side](https://raw.githubusercontent.com/rajeshkumarchandolu/ozone/tracing-proofs/om/sh-volume-create-side-by-side.png) | | `ozone sh bucket create` | 11 | 20 | [before](https://raw.githubusercontent.com/rajeshkumarchandolu/ozone/tracing-proofs/om/sh-bucket-create-before.png) / [after](https://raw.githubusercontent.com/rajeshkumarchandolu/ozone/tracing-proofs/om/sh-bucket-create-after.png) / [side by side](https://raw.githubusercontent.com/rajeshkumarchandolu/ozone/tracing-proofs/om/sh-bucket-create-side-by-side.png) | | `ozone sh key put` (4 MB) | 64 | 84 | [before](https://raw.githubusercontent.com/rajeshkumarchandolu/ozone/tracing-proofs/om/sh-key-put-4m-before.png) / [after](https://raw.githubusercontent.com/rajeshkumarchandolu/ozone/tracing-proofs/om/sh-key-put-4m-after.png) / [side by side](https://raw.githubusercontent.com/rajeshkumarchandolu/ozone/tracing-proofs/om/sh-key-put-4m-side-by-side.png) | | `ozone sh key info` | 10 | 21 | [before](https://raw.githubusercontent.com/rajeshkumarchandolu/ozone/tracing-proofs/om/sh-key-info-before.png) / [after](https://raw.githubusercontent.com/rajeshkumarchandolu/ozone/tracing-proofs/om/sh-key-info-after.png) / [side by side](https://raw.githubusercontent.com/rajeshkumarchandolu/ozone/tracing-proofs/om/sh-key-info-side-by-side.png) | | `ozone sh key info` on a missing key | 10 | 19 | [before](https://raw.githubusercontent.com/rajeshkumarchandolu/ozone/tracing-proofs/om/sh-key-info-missing-before.png) / [after](https://raw.githubusercontent.com/rajeshkumarchandolu/ozone/tracing-proofs/om/sh-key-info-missing-after.png) / [side by side](https://raw.githubusercontent.com/rajeshkumarchandolu/ozone/tracing-proofs/om/sh-key-info-missing-side-by-side.png) | | `ozone sh key list` | 14 | 28 | [before](https://raw.githubusercontent.com/rajeshkumarchandolu/ozone/tracing-proofs/om/sh-key-list-before.png) / [after](https://raw.githubusercontent.com/rajeshkumarchandolu/ozone/tracing-proofs/om/sh-key-list-after.png) / [side by side](https://raw.githubusercontent.com/rajeshkumarchandolu/ozone/tracing-proofs/om/sh-key-list-side-by-side.png) | | `ozone sh snapshot create` | 9 | 18 | [before](https://raw.githubusercontent.com/rajeshkumarchandolu/ozone/tracing-proofs/om/sh-snapshot-create-before.png) / [after](https://raw.githubusercontent.com/rajeshkumarchandolu/ozone/tracing-proofs/om/sh-snapshot-create-after.png) / [side by side](https://raw.githubusercontent.com/rajeshkumarchandolu/ozone/tracing-proofs/om/sh-snapshot-create-side-by-side.png) | | `ozone fs -mkdir -p` | 14 | 27 | [before](https://raw.githubusercontent.com/rajeshkumarchandolu/ozone/tracing-proofs/om/fs-mkdir-before.png) / [after](https://raw.githubusercontent.com/rajeshkumarchandolu/ozone/tracing-proofs/om/fs-mkdir-after.png) / [side by side](https://raw.githubusercontent.com/rajeshkumarchandolu/ozone/tracing-proofs/om/fs-mkdir-side-by-side.png) | SCM and datanode spans are unchanged by this PR. **ozone sh key put: OM request handling, lock times and double buffer commit** [](https://raw.githubusercontent.com/rajeshkumarchandolu/ozone/tracing-proofs/om/sh-key-put-4m-side-by-side.png) **ozone sh key info on a missing key: error status with ozone.status=KEY_NOT_FOUND** [](https://raw.githubusercontent.com/rajeshkumarchandolu/ozone/tracing-proofs/om/sh-key-info-missing-side-by-side.png) **With ACLs enabled, a denied user: om.acl.granted=False and the failing span** [](https://raw.githubusercontent.com/rajeshkumarchandolu/ozone/tracing-proofs/om/acl-user-key-info-denied-after.png) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
