rajeshkumarchandolu opened a new pull request, #11994:
URL: https://github.com/apache/ozone/pull/11994

   ## What changes are proposed in this pull request?
   
   Ozone Manager spans today stop at the RPC server span, so a slow or failed 
request cannot be attributed to a stage
   (validation, lock wait, ACL check, key lookup, double buffer commit). This 
PR adds OpenTelemetry coverage for OM on top
   of the tracing helpers merged in HDDS-17581 (#11852). It is part of the 
tracing epic HDDS-13679. The SCM, datanode,
   client and Freon changes are in separate PRs.
   
   **This branch includes the OM prerequisite commits**, which go away from the 
diff once they are merged:
   #11849 (HDDS-17595, `KeyManager` interface instead of `KeyManagerImpl` 
casts), #11850 (HDDS-17596, `TestOMSortDatanodes`
   through `KeyManager#listStatus`), #11851 (HDDS-17597, request handler fault 
injection as a test decorator), and
   `IOMClientRequest` for callers of 
`OzoneManagerRatisUtils#createClientRequest` (HDDS-17598). The OM tracing 
change itself
   is the last 9 commits.
   
   **Spans added (service `OzoneManager`)**
   
   | Area | Span | How |
   |---|---|---|
   | Request handler | `OzoneManagerRequestHandler.handleReadRequest` / 
`handleWriteRequest` / `validateRequest` | `createProxy` |
   | Client requests | `<Request>.preExecute` / `validateAndUpdateCache` / 
`handleRequestFailure`, for example 
`OMKeyCommitRequestWithFSO.validateAndUpdateCache` | `createProxy` through 
`IOMClientRequest` |
   | Managers | `KeyManagerImpl.getKeyInfo` / `getFileStatus` / `listStatus`, 
`BucketManagerImpl.getBucketInfo`, `VolumeManagerImpl.getVolumeInfo` / 
`checkAccess` | `createProxy` |
   | Authorizer | `OzoneNativeAuthorizer.checkAccess` (only when ACLs are 
enabled) | `createProxy` |
   | Key read steps | `OM.readKeyInfo`, `OM.refreshPipelineFromCache`, EDEK, 
bucket checks, snapshot checkpoint | local spans |
   | Double buffer | `DB-commitWriteBatch`, `DB-addWriteBatch-transactioninfo` 
(fixed names, the batch size is an attribute) | renamed spans |
   
   **Attributes added**
   
   | Where | Attributes |
   |---|---|
   | OM server span of each RPC | `om.cmd_type`, `om.client_id`, `om.call_id`, 
`ozone.status` |
   | Request and handler spans | `volume`, `bucket`, `key`, `dataSize`, 
`replicationType`, `replicationFactor`, `bucketLayout`, `quotaInBytes`, 
`quotaInNamespace`, `snapshotName`, `srcKey`, `dstKey`, `isVersionEnabled`, and 
the other values already put in the audit map |
   | Write requests | `om.lock.wait_ns`, `om.lock.read_ns`, `om.lock.write_ns`, 
`om.bucket_layout`, `om.role` |
   | ACL checks | `om.acl.granted` |
   | Key requests | `om.key.requested_blocks`, `om.key.committed_blocks`, 
`om.key.hsync`, `om.key.recovery`, `om.key.container_count`, 
`om.key.force_refresh` |
   | Double buffer | `om.batch_size` |
   | Failures | exception recorded as a span event, span status ERROR, 
`ozone.status` holds the `OMException` result code |
   
   Audit identity fields such as owner and admin are not added as span 
attributes.
   
   ## What is the link to the Apache JIRA
   
   https://issues.apache.org/jira/browse/HDDS-17586
   
   ## How was this patch tested?
   
   * `mvn clean install -DskipTests -DskipShade -DskipRecon -DskipDocs`: BUILD 
SUCCESS
   * Unit tests with `TestSpanCollector` assertions, 211 tests in 12 classes 
pass: `TestOMMetadataReader`,
     `TestOzoneManagerDoubleBuffer`, `TestOzoneManagerRatisRequest`, 
`TestOMResponse`, `TestOMBucketCreateRequest`,
     `TestOMBucketDeleteRequest`, `TestOMKeyCommitRequest(WithFSO)`, 
`TestOMKeyRenameRequest(WithFSO)`,
     `TestOMSnapshotCreateResponse`, `TestKeyManagerUnit`
   * `checkstyle.sh`: clean. `rat.sh`: no findings in the changed files
   * Build in the fork: [`build-branch` workflow run on 
`tracing-om-spans`](https://github.com/rajeshkumarchandolu/ozone/actions/runs/42639967763)
 (in progress)
   * Manual check in Jaeger, master versus this branch:
   
   The Jaeger runs used the docker compose cluster (`ozone` with 
`monitoring.yaml`, 3 datanodes) and the same
   commands on `origin/master` and on this branch. Screenshots are on the
   
[tracing-proofs](https://github.com/rajeshkumarchandolu/ozone/tree/tracing-proofs)
 branch. Click an image to open it
   at full size.
   
   | Command | Spans on master | Spans with this PR | Jaeger screenshot |
   |---|---|---|---|
   | `ozone sh volume create` | 9 | 16 | 
[before](https://raw.githubusercontent.com/rajeshkumarchandolu/ozone/tracing-proofs/om/sh-volume-create-before.png)
 / 
[after](https://raw.githubusercontent.com/rajeshkumarchandolu/ozone/tracing-proofs/om/sh-volume-create-after.png)
 / [side by 
side](https://raw.githubusercontent.com/rajeshkumarchandolu/ozone/tracing-proofs/om/sh-volume-create-side-by-side.png)
 |
   | `ozone sh bucket create` | 11 | 20 | 
[before](https://raw.githubusercontent.com/rajeshkumarchandolu/ozone/tracing-proofs/om/sh-bucket-create-before.png)
 / 
[after](https://raw.githubusercontent.com/rajeshkumarchandolu/ozone/tracing-proofs/om/sh-bucket-create-after.png)
 / [side by 
side](https://raw.githubusercontent.com/rajeshkumarchandolu/ozone/tracing-proofs/om/sh-bucket-create-side-by-side.png)
 |
   | `ozone sh key put` (4 MB) | 64 | 84 | 
[before](https://raw.githubusercontent.com/rajeshkumarchandolu/ozone/tracing-proofs/om/sh-key-put-4m-before.png)
 / 
[after](https://raw.githubusercontent.com/rajeshkumarchandolu/ozone/tracing-proofs/om/sh-key-put-4m-after.png)
 / [side by 
side](https://raw.githubusercontent.com/rajeshkumarchandolu/ozone/tracing-proofs/om/sh-key-put-4m-side-by-side.png)
 |
   | `ozone sh key info` | 10 | 21 | 
[before](https://raw.githubusercontent.com/rajeshkumarchandolu/ozone/tracing-proofs/om/sh-key-info-before.png)
 / 
[after](https://raw.githubusercontent.com/rajeshkumarchandolu/ozone/tracing-proofs/om/sh-key-info-after.png)
 / [side by 
side](https://raw.githubusercontent.com/rajeshkumarchandolu/ozone/tracing-proofs/om/sh-key-info-side-by-side.png)
 |
   | `ozone sh key info` on a missing key | 10 | 19 | 
[before](https://raw.githubusercontent.com/rajeshkumarchandolu/ozone/tracing-proofs/om/sh-key-info-missing-before.png)
 / 
[after](https://raw.githubusercontent.com/rajeshkumarchandolu/ozone/tracing-proofs/om/sh-key-info-missing-after.png)
 / [side by 
side](https://raw.githubusercontent.com/rajeshkumarchandolu/ozone/tracing-proofs/om/sh-key-info-missing-side-by-side.png)
 |
   | `ozone sh key list` | 14 | 28 | 
[before](https://raw.githubusercontent.com/rajeshkumarchandolu/ozone/tracing-proofs/om/sh-key-list-before.png)
 / 
[after](https://raw.githubusercontent.com/rajeshkumarchandolu/ozone/tracing-proofs/om/sh-key-list-after.png)
 / [side by 
side](https://raw.githubusercontent.com/rajeshkumarchandolu/ozone/tracing-proofs/om/sh-key-list-side-by-side.png)
 |
   | `ozone sh snapshot create` | 9 | 18 | 
[before](https://raw.githubusercontent.com/rajeshkumarchandolu/ozone/tracing-proofs/om/sh-snapshot-create-before.png)
 / 
[after](https://raw.githubusercontent.com/rajeshkumarchandolu/ozone/tracing-proofs/om/sh-snapshot-create-after.png)
 / [side by 
side](https://raw.githubusercontent.com/rajeshkumarchandolu/ozone/tracing-proofs/om/sh-snapshot-create-side-by-side.png)
 |
   | `ozone fs -mkdir -p` | 14 | 27 | 
[before](https://raw.githubusercontent.com/rajeshkumarchandolu/ozone/tracing-proofs/om/fs-mkdir-before.png)
 / 
[after](https://raw.githubusercontent.com/rajeshkumarchandolu/ozone/tracing-proofs/om/fs-mkdir-after.png)
 / [side by 
side](https://raw.githubusercontent.com/rajeshkumarchandolu/ozone/tracing-proofs/om/fs-mkdir-side-by-side.png)
 |
   
   SCM and datanode spans are unchanged by this PR.
   
   **ozone sh key put: OM request handling, lock times and double buffer 
commit**
   
   [![ozone sh key put: OM request handling, lock times and double buffer 
commit](https://raw.githubusercontent.com/rajeshkumarchandolu/ozone/tracing-proofs/om/sh-key-put-4m-side-by-side.png)](https://raw.githubusercontent.com/rajeshkumarchandolu/ozone/tracing-proofs/om/sh-key-put-4m-side-by-side.png)
   
   **ozone sh key info on a missing key: error status with 
ozone.status=KEY_NOT_FOUND**
   
   [![ozone sh key info on a missing key: error status with 
ozone.status=KEY_NOT_FOUND](https://raw.githubusercontent.com/rajeshkumarchandolu/ozone/tracing-proofs/om/sh-key-info-missing-side-by-side.png)](https://raw.githubusercontent.com/rajeshkumarchandolu/ozone/tracing-proofs/om/sh-key-info-missing-side-by-side.png)
   
   **With ACLs enabled, a denied user: om.acl.granted=False and the failing 
span**
   
   [![With ACLs enabled, a denied user: om.acl.granted=False and the failing 
span](https://raw.githubusercontent.com/rajeshkumarchandolu/ozone/tracing-proofs/om/acl-user-key-info-denied-after.png)](https://raw.githubusercontent.com/rajeshkumarchandolu/ozone/tracing-proofs/om/acl-user-key-info-denied-after.png)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to