rajeshkumarchandolu opened a new pull request, #11995: URL: https://github.com/apache/ozone/pull/11995
## What changes are proposed in this pull request? A datanode write or read currently shows one span per command, with nothing about the chunk or block work behind it or which container and block it touched. This PR adds OpenTelemetry coverage for the datanode container write and read path on top of the tracing helpers merged in HDDS-17581 (#11852). It is part of the tracing epic HDDS-13679. The OM, SCM, client and Freon changes are in separate PRs. **Spans added (service `HddsDatanodeService.<datanode uuid>`)** | Area | Span | How | |---|---|---| | Chunk manager | `ChunkManagerDispatcher.writeChunk` / `readChunk` / `finishWriteChunks`, `FilePerBlockStrategy.writeChunk` / `readChunk` / `finishWriteChunks` | `createProxy` | | Block manager | `BlockManagerImpl.putBlock` / `getBlock` | `createProxy` | | Token verifier | `NoopTokenVerifier.verify` (or the real verifier when security is enabled) | `createProxy` | | Volume choosing | `CapacityVolumeChoosingPolicy.chooseVolume` | `createProxy` | | Container create | `HddsDispatcher.createContainer` (auto-create on the first write) | local span | The existing dispatcher spans (`WriteChunk`, `PutBlock`, `ReadChunk`, `GetBlock`, and others) keep their names. **Attributes added** | Where | Attributes | |---|---| | Every dispatcher span | `dn.cmd_type`, `hdds.container.id`, `hdds.datanode.uuid`, `hdds.datanode.hostname`, `ozone.status` (the container `Result`) | | `WriteChunk` | `hdds.block.local_id`, `hdds.chunk.offset`, `hdds.chunk.length`, `dn.stage` (`WRITE_DATA`, `COMMIT_DATA` or `COMBINED`) | | `PutBlock` | `hdds.block.local_id`, `hdds.block.bcs_id` | | Container state | `dn.container.state` when a failed write checks the container state before marking it unhealthy | | Failures | exception recorded as a span event and span status ERROR | `hdds.block.bcs_id` is recorded in `KeyValueHandler#handlePutBlock` after the datanode assigns the block commit sequence ID, because the client always sends 0. It therefore shows the real value (2, 6, 10 and 14 for four blocks in the manual run). ## What is the link to the Apache JIRA https://issues.apache.org/jira/browse/HDDS-17594 ## How was this patch tested? * `mvn clean install -DskipTests -DskipShade -DskipRecon -DskipDocs`: BUILD SUCCESS * Unit tests with `TestSpanCollector` assertions, 147 run with 0 failures (20 skipped by existing assumptions): `TestHddsDispatcher` (status, exception, stage, container state, container auto-create, BCSID assigned by the datanode), `TestKeyValueHandler`, `TestBlockManagerImpl`, `TestChunkManagerDummyImpl` * `checkstyle.sh`: clean. `rat.sh`: no findings in the changed files * Build in the fork: [`build-branch` workflow run on `tracing-datanode-spans`](https://github.com/rajeshkumarchandolu/ozone/actions/runs/42643332452) (in progress) * Manual check in Jaeger, master versus this branch: The Jaeger runs used the docker compose cluster (`ozone` with `monitoring.yaml`, 3 datanodes) and the same commands on `origin/master` and on this branch. Screenshots are on the [tracing-proofs](https://github.com/rajeshkumarchandolu/ozone/tree/tracing-proofs) branch. Click an image to open it at full size. | Command | Spans on master | Spans with this PR | Jaeger screenshot | |---|---|---|---| | `ozone sh key put` (4 MB) | 64 | 154 | [before](https://raw.githubusercontent.com/rajeshkumarchandolu/ozone/tracing-proofs/datanode/sh-key-put-4m-before.png) / [after](https://raw.githubusercontent.com/rajeshkumarchandolu/ozone/tracing-proofs/datanode/sh-key-put-4m-after.png) / [side by side](https://raw.githubusercontent.com/rajeshkumarchandolu/ozone/tracing-proofs/datanode/sh-key-put-4m-side-by-side.png) | | `ozone sh key get` (4 MB) | 40 | 60 | [before](https://raw.githubusercontent.com/rajeshkumarchandolu/ozone/tracing-proofs/datanode/sh-key-get-4m-before.png) / [after](https://raw.githubusercontent.com/rajeshkumarchandolu/ozone/tracing-proofs/datanode/sh-key-get-4m-after.png) / [side by side](https://raw.githubusercontent.com/rajeshkumarchandolu/ozone/tracing-proofs/datanode/sh-key-get-4m-side-by-side.png) | | `ozone fs -put` (4 MB) | 1129 | 1213 | [before](https://raw.githubusercontent.com/rajeshkumarchandolu/ozone/tracing-proofs/datanode/fs-put-4m-before.png) / [after](https://raw.githubusercontent.com/rajeshkumarchandolu/ozone/tracing-proofs/datanode/fs-put-4m-after.png) / [side by side](https://raw.githubusercontent.com/rajeshkumarchandolu/ozone/tracing-proofs/datanode/fs-put-4m-side-by-side.png) | | `ozone fs -cat` (4 MB) | 2073 | 2849 | [before](https://raw.githubusercontent.com/rajeshkumarchandolu/ozone/tracing-proofs/datanode/fs-cat-4m-before.png) / [after](https://raw.githubusercontent.com/rajeshkumarchandolu/ozone/tracing-proofs/datanode/fs-cat-4m-after.png) / [side by side](https://raw.githubusercontent.com/rajeshkumarchandolu/ozone/tracing-proofs/datanode/fs-cat-4m-side-by-side.png) | | `ozone freon ockg -n 3 -s 4MB` | 191 | 443 | [before](https://raw.githubusercontent.com/rajeshkumarchandolu/ozone/tracing-proofs/datanode/freon-ockg-before.png) / [after](https://raw.githubusercontent.com/rajeshkumarchandolu/ozone/tracing-proofs/datanode/freon-ockg-after.png) / [side by side](https://raw.githubusercontent.com/rajeshkumarchandolu/ozone/tracing-proofs/datanode/freon-ockg-side-by-side.png) | OM and SCM spans are unchanged by this PR. Commands without datanode I/O (volume, bucket, key info, list) are identical. Each chunk read or write on each replica adds up to three internal spans. `ozone fs -cat` reads in small chunks, so the datanode part adds about 780 spans for a 4 MB key. **ozone sh key put: datanode WriteChunk and PutBlock subtree on each replica** [](https://raw.githubusercontent.com/rajeshkumarchandolu/ozone/tracing-proofs/datanode/sh-key-put-4m-side-by-side.png) **ozone sh key get: datanode GetBlock and ReadChunk** [](https://raw.githubusercontent.com/rajeshkumarchandolu/ozone/tracing-proofs/datanode/sh-key-get-4m-side-by-side.png) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
