rajeshkumarchandolu opened a new pull request, #11996: URL: https://github.com/apache/ozone/pull/11996
## What changes are proposed in this pull request? Client side spans are named after the full command line (`ozone sh key put /vol/bucket/key /tmp/file`), which gives one span name per key, and they carry no information about what the client wrote, which datanode served a read or when a Ratis request really finished. This PR improves the Ozone client libraries and the CLI (`ozone sh`, `ozone fs`) on top of the tracing helpers merged in HDDS-17581 (#11852). It is part of the tracing epic HDDS-13679. The OM, SCM, datanode and Freon changes are in separate PRs. **Root spans (CLI)** | Before (master) | After | |---|---| | `ozone sh key put /vol/bucket/key /tmp/file` (whole command line in the span name) | `ozone sh key put` (sub-command only) with `cli.*` attributes | | `ozone fs -put /tmp/file ofs://...` | `ozone fs -put` with `cli.*` attributes | Span names are now low cardinality. The arguments are attributes, so they can still be searched. **Spans added (service `shell` / `FsShell`)** | Area | Span | How | |---|---|---| | Streaming read | `XceiverClientGrpc.ReadBlock` (one per streaming ReadBlock request, ends when its data arrives) | local client span | | Ratis write | `XceiverClientRatis.WriteChunk` / `PutBlock` now end when the datanode reply arrives | fix of the existing span | | Connections | `XceiverClientGrpc.connect`, `XceiverClientRatis.connect` | local spans | | Setup | `XceiverClientMetrics.create`, `SecurityConfig.init` | local spans | | Block flush | `BlockOutputStream.flush` | local span | **Attributes added** | Where | Attributes | |---|---| | CLI root span | `cli.command`, `cli.args`, `cli.option.<name>`. Values of `--secret`, `--token`, `--set` and interactive options are replaced by `***` | | `ozone sh` root span | `volume`, `bucket`, `key`, `dataSize`, `replicationConfig` (when set) | | Key output stream (`ozone sh key put`, `ozone fs -put`, Freon) | `client.bytes_written`, `client.blocks_written`, `client.write.retry` | | `BlockOutputStream.flush` | `client.block.bytes_written`, `hdds.container.id`, `hdds.block.local_id`, `pipeline.id` | | `XceiverClientRatis.WriteChunk` / `PutBlock` | `hdds.container.id`, `hdds.block.local_id`, `hdds.chunk.offset`, `hdds.chunk.length`, `pipeline.id` | | `XceiverClientGrpc.connect` / `XceiverClientRatis.connect` | `pipeline.id` | | Datanode calls | `hdds.datanode`, `hdds.datanode.uuid` (serving datanode), and event `hdds.datanode.failed` for each datanode that failed before another one served the read | | Streaming ReadBlock | the trace ID is sent to the datanode, so the datanode `ReadBlock` span joins the client trace | ## What is the link to the Apache JIRA https://issues.apache.org/jira/browse/HDDS-17761 ## How was this patch tested? * `mvn clean install -DskipTests -DskipShade -DskipRecon -DskipDocs`: BUILD SUCCESS * Unit tests with `TestSpanCollector` assertions, 94 pass: `TestXceiverClientGrpcTracing`, `TestXceiverClientRatisTracing`, `TestStreamBlockInputStream`, `TestContainerProtocolCalls`, `TestTracingUtil`, `TestHandlerTracing`, `TestShell`, `TestPutKeyHandler`, `TestOzoneFsShell`, `TestOzoneClient` * `checkstyle.sh`: clean. `rat.sh`: no findings in the changed files * Build in the fork: [`build-branch` workflow run on `tracing-client-spans`](https://github.com/rajeshkumarchandolu/ozone/actions/runs/42644804308) (in progress) * Manual check in Jaeger, master versus this branch. The streaming read was checked with `-Dozone.client.stream.readblock.enable=true`: The Jaeger runs used the docker compose cluster (`ozone` with `monitoring.yaml`, 3 datanodes) and the same commands on `origin/master` and on this branch. Screenshots are on the [tracing-proofs](https://github.com/rajeshkumarchandolu/ozone/tree/tracing-proofs) branch. Click an image to open it at full size. | Command | Spans on master | Spans with this PR | Jaeger screenshot | |---|---|---|---| | `ozone sh key put` (4 MB) | 64 | 71 | [before](https://raw.githubusercontent.com/rajeshkumarchandolu/ozone/tracing-proofs/client/sh-key-put-4m-before.png) / [after](https://raw.githubusercontent.com/rajeshkumarchandolu/ozone/tracing-proofs/client/sh-key-put-4m-after.png) / [side by side](https://raw.githubusercontent.com/rajeshkumarchandolu/ozone/tracing-proofs/client/sh-key-put-4m-side-by-side.png) | | `ozone sh key get` (4 MB) | 40 | 42 | [before](https://raw.githubusercontent.com/rajeshkumarchandolu/ozone/tracing-proofs/client/sh-key-get-4m-before.png) / [after](https://raw.githubusercontent.com/rajeshkumarchandolu/ozone/tracing-proofs/client/sh-key-get-4m-after.png) / [side by side](https://raw.githubusercontent.com/rajeshkumarchandolu/ozone/tracing-proofs/client/sh-key-get-4m-side-by-side.png) | | `ozone fs -put` (4 MB) | 1129 | 1136 | [before](https://raw.githubusercontent.com/rajeshkumarchandolu/ozone/tracing-proofs/client/fs-put-4m-before.png) / [after](https://raw.githubusercontent.com/rajeshkumarchandolu/ozone/tracing-proofs/client/fs-put-4m-after.png) / [side by side](https://raw.githubusercontent.com/rajeshkumarchandolu/ozone/tracing-proofs/client/fs-put-4m-side-by-side.png) | | `ozone fs -cat` (4 MB) | 2073 | 2075 | [before](https://raw.githubusercontent.com/rajeshkumarchandolu/ozone/tracing-proofs/client/fs-cat-4m-before.png) / [after](https://raw.githubusercontent.com/rajeshkumarchandolu/ozone/tracing-proofs/client/fs-cat-4m-after.png) / [side by side](https://raw.githubusercontent.com/rajeshkumarchandolu/ozone/tracing-proofs/client/fs-cat-4m-side-by-side.png) | OM, SCM and datanode spans are unchanged by this PR. Commands that open no new connection or stream (volume, bucket, key info, list) show the new root span name and attributes with the same span count. A per-read span in `MultipartInputStream` is deliberately not part of this PR: it created about 3,900 spans for a 4 MB `ozone fs -cat`. **ozone sh key put: root span with cli.* and client.* attributes, connect and flush spans** [](https://raw.githubusercontent.com/rajeshkumarchandolu/ozone/tracing-proofs/client/sh-key-put-4m-side-by-side.png) **ozone sh key get with the streaming ReadBlock client spans (no master equivalent captured)** [](https://raw.githubusercontent.com/rajeshkumarchandolu/ozone/tracing-proofs/client/stream-key-get-4m-after.png) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
