rajeshkumarchandolu opened a new pull request, #11996:
URL: https://github.com/apache/ozone/pull/11996

   ## What changes are proposed in this pull request?
   
   Client side spans are named after the full command line (`ozone sh key put 
/vol/bucket/key /tmp/file`), which gives one
   span name per key, and they carry no information about what the client 
wrote, which datanode served a read or when a
   Ratis request really finished. This PR improves the Ozone client libraries 
and the CLI (`ozone sh`, `ozone fs`) on top of
   the tracing helpers merged in HDDS-17581 (#11852). It is part of the tracing 
epic HDDS-13679. The OM, SCM, datanode and
   Freon changes are in separate PRs.
   
   **Root spans (CLI)**
   
   | Before (master) | After |
   |---|---|
   | `ozone sh key put /vol/bucket/key /tmp/file` (whole command line in the 
span name) | `ozone sh key put` (sub-command only) with `cli.*` attributes |
   | `ozone fs -put /tmp/file ofs://...` | `ozone fs -put` with `cli.*` 
attributes |
   
   Span names are now low cardinality. The arguments are attributes, so they 
can still be searched.
   
   **Spans added (service `shell` / `FsShell`)**
   
   | Area | Span | How |
   |---|---|---|
   | Streaming read | `XceiverClientGrpc.ReadBlock` (one per streaming 
ReadBlock request, ends when its data arrives) | local client span |
   | Ratis write | `XceiverClientRatis.WriteChunk` / `PutBlock` now end when 
the datanode reply arrives | fix of the existing span |
   | Connections | `XceiverClientGrpc.connect`, `XceiverClientRatis.connect` | 
local spans |
   | Setup | `XceiverClientMetrics.create`, `SecurityConfig.init` | local spans 
|
   | Block flush | `BlockOutputStream.flush` | local span |
   
   **Attributes added**
   
   | Where | Attributes |
   |---|---|
   | CLI root span | `cli.command`, `cli.args`, `cli.option.<name>`. Values of 
`--secret`, `--token`, `--set` and interactive options are replaced by `***` |
   | `ozone sh` root span | `volume`, `bucket`, `key`, `dataSize`, 
`replicationConfig` (when set) |
   | Key output stream (`ozone sh key put`, `ozone fs -put`, Freon) | 
`client.bytes_written`, `client.blocks_written`, `client.write.retry` |
   | `BlockOutputStream.flush` | `client.block.bytes_written`, 
`hdds.container.id`, `hdds.block.local_id`, `pipeline.id` |
   | `XceiverClientRatis.WriteChunk` / `PutBlock` | `hdds.container.id`, 
`hdds.block.local_id`, `hdds.chunk.offset`, `hdds.chunk.length`, `pipeline.id` |
   | `XceiverClientGrpc.connect` / `XceiverClientRatis.connect` | `pipeline.id` 
|
   | Datanode calls | `hdds.datanode`, `hdds.datanode.uuid` (serving datanode), 
and event `hdds.datanode.failed` for each datanode that failed before another 
one served the read |
   | Streaming ReadBlock | the trace ID is sent to the datanode, so the 
datanode `ReadBlock` span joins the client trace |
   
   ## What is the link to the Apache JIRA
   
   https://issues.apache.org/jira/browse/HDDS-17761
   
   ## How was this patch tested?
   
   * `mvn clean install -DskipTests -DskipShade -DskipRecon -DskipDocs`: BUILD 
SUCCESS
   * Unit tests with `TestSpanCollector` assertions, 94 pass: 
`TestXceiverClientGrpcTracing`, `TestXceiverClientRatisTracing`,
     `TestStreamBlockInputStream`, `TestContainerProtocolCalls`, 
`TestTracingUtil`, `TestHandlerTracing`, `TestShell`,
     `TestPutKeyHandler`, `TestOzoneFsShell`, `TestOzoneClient`
   * `checkstyle.sh`: clean. `rat.sh`: no findings in the changed files
   * Build in the fork: [`build-branch` workflow run on 
`tracing-client-spans`](https://github.com/rajeshkumarchandolu/ozone/actions/runs/42644804308)
 (in progress)
   * Manual check in Jaeger, master versus this branch. The streaming read was 
checked with
     `-Dozone.client.stream.readblock.enable=true`:
   
   The Jaeger runs used the docker compose cluster (`ozone` with 
`monitoring.yaml`, 3 datanodes) and the same
   commands on `origin/master` and on this branch. Screenshots are on the
   
[tracing-proofs](https://github.com/rajeshkumarchandolu/ozone/tree/tracing-proofs)
 branch. Click an image to open it
   at full size.
   
   | Command | Spans on master | Spans with this PR | Jaeger screenshot |
   |---|---|---|---|
   | `ozone sh key put` (4 MB) | 64 | 71 | 
[before](https://raw.githubusercontent.com/rajeshkumarchandolu/ozone/tracing-proofs/client/sh-key-put-4m-before.png)
 / 
[after](https://raw.githubusercontent.com/rajeshkumarchandolu/ozone/tracing-proofs/client/sh-key-put-4m-after.png)
 / [side by 
side](https://raw.githubusercontent.com/rajeshkumarchandolu/ozone/tracing-proofs/client/sh-key-put-4m-side-by-side.png)
 |
   | `ozone sh key get` (4 MB) | 40 | 42 | 
[before](https://raw.githubusercontent.com/rajeshkumarchandolu/ozone/tracing-proofs/client/sh-key-get-4m-before.png)
 / 
[after](https://raw.githubusercontent.com/rajeshkumarchandolu/ozone/tracing-proofs/client/sh-key-get-4m-after.png)
 / [side by 
side](https://raw.githubusercontent.com/rajeshkumarchandolu/ozone/tracing-proofs/client/sh-key-get-4m-side-by-side.png)
 |
   | `ozone fs -put` (4 MB) | 1129 | 1136 | 
[before](https://raw.githubusercontent.com/rajeshkumarchandolu/ozone/tracing-proofs/client/fs-put-4m-before.png)
 / 
[after](https://raw.githubusercontent.com/rajeshkumarchandolu/ozone/tracing-proofs/client/fs-put-4m-after.png)
 / [side by 
side](https://raw.githubusercontent.com/rajeshkumarchandolu/ozone/tracing-proofs/client/fs-put-4m-side-by-side.png)
 |
   | `ozone fs -cat` (4 MB) | 2073 | 2075 | 
[before](https://raw.githubusercontent.com/rajeshkumarchandolu/ozone/tracing-proofs/client/fs-cat-4m-before.png)
 / 
[after](https://raw.githubusercontent.com/rajeshkumarchandolu/ozone/tracing-proofs/client/fs-cat-4m-after.png)
 / [side by 
side](https://raw.githubusercontent.com/rajeshkumarchandolu/ozone/tracing-proofs/client/fs-cat-4m-side-by-side.png)
 |
   
   OM, SCM and datanode spans are unchanged by this PR. Commands that open no 
new connection or stream (volume, bucket, key
   info, list) show the new root span name and attributes with the same span 
count.
   
   A per-read span in `MultipartInputStream` is deliberately not part of this 
PR: it created about 3,900 spans for a 4 MB
   `ozone fs -cat`.
   
   **ozone sh key put: root span with cli.* and client.* attributes, connect 
and flush spans**
   
   [![ozone sh key put: root span with cli.* and client.* attributes, connect 
and flush 
spans](https://raw.githubusercontent.com/rajeshkumarchandolu/ozone/tracing-proofs/client/sh-key-put-4m-side-by-side.png)](https://raw.githubusercontent.com/rajeshkumarchandolu/ozone/tracing-proofs/client/sh-key-put-4m-side-by-side.png)
   
   **ozone sh key get with the streaming ReadBlock client spans (no master 
equivalent captured)**
   
   [![ozone sh key get with the streaming ReadBlock client spans (no master 
equivalent 
captured)](https://raw.githubusercontent.com/rajeshkumarchandolu/ozone/tracing-proofs/client/stream-key-get-4m-after.png)](https://raw.githubusercontent.com/rajeshkumarchandolu/ozone/tracing-proofs/client/stream-key-get-4m-after.png)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to