[
https://issues.apache.org/jira/browse/HTRACE-298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15008089#comment-15008089
]
Colin Patrick McCabe commented on HTRACE-298:
---------------------------------------------
Two big changes here.
* The datastore now uses msgpack serialization rather than gob serialization.
gob serialization is slower and less compact-- partly because it contains
information we don't really need (type information about each field).
* The datastore now accepts "batches" of spans rather than individual spans.
This is more efficient because each time we send a span to the datastore
goroutines, that requires CPU-level synchronization.
Some smaller changes:
* Update {{github.com/ugorji/go/codec}} dependency to a new enough version to
get the {{Encoder#ResetBytes}} function
* Simplify {{MetricsSink}} so that after each batch of spans is processed, we
update the metrics, rather than using a message-passing system.
* Rename {{ClientDropped}} to {{ClientDroppedEstimate}} to reflect the fact
that it's just an estimate of the number of spans the client dropped
* Add {{BenchmarkWriteSpans}} and {{TestWriteSpansRpcs}}
* Fix some shutdown race conditions in datastore that really only could trigger
in unit tests (but were still annoying)
* Add global metrics for spans written and spans dropped by the server. Fix a
case where spans could be dropped by the server but not accounted for in the
per-host metrics.
I get 66374 ns/op when running {{./gobuild.sh test -test.run=SkipRunningTests
-test.bench=BenchmarkDatastoreWrites -cpu 16 -benchtime=1m}}
> htraced: improve datastore serialization
> ----------------------------------------
>
> Key: HTRACE-298
> URL: https://issues.apache.org/jira/browse/HTRACE-298
> Project: HTrace
> Issue Type: Improvement
> Components: htraced
> Affects Versions: 4.0
> Reporter: Colin Patrick McCabe
> Assignee: Colin Patrick McCabe
> Attachments: HTRACE-298.001.patch
>
>
> htraced: speed up datastore serialization
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
