Colin Patrick McCabe created HTRACE-308:
-------------------------------------------
Summary: Deserialize WriteSpans requests incrementally rather than
all at once to optimize GC
Key: HTRACE-308
URL: https://issues.apache.org/jira/browse/HTRACE-308
Project: HTrace
Issue Type: Improvement
Components: htraced
Affects Versions: 4.0
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
We should deserialize WriteSpans requests incrementally rather than all at
once. Currently, we can deserialize 63 MB of spans all at once, which
immediately creates somewhere between 60k and 600k spans, depending on span
size. This is hard on the garbage collector because it's a lot of allocations
all at once, and because it allocates a very large array to hold it all.
It would be better to deserialize spans one at a time and feed them into the
datastore via the BatchIngestor. This will ensure that we don't have to
allocate giant arrays of spans all at once. If the datastore lags behind the
rate of span ingestion, this will avoid us needing to allocate a bunch of
memory "up front" which can lead to further slowdowns due to GC.
Also, we should reuse buffers for the RPC handlers, and use buffering while
deserializing to avoid making lots of small reads from the socket.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
