[
https://issues.apache.org/jira/browse/HTRACE-308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Colin Patrick McCabe updated HTRACE-308:
----------------------------------------
Attachment: HTRACE-308.004.patch
> Deserialize WriteSpans requests incrementally rather than all at once to
> optimize GC
> ------------------------------------------------------------------------------------
>
> Key: HTRACE-308
> URL: https://issues.apache.org/jira/browse/HTRACE-308
> Project: HTrace
> Issue Type: Improvement
> Components: htraced
> Affects Versions: 4.0
> Reporter: Colin Patrick McCabe
> Assignee: Colin Patrick McCabe
> Attachments: HTRACE-308.001.patch, HTRACE-308.002.patch,
> HTRACE-308.003.patch, HTRACE-308.004.patch
>
>
> We should deserialize WriteSpans requests incrementally rather than all at
> once. Currently, we can deserialize 63 MB of spans all at once, which
> immediately creates somewhere between 60k and 600k spans, depending on span
> size. This is hard on the garbage collector because it's a lot of
> allocations all at once, and because it allocates a very large array to hold
> it all.
> It would be better to deserialize spans one at a time and feed them into the
> datastore via the BatchIngestor. This will ensure that we don't have to
> allocate giant arrays of spans all at once. If the datastore lags behind the
> rate of span ingestion, this will avoid us needing to allocate a bunch of
> memory "up front" which can lead to further slowdowns due to GC.
> Also, we should reuse buffers for the RPC handlers, and use buffering while
> deserializing to avoid making lots of small reads from the socket.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
