[
https://issues.apache.org/jira/browse/HTRACE-308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15030334#comment-15030334
]
Colin Patrick McCabe edited comment on HTRACE-308 at 11/28/15 1:07 AM:
-----------------------------------------------------------------------
I'm doing some testing on a 300-node cluster, and I can confirm that this
change makes a big difference. Memory consumption is staying mostly flat
around 350/375 MB or so rather than ballooning. I am running with a few other
optimizations, but I think this is the big one. I am generating about 416
spans per second and have about 300202 total in htraced. I am using 11 leveldb
instance on an 8 core machine with hyperthreading.
was (Author: cmccabe):
I'm doing some testing on a 300-node cluster, and I can confirm that this
change makes a big difference. Memory consumption is staying mostly flat
around 350 MB or so rather than ballooning. I am running with a few other
optimizations, but I think this is the big one.
> Deserialize WriteSpans requests incrementally rather than all at once to
> optimize GC
> ------------------------------------------------------------------------------------
>
> Key: HTRACE-308
> URL: https://issues.apache.org/jira/browse/HTRACE-308
> Project: HTrace
> Issue Type: Improvement
> Components: htraced
> Affects Versions: 4.0
> Reporter: Colin Patrick McCabe
> Assignee: Colin Patrick McCabe
> Attachments: HTRACE-308.001.patch, HTRACE-308.002.patch,
> HTRACE-308.003.patch, HTRACE-308.004.patch
>
>
> We should deserialize WriteSpans requests incrementally rather than all at
> once. Currently, we can deserialize 63 MB of spans all at once, which
> immediately creates somewhere between 60k and 600k spans, depending on span
> size. This is hard on the garbage collector because it's a lot of
> allocations all at once, and because it allocates a very large array to hold
> it all.
> It would be better to deserialize spans one at a time and feed them into the
> datastore via the BatchIngestor. This will ensure that we don't have to
> allocate giant arrays of spans all at once. If the datastore lags behind the
> rate of span ingestion, this will avoid us needing to allocate a bunch of
> memory "up front" which can lead to further slowdowns due to GC.
> Also, we should reuse buffers for the RPC handlers, and use buffering while
> deserializing to avoid making lots of small reads from the socket.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
