[
https://issues.apache.org/jira/browse/KUDU-3197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17235147#comment-17235147
]
wangningito edited comment on KUDU-3197 at 11/19/20, 4:02 AM:
--------------------------------------------------------------
A minor optimization may be applied in this case, I'll try to working on it
firstly.
We may alter schema many times without any data ingestion. So maybe only the
latest one schema is needed to replay during next bootstrap.
was (Author: wangning):
A minor optimization may be applied in this case, I'll try to working on it
firstly.
We may alter schema many times without any data ingestion. So only the latest
one schema is needed for next bootstrap's rewinding.
> Tablet keeps all history schemas in memory may result in high memory
> consumption
> --------------------------------------------------------------------------------
>
> Key: KUDU-3197
> URL: https://issues.apache.org/jira/browse/KUDU-3197
> Project: Kudu
> Issue Type: Improvement
> Components: tablet
> Affects Versions: 1.12.0
> Reporter: wangningito
> Assignee: wangningito
> Priority: Minor
> Attachments: image-2020-09-25-14-45-33-402.png,
> image-2020-09-25-14-49-30-913.png, image-2020-09-25-15-05-44-948.png
>
>
> In case of high frequency of updating table, memory consumption of
> kudu-tserver may be very high, and the memory in not tracked in the memory
> page.
> This is the memory usage of a tablet, the memory consumption of tablet-xxx‘s
> peak is 3.6G， but none of its' childrens' memory can reach.
> !image-2020-09-25-14-45-33-402.png!
> So I use pprof to get the heap sampling. The tserver started for long but the
> memory is still consuming by TabletBootstrap:PlayAlterSchemaRequest.
> !image-2020-09-25-14-49-30-913.png!
> I change the `old_schemas_` in tablet_metadata.h to a fixed size vector,
> // Previous values of 'schema_'.
> // These are currently kept alive forever, under the assumption that
> // a given tablet won't have thousands of "alter table" calls.
> // They are kept alive so that callers of schema() don't need to
> // worry about reference counting or locking.
> std::vector<Schema*> old_schemas_;
> The heap sampling then becomes
> !image-2020-09-25-15-05-44-948.png!
> So, to make application layer more flexible, it could be better to make the
> size of the old_schemas configurable.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
