[jira] [Commented] (KUDU-3463) KuduMaster leader consumes too much memory
[
https://issues.apache.org/jira/browse/KUDU-3463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17713282#comment-17713282
]
Alexey Serbin commented on KUDU-3463:
-
[~weizisheng],
Just two more cents in addition to what [~zhangyifan27] already pointed.
Even without deleting/creating tables in the system catalog, {{kudu-master}}
process might consume a lot of memory when receiving many updates on the tablet
status and DDLs like adding/dropping/changing columns in a table. See
[KUDU-3406|https://issues.apache.org/jira/browse/KUDU-3406] and
[KUDU-3429|https://issues.apache.org/jira/browse/KUDU-3429] for details.
You can easily address that by setting {{\-\-tablet_history_max_age_sec=900}}
for kudu-masters. In addition, consider picking up
[4537be98d|https://github.com/apache/kudu/commit/4537be98d0542d99ecbe4674197b9d894c3eb4e5]
if you don't want to override the {{\-\-tablet_history_max_age_sec}} for
{{kudu-master}} processes.
> KuduMaster leader consumes too much memory
> --
>
> Key: KUDU-3463
> URL: https://issues.apache.org/jira/browse/KUDU-3463
> Project: Kudu
> Issue Type: Bug
>Affects Versions: 1.12.0
>Reporter: Weizisheng
>Priority: Major
> Attachments: heap321.txt
>
>
> We rarely face a suspected memory leak on a cluster with 3-master and
> 4-tserver, 800 tables and 3000 tablets. Leader master consume 50GB memory
> while the other two only take one tenth of it ...
> From the last 5 days, leader's memory usage grows 3%+
> attachment pprof heap
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (KUDU-3463) KuduMaster leader consumes too much memory
[ https://issues.apache.org/jira/browse/KUDU-3463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17713075#comment-17713075 ] Weizisheng commented on KUDU-3463: -- Actually it is, data/data/* grows from 3.2G to 3.4G while memory usage decrease > KuduMaster leader consumes too much memory > -- > > Key: KUDU-3463 > URL: https://issues.apache.org/jira/browse/KUDU-3463 > Project: Kudu > Issue Type: Bug >Affects Versions: 1.12.0 >Reporter: Weizisheng >Priority: Major > Attachments: heap321.txt > > > We rarely face a suspected memory leak on a cluster with 3-master and > 4-tserver, 800 tables and 3000 tablets. Leader master consume 50GB memory > while the other two only take one tenth of it ... > From the last 5 days, leader's memory usage grows 3%+ > attachment pprof heap > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KUDU-3463) KuduMaster leader consumes too much memory
[ https://issues.apache.org/jira/browse/KUDU-3463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17713068#comment-17713068 ] YifanZhang commented on KUDU-3463: -- [~weizisheng] The metadata of deleted tables and tablets should be deleted from both memory and disk if the setting of 'enable_metadata_cleanup_for_deleted_tables_and_tablets' is true. For deleting them from persistent disks, we delete corresponding entries from the sys.catalog table where the metadata stored in: [https://github.com/apache/kudu/blob/a3a7c97be031f8fc32402e430eff1a89c19dbdfb/src/kudu/master/catalog_manager.cc#L6099.] Did you find that the data on disks did not decrease after deleting tables? > KuduMaster leader consumes too much memory > -- > > Key: KUDU-3463 > URL: https://issues.apache.org/jira/browse/KUDU-3463 > Project: Kudu > Issue Type: Bug >Affects Versions: 1.12.0 >Reporter: Weizisheng >Priority: Major > Attachments: heap321.txt > > > We rarely face a suspected memory leak on a cluster with 3-master and > 4-tserver, 800 tables and 3000 tablets. Leader master consume 50GB memory > while the other two only take one tenth of it ... > From the last 5 days, leader's memory usage grows 3%+ > attachment pprof heap > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KUDU-3463) KuduMaster leader consumes too much memory
[ https://issues.apache.org/jira/browse/KUDU-3463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17712949#comment-17712949 ] Weizisheng commented on KUDU-3463: -- [~zhangyifan27] when learning KUDU-3344 i find that only table and tablet was deleted in memory but not not in disk data. Did you intend to do this? or maybe i missed something > KuduMaster leader consumes too much memory > -- > > Key: KUDU-3463 > URL: https://issues.apache.org/jira/browse/KUDU-3463 > Project: Kudu > Issue Type: Bug >Affects Versions: 1.12.0 >Reporter: Weizisheng >Priority: Major > Attachments: heap321.txt > > > We rarely face a suspected memory leak on a cluster with 3-master and > 4-tserver, 800 tables and 3000 tablets. Leader master consume 50GB memory > while the other two only take one tenth of it ... > From the last 5 days, leader's memory usage grows 3%+ > attachment pprof heap > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KUDU-3463) KuduMaster leader consumes too much memory
[ https://issues.apache.org/jira/browse/KUDU-3463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17712858#comment-17712858 ] Weizisheng commented on KUDU-3463: -- [~zhangyifan27] KUDU-3344 works on Kudu 1.12 with no memory grows in the past two days, i'll test it a little longer, thanks > KuduMaster leader consumes too much memory > -- > > Key: KUDU-3463 > URL: https://issues.apache.org/jira/browse/KUDU-3463 > Project: Kudu > Issue Type: Bug >Affects Versions: 1.12.0 >Reporter: Weizisheng >Priority: Major > Attachments: heap321.txt > > > We rarely face a suspected memory leak on a cluster with 3-master and > 4-tserver, 800 tables and 3000 tablets. Leader master consume 50GB memory > while the other two only take one tenth of it ... > From the last 5 days, leader's memory usage grows 3%+ > attachment pprof heap > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KUDU-3463) KuduMaster leader consumes too much memory
[ https://issues.apache.org/jira/browse/KUDU-3463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17712214#comment-17712214 ] Weizisheng commented on KUDU-3463: -- [~zhangyifan27] thanks for reply. Actually it's weired that kudu master keep deleted table and tablet metadata in memory or even in the disk. [Kudu disk usage|https://kudu.apache.org/docs/troubleshooting.html#disk_space_usage] says kudu uses sparse file to store data, which i think that kudu master removes deleted table info from the disk... Anyway i'll have a look on KUDU-3344 > KuduMaster leader consumes too much memory > -- > > Key: KUDU-3463 > URL: https://issues.apache.org/jira/browse/KUDU-3463 > Project: Kudu > Issue Type: Bug >Affects Versions: 1.12.0 >Reporter: Weizisheng >Priority: Major > Attachments: heap321.txt > > > We rarely face a suspected memory leak on a cluster with 3-master and > 4-tserver, 800 tables and 3000 tablets. Leader master consume 50GB memory > while the other two only take one tenth of it ... > From the last 5 days, leader's memory usage grows 3%+ > attachment pprof heap > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KUDU-3463) KuduMaster leader consumes too much memory
[ https://issues.apache.org/jira/browse/KUDU-3463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17712196#comment-17712196 ] YifanZhang commented on KUDU-3463: -- [~weizisheng]This issue seems related to KUDU-3097 and KUDU-3344. The fix for KUDU-3344 will be included in Kudu 1.17.0 release. I think it's not a memory leak in the Kudu leader master. The reason is too many table and tablet metadata are preserved in the memory. Maybe you can try to pick the changes in KUDU-3344 and set --enable_metadata_cleanup_for_deleted_tables_and_tablets=true for Kudu masters, to see if the memory usage can be reduced. > KuduMaster leader consumes too much memory > -- > > Key: KUDU-3463 > URL: https://issues.apache.org/jira/browse/KUDU-3463 > Project: Kudu > Issue Type: Bug >Affects Versions: 1.12.0 >Reporter: Weizisheng >Priority: Major > Attachments: heap321.txt > > > We rarely face a suspected memory leak on a cluster with 3-master and > 4-tserver, 800 tables and 3000 tablets. Leader master consume 50GB memory > while the other two only take one tenth of it ... > From the last 5 days, leader's memory usage grows 3%+ > attachment pprof heap > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KUDU-3463) KuduMaster leader consumes too much memory
[ https://issues.apache.org/jira/browse/KUDU-3463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17712170#comment-17712170 ] Weizisheng commented on KUDU-3463: -- [~abukor] [~hahao] Do you have any ideas about this? thanks... > KuduMaster leader consumes too much memory > -- > > Key: KUDU-3463 > URL: https://issues.apache.org/jira/browse/KUDU-3463 > Project: Kudu > Issue Type: Bug >Affects Versions: 1.12.0 >Reporter: Weizisheng >Priority: Major > Attachments: heap321.txt > > > We rarely face a suspected memory leak on a cluster with 3-master and > 4-tserver, 800 tables and 3000 tablets. Leader master consume 50GB memory > while the other two only take one tenth of it ... > From the last 5 days, leader's memory usage grows 3%+ > attachment pprof heap > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KUDU-3463) KuduMaster leader consumes too much memory
[ https://issues.apache.org/jira/browse/KUDU-3463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17712169#comment-17712169 ] Weizisheng commented on KUDU-3463: -- Reproduced: creating & deleting table continuously, the leader memory usage grows from xxMB to xGB, and it will continues to grow... kill the leader, another one becomes leader and it;s memory usage becomes instantly grows to xGB, most of it's memory usage is consumed in TableLoader.VisitTable and TabletLoader.VisitTablet I'm wondering it's the normal impl or memory leak.. > KuduMaster leader consumes too much memory > -- > > Key: KUDU-3463 > URL: https://issues.apache.org/jira/browse/KUDU-3463 > Project: Kudu > Issue Type: Bug >Affects Versions: 1.12.0 >Reporter: Weizisheng >Priority: Major > Attachments: heap321.txt > > > We rarely face a suspected memory leak on a cluster with 3-master and > 4-tserver, 800 tables and 3000 tablets. Leader master consume 50GB memory > while the other two only take one tenth of it ... > From the last 5 days, leader's memory usage grows 3%+ > attachment pprof heap > -- This message was sent by Atlassian Jira (v8.20.10#820010)
