[
https://issues.apache.org/jira/browse/KUDU-2238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ZhangZhen updated KUDU-2238:
----------------------------
Description:
I have a table with many updates, its DMS consumes a lot memory and cause “Soft
Memory Limit Exceed”. I check the /mem-trackers on the tablet server, one of
its DMS consumes about 3G memory, but check the /maintenance-manager, its
FlushDeltaMemStoresOp can only free 763B anchored memory and perf_improvement
is 0. Is this normal? I know Kudu is not optimized for updates, but still
confused why the DMS won’t be flushed under memory pressure.
Infos from /mem-trackers:
tablet-5941a8bb934e4686abd1bfff9e35c860 server none 3.00G 3.00G
txn_tracker tablet-5941a8bb934e4686abd1bfff9e35c860 64.00M 0B 1.67M
MemRowSet-339 tablet-5941a8bb934e4686abd1bfff9e35c860 none 265B 265B
DeltaMemStores tablet-5941a8bb934e4686abd1bfff9e35c860 none 3.00G
3.00G
!memory_consumed.png|thumbnail!
Infos from /maintenance-manager
FlushDeltaMemStoresOp(5941a8bb934e4686abd1bfff9e35c860) true 763B
511.15M 0
!memory_anchored.png|thumbnail!
The tablet 5941a8bb934e4686abd1bfff9e35c860 has 16 RowSets in total
Some configs of MM:
--enable_maintenance_manager=true
--log_target_replay_size_mb=1024
--maintenance_manager_history_size=8
--maintenance_manager_num_threads=6
--maintenance_manager_polling_interval_ms=50
was:
I have a table with many updates, its DMS consumes a lot memory and cause “Soft
Memory Limit Exceed”. I check the /mem-trackers on the tablet server, one of
its DMS consumes about 3G memory, but check the /maintenance-manager, its
FlushDeltaMemStoresOp can only free 763B anchored memory and perf_improvement
is 0. Is this normal? I know Kudu is not optimized for updates, but still
confused why the DMS won’t be flushed under memory pressure.
Infos from /mem-trackers:
tablet-5941a8bb934e4686abd1bfff9e35c860 server none 3.00G 3.00G
txn_tracker tablet-5941a8bb934e4686abd1bfff9e35c860 64.00M 0B 1.67M
MemRowSet-339 tablet-5941a8bb934e4686abd1bfff9e35c860 none 265B 265B
DeltaMemStores tablet-5941a8bb934e4686abd1bfff9e35c860 none 3.00G
3.00G
!attachment-name.jpg|thumbnail!
Infos from /maintenance-manager
FlushDeltaMemStoresOp(5941a8bb934e4686abd1bfff9e35c860) true 763B
511.15M 0
!attachment-name.jpg|thumbnail!
The tablet 5941a8bb934e4686abd1bfff9e35c860 has 16 RowSets in total
Some configs of MM:
--enable_maintenance_manager=true
--log_target_replay_size_mb=1024
--maintenance_manager_history_size=8
--maintenance_manager_num_threads=6
--maintenance_manager_polling_interval_ms=50
> Big DMS not flush under memory pressure
> ---------------------------------------
>
> Key: KUDU-2238
> URL: https://issues.apache.org/jira/browse/KUDU-2238
> Project: Kudu
> Issue Type: Bug
> Affects Versions: 1.3.0
> Environment: CentOS6.5 Linux 2.6.32-431
> Kudu1.3.0
> GitCommit 00813f96b9cb
> Reporter: ZhangZhen
> Attachments: memory_anchored.png, memory_consumed.png
>
>
> I have a table with many updates, its DMS consumes a lot memory and cause
> “Soft Memory Limit Exceed”. I check the /mem-trackers on the tablet server,
> one of its DMS consumes about 3G memory, but check the /maintenance-manager,
> its FlushDeltaMemStoresOp can only free 763B anchored memory and
> perf_improvement is 0. Is this normal? I know Kudu is not optimized for
> updates, but still confused why the DMS won’t be flushed under memory
> pressure.
> Infos from /mem-trackers:
> tablet-5941a8bb934e4686abd1bfff9e35c860 server none 3.00G 3.00G
> txn_tracker tablet-5941a8bb934e4686abd1bfff9e35c860 64.00M 0B
> 1.67M
> MemRowSet-339 tablet-5941a8bb934e4686abd1bfff9e35c860 none 265B
> 265B
> DeltaMemStores tablet-5941a8bb934e4686abd1bfff9e35c860 none 3.00G
> 3.00G
> !memory_consumed.png|thumbnail!
> Infos from /maintenance-manager
> FlushDeltaMemStoresOp(5941a8bb934e4686abd1bfff9e35c860) true 763B
> 511.15M 0
> !memory_anchored.png|thumbnail!
> The tablet 5941a8bb934e4686abd1bfff9e35c860 has 16 RowSets in total
> Some configs of MM:
> --enable_maintenance_manager=true
> --log_target_replay_size_mb=1024
> --maintenance_manager_history_size=8
> --maintenance_manager_num_threads=6
> --maintenance_manager_polling_interval_ms=50
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
