Hi Hulk, I’d like to share some results from an experiment I ran related to Jemalloc memory tuning in Kvrocks, which may be helpful to the gradual memory growth over time.I read a Apache Doris jemalloc analysis article, where they seems to claim more or the the same issue as our. Jemalloc retained memory aggressively by default, leading to high memory usage that the OS could not easily reclaim. They suggest to restart Apache Doris after sometime to reclaim the memory, or use the retain:false Jemalloc option to force Jemalloc return the unused memory to OS.
I tried the option with Kvrocks using the environment variable MALLOC_CONF=retain:false. After applying retain=false and stress tests Kvrocks using redis-benchmark a few dozen times, I observed: - Docker stats memory: consistently under 800MB (was more than 1.3GB if retain:true) - Kvrocks-reported RSS memory: consistently under 500MB - When running other memory-intensive processes, Kvrocks memory usage dropped by ~150MB, suggesting part of the usage was reclaimable cache. This aligns with Jemalloc behavior where disabling retain makes memory more readily returnable to the OS. Maybe the gradual memory growth issue is more a Jemalloc's nature than Kvrocks issue itself? Regarding performance metrics, I did not deeply dig into it yet. A quick benchmark on my side showed no significant difference between the two settings. Please let me know your thoughts on this matter. Quan On May 3, 2025 9:10 PM, from Hulk <[email protected]>Hi Quan, I find a gap in the memory usage in Docker stats, which is about double that of Kvrocks's RSS. You might have a check as well using the command `info memory` via redis-cli. Not sure if it's caused by the different algorithm when counting the memory usage. *docker stats a68a50cf4a82* a68a50cf4a82 tender_jackson 143.39% 635MiB / 14.63GiB 4.24% 22.8GB / 4.7GB 844kB / 21GB 18 *hulk@instance-20241021-041032:~$ redis-cli -p 6666 info memory* # Memory used_memory_rss:247369728 used_memory_rss_human:235.91M used_memory_lua:319488 used_memory_lua_human:312.00K used_memory_startup:33468416 On Sat, 3 May 2025 at 19:02, hulk <[email protected]> wrote: > Hi, Quan > > Thanks for your detailed information. I can reproduce it on my side. I > ran the benchmark a few times, and its memory is around the same(about > 1.3GiB), > but I haven't dived deep into the Jemalloc stats yet. > > On Fri, 2 May 2025 at 17:31, Hong Quan TRAN <[email protected]> wrote: > > > > Hi Hulk, > > > > Sorry for the late reply (we were on holiday). > > > > > I have no clue about why it's up to 1.3 GiB, will try to reproduce > > this on my side and analyze the memory in Jemalloc. > > > > We use Kvrocks as Rspamd's bayes training database. I would attach the > Kvrocks used command stats for more details. > > > > BTW on my side, I tried to run the redis-benchmark tool against Kvrocks > (still the same configuration). After a dozen times running > `redis-benchmark -n 2000000 -c 50` command, Kvrocks memory is upto around > 1.1GB and does not seems to be released automatically after a while. > > > > I hope this would help you ease reproducing the issue. I would be happy > to do more diagnostic and share the output to you if needed. > > > > Regards, > > > > Quan > > > > > > On Apr 27, 2025 4:27 PM, from Hulk <[email protected]> > > > > Hi Benoit, Quan > > > > Sorry for the late reply. I took a look at your configuration, and it > > all looks good. > > > > I have no clue about why it's up to 1.3 GiB, will try to reproduce > > this on my side and analyze the memory in Jemalloc. > > > > Will update you once we have any progress. > > > > On Mon, 21 Apr 2025 at 11:00, hulk <[email protected]> wrote: > > > > > > Thanks all for taking care of this, will have a look. > > > > > > On Mon, 21 Apr 2025 at 10:56, Hong Quan TRAN <[email protected]> > wrote: > > >> > > >> Hi, > > >> > > >> Somehow the format is still bad (although I tested sending it to my > personal mail). > > >> > > >> I would send the Kvrocks config output via an attachment this time. > > >> > > >> Quan > > >> > > >> Vào ngày thg 4 21, 2025 9:39 SA, từ Hong Quan Tran < > [email protected]> > > >> > > >> Hi everyone, > > >> Sorry for the bad formatting output. > > >> I resend the config output. I hope it is fine this time. > > >> > > >> kvrocks@561e141f9b72:/$ redis-cli -p 6379 CONFIG GET '*' 1) > "auto-resize-block-and-sst" 2) "yes" 3) "backup-dir" 4) > "/var/lib/kvrocks/backup" 5) "bgsave-cron" 6) "" 7) "bind" 8) "0.0.0.0" 9) > "cluster-enabled" 10) "no" 11) "compact-cron" 12) "" 13) > "compaction-checker-cron" 14) "" 15) "compaction-checker-range" 16) "" 17) > "daemonize" 18) "no" 19) "db-name" 20) "change.me.db" 21) > "dbsize-scan-cron" 22) "" 23) "dir" 24) "/var/lib/kvrocks" 25) > "force-compact-file-age" 26) "172800" 27) > "force-compact-file-min-deleted-percentage" 28) "10" 29) > "fullsync-recv-file-delay" 30) "0" 31) "histogram-bucket-boundaries" 32) "" > 33) "json-max-nesting-depth" 34) "1024" 35) "json-storage-format" 36) > "json" 37) "log-dir" 38) "/var/lib/kvrocks" 39) "log-level" 40) "info" 41) > "log-retention-days" 42) "-1" 43) "master-use-repl-port" 44) "no" 45) > "masterauth" 46) "" 47) "max-backup-keep-hours" 48) "0" 49) > "max-backup-to-keep" 50) "1" 51) "max-bitmap-to-string-mb" 52) "16" 53) > "max-db-size" 54) "0" 55) "max-io-mb" 56) "0" 57) "max-replication-mb" 58) > "0" 59) "maxclients" 60) "10240" 61) "migrate-batch-rate-limit-mb" 62) "16" > 63) "migrate-batch-size-kb" 64) "16" 65) "migrate-pipeline-size" 66) "16" > 67) "migrate-sequence-gap" 68) "10000" 69) "migrate-speed" 70) "4096" 71) > "migrate-type" 72) "redis-command" 73) "persist-cluster-nodes-enabled" 74) > "yes" 75) "pidfile" 76) "/var/run/kvrocks/kvrocks.pid" 77) "port" 78) > "6379" 79) "profiling-sample-commands" 80) "" 81) "profiling-sample-ratio" > 82) "0" 83) "profiling-sample-record-max-len" 84) "256" 85) > "profiling-sample-record-threshold-ms" 86) "100" 87) "proto-max-bulk-len" > 88) "536870912" 89) "purge-backup-on-fullsync" 90) "no" 91) > "redis-cursor-compatible" 92) "yes" 93) "repl-namespace-enabled" 94) "no" > 95) "replica-announce-ip" 96) "" 97) "replica-announce-port" 98) "0" 99) > "replicaof"100) ""101) "replication-connect-timeout-ms"102) "3100"103) > "replication-recv-timeout-ms"104) "3200"105) "requirepass"106) ""107) > "resp3-enabled"108) "no"109) "rocksdb.avoid_unnecessary_blocking_io"110) > "yes"111) "rocksdb.blob_file_size"112) "268435456"113) > "rocksdb.blob_garbage_collection_age_cutoff"114) "25"115) > "rocksdb.block_cache_size"116) "256"117) "rocksdb.block_cache_type"118) > "lru"119) "rocksdb.block_size"120) "16384"121) > "rocksdb.cache_index_and_filter_blocks"122) "yes"123) > "rocksdb.compaction_readahead_size"124) "2097152"125) > "rocksdb.compression"126) "no"127) "rocksdb.compression_level"128) > "32767"129) "rocksdb.compression_start_level"130) "2"131) > "rocksdb.delayed_write_rate"132) "0"133) > "rocksdb.disable_auto_compactions"134) "no"135) > "rocksdb.dump_malloc_stats"136) "yes"137) "rocksdb.enable_blob_files"138) > "no"139) "rocksdb.enable_blob_garbage_collection"140) "yes"141) > "rocksdb.enable_pipelined_write"142) "no"143) > "rocksdb.level0_file_num_compaction_trigger"144) "4"145) > "rocksdb.level0_slowdown_writes_trigger"146) "20"147) > "rocksdb.level0_stop_writes_trigger"148) "40"149) > "rocksdb.level_compaction_dynamic_level_bytes"150) "no"151) > "rocksdb.max_background_compactions"152) "2"153) > "rocksdb.max_background_flushes"154) "2"155) > "rocksdb.max_background_jobs"156) "4"157) > "rocksdb.max_bytes_for_level_base"158) "268435456"159) > "rocksdb.max_bytes_for_level_multiplier"160) "10"161) > "rocksdb.max_open_files"162) "8096"163) "rocksdb.max_subcompactions"164) > "2"165) "rocksdb.max_total_wal_size"166) "256"167) > "rocksdb.max_write_buffer_number"168) "2"169) > "rocksdb.metadata_block_cache_size"170) "2048"171) > "rocksdb.min_blob_size"172) "4096"173) "rocksdb.partition_filters"174) > "yes"175) "rocksdb.rate_limiter_auto_tuned"176) "yes"177) > "rocksdb.read_options.async_io"178) "yes"179) "rocksdb.row_cache_size"180) > "0"181) "rocksdb.share_metadata_and_subkey_block_cache"182) "yes"183) > "rocksdb.stats_dump_period_sec"184) "0"185) > "rocksdb.subkey_block_cache_size"186) "2048"187) > "rocksdb.target_file_size_base"188) "128"189) "rocksdb.wal_compression"190) > "no"191) "rocksdb.wal_size_limit_mb"192) "16384"193) > "rocksdb.wal_ttl_seconds"194) "10800"195) "rocksdb.write_buffer_size"196) > "32"197) "rocksdb.write_options.disable_wal"198) "no"199) > "rocksdb.write_options.low_pri"200) "no"201) > "rocksdb.write_options.memtable_insert_hint_per_batch"202) "no"203) > "rocksdb.write_options.no_slowdown"204) "no"205) > "rocksdb.write_options.sync"206) "no"207) > "rocksdb.write_options.write_batch_max_bytes"208) "0"209) > "skip-block-cache-deallocation-on-close"210) "no"211) > "slave-empty-db-before-fullsync"212) "no"213) "slave-priority"214) > "100"215) "slave-read-only"216) "yes"217) "slave-serve-stale-data"218) > "yes"219) "slaveof"220) ""221) "slowlog-log-slower-than"222) "200000"223) > "slowlog-max-len"224) "128"225) "socket-fd"226) "-1"227) "supervised"228) > "no"229) "tcp-backlog"230) "511"231) "timeout"232) "0"233) > "tls-auth-clients"234) ""235) "tls-ca-cert-dir"236) ""237) > "tls-ca-cert-file"238) ""239) "tls-cert-file"240) ""241) "tls-ciphers"242) > ""243) "tls-ciphersuites"244) ""245) "tls-key-file"246) ""247) > "tls-key-file-pass"248) ""249) "tls-port"250) "0"251) > "tls-prefer-server-ciphers"252) "no"253) "tls-protocols"254) ""255) > "tls-replication"256) "no"257) "tls-session-cache-size"258) "20480"259) > "tls-session-cache-timeout"260) "300"261) "tls-session-caching"262) > "yes"263) "txn-context-enabled"264) "no"265) "unixsocket"266) ""267) > "unixsocketperm"268) "777"269) "use-rsid-psync"270) "no"271) "workers"272) > "8" > > >> Quan > > >> Vào ngày thg 4 21, 2025 1:52 SA, từ Benoit Tellier < > [email protected]>Careful > > >> > > >> Formating sucks > > >> > > >> Maybe resent it as a plain/text version to avoid this pitfall? > > >> > > >> -- > > >> > > >> Best regards, > > >> > > >> Benoit TELLIER > > >> > > >> General manager of Linagora VIETNAM. > > >> Product owner for Team-Mail product. > > >> Chairman of the Apache James project. > > >> > > >> Mail: [email protected] > > >> Tel: (0033) 6 77 26 04 58 (WhatsApp, Signal) > > >> > > >> > > >> On Apr 20, 2025 3:43 PM, from Hong Quan Tran <[email protected]>Hi > Hulk, > > >> > Thanks for your information, could you please provide the output by > running the command in redis-cli `config get *`? > > >> Please see the output below. > > >> > > >> kvrocks@561e141f9b72:/$ redis-cli -p 6379 CONFIG GET '*' 1) > "auto-resize-block-and-sst" 2) "yes" 3) "backup-dir" 4) > "/var/lib/kvrocks/backup" 5) "bgsave-cron" 6) "" 7) "bind" 8) "0.0.0.0" 9) > "cluster-enabled" 10) "no" 11) "compact-cron" 12) "" 13) > "compaction-checker-cron" 14) "" 15) "compaction-checker-range" 16) "" 17) > "daemonize" 18) "no" 19) "db-name" 20) "change.me.db" 21) > "dbsize-scan-cron" 22) "" 23) "dir" 24) "/var/lib/kvrocks" 25) > "force-compact-file-age" 26) "172800" 27) > "force-compact-file-min-deleted-percentage" 28) "10" 29) > "fullsync-recv-file-delay" 30) "0" 31) "histogram-bucket-boundaries" 32) "" > 33) "json-max-nesting-depth" 34) "1024" 35) "json-storage-format" 36) > "json" 37) "log-dir" 38) "/var/lib/kvrocks" 39) "log-level" 40) "info" 41) > "log-retention-days" 42) "-1" 43) "master-use-repl-port" 44) "no" 45) > "masterauth" 46) "" 47) "max-backup-keep-hours" 48) "0" 49) > "max-backup-to-keep" 50) "1" 51) "max-bitmap-to-string-mb" 52) "16" 53) > "max-db-size" 54) "0" 55) "max-io-mb" 56) "0" 57) "max-replication-mb" 58) > "0" 59) "maxclients" 60) "10240" 61) "migrate-batch-rate-limit-mb" 62) "16" > 63) "migrate-batch-size-kb" 64) "16" 65) "migrate-pipeline-size" 66) "16" > 67) "migrate-sequence-gap" 68) "10000" 69) "migrate-speed" 70) "4096" 71) > "migrate-type" 72) "redis-command" 73) "persist-cluster-nodes-enabled" 74) > "yes" 75) "pidfile" 76) "/var/run/kvrocks/kvrocks.pid" 77) "port" 78) > "6379" 79) "profiling-sample-commands" 80) "" 81) "profiling-sample-ratio" > 82) "0" 83) "profiling-sample-record-max-len" 84) "256" 85) > "profiling-sample-record-threshold-ms" 86) "100" 87) "proto-max-bulk-len" > 88) "536870912" 89) "purge-backup-on-fullsync" 90) "no" 91) > "redis-cursor-compatible" 92) "yes" 93) "repl-namespace-enabled" 94) "no" > 95) "replica-announce-ip" 96) "" 97) "replica-announce-port" 98) "0" 99) > "replicaof"100) ""101) "replication-connect-timeout-ms"102) "3100"103) > "replication-recv-timeout-ms"104) "3200"105) "requirepass"106) ""107) > "resp3-enabled"108) "no"109) "rocksdb.avoid_unnecessary_blocking_io"110) > "yes"111) "rocksdb.blob_file_size"112) "268435456"113) > "rocksdb.blob_garbage_collection_age_cutoff"114) "25"115) > "rocksdb.block_cache_size"116) "256"117) "rocksdb.block_cache_type"118) > "lru"119) "rocksdb.block_size"120) "16384"121) > "rocksdb.cache_index_and_filter_blocks"122) "yes"123) > "rocksdb.compaction_readahead_size"124) "2097152"125) > "rocksdb.compression"126) "no"127) "rocksdb.compression_level"128) > "32767"129) "rocksdb.compression_start_level"130) "2"131) > "rocksdb.delayed_write_rate"132) "0"133) > "rocksdb.disable_auto_compactions"134) "no"135) > "rocksdb.dump_malloc_stats"136) "yes"137) "rocksdb.enable_blob_files"138) > "no"139) "rocksdb.enable_blob_garbage_collection"140) "yes"141) > "rocksdb.enable_pipelined_write"142) "no"143) > "rocksdb.level0_file_num_compaction_trigger"144) "4"145) > "rocksdb.level0_slowdown_writes_trigger"146) "20"147) > "rocksdb.level0_stop_writes_trigger"148) "40"149) > "rocksdb.level_compaction_dynamic_level_bytes"150) "no"151) > "rocksdb.max_background_compactions"152) "2"153) > "rocksdb.max_background_flushes"154) "2"155) > "rocksdb.max_background_jobs"156) "4"157) > "rocksdb.max_bytes_for_level_base"158) "268435456"159) > "rocksdb.max_bytes_for_level_multiplier"160) "10"161) > "rocksdb.max_open_files"162) "8096"163) "rocksdb.max_subcompactions"164) > "2"165) "rocksdb.max_total_wal_size"166) "256"167) > "rocksdb.max_write_buffer_number"168) "2"169) > "rocksdb.metadata_block_cache_size"170) "2048"171) > "rocksdb.min_blob_size"172) "4096"173) "rocksdb.partition_filters"174) > "yes"175) "rocksdb.rate_limiter_auto_tuned"176) "yes"177) > "rocksdb.read_options.async_io"178) "yes"179) "rocksdb.row_cache_size"180) > "0"181) "rocksdb.share_metadata_and_subkey_block_cache"182) "yes"183) > "rocksdb.stats_dump_period_sec"184) "0"185) > "rocksdb.subkey_block_cache_size"186) "2048"187) > "rocksdb.target_file_size_base"188) "128"189) "rocksdb.wal_compression"190) > "no"191) "rocksdb.wal_size_limit_mb"192) "16384"193) > "rocksdb.wal_ttl_seconds"194) "10800"195) "rocksdb.write_buffer_size"196) > "32"197) "rocksdb.write_options.disable_wal"198) "no"199) > "rocksdb.write_options.low_pri"200) "no"201) > "rocksdb.write_options.memtable_insert_hint_per_batch"202) "no"203) > "rocksdb.write_options.no_slowdown"204) "no"205) > "rocksdb.write_options.sync"206) "no"207) > "rocksdb.write_options.write_batch_max_bytes"208) "0"209) > "skip-block-cache-deallocation-on-close"210) "no"211) > "slave-empty-db-before-fullsync"212) "no"213) "slave-priority"214) > "100"215) "slave-read-only"216) "yes"217) "slave-serve-stale-data"218) > "yes"219) "slaveof"220) ""221) "slowlog-log-slower-than"222) "200000"223) > "slowlog-max-len"224) "128"225) "socket-fd"226) "-1"227) "supervised"228) > "no"229) "tcp-backlog"230) "511"231) "timeout"232) "0"233) > "tls-auth-clients"234) ""235) "tls-ca-cert-dir"236) ""237) > "tls-ca-cert-file"238) ""239) "tls-cert-file"240) ""241) "tls-ciphers"242) > ""243) "tls-ciphersuites"244) ""245) "tls-key-file"246) ""247) > "tls-key-file-pass"248) ""249) "tls-port"250) "0"251) > "tls-prefer-server-ciphers"252) "no"253) "tls-protocols"254) ""255) > "tls-replication"256) "no"257) "tls-session-cache-size"258) "20480"259) > "tls-session-cache-timeout"260) "300"261) "tls-session-caching"262) > "yes"263) "txn-context-enabled"264) "no"265) "unixsocket"266) ""267) > "unixsocketperm"268) "777"269) "use-rsid-psync"270) "no"271) "workers"272) > "8" > > >> Vào ngày thg 4 18, 2025 5:49 CH, từ Hulk <[email protected]>Hi > Quan, > > >> > > >> Thanks for your information, could you please provide the output by > running > > >> the command in redis-cli `config get *`? > > >> > > >> On Fri, 18 Apr 2025 at 18:24, Hong Quan TRAN <[email protected]> > wrote: > > >> > > >> > Hi the Kvrocks community, > > >> > We would like to update you our progress on the integration topic. > > >> > We have started to migrate a Redis deployment to Kvrocks for a > week. The > > >> > Kvrocks is running stable I would say. However, we have concern > after > > >> > observing the Kvrocks's memory comsuption as it keeps slowly > consume more > > >> > memory without releasing them back to the OS. At first the memory > usage was > > >> > 500MB, then it is up to 700MB, 1GB, 1.3GB... > > >> > Please note that we already tuning the memory usage using the > > >> > recommendations at github.com/apache/kvrocks/discussions/2767. > > >> > ```kvrocks.conflog-level > > >> > <github.com/apache/kvrocks/discussions/2767.kvrocks.conflog-level> > > >> > infoport 6379rocksdb.max_total_wal_size > 256rocksdb.max_write_buffer_number > > >> > 2rocksdb.write_buffer_size 32rocksdb.block_cache_size 256``` > > >> > We expect the memory consumption would be low and stable with this > applied > > >> > configuration. However, it keeps slowly increasing, which makes us > worry. > > >> > Our memory stats so far: > > >> > ```docker statsCONTAINER ID NAME CPU % MEM USAGE / > > >> > LIMIT MEM % NET I/O BLOCK I/O PIDS561e141f9b72 > > >> > kvrocks 0.14% 1.292GiB / 14.3GiB 9.03% 4.77GB / > > >> > 1.51GB 270kB / 8.08GB 19``` > > >> > ```kvrocks@561e141f9b72:/$ redis-cli -p 6379 info memory# > > >> > > Memoryused_memory_rss:983244800used_memory_rss_human:937.70Mused_memory_lua:59392used_memory_lua_human:58.00Kused_memory_startup:63860736``` > > >> > FYI we are using Kvrocks version 2.11.1. > > >> > Could you help provide insights and advice on the Kvrocks memory > > >> > management topic? How to limit the Kvrocks memory usage effectively? > > >> > > > >> > Best regards, > > >> > Quan > > >> > > > >> > > > >> > > >> -- > > >> Best Regards, > > >> - *Hulk Lin* > > > > > > > > > > > > -- > > > Best Regards, > > > - Hulk Lin > > > > > > > > -- > > Best Regards, > > - Hulk Lin > > > > > -- > Best Regards, > - Hulk Lin > -- Best Regards, - *Hulk Lin*
