Hi Hulk, Sorry for the late reply (we were on holiday). > I have no clue about why it's up to 1.3 GiB, will try to reproducethis on my > side and analyze the memory in Jemalloc. We use Kvrocks as Rspamd's bayes training database. I would attach the Kvrocks used command stats for more details. BTW on my side, I tried to run the redis-benchmark tool against Kvrocks (still the same configuration). After a dozen times running `redis-benchmark -n 2000000 -c 50` command, Kvrocks memory is upto around 1.1GB and does not seems to be released automatically after a while. I hope this would help you ease reproducing the issue. I would be happy to do more diagnostic and share the output to you if needed. Regards, Quan
On Apr 27, 2025 4:27 PM, from Hulk <[email protected]>Hi Benoit, Quan Sorry for the late reply. I took a look at your configuration, and it all looks good. I have no clue about why it's up to 1.3 GiB, will try to reproduce this on my side and analyze the memory in Jemalloc. Will update you once we have any progress. On Mon, 21 Apr 2025 at 11:00, hulk <[email protected]> wrote: > > Thanks all for taking care of this, will have a look. > > On Mon, 21 Apr 2025 at 10:56, Hong Quan TRAN <[email protected]> wrote: >> >> Hi, >> >> Somehow the format is still bad (although I tested sending it to my personal >> mail). >> >> I would send the Kvrocks config output via an attachment this time. >> >> Quan >> >> Vào ngày thg 4 21, 2025 9:39 SA, từ Hong Quan Tran <[email protected]> >> >> Hi everyone, >> Sorry for the bad formatting output. >> I resend the config output. I hope it is fine this time. >> >> kvrocks@561e141f9b72:/$ redis-cli -p 6379 CONFIG GET '*' 1) >> "auto-resize-block-and-sst" 2) "yes" 3) "backup-dir" 4) >> "/var/lib/kvrocks/backup" 5) "bgsave-cron" 6) "" 7) "bind" 8) "0.0.0.0" >> 9) "cluster-enabled" 10) "no" 11) "compact-cron" 12) "" 13) >> "compaction-checker-cron" 14) "" 15) "compaction-checker-range" 16) "" 17) >> "daemonize" 18) "no" 19) "db-name" 20) "change.me.db" 21) "dbsize-scan-cron" >> 22) "" 23) "dir" 24) "/var/lib/kvrocks" 25) "force-compact-file-age" 26) >> "172800" 27) "force-compact-file-min-deleted-percentage" 28) "10" 29) >> "fullsync-recv-file-delay" 30) "0" 31) "histogram-bucket-boundaries" 32) "" >> 33) "json-max-nesting-depth" 34) "1024" 35) "json-storage-format" 36) "json" >> 37) "log-dir" 38) "/var/lib/kvrocks" 39) "log-level" 40) "info" 41) >> "log-retention-days" 42) "-1" 43) "master-use-repl-port" 44) "no" 45) >> "masterauth" 46) "" 47) "max-backup-keep-hours" 48) "0" 49) >> "max-backup-to-keep" 50) "1" 51) "max-bitmap-to-string-mb" 52) "16" 53) >> "max-db-size" 54) "0" 55) "max-io-mb" 56) "0" 57) "max-replication-mb" 58) >> "0" 59) "maxclients" 60) "10240" 61) "migrate-batch-rate-limit-mb" 62) "16" >> 63) "migrate-batch-size-kb" 64) "16" 65) "migrate-pipeline-size" 66) "16" >> 67) "migrate-sequence-gap" 68) "10000" 69) "migrate-speed" 70) "4096" 71) >> "migrate-type" 72) "redis-command" 73) "persist-cluster-nodes-enabled" 74) >> "yes" 75) "pidfile" 76) "/var/run/kvrocks/kvrocks.pid" 77) "port" 78) "6379" >> 79) "profiling-sample-commands" 80) "" 81) "profiling-sample-ratio" 82) "0" >> 83) "profiling-sample-record-max-len" 84) "256" 85) >> "profiling-sample-record-threshold-ms" 86) "100" 87) "proto-max-bulk-len" >> 88) "536870912" 89) "purge-backup-on-fullsync" 90) "no" 91) >> "redis-cursor-compatible" 92) "yes" 93) "repl-namespace-enabled" 94) "no" >> 95) "replica-announce-ip" 96) "" 97) "replica-announce-port" 98) "0" 99) >> "replicaof"100) ""101) "replication-connect-timeout-ms"102) "3100"103) >> "replication-recv-timeout-ms"104) "3200"105) "requirepass"106) ""107) >> "resp3-enabled"108) "no"109) "rocksdb.avoid_unnecessary_blocking_io"110) >> "yes"111) "rocksdb.blob_file_size"112) "268435456"113) >> "rocksdb.blob_garbage_collection_age_cutoff"114) "25"115) >> "rocksdb.block_cache_size"116) "256"117) "rocksdb.block_cache_type"118) >> "lru"119) "rocksdb.block_size"120) "16384"121) >> "rocksdb.cache_index_and_filter_blocks"122) "yes"123) >> "rocksdb.compaction_readahead_size"124) "2097152"125) >> "rocksdb.compression"126) "no"127) "rocksdb.compression_level"128) >> "32767"129) "rocksdb.compression_start_level"130) "2"131) >> "rocksdb.delayed_write_rate"132) "0"133) >> "rocksdb.disable_auto_compactions"134) "no"135) >> "rocksdb.dump_malloc_stats"136) "yes"137) "rocksdb.enable_blob_files"138) >> "no"139) "rocksdb.enable_blob_garbage_collection"140) "yes"141) >> "rocksdb.enable_pipelined_write"142) "no"143) >> "rocksdb.level0_file_num_compaction_trigger"144) "4"145) >> "rocksdb.level0_slowdown_writes_trigger"146) "20"147) >> "rocksdb.level0_stop_writes_trigger"148) "40"149) >> "rocksdb.level_compaction_dynamic_level_bytes"150) "no"151) >> "rocksdb.max_background_compactions"152) "2"153) >> "rocksdb.max_background_flushes"154) "2"155) >> "rocksdb.max_background_jobs"156) "4"157) >> "rocksdb.max_bytes_for_level_base"158) "268435456"159) >> "rocksdb.max_bytes_for_level_multiplier"160) "10"161) >> "rocksdb.max_open_files"162) "8096"163) "rocksdb.max_subcompactions"164) >> "2"165) "rocksdb.max_total_wal_size"166) "256"167) >> "rocksdb.max_write_buffer_number"168) "2"169) >> "rocksdb.metadata_block_cache_size"170) "2048"171) >> "rocksdb.min_blob_size"172) "4096"173) "rocksdb.partition_filters"174) >> "yes"175) "rocksdb.rate_limiter_auto_tuned"176) "yes"177) >> "rocksdb.read_options.async_io"178) "yes"179) "rocksdb.row_cache_size"180) >> "0"181) "rocksdb.share_metadata_and_subkey_block_cache"182) "yes"183) >> "rocksdb.stats_dump_period_sec"184) "0"185) >> "rocksdb.subkey_block_cache_size"186) "2048"187) >> "rocksdb.target_file_size_base"188) "128"189) "rocksdb.wal_compression"190) >> "no"191) "rocksdb.wal_size_limit_mb"192) "16384"193) >> "rocksdb.wal_ttl_seconds"194) "10800"195) "rocksdb.write_buffer_size"196) >> "32"197) "rocksdb.write_options.disable_wal"198) "no"199) >> "rocksdb.write_options.low_pri"200) "no"201) >> "rocksdb.write_options.memtable_insert_hint_per_batch"202) "no"203) >> "rocksdb.write_options.no_slowdown"204) "no"205) >> "rocksdb.write_options.sync"206) "no"207) >> "rocksdb.write_options.write_batch_max_bytes"208) "0"209) >> "skip-block-cache-deallocation-on-close"210) "no"211) >> "slave-empty-db-before-fullsync"212) "no"213) "slave-priority"214) "100"215) >> "slave-read-only"216) "yes"217) "slave-serve-stale-data"218) "yes"219) >> "slaveof"220) ""221) "slowlog-log-slower-than"222) "200000"223) >> "slowlog-max-len"224) "128"225) "socket-fd"226) "-1"227) "supervised"228) >> "no"229) "tcp-backlog"230) "511"231) "timeout"232) "0"233) >> "tls-auth-clients"234) ""235) "tls-ca-cert-dir"236) ""237) >> "tls-ca-cert-file"238) ""239) "tls-cert-file"240) ""241) "tls-ciphers"242) >> ""243) "tls-ciphersuites"244) ""245) "tls-key-file"246) ""247) >> "tls-key-file-pass"248) ""249) "tls-port"250) "0"251) >> "tls-prefer-server-ciphers"252) "no"253) "tls-protocols"254) ""255) >> "tls-replication"256) "no"257) "tls-session-cache-size"258) "20480"259) >> "tls-session-cache-timeout"260) "300"261) "tls-session-caching"262) >> "yes"263) "txn-context-enabled"264) "no"265) "unixsocket"266) ""267) >> "unixsocketperm"268) "777"269) "use-rsid-psync"270) "no"271) "workers"272) >> "8" >> Quan >> Vào ngày thg 4 21, 2025 1:52 SA, từ Benoit Tellier >> <[email protected]>Careful >> >> Formating sucks >> >> Maybe resent it as a plain/text version to avoid this pitfall? >> >> -- >> >> Best regards, >> >> Benoit TELLIER >> >> General manager of Linagora VIETNAM. >> Product owner for Team-Mail product. >> Chairman of the Apache James project. >> >> Mail: [email protected] >> Tel: (0033) 6 77 26 04 58 (WhatsApp, Signal) >> >> >> On Apr 20, 2025 3:43 PM, from Hong Quan Tran <[email protected]>Hi Hulk, >> > Thanks for your information, could you please provide the output by >> > running the command in redis-cli `config get *`? >> Please see the output below. >> >> kvrocks@561e141f9b72:/$ redis-cli -p 6379 CONFIG GET '*' 1) >> "auto-resize-block-and-sst" 2) "yes" 3) "backup-dir" 4) >> "/var/lib/kvrocks/backup" 5) "bgsave-cron" 6) "" 7) "bind" 8) "0.0.0.0" >> 9) "cluster-enabled" 10) "no" 11) "compact-cron" 12) "" 13) >> "compaction-checker-cron" 14) "" 15) "compaction-checker-range" 16) "" 17) >> "daemonize" 18) "no" 19) "db-name" 20) "change.me.db" 21) "dbsize-scan-cron" >> 22) "" 23) "dir" 24) "/var/lib/kvrocks" 25) "force-compact-file-age" 26) >> "172800" 27) "force-compact-file-min-deleted-percentage" 28) "10" 29) >> "fullsync-recv-file-delay" 30) "0" 31) "histogram-bucket-boundaries" 32) "" >> 33) "json-max-nesting-depth" 34) "1024" 35) "json-storage-format" 36) "json" >> 37) "log-dir" 38) "/var/lib/kvrocks" 39) "log-level" 40) "info" 41) >> "log-retention-days" 42) "-1" 43) "master-use-repl-port" 44) "no" 45) >> "masterauth" 46) "" 47) "max-backup-keep-hours" 48) "0" 49) >> "max-backup-to-keep" 50) "1" 51) "max-bitmap-to-string-mb" 52) "16" 53) >> "max-db-size" 54) "0" 55) "max-io-mb" 56) "0" 57) "max-replication-mb" 58) >> "0" 59) "maxclients" 60) "10240" 61) "migrate-batch-rate-limit-mb" 62) "16" >> 63) "migrate-batch-size-kb" 64) "16" 65) "migrate-pipeline-size" 66) "16" >> 67) "migrate-sequence-gap" 68) "10000" 69) "migrate-speed" 70) "4096" 71) >> "migrate-type" 72) "redis-command" 73) "persist-cluster-nodes-enabled" 74) >> "yes" 75) "pidfile" 76) "/var/run/kvrocks/kvrocks.pid" 77) "port" 78) "6379" >> 79) "profiling-sample-commands" 80) "" 81) "profiling-sample-ratio" 82) "0" >> 83) "profiling-sample-record-max-len" 84) "256" 85) >> "profiling-sample-record-threshold-ms" 86) "100" 87) "proto-max-bulk-len" >> 88) "536870912" 89) "purge-backup-on-fullsync" 90) "no" 91) >> "redis-cursor-compatible" 92) "yes" 93) "repl-namespace-enabled" 94) "no" >> 95) "replica-announce-ip" 96) "" 97) "replica-announce-port" 98) "0" 99) >> "replicaof"100) ""101) "replication-connect-timeout-ms"102) "3100"103) >> "replication-recv-timeout-ms"104) "3200"105) "requirepass"106) ""107) >> "resp3-enabled"108) "no"109) "rocksdb.avoid_unnecessary_blocking_io"110) >> "yes"111) "rocksdb.blob_file_size"112) "268435456"113) >> "rocksdb.blob_garbage_collection_age_cutoff"114) "25"115) >> "rocksdb.block_cache_size"116) "256"117) "rocksdb.block_cache_type"118) >> "lru"119) "rocksdb.block_size"120) "16384"121) >> "rocksdb.cache_index_and_filter_blocks"122) "yes"123) >> "rocksdb.compaction_readahead_size"124) "2097152"125) >> "rocksdb.compression"126) "no"127) "rocksdb.compression_level"128) >> "32767"129) "rocksdb.compression_start_level"130) "2"131) >> "rocksdb.delayed_write_rate"132) "0"133) >> "rocksdb.disable_auto_compactions"134) "no"135) >> "rocksdb.dump_malloc_stats"136) "yes"137) "rocksdb.enable_blob_files"138) >> "no"139) "rocksdb.enable_blob_garbage_collection"140) "yes"141) >> "rocksdb.enable_pipelined_write"142) "no"143) >> "rocksdb.level0_file_num_compaction_trigger"144) "4"145) >> "rocksdb.level0_slowdown_writes_trigger"146) "20"147) >> "rocksdb.level0_stop_writes_trigger"148) "40"149) >> "rocksdb.level_compaction_dynamic_level_bytes"150) "no"151) >> "rocksdb.max_background_compactions"152) "2"153) >> "rocksdb.max_background_flushes"154) "2"155) >> "rocksdb.max_background_jobs"156) "4"157) >> "rocksdb.max_bytes_for_level_base"158) "268435456"159) >> "rocksdb.max_bytes_for_level_multiplier"160) "10"161) >> "rocksdb.max_open_files"162) "8096"163) "rocksdb.max_subcompactions"164) >> "2"165) "rocksdb.max_total_wal_size"166) "256"167) >> "rocksdb.max_write_buffer_number"168) "2"169) >> "rocksdb.metadata_block_cache_size"170) "2048"171) >> "rocksdb.min_blob_size"172) "4096"173) "rocksdb.partition_filters"174) >> "yes"175) "rocksdb.rate_limiter_auto_tuned"176) "yes"177) >> "rocksdb.read_options.async_io"178) "yes"179) "rocksdb.row_cache_size"180) >> "0"181) "rocksdb.share_metadata_and_subkey_block_cache"182) "yes"183) >> "rocksdb.stats_dump_period_sec"184) "0"185) >> "rocksdb.subkey_block_cache_size"186) "2048"187) >> "rocksdb.target_file_size_base"188) "128"189) "rocksdb.wal_compression"190) >> "no"191) "rocksdb.wal_size_limit_mb"192) "16384"193) >> "rocksdb.wal_ttl_seconds"194) "10800"195) "rocksdb.write_buffer_size"196) >> "32"197) "rocksdb.write_options.disable_wal"198) "no"199) >> "rocksdb.write_options.low_pri"200) "no"201) >> "rocksdb.write_options.memtable_insert_hint_per_batch"202) "no"203) >> "rocksdb.write_options.no_slowdown"204) "no"205) >> "rocksdb.write_options.sync"206) "no"207) >> "rocksdb.write_options.write_batch_max_bytes"208) "0"209) >> "skip-block-cache-deallocation-on-close"210) "no"211) >> "slave-empty-db-before-fullsync"212) "no"213) "slave-priority"214) "100"215) >> "slave-read-only"216) "yes"217) "slave-serve-stale-data"218) "yes"219) >> "slaveof"220) ""221) "slowlog-log-slower-than"222) "200000"223) >> "slowlog-max-len"224) "128"225) "socket-fd"226) "-1"227) "supervised"228) >> "no"229) "tcp-backlog"230) "511"231) "timeout"232) "0"233) >> "tls-auth-clients"234) ""235) "tls-ca-cert-dir"236) ""237) >> "tls-ca-cert-file"238) ""239) "tls-cert-file"240) ""241) "tls-ciphers"242) >> ""243) "tls-ciphersuites"244) ""245) "tls-key-file"246) ""247) >> "tls-key-file-pass"248) ""249) "tls-port"250) "0"251) >> "tls-prefer-server-ciphers"252) "no"253) "tls-protocols"254) ""255) >> "tls-replication"256) "no"257) "tls-session-cache-size"258) "20480"259) >> "tls-session-cache-timeout"260) "300"261) "tls-session-caching"262) >> "yes"263) "txn-context-enabled"264) "no"265) "unixsocket"266) ""267) >> "unixsocketperm"268) "777"269) "use-rsid-psync"270) "no"271) "workers"272) >> "8" >> Vào ngày thg 4 18, 2025 5:49 CH, từ Hulk <[email protected]>Hi Quan, >> >> Thanks for your information, could you please provide the output by running >> the command in redis-cli `config get *`? >> >> On Fri, 18 Apr 2025 at 18:24, Hong Quan TRAN <[email protected]> wrote: >> >> > Hi the Kvrocks community, >> > We would like to update you our progress on the integration topic. >> > We have started to migrate a Redis deployment to Kvrocks for a week. The >> > Kvrocks is running stable I would say. However, we have concern after >> > observing the Kvrocks's memory comsuption as it keeps slowly consume more >> > memory without releasing them back to the OS. At first the memory usage was >> > 500MB, then it is up to 700MB, 1GB, 1.3GB... >> > Please note that we already tuning the memory usage using the >> > recommendations at github.com/apache/kvrocks/discussions/2767. >> > ```kvrocks.conflog-level >> > <github.com/apache/kvrocks/discussions/2767.kvrocks.conflog-level> >> > infoport 6379rocksdb.max_total_wal_size 256rocksdb.max_write_buffer_number >> > 2rocksdb.write_buffer_size 32rocksdb.block_cache_size 256``` >> > We expect the memory consumption would be low and stable with this applied >> > configuration. However, it keeps slowly increasing, which makes us worry. >> > Our memory stats so far: >> > ```docker statsCONTAINER ID NAME CPU % MEM USAGE / >> > LIMIT MEM % NET I/O BLOCK I/O PIDS561e141f9b72 >> > kvrocks 0.14% 1.292GiB / 14.3GiB 9.03% 4.77GB / >> > 1.51GB 270kB / 8.08GB 19``` >> > ```kvrocks@561e141f9b72:/$ redis-cli -p 6379 info memory# >> > Memoryused_memory_rss:983244800used_memory_rss_human:937.70Mused_memory_lua:59392used_memory_lua_human:58.00Kused_memory_startup:63860736``` >> > FYI we are using Kvrocks version 2.11.1. >> > Could you help provide insights and advice on the Kvrocks memory >> > management topic? How to limit the Kvrocks memory usage effectively? >> > >> > Best regards, >> > Quan >> > >> > >> >> -- >> Best Regards, >> - *Hulk Lin* > > > > -- > Best Regards, > - Hulk Lin -- Best Regards, - Hulk Lin
# Commandstats cmdstat_dbsize:calls=3,usec=117,usec_per_call=39 cmdstat_del:calls=160,usec=13499,usec_per_call=84 cmdstat_evalsha:calls=532198,usec=137389506,usec_per_call=258 cmdstat_exec:calls=44987,usec=613615272,usec_per_call=13639 cmdstat_expire:calls=241548,usec=8691547,usec_per_call=35 cmdstat_get:calls=97144,usec=6162397,usec_per_call=63 cmdstat_hdel:calls=64,usec=5398,usec_per_call=84 cmdstat_hget:calls=37941900,usec=988532559,usec_per_call=26 cmdstat_hincrby:calls=315525,usec=11315214,usec_per_call=35 cmdstat_hmget:calls=64,usec=70667,usec_per_call=1104 cmdstat_hset:calls=479810,usec=31809715,usec_per_call=66 cmdstat_info:calls=2,usec=813,usec_per_call=406 cmdstat_llen:calls=66836,usec=1435264,usec_per_call=21 cmdstat_lpush:calls=97202,usec=13559203,usec_per_call=139 cmdstat_ltrim:calls=97202,usec=5482668,usec_per_call=56 cmdstat_mget:calls=40961,usec=6175268,usec_per_call=150 cmdstat_multi:calls=44987,usec=95915,usec_per_call=2 cmdstat_ping:calls=29469,usec=91003,usec_per_call=3 cmdstat_psetex:calls=54202,usec=5078965,usec_per_call=93 cmdstat_quit:calls=329465,usec=373794,usec_per_call=1 cmdstat_sadd:calls=130740,usec=19215083,usec_per_call=146 cmdstat_scard:calls=29576,usec=1053479,usec_per_call=35 cmdstat_setex:calls=5949,usec=644179,usec_per_call=108 cmdstat_smembers:calls=64,usec=106728,usec_per_call=1667 cmdstat_sscan:calls=118394,usec=50885872,usec_per_call=429 cmdstat_zadd:calls=96,usec=35212,usec_per_call=366 cmdstat_zcard:calls=14788,usec=778559,usec_per_call=52 cmdstat_zrange:calls=32,usec=10871,usec_per_call=339 cmdstat_zremrangebyrank:calls=32,usec=13562,usec_per_call=423 cmdstat_zrevrange:calls=59152,usec=45027259,usec_per_call=761
