[ https://issues.apache.org/jira/browse/KUDU-3406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17849077#comment-17849077 ]
Alexey Serbin commented on KUDU-3406: ------------------------------------- A note: it's also necessary to pick up [ae7b08c00|https://github.com/apache/kudu/commit/ae7b08c006167da1ebb0c4302e5d6d7aa739a862] -- it contains the fix for the threshold MiBytes-to-bytes conversion. > CompactRowSetsOp can allocate much more memory than specified by the hard > memory limit > -------------------------------------------------------------------------------------- > > Key: KUDU-3406 > URL: https://issues.apache.org/jira/browse/KUDU-3406 > Project: Kudu > Issue Type: Bug > Components: master, tserver > Affects Versions: 1.10.0, 1.10.1, 1.11.0, 1.12.0, 1.11.1, 1.13.0, 1.14.0, > 1.15.0, 1.16.0 > Reporter: Alexey Serbin > Assignee: Ashwani Raina > Priority: Critical > Labels: compaction, stability > Fix For: 1.17.0 > > Attachments: 270.svg, 283.svg, 296.svg, 308.svg, 332.svg, 344.svg, > fs_list.before > > > In some scenarios, rowsets can accumulate a lot of data, so {{kudu-master}} > and {{kudu-tserver}} processes grow far beyond the hard memory limit > (controlled by the {{\-\-memory_limit_hard_bytes}} flag) when running > CompactRowSetsOp. In some cases, a Kudu server process consumes all the > available memory, so that the OS might invoke the OOM killer. > At this point I'm not yet sure about the exact versions affected, and what > leads to accumulating so much data in flushed rowsets, but I know that 1.13, > 1.14, 1.15 and 1.16 are affected. It's also not clear whether the actual > regression is in allowing the flushed rowsets growing that big. > There is a reproduction scenario for this bug with {{kudu-master}} using the > real data from the fields. With that data, {{kudu fs list}} reveals a rowset > with many UNDOs: see the attached {{fs_list.before}} file. When starting > {{kudu-master}} with the data, the process memory usage eventually peaked > with about 25GByte of RSS while running CompactRowSetsOp, and then the RSS > eventually subsides down to about 200MByte once the CompactRowSetsOp is > completed. > I also attached several SVG files generated by the TCMalloc's pprof from the > memory profile snapshots output by {{kudu-master}} when configured to dump > allocation stats every 512 MBytes. I generated the SVG reports for profiles > attributed to the highest memory usage: > {noformat} > Dumping heap profile to /opt/tmp/master/nn1/profile.0270.heap (24573 MB > currently in use) > Dumping heap profile to /opt/tmp/master/nn1/profile.0283.heap (64594 MB > allocated cumulatively, 13221 MB currently in use) > Dumping heap profile to /opt/tmp/master/nn1/profile.0296.heap (77908 MB > allocated cumulatively, 12110 MB currently in use) > Dumping heap profile to /opt/tmp/master/nn1/profile.0308.heap (90197 MB > allocated cumulatively, 12406 MB currently in use) > Dumping heap profile to /opt/tmp/master/nn1/profile.0332.heap (114775 MB > allocated cumulatively, 23884 MB currently in use) > Dumping heap profile to /opt/tmp/master/nn1/profile.0344.heap (127064 MB > allocated cumulatively, 12648 MB currently in use) > {noformat} > The report from the compaction doesn't look like anything extraordinary > (except for the duration): > {noformat} > I20221012 10:45:49.684247 101750 maintenance_manager.cc:603] P > 68dbea0ec022440d9fc282099a8656cb: > CompactRowSetsOp(00000000000000000000000000000000) complete. Timing: real > 522.617s user 471.783s sys 46.588s Metrics: > {"bytes_written":1665145,"cfile_cache_hit":846,"cfile_cache_hit_bytes":14723646,"cfile_cache_miss":1786556,"cfile_cache_miss_bytes":4065589152,"cfile_init":7,"delta_iterators_relevant":1558,"dirs.queue_time_us":220086,"dirs.run_cpu_time_us":89219,"dirs.run_wall_time_us":89163,"drs_written":1,"fdatasync":15,"fdatasync_us":150709,"lbm_read_time_us":11120726,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":1786583,"lbm_write_time_us":14120016,"lbm_writes_1-10_ms":3,"lbm_writes_lt_1ms":894069,"mutex_wait_us":108,"num_input_rowsets":5,"rows_written":4043,"spinlock_wait_cycles":14720,"thread_start_us":741,"threads_started":9,"wal-append.queue_time_us":307} > {noformat} -- This message was sent by Atlassian Jira (v8.20.10#820010)