On 11/3/20 4:53 PM, Ilya Maximets wrote: > Version 2: > - Skipping 3 patches that already applied. > - Fixed use of uninitialized backlog limits in jsonrpc_session, > i.e. added initialization on each jsorpc_session_open*() > - Added Acked-by from Dumitru to the last patch. > > Original cover letter: > > Under a heavy load or in a long run memory consumption if ovsdb-server > process that is part of a RAFT cluster could reach very high values. > From my experience it could be up to 60-100 GB. In these conditions > it's likely that ovsdb-server will be killed by OOM-killer or just > will not be able to work properly wasting time on processing outdated > or unneeded data. There are 3 main parts that consumes most of the > memory: > > 1. Backlog on RAFT connections between servers. > 2. Local RAFT log. > 3. Libc doesn't return memory back to system. > > Backlog could start growing if one of remote servers doesn't doing > well and is not able to process requests in time. This sending > backlog could contain snapshots or even just big number of big > append requests. It could grow to tens of GBs really fast and > most of this data might be even unnecessary if it becomes obsolete > by one of the previous requests or if current 'term' changes and > all the old messages should be dropped. Solution for this is > to monitor the size of the current backlog and disconnect if it grows > too big since it will be easier to just reconnect and send one new > snapshot. > > Local RAFT log contains all the DB changes that are not part of a > snapshot yet. Since snapshots are taken at most once in 10 minutes, > log could grow pretty big. Up to tens of thousands of entries and > each of these entries could be fairly big by themselves. That being > said RAFT log could grow up to tens of GBs too. > > One extra point for memory consumption is that memory likely doesn't > go away even after calling free() due to implementation of a C memory > allocators. And this happens a lot. ovsdb-server process usually > holds a lot of system memory even if the database is almost empty. > This heap memory might be returned back to OS by using malloc_trim(). > > -- > All of these issues was found on branch-2.13, but it always hard to > distinguish new features from the bug fix when we're talking about > scaling issues. Anyway, I think, it'll be good to have these > patches (if they are any good) backorted to 2.13, especially because > it's going to be our next LTS. Thoughts? > > > Ilya Maximets (2): > raft: Set threshold on backlog for raft connections. > raft: Make backlog thresholds configurable. > > NEWS | 3 +++ > lib/jsonrpc.c | 60 ++++++++++++++++++++++++++++++++++++++++- > lib/jsonrpc.h | 6 +++++ > ovsdb/ovsdb-server.1.in | 5 ++++ > ovsdb/raft.c | 50 ++++++++++++++++++++++++++++++++++ > 5 files changed, 123 insertions(+), 1 deletion(-) >
Dumitru, thanks for review! Applied to master and backported down to 2.13. Best regards, Ilya Maximets. _______________________________________________ dev mailing list d...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-dev