Version 2: - Skipping 3 patches that already applied. - Fixed use of uninitialized backlog limits in jsonrpc_session, i.e. added initialization on each jsorpc_session_open*() - Added Acked-by from Dumitru to the last patch.
Original cover letter: Under a heavy load or in a long run memory consumption if ovsdb-server process that is part of a RAFT cluster could reach very high values. >From my experience it could be up to 60-100 GB. In these conditions it's likely that ovsdb-server will be killed by OOM-killer or just will not be able to work properly wasting time on processing outdated or unneeded data. There are 3 main parts that consumes most of the memory: 1. Backlog on RAFT connections between servers. 2. Local RAFT log. 3. Libc doesn't return memory back to system. Backlog could start growing if one of remote servers doesn't doing well and is not able to process requests in time. This sending backlog could contain snapshots or even just big number of big append requests. It could grow to tens of GBs really fast and most of this data might be even unnecessary if it becomes obsolete by one of the previous requests or if current 'term' changes and all the old messages should be dropped. Solution for this is to monitor the size of the current backlog and disconnect if it grows too big since it will be easier to just reconnect and send one new snapshot. Local RAFT log contains all the DB changes that are not part of a snapshot yet. Since snapshots are taken at most once in 10 minutes, log could grow pretty big. Up to tens of thousands of entries and each of these entries could be fairly big by themselves. That being said RAFT log could grow up to tens of GBs too. One extra point for memory consumption is that memory likely doesn't go away even after calling free() due to implementation of a C memory allocators. And this happens a lot. ovsdb-server process usually holds a lot of system memory even if the database is almost empty. This heap memory might be returned back to OS by using malloc_trim(). -- All of these issues was found on branch-2.13, but it always hard to distinguish new features from the bug fix when we're talking about scaling issues. Anyway, I think, it'll be good to have these patches (if they are any good) backorted to 2.13, especially because it's going to be our next LTS. Thoughts? Ilya Maximets (2): raft: Set threshold on backlog for raft connections. raft: Make backlog thresholds configurable. NEWS | 3 +++ lib/jsonrpc.c | 60 ++++++++++++++++++++++++++++++++++++++++- lib/jsonrpc.h | 6 +++++ ovsdb/ovsdb-server.1.in | 5 ++++ ovsdb/raft.c | 50 ++++++++++++++++++++++++++++++++++ 5 files changed, 123 insertions(+), 1 deletion(-) -- 2.25.4 _______________________________________________ dev mailing list d...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-dev
