Hi Ben, With your memory fixes applied, I still see many test failures due to memory leaks when configured as:
./configure --enable-Werror --enable-sparse CFLAGS="-g -fsanitize=address" You can find more details about the memory leaks here - https://gist.github.com/numansiddique/e701777da977a89e24b49f159bb30d5d Thanks Numan On Wed, Mar 24, 2021 at 9:19 PM Ben Pfaff <b...@ovn.org> wrote: > > Thanks a lot for the report! I need a new test case so I'll give this > one a shot when I can. > > On Wed, Mar 24, 2021 at 04:03:07PM +0100, Dumitru Ceara wrote: > > Hi Ben, > > > > We discussed a bit about this during one of the recent IRC OVN meetings, > > but I didn't get around to properly reporting this until now. > > > > I've tried running ovn-northd-ddlog against some large OVN NB/DB > > databases extracted from one of our scale testing runs: > > > > http://people.redhat.com/~dceara/ovn-northd-ddlog-tests/20210324/existing-nb-sb/ > > > > It seems that ovn-northd-ddlog gets stuck in a busy loop and uses a lot > > of memory: > > > > 775734 root 10 -10 81.6g 80.8g 22396 S 99.7 64.2 3:50.79 > > ovn-northd-ddlog > > > > ovn-northd-ddlog is stuck in an (infinite?) loop in > > ddlog_transaction_commit_dump_changes(): > > > > (gdb) bt > > #0 0x00007f2762167e92 in pthread_cond_wait@@GLIBC_2.3.2 () from > > /lib64/libpthread.so.0 > > #1 0x0000000000891eb3 in std::thread::park () > > #2 0x0000000000530963 in crossbeam_channel::context::Context::wait_until () > > #3 0x0000000000530aa1 in > > crossbeam_channel::context::Context::with::{{closure}} () > > #4 0x0000000000531f5c in > > crossbeam_channel::flavors::list::Channel<T>::recv () > > #5 0x000000000075b35d in crossbeam_channel::channel::Receiver<T>::recv () > > #6 0x0000000000512156 in > > differential_datalog::program::RunningProgram::flush () > > #7 0x000000000050de67 in > > differential_datalog::program::RunningProgram::transaction_commit () > > #8 0x000000000093f17d in <ovn_northd_ddlog::api::HDDlog as > > differential_datalog::ddlog::DDlog>::transaction_commit_dump_changes () > > #9 0x000000000040cc40 in ddlog_transaction_commit_dump_changes () > > #10 0x000000000040a758 in ddlog_commit (ddlog=<optimized out>) at > > northd/ovn-northd-ddlog.c:435 > > #11 northd_parse_update (update=0x3dc0598, ctx=0x3d71880) at > > northd/ovn-northd-ddlog.c:435 > > #12 northd_run (ctx=0x3d71880) at northd/ovn-northd-ddlog.c:512 > > #13 0x0000000000408edd in main (argc=<optimized out>, argv=<optimized out>) > > at northd/ovn-northd-ddlog.c:1203 > > > > For comparison, running the C version of ovn-northd I get: > > > > 2021-03-24T14:48:06.556Z|00033|timeval|WARN|Unreasonably long 11290ms poll > > interval (10916ms user, 334ms system) > > 2021-03-24T14:48:14.678Z|00050|timeval|WARN|Unreasonably long 8122ms poll > > interval (8046ms user, 48ms system) > > > > But after the northd iteration ends, memory usage is OK: > > > > 777567 root 10 -10 1657308 1.6g 3496 S 0.0 1.2 0:49.08 > > ovn-northd > > > > The behavior above is consistent both with current OVN master and also > > when cherry-picking the ddlog-related patches that are pending in > > patchwork: > > http://patchwork.ozlabs.org/project/ovn/list/?series=233075 > > http://patchwork.ozlabs.org/project/ovn/list/?series=232480 > > http://patchwork.ozlabs.org/project/ovn/list/?series=233079 > > http://patchwork.ozlabs.org/project/ovn/list/?series=233080 > > > > I didn't try out the changes from the following series though as I > > understand they need a v2: > > http://patchwork.ozlabs.org/project/ovn/list/?series=232040 > > > > Regards, > > Dumitru > > > _______________________________________________ > dev mailing list > d...@openvswitch.org > https://mail.openvswitch.org/mailman/listinfo/ovs-dev > _______________________________________________ dev mailing list d...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-dev