** Description changed: + [Impact] + + * multipath crashes when device-mapper is modified. DM_NAME was being freed twice. + * expect multipath daemon to crash and not run any checkers on path groups. + * not checking path groups, in an event of failure, the mpath won't change path prios. + * openstack relies on flushing device maps frequently when using iscsi. + + [Test Case] + + * having a multipathed environment (4 paths, 2 and 2, to a lun): + - while true; do multipath -F ; multipath -r ; multipath -ll; done + * run multipath with valgrind and see: + + ==31831== Invalid read of size 1 + ==31831== at 0x4C2E902: strncmp (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so) + ==31831== by 0x56FC26E: find_mp_by_alias (structs.c:296) + ==31831== by 0x404B2F: ev_add_map (main.c:264) + ==31831== by 0x404A8B: uev_add_map (main.c:244) + ... + ==31831== Address 0x728d8d1 is 1 bytes inside a block of size 6 free'd + ==31831== at 0x4C2BDEC: free (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so) + ==31831== by 0x404A9A: uev_add_map (main.c:245) + ==31831== by 0x40623C: uev_trigger (main.c:756) + + [Regression Potential] + + * using strdup for this char *, if there was no double free - like i discovered, would cause a slight memory leak of the size of DM_NAME every time a device mapper disappears and is re-created. it wouldn't be an important regression. + * based on upstream commit and tested by the reported. fixes initial issue. + + [Other Info] + It has brought to my attention that multipath in trusty has been crashing randomly. Some dumps were given to me and I was able to generate some others. I have also generated valgrind output to help me with these random crashes. Crashes: #0 malloc_consolidate (av=av@entry=0x7f5b58000020) at malloc.c:4149 #1 0x00007f5b62df3cf8 in _int_malloc (av=0x7f5b58000020, bytes=16384) at malloc.c:3423 #2 0x00007f5b62df66d0 in __GI___libc_malloc (bytes=16384) at malloc.c:2891 #3 0x00007f5b638134d7 in dm_task_run () from /lib/x86_64-linux-gnu/libdevmapper.so.1.02.1 #4 0x00007f5b6314be5c in dm_map_present (str=0x7f5b58000990 "lun02") at devmapper.c:304 #5 0x0000000000404ac7 in ev_add_map (dev=, alias=, vecs=) at main.c:257 #6 0x0000000000000000 in ?? () And: #0 0x00007f13a5933c37 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56 #1 0x00007f13a5937028 in __GI_abort () at abort.c:89 #2 0x00007f13a59702a4 in __libc_message (do_abort=do_abort@entry=1, fmt=fmt@entry=0x7f13a5a81ef0 "") at ../sysdeps/posix/libc_fatal.c:175 #3 0x00007f13a597c56e in malloc_printerr (ptr=<optimized out>, str=0x7f13a5a82020 "double free or corruption (out)", action=1) at malloc.c:4996 #4 _int_free (av=<optimized out>, p=<optimized out>, have_lock=0) at malloc.c:3840 #5 0x00007f13a5cdbe86 in free_multipath (mpp=0x7f138c033d60, free_paths=0) at structs.c:174 #6 0x00007f13a5cfe117 in _remove_map (mpp=0x7f138c033d60, vecs=0x8adaa0, stop_waiter=1, purge_vec=1) at structs_vec.c:143 #7 0x00007f13a5cfe175 in remove_map_and_stop_waiter (mpp=0x7f138c033d60, vecs=0x8adaa0, purge_vec=1) at structs_vec.c:156 #8 0x0000000000406b4d in mpvec_garbage_collector (vecs=<error reading variable: can't compute CFA for this frame>) at main.c:950 ... #14 0x00000000004076b7 in checkerloop (ap=<error reading variable: can't compute CFA for this frame>) at main.c:1163 Please follow my analysis in the subsequent comments.
-- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1695789 Title: multipath random crashes on use-after-free To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/1695789/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
