Re: [PATCH] migration/rdma: prevent from double free the same mr

2021-07-13 Thread Dr. David Alan Gilbert 
* Li Zhijian (lizhij...@cn.fujitsu.com) wrote:
> backtrace:
> '0x75f44ec2 in __ibv_dereg_mr_1_1 (mr=0x7fff1007d390) at 
> /home/lizhijian/rdma-core/libibverbs/verbs.c:478
> 478 void *addr  = mr->addr;
> (gdb) bt
>  #0  0x75f44ec2 in __ibv_dereg_mr_1_1 (mr=0x7fff1007d390) at 
> /home/lizhijian/rdma-core/libibverbs/verbs.c:478
>  #1  0x55891fcc in rdma_delete_block (block=, 
> rdma=0x7fff38176010) at ../migration/rdma.c:691
>  #2  qemu_rdma_cleanup (rdma=0x7fff38176010) at ../migration/rdma.c:2365
>  #3  0x558925b0 in qio_channel_rdma_close_rcu (rcu=0x56b8b6c0) at 
> ../migration/rdma.c:3073
>  #4  0x55d652a3 in call_rcu_thread (opaque=opaque@entry=0x0) at 
> ../util/rcu.c:281
>  #5  0x55d5edf9 in qemu_thread_start (args=0x7fffe88bb4d0) at 
> ../util/qemu-thread-posix.c:541
>  #6  0x754c73f9 in start_thread () at /lib64/libpthread.so.0
>  #7  0x753f3b03 in clone () at /lib64/libc.so.6 '
> 
> Signed-off-by: Li Zhijian 

Queued

> ---
>  migration/rdma.c | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/migration/rdma.c b/migration/rdma.c
> index b6cc4bef4a8..0f22b8227c0 100644
> --- a/migration/rdma.c
> +++ b/migration/rdma.c
> @@ -1143,6 +1143,7 @@ static int qemu_rdma_reg_whole_ram_blocks(RDMAContext 
> *rdma)
>  
>  for (i--; i >= 0; i--) {
>  ibv_dereg_mr(local->block[i].mr);
> +local->block[i].mr = NULL;
>  rdma->total_registrations--;
>  }
>  
> -- 
> 2.30.2
> 
> 
> 
> 
-- 
Dr. David Alan Gilbert / dgilb...@redhat.com / Manchester, UK

Re: [PATCH] migration/rdma: prevent from double free the same mr

2021-07-12 Thread Dr. David Alan Gilbert 
* lizhij...@fujitsu.com (lizhij...@fujitsu.com) wrote:
> 
> 
> On 09/07/2021 03:11, Dr. David Alan Gilbert wrote:
> > * Li Zhijian (lizhij...@cn.fujitsu.com) wrote:
> >> backtrace:
> >> '0x75f44ec2 in __ibv_dereg_mr_1_1 (mr=0x7fff1007d390) at 
> >> /home/lizhijian/rdma-core/libibverbs/verbs.c:478
> >> 478 void *addr  = mr->addr;
> > ANy idea why it deletes the same mr twice?
> 
> It's easy to reproduce it if we specify a nvdimm backing to a fsdax 
> memory-backend-file which cannot support registering mr like:
> 
> [root@iaas-rpma ~]# mount | grep pmem0
> /dev/pmem0 on /mnt/pmem0 type ext4 (rw,relatime,seclabel,dax=always)
> 
> [root@iaas-rpma ~]# ndctl list -n namespace0.0
> [
>    {
>      "dev":"namespace0.0",
>      "mode":"fsdax",
>      "map":"mem",
>      "size":536870912,
>      "sector_size":512,
>      "blockdev":"pmem0"
>    }
> ]
> 
> 
> `-object 
> memory-backend-file,id=mem1,share=on,mem-path=/mnt/pmem0/nv-128m.img,size=128m,pmem=on,align=2m
>  -device nvdimm,memdev=mem1,id=nv1`
> 
> and then enable rdma-pin-all.
> 
> (qemu) migrate_set_capability rdma-pin-all on
> (qemu)
> 
> Now qemu has at least 2 ram block, pc.ram and mem1. the latter will be failed 
> to register mr:
> `Failed to register local dest ram block! : Invalid argument   `
> 
> in this case, the mr of pc.ram will be deleted twice.

Ah OK, that makes more sense - from your original description I hadn't
noticed it was the failure path.


Reviewed-by: Dr. David Alan Gilbert 

> Thanks
> Li
> >
> > Dave
> >
> >> (gdb) bt
> >>   #0  0x75f44ec2 in __ibv_dereg_mr_1_1 (mr=0x7fff1007d390) at 
> >> /home/lizhijian/rdma-core/libibverbs/verbs.c:478
> >>   #1  0x55891fcc in rdma_delete_block (block=, 
> >> rdma=0x7fff38176010) at ../migration/rdma.c:691
> >>   #2  qemu_rdma_cleanup (rdma=0x7fff38176010) at ../migration/rdma.c:2365
> >>   #3  0x558925b0 in qio_channel_rdma_close_rcu 
> >> (rcu=0x56b8b6c0) at ../migration/rdma.c:3073
> >>   #4  0x55d652a3 in call_rcu_thread (opaque=opaque@entry=0x0) at 
> >> ../util/rcu.c:281
> >>   #5  0x55d5edf9 in qemu_thread_start (args=0x7fffe88bb4d0) at 
> >> ../util/qemu-thread-posix.c:541
> >>   #6  0x754c73f9 in start_thread () at /lib64/libpthread.so.0
> >>   #7  0x753f3b03 in clone () at /lib64/libc.so.6 '
> >>
> >> Signed-off-by: Li Zhijian 
> >> ---
> >>   migration/rdma.c | 1 +
> >>   1 file changed, 1 insertion(+)
> >>
> >> diff --git a/migration/rdma.c b/migration/rdma.c
> >> index b6cc4bef4a8..0f22b8227c0 100644
> >> --- a/migration/rdma.c
> >> +++ b/migration/rdma.c
> >> @@ -1143,6 +1143,7 @@ static int 
> >> qemu_rdma_reg_whole_ram_blocks(RDMAContext *rdma)
> >>   
> >>   for (i--; i >= 0; i--) {
> >>   ibv_dereg_mr(local->block[i].mr);
> >> +local->block[i].mr = NULL;
> >>   rdma->total_registrations--;
> >>   }
> >>   
> >> -- 
> >> 2.30.2
> >>
> >>
> >>
-- 
Dr. David Alan Gilbert / dgilb...@redhat.com / Manchester, UK

Re: [PATCH] migration/rdma: prevent from double free the same mr

2021-07-08 Thread lizhij...@fujitsu.com 


On 09/07/2021 03:11, Dr. David Alan Gilbert wrote:
> * Li Zhijian (lizhij...@cn.fujitsu.com) wrote:
>> backtrace:
>> '0x75f44ec2 in __ibv_dereg_mr_1_1 (mr=0x7fff1007d390) at 
>> /home/lizhijian/rdma-core/libibverbs/verbs.c:478
>> 478 void *addr  = mr->addr;
> ANy idea why it deletes the same mr twice?

It's easy to reproduce it if we specify a nvdimm backing to a fsdax 
memory-backend-file which cannot support registering mr like:

[root@iaas-rpma ~]# mount | grep pmem0
/dev/pmem0 on /mnt/pmem0 type ext4 (rw,relatime,seclabel,dax=always)

[root@iaas-rpma ~]# ndctl list -n namespace0.0
[
   {
     "dev":"namespace0.0",
     "mode":"fsdax",
     "map":"mem",
     "size":536870912,
     "sector_size":512,
     "blockdev":"pmem0"
   }
]


`-object 
memory-backend-file,id=mem1,share=on,mem-path=/mnt/pmem0/nv-128m.img,size=128m,pmem=on,align=2m
 -device nvdimm,memdev=mem1,id=nv1`

and then enable rdma-pin-all.

(qemu) migrate_set_capability rdma-pin-all on
(qemu)

Now qemu has at least 2 ram block, pc.ram and mem1. the latter will be failed 
to register mr:
`Failed to register local dest ram block! : Invalid argument   `

in this case, the mr of pc.ram will be deleted twice.

Thanks
Li
>
> Dave
>
>> (gdb) bt
>>   #0  0x75f44ec2 in __ibv_dereg_mr_1_1 (mr=0x7fff1007d390) at 
>> /home/lizhijian/rdma-core/libibverbs/verbs.c:478
>>   #1  0x55891fcc in rdma_delete_block (block=, 
>> rdma=0x7fff38176010) at ../migration/rdma.c:691
>>   #2  qemu_rdma_cleanup (rdma=0x7fff38176010) at ../migration/rdma.c:2365
>>   #3  0x558925b0 in qio_channel_rdma_close_rcu (rcu=0x56b8b6c0) 
>> at ../migration/rdma.c:3073
>>   #4  0x55d652a3 in call_rcu_thread (opaque=opaque@entry=0x0) at 
>> ../util/rcu.c:281
>>   #5  0x55d5edf9 in qemu_thread_start (args=0x7fffe88bb4d0) at 
>> ../util/qemu-thread-posix.c:541
>>   #6  0x754c73f9 in start_thread () at /lib64/libpthread.so.0
>>   #7  0x753f3b03 in clone () at /lib64/libc.so.6 '
>>
>> Signed-off-by: Li Zhijian 
>> ---
>>   migration/rdma.c | 1 +
>>   1 file changed, 1 insertion(+)
>>
>> diff --git a/migration/rdma.c b/migration/rdma.c
>> index b6cc4bef4a8..0f22b8227c0 100644
>> --- a/migration/rdma.c
>> +++ b/migration/rdma.c
>> @@ -1143,6 +1143,7 @@ static int qemu_rdma_reg_whole_ram_blocks(RDMAContext 
>> *rdma)
>>   
>>   for (i--; i >= 0; i--) {
>>   ibv_dereg_mr(local->block[i].mr);
>> +local->block[i].mr = NULL;
>>   rdma->total_registrations--;
>>   }
>>   
>> -- 
>> 2.30.2
>>
>>
>>

Re: [PATCH] migration/rdma: prevent from double free the same mr

2021-07-08 Thread Dr. David Alan Gilbert 
* Li Zhijian (lizhij...@cn.fujitsu.com) wrote:
> backtrace:
> '0x75f44ec2 in __ibv_dereg_mr_1_1 (mr=0x7fff1007d390) at 
> /home/lizhijian/rdma-core/libibverbs/verbs.c:478
> 478 void *addr  = mr->addr;

ANy idea why it deletes the same mr twice?
What was your commandline?

Dave

> (gdb) bt
>  #0  0x75f44ec2 in __ibv_dereg_mr_1_1 (mr=0x7fff1007d390) at 
> /home/lizhijian/rdma-core/libibverbs/verbs.c:478
>  #1  0x55891fcc in rdma_delete_block (block=, 
> rdma=0x7fff38176010) at ../migration/rdma.c:691
>  #2  qemu_rdma_cleanup (rdma=0x7fff38176010) at ../migration/rdma.c:2365
>  #3  0x558925b0 in qio_channel_rdma_close_rcu (rcu=0x56b8b6c0) at 
> ../migration/rdma.c:3073
>  #4  0x55d652a3 in call_rcu_thread (opaque=opaque@entry=0x0) at 
> ../util/rcu.c:281
>  #5  0x55d5edf9 in qemu_thread_start (args=0x7fffe88bb4d0) at 
> ../util/qemu-thread-posix.c:541
>  #6  0x754c73f9 in start_thread () at /lib64/libpthread.so.0
>  #7  0x753f3b03 in clone () at /lib64/libc.so.6 '
> 
> Signed-off-by: Li Zhijian 
> ---
>  migration/rdma.c | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/migration/rdma.c b/migration/rdma.c
> index b6cc4bef4a8..0f22b8227c0 100644
> --- a/migration/rdma.c
> +++ b/migration/rdma.c
> @@ -1143,6 +1143,7 @@ static int qemu_rdma_reg_whole_ram_blocks(RDMAContext 
> *rdma)
>  
>  for (i--; i >= 0; i--) {
>  ibv_dereg_mr(local->block[i].mr);
> +local->block[i].mr = NULL;
>  rdma->total_registrations--;
>  }
>  
> -- 
> 2.30.2
> 
> 
> 
-- 
Dr. David Alan Gilbert / dgilb...@redhat.com / Manchester, UK

[PATCH] migration/rdma: prevent from double free the same mr

2021-07-08 Thread Li Zhijian 
backtrace:
'0x75f44ec2 in __ibv_dereg_mr_1_1 (mr=0x7fff1007d390) at 
/home/lizhijian/rdma-core/libibverbs/verbs.c:478
478 void *addr  = mr->addr;
(gdb) bt
 #0  0x75f44ec2 in __ibv_dereg_mr_1_1 (mr=0x7fff1007d390) at 
/home/lizhijian/rdma-core/libibverbs/verbs.c:478
 #1  0x55891fcc in rdma_delete_block (block=, 
rdma=0x7fff38176010) at ../migration/rdma.c:691
 #2  qemu_rdma_cleanup (rdma=0x7fff38176010) at ../migration/rdma.c:2365
 #3  0x558925b0 in qio_channel_rdma_close_rcu (rcu=0x56b8b6c0) at 
../migration/rdma.c:3073
 #4  0x55d652a3 in call_rcu_thread (opaque=opaque@entry=0x0) at 
../util/rcu.c:281
 #5  0x55d5edf9 in qemu_thread_start (args=0x7fffe88bb4d0) at 
../util/qemu-thread-posix.c:541
 #6  0x754c73f9 in start_thread () at /lib64/libpthread.so.0
 #7  0x753f3b03 in clone () at /lib64/libc.so.6 '

Signed-off-by: Li Zhijian 
---
 migration/rdma.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/migration/rdma.c b/migration/rdma.c
index b6cc4bef4a8..0f22b8227c0 100644
--- a/migration/rdma.c
+++ b/migration/rdma.c
@@ -1143,6 +1143,7 @@ static int qemu_rdma_reg_whole_ram_blocks(RDMAContext 
*rdma)
 
 for (i--; i >= 0; i--) {
 ibv_dereg_mr(local->block[i].mr);
+local->block[i].mr = NULL;
 rdma->total_registrations--;
 }
 
-- 
2.30.2