Re: [Ocfs2-devel] [PATCH] ocfs2: fix deadlock on mmapped page in ocfs2_write_begin_nolock()

2016-09-14 Thread Joseph Qi
Okay, IC.
So we have to take care of all errors for ocfs2_write_begin_nolock.

On 2016/9/14 16:43, Eric Ren wrote:
> Hi Joseph,
> 
> On 09/14/2016 04:25 PM, Joseph Qi wrote:
>> Hi Eric,
>> Sorry for the delayed response.
>> I have got your explanation. So we have to unlock the page only in case
>> of retry, right?
>> If so, I think the unlock should be right before "goto try_again".
> No, the mmapped page should be unlocked as long as we cannot return 
> VM_FAULT_LOCKED
> to do_page_mkpage(). Otherwise, the deadlock will happen in do_page_mkpage(). 
> Please
> see the recent 2 mails;-)
> 
> Eric
>>
>> Thanks,
>> Joseph
>>
>> On 2016/9/14 16:04, Eric Ren wrote:
>>> Hi Joseph,
>> In ocfs2_write_begin_nolock(), we first grab the pages and then
>> allocate disk space for this write; ocfs2_try_to_free_truncate_log()
>> will be called if ENOSPC is turned; if we're lucky to get enough 
>> clusters,
>> which is usually the case, we start over again. But in 
>> ocfs2_free_write_ctxt()
>> the target page isn't unlocked, so we will deadlock when trying to grab
>> the target page again.
> IMO, in ocfs2_grab_pages_for_write, mmap_page is mapping to w_pages and
> w_target_locked is set to true, and then will be unlocked by
> ocfs2_unlock_pages in ocfs2_free_write_ctxt.
> So I'm not getting the case "page isn't unlock". Could you please explain
> it in more detail?
 Thanks for review;-) Follow up the calling chain:

 ocfs2_free_write_ctxt()
   ->ocfs2_unlock_pages()

 in ocfs2_unlock_pages 
 (https://github.com/torvalds/linux/blob/master/fs/ocfs2/aops.c#L793), we
 can see the code just put_page(target_page), but not unlock it.
>>> Did this answer your question?
>>>
>>> Thanks,
>>> Eric
 Yeah, I will think this a bit more like:
 why not unlock the target_page there? Is there other potential problems if 
 the "ret" is not "-ENOSPC" but
 other possible error code?

 Thanks,
 Eric

> Thanks,
> Joseph
>
>> Fix this issue by unlocking the target page after we fail to allocate
>> enough space at the first time.
>>
>> Jan Kara helps me clear out the JBD2 part, and suggest the hint for root 
>> cause.
>>
>> Signed-off-by: Eric Ren 
>> ---
>>fs/ocfs2/aops.c | 7 +++
>>1 file changed, 7 insertions(+)
>>
>> diff --git a/fs/ocfs2/aops.c b/fs/ocfs2/aops.c
>> index 98d3654..78d1d67 100644
>> --- a/fs/ocfs2/aops.c
>> +++ b/fs/ocfs2/aops.c
>> @@ -1860,6 +1860,13 @@ out:
>> */
>>try_free = 0;
>>+/*
>> + * Unlock mmap_page because the page has been locked when we
>> + * are here.
>> + */
>> +if (mmap_page)
>> +unlock_page(mmap_page);
>> +
>>ret1 = ocfs2_try_to_free_truncate_log(osb, clusters_need);
>>if (ret1 == 1)
>>goto try_again;
>>
>



 ___
 Ocfs2-devel mailing list
 Ocfs2-devel@oss.oracle.com
 https://oss.oracle.com/mailman/listinfo/ocfs2-devel
>>>
>>
>>
> 
> 
> .
> 



___
Ocfs2-devel mailing list
Ocfs2-devel@oss.oracle.com
https://oss.oracle.com/mailman/listinfo/ocfs2-devel


Re: [Ocfs2-devel] [PATCH] ocfs2: fix deadlock on mmapped page in ocfs2_write_begin_nolock()

2016-09-14 Thread Eric Ren
Hi Joseph,

On 09/14/2016 04:25 PM, Joseph Qi wrote:
> Hi Eric,
> Sorry for the delayed response.
> I have got your explanation. So we have to unlock the page only in case
> of retry, right?
> If so, I think the unlock should be right before "goto try_again".
No, the mmapped page should be unlocked as long as we cannot return 
VM_FAULT_LOCKED
to do_page_mkpage(). Otherwise, the deadlock will happen in do_page_mkpage(). 
Please
see the recent 2 mails;-)

Eric
>
> Thanks,
> Joseph
>
> On 2016/9/14 16:04, Eric Ren wrote:
>> Hi Joseph,
> In ocfs2_write_begin_nolock(), we first grab the pages and then
> allocate disk space for this write; ocfs2_try_to_free_truncate_log()
> will be called if ENOSPC is turned; if we're lucky to get enough clusters,
> which is usually the case, we start over again. But in 
> ocfs2_free_write_ctxt()
> the target page isn't unlocked, so we will deadlock when trying to grab
> the target page again.
 IMO, in ocfs2_grab_pages_for_write, mmap_page is mapping to w_pages and
 w_target_locked is set to true, and then will be unlocked by
 ocfs2_unlock_pages in ocfs2_free_write_ctxt.
 So I'm not getting the case "page isn't unlock". Could you please explain
 it in more detail?
>>> Thanks for review;-) Follow up the calling chain:
>>>
>>> ocfs2_free_write_ctxt()
>>>   ->ocfs2_unlock_pages()
>>>
>>> in ocfs2_unlock_pages 
>>> (https://github.com/torvalds/linux/blob/master/fs/ocfs2/aops.c#L793), we
>>> can see the code just put_page(target_page), but not unlock it.
>> Did this answer your question?
>>
>> Thanks,
>> Eric
>>> Yeah, I will think this a bit more like:
>>> why not unlock the target_page there? Is there other potential problems if 
>>> the "ret" is not "-ENOSPC" but
>>> other possible error code?
>>>
>>> Thanks,
>>> Eric
>>>
 Thanks,
 Joseph

> Fix this issue by unlocking the target page after we fail to allocate
> enough space at the first time.
>
> Jan Kara helps me clear out the JBD2 part, and suggest the hint for root 
> cause.
>
> Signed-off-by: Eric Ren 
> ---
>fs/ocfs2/aops.c | 7 +++
>1 file changed, 7 insertions(+)
>
> diff --git a/fs/ocfs2/aops.c b/fs/ocfs2/aops.c
> index 98d3654..78d1d67 100644
> --- a/fs/ocfs2/aops.c
> +++ b/fs/ocfs2/aops.c
> @@ -1860,6 +1860,13 @@ out:
> */
>try_free = 0;
>+/*
> + * Unlock mmap_page because the page has been locked when we
> + * are here.
> + */
> +if (mmap_page)
> +unlock_page(mmap_page);
> +
>ret1 = ocfs2_try_to_free_truncate_log(osb, clusters_need);
>if (ret1 == 1)
>goto try_again;
>

>>>
>>>
>>>
>>> ___
>>> Ocfs2-devel mailing list
>>> Ocfs2-devel@oss.oracle.com
>>> https://oss.oracle.com/mailman/listinfo/ocfs2-devel
>>
>
>


___
Ocfs2-devel mailing list
Ocfs2-devel@oss.oracle.com
https://oss.oracle.com/mailman/listinfo/ocfs2-devel


Re: [Ocfs2-devel] [PATCH] ocfs2: fix deadlock on mmapped page in ocfs2_write_begin_nolock()

2016-09-14 Thread Joseph Qi
Hi Eric,
Sorry for the delayed response.
I have got your explanation. So we have to unlock the page only in case
of retry, right?
If so, I think the unlock should be right before "goto try_again".

Thanks,
Joseph

On 2016/9/14 16:04, Eric Ren wrote:
> Hi Joseph,
 In ocfs2_write_begin_nolock(), we first grab the pages and then
 allocate disk space for this write; ocfs2_try_to_free_truncate_log()
 will be called if ENOSPC is turned; if we're lucky to get enough clusters,
 which is usually the case, we start over again. But in 
 ocfs2_free_write_ctxt()
 the target page isn't unlocked, so we will deadlock when trying to grab
 the target page again.
>>> IMO, in ocfs2_grab_pages_for_write, mmap_page is mapping to w_pages and
>>> w_target_locked is set to true, and then will be unlocked by
>>> ocfs2_unlock_pages in ocfs2_free_write_ctxt.
>>> So I'm not getting the case "page isn't unlock". Could you please explain
>>> it in more detail?
>> Thanks for review;-) Follow up the calling chain:
>>
>> ocfs2_free_write_ctxt()
>>  ->ocfs2_unlock_pages()
>>
>> in ocfs2_unlock_pages 
>> (https://github.com/torvalds/linux/blob/master/fs/ocfs2/aops.c#L793), we
>> can see the code just put_page(target_page), but not unlock it.
> Did this answer your question?
> 
> Thanks,
> Eric
>>
>> Yeah, I will think this a bit more like:
>> why not unlock the target_page there? Is there other potential problems if 
>> the "ret" is not "-ENOSPC" but
>> other possible error code?
>>
>> Thanks,
>> Eric
>>
>>>
>>> Thanks,
>>> Joseph
>>>
 Fix this issue by unlocking the target page after we fail to allocate
 enough space at the first time.

 Jan Kara helps me clear out the JBD2 part, and suggest the hint for root 
 cause.

 Signed-off-by: Eric Ren 
 ---
   fs/ocfs2/aops.c | 7 +++
   1 file changed, 7 insertions(+)

 diff --git a/fs/ocfs2/aops.c b/fs/ocfs2/aops.c
 index 98d3654..78d1d67 100644
 --- a/fs/ocfs2/aops.c
 +++ b/fs/ocfs2/aops.c
 @@ -1860,6 +1860,13 @@ out:
*/
   try_free = 0;
   +/*
 + * Unlock mmap_page because the page has been locked when we
 + * are here.
 + */
 +if (mmap_page)
 +unlock_page(mmap_page);
 +
   ret1 = ocfs2_try_to_free_truncate_log(osb, clusters_need);
   if (ret1 == 1)
   goto try_again;

>>>
>>>
>>
>>
>>
>>
>> ___
>> Ocfs2-devel mailing list
>> Ocfs2-devel@oss.oracle.com
>> https://oss.oracle.com/mailman/listinfo/ocfs2-devel
> 
> 



___
Ocfs2-devel mailing list
Ocfs2-devel@oss.oracle.com
https://oss.oracle.com/mailman/listinfo/ocfs2-devel


Re: [Ocfs2-devel] [PATCH] ocfs2: fix deadlock on mmapped page in ocfs2_write_begin_nolock()

2016-09-14 Thread Eric Ren
Hi,

On 09/12/2016 11:06 AM, Eric Ren wrote:
> Hi,
>>> IMO, in ocfs2_grab_pages_for_write, mmap_page is mapping to w_pages and
>>> w_target_locked is set to true, and then will be unlocked by
>>> ocfs2_unlock_pages in ocfs2_free_write_ctxt.
>>> So I'm not getting the case "page isn't unlock". Could you please explain
>>> it in more detail?
>> Thanks for review;-) Follow up the calling chain:
>>
>> ocfs2_free_write_ctxt()
>>   ->ocfs2_unlock_pages()
>>
>> in ocfs2_unlock_pages
>> (https://github.com/torvalds/linux/blob/master/fs/ocfs2/aops.c#L793), we
>> can see the code just put_page(target_page), but not unlock it.
>>
>> Yeah, I will think this a bit more like:
>> why not unlock the target_page there? Is there other potential problems if 
>> the "ret" is
>> not "-ENOSPC" but
>> other possible error code?
> 1. ocfs2_unlock_pages() will be called in ocfs2_write_end_nolock(), in this 
> case, we
> definitely want to return a locked mmaped page
> to VM code (do_page_mkwrite) when VM_FAULT_LOCKED is set.
>
> 2. But there's indeed a potential existing deadlock situation:
>  ocfs2_grab_pages_for_write()  ==> return 
> -ENOMEM and with
> the mmaped page locked
>  ocfs2_free_write_ctxt() ==> leave the 
> mmapped page locked
>ocfs2_write_begin_nolock() ==> return -ENOMEM
>  __ocfs2_page_mkwrite() ==> return VM_FAULT_OMM
>__do_page_mkwrite() ==> deadlock here
> (https://github.com/torvalds/linux/blob/master/mm/memory.c#L2054)
> This is another corner case, right?
>
> Anyway, I think this patch is good for the -ENOSPC case. And another patch 
> should be
> proposed for -ENOMEM case?
Yes, I think we can catch both -ENOSPC and -ENOMEM cases in the failure path by 
unlocking the
mmaped page after ocfs2_free_write_ctx(), right?

Eric
>
> Thanks,
> Eric
>
>> Thanks,
>> Eric
>>
>>> Thanks,
>>> Joseph
>>>
 Fix this issue by unlocking the target page after we fail to allocate
 enough space at the first time.

 Jan Kara helps me clear out the JBD2 part, and suggest the hint for root 
 cause.

 Signed-off-by: Eric Ren 
 ---
fs/ocfs2/aops.c | 7 +++
1 file changed, 7 insertions(+)

 diff --git a/fs/ocfs2/aops.c b/fs/ocfs2/aops.c
 index 98d3654..78d1d67 100644
 --- a/fs/ocfs2/aops.c
 +++ b/fs/ocfs2/aops.c
 @@ -1860,6 +1860,13 @@ out:
 */
try_free = 0;
+/*
 + * Unlock mmap_page because the page has been locked when we
 + * are here.
 + */
 +if (mmap_page)
 +unlock_page(mmap_page);
 +
ret1 = ocfs2_try_to_free_truncate_log(osb, clusters_need);
if (ret1 == 1)
goto try_again;

>>>
>>
>
> ___
> Ocfs2-devel mailing list
> Ocfs2-devel@oss.oracle.com
> https://oss.oracle.com/mailman/listinfo/ocfs2-devel
>


___
Ocfs2-devel mailing list
Ocfs2-devel@oss.oracle.com
https://oss.oracle.com/mailman/listinfo/ocfs2-devel


Re: [Ocfs2-devel] [PATCH] ocfs2: fix deadlock on mmapped page in ocfs2_write_begin_nolock()

2016-09-14 Thread Eric Ren

Hi Joseph,

In ocfs2_write_begin_nolock(), we first grab the pages and then
allocate disk space for this write; ocfs2_try_to_free_truncate_log()
will be called if ENOSPC is turned; if we're lucky to get enough clusters,
which is usually the case, we start over again. But in ocfs2_free_write_ctxt()
the target page isn't unlocked, so we will deadlock when trying to grab
the target page again.

IMO, in ocfs2_grab_pages_for_write, mmap_page is mapping to w_pages and
w_target_locked is set to true, and then will be unlocked by
ocfs2_unlock_pages in ocfs2_free_write_ctxt.
So I'm not getting the case "page isn't unlock". Could you please explain
it in more detail?

Thanks for review;-) Follow up the calling chain:

ocfs2_free_write_ctxt()
 ->ocfs2_unlock_pages()

in ocfs2_unlock_pages 
(https://github.com/torvalds/linux/blob/master/fs/ocfs2/aops.c#L793), we

can see the code just put_page(target_page), but not unlock it.

Did this answer your question?

Thanks,
Eric


Yeah, I will think this a bit more like:
why not unlock the target_page there? Is there other potential problems if the "ret" is 
not "-ENOSPC" but

other possible error code?

Thanks,
Eric



Thanks,
Joseph


Fix this issue by unlocking the target page after we fail to allocate
enough space at the first time.

Jan Kara helps me clear out the JBD2 part, and suggest the hint for root cause.

Signed-off-by: Eric Ren 
---
  fs/ocfs2/aops.c | 7 +++
  1 file changed, 7 insertions(+)

diff --git a/fs/ocfs2/aops.c b/fs/ocfs2/aops.c
index 98d3654..78d1d67 100644
--- a/fs/ocfs2/aops.c
+++ b/fs/ocfs2/aops.c
@@ -1860,6 +1860,13 @@ out:
   */
  try_free = 0;
  +/*
+ * Unlock mmap_page because the page has been locked when we
+ * are here.
+ */
+if (mmap_page)
+unlock_page(mmap_page);
+
  ret1 = ocfs2_try_to_free_truncate_log(osb, clusters_need);
  if (ret1 == 1)
  goto try_again;









___
Ocfs2-devel mailing list
Ocfs2-devel@oss.oracle.com
https://oss.oracle.com/mailman/listinfo/ocfs2-devel



___
Ocfs2-devel mailing list
Ocfs2-devel@oss.oracle.com
https://oss.oracle.com/mailman/listinfo/ocfs2-devel

Re: [Ocfs2-devel] [PATCH] ocfs2: fix deadlock on mmapped page in ocfs2_write_begin_nolock()

2016-09-11 Thread Eric Ren
Hi,
>> IMO, in ocfs2_grab_pages_for_write, mmap_page is mapping to w_pages and
>> w_target_locked is set to true, and then will be unlocked by
>> ocfs2_unlock_pages in ocfs2_free_write_ctxt.
>> So I'm not getting the case "page isn't unlock". Could you please explain
>> it in more detail?
> Thanks for review;-) Follow up the calling chain:
>
> ocfs2_free_write_ctxt()
>  ->ocfs2_unlock_pages()
>
> in ocfs2_unlock_pages 
> (https://github.com/torvalds/linux/blob/master/fs/ocfs2/aops.c#L793), we
> can see the code just put_page(target_page), but not unlock it.
>
> Yeah, I will think this a bit more like:
> why not unlock the target_page there? Is there other potential problems if 
> the "ret" is 
> not "-ENOSPC" but
> other possible error code?
1. ocfs2_unlock_pages() will be called in ocfs2_write_end_nolock(), in this 
case, we 
definitely want to return a locked mmaped page
to VM code (do_page_mkwrite) when VM_FAULT_LOCKED is set.

2. But there's indeed a potential existing deadlock situation:
ocfs2_grab_pages_for_write()  ==> return 
-ENOMEM and with 
the mmaped page locked
ocfs2_free_write_ctxt() ==> leave the mmapped 
page locked
  ocfs2_write_begin_nolock() ==> return -ENOMEM
__ocfs2_page_mkwrite() ==> return VM_FAULT_OMM
  __do_page_mkwrite() ==> deadlock here 
(https://github.com/torvalds/linux/blob/master/mm/memory.c#L2054)
This is another corner case, right?

Anyway, I think this patch is good for the -ENOSPC case. And another patch 
should be 
proposed for -ENOMEM case?

Thanks,
Eric

>
> Thanks,
> Eric
>
>>
>> Thanks,
>> Joseph
>>
>>> Fix this issue by unlocking the target page after we fail to allocate
>>> enough space at the first time.
>>>
>>> Jan Kara helps me clear out the JBD2 part, and suggest the hint for root 
>>> cause.
>>>
>>> Signed-off-by: Eric Ren 
>>> ---
>>>   fs/ocfs2/aops.c | 7 +++
>>>   1 file changed, 7 insertions(+)
>>>
>>> diff --git a/fs/ocfs2/aops.c b/fs/ocfs2/aops.c
>>> index 98d3654..78d1d67 100644
>>> --- a/fs/ocfs2/aops.c
>>> +++ b/fs/ocfs2/aops.c
>>> @@ -1860,6 +1860,13 @@ out:
>>>*/
>>>   try_free = 0;
>>>   +/*
>>> + * Unlock mmap_page because the page has been locked when we
>>> + * are here.
>>> + */
>>> +if (mmap_page)
>>> +unlock_page(mmap_page);
>>> +
>>>   ret1 = ocfs2_try_to_free_truncate_log(osb, clusters_need);
>>>   if (ret1 == 1)
>>>   goto try_again;
>>>
>>
>>
>
>


___
Ocfs2-devel mailing list
Ocfs2-devel@oss.oracle.com
https://oss.oracle.com/mailman/listinfo/ocfs2-devel


Re: [Ocfs2-devel] [PATCH] ocfs2: fix deadlock on mmapped page in ocfs2_write_begin_nolock()

2016-09-11 Thread Eric Ren

Hi Joseph,

On 09/12/2016 09:37 AM, Joseph Qi wrote:

Hi Eric,

On 2016/9/10 17:55, Eric Ren wrote:

The testcase "mmaptruncate" of ocfs2-test deadlocked occasionally.

In this testcase, we create a 2*CLUSTER_SIZE file and mmap() on it;
there are 2 process repeatedly performing the following operations
respectively: one is doing memset(mmaped_addr + 2*CLUSTER_SIZE - 1,
'a', 1), while the another is playing ftruncate(fd, 2*CLUSTER_SIZE)
and then ftruncate(fd, CLUSTER_SIZE) again and again.

This is the backtrace when the deadlock happens:
[] __wait_on_bit_lock+0x50/0xa0
[] __lock_page+0xb7/0xc0
[] ? autoremove_wake_function+0x40/0x40
[] ocfs2_write_begin_nolock+0x163f/0x1790 [ocfs2]
[] ? ocfs2_allocate_extend_trans+0x180/0x180 [ocfs2]
[] ocfs2_page_mkwrite+0x1c7/0x2a0 [ocfs2]
[] do_page_mkwrite+0x66/0xc0
[] handle_mm_fault+0x685/0x1350
[] ? __fpu__restore_sig+0x70/0x530
[] __do_page_fault+0x1d8/0x4d0
[] trace_do_page_fault+0x37/0xf0
[] do_async_page_fault+0x19/0x70
[] async_page_fault+0x28/0x30

In ocfs2_write_begin_nolock(), we first grab the pages and then
allocate disk space for this write; ocfs2_try_to_free_truncate_log()
will be called if ENOSPC is turned; if we're lucky to get enough clusters,
which is usually the case, we start over again. But in ocfs2_free_write_ctxt()
the target page isn't unlocked, so we will deadlock when trying to grab
the target page again.

IMO, in ocfs2_grab_pages_for_write, mmap_page is mapping to w_pages and
w_target_locked is set to true, and then will be unlocked by
ocfs2_unlock_pages in ocfs2_free_write_ctxt.
So I'm not getting the case "page isn't unlock". Could you please explain
it in more detail?

Thanks for review;-) Follow up the calling chain:

ocfs2_free_write_ctxt()
 ->ocfs2_unlock_pages()

in ocfs2_unlock_pages 
(https://github.com/torvalds/linux/blob/master/fs/ocfs2/aops.c#L793), we
can see the code just put_page(target_page), but not unlock it.

Yeah, I will think this a bit more like:
why not unlock the target_page there? Is there other potential problems if the "ret" is not 
"-ENOSPC" but

other possible error code?

Thanks,
Eric



Thanks,
Joseph


Fix this issue by unlocking the target page after we fail to allocate
enough space at the first time.

Jan Kara helps me clear out the JBD2 part, and suggest the hint for root cause.

Signed-off-by: Eric Ren 
---
  fs/ocfs2/aops.c | 7 +++
  1 file changed, 7 insertions(+)

diff --git a/fs/ocfs2/aops.c b/fs/ocfs2/aops.c
index 98d3654..78d1d67 100644
--- a/fs/ocfs2/aops.c
+++ b/fs/ocfs2/aops.c
@@ -1860,6 +1860,13 @@ out:
 */
try_free = 0;
  
+		/*

+* Unlock mmap_page because the page has been locked when we
+* are here.
+*/
+   if (mmap_page)
+   unlock_page(mmap_page);
+
ret1 = ocfs2_try_to_free_truncate_log(osb, clusters_need);
if (ret1 == 1)
goto try_again;






___
Ocfs2-devel mailing list
Ocfs2-devel@oss.oracle.com
https://oss.oracle.com/mailman/listinfo/ocfs2-devel

Re: [Ocfs2-devel] [PATCH] ocfs2: fix deadlock on mmapped page in ocfs2_write_begin_nolock()

2016-09-11 Thread Gang He
Reviewed-by: Gang He 

Thanks
Gang


>>> 
> The testcase "mmaptruncate" of ocfs2-test deadlocked occasionally.
> 
> In this testcase, we create a 2*CLUSTER_SIZE file and mmap() on it;
> there are 2 process repeatedly performing the following operations
> respectively: one is doing memset(mmaped_addr + 2*CLUSTER_SIZE - 1,
> 'a', 1), while the another is playing ftruncate(fd, 2*CLUSTER_SIZE)
> and then ftruncate(fd, CLUSTER_SIZE) again and again.
> 
> This is the backtrace when the deadlock happens:
> [] __wait_on_bit_lock+0x50/0xa0
> [] __lock_page+0xb7/0xc0
> [] ? autoremove_wake_function+0x40/0x40
> [] ocfs2_write_begin_nolock+0x163f/0x1790 [ocfs2]
> [] ? ocfs2_allocate_extend_trans+0x180/0x180 [ocfs2]
> [] ocfs2_page_mkwrite+0x1c7/0x2a0 [ocfs2]
> [] do_page_mkwrite+0x66/0xc0
> [] handle_mm_fault+0x685/0x1350
> [] ? __fpu__restore_sig+0x70/0x530
> [] __do_page_fault+0x1d8/0x4d0
> [] trace_do_page_fault+0x37/0xf0
> [] do_async_page_fault+0x19/0x70
> [] async_page_fault+0x28/0x30
> 
> In ocfs2_write_begin_nolock(), we first grab the pages and then
> allocate disk space for this write; ocfs2_try_to_free_truncate_log()
> will be called if ENOSPC is turned; if we're lucky to get enough clusters,
> which is usually the case, we start over again. But in 
> ocfs2_free_write_ctxt()
> the target page isn't unlocked, so we will deadlock when trying to grab
> the target page again.
> 
> Fix this issue by unlocking the target page after we fail to allocate
> enough space at the first time.
> 
> Jan Kara helps me clear out the JBD2 part, and suggest the hint for root 
> cause.
> 
> Signed-off-by: Eric Ren 
> ---
>  fs/ocfs2/aops.c | 7 +++
>  1 file changed, 7 insertions(+)
> 
> diff --git a/fs/ocfs2/aops.c b/fs/ocfs2/aops.c
> index 98d3654..78d1d67 100644
> --- a/fs/ocfs2/aops.c
> +++ b/fs/ocfs2/aops.c
> @@ -1860,6 +1860,13 @@ out:
>*/
>   try_free = 0;
>  
> + /*
> +  * Unlock mmap_page because the page has been locked when we
> +  * are here.
> +  */
> + if (mmap_page)
> + unlock_page(mmap_page);
> +
>   ret1 = ocfs2_try_to_free_truncate_log(osb, clusters_need);
>   if (ret1 == 1)
>   goto try_again;
> -- 
> 2.6.6
> 
> 
> ___
> Ocfs2-devel mailing list
> Ocfs2-devel@oss.oracle.com 
> https://oss.oracle.com/mailman/listinfo/ocfs2-devel


___
Ocfs2-devel mailing list
Ocfs2-devel@oss.oracle.com
https://oss.oracle.com/mailman/listinfo/ocfs2-devel


Re: [Ocfs2-devel] [PATCH] ocfs2: fix deadlock on mmapped page in ocfs2_write_begin_nolock()

2016-09-11 Thread Joseph Qi
Hi Eric,

On 2016/9/10 17:55, Eric Ren wrote:
> The testcase "mmaptruncate" of ocfs2-test deadlocked occasionally.
> 
> In this testcase, we create a 2*CLUSTER_SIZE file and mmap() on it;
> there are 2 process repeatedly performing the following operations
> respectively: one is doing memset(mmaped_addr + 2*CLUSTER_SIZE - 1,
> 'a', 1), while the another is playing ftruncate(fd, 2*CLUSTER_SIZE)
> and then ftruncate(fd, CLUSTER_SIZE) again and again.
> 
> This is the backtrace when the deadlock happens:
> [] __wait_on_bit_lock+0x50/0xa0
> [] __lock_page+0xb7/0xc0
> [] ? autoremove_wake_function+0x40/0x40
> [] ocfs2_write_begin_nolock+0x163f/0x1790 [ocfs2]
> [] ? ocfs2_allocate_extend_trans+0x180/0x180 [ocfs2]
> [] ocfs2_page_mkwrite+0x1c7/0x2a0 [ocfs2]
> [] do_page_mkwrite+0x66/0xc0
> [] handle_mm_fault+0x685/0x1350
> [] ? __fpu__restore_sig+0x70/0x530
> [] __do_page_fault+0x1d8/0x4d0
> [] trace_do_page_fault+0x37/0xf0
> [] do_async_page_fault+0x19/0x70
> [] async_page_fault+0x28/0x30
> 
> In ocfs2_write_begin_nolock(), we first grab the pages and then
> allocate disk space for this write; ocfs2_try_to_free_truncate_log()
> will be called if ENOSPC is turned; if we're lucky to get enough clusters,
> which is usually the case, we start over again. But in ocfs2_free_write_ctxt()
> the target page isn't unlocked, so we will deadlock when trying to grab
> the target page again.
IMO, in ocfs2_grab_pages_for_write, mmap_page is mapping to w_pages and
w_target_locked is set to true, and then will be unlocked by
ocfs2_unlock_pages in ocfs2_free_write_ctxt.
So I'm not getting the case "page isn't unlock". Could you please explain
it in more detail?

Thanks,
Joseph

> 
> Fix this issue by unlocking the target page after we fail to allocate
> enough space at the first time.
> 
> Jan Kara helps me clear out the JBD2 part, and suggest the hint for root 
> cause.
> 
> Signed-off-by: Eric Ren 
> ---
>  fs/ocfs2/aops.c | 7 +++
>  1 file changed, 7 insertions(+)
> 
> diff --git a/fs/ocfs2/aops.c b/fs/ocfs2/aops.c
> index 98d3654..78d1d67 100644
> --- a/fs/ocfs2/aops.c
> +++ b/fs/ocfs2/aops.c
> @@ -1860,6 +1860,13 @@ out:
>*/
>   try_free = 0;
>  
> + /*
> +  * Unlock mmap_page because the page has been locked when we
> +  * are here.
> +  */
> + if (mmap_page)
> + unlock_page(mmap_page);
> +
>   ret1 = ocfs2_try_to_free_truncate_log(osb, clusters_need);
>   if (ret1 == 1)
>   goto try_again;
> 



___
Ocfs2-devel mailing list
Ocfs2-devel@oss.oracle.com
https://oss.oracle.com/mailman/listinfo/ocfs2-devel


[Ocfs2-devel] [PATCH] ocfs2: fix deadlock on mmapped page in ocfs2_write_begin_nolock()

2016-09-10 Thread Eric Ren
The testcase "mmaptruncate" of ocfs2-test deadlocked occasionally.

In this testcase, we create a 2*CLUSTER_SIZE file and mmap() on it;
there are 2 process repeatedly performing the following operations
respectively: one is doing memset(mmaped_addr + 2*CLUSTER_SIZE - 1,
'a', 1), while the another is playing ftruncate(fd, 2*CLUSTER_SIZE)
and then ftruncate(fd, CLUSTER_SIZE) again and again.

This is the backtrace when the deadlock happens:
[] __wait_on_bit_lock+0x50/0xa0
[] __lock_page+0xb7/0xc0
[] ? autoremove_wake_function+0x40/0x40
[] ocfs2_write_begin_nolock+0x163f/0x1790 [ocfs2]
[] ? ocfs2_allocate_extend_trans+0x180/0x180 [ocfs2]
[] ocfs2_page_mkwrite+0x1c7/0x2a0 [ocfs2]
[] do_page_mkwrite+0x66/0xc0
[] handle_mm_fault+0x685/0x1350
[] ? __fpu__restore_sig+0x70/0x530
[] __do_page_fault+0x1d8/0x4d0
[] trace_do_page_fault+0x37/0xf0
[] do_async_page_fault+0x19/0x70
[] async_page_fault+0x28/0x30

In ocfs2_write_begin_nolock(), we first grab the pages and then
allocate disk space for this write; ocfs2_try_to_free_truncate_log()
will be called if ENOSPC is turned; if we're lucky to get enough clusters,
which is usually the case, we start over again. But in ocfs2_free_write_ctxt()
the target page isn't unlocked, so we will deadlock when trying to grab
the target page again.

Fix this issue by unlocking the target page after we fail to allocate
enough space at the first time.

Jan Kara helps me clear out the JBD2 part, and suggest the hint for root cause.

Signed-off-by: Eric Ren 
---
 fs/ocfs2/aops.c | 7 +++
 1 file changed, 7 insertions(+)

diff --git a/fs/ocfs2/aops.c b/fs/ocfs2/aops.c
index 98d3654..78d1d67 100644
--- a/fs/ocfs2/aops.c
+++ b/fs/ocfs2/aops.c
@@ -1860,6 +1860,13 @@ out:
 */
try_free = 0;
 
+   /*
+* Unlock mmap_page because the page has been locked when we
+* are here.
+*/
+   if (mmap_page)
+   unlock_page(mmap_page);
+
ret1 = ocfs2_try_to_free_truncate_log(osb, clusters_need);
if (ret1 == 1)
goto try_again;
-- 
2.6.6


___
Ocfs2-devel mailing list
Ocfs2-devel@oss.oracle.com
https://oss.oracle.com/mailman/listinfo/ocfs2-devel